This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "".

The branch, next has been updated discards 472b66cd232bd0594ac235c3852bfbc04c3f19d7 (commit) discards 1becf47561d1230b5bba747d33b11eff829803de (commit) discards dce6e048d50757a650bd0cf3729b3919c9f6915d (commit) discards 905c2112acc9272cd0ad3965091d44b3f3c47a6d (commit) discards 8bbc5375156a3f0f3f3b0b5d4df47c7513445e68 (commit) discards d30a7b37145fb21281f077c422f7fc5a58d5aaff (commit) discards e7da02eefb0f38bc692492bf96a381310dd742d9 (commit) discards 730bcb8770720a95e41a7f9c916c0d5748096027 (commit) discards b2830822661cad2b4386bcbda9b3fb52d1cad733 (commit) discards 07e911908ce2bbc09c90aab8883ae35d5beccedd (commit) discards 35dd203fe80ea256bc7c0918c5883feb29b2d12e (commit) discards 623da62dbbdc9312deb9e7d0247fa71a82199c76 (commit) discards 5bcf0ca91ffd178b084e4ea3e5ccb45992b500cc (commit) discards e3a7535e29cd81a85c768d44ed80c9aea64cf0a9 (commit) discards 98d32ab8dcd596eb3d561c387bd627e4ac374c66 (commit) discards d506325e770abeb77854dac8e0486c73ee622350 (commit) discards 295eedde0f7c585451f2c4ecd2d610294669e6d9 (commit) discards 6f833db4f07d44f139c49d7ed2b3453efa71c13d (commit) discards e0955b89779956ec54fb246ec655f9f27e9b58ef (commit) discards 366bdf7d5a8eb4f386917af06de4b723e7cea5d0 (commit) discards d15eb7b5f902bb90e5d9ebb1bdb3867be7931057 (commit) discards 59a70e7715ab38522b83385fd7191ee5750c1bec (commit) discards cc70fb42ee97b8c190957642d4651efaa9c46497 (commit) discards 9cf58902e4393f0c8d1394ba3c731ecd0e9bd2f8 (commit) discards 350eed81c7310ccab6d89b1ed7fe9a5f59d0e376 (commit) discards 77a9f37df47411cd993da0279b3735b3d2318a68 (commit) discards 32339365cce880cb2fda74017ee439097cc8ecac (commit) discards 80688692326acc3b00be562832dc923018a57acc (commit) discards ac3756d327ddd32cbff5fff08561314dc1d62255 (commit) discards edd80c5086afd6614953e7aeac6d4eec472027aa (commit) discards 63341917c9d96b05ecb6c0da6713bdd7f0a6f408 (commit) discards a9ddc94935f7a3c1dc1f98f70abf47441a00c111 (commit) discards 8df5da4225642d7f39d52df1707e3b5e3d84822e (commit) discards 648d44b98a1bea63eaac8dadfad649cbc2d687e8 (commit) discards 409b75ee2eb570745ddbc739272c9d931f4a289c (commit) discards fce3590c5e9d40bff64c4d9cefaf49e8ba289ec5 (commit) discards 94d9da5090bbaf01b8266fff67e42d471c1b1e8d (commit) discards 3fee08e5f5f18e1a7d516fba2832efe54be0543a (commit) via f47f91e7e867fb55e5f41df66b0c4ef6fbf3a293 (commit) via c310ba4fe97a50d419274d709d9f57f2659fa3e6 (commit) via 0c33f00fa6ee0e7df4ce1276b160c6e51a74e546 (commit) via 62fdec5f1125eb881081ab5b520e65d71319fb71 (commit) via 0b7865e710caedbe88719c6ceeb79e94aa8a292d (commit) via 38e3b3d6ccf37d172fd7cdf691f0f769a64efb75 (commit) via f31f2fbb7d6cddec6f7f239ddff67cd1b23878a8 (commit) via 363747697803d1b081a425e62a26dce13a788852 (commit) via f6a1813955fbbfd3e710dc949d239ab2ec499a45 (commit) via 0f8ba06581b69c75b0567af14aa86ff7a1d1c20a (commit) via 57096147b4c4577ee2c3ff0b6ab10a8fbdd339b5 (commit) via 87e2afd61019341264dcb2dc7d4593c33fd426aa (commit) via d0771f708492246c8a8d9b10ff550f29196064dc (commit) via f06971cf35f58b6ed30a46446c185bd6a65ba388 (commit) via 7412355450f0a78bd7cfe112a8520869e91e9647 (commit) via 6e57710dc7eb24c3c91a30e6d1f5d9cd3ad9976a (commit) via 7e93b42f889e544bbf63576e36d2f1b313fd7a4e (commit) via 2b994eae58d69d84126bd9db613f630884220d77 (commit) via da5eb63ba1048b0f18e6c436bb655d0649dc1bd7 (commit) via 3c67b97bd06590235287d06d18b662c93aa64842 (commit) via 62164116ae9f072ddeaa3206eaf3b24c8dc9c884 (commit) via fdb1aa2981dbc68c63e11abd2663294338b13764 (commit) via 023d4a942e89f86e66c6d5226d6e792936c37df3 (commit) via e9b7c770c39c2409da68619a8ecc9090acda0c12 (commit) via 3c2cc60893a9b1fe62962df73bfe35b9dbc319c2 (commit) via af005d182ee0eb332cab3fa8f62cdb91135bcaa2 (commit) via 942908c478c41038bbf1d360b664d53b42f7f76e (commit) via 579fac5fead538bfb4f052e5842ebc8176bbf5ea (commit) via 36c59903f1f898567d2a8eafbf5dc0eee93b2a58 (commit) via 516c8bdf27bebf42ddc726b6ce6dd65201ee5bef (commit) via 642c8293c63752aa5f07e2e828cdfcf4ca3c2e62 (commit) via 804c1667832794c43a1a4867b778b31e7a104e9a (commit) via 9e4bca41f1df50b1cf1f324118bb669f9bee0369 (commit) via f27d3d3bdd745a000b58a7ce95d77707b957c903 (commit) via 55c4f518ecd510d2d812e3fc766d6690f65871c3 (commit) via 2deb9c98440a06f0b2e6733693f58d081059dc0d (commit) via cbc97ae9237f1a206d1f4e25ae7e98ad99d430cb (commit)

This update added new revisions after undoing existing revisions. That is to say, the old revision is not a strict subset of the new revision. This situation occurs when you --force push a change and generate a repository containing something like this:

* -- * -- B -- O -- O -- O (472b66cd232bd0594ac235c3852bfbc04c3f19d7) \ N -- N -- N (f47f91e7e867fb55e5f41df66b0c4ef6fbf3a293)

When this happens we assume that you've already had alert emails for all of the O revisions, and so we here report only the revisions in the N branch from the common base, B.

Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below.

- Log ----------------------------------------------------------------- commit f47f91e7e867fb55e5f41df66b0c4ef6fbf3a293 Author: Christophe Milard christophe.milard@linaro.org Date: Thu Nov 24 17:22:30 2016 +0100

test: linux-gen: api: shmem: test sharing memory between ODP instances

The platform tests odp/test/linux-generic/validation/api/shmem are updated to both test ODP<->linux process memory sharing, but also test ODP to ODP (different instances) memory sharing. shmem_linux is the main test process, and shmem_linux.c contains (at file top) a chart flow of the test procedure.

Signed-off-by: Christophe Milard christophe.milard@linaro.org Reviewed-and-tested-by: Bill Fischofer bill.fischofer@linaro.org Signed-off-by: Maxim Uvarov maxim.uvarov@linaro.org

diff --git a/test/linux-generic/validation/api/shmem/.gitignore b/test/linux-generic/validation/api/shmem/.gitignore index 7627079..74195f5 100644 --- a/test/linux-generic/validation/api/shmem/.gitignore +++ b/test/linux-generic/validation/api/shmem/.gitignore @@ -1,2 +1,3 @@ shmem_linux -shmem_odp +shmem_odp1 +shmem_odp2 diff --git a/test/linux-generic/validation/api/shmem/Makefile.am b/test/linux-generic/validation/api/shmem/Makefile.am index 341747f..b0ae627 100644 --- a/test/linux-generic/validation/api/shmem/Makefile.am +++ b/test/linux-generic/validation/api/shmem/Makefile.am @@ -2,19 +2,27 @@ include ../Makefile.inc

#the main test program is shmem_linux, which, in turn, starts a shmem_odp: test_PROGRAMS = shmem_linux$(EXEEXT) -test_extra_PROGRAMS = shmem_odp$(EXEEXT) +test_extra_PROGRAMS = shmem_odp1$(EXEEXT) shmem_odp2$(EXEEXT) test_extradir = $(testdir)

#shmem_linux is stand alone, pure linux (no ODP): dist_shmem_linux_SOURCES = shmem_linux.c shmem_linux_LDFLAGS = $(AM_LDFLAGS) -lrt

-#shmem_odp is the odp part: -dist_shmem_odp_SOURCES = shmem_odp.c -shmem_odp_CFLAGS = $(AM_CFLAGS) \ +#shmem_odp1 and shmem_odp2 are the 2 ODP processes: +dist_shmem_odp1_SOURCES = shmem_odp1.c +shmem_odp1_CFLAGS = $(AM_CFLAGS) \ $(INCCUNIT_COMMON) \ $(INCODP) -shmem_odp_LDFLAGS = $(AM_LDFLAGS) -shmem_odp_LDADD = $(LIBCUNIT_COMMON) $(LIBODP) +shmem_odp1_LDFLAGS = $(AM_LDFLAGS) +shmem_odp1_LDADD = $(LIBCUNIT_COMMON) $(LIBODP)

-noinst_HEADERS = shmem_common.h shmem_linux.h shmem_odp.h +dist_shmem_odp2_SOURCES = shmem_odp2.c +shmem_odp2_CFLAGS = $(AM_CFLAGS) \ + $(INCCUNIT_COMMON) \ + $(INCODP) +shmem_odp2_LDFLAGS = $(AM_LDFLAGS) +shmem_odp2_LDADD = $(LIBCUNIT_COMMON) $(LIBODP) + + +noinst_HEADERS = shmem_common.h shmem_linux.h shmem_odp1.h shmem_odp2.h diff --git a/test/linux-generic/validation/api/shmem/shmem_linux.c b/test/linux-generic/validation/api/shmem/shmem_linux.c index 9ab0e2b..39473f3 100644 --- a/test/linux-generic/validation/api/shmem/shmem_linux.c +++ b/test/linux-generic/validation/api/shmem/shmem_linux.c @@ -5,8 +5,10 @@ */

/* this test makes sure that odp shared memory created with the ODP_SHM_PROC - * flag is visible under linux. It therefore checks both that the device - * name under /dev/shm is correct, and also checks that the memory contents + * flag is visible under linux, and checks that memory created with the + * ODP_SHM_EXPORT flag is visible by other ODP instances. + * It therefore checks both that the link + * name under /tmp is correct, and also checks that the memory contents * is indeed shared. * we want: * -the odp test to run using C UNIT @@ -15,18 +17,47 @@ * * To achieve this, the flow of operations is as follows: * - * linux process (main, non odp) | ODP process - * (shmem_linux.c) | (shmem_odp.c) + * linux process (main, non odp) | + * (shmem_linux.c) | + * | + * | * | * main() | - * forks odp process | allocate shmem - * wait for named pipe creation | populate shmem + * forks odp_app1 process | + * wait for named pipe creation | + * | + * | ODP_APP1 process + * | (shmem_odp1.c) + * | + * | allocate shmem + * | populate shmem * | create named pipe - * read shared memory | wait for test report in fifo + * | wait for test report in fifo... + * read shared memory | * check if memory contents is OK | - * if OK, write "S" in fifo, else "F" | report success or failure to C-Unit - * wait for child terminaison & status| terminate with usual F/S status + * If not OK, write "F" in fifo and | + * exit with failure code. | ------------------- + * | + * forks odp app2 process | ODP APP2 process + * wait for child terminaison & status| (shmem_odp2.c) + * | lookup ODP_APP1 shared memory, + * | check if memory contents is OK + * | Exit(0) on success, exit(1) on fail + * If child failed, write "F" in fifo | + * exit with failure code. | ------------------- + * | + * OK, write "S" in fifo, | + * wait for child terminaison & status| * terminate with same status as child| + * | ODP APP1 process + * | (shmem_odp1.c) + * | + * | ...(continued) + * | read S(success) or F(fail) from fifo + * | report success or failure to C-Unit + * | Exit(0) on success, exit(1) on fail + * wait for child terminaison & status | + * terminate with same status as child | * | * |/ * time @@ -49,7 +80,8 @@ #include "shmem_linux.h" #include "shmem_common.h"

-#define ODP_APP_NAME "shmem_odp" /* name of the odp program, in this dir */ +#define ODP_APP1_NAME "shmem_odp1" /* name of the odp1 program, in this dir */ +#define ODP_APP2_NAME "shmem_odp2" /* name of the odp2 program, in this dir */ #define DEVNAME_FMT "/tmp/odp-%" PRIu64 "-shm-%s" /* odp-<pid>-shm-<name> */ #define MAX_FIFO_WAIT 30 /* Max time waiting for the fifo (sec) */

@@ -117,7 +149,7 @@ void test_success(char *fifo_name, int fd, pid_t odp_app) /* write "Success" to the FIFO */ nb_char = write(fd, &result, sizeof(char)); close(fd); - /* wait for the odp app to terminate */ + /* wait for the odp app1 to terminate */ waitpid(odp_app, &status, 0); /* if the write failed, report an error anyway */ if (nb_char != 1) @@ -134,10 +166,10 @@ void test_failure(char *fifo_name, int fd, pid_t odp_app) int nb_char __attribute__((unused)); /*ignored: we fail anyway */

result = TEST_FAILURE; - /* write "Success" to the FIFO */ + /* write "Failure" to the FIFO */ nb_char = write(fd, &result, sizeof(char)); close(fd); - /* wait for the odp app to terminate */ + /* wait for the odp app1 to terminate */ waitpid(odp_app, &status, 0); unlink(fifo_name); exit(1); /* error */ @@ -146,36 +178,43 @@ void test_failure(char *fifo_name, int fd, pid_t odp_app) int main(int argc __attribute__((unused)), char *argv[]) { char prg_name[PATH_MAX]; - char odp_name[PATH_MAX]; + char odp_name1[PATH_MAX]; + char odp_name2[PATH_MAX]; int nb_sec; - uint64_t size; - pid_t odp_app; - char *odp_params = NULL; + int size; + pid_t odp_app1; + pid_t odp_app2; + char *odp_params1 = NULL; + char *odp_params2[3]; + char pid1[10]; char fifo_name[PATH_MAX]; /* fifo for linux->odp feedback */ int fifo_fd = -1; - char shm_devname[PATH_MAX];/* shared mem device name.*/ + char shm_filename[PATH_MAX];/* shared mem device name, under /dev/shm */ uint64_t len; uint32_t flags; uint32_t align; int shm_fd; test_shared_linux_data_t *addr; + int app2_status;

- /* odp app is in the same directory as this file: */ + /* odp_app1 is in the same directory as this file: */ strncpy(prg_name, argv[0], PATH_MAX - 1); - sprintf(odp_name, "%s/%s", dirname(prg_name), ODP_APP_NAME); + sprintf(odp_name1, "%s/%s", dirname(prg_name), ODP_APP1_NAME);

/* start the ODP application: */ - odp_app = fork(); - if (odp_app < 0) /* error */ + odp_app1 = fork(); + if (odp_app1 < 0) /* error */ exit(1);

- if (odp_app == 0) /* child */ - execv(odp_name, &odp_params); + if (odp_app1 == 0) { /* child */ + execv(odp_name1, &odp_params1); /* no return unless error */ + fprintf(stderr, "execv failed: %s\n", strerror(errno)); + }

/* wait max 30 sec for the fifo to be created by the ODP side. * Just die if time expire as there is no fifo to communicate * through... */ - sprintf(fifo_name, FIFO_NAME_FMT, odp_app); + sprintf(fifo_name, FIFO_NAME_FMT, odp_app1); for (nb_sec = 0; nb_sec < MAX_FIFO_WAIT; nb_sec++) { fifo_fd = open(fifo_name, O_WRONLY); if (fifo_fd >= 0) @@ -191,17 +230,17 @@ int main(int argc __attribute__((unused)), char *argv[]) * check to see if linux can see the created shared memory: */

/* read the shared memory attributes (includes the shm filename): */ - if (read_shmem_attribues(odp_app, ODP_SHM_NAME, - shm_devname, &len, &flags, &align) != 0) - test_failure(fifo_name, fifo_fd, odp_app); + if (read_shmem_attribues(odp_app1, ODP_SHM_NAME, + shm_filename, &len, &flags, &align) != 0) + test_failure(fifo_name, fifo_fd, odp_app1);

/* open the shm filename (which is either on /tmp or on hugetlbfs) * O_CREAT flag not given => failure if shm_devname does not already * exist */ - shm_fd = open(shm_devname, O_RDONLY, + shm_fd = open(shm_filename, O_RDONLY, S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH); if (shm_fd == -1) - test_failure(fifo_name, fifo_fd, odp_app); + test_failure(fifo_name, fifo_fd, odp_app1); /* no return */

/* linux ODP guarantees page size alignement. Larger alignment may * fail as 2 different processes will have fully unrelated @@ -210,12 +249,41 @@ int main(int argc __attribute__((unused)), char *argv[]) size = sizeof(test_shared_linux_data_t);

addr = mmap(NULL, size, PROT_READ, MAP_SHARED, shm_fd, 0); - if (addr == MAP_FAILED) - test_failure(fifo_name, fifo_fd, odp_app); + if (addr == MAP_FAILED) { + printf("shmem_linux: map failed!\n"); + test_failure(fifo_name, fifo_fd, odp_app1); + }

/* check that we see what the ODP application wrote in the memory */ - if ((addr->foo == TEST_SHARE_FOO) && (addr->bar == TEST_SHARE_BAR)) - test_success(fifo_name, fifo_fd, odp_app); - else - test_failure(fifo_name, fifo_fd, odp_app); + if ((addr->foo != TEST_SHARE_FOO) || (addr->bar != TEST_SHARE_BAR)) + test_failure(fifo_name, fifo_fd, odp_app1); /* no return */ + + /* odp_app2 is in the same directory as this file: */ + strncpy(prg_name, argv[0], PATH_MAX - 1); + sprintf(odp_name2, "%s/%s", dirname(prg_name), ODP_APP2_NAME); + + /* start the second ODP application with pid of ODP_APP1 as parameter:*/ + sprintf(pid1, "%d", odp_app1); + odp_params2[0] = odp_name2; + odp_params2[1] = pid1; + odp_params2[2] = NULL; + odp_app2 = fork(); + if (odp_app2 < 0) /* error */ + exit(1); + + if (odp_app2 == 0) { /* child */ + execv(odp_name2, odp_params2); /* no return unless error */ + fprintf(stderr, "execv failed: %s\n", strerror(errno)); + } + + /* wait for the second ODP application to terminate: + * status is OK if that second ODP application could see the + * memory shared by the first one. */ + waitpid(odp_app2, &app2_status, 0); + + if (app2_status) + test_failure(fifo_name, fifo_fd, odp_app1); /* no return */ + + /* everything looked good: */ + test_success(fifo_name, fifo_fd, odp_app1); } diff --git a/test/linux-generic/validation/api/shmem/shmem_odp.c b/test/linux-generic/validation/api/shmem/shmem_odp1.c similarity index 81% rename from test/linux-generic/validation/api/shmem/shmem_odp.c rename to test/linux-generic/validation/api/shmem/shmem_odp1.c index a1f750f..3869c2e 100644 --- a/test/linux-generic/validation/api/shmem/shmem_odp.c +++ b/test/linux-generic/validation/api/shmem/shmem_odp1.c @@ -13,7 +13,7 @@ #include <fcntl.h>

#include <odp_cunit_common.h> -#include "shmem_odp.h" +#include "shmem_odp1.h" #include "shmem_common.h"

#define TEST_SHARE_FOO (0xf0f0f0f0) @@ -27,9 +27,10 @@ void shmem_test_odp_shm_proc(void) test_shared_data_t *test_shared_data; char test_result;

+ /* reminder: ODP_SHM_PROC => export to linux, ODP_SHM_EXPORT=>to odp */ shm = odp_shm_reserve(ODP_SHM_NAME, sizeof(test_shared_data_t), - ALIGN_SIZE, ODP_SHM_PROC); + ALIGN_SIZE, ODP_SHM_PROC | ODP_SHM_EXPORT); CU_ASSERT_FATAL(ODP_SHM_INVALID != shm); test_shared_data = odp_shm_addr(shm); CU_ASSERT_FATAL(NULL != test_shared_data); @@ -39,15 +40,18 @@ void shmem_test_odp_shm_proc(void) odp_mb_full();

/* open the fifo: this will indicate to linux process that it can - * start the shmem lookup and check if it sees the data */ + * start the shmem lookups and check if it sees the data */ sprintf(fifo_name, FIFO_NAME_FMT, getpid()); CU_ASSERT_FATAL(mkfifo(fifo_name, 0666) == 0);

/* read from the fifo: the linux process result: */ + printf("shmem_odp1: opening fifo: %s\n", fifo_name); fd = open(fifo_name, O_RDONLY); CU_ASSERT_FATAL(fd >= 0);

+ printf("shmem_odp1: reading fifo: %s\n", fifo_name); CU_ASSERT(read(fd, &test_result, sizeof(char)) == 1); + printf("shmem_odp1: closing fifo: %s\n", fifo_name); close(fd); CU_ASSERT_FATAL(test_result == TEST_SUCCESS);

diff --git a/test/linux-generic/validation/api/shmem/shmem_odp.h b/test/linux-generic/validation/api/shmem/shmem_odp1.h similarity index 100% copy from test/linux-generic/validation/api/shmem/shmem_odp.h copy to test/linux-generic/validation/api/shmem/shmem_odp1.h diff --git a/test/linux-generic/validation/api/shmem/shmem_odp2.c b/test/linux-generic/validation/api/shmem/shmem_odp2.c new file mode 100644 index 0000000..e39dc76 --- /dev/null +++ b/test/linux-generic/validation/api/shmem/shmem_odp2.c @@ -0,0 +1,95 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#include <odp.h> +#include <linux/limits.h> +#include <sys/types.h> +#include <unistd.h> +#include <stdio.h> +#include <sys/stat.h> +#include <fcntl.h> +#include <stdlib.h> + +#include <odp_cunit_common.h> +#include "shmem_odp2.h" +#include "shmem_common.h" + +#define TEST_SHARE_FOO (0xf0f0f0f0) +#define TEST_SHARE_BAR (0xf0f0f0f) + +/* The C unit test harness is run by ODP1 app which will be told the return + * staus of this process. See top of shmem_linux.c for chart flow of events + */ +int main(int argc, char *argv[]) +{ + odp_instance_t odp1; + odp_instance_t odp2; + odp_shm_t shm; + test_shared_data_t *test_shared_data; + + /* odp init: */ + if (0 != odp_init_global(&odp2, NULL, NULL)) { + fprintf(stderr, "error: odp_init_global() failed.\n"); + return 1; + } + if (0 != odp_init_local(odp2, ODP_THREAD_CONTROL)) { + fprintf(stderr, "error: odp_init_local() failed.\n"); + return 1; + } + + /* test: map ODP1 memory and check its contents: + * The pid of the ODP instantiation process sharing its memory + * is given as first arg. In linux-generic ODP, this pid is actually + * the ODP instance */ + if (argc != 2) { + fprintf(stderr, "One single parameter expected, %d found.\n", + argc); + return 1; + } + odp1 = (odp_instance_t)atoi(argv[1]); + + printf("shmem_odp2: trying to grab %s from pid %d\n", + ODP_SHM_NAME, (int)odp1); + shm = odp_shm_import(ODP_SHM_NAME, odp1, ODP_SHM_NAME); + if (shm == ODP_SHM_INVALID) { + fprintf(stderr, "error: odp_shm_lookup_external failed.\n"); + return 1; + } + + test_shared_data = odp_shm_addr(shm); + if (test_shared_data == NULL) { + fprintf(stderr, "error: odp_shm_addr failed.\n"); + return 1; + } + + if (test_shared_data->foo != TEST_SHARE_FOO) { + fprintf(stderr, "error: Invalid data TEST_SHARE_FOO.\n"); + return 1; + } + + if (test_shared_data->bar != TEST_SHARE_BAR) { + fprintf(stderr, "error: Invalid data TEST_SHARE_BAR.\n"); + return 1; + } + + if (odp_shm_free(shm) != 0) { + fprintf(stderr, "error: odp_shm_free() failed.\n"); + return 1; + } + + /* odp term: */ + if (0 != odp_term_local()) { + fprintf(stderr, "error: odp_term_local() failed.\n"); + return 1; + } + + if (0 != odp_term_global(odp2)) { + fprintf(stderr, "error: odp_term_global() failed.\n"); + return 1; + } + + return 0; +} diff --git a/test/linux-generic/validation/api/shmem/shmem_odp.h b/test/linux-generic/validation/api/shmem/shmem_odp2.h similarity index 76% rename from test/linux-generic/validation/api/shmem/shmem_odp.h rename to test/linux-generic/validation/api/shmem/shmem_odp2.h index 614bbf8..a8db909 100644 --- a/test/linux-generic/validation/api/shmem/shmem_odp.h +++ b/test/linux-generic/validation/api/shmem/shmem_odp2.h @@ -4,4 +4,4 @@ * SPDX-License-Identifier: BSD-3-Clause */

-void shmem_test_odp_shm_proc(void); +int main(int argc, char *argv[]);

commit c310ba4fe97a50d419274d709d9f57f2659fa3e6 Author: Bill Fischofer bill.fischofer@linaro.org Date: Mon Dec 12 09:06:17 2016 -0600

doc: userguide: expand crypto documentation to cover random apis

Clean up the crypto section of the User Guide and expand on the ODP random data APIs.

Signed-off-by: Bill Fischofer bill.fischofer@linaro.org Reviewed-by: Petri Savolainen petri.savolainen@nokia.com Signed-off-by: Maxim Uvarov maxim.uvarov@linaro.org

diff --git a/doc/users-guide/users-guide-crypto.adoc b/doc/users-guide/users-guide-crypto.adoc index 04b3e87..c18e369 100644 --- a/doc/users-guide/users-guide-crypto.adoc +++ b/doc/users-guide/users-guide-crypto.adoc @@ -1,7 +1,8 @@ == Cryptographic services

ODP provides APIs to perform cryptographic operations required by various -communication protocols (e.g. IPSec). ODP cryptographic APIs are session based. +communication protocols (_e.g.,_ IPsec). ODP cryptographic APIs are session +based.

ODP provides APIs for following cryptographic services:

@@ -19,24 +20,26 @@ ODP supports synchronous and asynchronous crypto sessions. For asynchronous sessions, the output of crypto operation is posted in a queue defined as the completion queue in its session parameters.

-ODP crypto APIs support chained operation sessions in which hashing and ciphering -both can be achieved using a single session and operation call. The order of -cipher and hashing can be controlled by the `auth_cipher_text` session parameter. +ODP crypto APIs support chained operation sessions in which hashing and +ciphering both can be achieved using a single session and operation call. The +order of cipher and hashing can be controlled by the `auth_cipher_text` +session parameter.

Other Session parameters include algorithms, keys, initialization vector -(optional), encode or decode, output queue for async mode and output packet pool -for allocation of an output packet if required. +(optional), encode or decode, output queue for async mode and output packet +pool for allocation of an output packet if required.

=== Crypto operations

After session creation, a cryptographic operation can be applied to a packet using the `odp_crypto_operation()` API. Applications may indicate a preference -for synchronous or asynchronous processing in the session's `pref_mode` parameter. -However crypto operations may complete synchronously even if an asynchronous -preference is indicated, and applications must examine the `posted` output -parameter from `odp_crypto_operation()` to determine whether the operation has -completed or if an `ODP_EVENT_CRYPTO_COMPL` notification is expected. In the case -of an async operation, the `posted` output parameter will be set to true. +for synchronous or asynchronous processing in the session's `pref_mode` +parameter. However crypto operations may complete synchronously even if an +asynchronous preference is indicated, and applications must examine the +`posted` output parameter from `odp_crypto_operation()` to determine whether +the operation has completed or if an `ODP_EVENT_CRYPTO_COMPL` notification is +expected. In the case of an async operation, the `posted` output parameter +will be set to true.

The operation arguments specify for each packet the areas that are to be @@ -49,9 +52,9 @@ In case of out-of-place mode output packet is different from input packet as specified by the application, while in new buffer mode implementation allocates a new output buffer from the session’s output pool.

-The application can also specify a context associated with a given operation that -will be retained during async operation and can be retrieved via the completion -event. +The application can also specify a context associated with a given operation +that will be retained during async operation and can be retrieved via the +completion event.

Results of an asynchronous session will be posted as completion events to the session’s completion queue, which can be accessed directly or via the ODP @@ -60,12 +63,60 @@ result. The application has the responsibility to free the completion event.

=== Random number Generation

-ODP provides an API `odp_random_data()` to generate random data bytes. It has -an argument to specify whether to use system entropy source for random number -generation or not. +ODP provides two APIs to generate various kinds of random data bytes. Random +data is characterized by _kind_, which specifies the "quality" of the +randomness required. ODP support three kinds of random data: + +ODP_RANDOM_BASIC:: No specific requirement other than the data appear to be +uniformly distributed. Suitable for load-balancing or other non-cryptographic +use. + +ODP_RANDOM_CRYPTO:: Data suitable for cryptographic use. This is a more +stringent requirement that the data pass tests for statistical randomness. + +ODP_RANDOM_TRUE:: Data generated from a hardware entropy source rather than +any software generated pseudo-random data. May not be available on all +platforms. + +These form a hierarchy with BASIC being the lowest kind of random and TRUE +behing the highest. The main API for accessing random data is: + +[source,c] +----- +int32_t odp_random_data(uint8_t buf, uint32_t len, odp_random_kind_t kind); +----- + +The expectation is that lesser-quality random is easier and faster to generate +while higher-quality random may take more time. Implementations are always free +to substitute a higher kind of random than the one requested if they are able +to do so more efficiently, however calls must return a failure indicator +(rc < 0) if a higher kind of data is requested than the implementation can +provide. This is most likely the case for ODP_RANDOM_TRUE since not all +platforms have access to a true hardware random number generator. + +The `odp_random_max_kind()` API returns the highest kind of random data +available on this implementation. + +For testing purposes it is often desirable to generate repeatable sequences +of "random" data. To address this need ODP provides the additional API: + +[source,c] +----- +int32_t odp_random_test_data(uint8_t buf, uint32_t len, uint64_t *seed); +----- + +This operates the same as `odp_random_data()` except that it always returns +data of kind `ODP_RANDOM_BASIC` and an additional thread-local `seed` +parameter is provide that specifies a seed value to use in generating the +data. This value is updated on each call, so repeated calls with the same +variable will generate a sequence of random data starting from the initial +specified seed. If another sequence of calls is made starting with the same +initial seed value, then `odp_random_test_data()` will return the same +sequence of data bytes.

=== Capability inquiries

-ODP provides an API interface `odp_crypto_capability()` to inquire implementation’s -crypto capabilities. This interface returns a bitmask for supported algorithms -and hardware backed algorithms. +ODP provides the API `odp_crypto_capability()` to inquire the implementation’s +crypto capabilities. This interface returns a the maximum number of crypto +sessions supported as well as bitmasks for supported algorithms and hardware +backed algorithms. \ No newline at end of file

commit 0c33f00fa6ee0e7df4ce1276b160c6e51a74e546 Author: Bill Fischofer bill.fischofer@linaro.org Date: Mon Dec 12 09:06:16 2016 -0600

doc: userguide: move crypto documentation to its own sub-document

Signed-off-by: Bill Fischofer bill.fischofer@linaro.org Reviewed-by: Petri Savolainen petri.savolainen@nokia.com Signed-off-by: Maxim Uvarov maxim.uvarov@linaro.org

diff --git a/doc/users-guide/Makefile.am b/doc/users-guide/Makefile.am index a01c717..01b4df3 100644 --- a/doc/users-guide/Makefile.am +++ b/doc/users-guide/Makefile.am @@ -2,6 +2,7 @@ include ../Makefile.inc

SRC = $(top_srcdir)/doc/users-guide/users-guide.adoc \ $(top_srcdir)/doc/users-guide/users-guide-cls.adoc \ + $(top_srcdir)/doc/users-guide/users-guide-crypto.adoc \ $(top_srcdir)/doc/users-guide/users-guide-packet.adoc \ $(top_srcdir)/doc/users-guide/users-guide-pktio.adoc \ $(top_srcdir)/doc/users-guide/users-guide-timer.adoc \ diff --git a/doc/users-guide/users-guide-crypto.adoc b/doc/users-guide/users-guide-crypto.adoc new file mode 100644 index 0000000..04b3e87 --- /dev/null +++ b/doc/users-guide/users-guide-crypto.adoc @@ -0,0 +1,71 @@ +== Cryptographic services + +ODP provides APIs to perform cryptographic operations required by various +communication protocols (e.g. IPSec). ODP cryptographic APIs are session based. + +ODP provides APIs for following cryptographic services: + +* Ciphering +* Authentication/data integrity via Keyed-Hashing (HMAC) +* Random number generation +* Crypto capability inquiries + +=== Crypto Sessions + +To apply a cryptographic operation to a packet a session must be created. All +packets processed by a session share the parameters that define the session. + +ODP supports synchronous and asynchronous crypto sessions. For asynchronous +sessions, the output of crypto operation is posted in a queue defined as +the completion queue in its session parameters. + +ODP crypto APIs support chained operation sessions in which hashing and ciphering +both can be achieved using a single session and operation call. The order of +cipher and hashing can be controlled by the `auth_cipher_text` session parameter. + +Other Session parameters include algorithms, keys, initialization vector +(optional), encode or decode, output queue for async mode and output packet pool +for allocation of an output packet if required. + +=== Crypto operations + +After session creation, a cryptographic operation can be applied to a packet +using the `odp_crypto_operation()` API. Applications may indicate a preference +for synchronous or asynchronous processing in the session's `pref_mode` parameter. +However crypto operations may complete synchronously even if an asynchronous +preference is indicated, and applications must examine the `posted` output +parameter from `odp_crypto_operation()` to determine whether the operation has +completed or if an `ODP_EVENT_CRYPTO_COMPL` notification is expected. In the case +of an async operation, the `posted` output parameter will be set to true. + + +The operation arguments specify for each packet the areas that are to be +encrypted or decrypted and authenticated. Also, there is an option of overriding +the initialization vector specified in session parameters. + +An operation can be executed in in-place, out-of-place or new buffer mode. +In in-place mode output packet is same as the input packet. +In case of out-of-place mode output packet is different from input packet as +specified by the application, while in new buffer mode implementation allocates +a new output buffer from the session’s output pool. + +The application can also specify a context associated with a given operation that +will be retained during async operation and can be retrieved via the completion +event. + +Results of an asynchronous session will be posted as completion events to the +session’s completion queue, which can be accessed directly or via the ODP +scheduler. The completion event contains the status of the operation and the +result. The application has the responsibility to free the completion event. + +=== Random number Generation + +ODP provides an API `odp_random_data()` to generate random data bytes. It has +an argument to specify whether to use system entropy source for random number +generation or not. + +=== Capability inquiries + +ODP provides an API interface `odp_crypto_capability()` to inquire implementation’s +crypto capabilities. This interface returns a bitmask for supported algorithms +and hardware backed algorithms. diff --git a/doc/users-guide/users-guide.adoc b/doc/users-guide/users-guide.adoc index 9a427fa..41c57d1 100755 --- a/doc/users-guide/users-guide.adoc +++ b/doc/users-guide/users-guide.adoc @@ -1018,77 +1018,7 @@ include::users-guide-pktio.adoc[]

include::users-guide-timer.adoc[]

-== Cryptographic services - -ODP provides APIs to perform cryptographic operations required by various -communication protocols (e.g. IPSec). ODP cryptographic APIs are session based. - -ODP provides APIs for following cryptographic services: - -* Ciphering -* Authentication/data integrity via Keyed-Hashing (HMAC) -* Random number generation -* Crypto capability inquiries - -=== Crypto Sessions - -To apply a cryptographic operation to a packet a session must be created. All -packets processed by a session share the parameters that define the session. - -ODP supports synchronous and asynchronous crypto sessions. For asynchronous -sessions, the output of crypto operation is posted in a queue defined as -the completion queue in its session parameters. - -ODP crypto APIs support chained operation sessions in which hashing and ciphering -both can be achieved using a single session and operation call. The order of -cipher and hashing can be controlled by the `auth_cipher_text` session parameter. - -Other Session parameters include algorithms, keys, initialization vector -(optional), encode or decode, output queue for async mode and output packet pool -for allocation of an output packet if required. - -=== Crypto operations - -After session creation, a cryptographic operation can be applied to a packet -using the `odp_crypto_operation()` API. Applications may indicate a preference -for synchronous or asynchronous processing in the session's `pref_mode` parameter. -However crypto operations may complete synchronously even if an asynchronous -preference is indicated, and applications must examine the `posted` output -parameter from `odp_crypto_operation()` to determine whether the operation has -completed or if an `ODP_EVENT_CRYPTO_COMPL` notification is expected. In the case -of an async operation, the `posted` output parameter will be set to true. - - -The operation arguments specify for each packet the areas that are to be -encrypted or decrypted and authenticated. Also, there is an option of overriding -the initialization vector specified in session parameters. - -An operation can be executed in in-place, out-of-place or new buffer mode. -In in-place mode output packet is same as the input packet. -In case of out-of-place mode output packet is different from input packet as -specified by the application, while in new buffer mode implementation allocates -a new output buffer from the session’s output pool. - -The application can also specify a context associated with a given operation that -will be retained during async operation and can be retrieved via the completion -event. - -Results of an asynchronous session will be posted as completion events to the -session’s completion queue, which can be accessed directly or via the ODP -scheduler. The completion event contains the status of the operation and the -result. The application has the responsibility to free the completion event. - -=== Random number Generation - -ODP provides an API `odp_random_data()` to generate random data bytes. It has -an argument to specify whether to use system entropy source for random number -generation or not. - -=== Capability inquiries - -ODP provides an API interface `odp_crypto_capability()` to inquire implementation’s -crypto capabilities. This interface returns a bitmask for supported algorithms -and hardware backed algorithms. +include::users-guide-crypto.adoc[]

include::users-guide-tm.adoc[]

commit 62fdec5f1125eb881081ab5b520e65d71319fb71 Author: Bill Fischofer bill.fischofer@linaro.org Date: Mon Dec 12 09:06:15 2016 -0600

api: random: add explicit controls over random data

Rework the odp_random_data() API to replace the use_entropy with an explicit odp_random_kind parameter that controls the type of random desired. Two new APIs are also introduced:

- odp_random_max_kind() returns the maximum kind of random data available

- odp_random_test_data() permits applications to generate repeatable random sequences for testing purposes

Signed-off-by: Bill Fischofer bill.fischofer@linaro.org Reviewed-by: Petri Savolainen petri.savolainen@nokia.com Signed-off-by: Maxim Uvarov maxim.uvarov@linaro.org

diff --git a/include/odp/api/spec/random.h b/include/odp/api/spec/random.h index 00fa15b..4765475 100644 --- a/include/odp/api/spec/random.h +++ b/include/odp/api/spec/random.h @@ -24,18 +24,82 @@ extern "C" { */

/** + * Random kind selector + * + * The kind of random denotes the statistical quality of the random data + * returned. Basic random simply appears uniformly distributed, Cryptographic + * random is statistically random and suitable for use by cryptographic + * functions. True random is generated from a hardware entropy source rather + * than an algorithm and is thus completely unpredictable. These form a + * hierarchy where higher quality data is presumably more costly to generate + * than lower quality data. + */ +typedef enum { + /** Basic random, presumably pseudo-random generated by SW. This + * is the lowest kind of random */ + ODP_RANDOM_BASIC, + /** Cryptographic quality random */ + ODP_RANDOM_CRYPTO, + /** True random, generated from a HW entropy source. This is the + * highest kind of random */ + ODP_RANDOM_TRUE, +} odp_random_kind_t; + +/** + * Query random max kind + * + * Implementations support the returned max kind and all kinds weaker than it. + * + * @return kind The maximum odp_random_kind_t supported by this implementation + */ +odp_random_kind_t odp_random_max_kind(void); + +/** * Generate random byte data * + * The intent in supporting different kinds of random data is to allow + * tradeoffs between performance and the quality of random data needed. The + * assumption is that basic random is cheap while true random is relatively + * expensive in terms of time to generate, with cryptographic random being + * something in between. Implementations that support highly efficient true + * random are free to use this for all requested kinds. So it is always + * permissible to "upgrade" a random data request, but never to "downgrade" + * such requests. + * * @param[out] buf Output buffer - * @param size Size of output buffer - * @param use_entropy Use entropy + * @param len Length of output buffer in bytes + * @param kind Specifies the type of random data required. Request + * is expected to fail if the implementation is unable to + * provide the requested type. + * + * @return Number of bytes written + * @retval <0 on failure + */ +int32_t odp_random_data(uint8_t *buf, uint32_t len, odp_random_kind_t kind); + +/** + * Generate repeatable random data for testing purposes + * + * For testing purposes it is often useful to generate "random" sequences that + * are repeatable. This is accomplished by supplying a seed value that is used + * for pseudo-random data generation. The caller-provided seed value is + * updated for each call to continue the sequence. Restarting a series of + * calls with the same initial seed value will generate the same sequence of + * random test data. + * + * This function returns data of ODP_RANDOM_BASIC quality and should be used + * only for testing purposes. Use odp_random_data() for production. * - * @todo Define the implication of the use_entropy parameter + * @param[out] buf Output buffer + * @param len Length of output buffer in bytes + * @param[in,out] seed Seed value to use. This must be a thread-local + * variable. Results are undefined if multiple threads + * call this routine with the same seed variable. * * @return Number of bytes written * @retval <0 on failure */ -int32_t odp_random_data(uint8_t *buf, int32_t size, odp_bool_t use_entropy); +int32_t odp_random_test_data(uint8_t *buf, uint32_t len, uint64_t *seed);

/** * @} diff --git a/platform/linux-generic/odp_crypto.c b/platform/linux-generic/odp_crypto.c index 6b7d60e..5808d16 100644 --- a/platform/linux-generic/odp_crypto.c +++ b/platform/linux-generic/odp_crypto.c @@ -4,6 +4,7 @@ * SPDX-License-Identifier: BSD-3-Clause */

+#include <odp_posix_extensions.h> #include <odp/api/crypto.h> #include <odp_internal.h> #include <odp/api/atomic.h> @@ -19,6 +20,7 @@ #include <odp_packet_internal.h>

#include <string.h> +#include <stdlib.h>

#include <openssl/des.h> #include <openssl/rand.h> @@ -999,12 +1001,48 @@ int odp_crypto_term_global(void) return rc; }

-int32_t -odp_random_data(uint8_t *buf, int32_t len, odp_bool_t use_entropy ODP_UNUSED) +odp_random_kind_t odp_random_max_kind(void) { - int32_t rc; - rc = RAND_bytes(buf, len); - return (1 == rc) ? len /*success*/: -1 /*failure*/; + return ODP_RANDOM_CRYPTO; +} + +int32_t odp_random_data(uint8_t *buf, uint32_t len, odp_random_kind_t kind) +{ + int rc; + + switch (kind) { + case ODP_RANDOM_BASIC: + RAND_pseudo_bytes(buf, len); + return len; + + case ODP_RANDOM_CRYPTO: + rc = RAND_bytes(buf, len); + return (1 == rc) ? (int)len /*success*/: -1 /*failure*/; + + case ODP_RANDOM_TRUE: + default: + return -1; + } +} + +int32_t odp_random_test_data(uint8_t *buf, uint32_t len, uint64_t *seed) +{ + union { + uint32_t rand_word; + uint8_t rand_byte[4]; + } u; + uint32_t i = 0, j; + uint32_t seed32 = (*seed) & 0xffffffff; + + while (i < len) { + u.rand_word = rand_r(&seed32); + + for (j = 0; j < 4 && i < len; j++, i++) + *buf++ = u.rand_byte[j]; + } + + *seed = seed32; + return len; }

odp_crypto_compl_t odp_crypto_compl_from_event(odp_event_t ev) diff --git a/test/common_plat/validation/api/random/random.c b/test/common_plat/validation/api/random/random.c index 7572366..a0e2ef7 100644 --- a/test/common_plat/validation/api/random/random.c +++ b/test/common_plat/validation/api/random/random.c @@ -13,12 +13,58 @@ void random_test_get_size(void) int32_t ret; uint8_t buf[32];

- ret = odp_random_data(buf, sizeof(buf), false); + ret = odp_random_data(buf, sizeof(buf), ODP_RANDOM_BASIC); CU_ASSERT(ret == sizeof(buf)); }

+void random_test_kind(void) +{ + int32_t rc; + uint8_t buf[4096]; + uint32_t buf_size = sizeof(buf); + odp_random_kind_t max_kind = odp_random_max_kind(); + + rc = odp_random_data(buf, buf_size, max_kind); + CU_ASSERT(rc > 0); + + switch (max_kind) { + case ODP_RANDOM_BASIC: + rc = odp_random_data(buf, 4, ODP_RANDOM_CRYPTO); + CU_ASSERT(rc < 0); + /* Fall through */ + + case ODP_RANDOM_CRYPTO: + rc = odp_random_data(buf, 4, ODP_RANDOM_TRUE); + CU_ASSERT(rc < 0); + break; + + default: + break; + } +} + +void random_test_repeat(void) +{ + uint8_t buf1[1024]; + uint8_t buf2[1024]; + int32_t rc; + uint64_t seed1 = 12345897; + uint64_t seed2 = seed1; + + rc = odp_random_test_data(buf1, sizeof(buf1), &seed1); + CU_ASSERT(rc == sizeof(buf1)); + + rc = odp_random_test_data(buf2, sizeof(buf2), &seed2); + CU_ASSERT(rc == sizeof(buf2)); + + CU_ASSERT(seed1 == seed2); + CU_ASSERT(memcmp(buf1, buf2, sizeof(buf1)) == 0); +} + odp_testinfo_t random_suite[] = { ODP_TEST_INFO(random_test_get_size), + ODP_TEST_INFO(random_test_kind), + ODP_TEST_INFO(random_test_repeat), ODP_TEST_INFO_NULL, };

diff --git a/test/common_plat/validation/api/random/random.h b/test/common_plat/validation/api/random/random.h index 26202cc..c4bca78 100644 --- a/test/common_plat/validation/api/random/random.h +++ b/test/common_plat/validation/api/random/random.h @@ -11,6 +11,8 @@

/* test functions: */ void random_test_get_size(void); +void random_test_kind(void); +void random_test_repeat(void);

/* test arrays: */ extern odp_testinfo_t random_suite[];

commit 0b7865e710caedbe88719c6ceeb79e94aa8a292d Author: Christophe Milard christophe.milard@linaro.org Date: Thu Nov 24 17:22:31 2016 +0100

linux-gen: _ishm: cleaning remaining block at odp_term_global

Remaining (forgotten, not freed) blocks are gathered and related files cleaned when odp_term_global() is called. An error message is also issued so the application writters get to know about these blocks

Signed-off-by: Christophe Milard christophe.milard@linaro.org Reviewed-and-tested-by: Bill Fischofer bill.fischofer@linaro.org Signed-off-by: Maxim Uvarov maxim.uvarov@linaro.org

diff --git a/platform/linux-generic/_ishm.c b/platform/linux-generic/_ishm.c index 92575bc..a0188ad 100644 --- a/platform/linux-generic/_ishm.c +++ b/platform/linux-generic/_ishm.c @@ -1507,12 +1507,25 @@ int _odp_ishm_term_local(void) int _odp_ishm_term_global(void) { int ret = 0; + int index; + ishm_block_t *block;

if ((getpid() != odp_global_data.main_pid) || (syscall(SYS_gettid) != getpid())) ODP_ERR("odp_term_global() must be performed by the main " "ODP process!\n.");

+ /* cleanup possibly non freed memory (and complain a bit): */ + for (index = 0; index < ISHM_MAX_NB_BLOCKS; index++) { + block = &ishm_tbl->block[index]; + if (block->len != 0) { + ODP_ERR("block '%s' (file %s) was never freed " + "(cleaning up...).\n", + block->name, block->filename); + delete_file(block); + } + } + /* perform the last thread terminate which was postponed: */ ret = do_odp_ishm_term_local();

commit 38e3b3d6ccf37d172fd7cdf691f0f769a64efb75 Author: Christophe Milard christophe.milard@linaro.org Date: Thu Nov 24 17:22:22 2016 +0100

linux-gen: _ishm: allow memory alloc/free at global init/term

_ishm.c assumed that both _ishm_init_global() and _ishm_init_local() had been run to work properly. This assumption turns out the be a problem if _ishm is to be used as main memory allocator, as many modules init_global functions assume the availability of the odp_reserve() function before any init_local function is called. Likewise, many term_global() functions assume the availability of the odp_shm_free() function after all odp_term_local() have run. This patch runs _ishm_init_local() in advance for the main ODP thread and postpones the execution of_ishm_term_local() to init_global() time for the main process, hence making the ishm_reserve() and ishm_free() functions available at init_global/term_global time.

Signed-off-by: Christophe Milard christophe.milard@linaro.org Reviewed-and-tested-by: Bill Fischofer bill.fischofer@linaro.org Signed-off-by: Maxim Uvarov maxim.uvarov@linaro.org

diff --git a/platform/linux-generic/_ishm.c b/platform/linux-generic/_ishm.c index 33229e8..92575bc 100644 --- a/platform/linux-generic/_ishm.c +++ b/platform/linux-generic/_ishm.c @@ -157,7 +157,6 @@ typedef struct ishm_block { char name[ISHM_NAME_MAXLEN]; /* name for the ishm block (if any) */ char filename[ISHM_FILENAME_MAXLEN]; /* name of the .../odp-* file */ char exptname[ISHM_FILENAME_MAXLEN]; /* name of the export file */ - int main_odpthread; /* The thread which did the initial reserve*/ uint32_t user_flags; /* any flags the user want to remember. */ uint32_t flags; /* block creation flags. */ uint64_t user_len; /* length, as requested at reserve time. */ @@ -179,6 +178,7 @@ typedef struct ishm_block { typedef struct { odp_spinlock_t lock; uint64_t dev_seq; /* used when creating device names */ + uint32_t odpthread_cnt; /* number of running ODP threads */ ishm_block_t block[ISHM_MAX_NB_BLOCKS]; } ishm_table_t; static ishm_table_t *ishm_tbl; @@ -879,7 +879,6 @@ int _odp_ishm_reserve(const char *name, uint64_t size, int fd, new_block->user_flags = user_flags; new_block->seq++; new_block->refcnt = 1; - new_block->main_odpthread = odp_thread_id(); new_block->start = addr; /* only for SINGLE_VA*/

/* the allocation succeeded: update the process local view */ @@ -999,10 +998,8 @@ static int block_free(int block_index)

proc_index = procfind_block(block_index); if (proc_index >= 0) { - /* close the fd, unless if it was externaly provided */ - if ((block->filename[0] != 0) || - (odp_thread_id() != block->main_odpthread)) - close(ishm_proctable->entry[proc_index].fd); + /* close the related fd */ + close(ishm_proctable->entry[proc_index].fd);

/* remove the mapping and possible fragment */ do_unmap(ishm_proctable->entry[proc_index].start, @@ -1293,12 +1290,62 @@ int _odp_ishm_info(int block_index, _odp_ishm_info_t *info) return 0; }

+static int do_odp_ishm_init_local(void) +{ + int i; + int block_index; + + /* + * the ishm_process table is local to each linux process + * Check that no other linux threads (of same or ancestor processes) + * have already created the table, and create it if needed. + * We protect this with the general ishm lock to avoid + * init race condition of different running threads. + */ + odp_spinlock_lock(&ishm_tbl->lock); + ishm_tbl->odpthread_cnt++; /* count ODPthread (pthread or process) */ + if (!ishm_proctable) { + ishm_proctable = malloc(sizeof(ishm_proctable_t)); + if (!ishm_proctable) { + odp_spinlock_unlock(&ishm_tbl->lock); + return -1; + } + memset(ishm_proctable, 0, sizeof(ishm_proctable_t)); + } + if (syscall(SYS_gettid) != getpid()) + ishm_proctable->thrd_refcnt++; /* new linux thread */ + else + ishm_proctable->thrd_refcnt = 1;/* new linux process */ + + /* + * if this ODP thread is actually a new linux process, (as opposed + * to a pthread), i.e, we just forked, then all shmem blocks + * of the parent process are mapped into this child by inheritance. + * (The process local table is inherited as well). We hence have to + * increase the process refcount for each of the inherited mappings: + */ + if (syscall(SYS_gettid) == getpid()) { + for (i = 0; i < ishm_proctable->nb_entries; i++) { + block_index = ishm_proctable->entry[i].block_index; + ishm_tbl->block[block_index].refcnt++; + } + } + + odp_spinlock_unlock(&ishm_tbl->lock); + return 0; +} + int _odp_ishm_init_global(void) { void *addr; void *spce_addr; int i;

+ if ((getpid() != odp_global_data.main_pid) || + (syscall(SYS_gettid) != getpid())) + ODP_ERR("odp_init_global() must be performed by the main " + "ODP process!\n."); + if (!odp_global_data.hugepage_info.default_huge_page_dir) ODP_DBG("NOTE: No support for huge pages\n"); else @@ -1315,6 +1362,7 @@ int _odp_ishm_init_global(void) ishm_tbl = addr; memset(ishm_tbl, 0, sizeof(ishm_table_t)); ishm_tbl->dev_seq = 0; + ishm_tbl->odpthread_cnt = 0; odp_spinlock_init(&ishm_tbl->lock);

/* allocate space for the internal shared mem fragment table: */ @@ -1355,7 +1403,13 @@ int _odp_ishm_init_global(void) ishm_ftbl->fragment[ISHM_NB_FRAGMNTS - 1].next = NULL; ishm_ftbl->unused_fragmnts = &ishm_ftbl->fragment[1];

- return 0; + /* + * We run _odp_ishm_init_local() directely here to give the + * possibility to run shm_reserve() before the odp_init_local() + * is performed for the main thread... Many init_global() functions + * indeed assume the availability of odp_shm_reserve()...: + */ + return do_odp_ishm_init_local();

init_glob_err3: if (munmap(ishm_ftbl, sizeof(ishm_ftable_t)) < 0) @@ -1369,80 +1423,28 @@ init_glob_err1:

int _odp_ishm_init_local(void) { - int i; - int block_index; - /* - * the ishm_process table is local to each linux process - * Check that no other linux threads (of same or ancestor processes) - * have already created the table, and create it if needed. - * We protect this with the general ishm lock to avoid - * init race condition of different running threads. + * Do not re-run this for the main ODP process, as it has already + * been done in advance at _odp_ishm_init_global() time: */ - odp_spinlock_lock(&ishm_tbl->lock); - if (!ishm_proctable) { - ishm_proctable = malloc(sizeof(ishm_proctable_t)); - if (!ishm_proctable) { - odp_spinlock_unlock(&ishm_tbl->lock); - return -1; - } - memset(ishm_proctable, 0, sizeof(ishm_proctable_t)); - } - if (syscall(SYS_gettid) != getpid()) - ishm_proctable->thrd_refcnt++; /* new linux thread */ - else - ishm_proctable->thrd_refcnt = 1;/* new linux process */ + if ((getpid() == odp_global_data.main_pid) && + (syscall(SYS_gettid) == getpid())) + return 0;

- /* - * if this ODP thread is actually a new linux process, (as opposed - * to a pthread), i.e, we just forked, then all shmem blocks - * of the parent process are mapped into this child by inheritance. - * (The process local table is inherited as well). We hence have to - * increase the process refcount for each of the inherited mappings: - */ - if (syscall(SYS_gettid) == getpid()) { - for (i = 0; i < ishm_proctable->nb_entries; i++) { - block_index = ishm_proctable->entry[i].block_index; - ishm_tbl->block[block_index].refcnt++; - } - } - - odp_spinlock_unlock(&ishm_tbl->lock); - return 0; + return do_odp_ishm_init_local(); }

-int _odp_ishm_term_global(void) -{ - int ret = 0; - - /* free the fragment table */ - if (munmap(ishm_ftbl, sizeof(ishm_ftable_t)) < 0) { - ret = -1; - ODP_ERR("unable to munmap fragment table\n."); - } - /* free the block table */ - if (munmap(ishm_tbl, sizeof(ishm_table_t)) < 0) { - ret = -1; - ODP_ERR("unable to munmap main table\n."); - } - - /* free the reserved VA space */ - if (_odp_ishmphy_unbook_va()) - ret = -1; - - return ret; -} - -int _odp_ishm_term_local(void) +static int do_odp_ishm_term_local(void) { int i; int proc_table_refcnt = 0; int block_index; ishm_block_t *block;

- odp_spinlock_lock(&ishm_tbl->lock); procsync();

+ ishm_tbl->odpthread_cnt--; /* decount ODPthread (pthread or process) */ + /* * The ishm_process table is local to each linux process * Check that no other linux threads (of this linux process) @@ -1482,10 +1484,56 @@ int _odp_ishm_term_local(void) ishm_proctable = NULL; }

- odp_spinlock_unlock(&ishm_tbl->lock); return 0; }

+int _odp_ishm_term_local(void) +{ + int ret; + + odp_spinlock_lock(&ishm_tbl->lock); + + /* postpone last thread term to allow free() by global term functions:*/ + if (ishm_tbl->odpthread_cnt == 1) { + odp_spinlock_unlock(&ishm_tbl->lock); + return 0; + } + + ret = do_odp_ishm_term_local(); + odp_spinlock_unlock(&ishm_tbl->lock); + return ret; +} + +int _odp_ishm_term_global(void) +{ + int ret = 0; + + if ((getpid() != odp_global_data.main_pid) || + (syscall(SYS_gettid) != getpid())) + ODP_ERR("odp_term_global() must be performed by the main " + "ODP process!\n."); + + /* perform the last thread terminate which was postponed: */ + ret = do_odp_ishm_term_local(); + + /* free the fragment table */ + if (munmap(ishm_ftbl, sizeof(ishm_ftable_t)) < 0) { + ret |= -1; + ODP_ERR("unable to munmap fragment table\n."); + } + /* free the block table */ + if (munmap(ishm_tbl, sizeof(ishm_table_t)) < 0) { + ret |= -1; + ODP_ERR("unable to munmap main table\n."); + } + + /* free the reserved VA space */ + if (_odp_ishmphy_unbook_va()) + ret |= -1; + + return ret; +} + /* * Print the current ishm status (allocated blocks and VA space map) * Return the number of allocated blocks (including those not mapped @@ -1541,13 +1589,12 @@ int _odp_ishm_status(const char *title) huge = '?'; } proc_index = procfind_block(i); - ODP_DBG("%-3d: name:%-.24s file:%-.24s tid:%-3d" + ODP_DBG("%-3d: name:%-.24s file:%-.24s" " flags:%s,%c len:0x%-08lx" " user_len:%-8ld seq:%-3ld refcnt:%-4d\n", i, ishm_tbl->block[i].name, ishm_tbl->block[i].filename, - ishm_tbl->block[i].main_odpthread, flags, huge, ishm_tbl->block[i].len, ishm_tbl->block[i].user_len,

commit f31f2fbb7d6cddec6f7f239ddff67cd1b23878a8 Author: Christophe Milard christophe.milard@linaro.org Date: Thu Nov 24 17:22:19 2016 +0100

linux-gen: _ishm: fix for alignment request matching page size

There is no reason to toggle the _ODP_ISHM_SINGLE_VA flag when the alignment exactly matches the page size. This just results in wasting the odp-common virtual space

Signed-off-by: Christophe Milard christophe.milard@linaro.org Reviewed-and-tested-by: Bill Fischofer bill.fischofer@linaro.org Signed-off-by: Maxim Uvarov maxim.uvarov@linaro.org

diff --git a/platform/linux-generic/_ishm.c b/platform/linux-generic/_ishm.c index 40bfb96..33229e8 100644 --- a/platform/linux-generic/_ishm.c +++ b/platform/linux-generic/_ishm.c @@ -824,7 +824,7 @@ int _odp_ishm_reserve(const char *name, uint64_t size, int fd, * the same address every where, otherwise alignment may be * be wrong for some process */ hp_align = align; - if (hp_align < odp_sys_huge_page_size()) + if (hp_align <= odp_sys_huge_page_size()) hp_align = odp_sys_huge_page_size(); else flags |= _ODP_ISHM_SINGLE_VA; @@ -852,7 +852,7 @@ int _odp_ishm_reserve(const char *name, uint64_t size, int fd, * size then we have to make sure the block will be mapped at * the same address every where, otherwise alignment may be * be wrong for some process */ - if (align < odp_sys_page_size()) + if (align <= odp_sys_page_size()) align = odp_sys_page_size(); else flags |= _ODP_ISHM_SINGLE_VA;

commit 363747697803d1b081a425e62a26dce13a788852 Author: Christophe Milard christophe.milard@linaro.org Date: Thu Nov 24 17:22:32 2016 +0100

linux_gen: _ishm: decreasing the number of error messages when no huge pages

Signed-off-by: Christophe Milard christophe.milard@linaro.org Reviewed-and-tested-by: Bill Fischofer bill.fischofer@linaro.org Signed-off-by: Maxim Uvarov maxim.uvarov@linaro.org

diff --git a/platform/linux-generic/_ishm.c b/platform/linux-generic/_ishm.c index 82478e5..40bfb96 100644 --- a/platform/linux-generic/_ishm.c +++ b/platform/linux-generic/_ishm.c @@ -439,8 +439,12 @@ static int create_file(int block_index, huge_flag_t huge, uint64_t len,

fd = open(filename, oflag, S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH); if (fd < 0) { - ODP_ERR("open failed for %s: %s.\n", - filename, strerror(errno)); + if (huge == HUGE) + ODP_DBG("open failed for %s: %s.\n", + filename, strerror(errno)); + else + ODP_ERR("open failed for %s: %s.\n", + filename, strerror(errno)); return -1; }

@@ -762,6 +766,7 @@ int _odp_ishm_reserve(const char *name, uint64_t size, int fd, void *addr = NULL; /* mapping address */ int new_proc_entry; struct stat statbuf; + static int huge_error_printed; /* to avoid millions of error...*/

odp_spinlock_lock(&ishm_tbl->lock);

@@ -828,11 +833,16 @@ int _odp_ishm_reserve(const char *name, uint64_t size, int fd, len = (size + (page_hp_size - 1)) & (-page_hp_size); addr = do_map(new_index, len, hp_align, flags, HUGE, &fd);

- if (addr == NULL) - ODP_DBG("No huge pages, fall back to normal pages, " - "check: /proc/sys/vm/nr_hugepages.\n"); - else + if (addr == NULL) { + if (!huge_error_printed) { + ODP_ERR("No huge pages, fall back to normal " + "pages. " + "check: /proc/sys/vm/nr_hugepages.\n"); + huge_error_printed = 1; + } + } else { new_block->huge = HUGE; + } }

/* Try normal pages if huge pages failed */

commit f6a1813955fbbfd3e710dc949d239ab2ec499a45 Author: Christophe Milard christophe.milard@linaro.org Date: Tue Nov 8 10:49:28 2016 +0100

linux-gen: _ishm: accept multiple usage of same block name

This is following the request that using the same name for multiple memory blocks should be allowed on north API. The change made here will affect any _ishm users (i.e. both north and south API), which is probably better for consistency.

Signed-off-by: Christophe Milard christophe.milard@linaro.org Reviewed-by: Bill Fischofer bill.fischofer@linaro.org Signed-off-by: Maxim Uvarov maxim.uvarov@linaro.org

diff --git a/platform/linux-generic/_ishm.c b/platform/linux-generic/_ishm.c index 88282ae..82478e5 100644 --- a/platform/linux-generic/_ishm.c +++ b/platform/linux-generic/_ishm.c @@ -772,14 +772,6 @@ int _odp_ishm_reserve(const char *name, uint64_t size, int fd, page_sz = odp_sys_page_size(); page_hp_size = odp_sys_huge_page_size();

- /* check if name already exists */ - if (name && (find_block_by_name(name) >= 0)) { - /* Found a block with the same name */ - odp_spinlock_unlock(&ishm_tbl->lock); - ODP_ERR("name "%s" already used.\n", name); - return -1; - } - /* grab a new entry: */ for (new_index = 0; new_index < ISHM_MAX_NB_BLOCKS; new_index++) { if (ishm_tbl->block[new_index].len == 0) {

commit 0f8ba06581b69c75b0567af14aa86ff7a1d1c20a Author: Christophe Milard christophe.milard@linaro.org Date: Mon Dec 5 19:26:55 2016 +0100

linux-gen: shared_memory: remove flag forcing mlock

The _ishm flag _ODP_ISHM_LOCK is no longer set when doing shm_reserve(), hence enabling non-root user to exceed the 64MB mlock memory limit (ulimit).

Signed-off-by: Christophe Milard christophe.milard@linaro.org Signed-off-by: Maxim Uvarov maxim.uvarov@linaro.org

diff --git a/platform/linux-generic/odp_shared_memory.c b/platform/linux-generic/odp_shared_memory.c index d2bb74c..ba32dee 100644 --- a/platform/linux-generic/odp_shared_memory.c +++ b/platform/linux-generic/odp_shared_memory.c @@ -58,9 +58,6 @@ odp_shm_t odp_shm_reserve(const char *name, uint64_t size, uint64_t align,

flgs = get_ishm_flags(flags);

- /* all mem reserved through this interface is requested to be locked: */ - flgs |= (flags & _ODP_ISHM_LOCK); - block_index = _odp_ishm_reserve(name, size, -1, align, flgs, flags); if (block_index >= 0) return to_handle(block_index);

commit 57096147b4c4577ee2c3ff0b6ab10a8fbdd339b5 Author: Christophe Milard christophe.milard@linaro.org Date: Thu Nov 24 17:22:29 2016 +0100

linux-gen: shm: add flag and function to share memory between ODP instances

Implemented by calling the related functions from _ishm.

Signed-off-by: Christophe Milard christophe.milard@linaro.org Reviewed-and-tested-by: Bill Fischofer bill.fischofer@linaro.org Signed-off-by: Maxim Uvarov maxim.uvarov@linaro.org

diff --git a/platform/linux-generic/odp_shared_memory.c b/platform/linux-generic/odp_shared_memory.c index 2377f16..d2bb74c 100644 --- a/platform/linux-generic/odp_shared_memory.c +++ b/platform/linux-generic/odp_shared_memory.c @@ -24,6 +24,21 @@ static inline odp_shm_t to_handle(uint32_t index) return _odp_cast_scalar(odp_shm_t, index + 1); }

+static uint32_t get_ishm_flags(uint32_t flags) +{ + uint32_t f = 0; /* internal ishm flags */ + + /* set internal ishm flags according to API flags: + * note that both ODP_SHM_PROC and ODP_SHM_EXPORT maps to + * _ODP_ISHM_LINK as in the linux-gen implementation there is + * no difference between exporting to another ODP instance or + * another linux process */ + f |= (flags & (ODP_SHM_PROC | ODP_SHM_EXPORT)) ? _ODP_ISHM_EXPORT : 0; + f |= (flags & ODP_SHM_SINGLE_VA) ? _ODP_ISHM_SINGLE_VA : 0; + + return f; +} + int odp_shm_capability(odp_shm_capability_t *capa) { memset(capa, 0, sizeof(odp_shm_capability_t)); @@ -41,9 +56,7 @@ odp_shm_t odp_shm_reserve(const char *name, uint64_t size, uint64_t align, int block_index; int flgs = 0; /* internal ishm flags */

- /* set internal ishm flags according to API flags: */ - flgs |= (flags & ODP_SHM_PROC) ? _ODP_ISHM_EXPORT : 0; - flgs |= (flags & ODP_SHM_SINGLE_VA) ? _ODP_ISHM_SINGLE_VA : 0; + flgs = get_ishm_flags(flags);

/* all mem reserved through this interface is requested to be locked: */ flgs |= (flags & _ODP_ISHM_LOCK); @@ -55,6 +68,18 @@ odp_shm_t odp_shm_reserve(const char *name, uint64_t size, uint64_t align, return ODP_SHM_INVALID; }

+odp_shm_t odp_shm_import(const char *remote_name, + odp_instance_t odp_inst, + const char *local_name) +{ + int ret; + + ret = _odp_ishm_find_exported(remote_name, (pid_t)odp_inst, + local_name); + + return to_handle(ret); +} + int odp_shm_free(odp_shm_t shm) { return _odp_ishm_free_by_index(from_handle(shm));

commit 87e2afd61019341264dcb2dc7d4593c33fd426aa Author: Christophe Milard christophe.milard@linaro.org Date: Thu Nov 24 17:22:26 2016 +0100

linux-gen: shm: new ODP_SHM_SINGLE_VA flag implementation

This flag guarentess the unicity the the block address on all ODP threads. The patch just exposes the _ODP_ISHM_SINGLE_VA flag of the internal memory allocator, ishm.

Signed-off-by: Christophe Milard christophe.milard@linaro.org Reviewed-and-tested-by: Bill Fischofer bill.fischofer@linaro.org Signed-off-by: Maxim Uvarov maxim.uvarov@linaro.org

diff --git a/platform/linux-generic/odp_shared_memory.c b/platform/linux-generic/odp_shared_memory.c index 9e916e9..2377f16 100644 --- a/platform/linux-generic/odp_shared_memory.c +++ b/platform/linux-generic/odp_shared_memory.c @@ -43,6 +43,7 @@ odp_shm_t odp_shm_reserve(const char *name, uint64_t size, uint64_t align,

/* set internal ishm flags according to API flags: */ flgs |= (flags & ODP_SHM_PROC) ? _ODP_ISHM_EXPORT : 0; + flgs |= (flags & ODP_SHM_SINGLE_VA) ? _ODP_ISHM_SINGLE_VA : 0;

/* all mem reserved through this interface is requested to be locked: */ flgs |= (flags & _ODP_ISHM_LOCK);

commit d0771f708492246c8a8d9b10ff550f29196064dc Author: Christophe Milard christophe.milard@linaro.org Date: Thu Nov 24 17:22:28 2016 +0100

linux-gen: _ishm: adding function to map memory from other ODP

functionality to export and map memory between ODP instance is added: This includes: - a bit of simplification in _odp_ishm_reserve() for externaly provided file descriptors. - a new function, _odp_ishm_find_exported() to map memory from other ODP instances (On same OS)

Signed-off-by: Christophe Milard christophe.milard@linaro.org Reviewed-and-tested-by: Bill Fischofer bill.fischofer@linaro.org Signed-off-by: Maxim Uvarov maxim.uvarov@linaro.org

diff --git a/platform/linux-generic/_ishm.c b/platform/linux-generic/_ishm.c index 37e56d4..88282ae 100644 --- a/platform/linux-generic/_ishm.c +++ b/platform/linux-generic/_ishm.c @@ -152,6 +152,7 @@ typedef struct ishm_fragment { * will allocate both a block and a fragment. * Blocks contain only global data common to all processes. */ +typedef enum {UNKNOWN, HUGE, NORMAL, EXTERNAL} huge_flag_t; typedef struct ishm_block { char name[ISHM_NAME_MAXLEN]; /* name for the ishm block (if any) */ char filename[ISHM_FILENAME_MAXLEN]; /* name of the .../odp-* file */ @@ -163,7 +164,7 @@ typedef struct ishm_block { void *start; /* only valid if _ODP_ISHM_SINGLE_VA is set*/ uint64_t len; /* length. multiple of page size. 0 if free*/ ishm_fragment_t *fragment; /* used when _ODP_ISHM_SINGLE_VA is used */ - int huge; /* true if this segment is mapped using huge pages */ + huge_flag_t huge; /* page type: external means unknown here. */ uint64_t seq; /* sequence number, incremented on alloc and free */ uint64_t refcnt;/* number of linux processes mapping this block */ } ishm_block_t; @@ -400,7 +401,7 @@ static void free_fragment(ishm_fragment_t *fragmnt) * or /mnt/huge/odp-<pid>-<sequence_or_name> (for huge pages) * Return the new file descriptor, or -1 on error. */ -static int create_file(int block_index, int huge, uint64_t len, +static int create_file(int block_index, huge_flag_t huge, uint64_t len, uint32_t flags, uint32_t align) { char *name; @@ -419,10 +420,11 @@ static int create_file(int block_index, int huge, uint64_t len, ishm_tbl->dev_seq++);

/* huge dir must be known to create files there!: */ - if (huge && !odp_global_data.hugepage_info.default_huge_page_dir) + if ((huge == HUGE) && + (!odp_global_data.hugepage_info.default_huge_page_dir)) return -1;

- if (huge) + if (huge == HUGE) snprintf(filename, ISHM_FILENAME_MAXLEN, ISHM_FILENAME_FORMAT, odp_global_data.hugepage_info.default_huge_page_dir, @@ -502,7 +504,7 @@ static void delete_file(ishm_block_t *block) * Mutex must be assured by the caller. */ static void *do_map(int block_index, uint64_t len, uint32_t align, - uint32_t flags, int huge, int *fd) + uint32_t flags, huge_flag_t huge, int *fd) { ishm_block_t *new_block; /* entry in the main block table */ void *addr = NULL; @@ -552,8 +554,6 @@ static void *do_map(int block_index, uint64_t len, uint32_t align, return NULL; }

- new_block->huge = huge; - return mapped_addr; }

@@ -756,27 +756,21 @@ int _odp_ishm_reserve(const char *name, uint64_t size, int fd, int new_index; /* index in the main block table*/ ishm_block_t *new_block; /* entry in the main block table*/ uint64_t page_sz; /* normal page size. usually 4K*/ - uint64_t alloc_size; /* includes extra for alignement*/ uint64_t page_hp_size; /* huge page size */ - uint64_t alloc_hp_size; /* includes extra for alignement*/ uint32_t hp_align; uint64_t len; /* mapped length */ void *addr = NULL; /* mapping address */ int new_proc_entry; - - page_sz = odp_sys_page_size(); + struct stat statbuf;

odp_spinlock_lock(&ishm_tbl->lock);

/* update this process view... */ procsync();

- /* roundup to page size */ - alloc_size = (size + (page_sz - 1)) & (-page_sz); - + /* Get system page sizes: page_hp_size is 0 if no huge page available*/ + page_sz = odp_sys_page_size(); page_hp_size = odp_sys_huge_page_size(); - /* roundup to page size */ - alloc_hp_size = (size + (page_hp_size - 1)) & (-page_hp_size);

/* check if name already exists */ if (name && (find_block_by_name(name) >= 0)) { @@ -809,8 +803,24 @@ int _odp_ishm_reserve(const char *name, uint64_t size, int fd, else new_block->name[0] = 0;

- /* Try first huge pages when possible and needed: */ - if (page_hp_size && (alloc_size > page_sz)) { + /* If a file descriptor is provided, get the real size and map: */ + if (fd >= 0) { + fstat(fd, &statbuf); + len = statbuf.st_size; + /* note that the huge page flag is meningless here as huge + * page is determined by the provided file descriptor: */ + addr = do_map(new_index, len, align, flags, EXTERNAL, &fd); + if (addr == NULL) { + close(fd); + odp_spinlock_unlock(&ishm_tbl->lock); + ODP_ERR("_ishm_reserve failed.\n"); + return -1; + } + new_block->huge = EXTERNAL; + } + + /* Otherwise, Try first huge pages when possible and needed: */ + if ((fd < 0) && page_hp_size && (size > page_sz)) { /* at least, alignment in VA should match page size, but user * can request more: If the user requirement exceeds the page * size then we have to make sure the block will be mapped at @@ -821,18 +831,20 @@ int _odp_ishm_reserve(const char *name, uint64_t size, int fd, hp_align = odp_sys_huge_page_size(); else flags |= _ODP_ISHM_SINGLE_VA; - len = alloc_hp_size; - addr = do_map(new_index, len, hp_align, flags, 1, &fd); + + /* roundup to page size */ + len = (size + (page_hp_size - 1)) & (-page_hp_size); + addr = do_map(new_index, len, hp_align, flags, HUGE, &fd);

if (addr == NULL) ODP_DBG("No huge pages, fall back to normal pages, " "check: /proc/sys/vm/nr_hugepages.\n"); else - new_block->huge = 1; + new_block->huge = HUGE; }

- /* try normal pages if huge pages failed */ - if (addr == NULL) { + /* Try normal pages if huge pages failed */ + if (fd < 0) { /* at least, alignment in VA should match page size, but user * can request more: If the user requirement exceeds the page * size then we have to make sure the block will be mapped at @@ -843,13 +855,14 @@ int _odp_ishm_reserve(const char *name, uint64_t size, int fd, else flags |= _ODP_ISHM_SINGLE_VA;

- len = alloc_size; - addr = do_map(new_index, len, align, flags, 0, &fd); - new_block->huge = 0; + /* roundup to page size */ + len = (size + (page_sz - 1)) & (-page_sz); + addr = do_map(new_index, len, align, flags, NORMAL, &fd); + new_block->huge = NORMAL; }

/* if neither huge pages or normal pages works, we cannot proceed: */ - if ((addr == NULL) || (len == 0)) { + if ((fd < 0) || (addr == NULL) || (len == 0)) { if ((new_block->filename[0]) && (fd >= 0)) close(fd); odp_spinlock_unlock(&ishm_tbl->lock); @@ -884,6 +897,83 @@ int _odp_ishm_reserve(const char *name, uint64_t size, int fd, }

/* + * Try to map an memory block mapped by another ODP instance into the + * current ODP instance. + * returns 0 on success. + */ +int _odp_ishm_find_exported(const char *remote_name, pid_t external_odp_pid, + const char *local_name) +{ + char export_filename[ISHM_FILENAME_MAXLEN]; + char blockname[ISHM_FILENAME_MAXLEN]; + char filename[ISHM_FILENAME_MAXLEN]; + FILE *export_file; + uint64_t len; + uint32_t flags; + uint32_t align; + int fd; + int ret; + + /* try to read the block description file: */ + snprintf(export_filename, ISHM_FILENAME_MAXLEN, + ISHM_EXPTNAME_FORMAT, + external_odp_pid, + remote_name); + + export_file = fopen(export_filename, "r"); + + if (export_file == NULL) { + ODP_ERR("Error opening %s.\n", export_filename); + return -1; + } + + if (fscanf(export_file, EXPORT_FILE_LINE1_FMT " ") != 0) + goto error_exp_file; + + if (fscanf(export_file, EXPORT_FILE_LINE2_FMT " ", blockname) != 1) + goto error_exp_file; + + if (fscanf(export_file, EXPORT_FILE_LINE3_FMT " ", filename) != 1) + goto error_exp_file; + + if (fscanf(export_file, EXPORT_FILE_LINE4_FMT " ", &len) != 1) + goto error_exp_file; + + if (fscanf(export_file, EXPORT_FILE_LINE5_FMT " ", &flags) != 1) + goto error_exp_file; + + if (fscanf(export_file, EXPORT_FILE_LINE6_FMT " ", &align) != 1) + goto error_exp_file; + + fclose(export_file); + + /* now open the filename given in the description file: */ + fd = open(filename, O_RDWR, S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH); + if (fd == -1) { + ODP_ERR("open failed for %s: %s.\n", + filename, strerror(errno)); + return -1; + } + + /* clear the _ODP_ISHM_EXPORT flag so we don't export that again*/ + flags &= ~(uint32_t)_ODP_ISHM_EXPORT; + + /* reserve the memory, providing the opened file descriptor: */ + ret = _odp_ishm_reserve(local_name, 0, fd, align, flags, 0); + if (ret < 0) { + close(fd); + return ret; + } + + return ret; + +error_exp_file: + fclose(export_file); + ODP_ERR("Error reading %s.\n", export_filename); + return -1; +} + +/* * Free and unmap internal shared memory: * The file descriptor is closed and the .../odp-* file deleted, * unless fd was externally provided at reserve() time. @@ -1192,7 +1282,7 @@ int _odp_ishm_info(int block_index, _odp_ishm_info_t *info) info->name = ishm_tbl->block[block_index].name; info->addr = ishm_proctable->entry[proc_index].start; info->size = ishm_tbl->block[block_index].user_len; - info->page_size = ishm_tbl->block[block_index].huge ? + info->page_size = (ishm_tbl->block[block_index].huge == HUGE) ? odp_sys_huge_page_size() : odp_sys_page_size(); info->flags = ishm_tbl->block[block_index].flags; info->user_flags = ishm_tbl->block[block_index].user_flags; @@ -1435,7 +1525,19 @@ int _odp_ishm_status(const char *title) flags[1] = (ishm_tbl->block[i].flags & _ODP_ISHM_LOCK) ? 'L' : '.'; flags[2] = 0; - huge = (ishm_tbl->block[i].huge) ? 'H' : '.'; + switch (ishm_tbl->block[i].huge) { + case HUGE: + huge = 'H'; + break; + case NORMAL: + huge = 'N'; + break; + case EXTERNAL: + huge = 'E'; + break; + default: + huge = '?'; + } proc_index = procfind_block(i); ODP_DBG("%-3d: name:%-.24s file:%-.24s tid:%-3d" " flags:%s,%c len:0x%-08lx" diff --git a/platform/linux-generic/include/_ishm_internal.h b/platform/linux-generic/include/_ishm_internal.h index f5de26e..c7c3307 100644 --- a/platform/linux-generic/include/_ishm_internal.h +++ b/platform/linux-generic/include/_ishm_internal.h @@ -11,6 +11,8 @@ extern "C" { #endif

+#include <sys/types.h> + /* flags available at ishm_reserve: */ #define _ODP_ISHM_SINGLE_VA 1 #define _ODP_ISHM_LOCK 2 @@ -36,6 +38,9 @@ int _odp_ishm_free_by_address(void *addr); void *_odp_ishm_lookup_by_index(int block_index); int _odp_ishm_lookup_by_name(const char *name); int _odp_ishm_lookup_by_address(void *addr); +int _odp_ishm_find_exported(const char *remote_name, + pid_t external_odp_pid, + const char *local_name); void *_odp_ishm_address(int block_index); int _odp_ishm_info(int block_index, _odp_ishm_info_t *info); int _odp_ishm_status(const char *title);

commit f06971cf35f58b6ed30a46446c185bd6a65ba388 Author: Christophe Milard christophe.milard@linaro.org Date: Sat Aug 20 09:45:59 2016 +0200

linux-gen: ishm: adding debug function

A debug function, printing the internal mem alloc status is added.

Signed-off-by: Christophe Milard christophe.milard@linaro.org Reviewed-and-tested-by: Bill Fischofer bill.fischofer@linaro.org Reviewed-by: Brian Brooks brian.brooks@linaro.org Signed-off-by: Maxim Uvarov maxim.uvarov@linaro.org

diff --git a/platform/linux-generic/_ishm.c b/platform/linux-generic/_ishm.c index 6ceda80..37e56d4 100644 --- a/platform/linux-generic/_ishm.c +++ b/platform/linux-generic/_ishm.c @@ -1393,3 +1393,128 @@ int _odp_ishm_term_local(void) odp_spinlock_unlock(&ishm_tbl->lock); return 0; } + +/* + * Print the current ishm status (allocated blocks and VA space map) + * Return the number of allocated blocks (including those not mapped + * by the current odp thread). Also perform a number of sanity check. + * For debug. + */ +int _odp_ishm_status(const char *title) +{ + int i; + char flags[3]; + char huge; + int proc_index; + ishm_fragment_t *fragmnt; + int consecutive_unallocated = 0; /* should never exceed 1 */ + uintptr_t last_address = 0; + ishm_fragment_t *previous = NULL; + int nb_used_frgments = 0; + int nb_unused_frgments = 0; /* nb frag describing a VA area */ + int nb_allocated_frgments = 0; /* nb frag describing an allocated VA */ + int nb_blocks = 0; + int single_va_blocks = 0; + + odp_spinlock_lock(&ishm_tbl->lock); + procsync(); + + ODP_DBG("ishm blocks allocated at: %s\n", title); + + /* display block table: 1 line per entry +1 extra line if mapped here */ + for (i = 0; i < ISHM_MAX_NB_BLOCKS; i++) { + if (ishm_tbl->block[i].len <= 0) + continue; /* unused block */ + + nb_blocks++; + if (ishm_tbl->block[i].flags & _ODP_ISHM_SINGLE_VA) + single_va_blocks++; + + flags[0] = (ishm_tbl->block[i].flags & _ODP_ISHM_SINGLE_VA) ? + 'S' : '.'; + flags[1] = (ishm_tbl->block[i].flags & _ODP_ISHM_LOCK) ? + 'L' : '.'; + flags[2] = 0; + huge = (ishm_tbl->block[i].huge) ? 'H' : '.'; + proc_index = procfind_block(i); + ODP_DBG("%-3d: name:%-.24s file:%-.24s tid:%-3d" + " flags:%s,%c len:0x%-08lx" + " user_len:%-8ld seq:%-3ld refcnt:%-4d\n", + i, + ishm_tbl->block[i].name, + ishm_tbl->block[i].filename, + ishm_tbl->block[i].main_odpthread, + flags, huge, + ishm_tbl->block[i].len, + ishm_tbl->block[i].user_len, + ishm_tbl->block[i].seq, + ishm_tbl->block[i].refcnt); + + if (proc_index < 0) + continue; + + ODP_DBG(" start:%-08lx fd:%-3d\n", + ishm_proctable->entry[proc_index].start, + ishm_proctable->entry[proc_index].fd); + } + + /* display the virtual space allocations... : */ + ODP_DBG("ishm virtual space:\n"); + for (fragmnt = ishm_ftbl->used_fragmnts; + fragmnt; fragmnt = fragmnt->next) { + if (fragmnt->block_index >= 0) { + nb_allocated_frgments++; + ODP_DBG(" %08p - %08p: ALLOCATED by block:%d\n", + (uintptr_t)fragmnt->start, + (uintptr_t)fragmnt->start + fragmnt->len - 1, + fragmnt->block_index); + consecutive_unallocated = 0; + } else { + ODP_DBG(" %08p - %08p: NOT ALLOCATED\n", + (uintptr_t)fragmnt->start, + (uintptr_t)fragmnt->start + fragmnt->len - 1); + if (consecutive_unallocated++) + ODP_ERR("defragmentation error\n"); + } + + /* some other sanity checks: */ + if (fragmnt->prev != previous) + ODP_ERR("chaining error\n"); + + if (fragmnt != ishm_ftbl->used_fragmnts) { + if ((uintptr_t)fragmnt->start != last_address + 1) + ODP_ERR("lost space error\n"); + } + + last_address = (uintptr_t)fragmnt->start + fragmnt->len - 1; + previous = fragmnt; + nb_used_frgments++; + } + + /* + * the number of blocks with the single_VA flag set should match + * the number of used fragments: + */ + if (single_va_blocks != nb_allocated_frgments) + ODP_ERR("single_va_blocks != nb_allocated_fragments!\n"); + + /* compute the number of unused fragments*/ + for (fragmnt = ishm_ftbl->unused_fragmnts; + fragmnt; fragmnt = fragmnt->next) + nb_unused_frgments++; + + ODP_DBG("ishm: %d fragment used. %d fragements unused. (total=%d)\n", + nb_used_frgments, nb_unused_frgments, + nb_used_frgments + nb_unused_frgments); + + if ((nb_used_frgments + nb_unused_frgments) != ISHM_NB_FRAGMNTS) + ODP_ERR("lost fragments!\n"); + + if (nb_blocks < ishm_proctable->nb_entries) + ODP_ERR("process known block cannot exceed main total sum!\n"); + + ODP_DBG("\n"); + + odp_spinlock_unlock(&ishm_tbl->lock); + return nb_blocks; +} diff --git a/platform/linux-generic/include/_ishm_internal.h b/platform/linux-generic/include/_ishm_internal.h index d348b41..f5de26e 100644 --- a/platform/linux-generic/include/_ishm_internal.h +++ b/platform/linux-generic/include/_ishm_internal.h @@ -38,6 +38,7 @@ int _odp_ishm_lookup_by_name(const char *name); int _odp_ishm_lookup_by_address(void *addr); void *_odp_ishm_address(int block_index); int _odp_ishm_info(int block_index, _odp_ishm_info_t *info); +int _odp_ishm_status(const char *title);

#ifdef __cplusplus }

commit 7412355450f0a78bd7cfe112a8520869e91e9647 Author: Christophe Milard christophe.milard@linaro.org Date: Thu Nov 24 17:22:23 2016 +0100

linux-gen: use ishm as north API mem allocator

The odp shared_memory API is changed to use the ODP internal memory allocator: _ishm. _ishm supports memory sharing between processes, regardless of fork time. The test testing the ODP_SHM_PROC flag is also changed to cope with the new OS sharing interface used by _ishm (link in /tmp).

Signed-off-by: Christophe Milard christophe.milard@linaro.org Reviewed-and-tested-by: Bill Fischofer bill.fischofer@linaro.org Signed-off-by: Maxim Uvarov maxim.uvarov@linaro.org

diff --git a/platform/linux-generic/include/odp_internal.h b/platform/linux-generic/include/odp_internal.h index 5698fb0..b313b1f 100644 --- a/platform/linux-generic/include/odp_internal.h +++ b/platform/linux-generic/include/odp_internal.h @@ -59,7 +59,6 @@ enum init_stage { SYSINFO_INIT, FDSERVER_INIT, ISHM_INIT, - SHM_INIT, THREAD_INIT, POOL_INIT, QUEUE_INIT, @@ -89,10 +88,6 @@ int odp_thread_init_local(odp_thread_type_t type); int odp_thread_term_local(void); int odp_thread_term_global(void);

-int odp_shm_init_global(void); -int odp_shm_term_global(void); -int odp_shm_init_local(void); - int odp_pool_init_global(void); int odp_pool_init_local(void); int odp_pool_term_global(void); diff --git a/platform/linux-generic/odp_init.c b/platform/linux-generic/odp_init.c index 43d9e40..1b0d8f8 100644 --- a/platform/linux-generic/odp_init.c +++ b/platform/linux-generic/odp_init.c @@ -120,12 +120,6 @@ int odp_init_global(odp_instance_t *instance, } stage = ISHM_INIT;

- if (odp_shm_init_global()) { - ODP_ERR("ODP shm init failed.\n"); - goto init_failed; - } - stage = SHM_INIT; - if (odp_thread_init_global()) { ODP_ERR("ODP thread init failed.\n"); goto init_failed; @@ -279,13 +273,6 @@ int _odp_term_global(enum init_stage stage) } /* Fall through */

- case SHM_INIT: - if (odp_shm_term_global()) { - ODP_ERR("ODP shm term failed.\n"); - rc = -1; - } - /* Fall through */ - case ISHM_INIT: if (_odp_ishm_term_global()) { ODP_ERR("ODP ishm term failed.\n"); @@ -343,12 +330,6 @@ int odp_init_local(odp_instance_t instance, odp_thread_type_t thr_type) } stage = ISHM_INIT;

- if (odp_shm_init_local()) { - ODP_ERR("ODP shm local init failed.\n"); - goto init_fail; - } - stage = SHM_INIT; - if (odp_thread_init_local(thr_type)) { ODP_ERR("ODP thread local init failed.\n"); goto init_fail; diff --git a/platform/linux-generic/odp_shared_memory.c b/platform/linux-generic/odp_shared_memory.c index 550af27..9e916e9 100644 --- a/platform/linux-generic/odp_shared_memory.c +++ b/platform/linux-generic/odp_shared_memory.c @@ -4,434 +4,88 @@ * SPDX-License-Identifier: BSD-3-Clause */

-#include <odp_posix_extensions.h> - -#include <odp/api/shared_memory.h> -#include <odp_internal.h> -#include <odp/api/spinlock.h> -#include <odp/api/align.h> -#include <odp/api/system_info.h> -#include <odp/api/debug.h> -#include <odp_shm_internal.h> -#include <odp_debug_internal.h> -#include <odp_align_internal.h> #include <odp_config_internal.h> - -#include <unistd.h> -#include <sys/mman.h> -#include <sys/stat.h> -#include <asm/mman.h> -#include <fcntl.h> - -#include <stdio.h> +#include <odp/api/debug.h> +#include <odp/api/std_types.h> +#include <odp/api/shared_memory.h> +#include <_ishm_internal.h> #include <string.h> -#include <errno.h> -#include <inttypes.h>

ODP_STATIC_ASSERT(ODP_CONFIG_SHM_BLOCKS >= ODP_CONFIG_POOLS, "ODP_CONFIG_SHM_BLOCKS < ODP_CONFIG_POOLS");

-typedef struct { - char name[ODP_SHM_NAME_LEN]; - uint64_t size; - uint64_t align; - uint64_t alloc_size; - void *addr_orig; - void *addr; - int huge; - odp_shm_t hdl; - uint32_t flags; - uint64_t page_sz; - int fd; - -} odp_shm_block_t; - - -typedef struct { - odp_shm_block_t block[ODP_CONFIG_SHM_BLOCKS]; - odp_spinlock_t lock; - -} odp_shm_table_t; - - -#ifndef MAP_ANONYMOUS -#define MAP_ANONYMOUS MAP_ANON -#endif - - -/* Global shared memory table */ -static odp_shm_table_t *odp_shm_tbl; - - static inline uint32_t from_handle(odp_shm_t shm) { return _odp_typeval(shm) - 1; }

- static inline odp_shm_t to_handle(uint32_t index) { return _odp_cast_scalar(odp_shm_t, index + 1); }

- -int odp_shm_init_global(void) -{ - void *addr; - -#ifndef MAP_HUGETLB - ODP_DBG("NOTE: mmap does not support huge pages\n"); -#endif - - addr = mmap(NULL, sizeof(odp_shm_table_t), - PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0); - - if (addr == MAP_FAILED) - return -1; - - odp_shm_tbl = addr; - - memset(odp_shm_tbl, 0, sizeof(odp_shm_table_t)); - odp_spinlock_init(&odp_shm_tbl->lock); - - return 0; -} - -int odp_shm_term_global(void) -{ - int ret; - - ret = munmap(odp_shm_tbl, sizeof(odp_shm_table_t)); - if (ret) - ODP_ERR("unable to munmap\n."); - - return ret; -} - - -int odp_shm_init_local(void) -{ - return 0; -} - int odp_shm_capability(odp_shm_capability_t *capa) { memset(capa, 0, sizeof(odp_shm_capability_t));

capa->max_blocks = ODP_CONFIG_SHM_BLOCKS; - capa->max_size = 0; - capa->max_align = 0; + capa->max_size = 0; + capa->max_align = 0;

return 0; }

-static int find_block(const char *name, uint32_t *index) -{ - uint32_t i; - - for (i = 0; i < ODP_CONFIG_SHM_BLOCKS; i++) { - if (strcmp(name, odp_shm_tbl->block[i].name) == 0) { - /* found it */ - if (index != NULL) - *index = i; - - return 1; - } - } - - return 0; -} - -int odp_shm_free(odp_shm_t shm) -{ - uint32_t i; - int ret; - odp_shm_block_t *block; - char shm_devname[SHM_DEVNAME_MAXLEN]; - - if (shm == ODP_SHM_INVALID) { - ODP_DBG("odp_shm_free: Invalid handle\n"); - return -1; - } - - i = from_handle(shm); - - if (i >= ODP_CONFIG_SHM_BLOCKS) { - ODP_DBG("odp_shm_free: Bad handle\n"); - return -1; - } - - odp_spinlock_lock(&odp_shm_tbl->lock); - - block = &odp_shm_tbl->block[i]; - - if (block->addr == NULL) { - ODP_DBG("odp_shm_free: Free block\n"); - odp_spinlock_unlock(&odp_shm_tbl->lock); - return 0; - } - - ret = munmap(block->addr_orig, block->alloc_size); - if (0 != ret) { - ODP_DBG("odp_shm_free: munmap failed: %s, id %u, addr %p\n", - strerror(errno), i, block->addr_orig); - odp_spinlock_unlock(&odp_shm_tbl->lock); - return -1; - } - - if (block->flags & ODP_SHM_PROC || block->flags & _ODP_SHM_PROC_NOCREAT) { - int shm_ns_id; - - if (odp_global_data.ipc_ns) - shm_ns_id = odp_global_data.ipc_ns; - else - shm_ns_id = odp_global_data.main_pid; - - snprintf(shm_devname, SHM_DEVNAME_MAXLEN, - SHM_DEVNAME_FORMAT, shm_ns_id, block->name); - ret = shm_unlink(shm_devname); - if (0 != ret) { - ODP_DBG("odp_shm_free: shm_unlink failed\n"); - odp_spinlock_unlock(&odp_shm_tbl->lock); - return -1; - } - } - memset(block, 0, sizeof(odp_shm_block_t)); - odp_spinlock_unlock(&odp_shm_tbl->lock); - return 0; -} - odp_shm_t odp_shm_reserve(const char *name, uint64_t size, uint64_t align, uint32_t flags) { - uint32_t i; - char shm_devname[SHM_DEVNAME_MAXLEN]; - odp_shm_block_t *block; - void *addr; - int fd = -1; - int map_flag = MAP_SHARED; - /* If already exists: O_EXCL: error, O_TRUNC: truncate to zero */ - int oflag = O_RDWR; - uint64_t alloc_size; - uint64_t page_sz, huge_sz; -#ifdef MAP_HUGETLB - int need_huge_page = 0; - uint64_t alloc_hp_size; -#endif - - page_sz = odp_sys_page_size(); - alloc_size = size + align; + int block_index; + int flgs = 0; /* internal ishm flags */

-#ifdef MAP_HUGETLB - huge_sz = odp_sys_huge_page_size(); - need_huge_page = (huge_sz && alloc_size > page_sz); - /* munmap for huge pages requires sizes round up by page */ - alloc_hp_size = (size + align + (huge_sz - 1)) & (-huge_sz); -#endif + /* set internal ishm flags according to API flags: */ + flgs |= (flags & ODP_SHM_PROC) ? _ODP_ISHM_EXPORT : 0;

- if (flags & ODP_SHM_PROC) - oflag |= O_CREAT | O_TRUNC; - if (flags & _ODP_SHM_O_EXCL) - oflag |= O_EXCL; + /* all mem reserved through this interface is requested to be locked: */ + flgs |= (flags & _ODP_ISHM_LOCK);

- if (flags & (ODP_SHM_PROC | _ODP_SHM_PROC_NOCREAT)) { - int shm_ns_id; - - if (odp_global_data.ipc_ns) - shm_ns_id = odp_global_data.ipc_ns; - else - shm_ns_id = odp_global_data.main_pid; - - need_huge_page = 0; - - /* Creates a file to /dev/shm/odp */ - snprintf(shm_devname, SHM_DEVNAME_MAXLEN, - SHM_DEVNAME_FORMAT, shm_ns_id, name); - fd = shm_open(shm_devname, oflag, - S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH); - if (fd == -1) { - ODP_DBG("%s: shm_open failed.\n", shm_devname); - return ODP_SHM_INVALID; - } - } else { - map_flag |= MAP_ANONYMOUS; - } - - odp_spinlock_lock(&odp_shm_tbl->lock); - - if (find_block(name, NULL)) { - /* Found a block with the same name */ - odp_spinlock_unlock(&odp_shm_tbl->lock); - ODP_DBG("name "%s" already used.\n", name); + block_index = _odp_ishm_reserve(name, size, -1, align, flgs, flags); + if (block_index >= 0) + return to_handle(block_index); + else return ODP_SHM_INVALID; - } - - for (i = 0; i < ODP_CONFIG_SHM_BLOCKS; i++) { - if (odp_shm_tbl->block[i].addr == NULL) { - /* Found free block */ - break; - } - } - - if (i > ODP_CONFIG_SHM_BLOCKS - 1) { - /* Table full */ - odp_spinlock_unlock(&odp_shm_tbl->lock); - ODP_DBG("%s: no more blocks.\n", name); - return ODP_SHM_INVALID; - } - - block = &odp_shm_tbl->block[i]; - - block->hdl = to_handle(i); - addr = MAP_FAILED; - -#ifdef MAP_HUGETLB - /* Try first huge pages */ - if (need_huge_page) { - if ((flags & ODP_SHM_PROC) && - (ftruncate(fd, alloc_hp_size) == -1)) { - odp_spinlock_unlock(&odp_shm_tbl->lock); - ODP_DBG("%s: ftruncate huge pages failed.\n", name); - return ODP_SHM_INVALID; - } - - addr = mmap(NULL, alloc_hp_size, PROT_READ | PROT_WRITE, - map_flag | MAP_HUGETLB, fd, 0); - if (addr == MAP_FAILED) { - ODP_DBG(" %s:\n" - "\tNo huge pages, fall back to normal pages,\n" - "\tcheck: /proc/sys/vm/nr_hugepages.\n", name); - } else { - block->alloc_size = alloc_hp_size; - block->huge = 1; - block->page_sz = huge_sz; - } - } -#endif - - /* Use normal pages for small or failed huge page allocations */ - if (addr == MAP_FAILED) { - if ((flags & ODP_SHM_PROC) && - (ftruncate(fd, alloc_size) == -1)) { - odp_spinlock_unlock(&odp_shm_tbl->lock); - ODP_ERR("%s: ftruncate failed.\n", name); - return ODP_SHM_INVALID; - } - - addr = mmap(NULL, alloc_size, PROT_READ | PROT_WRITE, - map_flag, fd, 0); - if (addr == MAP_FAILED) { - odp_spinlock_unlock(&odp_shm_tbl->lock); - ODP_DBG("%s mmap failed.\n", name); - return ODP_SHM_INVALID; - } else { - block->alloc_size = alloc_size; - block->huge = 0; - block->page_sz = page_sz; - } - } - - block->addr_orig = addr; - - /* move to correct alignment */ - addr = ODP_ALIGN_ROUNDUP_PTR(addr, align); - - strncpy(block->name, name, ODP_SHM_NAME_LEN - 1); - block->name[ODP_SHM_NAME_LEN - 1] = 0; - block->size = size; - block->align = align; - block->flags = flags; - block->fd = fd; - block->addr = addr; +}

- odp_spinlock_unlock(&odp_shm_tbl->lock); - return block->hdl; +int odp_shm_free(odp_shm_t shm) +{ + return _odp_ishm_free_by_index(from_handle(shm)); }

odp_shm_t odp_shm_lookup(const char *name) { - uint32_t i; - odp_shm_t hdl; - - odp_spinlock_lock(&odp_shm_tbl->lock); - - if (find_block(name, &i) == 0) { - odp_spinlock_unlock(&odp_shm_tbl->lock); - return ODP_SHM_INVALID; - } - - hdl = odp_shm_tbl->block[i].hdl; - odp_spinlock_unlock(&odp_shm_tbl->lock); - - return hdl; + return to_handle(_odp_ishm_lookup_by_name(name)); }

- void *odp_shm_addr(odp_shm_t shm) { - uint32_t i; - - i = from_handle(shm); - - if (i > (ODP_CONFIG_SHM_BLOCKS - 1)) - return NULL; - - return odp_shm_tbl->block[i].addr; + return _odp_ishm_address(from_handle(shm)); }

- int odp_shm_info(odp_shm_t shm, odp_shm_info_t *info) { - odp_shm_block_t *block; - uint32_t i; + _odp_ishm_info_t ishm_info;

- i = from_handle(shm); - - if (i > (ODP_CONFIG_SHM_BLOCKS - 1)) + if (_odp_ishm_info(from_handle(shm), &ishm_info)) return -1;

- block = &odp_shm_tbl->block[i]; - - info->name = block->name; - info->addr = block->addr; - info->size = block->size; - info->page_size = block->page_sz; - info->flags = block->flags; + info->name = ishm_info.name; + info->addr = ishm_info.addr; + info->size = ishm_info.size; + info->page_size = ishm_info.page_size; + info->flags = ishm_info.user_flags;

return 0; }

- void odp_shm_print_all(void) { - int i; - - ODP_PRINT("\nShared memory\n"); - ODP_PRINT("--------------\n"); - ODP_PRINT(" page size: %"PRIu64" kB\n", - odp_sys_page_size() / 1024); - ODP_PRINT(" huge page size: %"PRIu64" kB\n", - odp_sys_huge_page_size() / 1024); - ODP_PRINT("\n"); - - ODP_PRINT(" id name kB align huge addr\n"); - - for (i = 0; i < ODP_CONFIG_SHM_BLOCKS; i++) { - odp_shm_block_t *block; - - block = &odp_shm_tbl->block[i]; - - if (block->addr) { - ODP_PRINT(" %2i %-24s %4"PRIu64" %4"PRIu64 - " %2c %p\n", - i, - block->name, - block->size/1024, - block->align, - (block->huge ? '*' : ' '), - block->addr); - } - } - - ODP_PRINT("\n"); + _odp_ishm_status("Memory allocation status:"); } diff --git a/test/linux-generic/validation/api/shmem/shmem_linux.c b/test/linux-generic/validation/api/shmem/shmem_linux.c index 212a6c1..9ab0e2b 100644 --- a/test/linux-generic/validation/api/shmem/shmem_linux.c +++ b/test/linux-generic/validation/api/shmem/shmem_linux.c @@ -45,12 +45,69 @@ #include <sys/mman.h> #include <libgen.h> #include <linux/limits.h> +#include <inttypes.h> #include "shmem_linux.h" #include "shmem_common.h"

-#define ODP_APP_NAME "shmem_odp" /* name of the odp program, in this dir */ -#define DEVNAME_FMT "odp-%d-%s" /* shm device format: odp-<pid>-<name> */ -#define MAX_FIFO_WAIT 30 /* Max time waiting for the fifo (sec) */ +#define ODP_APP_NAME "shmem_odp" /* name of the odp program, in this dir */ +#define DEVNAME_FMT "/tmp/odp-%" PRIu64 "-shm-%s" /* odp-<pid>-shm-<name> */ +#define MAX_FIFO_WAIT 30 /* Max time waiting for the fifo (sec) */ + +/* + * read the attributes of a externaly shared mem object: + * input: ext_odp_pid, blockname: the remote ODP instance and the exported + * block name to be searched. + * Output: filename: the memory block underlaying file to be opened + * (the given buffer should be big enough i.e. at + * least ISHM_FILENAME_MAXLEN bytes) + * The 3 following parameters are really here for debug + * as they are really meaningles in a non-odp process: + * len: the block real length (bytes, multiple of page sz) + * flags: the _ishm flags setting the block was created with + * align: the alignement setting the block was created with + * + * return 0 on success, non zero on error + */ +static int read_shmem_attribues(uint64_t ext_odp_pid, const char *blockname, + char *filename, uint64_t *len, + uint32_t *flags, uint32_t *align) +{ + char shm_attr_filename[PATH_MAX]; + FILE *export_file; + + sprintf(shm_attr_filename, DEVNAME_FMT, ext_odp_pid, blockname); + + /* O_CREAT flag not given => failure if shm_attr_filename does not + * already exist */ + export_file = fopen(shm_attr_filename, "r"); + if (export_file == NULL) + return -1; + + if (fscanf(export_file, "ODP exported shm block info: ") != 0) + goto export_file_read_err; + + if (fscanf(export_file, "ishm_blockname: %*s ") != 0) + goto export_file_read_err; + + if (fscanf(export_file, "file: %s ", filename) != 1) + goto export_file_read_err; + + if (fscanf(export_file, "length: %" PRIu64 " ", len) != 1) + goto export_file_read_err; + + if (fscanf(export_file, "flags: %" PRIu32 " ", flags) != 1) + goto export_file_read_err; + + if (fscanf(export_file, "align: %" PRIu32 " ", align) != 1) + goto export_file_read_err; + + fclose(export_file); + return 0; + +export_file_read_err: + fclose(export_file); + return -1; +}

void test_success(char *fifo_name, int fd, pid_t odp_app) { @@ -91,12 +148,15 @@ int main(int argc __attribute__((unused)), char *argv[]) char prg_name[PATH_MAX]; char odp_name[PATH_MAX]; int nb_sec; - int size; + uint64_t size; pid_t odp_app; char *odp_params = NULL; char fifo_name[PATH_MAX]; /* fifo for linux->odp feedback */ int fifo_fd = -1; - char shm_devname[PATH_MAX];/* shared mem device name, under /dev/shm */ + char shm_devname[PATH_MAX];/* shared mem device name.*/ + uint64_t len; + uint32_t flags; + uint32_t align; int shm_fd; test_shared_linux_data_t *addr;

@@ -130,26 +190,28 @@ int main(int argc __attribute__((unused)), char *argv[]) * ODP application is up and running, and has allocated shmem. * check to see if linux can see the created shared memory: */

- sprintf(shm_devname, DEVNAME_FMT, odp_app, ODP_SHM_NAME); + /* read the shared memory attributes (includes the shm filename): */ + if (read_shmem_attribues(odp_app, ODP_SHM_NAME, + shm_devname, &len, &flags, &align) != 0) + test_failure(fifo_name, fifo_fd, odp_app);

- /* O_CREAT flag not given => failure if shm_devname does not already + /* open the shm filename (which is either on /tmp or on hugetlbfs) + * O_CREAT flag not given => failure if shm_devname does not already * exist */ - shm_fd = shm_open(shm_devname, O_RDONLY, - S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH); + shm_fd = open(shm_devname, O_RDONLY, + S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH); if (shm_fd == -1) - test_failure(fifo_name, shm_fd, odp_app); + test_failure(fifo_name, fifo_fd, odp_app); + + /* linux ODP guarantees page size alignement. Larger alignment may + * fail as 2 different processes will have fully unrelated + * virtual spaces. + */ + size = sizeof(test_shared_linux_data_t);

- /* we know that the linux generic ODP actually allocates the required - * size + alignment and aligns the returned address after. - * we must do the same here: */ - size = sizeof(test_shared_linux_data_t) + ALIGN_SIZE; addr = mmap(NULL, size, PROT_READ, MAP_SHARED, shm_fd, 0); if (addr == MAP_FAILED) - test_failure(fifo_name, shm_fd, odp_app); - - /* perform manual alignment */ - addr = (test_shared_linux_data_t *)((((unsigned long int)addr + - ALIGN_SIZE - 1) / ALIGN_SIZE) * ALIGN_SIZE); + test_failure(fifo_name, fifo_fd, odp_app);

/* check that we see what the ODP application wrote in the memory */ if ((addr->foo == TEST_SHARE_FOO) && (addr->bar == TEST_SHARE_BAR))

commit 6e57710dc7eb24c3c91a30e6d1f5d9cd3ad9976a Author: Christophe Milard christophe.milard@linaro.org Date: Thu Nov 24 17:22:21 2016 +0100

linux-gen: _ishm: create description file for external memory sharing

A new flag called _ODP_ISHM_EXPORT is added to _ishm. When this flag is specified at reserve() time, an extra file ("/tmp/odp-<pid>-shm-<blockname>", where <pid> is the process ID of the main ODP instatiation process and <blockname> is the block name given at reserve time) is created, describing to the underlying block attributes. This file is meant to be used by processes external to ODP willing to share this memory.

Signed-off-by: Christophe Milard christophe.milard@linaro.org Reviewed-and-tested-by: Bill Fischofer bill.fischofer@linaro.org Signed-off-by: Maxim Uvarov maxim.uvarov@linaro.org

diff --git a/platform/linux-generic/_ishm.c b/platform/linux-generic/_ishm.c index c8156aa..6ceda80 100644 --- a/platform/linux-generic/_ishm.c +++ b/platform/linux-generic/_ishm.c @@ -99,6 +99,14 @@ #define ISHM_FILENAME_NORMAL_PAGE_DIR "/tmp"

/* + * when the memory is to be shared with an external entity (such as another + * ODP instance or an OS process not part of this ODP instance) then a + * export file is created describing the exported memory: this defines the + * location and the filename format of this description file + */ +#define ISHM_EXPTNAME_FORMAT "/tmp/odp-%d-shm-%s" + +/* * At worse case the virtual space gets so fragmented that there is * a unallocated fragment between each allocated fragment: * In that case, the number of fragments to take care of is twice the @@ -107,6 +115,17 @@ #define ISHM_NB_FRAGMNTS (ISHM_MAX_NB_BLOCKS * 2 + 1)

/* + * when a memory block is to be exported outside its ODP instance, + * an block 'attribute file' is created in /tmp/odp-<pid>-shm-<name>. + * The information given in this file is according to the following: + */ +#define EXPORT_FILE_LINE1_FMT "ODP exported shm block info:" +#define EXPORT_FILE_LINE2_FMT "ishm_blockname: %s" +#define EXPORT_FILE_LINE3_FMT "file: %s" +#define EXPORT_FILE_LINE4_FMT "length: %" PRIu64 +#define EXPORT_FILE_LINE5_FMT "flags: %" PRIu32 +#define EXPORT_FILE_LINE6_FMT "align: %" PRIu32 +/* * A fragment describes a piece of the shared virtual address space, * and is allocated only when allocation is done with the _ODP_ISHM_SINGLE_VA * flag: @@ -136,6 +155,7 @@ typedef struct ishm_fragment { typedef struct ishm_block { char name[ISHM_NAME_MAXLEN]; /* name for the ishm block (if any) */ char filename[ISHM_FILENAME_MAXLEN]; /* name of the .../odp-* file */ + char exptname[ISHM_FILENAME_MAXLEN]; /* name of the export file */ int main_odpthread; /* The thread which did the initial reserve*/ uint32_t user_flags; /* any flags the user want to remember. */ uint32_t flags; /* block creation flags. */ @@ -380,7 +400,8 @@ static void free_fragment(ishm_fragment_t *fragmnt) * or /mnt/huge/odp-<pid>-<sequence_or_name> (for huge pages) * Return the new file descriptor, or -1 on error. */ -static int create_file(int block_index, int huge, uint64_t len) +static int create_file(int block_index, int huge, uint64_t len, + uint32_t flags, uint32_t align) { char *name; int fd; @@ -388,6 +409,7 @@ static int create_file(int block_index, int huge, uint64_t len) char seq_string[ISHM_FILENAME_MAXLEN]; /* used to construct filename*/ char filename[ISHM_FILENAME_MAXLEN];/* filename in /tmp/ or /mnt/huge */ int oflag = O_RDWR | O_CREAT | O_TRUNC; /* flags for open */ + FILE *export_file;

new_block = &ishm_tbl->block[block_index]; name = new_block->name; @@ -429,9 +451,48 @@ static int create_file(int block_index, int huge, uint64_t len)

strncpy(new_block->filename, filename, ISHM_FILENAME_MAXLEN - 1);

+ /* if _ODP_ISHM_EXPORT is set, create a description file for + * external ref: + */ + if (flags & _ODP_ISHM_EXPORT) { + snprintf(new_block->exptname, ISHM_FILENAME_MAXLEN, + ISHM_EXPTNAME_FORMAT, + odp_global_data.main_pid, + (name && name[0]) ? name : seq_string); + export_file = fopen(new_block->exptname, "w"); + if (export_file == NULL) { + ODP_ERR("open failed: err=%s.\n", + strerror(errno)); + new_block->exptname[0] = 0; + } else { + fprintf(export_file, EXPORT_FILE_LINE1_FMT "\n"); + fprintf(export_file, EXPORT_FILE_LINE2_FMT "\n", name); + fprintf(export_file, EXPORT_FILE_LINE3_FMT "\n", + new_block->filename); + fprintf(export_file, EXPORT_FILE_LINE4_FMT "\n", len); + fprintf(export_file, EXPORT_FILE_LINE5_FMT "\n", flags); + fprintf(export_file, EXPORT_FILE_LINE6_FMT "\n", align); + + fclose(export_file); + } + } else { + new_block->exptname[0] = 0; + } + return fd; }

+/* delete the files related to a given ishm block: */ +static void delete_file(ishm_block_t *block) +{ + /* remove the .../odp-* file, unless fd was external: */ + if (block->filename[0] != 0) + unlink(block->filename); + /* also remove possible description file (if block was exported): */ + if (block->exptname[0] != 0) + unlink(block->exptname); +} + /* * performs the mapping, possibly allocating a fragment of the pre-reserved * VA space if the _ODP_ISHM_SINGLE_VA flag was given. @@ -456,7 +517,7 @@ static void *do_map(int block_index, uint64_t len, uint32_t align, * unless a fd was already given */ if (*fd < 0) { - *fd = create_file(block_index, huge, len); + *fd = create_file(block_index, huge, len, flags, align); if (*fd < 0) return NULL; } else { @@ -471,7 +532,7 @@ static void *do_map(int block_index, uint64_t len, uint32_t align, if (new_block->filename[0]) { close(*fd); *fd = -1; - unlink(new_block->filename); + delete_file(new_block); } return NULL; } @@ -486,7 +547,7 @@ static void *do_map(int block_index, uint64_t len, uint32_t align, if (new_block->filename[0]) { close(*fd); *fd = -1; - unlink(new_block->filename); + delete_file(new_block); } return NULL; } @@ -867,9 +928,8 @@ static int block_free(int block_index) do_unmap(NULL, 0, block->flags, block_index); }

- /* remove the .../odp-* file, unless fd was external: */ - if (block->filename[0] != 0) - unlink(block->filename); + /* remove all files related to this block: */ + delete_file(block);

/* deregister the file descriptor from the file descriptor server. */ _odp_fdserver_deregister_fd(FD_SRV_CTX_ISHM, block_index); diff --git a/platform/linux-generic/include/_ishm_internal.h b/platform/linux-generic/include/_ishm_internal.h index 7d27477..d348b41 100644 --- a/platform/linux-generic/include/_ishm_internal.h +++ b/platform/linux-generic/include/_ishm_internal.h @@ -14,6 +14,7 @@ extern "C" { /* flags available at ishm_reserve: */ #define _ODP_ISHM_SINGLE_VA 1 #define _ODP_ISHM_LOCK 2 +#define _ODP_ISHM_EXPORT 4 /*create export descr file in /tmp */

/** * Shared memory block info

commit 7e93b42f889e544bbf63576e36d2f1b313fd7a4e Author: Christophe Milard christophe.milard@linaro.org Date: Sat Aug 20 09:45:58 2016 +0200

linux-gen: ishm: internal shared memory allocator (ishm) added

A new ODP internal memory allocator, called ishm (for internal shmem) is introduced here. This memory allocator enables the following: - works for odpthreads being linux processes, regardless for fork time. - guarantees the uniqueness of the virtual space mapping address over all ODP threads (even processes and regardless of fork time), when the required _ODP_ISHM_SINGLE_VA flag is used.

Signed-off-by: Christophe Milard christophe.milard@linaro.org Reviewed-and-tested-by: Bill Fischofer bill.fischofer@linaro.org Reviewed-by: Brian Brooks brian.brooks@linaro.org Signed-off-by: Maxim Uvarov maxim.uvarov@linaro.org

diff --git a/platform/linux-generic/Makefile.am b/platform/linux-generic/Makefile.am index 434e530..0bc9842 100644 --- a/platform/linux-generic/Makefile.am +++ b/platform/linux-generic/Makefile.am @@ -105,6 +105,8 @@ odpdrvinclude_HEADERS = \

noinst_HEADERS = \ ${srcdir}/include/_fdserver_internal.h \ + ${srcdir}/include/_ishm_internal.h \ + ${srcdir}/include/_ishmphy_internal.h \ ${srcdir}/include/odp_align_internal.h \ ${srcdir}/include/odp_atomic_internal.h \ ${srcdir}/include/odp_buffer_inlines.h \ @@ -147,6 +149,8 @@ noinst_HEADERS = \

__LIB__libodp_linux_la_SOURCES = \ _fdserver.c \ + _ishm.c \ + _ishmphy.c \ odp_atomic.c \ odp_barrier.c \ odp_buffer.c \ diff --git a/platform/linux-generic/_ishm.c b/platform/linux-generic/_ishm.c new file mode 100644 index 0000000..c8156aa --- /dev/null +++ b/platform/linux-generic/_ishm.c @@ -0,0 +1,1335 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +/* This file handles the internal shared memory: internal shared memory + * is memory which is sharable by all ODP threads regardless of how the + * ODP thread is implemented (pthread or process) and regardless of fork() + * time. + * Moreover, when reserved with the _ODP_ISHM_SINGLE_VA flag, + * internal shared memory is guaranteed to always be located at the same virtual + * address, i.e. pointers to internal shared memory are fully shareable + * between odp threads (regardless of thread type or fork time) in that case. + * Internal shared memory is mainly meant to be used internaly within ODP + * (hence its name), but may also be allocated by odp applications and drivers, + * in the future (through these interfaces). + * To guarrentee this full pointer shareability (when reserved with the + * _ODP_ISHM_SINGLE_VA flag) internal shared memory is handled as follows: + * At global_init time, a huge virtual address space reservation is performed. + * Note that this is just reserving virtual space, not physical memory. + * Because all ODP threads (pthreads or processes) are descendants of the ODP + * instantiation process, this VA space is inherited by all ODP threads. + * When internal shmem reservation actually occurs, and + * when reserved with the _ODP_ISHM_SINGLE_VA flag, physical memory is + * allocated, and mapped (MAP_FIXED) to some part in the huge preallocated + * address space area: + * because this virtual address space is common to all ODP threads, we + * know this mapping will succeed, and not clash with anything else. + * Hence, an ODP threads which perform a lookup for the same ishm block + * can map it at the same VA address. + * When internal shared memory is released, the physical memory is released + * and the corresponding virtual space returned to its "pool" of preallocated + * virtual space (assuming it was allocated from there). + * Note, though, that, if 2 linux processes share the same ishm block, + * the virtual space is marked as released as soon as one of the processes + * releases the ishm block, but the physical memory space is actually released + * by the kernel once all processes have done a ishm operation (i,e. a sync). + * This is due to the fact that linux does not contain any syscall to unmap + * memory from a different process. + * + * This file contains functions to handle the VA area (handling fragmentation + * and defragmentation resulting from different allocs/release) and also + * define the functions to allocate, release and lookup internal shared + * memory: + * _odp_ishm_reserve(), _odp_ishm_free*() and _odp_ishm_lookup*()... + */ +#include <odp_posix_extensions.h> +#include <odp_config_internal.h> +#include <odp_internal.h> +#include <odp/api/spinlock.h> +#include <odp/api/align.h> +#include <odp/api/system_info.h> +#include <odp/api/debug.h> +#include <odp_shm_internal.h> +#include <odp_debug_internal.h> +#include <odp_align_internal.h> +#include <_fdserver_internal.h> +#include <_ishm_internal.h> +#include <_ishmphy_internal.h> +#include <stdlib.h> +#include <stdio.h> +#include <unistd.h> +#include <string.h> +#include <errno.h> +#include <sys/mman.h> +#include <sys/stat.h> +#include <sys/syscall.h> +#include <fcntl.h> +#include <sys/types.h> +#include <inttypes.h> +#include <sys/wait.h> + +/* + * Maximum number of internal shared memory blocks. + * + * This the the number of separate ISHM areas that can be reserved concurrently + * (Note that freeing such blocks may take time, or possibly never happen + * if some of the block ownwers never procsync() after free). This number + * should take that into account) + */ +#define ISHM_MAX_NB_BLOCKS 128 + +/* + * Maximum internal shared memory block name length in chars + * probably taking the same number as SHM name size make sense at this stage + */ +#define ISHM_NAME_MAXLEN 32 + +/* + * Linux underlying file name: <directory>/odp-<odp_pid>-ishm-<name> + * The <name> part may be replaced by a sequence number if no specific + * name is given at reserve time + * <directory> is either /tmp or the hugepagefs mount point for default size. + * (searched at init time) + */ +#define ISHM_FILENAME_MAXLEN (ISHM_NAME_MAXLEN + 64) +#define ISHM_FILENAME_FORMAT "%s/odp-%d-ishm-%s" +#define ISHM_FILENAME_NORMAL_PAGE_DIR "/tmp" + +/* + * At worse case the virtual space gets so fragmented that there is + * a unallocated fragment between each allocated fragment: + * In that case, the number of fragments to take care of is twice the + * number of ISHM blocks + 1. + */ +#define ISHM_NB_FRAGMNTS (ISHM_MAX_NB_BLOCKS * 2 + 1) + +/* + * A fragment describes a piece of the shared virtual address space, + * and is allocated only when allocation is done with the _ODP_ISHM_SINGLE_VA + * flag: + * A fragment is said to be used when it actually does represent some + * portion of the virtual address space, and is said to be unused when + * it does not (so at start, one single fragment is used -describing the + * whole address space as unallocated-, and all others are unused). + * Fragments get used as address space fragmentation increases. + * A fragment is allocated if the piece of address space it + * describes is actually used by a shared memory block. + * Allocated fragments get their block_index set >=0. + */ +typedef struct ishm_fragment { + struct ishm_fragment *prev; /* not used when the fragment is unused */ + struct ishm_fragment *next; + void *start; /* start of segment (VA) */ + uintptr_t len; /* length of segment. multiple of page size */ + int block_index; /* -1 for unallocated fragments */ +} ishm_fragment_t; + +/* + * A block describes a piece of reserved memory: Any successful ishm_reserve() + * will allocate a block. A ishm_reserve() with the _ODP_ISHM_SINGLE_VA flag set + * will allocate both a block and a fragment. + * Blocks contain only global data common to all processes. + */ +typedef struct ishm_block { + char name[ISHM_NAME_MAXLEN]; /* name for the ishm block (if any) */ + char filename[ISHM_FILENAME_MAXLEN]; /* name of the .../odp-* file */ + int main_odpthread; /* The thread which did the initial reserve*/ + uint32_t user_flags; /* any flags the user want to remember. */ + uint32_t flags; /* block creation flags. */ + uint64_t user_len; /* length, as requested at reserve time. */ + void *start; /* only valid if _ODP_ISHM_SINGLE_VA is set*/ + uint64_t len; /* length. multiple of page size. 0 if free*/ + ishm_fragment_t *fragment; /* used when _ODP_ISHM_SINGLE_VA is used */ + int huge; /* true if this segment is mapped using huge pages */ + uint64_t seq; /* sequence number, incremented on alloc and free */ + uint64_t refcnt;/* number of linux processes mapping this block */ +} ishm_block_t; + +/* + * Table of blocks describing allocated internal shared memory + * This table is visible to every ODP thread (linux process or pthreads). + * (it is allocated shared at odp init time and is therefore inherited by all) + * Table index is used as handle, so it cannot move!. Entry is regarded as + * free when len==0 + */ +typedef struct { + odp_spinlock_t lock; + uint64_t dev_seq; /* used when creating device names */ + ishm_block_t block[ISHM_MAX_NB_BLOCKS]; +} ishm_table_t; +static ishm_table_t *ishm_tbl; + +/* + * Process local table containing the list of (believed) allocated blocks seen + * from the current process. There is one such table per linux process. linux + * threads within a process shares this table. + * The contents within this table may become obsolete when other processes + * reserve/free ishm blocks. This is what the procsync() function + * catches by comparing the block sequence number with the one in this table. + * This table is filled at ishm_reserve and ishm_lookup time. + * Entries are removed at ishm_free or procsync time. + * Note that flags and len are present in this table and seems to be redundant + * with those present in the ishm block table: but this is not fully true: + * When ishm_sync() detects obsolete mappings and tries to remove them, + * the entry in the ishm block table is then obsolete, and the values which are + * found in this table must be used to perform the ummap. + * (and the values in the block tables are needed at lookup time...) + */ +typedef struct { + int thrd_refcnt; /* number of pthreads in this process, really */ + struct { + int block_index; /* entry in the ishm_tbl */ + uint32_t flags; /* flags used at creation time */ + uint64_t seq; + void *start; /* start of block (VA) */ + uint64_t len; /* length of block. multiple of page size */ + int fd; /* file descriptor used for this block */ + } entry[ISHM_MAX_NB_BLOCKS]; + int nb_entries; +} ishm_proctable_t; +static ishm_proctable_t *ishm_proctable; + +/* + * Table of fragments describing the common virtual address space: + * This table is visible to every ODP thread (linux process or pthreads). + * (it is allocated at odp init time and is therefore inherited by all) + */ +typedef struct { + ishm_fragment_t fragment[ISHM_NB_FRAGMNTS]; + ishm_fragment_t *used_fragmnts; /* ordered by increasing start addr */ + ishm_fragment_t *unused_fragmnts; +} ishm_ftable_t; +static ishm_ftable_t *ishm_ftbl; + +#ifndef MAP_ANONYMOUS +#define MAP_ANONYMOUS MAP_ANON +#endif + +/* prototypes: */ +static void procsync(void); + +/* + * Take a piece of the preallocated virtual space to fit "size" bytes. + * (best fit). Size must be rounded up to an integer number of pages size. + * Possibly split the fragment to keep track of remaining space. + * Returns the allocated fragment (best_fragmnt) and the corresponding address. + * External caller must ensure mutex before the call! + */ +static void *alloc_fragment(uintptr_t size, int block_index, intptr_t align, + ishm_fragment_t **best_fragmnt) +{ + ishm_fragment_t *fragmnt; + *best_fragmnt = NULL; + ishm_fragment_t *rem_fragmnt; + uintptr_t border;/* possible start of new fragment (next alignement) */ + intptr_t left; /* room remaining after, if the segment is allocated */ + uintptr_t remainder = ODP_CONFIG_ISHM_VA_PREALLOC_SZ; + + /* + * search for the best bit, i.e. search for the unallocated fragment + * would give less remainder if the new fragment was allocated within + * it: + */ + for (fragmnt = ishm_ftbl->used_fragmnts; + fragmnt; fragmnt = fragmnt->next) { + /* skip allocated segment: */ + if (fragmnt->block_index >= 0) + continue; + /* skip too short segment: */ + border = ((uintptr_t)fragmnt->start + align - 1) & (-align); + left = + ((uintptr_t)fragmnt->start + fragmnt->len) - (border + size); + if (left < 0) + continue; + /* remember best fit: */ + if ((uintptr_t)left < remainder) { + remainder = left; /* best, so far */ + *best_fragmnt = fragmnt; + } + } + + if (!(*best_fragmnt)) { + ODP_ERR("unable to get virtual address for shmem block!\n."); + return NULL; + } + + (*best_fragmnt)->block_index = block_index; + border = ((uintptr_t)(*best_fragmnt)->start + align - 1) & (-align); + + /* + * if there is room between previous fragment and new one, (due to + * alignement requirement) then fragment (split) the space between + * the end of the previous fragment and the beginning of the new one: + */ + if (border - (uintptr_t)(*best_fragmnt)->start > 0) { + /* frangment space, i.e. take a new fragment descriptor... */ + rem_fragmnt = ishm_ftbl->unused_fragmnts; + if (!rem_fragmnt) { + ODP_ERR("unable to get shmem fragment descriptor!\n."); + return NULL; + } + ishm_ftbl->unused_fragmnts = rem_fragmnt->next; + + /* and link it between best_fragmnt->prev and best_fragmnt */ + if ((*best_fragmnt)->prev) + (*best_fragmnt)->prev->next = rem_fragmnt; + else + ishm_ftbl->used_fragmnts = rem_fragmnt; + rem_fragmnt->prev = (*best_fragmnt)->prev; + (*best_fragmnt)->prev = rem_fragmnt; + rem_fragmnt->next = (*best_fragmnt); + + /* update length: rem_fragmnt getting space before border */ + rem_fragmnt->block_index = -1; + rem_fragmnt->start = (*best_fragmnt)->start; + rem_fragmnt->len = border - (uintptr_t)(*best_fragmnt)->start; + (*best_fragmnt)->start = + (void *)((uintptr_t)rem_fragmnt->start + rem_fragmnt->len); + (*best_fragmnt)->len -= rem_fragmnt->len; + } + + /* if this was a perfect fit, i.e. no free space follows, we are done */ + if (remainder == 0) + return (*best_fragmnt)->start; + + /* otherwise, frangment space, i.e. take a new fragment descriptor... */ + rem_fragmnt = ishm_ftbl->unused_fragmnts; + if (!rem_fragmnt) { + ODP_ERR("unable to get shmem fragment descriptor!\n."); + return (*best_fragmnt)->start; + } + ishm_ftbl->unused_fragmnts = rem_fragmnt->next; + + /* ... double link it... */ + rem_fragmnt->next = (*best_fragmnt)->next; + rem_fragmnt->prev = (*best_fragmnt); + if ((*best_fragmnt)->next) + (*best_fragmnt)->next->prev = rem_fragmnt; + (*best_fragmnt)->next = rem_fragmnt; + + /* ... and keep track of the remainder */ + (*best_fragmnt)->len = size; + rem_fragmnt->len = remainder; + rem_fragmnt->start = (void *)((char *)(*best_fragmnt)->start + size); + rem_fragmnt->block_index = -1; + + return (*best_fragmnt)->start; +} + +/* + * Free a portion of virtual space. + * Possibly defragment, if the freed fragment is adjacent to another + * free virtual fragment. + * External caller must ensure mutex before the call! + */ +static void free_fragment(ishm_fragment_t *fragmnt) +{ + ishm_fragment_t *prev_f; + ishm_fragment_t *next_f; + + /* sanity check */ + if (!fragmnt) + return; + + prev_f = fragmnt->prev; + next_f = fragmnt->next; + + /* free the fragment */ + fragmnt->block_index = -1; + + /* check if the previous fragment is also free: if so, defragment */ + if (prev_f && (prev_f->block_index < 0)) { + fragmnt->start = prev_f->start; + fragmnt->len += prev_f->len; + if (prev_f->prev) { + prev_f->prev->next = fragmnt; + } else { + if (ishm_ftbl->used_fragmnts == prev_f) + ishm_ftbl->used_fragmnts = fragmnt; + else + ODP_ERR("corrupted fragment list!.\n"); + } + fragmnt->prev = prev_f->prev; + + /* put removed fragment in free list */ + prev_f->prev = NULL; + prev_f->next = ishm_ftbl->unused_fragmnts; + ishm_ftbl->unused_fragmnts = prev_f; + } + + /* check if the next fragment is also free: if so, defragment */ + if (next_f && (next_f->block_index < 0)) { + fragmnt->len += next_f->len; + if (next_f->next) + next_f->next->prev = fragmnt; + fragmnt->next = next_f->next; + + /* put removed fragment in free list */ + next_f->prev = NULL; + next_f->next = ishm_ftbl->unused_fragmnts; + ishm_ftbl->unused_fragmnts = next_f; + } +} + +/* + * Create file with size len. returns -1 on error + * Creates a file to /tmp/odp-<pid>-<sequence_or_name> (for normal pages) + * or /mnt/huge/odp-<pid>-<sequence_or_name> (for huge pages) + * Return the new file descriptor, or -1 on error. + */ +static int create_file(int block_index, int huge, uint64_t len) +{ + char *name; + int fd; + ishm_block_t *new_block; /* entry in the main block table */ + char seq_string[ISHM_FILENAME_MAXLEN]; /* used to construct filename*/ + char filename[ISHM_FILENAME_MAXLEN];/* filename in /tmp/ or /mnt/huge */ + int oflag = O_RDWR | O_CREAT | O_TRUNC; /* flags for open */ + + new_block = &ishm_tbl->block[block_index]; + name = new_block->name; + + /* create the filename: */ + snprintf(seq_string, ISHM_FILENAME_MAXLEN, "%08" PRIu64, + ishm_tbl->dev_seq++); + + /* huge dir must be known to create files there!: */ + if (huge && !odp_global_data.hugepage_info.default_huge_page_dir) + return -1; + + if (huge) + snprintf(filename, ISHM_FILENAME_MAXLEN, + ISHM_FILENAME_FORMAT, + odp_global_data.hugepage_info.default_huge_page_dir, + odp_global_data.main_pid, + (name && name[0]) ? name : seq_string); + else + snprintf(filename, ISHM_FILENAME_MAXLEN, + ISHM_FILENAME_FORMAT, + ISHM_FILENAME_NORMAL_PAGE_DIR, + odp_global_data.main_pid, + (name && name[0]) ? name : seq_string); + + fd = open(filename, oflag, S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH); + if (fd < 0) { + ODP_ERR("open failed for %s: %s.\n", + filename, strerror(errno)); + return -1; + } + + if (ftruncate(fd, len) == -1) { + ODP_ERR("ftruncate failed: fd=%d, err=%s.\n", + fd, strerror(errno)); + close(fd); + return -1; + } + + strncpy(new_block->filename, filename, ISHM_FILENAME_MAXLEN - 1); + + return fd; +} + +/* + * performs the mapping, possibly allocating a fragment of the pre-reserved + * VA space if the _ODP_ISHM_SINGLE_VA flag was given. + * Sets fd, and returns the mapping address. + * This funstion will also set the _ODP_ISHM_SINGLE_VA flag if the alignment + * requires it + * Mutex must be assured by the caller. + */ +static void *do_map(int block_index, uint64_t len, uint32_t align, + uint32_t flags, int huge, int *fd) +{ + ishm_block_t *new_block; /* entry in the main block table */ + void *addr = NULL; + void *mapped_addr; + ishm_fragment_t *fragment = NULL; + + new_block = &ishm_tbl->block[block_index]; + + /* + * Creates a file to /tmp/odp-<pid>-<sequence> (for normal pages) + * or /mnt/huge/odp-<pid>-<sequence> (for huge pages) + * unless a fd was already given + */ + if (*fd < 0) { + *fd = create_file(block_index, huge, len); + if (*fd < 0) + return NULL; + } else { + new_block->filename[0] = 0; + } + + /* allocate an address range in the prebooked VA area if needed */ + if (flags & _ODP_ISHM_SINGLE_VA) { + addr = alloc_fragment(len, block_index, align, &fragment); + if (!addr) { + ODP_ERR("alloc_fragment failed.\n"); + if (new_block->filename[0]) { + close(*fd); + *fd = -1; + unlink(new_block->filename); + } + return NULL; + } + ishm_tbl->block[block_index].fragment = fragment; + } + + /* try to mmap: */ + mapped_addr = _odp_ishmphy_map(*fd, addr, len, flags); + if (mapped_addr == NULL) { + if (flags & _ODP_ISHM_SINGLE_VA) + free_fragment(fragment); + if (new_block->filename[0]) { + close(*fd); + *fd = -1; + unlink(new_block->filename); + } + return NULL; + } + + new_block->huge = huge; + + return mapped_addr; +} + +/* + * Performs an extra mapping (for a process trying to see an existing block + * i.e. performing a lookup). + * Mutex must be assured by the caller. + */ +static void *do_remap(int block_index, int fd) +{ + void *mapped_addr; + ishm_fragment_t *fragment; + uint64_t len; + uint32_t flags; + + len = ishm_tbl->block[block_index].len; + flags = ishm_tbl->block[block_index].flags; + + if (flags & _ODP_ISHM_SINGLE_VA) { + fragment = ishm_tbl->block[block_index].fragment; + if (!fragment) { + ODP_ERR("invalid fragment failure.\n"); + return NULL; + } + + /* try to mmap: */ + mapped_addr = _odp_ishmphy_map(fd, fragment->start, len, flags); + if (mapped_addr == NULL) + return NULL; + return mapped_addr; + } + + /* try to mmap: */ + mapped_addr = _odp_ishmphy_map(fd, NULL, len, flags); + if (mapped_addr == NULL) + return NULL; + + return mapped_addr; +} + +/* + * Performs unmapping, possibly freeing a prereserved VA space fragment, + * if the _ODP_ISHM_SINGLE_VA flag was set at alloc time + * Mutex must be assured by the caller. + */ +static int do_unmap(void *start, uint64_t size, uint32_t flags, + int block_index) +{ + int ret; + + if (start) + ret = _odp_ishmphy_unmap(start, size, flags); + else + ret = 0; + + if ((block_index >= 0) && (flags & _ODP_ISHM_SINGLE_VA)) { + /* mark reserved address space as free */ + free_fragment(ishm_tbl->block[block_index].fragment); + } + + return ret; +} + +/* + * Search for a given used and allocated block name. + * (search is performed in the global ishm table) + * Returns the index of the found block (if any) or -1 if none. + * Mutex must be assured by the caller. + */ +static int find_block_by_name(const char *name) +{ + int i; + + if (name == NULL || name[0] == 0) + return -1; + + for (i = 0; i < ISHM_MAX_NB_BLOCKS; i++) { + if ((ishm_tbl->block[i].len) && + (strcmp(name, ishm_tbl->block[i].name) == 0)) + return i; + } + + return -1; +} + +/* + * Search for a block by address (only works when flag _ODP_ISHM_SINGLE_VA + * was set at reserve() time, or if the block is already known by this + * process). + * Search is performed in the process table and in the global ishm table. + * The provided address does not have to be at start: any address + * within the fragment is OK. + * Returns the index to the found block (if any) or -1 if none. + * Mutex must be assured by the caller. + */ +static int find_block_by_address(void *addr) +{ + int block_index; + int i; + ishm_fragment_t *fragmnt; + + /* + * first check if there is already a process known block for this + * address + */ + for (i = 0; i < ishm_proctable->nb_entries; i++) { + block_index = ishm_proctable->entry[i].block_index; + if ((addr > ishm_proctable->entry[i].start) && + ((char *)addr < ((char *)ishm_proctable->entry[i].start + + ishm_tbl->block[block_index].len))) + return block_index; + } + + /* + * then check if there is a existing single VA block known by some other + * process and containing the given address + */ + for (i = 0; i < ISHM_MAX_NB_BLOCKS; i++) { + if ((!ishm_tbl->block[i].len) || + (!(ishm_tbl->block[i].flags & _ODP_ISHM_SINGLE_VA))) + continue; + fragmnt = ishm_tbl->block[i].fragment; + if (!fragmnt) { + ODP_ERR("find_fragment: invalid NULL fragment\n"); + return -1; + } + if ((addr >= fragmnt->start) && + ((char *)addr < ((char *)fragmnt->start + fragmnt->len))) + return i; + } + + /* address does not belong to any accessible block: */ + return -1; +} + +/* + * Search a given ishm block in the process local table. Return its index + * in the process table or -1 if not found (meaning that the ishm table + * block index was not referenced in the process local table, i.e. the + * block is known by some other process, but not by the current process). + * Caller must assure mutex. + */ +static int procfind_block(int block_index) +{ + int i; + + for (i = 0; i < ishm_proctable->nb_entries; i++) { + if (ishm_proctable->entry[i].block_index == block_index) + return i; + } + return -1; +} + +/* + * Release the physical memory mapping for blocks which have been freed + * by other processes. Caller must ensure mutex. + * Mutex must be assured by the caller. + */ +static void procsync(void) +{ + int i = 0; + int last; + ishm_block_t *block; + + last = ishm_proctable->nb_entries; + while (i < last) { + /* if the procecess sequence number doesn't match the main + * table seq number, this entry is obsolete + */ + block = &ishm_tbl->block[ishm_proctable->entry[i].block_index]; + if (ishm_proctable->entry[i].seq != block->seq) { + /* obsolete entry: free memory and remove proc entry */ + close(ishm_proctable->entry[i].fd); + _odp_ishmphy_unmap(ishm_proctable->entry[i].start, + ishm_proctable->entry[i].len, + ishm_proctable->entry[i].flags); + ishm_proctable->entry[i] = + ishm_proctable->entry[--last]; + } else { + i++; + } + } + ishm_proctable->nb_entries = last; +} + +/* + * Allocate and map internal shared memory, or other objects: + * If a name is given, check that this name is not already in use. + * If ok, allocate a new shared memory block and map the + * provided fd in it (if fd >=0 was given). + * If no fd is provided, a shared memory file desc named + * /tmp/odp-<pid>-ishm-<name_or_sequence> is created and mapped. + * (the name is different for huge page file as they must be on hugepagefs) + * The function returns the index of the newly created block in the + * main block table (>=0) or -1 on error. + */ +int _odp_ishm_reserve(const char *name, uint64_t size, int fd, + uint32_t align, uint32_t flags, uint32_t user_flags) +{ + int new_index; /* index in the main block table*/ + ishm_block_t *new_block; /* entry in the main block table*/ + uint64_t page_sz; /* normal page size. usually 4K*/ + uint64_t alloc_size; /* includes extra for alignement*/ + uint64_t page_hp_size; /* huge page size */ + uint64_t alloc_hp_size; /* includes extra for alignement*/ + uint32_t hp_align; + uint64_t len; /* mapped length */ + void *addr = NULL; /* mapping address */ + int new_proc_entry; + + page_sz = odp_sys_page_size(); + + odp_spinlock_lock(&ishm_tbl->lock); + + /* update this process view... */ + procsync(); + + /* roundup to page size */ + alloc_size = (size + (page_sz - 1)) & (-page_sz); + + page_hp_size = odp_sys_huge_page_size(); + /* roundup to page size */ + alloc_hp_size = (size + (page_hp_size - 1)) & (-page_hp_size); + + /* check if name already exists */ + if (name && (find_block_by_name(name) >= 0)) { + /* Found a block with the same name */ + odp_spinlock_unlock(&ishm_tbl->lock); + ODP_ERR("name "%s" already used.\n", name); + return -1; + } + + /* grab a new entry: */ + for (new_index = 0; new_index < ISHM_MAX_NB_BLOCKS; new_index++) { + if (ishm_tbl->block[new_index].len == 0) { + /* Found free block */ + break; + } + } + + /* check if we have reached the maximum number of allocation: */ + if (new_index >= ISHM_MAX_NB_BLOCKS) { + odp_spinlock_unlock(&ishm_tbl->lock); + ODP_ERR("ISHM_MAX_NB_BLOCKS limit reached!\n"); + return -1; + } + + new_block = &ishm_tbl->block[new_index]; + + /* save block name (if any given): */ + if (name) + strncpy(new_block->name, name, ISHM_NAME_MAXLEN - 1); + else + new_block->name[0] = 0; + + /* Try first huge pages when possible and needed: */ + if (page_hp_size && (alloc_size > page_sz)) { + /* at least, alignment in VA should match page size, but user + * can request more: If the user requirement exceeds the page + * size then we have to make sure the block will be mapped at + * the same address every where, otherwise alignment may be + * be wrong for some process */ + hp_align = align; + if (hp_align < odp_sys_huge_page_size()) + hp_align = odp_sys_huge_page_size(); + else + flags |= _ODP_ISHM_SINGLE_VA; + len = alloc_hp_size; + addr = do_map(new_index, len, hp_align, flags, 1, &fd); + + if (addr == NULL) + ODP_DBG("No huge pages, fall back to normal pages, " + "check: /proc/sys/vm/nr_hugepages.\n"); + else + new_block->huge = 1; + } + + /* try normal pages if huge pages failed */ + if (addr == NULL) { + /* at least, alignment in VA should match page size, but user + * can request more: If the user requirement exceeds the page + * size then we have to make sure the block will be mapped at + * the same address every where, otherwise alignment may be + * be wrong for some process */ + if (align < odp_sys_page_size()) + align = odp_sys_page_size(); + else + flags |= _ODP_ISHM_SINGLE_VA; + + len = alloc_size; + addr = do_map(new_index, len, align, flags, 0, &fd); + new_block->huge = 0; + } + + /* if neither huge pages or normal pages works, we cannot proceed: */ + if ((addr == NULL) || (len == 0)) { + if ((new_block->filename[0]) && (fd >= 0)) + close(fd); + odp_spinlock_unlock(&ishm_tbl->lock); + ODP_ERR("_ishm_reserve failed.\n"); + return -1; + } + + /* remember block data and increment block seq number to mark change */ + new_block->len = len; + new_block->user_len = size; + new_block->flags = flags; + new_block->user_flags = user_flags; + new_block->seq++; + new_block->refcnt = 1; + new_block->main_odpthread = odp_thread_id(); + new_block->start = addr; /* only for SINGLE_VA*/ + + /* the allocation succeeded: update the process local view */ + new_proc_entry = ishm_proctable->nb_entries++; + ishm_proctable->entry[new_proc_entry].block_index = new_index; + ishm_proctable->entry[new_proc_entry].flags = flags; + ishm_proctable->entry[new_proc_entry].seq = new_block->seq; + ishm_proctable->entry[new_proc_entry].start = addr; + ishm_proctable->entry[new_proc_entry].len = len; + ishm_proctable->entry[new_proc_entry].fd = fd; + + /* register the file descriptor to the file descriptor server. */ + _odp_fdserver_register_fd(FD_SRV_CTX_ISHM, new_index, fd); + + odp_spinlock_unlock(&ishm_tbl->lock); + return new_index; +} + +/* + * Free and unmap internal shared memory: + * The file descriptor is closed and the .../odp-* file deleted, + * unless fd was externally provided at reserve() time. + * return 0 if OK, and -1 on error. + * Mutex must be assured by the caller. + */ +static int block_free(int block_index) +{ + int proc_index; + ishm_block_t *block; /* entry in the main block table*/ + int last; + + if ((block_index < 0) || + (block_index >= ISHM_MAX_NB_BLOCKS) || + (ishm_tbl->block[block_index].len == 0)) { + ODP_ERR("Request to free an invalid block\n"); + return -1; + } + + block = &ishm_tbl->block[block_index]; + + proc_index = procfind_block(block_index); + if (proc_index >= 0) { + /* close the fd, unless if it was externaly provided */ + if ((block->filename[0] != 0) || + (odp_thread_id() != block->main_odpthread)) + close(ishm_proctable->entry[proc_index].fd); + + /* remove the mapping and possible fragment */ + do_unmap(ishm_proctable->entry[proc_index].start, + block->len, + ishm_proctable->entry[proc_index].flags, + block_index); + + /* remove entry from process local table: */ + last = ishm_proctable->nb_entries - 1; + ishm_proctable->entry[proc_index] = + ishm_proctable->entry[last]; + ishm_proctable->nb_entries = last; + } else { + /* just possibly free the fragment as no mapping exist here: */ + do_unmap(NULL, 0, block->flags, block_index); + } + + /* remove the .../odp-* file, unless fd was external: */ + if (block->filename[0] != 0) + unlink(block->filename); + + /* deregister the file descriptor from the file descriptor server. */ + _odp_fdserver_deregister_fd(FD_SRV_CTX_ISHM, block_index); + + /* mark the block as free in the main block table: */ + block->len = 0; + + /* mark the change so other processes see this entry as obsolete: */ + block->seq++; + + return 0; +} + +/* + * Free and unmap internal shared memory, intentified by its block number: + * return -1 on error. 0 if OK. + */ +int _odp_ishm_free_by_index(int block_index) +{ + int ret; + + odp_spinlock_lock(&ishm_tbl->lock); + procsync(); + + ret = block_free(block_index); + odp_spinlock_unlock(&ishm_tbl->lock); + return ret; +} + +/* + * free and unmap internal shared memory, intentified by its block name: + * return -1 on error. 0 if OK. + */ +int _odp_ishm_free_by_name(const char *name) +{ + int block_index; + int ret; + + odp_spinlock_lock(&ishm_tbl->lock); + procsync(); + + /* search the block in main ishm table */ + block_index = find_block_by_name(name); + if (block_index < 0) { + ODP_ERR("Request to free an non existing block..." + " (double free?)\n"); + odp_spinlock_unlock(&ishm_tbl->lock); + return -1; + } + + ret = block_free(block_index); + odp_spinlock_unlock(&ishm_tbl->lock); + return ret; +} + +/* + * Free and unmap internal shared memory identified by address: + * return -1 on error. 0 if OK. + */ +int _odp_ishm_free_by_address(void *addr) +{ + int block_index; + int ret; + + odp_spinlock_lock(&ishm_tbl->lock); + procsync(); + + /* search the block in main ishm table */ + block_index = find_block_by_address(addr); + if (block_index < 0) { + ODP_ERR("Request to free an non existing block..." + " (double free?)\n"); + odp_spinlock_unlock(&ishm_tbl->lock); + return -1; + } + + ret = block_free(block_index); + + odp_spinlock_unlock(&ishm_tbl->lock); + return ret; +} + +/* + * Lookup for an ishm shared memory, identified by its block index + * in the main ishm block table. + * Map this ishm area in the process VA (if not already present). + * Returns the block user address or NULL on error. + * Mutex must be assured by the caller. + */ +static void *block_lookup(int block_index) +{ + int proc_index; + int fd = -1; + ishm_block_t *block; + void *mapped_addr; + int new_entry; + + if ((block_index < 0) || + (block_index >= ISHM_MAX_NB_BLOCKS) || + (ishm_tbl->block[block_index].len == 0)) { + ODP_ERR("Request to lookup an invalid block\n"); + return NULL; + } + + /* search it in process table: if there, this process knows it already*/ + proc_index = procfind_block(block_index); + if (proc_index >= 0) + return ishm_proctable->entry[proc_index].start; + + /* this ishm is not known by this process, yet: we create the mapping.*/ + fd = _odp_fdserver_lookup_fd(FD_SRV_CTX_ISHM, block_index); + if (fd < 0) { + ODP_ERR("Could not find ishm file descriptor (BUG!)\n"); + return NULL; + } + + /* perform the mapping */ + block = &ishm_tbl->block[block_index]; + + mapped_addr = do_remap(block_index, fd); + if (mapped_addr == NULL) { + ODP_ERR(" lookup: Could not map existing shared memory!\n"); + return NULL; + } + + /* the mapping succeeded: update the process local view */ + new_entry = ishm_proctable->nb_entries++; + ishm_proctable->entry[new_entry].block_index = block_index; + ishm_proctable->entry[new_entry].flags = block->flags; + ishm_proctable->entry[new_entry].seq = block->seq; + ishm_proctable->entry[new_entry].start = mapped_addr; + ishm_proctable->entry[new_entry].len = block->len; + ishm_proctable->entry[new_entry].fd = fd; + block->refcnt++; + + return mapped_addr; +} + +/* + * Lookup for an ishm shared memory, identified by its block_index. + * Maps this ishmem area in the process VA (if not already present). + * Returns the block user address, or NULL if the index + * does not match any known ishm blocks. + */ +void *_odp_ishm_lookup_by_index(int block_index) +{ + void *ret; + + odp_spinlock_lock(&ishm_tbl->lock); + procsync(); + + ret = block_lookup(block_index); + odp_spinlock_unlock(&ishm_tbl->lock); + return ret; +} + +/* + * Lookup for an ishm shared memory, identified by its block name. + * Map this ishm area in the process VA (if not already present). + * Return the block index, or -1 if the index + * does not match any known ishm blocks. + */ +int _odp_ishm_lookup_by_name(const char *name) +{ + int block_index; + + odp_spinlock_lock(&ishm_tbl->lock); + procsync(); + + /* search the block in main ishm table: return -1 if not found: */ + block_index = find_block_by_name(name); + if ((block_index < 0) || (!block_lookup(block_index))) { + odp_spinlock_unlock(&ishm_tbl->lock); + return -1; + } + + odp_spinlock_unlock(&ishm_tbl->lock); + return block_index; +} + +/* + * Lookup for an ishm shared memory block, identified by its VA address. + * This works only if the block has already been looked-up (mapped) by the + * current process or it it was created with the _ODP_ISHM_SINGLE_VA flag. + * Map this ishm area in the process VA (if not already present). + * Return the block index, or -1 if the address + * does not match any known ishm blocks. + */ +int _odp_ishm_lookup_by_address(void *addr) +{ + int block_index; + + odp_spinlock_lock(&ishm_tbl->lock); + procsync(); + + /* search the block in main ishm table: return -1 if not found: */ + block_index = find_block_by_address(addr); + if ((block_index < 0) || (!block_lookup(block_index))) { + odp_spinlock_unlock(&ishm_tbl->lock); + return -1; + } + + odp_spinlock_unlock(&ishm_tbl->lock); + return block_index; +} + +/* + * Returns the VA address of a given block (which has to be known in the current + * process). Returns NULL if the block is unknown. + */ +void *_odp_ishm_address(int block_index) +{ + int proc_index; + void *addr; + + odp_spinlock_lock(&ishm_tbl->lock); + procsync(); + + if ((block_index < 0) || + (block_index >= ISHM_MAX_NB_BLOCKS) || + (ishm_tbl->block[block_index].len == 0)) { + ODP_ERR("Request for address on an invalid block\n"); + odp_spinlock_unlock(&ishm_tbl->lock); + return NULL; + } + + proc_index = procfind_block(block_index); + if (proc_index < 0) { + odp_spinlock_unlock(&ishm_tbl->lock); + return NULL; + } + + addr = ishm_proctable->entry[proc_index].start; + odp_spinlock_unlock(&ishm_tbl->lock); + return addr; +} + +int _odp_ishm_info(int block_index, _odp_ishm_info_t *info) +{ + int proc_index; + + odp_spinlock_lock(&ishm_tbl->lock); + procsync(); + + if ((block_index < 0) || + (block_index >= ISHM_MAX_NB_BLOCKS) || + (ishm_tbl->block[block_index].len == 0)) { + odp_spinlock_unlock(&ishm_tbl->lock); + ODP_ERR("Request for info on an invalid block\n"); + return -1; + } + + /* search it in process table: if not there, need to map*/ + proc_index = procfind_block(block_index); + if (proc_index < 0) { + odp_spinlock_unlock(&ishm_tbl->lock); + return -1; + } + + info->name = ishm_tbl->block[block_index].name; + info->addr = ishm_proctable->entry[proc_index].start; + info->size = ishm_tbl->block[block_index].user_len; + info->page_size = ishm_tbl->block[block_index].huge ? + odp_sys_huge_page_size() : odp_sys_page_size(); + info->flags = ishm_tbl->block[block_index].flags; + info->user_flags = ishm_tbl->block[block_index].user_flags; + + odp_spinlock_unlock(&ishm_tbl->lock); + return 0; +} + +int _odp_ishm_init_global(void) +{ + void *addr; + void *spce_addr; + int i; + + if (!odp_global_data.hugepage_info.default_huge_page_dir) + ODP_DBG("NOTE: No support for huge pages\n"); + else + ODP_DBG("Huge pages mount point is: %s\n", + odp_global_data.hugepage_info.default_huge_page_dir); + + /* allocate space for the internal shared mem block table: */ + addr = mmap(NULL, sizeof(ishm_table_t), + PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0); + if (addr == MAP_FAILED) { + ODP_ERR("unable to mmap the main block table\n."); + goto init_glob_err1; + } + ishm_tbl = addr; + memset(ishm_tbl, 0, sizeof(ishm_table_t)); + ishm_tbl->dev_seq = 0; + odp_spinlock_init(&ishm_tbl->lock); + + /* allocate space for the internal shared mem fragment table: */ + addr = mmap(NULL, sizeof(ishm_ftable_t), + PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0); + if (addr == MAP_FAILED) { + ODP_ERR("unable to mmap the main fragment table\n."); + goto init_glob_err2; + } + ishm_ftbl = addr; + memset(ishm_ftbl, 0, sizeof(ishm_ftable_t)); + + /* + *reserve the address space for _ODP_ISHM_SINGLE_VA reserved blocks, + * only address space! + */ + spce_addr = _odp_ishmphy_book_va(ODP_CONFIG_ISHM_VA_PREALLOC_SZ, + odp_sys_huge_page_size()); + if (!spce_addr) { + ODP_ERR("unable to reserve virtual space\n."); + goto init_glob_err3; + } + + /* use the first fragment descriptor to describe to whole VA space: */ + ishm_ftbl->fragment[0].block_index = -1; + ishm_ftbl->fragment[0].start = spce_addr; + ishm_ftbl->fragment[0].len = ODP_CONFIG_ISHM_VA_PREALLOC_SZ; + ishm_ftbl->fragment[0].prev = NULL; + ishm_ftbl->fragment[0].next = NULL; + ishm_ftbl->used_fragmnts = &ishm_ftbl->fragment[0]; + + /* and put all other fragment descriptors in the unused list: */ + for (i = 1; i < ISHM_NB_FRAGMNTS - 1; i++) { + ishm_ftbl->fragment[i].prev = NULL; + ishm_ftbl->fragment[i].next = &ishm_ftbl->fragment[i + 1]; + } + ishm_ftbl->fragment[ISHM_NB_FRAGMNTS - 1].prev = NULL; + ishm_ftbl->fragment[ISHM_NB_FRAGMNTS - 1].next = NULL; + ishm_ftbl->unused_fragmnts = &ishm_ftbl->fragment[1]; + + return 0; + +init_glob_err3: + if (munmap(ishm_ftbl, sizeof(ishm_ftable_t)) < 0) + ODP_ERR("unable to munmap main fragment table\n."); +init_glob_err2: + if (munmap(ishm_tbl, sizeof(ishm_table_t)) < 0) + ODP_ERR("unable to munmap main block table\n."); +init_glob_err1: + return -1; +} + +int _odp_ishm_init_local(void) +{ + int i; + int block_index; + + /* + * the ishm_process table is local to each linux process + * Check that no other linux threads (of same or ancestor processes) + * have already created the table, and create it if needed. + * We protect this with the general ishm lock to avoid + * init race condition of different running threads. + */ + odp_spinlock_lock(&ishm_tbl->lock); + if (!ishm_proctable) { + ishm_proctable = malloc(sizeof(ishm_proctable_t)); + if (!ishm_proctable) { + odp_spinlock_unlock(&ishm_tbl->lock); + return -1; + } + memset(ishm_proctable, 0, sizeof(ishm_proctable_t)); + } + if (syscall(SYS_gettid) != getpid()) + ishm_proctable->thrd_refcnt++; /* new linux thread */ + else + ishm_proctable->thrd_refcnt = 1;/* new linux process */ + + /* + * if this ODP thread is actually a new linux process, (as opposed + * to a pthread), i.e, we just forked, then all shmem blocks + * of the parent process are mapped into this child by inheritance. + * (The process local table is inherited as well). We hence have to + * increase the process refcount for each of the inherited mappings: + */ + if (syscall(SYS_gettid) == getpid()) { + for (i = 0; i < ishm_proctable->nb_entries; i++) { + block_index = ishm_proctable->entry[i].block_index; + ishm_tbl->block[block_index].refcnt++; + } + } + + odp_spinlock_unlock(&ishm_tbl->lock); + return 0; +} + +int _odp_ishm_term_global(void) +{ + int ret = 0; + + /* free the fragment table */ + if (munmap(ishm_ftbl, sizeof(ishm_ftable_t)) < 0) { + ret = -1; + ODP_ERR("unable to munmap fragment table\n."); + } + /* free the block table */ + if (munmap(ishm_tbl, sizeof(ishm_table_t)) < 0) { + ret = -1; + ODP_ERR("unable to munmap main table\n."); + } + + /* free the reserved VA space */ + if (_odp_ishmphy_unbook_va()) + ret = -1; + + return ret; +} + +int _odp_ishm_term_local(void) +{ + int i; + int proc_table_refcnt = 0; + int block_index; + ishm_block_t *block; + + odp_spinlock_lock(&ishm_tbl->lock); + procsync(); + + /* + * The ishm_process table is local to each linux process + * Check that no other linux threads (of this linux process) + * still needs the table, and free it if so. + * We protect this with the general ishm lock to avoid + * term race condition of different running threads. + */ + proc_table_refcnt = --ishm_proctable->thrd_refcnt; + if (!proc_table_refcnt) { + /* + * this is the last thread of this process... + * All mappings for this process are about to be lost... + * Go through the table of visible blocks for this process, + * decreasing the refcnt of each visible blocks, and issuing + * warning for those no longer referenced by any process. + * Note that non-referenced blocks are nor freeed: this is + * deliberate as this would imply that the sementic of the + * freeing function would differ depending on whether we run + * with odp_thread as processes or pthreads. With this approach, + * the user should always free the blocks manually, which is + * more consistent + */ + for (i = 0; i < ishm_proctable->nb_entries; i++) { + block_index = ishm_proctable->entry[i].block_index; + block = &ishm_tbl->block[block_index]; + if ((--block->refcnt) <= 0) { + block->refcnt = 0; + ODP_DBG("Warning: block %d: name:%s " + "no longer referenced\n", + i, + ishm_tbl->block[i].name[0] ? + ishm_tbl->block[i].name : "<no name>"); + } + } + + free(ishm_proctable); + ishm_proctable = NULL; + } + + odp_spinlock_unlock(&ishm_tbl->lock); + return 0; +} diff --git a/platform/linux-generic/_ishmphy.c b/platform/linux-generic/_ishmphy.c new file mode 100644 index 0000000..2b2d100 --- /dev/null +++ b/platform/linux-generic/_ishmphy.c @@ -0,0 +1,185 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +/* + * This file handles the lower end of the ishm memory allocator: + * It performs the physical mappings. + */ +#include <odp_posix_extensions.h> +#include <odp_config_internal.h> +#include <odp_internal.h> +#include <odp/api/align.h> +#include <odp/api/system_info.h> +#include <odp/api/debug.h> +#include <odp_debug_internal.h> +#include <odp_align_internal.h> +#include <_ishm_internal.h> +#include <_ishmphy_internal.h> + +#include <stdlib.h> +#include <stdio.h> +#include <unistd.h> +#include <string.h> +#include <errno.h> +#include <sys/mman.h> +#include <sys/stat.h> +#include <fcntl.h> +#include <sys/types.h> +#include <sys/wait.h> +#include <_ishmphy_internal.h> + +static void *common_va_address; +static uint64_t common_va_len; + +#ifndef MAP_ANONYMOUS +#define MAP_ANONYMOUS MAP_ANON +#endif + +/* Book some virtual address space + * This function is called at odp_init_global() time to pre-book some + * virtual address space inherited by all odpthreads (i.e. descendant + * processes and threads) and later used to guarantee the unicity the + * the mapping VA address when memory is reserver with the _ODP_ISHM_SINGLE_VA + * flag. + * returns the address of the mapping or NULL on error. + */ +void *_odp_ishmphy_book_va(uintptr_t len, intptr_t align) +{ + void *addr; + + addr = mmap(NULL, len + align, PROT_NONE, + MAP_SHARED | MAP_ANONYMOUS | MAP_NORESERVE, -1, 0); + if (addr == MAP_FAILED) { + ODP_ERR("_ishmphy_book_va failure\n"); + return NULL; + } + + if (mprotect(addr, len, PROT_NONE)) + ODP_ERR("failure for protect\n"); + + ODP_DBG("VA Reserved: %p, len=%p\n", addr, len + align); + + common_va_address = addr; + common_va_len = len; + + /* return the nearest aligned address: */ + return (void *)(((uintptr_t)addr + align - 1) & (-align)); +} + +/* Un-book some virtual address space + * This function is called at odp_term_global() time to unbook + * the virtual address space booked by _ishmphy_book_va() + */ +int _odp_ishmphy_unbook_va(void) +{ + int ret; + + ret = munmap(common_va_address, common_va_len); + if (ret) + ODP_ERR("_unishmphy_book_va failure\n"); + return ret; +} + +/* + * do a mapping: + * Performs a mapping of the provided file descriptor to the process VA + * space. If the _ODP_ISHM_SINGLE_VA flag is set, 'start' is assumed to be + * the VA address where the mapping is to be done. + * If the flag is not set, a new VA address is taken. + * returns the address of the mapping or NULL on error. + */ +void *_odp_ishmphy_map(int fd, void *start, uint64_t size, + int flags) +{ + void *mapped_addr; + int mmap_flags = 0; + + if (flags & _ODP_ISHM_SINGLE_VA) { + if (!start) { + ODP_ERR("failure: missing address\n"); + return NULL; + } + /* maps over fragment of reserved VA: */ + mapped_addr = mmap(start, size, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_FIXED | mmap_flags, fd, 0); + /* if mapping fails, re-block the space we tried to take + * as it seems a mapping failure still affect what was there??*/ + if (mapped_addr == MAP_FAILED) { + mmap_flags = MAP_SHARED | MAP_FIXED | + MAP_ANONYMOUS | MAP_NORESERVE; + mmap(start, size, PROT_NONE, mmap_flags, -1, 0); + mprotect(start, size, PROT_NONE); + } + } else { + /* just do a new mapping in the VA space: */ + mapped_addr = mmap(NULL, size, PROT_READ | PROT_WRITE, + MAP_SHARED | mmap_flags, fd, 0); + if ((mapped_addr >= common_va_address) && + ((char *)mapped_addr < + (char *)common_va_address + common_va_len)) { + ODP_ERR("VA SPACE OVERLAP!\n"); + } + } + + if (mapped_addr == MAP_FAILED) { + ODP_ERR("mmap failed:%s\n", strerror(errno)); + return NULL; + } + + /* if locking is requested, lock it...*/ + if (flags & _ODP_ISHM_LOCK) { + if (mlock(mapped_addr, size)) { + if (munmap(mapped_addr, size)) + ODP_ERR("munmap failed:%s\n", strerror(errno)); + ODP_ERR("mlock failed:%s\n", strerror(errno)); + return NULL; + } + } + return mapped_addr; +} + +/* free a mapping: + * If the _ODP_ISHM_SINGLE_VA flag was given at creation time the virtual + * address range must be returned to the preoallocated "pool". this is + * done by mapping non accessibly memory there (hence blocking the VA but + * releasing the physical memory). + * If the _ODP_ISHM_SINGLE_VA flag was not given, both physical memory and + * virtual address space are realeased by calling the normal munmap. + * return 0 on success or -1 on error. + */ +int _odp_ishmphy_unmap(void *start, uint64_t len, int flags) +{ + void *addr; + int ret; + int mmap_flgs; + + mmap_flgs = MAP_SHARED | MAP_FIXED | MAP_ANONYMOUS | MAP_NORESERVE; + + /* if locking was requested, unlock...*/ + if (flags & _ODP_ISHM_LOCK) + munlock(start, len); + + if (flags & _ODP_ISHM_SINGLE_VA) { + /* map unnaccessible memory overwrites previous mapping + * and free the physical memory, but guarantees to block + * the VA range from other mappings + */ + addr = mmap(start, len, PROT_NONE, mmap_flgs, -1, 0); + if (addr == MAP_FAILED) { + ODP_ERR("_ishmphy_free failure for ISHM_SINGLE_VA\n"); + return -1; + } + if (mprotect(start, len, PROT_NONE)) + ODP_ERR("_ishmphy_free failure for protect\n"); + return 0; + } + + /* just release the mapping */ + ret = munmap(start, len); + if (ret) + ODP_ERR("_ishmphy_free failure: %s\n", strerror(errno)); + return ret; +} diff --git a/platform/linux-generic/arch/arm/odp/api/cpu_arch.h b/platform/linux-generic/arch/arm/odp/api/cpu_arch.h deleted file mode 120000 index e86e132..0000000 --- a/platform/linux-generic/arch/arm/odp/api/cpu_arch.h +++ /dev/null @@ -1 +0,0 @@ -../../../default/odp/api/cpu_arch.h \ No newline at end of file diff --git a/platform/linux-generic/arch/arm/odp/api/cpu_arch.h b/platform/linux-generic/arch/arm/odp/api/cpu_arch.h new file mode 100644 index 0000000..22b1da2 --- /dev/null +++ b/platform/linux-generic/arch/arm/odp/api/cpu_arch.h @@ -0,0 +1,24 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#ifndef ODP_PLAT_CPU_ARCH_H_ +#define ODP_PLAT_CPU_ARCH_H_ + +#ifdef __cplusplus +extern "C" { +#endif + +#define _ODP_CACHE_LINE_SIZE 64 + +static inline void odp_cpu_pause(void) +{ +} + +#ifdef __cplusplus +} +#endif + +#endif diff --git a/platform/linux-generic/arch/arm/odp_cpu_arch.c b/platform/linux-generic/arch/arm/odp_cpu_arch.c deleted file mode 120000 index deebc47..0000000 --- a/platform/linux-generic/arch/arm/odp_cpu_arch.c +++ /dev/null @@ -1 +0,0 @@ -../default/odp_cpu_arch.c \ No newline at end of file diff --git a/platform/linux-generic/arch/arm/odp_cpu_arch.c b/platform/linux-generic/arch/arm/odp_cpu_arch.c new file mode 100644 index 0000000..2ac223e --- /dev/null +++ b/platform/linux-generic/arch/arm/odp_cpu_arch.c @@ -0,0 +1,48 @@ +/* Copyright (c) 2015, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#include <odp_posix_extensions.h> + +#include <stdlib.h> +#include <time.h> + +#include <odp/api/cpu.h> +#include <odp/api/hints.h> +#include <odp/api/system_info.h> +#include <odp_debug_internal.h> + +#define GIGA 1000000000 + +uint64_t odp_cpu_cycles(void) +{ + struct timespec time; + uint64_t sec, ns, hz, cycles; + int ret; + + ret = clock_gettime(CLOCK_MONOTONIC_RAW, &time); + + if (ret != 0) + ODP_ABORT("clock_gettime failed\n"); + + hz = odp_cpu_hz_max(); + sec = (uint64_t)time.tv_sec; + ns = (uint64_t)time.tv_nsec; + + cycles = sec * hz; + cycles += (ns * hz) / GIGA; + + return cycles; +} + +uint64_t odp_cpu_cycles_max(void) +{ + return UINT64_MAX; +} + +uint64_t odp_cpu_cycles_resolution(void) +{ + return 1; +} diff --git a/platform/linux-generic/arch/arm/odp_sysinfo_parse.c b/platform/linux-generic/arch/arm/odp_sysinfo_parse.c deleted file mode 120000 index 39962b8..0000000 --- a/platform/linux-generic/arch/arm/odp_sysinfo_parse.c +++ /dev/null @@ -1 +0,0 @@ -../default/odp_sysinfo_parse.c \ No newline at end of file diff --git a/platform/linux-generic/arch/arm/odp_sysinfo_parse.c b/platform/linux-generic/arch/arm/odp_sysinfo_parse.c new file mode 100644 index 0000000..53e2aae --- /dev/null +++ b/platform/linux-generic/arch/arm/odp_sysinfo_parse.c @@ -0,0 +1,27 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#include <odp_internal.h> +#include <odp_debug_internal.h> +#include <string.h> + +int cpuinfo_parser(FILE *file ODP_UNUSED, system_info_t *sysinfo) +{ + int i; + + ODP_DBG("Warning: use dummy values for freq and model string\n"); + for (i = 0; i < MAX_CPU_NUMBER; i++) { + sysinfo->cpu_hz_max[i] = 1400000000; + strcpy(sysinfo->model_str[i], "UNKNOWN"); + } + + return 0; +} + +uint64_t odp_cpu_hz_current(int id ODP_UNUSED) +{ + return 0; +} diff --git a/platform/linux-generic/arch/powerpc/odp_cpu_arch.c b/platform/linux-generic/arch/powerpc/odp_cpu_arch.c deleted file mode 120000 index deebc47..0000000 --- a/platform/linux-generic/arch/powerpc/odp_cpu_arch.c +++ /dev/null @@ -1 +0,0 @@ -../default/odp_cpu_arch.c \ No newline at end of file diff --git a/platform/linux-generic/arch/powerpc/odp_cpu_arch.c b/platform/linux-generic/arch/powerpc/odp_cpu_arch.c new file mode 100644 index 0000000..2ac223e --- /dev/null +++ b/platform/linux-generic/arch/powerpc/odp_cpu_arch.c @@ -0,0 +1,48 @@ +/* Copyright (c) 2015, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#include <odp_posix_extensions.h> + +#include <stdlib.h> +#include <time.h> + +#include <odp/api/cpu.h> +#include <odp/api/hints.h> +#include <odp/api/system_info.h> +#include <odp_debug_internal.h> + +#define GIGA 1000000000 + +uint64_t odp_cpu_cycles(void) +{ + struct timespec time; + uint64_t sec, ns, hz, cycles; + int ret; + + ret = clock_gettime(CLOCK_MONOTONIC_RAW, &time); + + if (ret != 0) + ODP_ABORT("clock_gettime failed\n"); + + hz = odp_cpu_hz_max(); + sec = (uint64_t)time.tv_sec; + ns = (uint64_t)time.tv_nsec; + + cycles = sec * hz; + cycles += (ns * hz) / GIGA; + + return cycles; +} + +uint64_t odp_cpu_cycles_max(void) +{ + return UINT64_MAX; +} + +uint64_t odp_cpu_cycles_resolution(void) +{ + return 1; +} diff --git a/platform/linux-generic/include/_fdserver_internal.h b/platform/linux-generic/include/_fdserver_internal.h index 480ac02..22b2802 100644 --- a/platform/linux-generic/include/_fdserver_internal.h +++ b/platform/linux-generic/include/_fdserver_internal.h @@ -23,6 +23,7 @@ extern "C" { */ typedef enum fd_server_context { FD_SRV_CTX_NA, /* Not Applicable */ + FD_SRV_CTX_ISHM, FD_SRV_CTX_END, /* upper enum limit */ } fd_server_context_e;

diff --git a/platform/linux-generic/include/_ishm_internal.h b/platform/linux-generic/include/_ishm_internal.h new file mode 100644 index 0000000..7d27477 --- /dev/null +++ b/platform/linux-generic/include/_ishm_internal.h @@ -0,0 +1,45 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#ifndef ODP_ISHM_INTERNAL_H_ +#define ODP_ISHM_INTERNAL_H_ + +#ifdef __cplusplus +extern "C" { +#endif + +/* flags available at ishm_reserve: */ +#define _ODP_ISHM_SINGLE_VA 1 +#define _ODP_ISHM_LOCK 2 + +/** + * Shared memory block info + */ +typedef struct _odp_ishm_info_t { + const char *name; /**< Block name */ + void *addr; /**< Block address */ + uint64_t size; /**< Block size in bytes */ + uint64_t page_size; /**< Memory page size */ + uint32_t flags; /**< _ODP_ISHM_* flags */ + uint32_t user_flags;/**< user specific flags */ +} _odp_ishm_info_t; + +int _odp_ishm_reserve(const char *name, uint64_t size, int fd, uint32_t align, + uint32_t flags, uint32_t user_flags); +int _odp_ishm_free_by_index(int block_index); +int _odp_ishm_free_by_name(const char *name); +int _odp_ishm_free_by_address(void *addr); +void *_odp_ishm_lookup_by_index(int block_index); +int _odp_ishm_lookup_by_name(const char *name); +int _odp_ishm_lookup_by_address(void *addr); +void *_odp_ishm_address(int block_index); +int _odp_ishm_info(int block_index, _odp_ishm_info_t *info); + +#ifdef __cplusplus +} +#endif + +#endif diff --git a/platform/linux-generic/include/_ishmphy_internal.h b/platform/linux-generic/include/_ishmphy_internal.h new file mode 100644 index 0000000..4fe560f --- /dev/null +++ b/platform/linux-generic/include/_ishmphy_internal.h @@ -0,0 +1,25 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#ifndef _ISHMPHY_INTERNAL_H +#define _ISHMPHY_INTERNAL_H + +#ifdef __cplusplus +extern "C" { +#endif + +#include <stdint.h> + +void *_odp_ishmphy_book_va(uintptr_t len, intptr_t align); +int _odp_ishmphy_unbook_va(void); +void *_odp_ishmphy_map(int fd, void *start, uint64_t size, int flags); +int _odp_ishmphy_unmap(void *start, uint64_t len, int flags); + +#ifdef __cplusplus +} +#endif + +#endif diff --git a/platform/linux-generic/include/ishmphy_internal.h b/platform/linux-generic/include/ishmphy_internal.h new file mode 100644 index 0000000..0bc4207 --- /dev/null +++ b/platform/linux-generic/include/ishmphy_internal.h @@ -0,0 +1,24 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#ifndef _ISHMPHY_INTERNAL_H_ +#define _ISHMPHY_INTERNAL_H_ + +#ifdef __cplusplus +extern "C" { +#endif + +void *_ishmphy_book_va(uint64_t len); +int _ishmphy_unbook_va(void); +void *_ishmphy_map(int fd, void *start, uint64_t size, + int flags, int mmap_flags); +int _ishmphy_unmap(void *start, uint64_t len, int flags); + +#ifdef __cplusplus +} +#endif + +#endif diff --git a/platform/linux-generic/include/odp_config_internal.h b/platform/linux-generic/include/odp_config_internal.h index 06550e6..e89a6a3 100644 --- a/platform/linux-generic/include/odp_config_internal.h +++ b/platform/linux-generic/include/odp_config_internal.h @@ -124,6 +124,16 @@ extern "C" { */ #define CONFIG_POOL_CACHE_SIZE 256

+/* + * Size of the virtual address space pre-reserver for ISHM + * + * This is just virtual space preallocation size, not memory allocation. + * This address space is used by ISHM to map things at a common address in + * all ODP threads (when the _ODP_ISHM_SINGLE_VA flag is used). + * In bytes. + */ +#define ODP_CONFIG_ISHM_VA_PREALLOC_SZ (536870912L) + #ifdef __cplusplus } #endif diff --git a/platform/linux-generic/include/odp_internal.h b/platform/linux-generic/include/odp_internal.h index 363dc6f..5698fb0 100644 --- a/platform/linux-generic/include/odp_internal.h +++ b/platform/linux-generic/include/odp_internal.h @@ -58,6 +58,7 @@ enum init_stage { TIME_INIT, SYSINFO_INIT, FDSERVER_INIT, + ISHM_INIT, SHM_INIT, THREAD_INIT, POOL_INIT, @@ -126,6 +127,11 @@ int _odp_int_name_tbl_term_global(void); int _odp_fdserver_init_global(void); int _odp_fdserver_term_global(void);

+int _odp_ishm_init_global(void); +int _odp_ishm_init_local(void); +int _odp_ishm_term_global(void); +int _odp_ishm_term_local(void); + int cpuinfo_parser(FILE *file, system_info_t *sysinfo); uint64_t odp_cpu_hz_current(int id);

diff --git a/platform/linux-generic/odp_init.c b/platform/linux-generic/odp_init.c index a078533..43d9e40 100644 --- a/platform/linux-generic/odp_init.c +++ b/platform/linux-generic/odp_init.c @@ -114,6 +114,12 @@ int odp_init_global(odp_instance_t *instance, } stage = FDSERVER_INIT;

+ if (_odp_ishm_init_global()) { + ODP_ERR("ODP ishm init failed.\n"); + goto init_failed; + } + stage = ISHM_INIT; + if (odp_shm_init_global()) { ODP_ERR("ODP shm init failed.\n"); goto init_failed; @@ -280,6 +286,13 @@ int _odp_term_global(enum init_stage stage) } /* Fall through */

+ case ISHM_INIT: + if (_odp_ishm_term_global()) { + ODP_ERR("ODP ishm term failed.\n"); + rc = -1; + } + /* Fall through */ + case FDSERVER_INIT: if (_odp_fdserver_term_global()) { ODP_ERR("ODP fdserver term failed.\n"); @@ -324,6 +337,12 @@ int odp_init_local(odp_instance_t instance, odp_thread_type_t thr_type) goto init_fail; }

+ if (_odp_ishm_init_local()) { + ODP_ERR("ODP ishm local init failed.\n"); + goto init_fail; + } + stage = ISHM_INIT; + if (odp_shm_init_local()) { ODP_ERR("ODP shm local init failed.\n"); goto init_fail; @@ -399,6 +418,13 @@ int _odp_term_local(enum init_stage stage) } /* Fall through */

+ case ISHM_INIT: + if (_odp_ishm_term_local()) { + ODP_ERR("ODP ishm local term failed.\n"); + rc = -1; + } + /* Fall through */ + default: break; }

commit 2b994eae58d69d84126bd9db613f630884220d77 Author: Maxim Uvarov maxim.uvarov@linaro.org Date: Fri Dec 16 17:27:35 2016 +0300

linux-gen: pktio ipc: fix clang build

clang is more clever on setting and not using variables, so it traps compilation. Also buffers header almost everywhere reference by pointer so size of it should not impact on performance.

Signed-off-by: Maxim Uvarov maxim.uvarov@linaro.org Reviewed-and-tested-by: Bill Fischofer bill.fischofer@linaro.org

diff --git a/platform/linux-generic/include/odp_buffer_internal.h b/platform/linux-generic/include/odp_buffer_internal.h index 903f0a7..4cc51d3 100644 --- a/platform/linux-generic/include/odp_buffer_internal.h +++ b/platform/linux-generic/include/odp_buffer_internal.h @@ -64,11 +64,10 @@ struct odp_buffer_hdr_t { struct { void *hdr; uint8_t *data; -#ifdef _ODP_PKTIO_IPC - /* ipc mapped process can not walk over pointers, - * offset has to be used */ + /* Used only if _ODP_PKTIO_IPC is set. + * ipc mapped process can not walk over pointers, + * offset has to be used */ uint64_t ipc_data_offset; -#endif uint32_t len; } seg[CONFIG_PACKET_MAX_SEGS];

diff --git a/platform/linux-generic/pktio/ipc.c b/platform/linux-generic/pktio/ipc.c index 5f26b56..c9df043 100644 --- a/platform/linux-generic/pktio/ipc.c +++ b/platform/linux-generic/pktio/ipc.c @@ -459,12 +459,7 @@ static int ipc_pktio_recv_lockless(pktio_entry_t *pktio_entry, if (odp_unlikely(pool == ODP_POOL_INVALID)) ODP_ABORT("invalid pool");

-#ifdef _ODP_PKTIO_IPC data_pool_off = phdr->buf_hdr.seg[0].ipc_data_offset; -#else - /* compile all function code even if ipc disabled with config */ - data_pool_off = 0; -#endif

pkt = odp_packet_alloc(pool, phdr->frame_len); if (odp_unlikely(pkt == ODP_PACKET_INVALID)) { @@ -590,7 +585,6 @@ static int ipc_pktio_send_lockless(pktio_entry_t *pktio_entry, data_pool_off = (uint8_t *)pkt_hdr->buf_hdr.seg[0].data - (uint8_t *)odp_shm_addr(pool->shm);

-#ifdef _ODP_PKTIO_IPC /* compile all function code even if ipc disabled with config */ pkt_hdr->buf_hdr.seg[0].ipc_data_offset = data_pool_off; IPC_ODP_DBG("%d/%d send packet %llx, pool %llx," @@ -598,7 +592,6 @@ static int ipc_pktio_send_lockless(pktio_entry_t *pktio_entry, i, len, odp_packet_to_u64(pkt), odp_pool_to_u64(pool_hdl), pkt_hdr, pkt_hdr->buf_hdr.seg[0].ipc_data_offset); -#endif }

/* Put packets to ring to be processed by other process. */

commit da5eb63ba1048b0f18e6c436bb655d0649dc1bd7 Author: Maxim Uvarov maxim.uvarov@linaro.org Date: Wed Dec 14 22:57:57 2016 +0300

linux-gen: pktio ipc: tests: remove comment about master-slave

Implementation take care which process to name master and which slave. Comment is useless.

Signed-off-by: Maxim Uvarov maxim.uvarov@linaro.org Reviewed-and-tested-by: Bill Fischofer bill.fischofer@linaro.org

diff --git a/test/linux-generic/pktio_ipc/pktio_ipc1.c b/test/linux-generic/pktio_ipc/pktio_ipc1.c index 838b672..705c205 100644 --- a/test/linux-generic/pktio_ipc/pktio_ipc1.c +++ b/test/linux-generic/pktio_ipc/pktio_ipc1.c @@ -52,9 +52,6 @@ static int pktio_run_loop(odp_pool_t pool) start_cycle = odp_time_local(); current_cycle = start_cycle;

- /* slave process should always be run after master process to be - * able to create the same pktio. - */ for (;;) { if (run_time_sec) { cycle = odp_time_local(); diff --git a/test/linux-generic/pktio_ipc/pktio_ipc2.c b/test/linux-generic/pktio_ipc/pktio_ipc2.c index fb6f994..daf3841 100644 --- a/test/linux-generic/pktio_ipc/pktio_ipc2.c +++ b/test/linux-generic/pktio_ipc/pktio_ipc2.c @@ -49,9 +49,6 @@ static int ipc_second_process(int master_pid) wait = odp_time_local_from_ns(run_time_sec * ODP_TIME_SEC_IN_NS); start_cycle = odp_time_local();

- /* slave process should always be run after master process to be - * able to create the same pktio. - */ for (;;) { /* exit loop if time specified */ if (run_time_sec) {

commit 3c67b97bd06590235287d06d18b662c93aa64842 Author: Maxim Uvarov maxim.uvarov@linaro.org Date: Wed Dec 14 22:57:56 2016 +0300

linux-gen: pktio ipc: make it work again

Signed-off-by: Maxim Uvarov maxim.uvarov@linaro.org Reviewed-and-tested-by: Bill Fischofer bill.fischofer@linaro.org

diff --git a/platform/linux-generic/include/odp_buffer_internal.h b/platform/linux-generic/include/odp_buffer_internal.h index 2064f7c..903f0a7 100644 --- a/platform/linux-generic/include/odp_buffer_internal.h +++ b/platform/linux-generic/include/odp_buffer_internal.h @@ -64,6 +64,11 @@ struct odp_buffer_hdr_t { struct { void *hdr; uint8_t *data; +#ifdef _ODP_PKTIO_IPC + /* ipc mapped process can not walk over pointers, + * offset has to be used */ + uint64_t ipc_data_offset; +#endif uint32_t len; } seg[CONFIG_PACKET_MAX_SEGS];

@@ -94,11 +99,6 @@ struct odp_buffer_hdr_t { uint32_t uarea_size; /* size of user area */ uint32_t segcount; /* segment count */ uint32_t segsize; /* segment size */ -#ifdef _ODP_PKTIO_IPC - /* ipc mapped process can not walk over pointers, - * offset has to be used */ - uint64_t ipc_addr_offset[ODP_CONFIG_PACKET_MAX_SEGS]; -#endif

/* Data or next header */ uint8_t data[0]; diff --git a/platform/linux-generic/include/odp_internal.h b/platform/linux-generic/include/odp_internal.h index 6063b0f..363dc6f 100644 --- a/platform/linux-generic/include/odp_internal.h +++ b/platform/linux-generic/include/odp_internal.h @@ -50,7 +50,6 @@ struct odp_global_data_s { odp_cpumask_t control_cpus; odp_cpumask_t worker_cpus; int num_cpus_installed; - int ipc_ns; };

enum init_stage { diff --git a/platform/linux-generic/include/odp_packet_io_internal.h b/platform/linux-generic/include/odp_packet_io_internal.h index bdf6316..2001c42 100644 --- a/platform/linux-generic/include/odp_packet_io_internal.h +++ b/platform/linux-generic/include/odp_packet_io_internal.h @@ -102,6 +102,8 @@ typedef struct { packet, 0 - not yet ready */ void *pinfo; odp_shm_t pinfo_shm; + odp_shm_t remote_pool_shm; /**< shm of remote pool get with + _ipc_map_remote_pool() */ } _ipc_pktio_t;

struct pktio_entry { diff --git a/platform/linux-generic/include/odp_packet_io_ipc_internal.h b/platform/linux-generic/include/odp_packet_io_ipc_internal.h index 851114d..7cd2948 100644 --- a/platform/linux-generic/include/odp_packet_io_ipc_internal.h +++ b/platform/linux-generic/include/odp_packet_io_ipc_internal.h @@ -26,22 +26,31 @@ */ struct pktio_info { struct { - /* number of buffer in remote pool */ - int shm_pool_bufs_num; - /* size of remote pool */ - size_t shm_pkt_pool_size; + /* number of buffer*/ + int num; /* size of packet/segment in remote pool */ - uint32_t shm_pkt_size; + uint32_t block_size; /* offset from shared memory block start - * to pool_mdata_addr (odp-linux pool specific) */ - size_t mdata_offset; + * to pool *base_addr in remote process. + * (odp-linux pool specific) */ + size_t base_addr_offset; char pool_name[ODP_POOL_NAME_LEN]; + /* 1 if master finished creation of all shared objects */ + int init_done; } master; struct { /* offset from shared memory block start - * to pool_mdata_addr in remote process. + * to pool *base_addr in remote process. * (odp-linux pool specific) */ - size_t mdata_offset; + size_t base_addr_offset; + void *base_addr; + uint32_t block_size; char pool_name[ODP_POOL_NAME_LEN]; + /* pid of the slave process written to shm and + * used by master to look up memory created by + * slave + */ + int pid; + int init_done; } slave; } ODP_PACKED; diff --git a/platform/linux-generic/odp_init.c b/platform/linux-generic/odp_init.c index fb85cc1..a078533 100644 --- a/platform/linux-generic/odp_init.c +++ b/platform/linux-generic/odp_init.c @@ -67,7 +67,7 @@ static int cleanup_files(const char *dirpath, int odp_pid)

int odp_init_global(odp_instance_t *instance, const odp_init_t *params, - const odp_platform_init_t *platform_params) + const odp_platform_init_t *platform_params ODP_UNUSED) { char *hpdir;

@@ -75,9 +75,6 @@ int odp_init_global(odp_instance_t *instance, odp_global_data.main_pid = getpid(); cleanup_files(_ODP_TMPDIR, odp_global_data.main_pid);

- if (platform_params) - odp_global_data.ipc_ns = platform_params->ipc_ns; - enum init_stage stage = NO_INIT; odp_global_data.log_fn = odp_override_log; odp_global_data.abort_fn = odp_override_abort; diff --git a/platform/linux-generic/pktio/ipc.c b/platform/linux-generic/pktio/ipc.c index 0e99c6e..5f26b56 100644 --- a/platform/linux-generic/pktio/ipc.c +++ b/platform/linux-generic/pktio/ipc.c @@ -3,150 +3,85 @@ * * SPDX-License-Identifier: BSD-3-Clause */ -#ifdef _ODP_PKTIO_IPC #include <odp_packet_io_ipc_internal.h> #include <odp_debug_internal.h> #include <odp_packet_io_internal.h> #include <odp/api/system_info.h> #include <odp_shm_internal.h> +#include <_ishm_internal.h>

#include <sys/mman.h> #include <sys/stat.h> #include <fcntl.h>

+#define IPC_ODP_DEBUG_PRINT 0 + +#define IPC_ODP_DBG(fmt, ...) \ + do { \ + if (IPC_ODP_DEBUG_PRINT == 1) \ + ODP_DBG(fmt, ##__VA_ARGS__);\ + } while (0) + /* MAC address for the "ipc" interface */ static const char pktio_ipc_mac[] = {0x12, 0x12, 0x12, 0x12, 0x12, 0x12};

-static void *_ipc_map_remote_pool(const char *name, size_t size); +static odp_shm_t _ipc_map_remote_pool(const char *name, int pid);

static const char *_ipc_odp_buffer_pool_shm_name(odp_pool_t pool_hdl) { - pool_entry_t *pool; - uint32_t pool_id; + pool_t *pool; odp_shm_t shm; odp_shm_info_t info;

- pool_id = pool_handle_to_index(pool_hdl); - pool = get_pool_entry(pool_id); - shm = pool->s.pool_shm; + pool = pool_entry_from_hdl(pool_hdl); + shm = pool->shm;

odp_shm_info(shm, &info);

return info.name; }

-/** -* Look up for shared memory object. -* -* @param name name of shm object -* -* @return 0 on success, otherwise non-zero -*/ -static int _ipc_shm_lookup(const char *name) -{ - int shm; - char shm_devname[SHM_DEVNAME_MAXLEN]; - - if (!odp_global_data.ipc_ns) - ODP_ABORT("ipc_ns not set\n"); - - snprintf(shm_devname, SHM_DEVNAME_MAXLEN, - SHM_DEVNAME_FORMAT, - odp_global_data.ipc_ns, name); - - shm = shm_open(shm_devname, O_RDWR, S_IRUSR | S_IWUSR); - if (shm == -1) { - if (errno == ENOENT) { - ODP_DBG("no file %s\n", shm_devname); - return -1; - } - ODP_ABORT("shm_open for %s err %s\n", - shm_devname, strerror(errno)); - } - close(shm); - return 0; -} - -static int _ipc_map_pktio_info(pktio_entry_t *pktio_entry, - const char *dev, - int *slave) -{ - struct pktio_info *pinfo; - char name[ODP_POOL_NAME_LEN + sizeof("_info")]; - uint32_t flags; - odp_shm_t shm; - - /* Create info about remote pktio */ - snprintf(name, sizeof(name), "%s_info", dev); - - flags = ODP_SHM_PROC | _ODP_SHM_O_EXCL; - - shm = odp_shm_reserve(name, sizeof(struct pktio_info), - ODP_CACHE_LINE_SIZE, - flags); - if (ODP_SHM_INVALID != shm) { - pinfo = odp_shm_addr(shm); - pinfo->master.pool_name[0] = 0; - *slave = 0; - } else { - flags = _ODP_SHM_PROC_NOCREAT | _ODP_SHM_O_EXCL; - shm = odp_shm_reserve(name, sizeof(struct pktio_info), - ODP_CACHE_LINE_SIZE, - flags); - if (ODP_SHM_INVALID == shm) - ODP_ABORT("can not connect to shm\n"); - - pinfo = odp_shm_addr(shm); - *slave = 1; - } - - pktio_entry->s.ipc.pinfo = pinfo; - pktio_entry->s.ipc.pinfo_shm = shm; - - return 0; -} - static int _ipc_master_start(pktio_entry_t *pktio_entry) { struct pktio_info *pinfo = pktio_entry->s.ipc.pinfo; - int ret; - void *ipc_pool_base; + odp_shm_t shm;

- if (pinfo->slave.mdata_offset == 0) + if (pinfo->slave.init_done == 0) return -1;

- ret = _ipc_shm_lookup(pinfo->slave.pool_name); - if (ret) { - ODP_DBG("no pool file %s\n", pinfo->slave.pool_name); + shm = _ipc_map_remote_pool(pinfo->slave.pool_name, + pinfo->slave.pid); + if (shm == ODP_SHM_INVALID) { + ODP_DBG("no pool file %s for pid %d\n", + pinfo->slave.pool_name, pinfo->slave.pid); return -1; }

- ipc_pool_base = _ipc_map_remote_pool(pinfo->slave.pool_name, - pinfo->master.shm_pkt_pool_size); - pktio_entry->s.ipc.pool_mdata_base = (char *)ipc_pool_base + - pinfo->slave.mdata_offset; + pktio_entry->s.ipc.remote_pool_shm = shm; + pktio_entry->s.ipc.pool_base = odp_shm_addr(shm); + pktio_entry->s.ipc.pool_mdata_base = (char *)odp_shm_addr(shm) + + pinfo->slave.base_addr_offset;

odp_atomic_store_u32(&pktio_entry->s.ipc.ready, 1);

- ODP_DBG("%s started.\n", pktio_entry->s.name); + IPC_ODP_DBG("%s started.\n", pktio_entry->s.name); return 0; }

static int _ipc_init_master(pktio_entry_t *pktio_entry, const char *dev, - odp_pool_t pool) + odp_pool_t pool_hdl) { char ipc_shm_name[ODP_POOL_NAME_LEN + sizeof("_m_prod")]; - pool_entry_t *pool_entry; - uint32_t pool_id; + pool_t *pool; struct pktio_info *pinfo; const char *pool_name;

- pool_id = pool_handle_to_index(pool); - pool_entry = get_pool_entry(pool_id); + pool = pool_entry_from_hdl(pool_hdl); + (void)pool;

if (strlen(dev) > (ODP_POOL_NAME_LEN - sizeof("_m_prod"))) { - ODP_DBG("too big ipc name\n"); + ODP_ERR("too big ipc name\n"); return -1; }

@@ -158,7 +93,7 @@ static int _ipc_init_master(pktio_entry_t *pktio_entry, PKTIO_IPC_ENTRIES, _RING_SHM_PROC | _RING_NO_LIST); if (!pktio_entry->s.ipc.tx.send) { - ODP_DBG("pid %d unable to create ipc ring %s name\n", + ODP_ERR("pid %d unable to create ipc ring %s name\n", getpid(), ipc_shm_name); return -1; } @@ -174,7 +109,7 @@ static int _ipc_init_master(pktio_entry_t *pktio_entry, PKTIO_IPC_ENTRIES, _RING_SHM_PROC | _RING_NO_LIST); if (!pktio_entry->s.ipc.tx.free) { - ODP_DBG("pid %d unable to create ipc ring %s name\n", + ODP_ERR("pid %d unable to create ipc ring %s name\n", getpid(), ipc_shm_name); goto free_m_prod; } @@ -187,7 +122,7 @@ static int _ipc_init_master(pktio_entry_t *pktio_entry, PKTIO_IPC_ENTRIES, _RING_SHM_PROC | _RING_NO_LIST); if (!pktio_entry->s.ipc.rx.recv) { - ODP_DBG("pid %d unable to create ipc ring %s name\n", + ODP_ERR("pid %d unable to create ipc ring %s name\n", getpid(), ipc_shm_name); goto free_m_cons; } @@ -200,7 +135,7 @@ static int _ipc_init_master(pktio_entry_t *pktio_entry, PKTIO_IPC_ENTRIES, _RING_SHM_PROC | _RING_NO_LIST); if (!pktio_entry->s.ipc.rx.free) { - ODP_DBG("pid %d unable to create ipc ring %s name\n", + ODP_ERR("pid %d unable to create ipc ring %s name\n", getpid(), ipc_shm_name); goto free_s_prod; } @@ -210,24 +145,23 @@ static int _ipc_init_master(pktio_entry_t *pktio_entry,

/* Set up pool name for remote info */ pinfo = pktio_entry->s.ipc.pinfo; - pool_name = _ipc_odp_buffer_pool_shm_name(pool); + pool_name = _ipc_odp_buffer_pool_shm_name(pool_hdl); if (strlen(pool_name) > ODP_POOL_NAME_LEN) { - ODP_DBG("pid %d ipc pool name %s is too big %d\n", + ODP_ERR("pid %d ipc pool name %s is too big %d\n", getpid(), pool_name, strlen(pool_name)); goto free_s_prod; }

memcpy(pinfo->master.pool_name, pool_name, strlen(pool_name)); - pinfo->master.shm_pkt_pool_size = pool_entry->s.pool_size; - pinfo->master.shm_pool_bufs_num = pool_entry->s.buf_num; - pinfo->master.shm_pkt_size = pool_entry->s.seg_size; - pinfo->master.mdata_offset = pool_entry->s.pool_mdata_addr - - pool_entry->s.pool_base_addr; - pinfo->slave.mdata_offset = 0; + pinfo->slave.base_addr_offset = 0; + pinfo->slave.base_addr = 0; + pinfo->slave.pid = 0; + pinfo->slave.init_done = 0;

- pktio_entry->s.ipc.pool = pool; + pktio_entry->s.ipc.pool = pool_hdl;

ODP_DBG("Pre init... DONE.\n"); + pinfo->master.init_done = 1;

_ipc_master_start(pktio_entry);

@@ -246,55 +180,42 @@ free_m_prod: }

static void _ipc_export_pool(struct pktio_info *pinfo, - odp_pool_t pool) + odp_pool_t pool_hdl) { - pool_entry_t *pool_entry; - - pool_entry = odp_pool_to_entry(pool); - if (pool_entry->s.blk_size != pinfo->master.shm_pkt_size) - ODP_ABORT("pktio for same name should have the same pool size\n"); - if (pool_entry->s.buf_num != (unsigned)pinfo->master.shm_pool_bufs_num) - ODP_ABORT("pktio for same name should have the same pool size\n"); + pool_t *pool = pool_entry_from_hdl(pool_hdl);

snprintf(pinfo->slave.pool_name, ODP_POOL_NAME_LEN, "%s", - pool_entry->s.name); - pinfo->slave.mdata_offset = pool_entry->s.pool_mdata_addr - - pool_entry->s.pool_base_addr; + _ipc_odp_buffer_pool_shm_name(pool_hdl)); + pinfo->slave.pid = odp_global_data.main_pid; + pinfo->slave.block_size = pool->block_size; + pinfo->slave.base_addr = pool->base_addr; }

-static void *_ipc_map_remote_pool(const char *name, size_t size) +static odp_shm_t _ipc_map_remote_pool(const char *name, int pid) { odp_shm_t shm; - void *addr; - - ODP_DBG("Mapping remote pool %s, size %ld\n", name, size); - shm = odp_shm_reserve(name, - size, - ODP_CACHE_LINE_SIZE, - _ODP_SHM_PROC_NOCREAT); - if (shm == ODP_SHM_INVALID) - ODP_ABORT("unable map %s\n", name); - - addr = odp_shm_addr(shm); - ODP_DBG("MAP master: %p - %p size %ld, pool %s\n", - addr, (char *)addr + size, size, name); - return addr; + char rname[ODP_SHM_NAME_LEN]; + + snprintf(rname, ODP_SHM_NAME_LEN, "remote-%s", name); + shm = odp_shm_import(name, pid, rname); + if (shm == ODP_SHM_INVALID) { + ODP_ERR("unable map %s\n", name); + return ODP_SHM_INVALID; + } + + IPC_ODP_DBG("Mapped remote pool %s to local %s\n", name, rname); + return shm; }

-static void *_ipc_shm_map(char *name, size_t size) +static void *_ipc_shm_map(char *name, int pid) { odp_shm_t shm; - int ret;

- ret = _ipc_shm_lookup(name); - if (ret == -1) + shm = odp_shm_import(name, pid, name); + if (ODP_SHM_INVALID == shm) { + ODP_ERR("unable to map: %s\n", name); return NULL; - - shm = odp_shm_reserve(name, size, - ODP_CACHE_LINE_SIZE, - _ODP_SHM_PROC_NOCREAT); - if (ODP_SHM_INVALID == shm) - ODP_ABORT("unable to map: %s\n", name); + }

return odp_shm_addr(shm); } @@ -313,15 +234,21 @@ static int _ipc_init_slave(const char *dev, static int _ipc_slave_start(pktio_entry_t *pktio_entry) { char ipc_shm_name[ODP_POOL_NAME_LEN + sizeof("_slave_r")]; - size_t ring_size = PKTIO_IPC_ENTRIES * sizeof(void *) + - sizeof(_ring_t); struct pktio_info *pinfo; - void *ipc_pool_base; odp_shm_t shm; - const char *dev = pktio_entry->s.name; + char tail[ODP_POOL_NAME_LEN]; + char dev[ODP_POOL_NAME_LEN]; + int pid; + + if (sscanf(pktio_entry->s.name, "ipc:%d:%s", &pid, tail) != 2) { + ODP_ERR("wrong pktio name\n"); + return -1; + } + + sprintf(dev, "ipc:%s", tail);

snprintf(ipc_shm_name, sizeof(ipc_shm_name), "%s_m_prod", dev); - pktio_entry->s.ipc.rx.recv = _ipc_shm_map(ipc_shm_name, ring_size); + pktio_entry->s.ipc.rx.recv = _ipc_shm_map(ipc_shm_name, pid); if (!pktio_entry->s.ipc.rx.recv) { ODP_DBG("pid %d unable to find ipc ring %s name\n", getpid(), dev); @@ -333,9 +260,9 @@ static int _ipc_slave_start(pktio_entry_t *pktio_entry) _ring_free_count(pktio_entry->s.ipc.rx.recv));

snprintf(ipc_shm_name, sizeof(ipc_shm_name), "%s_m_cons", dev); - pktio_entry->s.ipc.rx.free = _ipc_shm_map(ipc_shm_name, ring_size); + pktio_entry->s.ipc.rx.free = _ipc_shm_map(ipc_shm_name, pid); if (!pktio_entry->s.ipc.rx.free) { - ODP_DBG("pid %d unable to find ipc ring %s name\n", + ODP_ERR("pid %d unable to find ipc ring %s name\n", getpid(), dev); goto free_m_prod; } @@ -344,9 +271,9 @@ static int _ipc_slave_start(pktio_entry_t *pktio_entry) _ring_free_count(pktio_entry->s.ipc.rx.free));

snprintf(ipc_shm_name, sizeof(ipc_shm_name), "%s_s_prod", dev); - pktio_entry->s.ipc.tx.send = _ipc_shm_map(ipc_shm_name, ring_size); + pktio_entry->s.ipc.tx.send = _ipc_shm_map(ipc_shm_name, pid); if (!pktio_entry->s.ipc.tx.send) { - ODP_DBG("pid %d unable to find ipc ring %s name\n", + ODP_ERR("pid %d unable to find ipc ring %s name\n", getpid(), dev); goto free_m_cons; } @@ -355,9 +282,9 @@ static int _ipc_slave_start(pktio_entry_t *pktio_entry) _ring_free_count(pktio_entry->s.ipc.tx.send));

snprintf(ipc_shm_name, sizeof(ipc_shm_name), "%s_s_cons", dev); - pktio_entry->s.ipc.tx.free = _ipc_shm_map(ipc_shm_name, ring_size); + pktio_entry->s.ipc.tx.free = _ipc_shm_map(ipc_shm_name, pid); if (!pktio_entry->s.ipc.tx.free) { - ODP_DBG("pid %d unable to find ipc ring %s name\n", + ODP_ERR("pid %d unable to find ipc ring %s name\n", getpid(), dev); goto free_s_prod; } @@ -367,15 +294,17 @@ static int _ipc_slave_start(pktio_entry_t *pktio_entry)

/* Get info about remote pool */ pinfo = pktio_entry->s.ipc.pinfo; - ipc_pool_base = _ipc_map_remote_pool(pinfo->master.pool_name, - pinfo->master.shm_pkt_pool_size); - pktio_entry->s.ipc.pool_mdata_base = (char *)ipc_pool_base + - pinfo->master.mdata_offset; - pktio_entry->s.ipc.pkt_size = pinfo->master.shm_pkt_size; + shm = _ipc_map_remote_pool(pinfo->master.pool_name, + pid); + pktio_entry->s.ipc.remote_pool_shm = shm; + pktio_entry->s.ipc.pool_mdata_base = (char *)odp_shm_addr(shm) + + pinfo->master.base_addr_offset; + pktio_entry->s.ipc.pkt_size = pinfo->master.block_size;

_ipc_export_pool(pinfo, pktio_entry->s.ipc.pool);

odp_atomic_store_u32(&pktio_entry->s.ipc.ready, 1); + pinfo->slave.init_done = 1;

ODP_DBG("%s started.\n", pktio_entry->s.name); return 0; @@ -401,7 +330,11 @@ static int ipc_pktio_open(odp_pktio_t id ODP_UNUSED, odp_pool_t pool) { int ret = -1; - int slave; + int pid ODP_UNUSED; + struct pktio_info *pinfo; + char name[ODP_POOL_NAME_LEN + sizeof("_info")]; + char tail[ODP_POOL_NAME_LEN]; + odp_shm_t shm;

ODP_STATIC_ASSERT(ODP_POOL_NAME_LEN == _RING_NAMESIZE, "mismatch pool and ring name arrays"); @@ -411,65 +344,59 @@ static int ipc_pktio_open(odp_pktio_t id ODP_UNUSED,

odp_atomic_init_u32(&pktio_entry->s.ipc.ready, 0);

- _ipc_map_pktio_info(pktio_entry, dev, &slave); - pktio_entry->s.ipc.type = (slave == 0) ? PKTIO_TYPE_IPC_MASTER : - PKTIO_TYPE_IPC_SLAVE; + /* Shared info about remote pktio */ + if (sscanf(dev, "ipc:%d:%s", &pid, tail) == 2) { + pktio_entry->s.ipc.type = PKTIO_TYPE_IPC_SLAVE;

- if (pktio_entry->s.ipc.type == PKTIO_TYPE_IPC_MASTER) { + snprintf(name, sizeof(name), "ipc:%s_info", tail); + IPC_ODP_DBG("lookup for name %s for pid %d\n", name, pid); + shm = odp_shm_import(name, pid, name); + if (ODP_SHM_INVALID == shm) + return -1; + pinfo = odp_shm_addr(shm); + + if (!pinfo->master.init_done) { + odp_shm_free(shm); + return -1; + } + pktio_entry->s.ipc.pinfo = pinfo; + pktio_entry->s.ipc.pinfo_shm = shm; + ODP_DBG("process %d is slave\n", getpid()); + ret = _ipc_init_slave(name, pktio_entry, pool); + } else { + pktio_entry->s.ipc.type = PKTIO_TYPE_IPC_MASTER; + snprintf(name, sizeof(name), "%s_info", dev); + shm = odp_shm_reserve(name, sizeof(struct pktio_info), + ODP_CACHE_LINE_SIZE, + _ODP_ISHM_EXPORT | _ODP_ISHM_LOCK); + if (ODP_SHM_INVALID == shm) { + ODP_ERR("can not create shm %s\n", name); + return -1; + } + + pinfo = odp_shm_addr(shm); + pinfo->master.init_done = 0; + pinfo->master.pool_name[0] = 0; + pktio_entry->s.ipc.pinfo = pinfo; + pktio_entry->s.ipc.pinfo_shm = shm; ODP_DBG("process %d is master\n", getpid()); ret = _ipc_init_master(pktio_entry, dev, pool); - } else { - ODP_DBG("process %d is slave\n", getpid()); - ret = _ipc_init_slave(dev, pktio_entry, pool); }

return ret; }

-static inline void *_ipc_buffer_map(odp_buffer_hdr_t *buf, - uint32_t offset, - uint32_t *seglen, - uint32_t limit) +static void _ipc_free_ring_packets(pktio_entry_t *pktio_entry, _ring_t *r) { - int seg_index = offset / buf->segsize; - int seg_offset = offset % buf->segsize; -#ifdef _ODP_PKTIO_IPC - void *addr = (char *)buf - buf->ipc_addr_offset[seg_index]; -#else - /** buf_hdr.ipc_addr_offset defined only when ipc is - * enabled. */ - void *addr = NULL; - - (void)seg_index; -#endif - if (seglen) { - uint32_t buf_left = limit - offset; - *seglen = seg_offset + buf_left <= buf->segsize ? - buf_left : buf->segsize - seg_offset; - } - - return (void *)(seg_offset + (uint8_t *)addr); -} - -static inline void *_ipc_packet_map(odp_packet_hdr_t *pkt_hdr, - uint32_t offset, uint32_t *seglen) -{ - if (offset > pkt_hdr->frame_len) - return NULL; - - return _ipc_buffer_map(&pkt_hdr->buf_hdr, - pkt_hdr->headroom + offset, seglen, - pkt_hdr->headroom + pkt_hdr->frame_len); -} - -static void _ipc_free_ring_packets(_ring_t *r) -{ - odp_packet_t r_p_pkts[PKTIO_IPC_ENTRIES]; + uintptr_t offsets[PKTIO_IPC_ENTRIES]; int ret; void **rbuf_p; int i;

- rbuf_p = (void *)&r_p_pkts; + if (!r) + return; + + rbuf_p = (void *)&offsets;

while (1) { ret = _ring_mc_dequeue_burst(r, rbuf_p, @@ -477,8 +404,13 @@ static void _ipc_free_ring_packets(_ring_t *r) if (0 == ret) break; for (i = 0; i < ret; i++) { - if (r_p_pkts[i] != ODP_PACKET_INVALID) - odp_packet_free(r_p_pkts[i]); + odp_packet_hdr_t *phdr; + odp_packet_t pkt; + void *mbase = pktio_entry->s.ipc.pool_mdata_base; + + phdr = (void *)((uint8_t *)mbase + offsets[i]); + pkt = (odp_packet_t)phdr->buf_hdr.handle.handle; + odp_packet_free(pkt); } } } @@ -490,22 +422,23 @@ static int ipc_pktio_recv_lockless(pktio_entry_t *pktio_entry, int i; _ring_t *r; _ring_t *r_p; + uintptr_t offsets[PKTIO_IPC_ENTRIES]; + void **ipcbufs_p = (void *)&offsets; + uint32_t ready; + int pkts_ring;

- odp_packet_t remote_pkts[PKTIO_IPC_ENTRIES]; - void **ipcbufs_p = (void *)&remote_pkts; - uint32_t ready = odp_atomic_load_u32(&pktio_entry->s.ipc.ready); - + ready = odp_atomic_load_u32(&pktio_entry->s.ipc.ready); if (odp_unlikely(!ready)) { - ODP_DBG("start pktio is missing before usage?\n"); - return -1; + IPC_ODP_DBG("start pktio is missing before usage?\n"); + return 0; }

- _ipc_free_ring_packets(pktio_entry->s.ipc.tx.free); + _ipc_free_ring_packets(pktio_entry, pktio_entry->s.ipc.tx.free);

r = pktio_entry->s.ipc.rx.recv; pkts = _ring_mc_dequeue_burst(r, ipcbufs_p, len); if (odp_unlikely(pkts < 0)) - ODP_ABORT("error to dequeue no packets\n"); + ODP_ABORT("internal error dequeue\n");

/* fast path */ if (odp_likely(0 == pkts)) @@ -514,36 +447,26 @@ static int ipc_pktio_recv_lockless(pktio_entry_t *pktio_entry, for (i = 0; i < pkts; i++) { odp_pool_t pool; odp_packet_t pkt; - odp_packet_hdr_t phdr; - void *ptr; - odp_buffer_bits_t handle; - int idx; /* Remote packet has coded pool and index. - * We need only index.*/ + odp_packet_hdr_t *phdr; void *pkt_data; - void *remote_pkt_data; + uint64_t data_pool_off; + void *rmt_data_ptr;

- if (remote_pkts[i] == ODP_PACKET_INVALID) - continue; + phdr = (void *)((uint8_t *)pktio_entry->s.ipc.pool_mdata_base + + offsets[i]);

- handle.handle = _odp_packet_to_buffer(remote_pkts[i]); - idx = handle.index; - - /* Link to packed data. To this line we have Zero-Copy between - * processes, to simplify use packet copy in that version which - * can be removed later with more advance buffer management - * (ref counters). - */ - /* reverse odp_buf_to_hdr() */ - ptr = (char *)pktio_entry->s.ipc.pool_mdata_base + - (idx * ODP_CACHE_LINE_SIZE); - memcpy(&phdr, ptr, sizeof(odp_packet_hdr_t)); - - /* Allocate new packet. Select*/ pool = pktio_entry->s.ipc.pool; if (odp_unlikely(pool == ODP_POOL_INVALID)) ODP_ABORT("invalid pool");

- pkt = odp_packet_alloc(pool, phdr.frame_len); +#ifdef _ODP_PKTIO_IPC + data_pool_off = phdr->buf_hdr.seg[0].ipc_data_offset; +#else + /* compile all function code even if ipc disabled with config */ + data_pool_off = 0; +#endif + + pkt = odp_packet_alloc(pool, phdr->frame_len); if (odp_unlikely(pkt == ODP_PACKET_INVALID)) { /* Original pool might be smaller then * PKTIO_IPC_ENTRIES. If packet can not be @@ -562,30 +485,40 @@ static int ipc_pktio_recv_lockless(pktio_entry_t *pktio_entry, (PKTIO_TYPE_IPC_SLAVE == pktio_entry->s.ipc.type));

- remote_pkt_data = _ipc_packet_map(ptr, 0, NULL); - if (odp_unlikely(!remote_pkt_data)) - ODP_ABORT("unable to map remote_pkt_data, ipc_slave %d\n", - (PKTIO_TYPE_IPC_SLAVE == - pktio_entry->s.ipc.type)); - /* Copy packet data from shared pool to local pool. */ - memcpy(pkt_data, remote_pkt_data, phdr.frame_len); + rmt_data_ptr = (uint8_t *)pktio_entry->s.ipc.pool_mdata_base + + data_pool_off; + memcpy(pkt_data, rmt_data_ptr, phdr->frame_len);

/* Copy packets L2, L3 parsed offsets and size */ - copy_packet_cls_metadata(&phdr, odp_packet_hdr(pkt)); + copy_packet_cls_metadata(phdr, odp_packet_hdr(pkt)); + + odp_packet_hdr(pkt)->frame_len = phdr->frame_len; + odp_packet_hdr(pkt)->headroom = phdr->headroom; + odp_packet_hdr(pkt)->tailroom = phdr->tailroom; + + /* Take classification fields */ + odp_packet_hdr(pkt)->p = phdr->p;

- odp_packet_hdr(pkt)->frame_len = phdr.frame_len; - odp_packet_hdr(pkt)->headroom = phdr.headroom; - odp_packet_hdr(pkt)->tailroom = phdr.tailroom; - odp_packet_hdr(pkt)->input = pktio_entry->s.handle; pkt_table[i] = pkt; }

/* Now tell other process that we no longer need that buffers.*/ r_p = pktio_entry->s.ipc.rx.free; - pkts = _ring_mp_enqueue_burst(r_p, ipcbufs_p, i); + +repeat: + pkts_ring = _ring_mp_enqueue_burst(r_p, ipcbufs_p, pkts); if (odp_unlikely(pkts < 0)) ODP_ABORT("ipc: odp_ring_mp_enqueue_bulk r_p fail\n"); + if (odp_unlikely(pkts != pkts_ring)) { + IPC_ODP_DBG("odp_ring_full: %d, odp_ring_count %d," + " _ring_free_count %d\n", + _ring_full(r_p), _ring_count(r_p), + _ring_free_count(r_p)); + ipcbufs_p = (void *)&offsets[pkts_ring - 1]; + pkts = pkts - pkts_ring; + goto repeat; + }

return pkts; } @@ -614,26 +547,23 @@ static int ipc_pktio_send_lockless(pktio_entry_t *pktio_entry, uint32_t ready = odp_atomic_load_u32(&pktio_entry->s.ipc.ready); odp_packet_t pkt_table_mapped[len]; /**< Ready to send packet has to be * in memory mapped pool. */ + uintptr_t offsets[len];

if (odp_unlikely(!ready)) return 0;

- _ipc_free_ring_packets(pktio_entry->s.ipc.tx.free); + _ipc_free_ring_packets(pktio_entry, pktio_entry->s.ipc.tx.free);

- /* Prepare packets: calculate offset from address. */ + /* Copy packets to shm shared pool if they are in different */ for (i = 0; i < len; i++) { - int j; odp_packet_t pkt = pkt_table[i]; - odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt); + pool_t *ipc_pool = pool_entry_from_hdl(pktio_entry->s.ipc.pool); odp_buffer_bits_t handle; - uint32_t cur_mapped_pool_id = - pool_handle_to_index(pktio_entry->s.ipc.pool); - uint32_t pool_id; + uint32_t pkt_pool_id;

- /* do copy if packet was allocated from not mapped pool */ handle.handle = _odp_packet_to_buffer(pkt); - pool_id = handle.pool_id; - if (pool_id != cur_mapped_pool_id) { + pkt_pool_id = handle.pool_id; + if (pkt_pool_id != ipc_pool->pool_idx) { odp_packet_t newpkt;

newpkt = odp_packet_copy(pkt, pktio_entry->s.ipc.pool); @@ -645,24 +575,34 @@ static int ipc_pktio_send_lockless(pktio_entry_t *pktio_entry, } else { pkt_table_mapped[i] = pkt; } + } + + /* Set offset to phdr for outgoing packets */ + for (i = 0; i < len; i++) { + uint64_t data_pool_off; + odp_packet_t pkt = pkt_table_mapped[i]; + odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt); + odp_pool_t pool_hdl = odp_packet_pool(pkt); + pool_t *pool = pool_entry_from_hdl(pool_hdl); + + offsets[i] = (uint8_t *)pkt_hdr - + (uint8_t *)odp_shm_addr(pool->shm); + data_pool_off = (uint8_t *)pkt_hdr->buf_hdr.seg[0].data - + (uint8_t *)odp_shm_addr(pool->shm);

- /* buf_hdr.addr can not be used directly in remote process, - * convert it to offset - */ - for (j = 0; j < ODP_BUFFER_MAX_SEG; j++) { #ifdef _ODP_PKTIO_IPC - pkt_hdr->buf_hdr.ipc_addr_offset[j] = (char *)pkt_hdr - - (char *)pkt_hdr->buf_hdr.addr[j]; -#else - /** buf_hdr.ipc_addr_offset defined only when ipc is - * enabled. */ - (void)pkt_hdr; + /* compile all function code even if ipc disabled with config */ + pkt_hdr->buf_hdr.seg[0].ipc_data_offset = data_pool_off; + IPC_ODP_DBG("%d/%d send packet %llx, pool %llx," + "phdr = %p, offset %x\n", + i, len, + odp_packet_to_u64(pkt), odp_pool_to_u64(pool_hdl), + pkt_hdr, pkt_hdr->buf_hdr.seg[0].ipc_data_offset); #endif - } }

/* Put packets to ring to be processed by other process. */ - rbuf_p = (void *)&pkt_table_mapped[0]; + rbuf_p = (void *)&offsets[0]; r = pktio_entry->s.ipc.tx.send; ret = _ring_mp_enqueue_burst(r, rbuf_p, len); if (odp_unlikely(ret < 0)) { @@ -673,6 +613,7 @@ static int ipc_pktio_send_lockless(pktio_entry_t *pktio_entry, ODP_ERR("odp_ring_full: %d, odp_ring_count %d, _ring_free_count %d\n", _ring_full(r), _ring_count(r), _ring_free_count(r)); + ODP_ABORT("Unexpected!\n"); }

return ret; @@ -722,22 +663,25 @@ static int ipc_start(pktio_entry_t *pktio_entry)

static int ipc_stop(pktio_entry_t *pktio_entry) { - unsigned tx_send, tx_free; + unsigned tx_send = 0, tx_free = 0;

odp_atomic_store_u32(&pktio_entry->s.ipc.ready, 0);

- _ipc_free_ring_packets(pktio_entry->s.ipc.tx.send); + if (pktio_entry->s.ipc.tx.send) + _ipc_free_ring_packets(pktio_entry, pktio_entry->s.ipc.tx.send); /* other process can transfer packets from one ring to * other, use delay here to free that packets. */ sleep(1); - _ipc_free_ring_packets(pktio_entry->s.ipc.tx.free); + if (pktio_entry->s.ipc.tx.free) + _ipc_free_ring_packets(pktio_entry, pktio_entry->s.ipc.tx.free);

- tx_send = _ring_count(pktio_entry->s.ipc.tx.send); - tx_free = _ring_count(pktio_entry->s.ipc.tx.free); + if (pktio_entry->s.ipc.tx.send) + tx_send = _ring_count(pktio_entry->s.ipc.tx.send); + if (pktio_entry->s.ipc.tx.free) + tx_free = _ring_count(pktio_entry->s.ipc.tx.free); if (tx_send | tx_free) { ODP_DBG("IPC rings: tx send %d tx free %d\n", - _ring_free_count(pktio_entry->s.ipc.tx.send), - _ring_free_count(pktio_entry->s.ipc.tx.free)); + tx_send, tx_free); }

return 0; @@ -747,23 +691,31 @@ static int ipc_close(pktio_entry_t *pktio_entry) { char ipc_shm_name[ODP_POOL_NAME_LEN + sizeof("_m_prod")]; char *dev = pktio_entry->s.name; + char name[ODP_POOL_NAME_LEN]; + char tail[ODP_POOL_NAME_LEN]; + int pid = 0;

ipc_stop(pktio_entry);

- if (pktio_entry->s.ipc.type == PKTIO_TYPE_IPC_MASTER) { - /* unlink this pktio info for both master and slave */ - odp_shm_free(pktio_entry->s.ipc.pinfo_shm); - - /* destroy rings */ - snprintf(ipc_shm_name, sizeof(ipc_shm_name), "%s_s_cons", dev); - _ring_destroy(ipc_shm_name); - snprintf(ipc_shm_name, sizeof(ipc_shm_name), "%s_s_prod", dev); - _ring_destroy(ipc_shm_name); - snprintf(ipc_shm_name, sizeof(ipc_shm_name), "%s_m_cons", dev); - _ring_destroy(ipc_shm_name); - snprintf(ipc_shm_name, sizeof(ipc_shm_name), "%s_m_prod", dev); - _ring_destroy(ipc_shm_name); - } + odp_shm_free(pktio_entry->s.ipc.remote_pool_shm); + + if (sscanf(dev, "ipc:%d:%s", &pid, tail) == 2) + snprintf(name, sizeof(name), "ipc:%s", tail); + else + snprintf(name, sizeof(name), "%s", dev); + + /* unlink this pktio info for both master and slave */ + odp_shm_free(pktio_entry->s.ipc.pinfo_shm); + + /* destroy rings */ + snprintf(ipc_shm_name, sizeof(ipc_shm_name), "%s_s_cons", name); + _ring_destroy(ipc_shm_name); + snprintf(ipc_shm_name, sizeof(ipc_shm_name), "%s_s_prod", name); + _ring_destroy(ipc_shm_name); + snprintf(ipc_shm_name, sizeof(ipc_shm_name), "%s_m_cons", name); + _ring_destroy(ipc_shm_name); + snprintf(ipc_shm_name, sizeof(ipc_shm_name), "%s_m_prod", name); + _ring_destroy(ipc_shm_name);

return 0; } @@ -795,4 +747,3 @@ const pktio_if_ops_t ipc_pktio_ops = { .pktin_ts_from_ns = NULL, .config = NULL }; -#endif diff --git a/test/linux-generic/pktio_ipc/pktio_ipc_run.sh b/test/linux-generic/pktio_ipc/pktio_ipc_run.sh index 3cd28f5..52e8d42 100755 --- a/test/linux-generic/pktio_ipc/pktio_ipc_run.sh +++ b/test/linux-generic/pktio_ipc/pktio_ipc_run.sh @@ -25,19 +25,23 @@ run() rm -rf /tmp/odp-* 2>&1 > /dev/null

echo "==== run pktio_ipc1 then pktio_ipc2 ====" - pktio_ipc1${EXEEXT} -t 30 & + pktio_ipc1${EXEEXT} -t 10 & IPC_PID=$!

- pktio_ipc2${EXEEXT} -p ${IPC_PID} -t 10 + pktio_ipc2${EXEEXT} -p ${IPC_PID} -t 5 ret=$? # pktio_ipc1 should do clean up and exit just # after pktio_ipc2 exited. If it does not happen # kill him in test. - sleep 1 - kill ${IPC_PID} 2>&1 > /dev/null + sleep 13 + (kill ${IPC_PID} 2>&1 > /dev/null ) > /dev/null if [ $? -eq 0 ]; then - ls -l /tmp/odp* + echo "pktio_ipc1${EXEEXT} was killed" + ls -l /tmp/odp* 2> /dev/null rm -rf /tmp/odp-${IPC_PID}* 2>&1 > /dev/null + else + echo "normal exit of 2 application" + ls -l /tmp/odp* 2> /dev/null fi

if [ $ret -ne 0 ]; then @@ -47,21 +51,32 @@ run() echo "First stage PASSED" fi

- echo "==== run pktio_ipc2 then pktio_ipc1 ====" - pktio_ipc2${EXEEXT} -t 20 & + pktio_ipc2${EXEEXT} -t 10 & IPC_PID=$!

- pktio_ipc1${EXEEXT} -p ${IPC_PID} -t 10 + pktio_ipc1${EXEEXT} -p ${IPC_PID} -t 5 ret=$? - (kill ${IPC_PID} 2>&1 > /dev/null) > /dev/null || true + # pktio_ipc2 do not exit on pktio_ipc1 disconnect + # wait until it exits cleanly + sleep 13 + (kill ${IPC_PID} 2>&1 > /dev/null ) > /dev/null + if [ $? -eq 0 ]; then + echo "pktio_ipc2${EXEEXT} was killed" + ls -l /tmp/odp* 2> /dev/null + rm -rf /tmp/odp-${IPC_PID}* 2>&1 > /dev/null + else + echo "normal exit of 2 application" + ls -l /tmp/odp* 2> /dev/null + fi

if [ $ret -ne 0 ]; then echo "!!! FAILED !!!" - ls -l /tmp/odp* + ls -l /tmp/odp* 2> /dev/null rm -rf /tmp/odp-${IPC_PID}* 2>&1 > /dev/null exit $ret else + ls -l /tmp/odp* 2> /dev/null echo "Second stage PASSED" fi

commit 62164116ae9f072ddeaa3206eaf3b24c8dc9c884 Author: Maxim Uvarov maxim.uvarov@linaro.org Date: Wed Dec 14 22:57:55 2016 +0300

linux-gen: pktio ipc: update tests

Signed-off-by: Maxim Uvarov maxim.uvarov@linaro.org Reviewed-and-tested-by: Bill Fischofer bill.fischofer@linaro.org

diff --git a/test/linux-generic/pktio_ipc/ipc_common.c b/test/linux-generic/pktio_ipc/ipc_common.c index 387c921..85cbc8b 100644 --- a/test/linux-generic/pktio_ipc/ipc_common.c +++ b/test/linux-generic/pktio_ipc/ipc_common.c @@ -8,7 +8,8 @@

/** Run time in seconds */ int run_time_sec; -int ipc_name_space; +/** Pid of the master process */ +int master_pid;

int ipc_odp_packet_send_or_free(odp_pktio_t pktio, odp_packet_t pkt_tbl[], int num) @@ -33,6 +34,7 @@ int ipc_odp_packet_send_or_free(odp_pktio_t pktio, while (sent != num) { ret = odp_pktout_send(pktout, &pkt_tbl[sent], num - sent); if (ret < 0) { + EXAMPLE_ERR("odp_pktout_send return %d\n", ret); for (i = sent; i < num; i++) odp_packet_free(pkt_tbl[i]); return -1; @@ -43,6 +45,7 @@ int ipc_odp_packet_send_or_free(odp_pktio_t pktio, if (odp_time_cmp(end_time, odp_time_local()) < 0) { for (i = sent; i < num; i++) odp_packet_free(pkt_tbl[i]); + EXAMPLE_ERR("Send Timeout!\n"); return -1; } } @@ -50,17 +53,25 @@ int ipc_odp_packet_send_or_free(odp_pktio_t pktio, return 0; }

-odp_pktio_t create_pktio(odp_pool_t pool) +odp_pktio_t create_pktio(odp_pool_t pool, int master_pid) { odp_pktio_param_t pktio_param; odp_pktio_t ipc_pktio; + char name[30];

odp_pktio_param_init(&pktio_param);

- printf("pid: %d, create IPC pktio\n", getpid()); - ipc_pktio = odp_pktio_open("ipc_pktio", pool, &pktio_param); - if (ipc_pktio == ODP_PKTIO_INVALID) - EXAMPLE_ABORT("Error: ipc pktio create failed.\n"); + if (master_pid) + sprintf(name, TEST_IPC_PKTIO_PID_NAME, master_pid); + else + sprintf(name, TEST_IPC_PKTIO_NAME); + + printf("pid: %d, create IPC pktio %s\n", getpid(), name); + ipc_pktio = odp_pktio_open(name, pool, &pktio_param); + if (ipc_pktio == ODP_PKTIO_INVALID) { + EXAMPLE_ERR("Error: ipc pktio %s create failed.\n", name); + return ODP_PKTIO_INVALID; + }

if (odp_pktin_queue_config(ipc_pktio, NULL)) { EXAMPLE_ERR("Input queue config failed\n"); @@ -88,16 +99,16 @@ void parse_args(int argc, char *argv[]) int long_index; static struct option longopts[] = { {"time", required_argument, NULL, 't'}, - {"ns", required_argument, NULL, 'n'}, /* ipc name space */ + {"pid", required_argument, NULL, 'p'}, /* master process pid */ {"help", no_argument, NULL, 'h'}, /* return 'h' */ {NULL, 0, NULL, 0} };

run_time_sec = 0; /* loop forever if time to run is 0 */ - ipc_name_space = 0; + master_pid = 0;

while (1) { - opt = getopt_long(argc, argv, "+t:n:h", + opt = getopt_long(argc, argv, "+t:p:h", longopts, &long_index);

if (opt == -1) @@ -107,24 +118,18 @@ void parse_args(int argc, char *argv[]) case 't': run_time_sec = atoi(optarg); break; - case 'n': - ipc_name_space = atoi(optarg); + case 'p': + master_pid = atoi(optarg); break; case 'h': + default: usage(argv[0]); exit(EXIT_SUCCESS); break; - default: - break; } }

optind = 1; /* reset 'extern optind' from the getopt lib */ - - if (!ipc_name_space) { - usage(argv[0]); - exit(1); - } }

/** diff --git a/test/linux-generic/pktio_ipc/ipc_common.h b/test/linux-generic/pktio_ipc/ipc_common.h index 99276b5..8804994 100644 --- a/test/linux-generic/pktio_ipc/ipc_common.h +++ b/test/linux-generic/pktio_ipc/ipc_common.h @@ -30,7 +30,7 @@ /** @def SHM_PKT_POOL_BUF_SIZE * @brief Buffer size of the packet pool buffer */ -#define SHM_PKT_POOL_BUF_SIZE 1856 +#define SHM_PKT_POOL_BUF_SIZE 100

/** @def MAX_PKT_BURST * @brief Maximum number of packet bursts @@ -46,6 +46,12 @@

#define TEST_ALLOC_MAGIC 0x1234adcd

+#define TEST_IPC_PKTIO_NAME "ipc:ipktio" +#define TEST_IPC_PKTIO_PID_NAME "ipc:%d:ipktio" + +/** Can be any name, same or not the same. */ +#define TEST_IPC_POOL_NAME "ipc_packet_pool" + /** magic number and sequence at start of packet payload */ typedef struct ODP_PACKED { odp_u32be_t magic; @@ -63,8 +69,8 @@ char *pktio_name; /** Run time in seconds */ int run_time_sec;

-/** IPC name space id /dev/shm/odp-nsid-objname */ -int ipc_name_space; +/** PID of the master process */ +int master_pid;

/* helper funcs */ void parse_args(int argc, char *argv[]); @@ -75,11 +81,12 @@ void usage(char *progname); * Create a ipc pktio handle. * * @param pool Pool to associate with device for packet RX/TX + * @param master_pid Pid of master process * * @return The handle of the created pktio object. * @retval ODP_PKTIO_INVALID if the create fails. */ -odp_pktio_t create_pktio(odp_pool_t pool); +odp_pktio_t create_pktio(odp_pool_t pool, int master_pid);

/** Spin and send all packet from table * diff --git a/test/linux-generic/pktio_ipc/pktio_ipc1.c b/test/linux-generic/pktio_ipc/pktio_ipc1.c index 5c1da23..838b672 100644 --- a/test/linux-generic/pktio_ipc/pktio_ipc1.c +++ b/test/linux-generic/pktio_ipc/pktio_ipc1.c @@ -23,9 +23,8 @@ */ static int pktio_run_loop(odp_pool_t pool) { - int thr; int pkts; - odp_pktio_t ipc_pktio; + odp_pktio_t ipc_pktio = ODP_PKTIO_INVALID; odp_packet_t pkt_tbl[MAX_PKT_BURST]; uint64_t cnt = 0; /* increasing counter on each send packet */ uint64_t cnt_recv = 0; /* increasing counter to validate @@ -42,22 +41,41 @@ static int pktio_run_loop(odp_pool_t pool) odp_time_t wait; int ret; odp_pktin_queue_t pktin; + char name[30];

- thr = odp_thread_id(); - - ipc_pktio = odp_pktio_lookup("ipc_pktio"); - if (ipc_pktio == ODP_PKTIO_INVALID) { - EXAMPLE_ERR(" [%02i] Error: lookup of pktio %s failed\n", - thr, "ipc_pktio"); - return -2; - } - printf(" [%02i] looked up ipc_pktio:%02" PRIu64 ", burst mode\n", - thr, odp_pktio_to_u64(ipc_pktio)); + if (master_pid) + sprintf(name, TEST_IPC_PKTIO_PID_NAME, master_pid); + else + sprintf(name, TEST_IPC_PKTIO_NAME);

wait = odp_time_local_from_ns(run_time_sec * ODP_TIME_SEC_IN_NS); start_cycle = odp_time_local(); current_cycle = start_cycle;

+ /* slave process should always be run after master process to be + * able to create the same pktio. + */ + for (;;) { + if (run_time_sec) { + cycle = odp_time_local(); + diff = odp_time_diff(cycle, start_cycle); + if (odp_time_cmp(wait, diff) < 0) { + printf("timeout exit, run_time_sec %d\n", + run_time_sec); + return -1; + } + } + + ipc_pktio = create_pktio(pool, master_pid); + if (ipc_pktio != ODP_PKTIO_INVALID) + break; + if (!master_pid) + break; + } + + if (ipc_pktio == ODP_PKTIO_INVALID) + return -1; + if (odp_pktin_queue(ipc_pktio, &pktin, 1) != 1) { EXAMPLE_ERR("no input queue\n"); return -1; @@ -110,8 +128,12 @@ static int pktio_run_loop(odp_pool_t pool) size_t off;

off = odp_packet_l4_offset(pkt); - if (off == ODP_PACKET_OFFSET_INVALID) - EXAMPLE_ABORT("invalid l4 offset\n"); + if (off == ODP_PACKET_OFFSET_INVALID) { + stat_errors++; + stat_free++; + odp_packet_free(pkt); + EXAMPLE_ERR("invalid l4 offset\n"); + }

off += ODPH_UDPHDR_LEN; ret = odp_packet_copy_to_mem(pkt, off, @@ -279,17 +301,13 @@ int main(int argc, char *argv[]) odp_pool_t pool; odp_pool_param_t params; odp_instance_t instance; - odp_platform_init_t plat_idata; int ret;

/* Parse and store the application arguments */ parse_args(argc, argv);

- memset(&plat_idata, 0, sizeof(odp_platform_init_t)); - plat_idata.ipc_ns = ipc_name_space; - /* Init ODP before calling anything else */ - if (odp_init_global(&instance, NULL, &plat_idata)) { + if (odp_init_global(&instance, NULL, NULL)) { EXAMPLE_ERR("Error: ODP global init failed.\n"); exit(EXIT_FAILURE); } @@ -310,7 +328,7 @@ int main(int argc, char *argv[]) params.pkt.num = SHM_PKT_POOL_SIZE; params.type = ODP_POOL_PACKET;

- pool = odp_pool_create("packet_pool1", &params); + pool = odp_pool_create(TEST_IPC_POOL_NAME, &params); if (pool == ODP_POOL_INVALID) { EXAMPLE_ERR("Error: packet pool create failed.\n"); exit(EXIT_FAILURE); @@ -318,8 +336,6 @@ int main(int argc, char *argv[])

odp_pool_print(pool);

- create_pktio(pool); - ret = pktio_run_loop(pool);

if (odp_pool_destroy(pool)) { diff --git a/test/linux-generic/pktio_ipc/pktio_ipc2.c b/test/linux-generic/pktio_ipc/pktio_ipc2.c index 5c1f142..fb6f994 100644 --- a/test/linux-generic/pktio_ipc/pktio_ipc2.c +++ b/test/linux-generic/pktio_ipc/pktio_ipc2.c @@ -16,9 +16,9 @@

#include "ipc_common.h"

-static int ipc_second_process(void) +static int ipc_second_process(int master_pid) { - odp_pktio_t ipc_pktio; + odp_pktio_t ipc_pktio = ODP_PKTIO_INVALID; odp_pool_param_t params; odp_pool_t pool; odp_packet_t pkt_tbl[MAX_PKT_BURST]; @@ -40,18 +40,44 @@ static int ipc_second_process(void) params.pkt.num = SHM_PKT_POOL_SIZE; params.type = ODP_POOL_PACKET;

- pool = odp_pool_create("packet_pool2", &params); + pool = odp_pool_create(TEST_IPC_POOL_NAME, &params); if (pool == ODP_POOL_INVALID) { EXAMPLE_ERR("Error: packet pool create failed.\n"); exit(EXIT_FAILURE); }

- ipc_pktio = create_pktio(pool); - wait = odp_time_local_from_ns(run_time_sec * ODP_TIME_SEC_IN_NS); start_cycle = odp_time_local();

+ /* slave process should always be run after master process to be + * able to create the same pktio. + */ + for (;;) { + /* exit loop if time specified */ + if (run_time_sec) { + cycle = odp_time_local(); + diff = odp_time_diff(cycle, start_cycle); + if (odp_time_cmp(wait, diff) < 0) { + printf("timeout exit, run_time_sec %d\n", + run_time_sec); + goto not_started; + } + } + + ipc_pktio = create_pktio(pool, master_pid); + if (ipc_pktio != ODP_PKTIO_INVALID) + break; + if (!master_pid) + break; + } + + if (ipc_pktio == ODP_PKTIO_INVALID) { + odp_pool_destroy(pool); + return -1; + } + if (odp_pktin_queue(ipc_pktio, &pktin, 1) != 1) { + odp_pool_destroy(pool); EXAMPLE_ERR("no input queue\n"); return -1; } @@ -97,8 +123,12 @@ static int ipc_second_process(void) size_t off;

off = odp_packet_l4_offset(pkt); - if (off == ODP_PACKET_OFFSET_INVALID) - EXAMPLE_ABORT("invalid l4 offset\n"); + if (off == ODP_PACKET_OFFSET_INVALID) { + EXAMPLE_ERR("invalid l4 offset\n"); + for (int j = i; j < pkts; j++) + odp_packet_free(pkt_tbl[j]); + break; + }

off += ODPH_UDPHDR_LEN; ret = odp_packet_copy_to_mem(pkt, off, sizeof(head), @@ -106,8 +136,12 @@ static int ipc_second_process(void) if (ret) EXAMPLE_ABORT("unable copy out head data");

- if (head.magic != TEST_SEQ_MAGIC) - EXAMPLE_ABORT("Wrong head magic!"); + if (head.magic != TEST_SEQ_MAGIC) { + EXAMPLE_ERR("Wrong head magic! %x", head.magic); + for (int j = i; j < pkts; j++) + odp_packet_free(pkt_tbl[j]); + break; + }

/* Modify magic number in packet */ head.magic = TEST_SEQ_MAGIC_2; @@ -118,7 +152,7 @@ static int ipc_second_process(void) }

/* send all packets back */ - ret = ipc_odp_packet_send_or_free(ipc_pktio, pkt_tbl, pkts); + ret = ipc_odp_packet_send_or_free(ipc_pktio, pkt_tbl, i); if (ret < 0) EXAMPLE_ABORT("can not send packets\n");

@@ -176,16 +210,12 @@ not_started: int main(int argc, char *argv[]) { odp_instance_t instance; - odp_platform_init_t plat_idata; int ret;

/* Parse and store the application arguments */ parse_args(argc, argv);

- memset(&plat_idata, 0, sizeof(odp_platform_init_t)); - plat_idata.ipc_ns = ipc_name_space; - - if (odp_init_global(&instance, NULL, &plat_idata)) { + if (odp_init_global(&instance, NULL, NULL)) { EXAMPLE_ERR("Error: ODP global init failed.\n"); exit(EXIT_FAILURE); } @@ -196,7 +226,7 @@ int main(int argc, char *argv[]) exit(EXIT_FAILURE); }

- ret = ipc_second_process(); + ret = ipc_second_process(master_pid);

if (odp_term_local()) { EXAMPLE_ERR("Error: odp_term_local() failed.\n"); diff --git a/test/linux-generic/pktio_ipc/pktio_ipc_run.sh b/test/linux-generic/pktio_ipc/pktio_ipc_run.sh index bd64baf..3cd28f5 100755 --- a/test/linux-generic/pktio_ipc/pktio_ipc_run.sh +++ b/test/linux-generic/pktio_ipc/pktio_ipc_run.sh @@ -20,20 +20,15 @@ PATH=.:$PATH run() { local ret=0 - IPC_NS=`expr $$ + 5000` - IPC_NS=`expr ${IPC_NS} % 65000` - IPC_NS=`expr ${IPC_NS} + 2` - echo "Using ns ${IPC_NS}" - #if test was interrupted with CTRL+c than files #might remain in shm. Needed cleanely delete them. - rm -rf /dev/shm/odp-${IPC_NS}* 2>&1 > /dev/null + rm -rf /tmp/odp-* 2>&1 > /dev/null

echo "==== run pktio_ipc1 then pktio_ipc2 ====" - pktio_ipc1${EXEEXT} -n ${IPC_NS} -t 30 & + pktio_ipc1${EXEEXT} -t 30 & IPC_PID=$!

- pktio_ipc2${EXEEXT} -n ${IPC_NS} -t 10 + pktio_ipc2${EXEEXT} -p ${IPC_PID} -t 10 ret=$? # pktio_ipc1 should do clean up and exit just # after pktio_ipc2 exited. If it does not happen @@ -41,12 +36,12 @@ run() sleep 1 kill ${IPC_PID} 2>&1 > /dev/null if [ $? -eq 0 ]; then - rm -rf /dev/shm/odp-${IPC_NS}* 2>&1 > /dev/null + ls -l /tmp/odp* + rm -rf /tmp/odp-${IPC_PID}* 2>&1 > /dev/null fi

if [ $ret -ne 0 ]; then echo "!!!First stage FAILED $ret!!!" - ls -l /dev/shm/ exit $ret else echo "First stage PASSED" @@ -54,19 +49,17 @@ run()

echo "==== run pktio_ipc2 then pktio_ipc1 ====" - IPC_NS=`expr $IPC_NS - 1` - echo "Using ns ${IPC_NS}" - - pktio_ipc2${EXEEXT} -n ${IPC_NS} -t 10 & + pktio_ipc2${EXEEXT} -t 20 & IPC_PID=$!

- pktio_ipc1${EXEEXT} -n ${IPC_NS} -t 20 + pktio_ipc1${EXEEXT} -p ${IPC_PID} -t 10 ret=$? (kill ${IPC_PID} 2>&1 > /dev/null) > /dev/null || true

if [ $ret -ne 0 ]; then echo "!!! FAILED !!!" - ls -l /dev/shm/ + ls -l /tmp/odp* + rm -rf /tmp/odp-${IPC_PID}* 2>&1 > /dev/null exit $ret else echo "Second stage PASSED"

commit fdb1aa2981dbc68c63e11abd2663294338b13764 Author: Maxim Uvarov maxim.uvarov@linaro.org Date: Wed Dec 14 22:57:54 2016 +0300

linux-gen: pktio ipc: more accurate settings of pool flags

Make code more accurate and nice view, no functional changes.

Signed-off-by: Maxim Uvarov maxim.uvarov@linaro.org Reviewed-and-tested-by: Bill Fischofer bill.fischofer@linaro.org

diff --git a/platform/linux-generic/odp_pool.c b/platform/linux-generic/odp_pool.c index 4be3827..cae2759 100644 --- a/platform/linux-generic/odp_pool.c +++ b/platform/linux-generic/odp_pool.c @@ -497,14 +497,17 @@ static int check_params(odp_pool_param_t *params)

odp_pool_t odp_pool_create(const char *name, odp_pool_param_t *params) { + uint32_t shm_flags = 0; + + if (check_params(params)) + return ODP_POOL_INVALID; + #ifdef _ODP_PKTIO_IPC if (params && (params->type == ODP_POOL_PACKET)) - return pool_create(name, params, ODP_SHM_PROC); + shm_flags = ODP_SHM_PROC; #endif - if (check_params(params)) - return ODP_POOL_INVALID;

- return pool_create(name, params, 0); + return pool_create(name, params, shm_flags); }

int odp_pool_destroy(odp_pool_t pool_hdl)

commit 023d4a942e89f86e66c6d5226d6e792936c37df3 Author: Maxim Uvarov maxim.uvarov@linaro.org Date: Wed Dec 14 22:57:53 2016 +0300

linux-gen: pktio ipc: ring changes

Make rings visible by other processes.

Signed-off-by: Maxim Uvarov maxim.uvarov@linaro.org Reviewed-and-tested-by: Bill Fischofer bill.fischofer@linaro.org

diff --git a/platform/linux-generic/pktio/ring.c b/platform/linux-generic/pktio/ring.c index cc84e8a..aeda04b 100644 --- a/platform/linux-generic/pktio/ring.c +++ b/platform/linux-generic/pktio/ring.c @@ -160,7 +160,7 @@ _ring_create(const char *name, unsigned count, unsigned flags) odp_shm_t shm;

if (flags & _RING_SHM_PROC) - shm_flag = ODP_SHM_PROC; + shm_flag = ODP_SHM_PROC | ODP_SHM_EXPORT; else shm_flag = 0;

commit e9b7c770c39c2409da68619a8ecc9090acda0c12 Author: Maxim Uvarov maxim.uvarov@linaro.org Date: Tue Dec 13 16:34:31 2016 +0300

configure.ac: do not disable shared lib for non abi mode

After discussion on mailing list there is case to support shared library for non abi compatibility mode.

Signed-off-by: Maxim Uvarov maxim.uvarov@linaro.org Reviewed-by: Bill Fischofer bill.fischofer@linaro.org

diff --git a/configure.ac b/configure.ac index 3ddd238..3a20959 100644 --- a/configure.ac +++ b/configure.ac @@ -255,7 +255,6 @@ AC_ARG_ENABLE([abi-compat], [if test "x$enableval" = "xno"; then ODP_ABI_COMPAT=0 abi_compat=no - enable_shared=no #if there is no ABI compatibility the .so numbers are meaningless ODP_LIBSO_VERSION=0:0:0 fi])

commit 3c2cc60893a9b1fe62962df73bfe35b9dbc319c2 Author: Nicolas Morey-Chaisemartin nmorey@kalray.eu Date: Tue Dec 13 17:55:50 2016 +0100

example: traffic_mgmt: use PRIu32 instead of %u

Signed-off-by: Nicolas Morey-Chaisemartin nmorey@kalray.eu Reviewed-and-tested-by: Bill Fischofer bill.fischofer@linaro.org Signed-off-by: Maxim Uvarov maxim.uvarov@linaro.org

diff --git a/example/traffic_mgmt/odp_traffic_mgmt.c b/example/traffic_mgmt/odp_traffic_mgmt.c index c4f5356..c83b615 100644 --- a/example/traffic_mgmt/odp_traffic_mgmt.c +++ b/example/traffic_mgmt/odp_traffic_mgmt.c @@ -10,6 +10,7 @@

#include <unistd.h> #include <signal.h> +#include <inttypes.h> #include <sys/resource.h> #include <execinfo.h> #include <odp_api.h> @@ -261,7 +262,8 @@ static uint32_t create_profile_set(profile_params_set_t *profile_params_set, if (name_idx == 0) snprintf(name, sizeof(name), "%s", base_name); else - snprintf(name, sizeof(name), "%s-%u", base_name, name_idx); + snprintf(name, sizeof(name), "%s-%" PRIu32, + base_name, name_idx);

odp_tm_shaper_params_init(&shaper_params); shaper = &profile_params_set->shaper_params; @@ -289,7 +291,8 @@ static uint32_t create_profile_set(profile_params_set_t *profile_params_set, err_cnt++;

for (color = 0; color < ODP_NUM_PACKET_COLORS; color++) { - snprintf(wred_name, sizeof(wred_name), "%s-%u", name, color); + snprintf(wred_name, sizeof(wred_name), "%s-%" PRIu32, + name, color);

odp_tm_wred_params_init(&wred_params); wred = &profile_params_set->wred_params[color]; @@ -400,7 +403,7 @@ static int config_example_user(odp_tm_node_t cos_tm_node, profile_set->wred_profiles[2]; tm_node_params.level = 2;

- snprintf(user_name, sizeof(user_name), "Subscriber-%u", user_num); + snprintf(user_name, sizeof(user_name), "Subscriber-%" PRIu32, user_num); user_tm_node = odp_tm_node_create(odp_tm_test, user_name, &tm_node_params); odp_tm_node_connect(user_tm_node, cos_tm_node); @@ -478,8 +481,8 @@ static int config_company_node(const char *company_name) tm_node_params.wred_profile[ODP_PACKET_RED] = profile_set->wred_profiles[ODP_PACKET_RED];

- snprintf(cos_node_name, sizeof(cos_node_name), "%s-Class-%u", - company_name, cos_idx); + snprintf(cos_node_name, sizeof(cos_node_name), + "%s-Class-%" PRIu32, company_name, cos_idx); cos_tm_node = odp_tm_node_create(odp_tm_test, cos_node_name, &tm_node_params); odp_tm_node_connect(cos_tm_node, company_tm_node); @@ -528,7 +531,7 @@ static int create_and_config_tm(void) odp_tm_test = odp_tm_create("TM test", &requirements, &egress); err_cnt = init_profile_sets(); if (err_cnt != 0) - printf("%s init_profile_sets encountered %u errors\n", + printf("%s init_profile_sets encountered %" PRIu32 " errors\n", __func__, err_cnt);

config_company_node("TestCompany"); @@ -644,7 +647,7 @@ static int traffic_generator(uint32_t pkts_to_send) odp_atomic_inc_u32(&atomic_pkts_into_tm); }

- printf("%s odp_tm_enq_errs=%u\n", __func__, odp_tm_enq_errs); + printf("%s odp_tm_enq_errs=%" PRIu32 "\n", __func__, odp_tm_enq_errs);

/* Wait until the main traffic mgmt worker thread is idle and has no * outstanding events (i.e. no timers, empty work queue, etc), but @@ -786,7 +789,8 @@ int main(int argc, char *argv[])

pkts_into_tm = odp_atomic_load_u32(&atomic_pkts_into_tm); pkts_from_tm = odp_atomic_load_u32(&atomic_pkts_from_tm); - printf("pkts_into_tm=%u pkts_from_tm=%u\n", pkts_into_tm, pkts_from_tm); + printf("pkts_into_tm=%" PRIu32 " pkts_from_tm=%" PRIu32 "\n", + pkts_into_tm, pkts_from_tm);

odp_tm_stats_print(odp_tm_test); return 0;

commit af005d182ee0eb332cab3fa8f62cdb91135bcaa2 Author: Petri Savolainen petri.savolainen@nokia.com Date: Thu Dec 8 16:28:53 2016 +0200

example: ipsec: use op_param_t instead of op_params_t

Type name odp_crypto_op_params_t is deprecated.

Signed-off-by: Petri Savolainen petri.savolainen@nokia.com Reviewed-by: Bill Fischofer bill.fischofer@linaro.org Reviewed-by: Balasubramanian Manoharan bala.manoharan@linaro.org Reviewed-by: Nikhil Agarwal nikhil.agarwal@linaro.org Signed-off-by: Maxim Uvarov maxim.uvarov@linaro.org

diff --git a/example/ipsec/odp_ipsec.c b/example/ipsec/odp_ipsec.c index 6a9a9fe..c51922a 100644 --- a/example/ipsec/odp_ipsec.c +++ b/example/ipsec/odp_ipsec.c @@ -148,7 +148,7 @@ typedef struct { uint32_t dst_ip; /**< SA dest IP address */

/* Output only */ - odp_crypto_op_params_t params; /**< Parameters for crypto call */ + odp_crypto_op_param_t params; /**< Parameters for crypto call */ uint32_t *ah_seq; /**< AH sequence number location */ uint32_t *esp_seq; /**< ESP sequence number location */ uint16_t *tun_hdr_id; /**< Tunnel header ID > */ @@ -644,7 +644,7 @@ pkt_disposition_e do_ipsec_in_classify(odp_packet_t pkt, odph_ahhdr_t *ah = NULL; odph_esphdr_t *esp = NULL; ipsec_cache_entry_t *entry; - odp_crypto_op_params_t params; + odp_crypto_op_param_t params; odp_bool_t posted = 0;

/* Default to skip IPsec */ @@ -823,7 +823,7 @@ pkt_disposition_e do_ipsec_out_classify(odp_packet_t pkt, uint16_t ip_data_len = ipv4_data_len(ip); uint8_t *ip_data = ipv4_data_p(ip); ipsec_cache_entry_t *entry; - odp_crypto_op_params_t params; + odp_crypto_op_param_t params; int hdr_len = 0; int trl_len = 0; odph_ahhdr_t *ah = NULL;

commit 942908c478c41038bbf1d360b664d53b42f7f76e Author: Petri Savolainen petri.savolainen@nokia.com Date: Thu Dec 8 16:05:30 2016 +0200

validation: crypto: use algorithm capability

Use new algorithm enumerations and capability functions.

Signed-off-by: Petri Savolainen petri.savolainen@nokia.com Reviewed-by: Bill Fischofer bill.fischofer@linaro.org Reviewed-by: Balasubramanian Manoharan bala.manoharan@linaro.org Reviewed-by: Nikhil Agarwal nikhil.agarwal@linaro.org Signed-off-by: Maxim Uvarov maxim.uvarov@linaro.org

diff --git a/test/common_plat/validation/api/crypto/odp_crypto_test_inp.c b/test/common_plat/validation/api/crypto/odp_crypto_test_inp.c index 55fc6aa..de9d6e4 100644 --- a/test/common_plat/validation/api/crypto/odp_crypto_test_inp.c +++ b/test/common_plat/validation/api/crypto/odp_crypto_test_inp.c @@ -11,6 +11,8 @@ #include "odp_crypto_test_inp.h" #include "crypto.h"

+#define MAX_ALG_CAPA 32 + struct suite_context_s { odp_crypto_op_mode_t pref_mode; odp_pool_t pool; @@ -42,8 +44,7 @@ static void alg_test(odp_crypto_op_t op, const uint8_t *ciphertext, unsigned int ciphertext_len, const uint8_t *digest, - unsigned int digest_len - ) + uint32_t digest_len) { odp_crypto_session_t session; odp_crypto_capability_t capability; @@ -57,6 +58,10 @@ static void alg_test(odp_crypto_op_t op, odp_crypto_op_param_t op_params; uint8_t *data_addr; int data_off; + odp_crypto_cipher_capability_t cipher_capa[MAX_ALG_CAPA]; + odp_crypto_auth_capability_t auth_capa[MAX_ALG_CAPA]; + int num, i; + int found;

rc = odp_crypto_capability(&capability); CU_ASSERT(!rc); @@ -65,36 +70,36 @@ static void alg_test(odp_crypto_op_t op, if (cipher_alg == ODP_CIPHER_ALG_3DES_CBC && !(capability.hw_ciphers.bit.trides_cbc)) rc = -1; - if (cipher_alg == ODP_CIPHER_ALG_AES128_CBC && - !(capability.hw_ciphers.bit.aes128_cbc)) + if (cipher_alg == ODP_CIPHER_ALG_AES_CBC && + !(capability.hw_ciphers.bit.aes_cbc)) rc = -1; - if (cipher_alg == ODP_CIPHER_ALG_AES128_GCM && - !(capability.hw_ciphers.bit.aes128_gcm)) + if (cipher_alg == ODP_CIPHER_ALG_AES_GCM && + !(capability.hw_ciphers.bit.aes_gcm)) rc = -1; } else { if (cipher_alg == ODP_CIPHER_ALG_3DES_CBC && !(capability.ciphers.bit.trides_cbc)) rc = -1; - if (cipher_alg == ODP_CIPHER_ALG_AES128_CBC && - !(capability.ciphers.bit.aes128_cbc)) + if (cipher_alg == ODP_CIPHER_ALG_AES_CBC && + !(capability.ciphers.bit.aes_cbc)) rc = -1; - if (cipher_alg == ODP_CIPHER_ALG_AES128_GCM && - !(capability.ciphers.bit.aes128_gcm)) + if (cipher_alg == ODP_CIPHER_ALG_AES_GCM && + !(capability.ciphers.bit.aes_gcm)) rc = -1; }

CU_ASSERT(!rc);

if (capability.hw_auths.all_bits) { - if (auth_alg == ODP_AUTH_ALG_AES128_GCM && - !(capability.hw_auths.bit.aes128_gcm)) + if (auth_alg == ODP_AUTH_ALG_AES_GCM && + !(capability.hw_auths.bit.aes_gcm)) rc = -1; if (auth_alg == ODP_AUTH_ALG_NULL && !(capability.hw_auths.bit.null)) rc = -1; } else { - if (auth_alg == ODP_AUTH_ALG_AES128_GCM && - !(capability.auths.bit.aes128_gcm)) + if (auth_alg == ODP_AUTH_ALG_AES_GCM && + !(capability.auths.bit.aes_gcm)) rc = -1; if (auth_alg == ODP_AUTH_ALG_NULL && !(capability.auths.bit.null)) @@ -103,6 +108,59 @@ static void alg_test(odp_crypto_op_t op,

CU_ASSERT(!rc);

+ num = odp_crypto_cipher_capability(cipher_alg, cipher_capa, + MAX_ALG_CAPA); + + if (cipher_alg != ODP_CIPHER_ALG_NULL) { + CU_ASSERT(num > 0); + found = 0; + } else { + CU_ASSERT(num == 0); + found = 1; + } + + CU_ASSERT(num <= MAX_ALG_CAPA); + + if (num > MAX_ALG_CAPA) + num = MAX_ALG_CAPA; + + /* Search for the test case */ + for (i = 0; i < num; i++) { + if (cipher_capa[i].key_len == cipher_key.length && + cipher_capa[i].iv_len == ses_iv.length) { + found = 1; + break; + } + } + + CU_ASSERT(found); + + num = odp_crypto_auth_capability(auth_alg, auth_capa, MAX_ALG_CAPA); + + if (auth_alg != ODP_AUTH_ALG_NULL) { + CU_ASSERT(num > 0); + found = 0; + } else { + CU_ASSERT(num == 0); + found = 1; + } + + CU_ASSERT(num <= MAX_ALG_CAPA); + + if (num > MAX_ALG_CAPA) + num = MAX_ALG_CAPA; + + /* Search for the test case */ + for (i = 0; i < num; i++) { + if (auth_capa[i].digest_len == digest_len && + auth_capa[i].key_len == auth_key.length) { + found = 1; + break; + } + } + + CU_ASSERT(found); + /* Create a crypto session */ odp_crypto_session_param_init(&ses_params); ses_params.op = op; @@ -345,11 +403,11 @@ void crypto_test_enc_alg_aes128_gcm(void) iv.length = sizeof(aes128_gcm_reference_iv[i]);

alg_test(ODP_CRYPTO_OP_ENCODE, - ODP_CIPHER_ALG_AES128_GCM, + ODP_CIPHER_ALG_AES_GCM, iv, NULL, cipher_key, - ODP_AUTH_ALG_AES128_GCM, + ODP_AUTH_ALG_AES_GCM, auth_key, &aes128_gcm_cipher_range[i], &aes128_gcm_auth_range[i], @@ -381,11 +439,11 @@ void crypto_test_enc_alg_aes128_gcm_ovr_iv(void) cipher_key.length = sizeof(aes128_gcm_reference_key[i]);

alg_test(ODP_CRYPTO_OP_ENCODE, - ODP_CIPHER_ALG_AES128_GCM, + ODP_CIPHER_ALG_AES_GCM, iv, aes128_gcm_reference_iv[i], cipher_key, - ODP_AUTH_ALG_AES128_GCM, + ODP_AUTH_ALG_AES_GCM, auth_key, &aes128_gcm_cipher_range[i], &aes128_gcm_auth_range[i], @@ -420,11 +478,11 @@ void crypto_test_dec_alg_aes128_gcm(void) iv.length = sizeof(aes128_gcm_reference_iv[i]);

alg_test(ODP_CRYPTO_OP_DECODE, - ODP_CIPHER_ALG_AES128_GCM, + ODP_CIPHER_ALG_AES_GCM, iv, NULL, cipher_key, - ODP_AUTH_ALG_AES128_GCM, + ODP_AUTH_ALG_AES_GCM, auth_key, &aes128_gcm_cipher_range[i], &aes128_gcm_auth_range[i], @@ -457,11 +515,11 @@ void crypto_test_dec_alg_aes128_gcm_ovr_iv(void) cipher_key.length = sizeof(aes128_gcm_reference_key[i]);

alg_test(ODP_CRYPTO_OP_DECODE, - ODP_CIPHER_ALG_AES128_GCM, + ODP_CIPHER_ALG_AES_GCM, iv, aes128_gcm_reference_iv[i], cipher_key, - ODP_AUTH_ALG_AES128_GCM, + ODP_AUTH_ALG_AES_GCM, auth_key, &aes128_gcm_cipher_range[i], &aes128_gcm_auth_range[i], @@ -495,7 +553,7 @@ void crypto_test_enc_alg_aes128_cbc(void) iv.length = sizeof(aes128_cbc_reference_iv[i]);

alg_test(ODP_CRYPTO_OP_ENCODE, - ODP_CIPHER_ALG_AES128_CBC, + ODP_CIPHER_ALG_AES_CBC, iv, NULL, cipher_key, @@ -526,7 +584,7 @@ void crypto_test_enc_alg_aes128_cbc_ovr_iv(void) cipher_key.length = sizeof(aes128_cbc_reference_key[i]);

alg_test(ODP_CRYPTO_OP_ENCODE, - ODP_CIPHER_ALG_AES128_CBC, + ODP_CIPHER_ALG_AES_CBC, iv, aes128_cbc_reference_iv[i], cipher_key, @@ -561,7 +619,7 @@ void crypto_test_dec_alg_aes128_cbc(void) iv.length = sizeof(aes128_cbc_reference_iv[i]);

alg_test(ODP_CRYPTO_OP_DECODE, - ODP_CIPHER_ALG_AES128_CBC, + ODP_CIPHER_ALG_AES_CBC, iv, NULL, cipher_key, @@ -594,7 +652,7 @@ void crypto_test_dec_alg_aes128_cbc_ovr_iv(void) cipher_key.length = sizeof(aes128_cbc_reference_key[i]);

alg_test(ODP_CRYPTO_OP_DECODE, - ODP_CIPHER_ALG_AES128_CBC, + ODP_CIPHER_ALG_AES_CBC, iv, aes128_cbc_reference_iv[i], cipher_key, @@ -634,7 +692,7 @@ void crypto_test_alg_hmac_md5(void) iv, iv.data, cipher_key, - ODP_AUTH_ALG_MD5_96, + ODP_AUTH_ALG_MD5_HMAC, auth_key, NULL, NULL, hmac_md5_reference_plaintext[i], @@ -672,7 +730,7 @@ void crypto_test_alg_hmac_sha256(void) iv, iv.data, cipher_key, - ODP_AUTH_ALG_SHA256_128, + ODP_AUTH_ALG_SHA256_HMAC, auth_key, NULL, NULL, hmac_sha256_reference_plaintext[i],

commit 579fac5fead538bfb4f052e5842ebc8176bbf5ea Author: Petri Savolainen petri.savolainen@nokia.com Date: Thu Dec 8 16:05:29 2016 +0200

test: crypto: use odp_crypto_session_param_init

Use session param init function instead of memset() to zero.

Signed-off-by: Petri Savolainen petri.savolainen@nokia.com Reviewed-by: Bill Fischofer bill.fischofer@linaro.org Reviewed-by: Balasubramanian Manoharan bala.manoharan@linaro.org Reviewed-by: Nikhil Agarwal nikhil.agarwal@linaro.org Signed-off-by: Maxim Uvarov maxim.uvarov@linaro.org

diff --git a/example/ipsec/odp_ipsec_cache.c b/example/ipsec/odp_ipsec_cache.c index 2bd44cf..b2a91c2 100644 --- a/example/ipsec/odp_ipsec_cache.c +++ b/example/ipsec/odp_ipsec_cache.c @@ -44,7 +44,7 @@ int create_ipsec_cache_entry(sa_db_entry_t *cipher_sa, odp_queue_t completionq, odp_pool_t out_pool) { - odp_crypto_session_params_t params; + odp_crypto_session_param_t params; ipsec_cache_entry_t *entry; odp_crypto_ses_create_err_t ses_create_rc; odp_crypto_session_t session; @@ -60,6 +60,8 @@ int create_ipsec_cache_entry(sa_db_entry_t *cipher_sa, (cipher_sa->mode != auth_sa->mode)) return -1;

+ odp_crypto_session_param_init(&params); + /* Setup parameters and call crypto library to create session */ params.op = (in) ? ODP_CRYPTO_OP_DECODE : ODP_CRYPTO_OP_ENCODE; params.auth_cipher_text = TRUE; diff --git a/test/common_plat/performance/odp_crypto.c b/test/common_plat/performance/odp_crypto.c index 39df78b..9936288 100644 --- a/test/common_plat/performance/odp_crypto.c +++ b/test/common_plat/performance/odp_crypto.c @@ -49,7 +49,7 @@ static uint8_t test_key24[24] = { 0x01, 0x02, 0x03, 0x04, 0x05, */ typedef struct { const char *name; /**< Algorithm name */ - odp_crypto_session_params_t session; /**< Prefilled crypto session params */ + odp_crypto_session_param_t session; /**< Prefilled crypto session params */ unsigned int hash_adjust; /**< Size of hash */ } crypto_alg_config_t;

@@ -420,12 +420,13 @@ create_session_from_config(odp_crypto_session_t *session, crypto_alg_config_t *config, crypto_args_t *cargs) { - odp_crypto_session_params_t params; + odp_crypto_session_param_t params; odp_crypto_ses_create_err_t ses_create_rc; odp_pool_t pkt_pool; odp_queue_t out_queue;

- memcpy(&params, &config->session, sizeof(odp_crypto_session_params_t)); + odp_crypto_session_param_init(&params); + memcpy(&params, &config->session, sizeof(odp_crypto_session_param_t)); params.op = ODP_CRYPTO_OP_ENCODE; params.pref_mode = ODP_CRYPTO_SYNC;

@@ -468,7 +469,7 @@ run_measure_one(crypto_args_t *cargs, unsigned int payload_length, crypto_run_result_t *result) { - odp_crypto_op_params_t params; + odp_crypto_op_param_t params;

odp_pool_t pkt_pool; odp_queue_t out_queue; diff --git a/test/common_plat/validation/api/crypto/odp_crypto_test_inp.c b/test/common_plat/validation/api/crypto/odp_crypto_test_inp.c index 4ac4a07..55fc6aa 100644 --- a/test/common_plat/validation/api/crypto/odp_crypto_test_inp.c +++ b/test/common_plat/validation/api/crypto/odp_crypto_test_inp.c @@ -53,8 +53,8 @@ static void alg_test(odp_crypto_op_t op, odp_event_t event; odp_crypto_compl_t compl_event; odp_crypto_op_result_t result; - odp_crypto_session_params_t ses_params; - odp_crypto_op_params_t op_params; + odp_crypto_session_param_t ses_params; + odp_crypto_op_param_t op_params; uint8_t *data_addr; int data_off;

@@ -104,7 +104,7 @@ static void alg_test(odp_crypto_op_t op, CU_ASSERT(!rc);

/* Create a crypto session */ - memset(&ses_params, 0, sizeof(ses_params)); + odp_crypto_session_param_init(&ses_params); ses_params.op = op; ses_params.auth_cipher_text = false; ses_params.pref_mode = suite_context.pref_mode;

commit 36c59903f1f898567d2a8eafbf5dc0eee93b2a58 Author: Petri Savolainen petri.savolainen@nokia.com Date: Thu Dec 8 16:05:28 2016 +0200

api: crypto: documentation clean up

Moved documentation of struct fields over each field. Removed references to buffers as crypto API works only with packets.

Signed-off-by: Petri Savolainen petri.savolainen@nokia.com Reviewed-by: Bill Fischofer bill.fischofer@linaro.org Reviewed-by: Balasubramanian Manoharan bala.manoharan@linaro.org Reviewed-by: Nikhil Agarwal nikhil.agarwal@linaro.org Signed-off-by: Maxim Uvarov maxim.uvarov@linaro.org

diff --git a/include/odp/api/spec/crypto.h b/include/odp/api/spec/crypto.h index 0fb6d05..9855bf9 100644 --- a/include/odp/api/spec/crypto.h +++ b/include/odp/api/spec/crypto.h @@ -198,117 +198,166 @@ typedef union odp_crypto_auth_algos_t { * Crypto API key structure */ typedef struct odp_crypto_key { - uint8_t *data; /**< Key data */ - uint32_t length; /**< Key length in bytes */ + /** Key data */ + uint8_t *data; + + /** Key length in bytes */ + uint32_t length; + } odp_crypto_key_t;

/** * Crypto API IV structure */ typedef struct odp_crypto_iv { - uint8_t *data; /**< IV data */ - uint32_t length; /**< IV length in bytes */ + /** IV data */ + uint8_t *data; + + /** IV length in bytes */ + uint32_t length; + } odp_crypto_iv_t;

/** * Crypto API data range specifier */ typedef struct odp_crypto_data_range { - uint32_t offset; /**< Offset from beginning of buffer (chain) */ - uint32_t length; /**< Length of data to operate on */ + /** Offset from beginning of packet */ + uint32_t offset; + + /** Length of data to operate on */ + uint32_t length; + } odp_crypto_data_range_t;

/** * Crypto API session creation parameters */ typedef struct odp_crypto_session_param_t { - odp_crypto_op_t op; /**< Encode versus decode */ - odp_bool_t auth_cipher_text; /**< Authenticate/cipher ordering */ - odp_crypto_op_mode_t pref_mode; /**< Preferred sync vs async */ - odp_cipher_alg_t cipher_alg; /**< Cipher algorithm */ - odp_crypto_key_t cipher_key; /**< Cipher key */ - odp_crypto_iv_t iv; /**< Cipher Initialization Vector (IV) */ - odp_auth_alg_t auth_alg; /**< Authentication algorithm */ - odp_crypto_key_t auth_key; /**< Authentication key */ - odp_queue_t compl_queue; /**< Async mode completion event queue */ - odp_pool_t output_pool; /**< Output buffer pool */ + /** Encode vs. decode operation */ + odp_crypto_op_t op; + + /** Authenticate cipher vs. plain text + * + * Controls ordering of authentication and cipher operations, + * and is relative to the operation (encode vs decode). When encoding, + * TRUE indicates the authentication operation should be performed + * after the cipher operation else before. When decoding, TRUE + * indicates the reverse order of operation. + * + * true: Authenticate cipher text + * false: Authenticate plain text + */ + odp_bool_t auth_cipher_text; + + /** Preferred sync vs. async */ + odp_crypto_op_mode_t pref_mode; + + /** Cipher algorithm + * + * Use odp_crypto_capability() for supported algorithms. + */ + odp_cipher_alg_t cipher_alg; + + /** Cipher key + * + * Use odp_crypto_cipher_capa() for supported key and IV lengths. + */ + odp_crypto_key_t cipher_key; + + /** Cipher Initialization Vector (IV) */ + odp_crypto_iv_t iv; + + /** Authentication algorithm + * + * Use odp_crypto_capability() for supported algorithms. + */ + odp_auth_alg_t auth_alg; + + /** Authentication key + * + * Use odp_crypto_auth_capa() for supported digest and key lengths. + */ + odp_crypto_key_t auth_key; + + /** Async mode completion event queue + * + * When odp_crypto_operation() is asynchronous, the completion queue is + * used to return the completion status of the operation to the + * application. + */ + odp_queue_t compl_queue; + + /** Output pool + * + * When the output packet is not specified during the call to + * odp_crypto_operation(), the output packet will be allocated + * from this pool. + */ + odp_pool_t output_pool; + } odp_crypto_session_param_t;

/** @deprecated Use odp_crypto_session_param_t instead */ typedef odp_crypto_session_param_t odp_crypto_session_params_t;

/** - * @var odp_crypto_session_params_t::auth_cipher_text - * - * Controls ordering of authentication and cipher operations, - * and is relative to the operation (encode vs decode). - * When encoding, @c TRUE indicates the authentication operation - * should be performed @b after the cipher operation else before. - * When decoding, @c TRUE indicates the reverse order of operation. - * - * @var odp_crypto_session_params_t::compl_queue - * - * When the API operates asynchronously, the completion queue is - * used to return the completion status of the operation to the - * application. - * - * @var odp_crypto_session_params_t::output_pool - * - * When the output packet is not specified during the call to - * odp_crypto_operation, the output packet buffer will be allocated - * from this pool. - */ - -/** * Crypto API per packet operation parameters */ typedef struct odp_crypto_op_param_t { - odp_crypto_session_t session; /**< Session handle from creation */ - void *ctx; /**< User context */ - odp_packet_t pkt; /**< Input packet buffer */ - odp_packet_t out_pkt; /**< Output packet buffer */ - uint8_t *override_iv_ptr; /**< Override session IV pointer */ - uint32_t hash_result_offset; /**< Offset from start of packet buffer for hash result */ - odp_crypto_data_range_t cipher_range; /**< Data range to apply cipher */ - odp_crypto_data_range_t auth_range; /**< Data range to authenticate */ + /** Session handle from creation */ + odp_crypto_session_t session; + + /** User context */ + void *ctx; + + /** Input packet + * + * Specifies the input packet for the crypto operation. When the + * 'out_pkt' variable is set to ODP_PACKET_INVALID (indicating a new + * packet should be allocated for the resulting packet). + */ + odp_packet_t pkt; + + /** Output packet + * + * Both "in place" (the original packet 'pkt' is modified) and + * "copy" (the packet is replicated to a new packet which contains + * the modified data) modes are supported. The "in place" mode of + * operation is indicated by setting 'out_pkt' equal to 'pkt'. + * For the copy mode of operation, setting 'out_pkt' to a valid packet + * value indicates the caller wishes to specify the destination packet. + * Setting 'out_pkt' to ODP_PACKET_INVALID indicates the caller wishes + * the destination packet be allocated from the output pool specified + * during session creation. + */ + odp_packet_t out_pkt; + + /** Override session IV pointer */ + uint8_t *override_iv_ptr; + + /** Offset from start of packet for hash result + * + * Specifies the offset where the hash result is to be stored. In case + * of decode sessions, input hash values will be read from this offset, + * and overwritten with hash results. If this offset lies within + * specified 'auth_range', implementation will mute this field before + * calculating the hash result. + */ + uint32_t hash_result_offset; + + /** Data range to apply cipher */ + odp_crypto_data_range_t cipher_range; + + /** Data range to authenticate */ + odp_crypto_data_range_t auth_range; + } odp_crypto_op_param_t;

/** @deprecated Use odp_crypto_op_param_t instead */ typedef odp_crypto_op_param_t odp_crypto_op_params_t;

/** - * @var odp_crypto_op_params_t::pkt - * Specifies the input packet buffer for the crypto operation. When the - * @c out_pkt variable is set to @c ODP_PACKET_INVALID (indicating a new - * buffer should be allocated for the resulting packet), the #define TBD - * indicates whether the implementation will free the input packet buffer - * or if it becomes the responsibility of the caller. - * - * @var odp_crypto_op_params_t::out_pkt - * - * The API supports both "in place" (the original packet "pkt" is - * modified) and "copy" (the packet is replicated to a new buffer - * which contains the modified data). - * - * The "in place" mode of operation is indicated by setting @c out_pkt - * equal to @c pkt. For the copy mode of operation, setting @c out_pkt - * to a valid packet buffer value indicates the caller wishes to specify - * the destination buffer. Setting @c out_pkt to @c ODP_PACKET_INVALID - * indicates the caller wishes the destination packet buffer be allocated - * from the output pool specified during session creation. - * - * @var odp_crypto_op_params_t::hash_result_offset - * - * Specifies the offset where the hash result is to be stored. In case of - * decode sessions, input hash values will be read from this offset, and - * overwritten with hash results. If this offset lies within specified - * auth_range, implementation will mute this field before calculating the hash - * result. - * - * @sa odp_crypto_session_params_t::output_pool. - */ - -/** * Crypto API session creation return code */ typedef enum { @@ -346,7 +395,7 @@ typedef enum { ODP_CRYPTO_HW_ERR_NONE, /** Error detected during DMA of data */ ODP_CRYPTO_HW_ERR_DMA, - /** Operation failed due to buffer pool depletion */ + /** Operation failed due to pool depletion */ ODP_CRYPTO_HW_ERR_BP_DEPLETED, } odp_crypto_hw_err_t;

@@ -354,19 +403,33 @@ typedef enum { * Cryto API per packet operation completion status */ typedef struct odp_crypto_compl_status { - odp_crypto_alg_err_t alg_err; /**< Algorithm specific return code */ - odp_crypto_hw_err_t hw_err; /**< Hardware specific return code */ + /** Algorithm specific return code */ + odp_crypto_alg_err_t alg_err; + + /** Hardware specific return code */ + odp_crypto_hw_err_t hw_err; + } odp_crypto_compl_status_t;

/** * Crypto API operation result */ typedef struct odp_crypto_op_result { - odp_bool_t ok; /**< Request completed successfully */ - void *ctx; /**< User context from request */ - odp_packet_t pkt; /**< Output packet */ - odp_crypto_compl_status_t cipher_status; /**< Cipher status */ - odp_crypto_compl_status_t auth_status; /**< Authentication status */ + /** Request completed successfully */ + odp_bool_t ok; + + /** User context from request */ + void *ctx; + + /** Output packet */ + odp_packet_t pkt; + + /** Cipher status */ + odp_crypto_compl_status_t cipher_status; + + /** Authentication status */ + odp_crypto_compl_status_t auth_status; + } odp_crypto_op_result_t;

/**

commit 516c8bdf27bebf42ddc726b6ce6dd65201ee5bef Author: Petri Savolainen petri.savolainen@nokia.com Date: Thu Dec 8 16:05:27 2016 +0200

api: crypto: added session param init

Added session parameter init function which should be used to initialize the structure before calling odp_crypto_session_create().

Signed-off-by: Petri Savolainen petri.savolainen@nokia.com Reviewed-by: Bill Fischofer bill.fischofer@linaro.org Reviewed-by: Balasubramanian Manoharan bala.manoharan@linaro.org Reviewed-by: Nikhil Agarwal nikhil.agarwal@linaro.org Signed-off-by: Maxim Uvarov maxim.uvarov@linaro.org

diff --git a/include/odp/api/spec/crypto.h b/include/odp/api/spec/crypto.h index 4b94824..0fb6d05 100644 --- a/include/odp/api/spec/crypto.h +++ b/include/odp/api/spec/crypto.h @@ -479,7 +479,11 @@ int odp_crypto_auth_capability(odp_auth_alg_t auth, odp_crypto_auth_capability_t capa[], int num);

/** - * Crypto session creation (synchronous) + * Crypto session creation + * + * Create a crypto session according to the session parameters. Use + * odp_crypto_session_param_init() to initialize parameters into their + * default values. * * @param param Session parameters * @param session Created session else ODP_CRYPTO_SESSION_INVALID @@ -589,6 +593,16 @@ uint64_t odp_crypto_session_to_u64(odp_crypto_session_t hdl); uint64_t odp_crypto_compl_to_u64(odp_crypto_compl_t hdl);

/** + * Initialize crypto session parameters + * + * Initialize an odp_crypto_session_param_t to its default values for + * all fields. + * + * @param param Pointer to odp_crypto_session_param_t to be initialized + */ +void odp_crypto_session_param_init(odp_crypto_session_param_t *param); + +/** * @} */

diff --git a/platform/linux-generic/odp_crypto.c b/platform/linux-generic/odp_crypto.c index fd121c8..6b7d60e 100644 --- a/platform/linux-generic/odp_crypto.c +++ b/platform/linux-generic/odp_crypto.c @@ -1042,3 +1042,8 @@ odp_crypto_compl_free(odp_crypto_compl_t completion_event) odp_buffer_from_event((odp_event_t)completion_event), ODP_EVENT_PACKET); } + +void odp_crypto_session_param_init(odp_crypto_session_param_t *param) +{ + memset(param, 0, sizeof(odp_crypto_session_param_t)); +}

commit 642c8293c63752aa5f07e2e828cdfcf4ca3c2e62 Author: Petri Savolainen petri.savolainen@nokia.com Date: Thu Dec 8 16:05:26 2016 +0200

linux-gen: crypto: add support to new enumerations

Added support for new algorithm enumerations and algorithm capability functions.

Signed-off-by: Petri Savolainen petri.savolainen@nokia.com Reviewed-by: Bill Fischofer bill.fischofer@linaro.org Reviewed-by: Balasubramanian Manoharan bala.manoharan@linaro.org Reviewed-by: Nikhil Agarwal nikhil.agarwal@linaro.org Signed-off-by: Maxim Uvarov maxim.uvarov@linaro.org

diff --git a/platform/linux-generic/include/odp_crypto_internal.h b/platform/linux-generic/include/odp_crypto_internal.h index 7b4eb61..c7b893a 100644 --- a/platform/linux-generic/include/odp_crypto_internal.h +++ b/platform/linux-generic/include/odp_crypto_internal.h @@ -14,6 +14,7 @@ extern "C" { #include <openssl/des.h> #include <openssl/aes.h>

+#define MAX_IV_LEN 64 #define OP_RESULT_MAGIC 0x91919191

/** Forward declaration of session structure */ @@ -31,16 +32,16 @@ odp_crypto_alg_err_t (*crypto_func_t)(odp_crypto_op_param_t *param, */ struct odp_crypto_generic_session { struct odp_crypto_generic_session *next; - odp_crypto_op_t op; + + /* Session creation parameters */ + odp_crypto_session_param_t p; + odp_bool_t do_cipher_first; - odp_queue_t compl_queue; - odp_pool_t output_pool; + struct { - odp_cipher_alg_t alg; - struct { - uint8_t *data; - size_t len; - } iv; + /* Copy of session IV data */ + uint8_t iv_data[MAX_IV_LEN]; + union { struct { DES_key_schedule ks1; @@ -56,8 +57,8 @@ struct odp_crypto_generic_session { } data; crypto_func_t func; } cipher; + struct { - odp_auth_alg_t alg; union { struct { uint8_t key[16]; diff --git a/platform/linux-generic/odp_crypto.c b/platform/linux-generic/odp_crypto.c index 44b8e06..fd121c8 100644 --- a/platform/linux-generic/odp_crypto.c +++ b/platform/linux-generic/odp_crypto.c @@ -249,8 +249,8 @@ odp_crypto_alg_err_t aes_encrypt(odp_crypto_op_param_t *param,

if (param->override_iv_ptr) iv_ptr = param->override_iv_ptr; - else if (session->cipher.iv.data) - iv_ptr = session->cipher.iv.data; + else if (session->p.iv.data) + iv_ptr = session->cipher.iv_data; else return ODP_CRYPTO_ALG_ERR_IV_INVALID;

@@ -281,8 +281,8 @@ odp_crypto_alg_err_t aes_decrypt(odp_crypto_op_param_t *param,

if (param->override_iv_ptr) iv_ptr = param->override_iv_ptr; - else if (session->cipher.iv.data) - iv_ptr = session->cipher.iv.data; + else if (session->p.iv.data) + iv_ptr = session->cipher.iv_data; else return ODP_CRYPTO_ALG_ERR_IV_INVALID;

@@ -302,22 +302,20 @@ odp_crypto_alg_err_t aes_decrypt(odp_crypto_op_param_t *param, return ODP_CRYPTO_ALG_ERR_NONE; }

-static -int process_aes_param(odp_crypto_generic_session_t *session, - odp_crypto_session_param_t *param) +static int process_aes_param(odp_crypto_generic_session_t *session) { /* Verify IV len is either 0 or 16 */ - if (!((0 == param->iv.length) || (16 == param->iv.length))) + if (!((0 == session->p.iv.length) || (16 == session->p.iv.length))) return -1;

/* Set function */ - if (ODP_CRYPTO_OP_ENCODE == param->op) { + if (ODP_CRYPTO_OP_ENCODE == session->p.op) { session->cipher.func = aes_encrypt; - AES_set_encrypt_key(param->cipher_key.data, 128, + AES_set_encrypt_key(session->p.cipher_key.data, 128, &session->cipher.data.aes.key); } else { session->cipher.func = aes_decrypt; - AES_set_decrypt_key(param->cipher_key.data, 128, + AES_set_decrypt_key(session->p.cipher_key.data, 128, &session->cipher.data.aes.key); }

@@ -340,8 +338,8 @@ odp_crypto_alg_err_t aes_gcm_encrypt(odp_crypto_op_param_t *param,

if (param->override_iv_ptr) iv_ptr = param->override_iv_ptr; - else if (session->cipher.iv.data) - iv_ptr = session->cipher.iv.data; + else if (session->p.iv.data) + iv_ptr = session->cipher.iv_data; else return ODP_CRYPTO_ALG_ERR_IV_INVALID;

@@ -405,8 +403,8 @@ odp_crypto_alg_err_t aes_gcm_decrypt(odp_crypto_op_param_t *param,

if (param->override_iv_ptr) iv_ptr = param->override_iv_ptr; - else if (session->cipher.iv.data) - iv_ptr = session->cipher.iv.data; + else if (session->p.iv.data) + iv_ptr = session->cipher.iv_data; else return ODP_CRYPTO_ALG_ERR_IV_INVALID;

@@ -455,19 +453,17 @@ odp_crypto_alg_err_t aes_gcm_decrypt(odp_crypto_op_param_t *param, return ODP_CRYPTO_ALG_ERR_NONE; }

-static -int process_aes_gcm_param(odp_crypto_generic_session_t *session, - odp_crypto_session_param_t *param) +static int process_aes_gcm_param(odp_crypto_generic_session_t *session) { /* Verify Key len is 16 */ - if (param->cipher_key.length != 16) + if (session->p.cipher_key.length != 16) return -1;

/* Set function */ EVP_CIPHER_CTX *ctx = session->cipher.data.aes_gcm.ctx = EVP_CIPHER_CTX_new();

- if (ODP_CRYPTO_OP_ENCODE == param->op) { + if (ODP_CRYPTO_OP_ENCODE == session->p.op) { session->cipher.func = aes_gcm_encrypt; EVP_EncryptInit_ex(ctx, EVP_aes_128_gcm(), NULL, NULL, NULL); } else { @@ -476,13 +472,13 @@ int process_aes_gcm_param(odp_crypto_generic_session_t *session, }

EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_IVLEN, - param->iv.length, NULL); - if (ODP_CRYPTO_OP_ENCODE == param->op) { + session->p.iv.length, NULL); + if (ODP_CRYPTO_OP_ENCODE == session->p.op) { EVP_EncryptInit_ex(ctx, NULL, NULL, - param->cipher_key.data, NULL); + session->p.cipher_key.data, NULL); } else { EVP_DecryptInit_ex(ctx, NULL, NULL, - param->cipher_key.data, NULL); + session->p.cipher_key.data, NULL); }

return 0; @@ -499,8 +495,8 @@ odp_crypto_alg_err_t des_encrypt(odp_crypto_op_param_t *param,

if (param->override_iv_ptr) iv_ptr = param->override_iv_ptr; - else if (session->cipher.iv.data) - iv_ptr = session->cipher.iv.data; + else if (session->p.iv.data) + iv_ptr = session->cipher.iv_data; else return ODP_CRYPTO_ALG_ERR_IV_INVALID;

@@ -537,8 +533,8 @@ odp_crypto_alg_err_t des_decrypt(odp_crypto_op_param_t *param,

if (param->override_iv_ptr) iv_ptr = param->override_iv_ptr; - else if (session->cipher.iv.data) - iv_ptr = session->cipher.iv.data; + else if (session->p.iv.data) + iv_ptr = session->cipher.iv_data; else return ODP_CRYPTO_ALG_ERR_IV_INVALID;

@@ -565,38 +561,34 @@ odp_crypto_alg_err_t des_decrypt(odp_crypto_op_param_t *param, return ODP_CRYPTO_ALG_ERR_NONE; }

-static -int process_des_param(odp_crypto_generic_session_t *session, - odp_crypto_session_param_t *param) +static int process_des_param(odp_crypto_generic_session_t *session) { /* Verify IV len is either 0 or 8 */ - if (!((0 == param->iv.length) || (8 == param->iv.length))) + if (!((0 == session->p.iv.length) || (8 == session->p.iv.length))) return -1;

/* Set function */ - if (ODP_CRYPTO_OP_ENCODE == param->op) + if (ODP_CRYPTO_OP_ENCODE == session->p.op) session->cipher.func = des_encrypt; else session->cipher.func = des_decrypt;

/* Convert keys */ - DES_set_key((DES_cblock *)&param->cipher_key.data[0], + DES_set_key((DES_cblock *)&session->p.cipher_key.data[0], &session->cipher.data.des.ks1); - DES_set_key((DES_cblock *)&param->cipher_key.data[8], + DES_set_key((DES_cblock *)&session->p.cipher_key.data[8], &session->cipher.data.des.ks2); - DES_set_key((DES_cblock *)&param->cipher_key.data[16], + DES_set_key((DES_cblock *)&session->p.cipher_key.data[16], &session->cipher.data.des.ks3);

return 0; }

-static -int process_md5_param(odp_crypto_generic_session_t *session, - odp_crypto_session_param_t *param, - uint32_t bits) +static int process_md5_param(odp_crypto_generic_session_t *session, + uint32_t bits) { /* Set function */ - if (ODP_CRYPTO_OP_ENCODE == param->op) + if (ODP_CRYPTO_OP_ENCODE == session->p.op) session->auth.func = md5_gen; else session->auth.func = md5_check; @@ -605,18 +597,16 @@ int process_md5_param(odp_crypto_generic_session_t *session, session->auth.data.md5.bytes = bits / 8;

/* Convert keys */ - memcpy(session->auth.data.md5.key, param->auth_key.data, 16); + memcpy(session->auth.data.md5.key, session->p.auth_key.data, 16);

return 0; }

-static -int process_sha256_param(odp_crypto_generic_session_t *session, - odp_crypto_session_param_t *param, - uint32_t bits) +static int process_sha256_param(odp_crypto_generic_session_t *session, + uint32_t bits) { /* Set function */ - if (ODP_CRYPTO_OP_ENCODE == param->op) + if (ODP_CRYPTO_OP_ENCODE == session->p.op) session->auth.func = sha256_gen; else session->auth.func = sha256_check; @@ -625,7 +615,7 @@ int process_sha256_param(odp_crypto_generic_session_t *session, session->auth.data.sha256.bytes = bits / 8;

/* Convert keys */ - memcpy(session->auth.data.sha256.key, param->auth_key.data, 32); + memcpy(session->auth.data.sha256.key, session->p.auth_key.data, 32);

return 0; } @@ -638,16 +628,23 @@ int odp_crypto_capability(odp_crypto_capability_t *capa) /* Initialize crypto capability structure */ memset(capa, 0, sizeof(odp_crypto_capability_t));

- capa->ciphers.bit.null = 1; - capa->ciphers.bit.des = 1; - capa->ciphers.bit.trides_cbc = 1; - capa->ciphers.bit.aes128_cbc = 1; - capa->ciphers.bit.aes128_gcm = 1; + capa->ciphers.bit.null = 1; + capa->ciphers.bit.des = 1; + capa->ciphers.bit.trides_cbc = 1; + capa->ciphers.bit.aes_cbc = 1; + capa->ciphers.bit.aes_gcm = 1; + + capa->auths.bit.null = 1; + capa->auths.bit.md5_hmac = 1; + capa->auths.bit.sha256_hmac = 1; + capa->auths.bit.aes_gcm = 1;

- capa->auths.bit.null = 1; - capa->auths.bit.md5_96 = 1; - capa->auths.bit.sha256_128 = 1; - capa->auths.bit.aes128_gcm = 1; + /* Deprecated */ + capa->ciphers.bit.aes128_cbc = 1; + capa->ciphers.bit.aes128_gcm = 1; + capa->auths.bit.md5_96 = 1; + capa->auths.bit.sha256_128 = 1; + capa->auths.bit.aes128_gcm = 1;

capa->max_sessions = MAX_SESSIONS;

@@ -749,21 +746,26 @@ odp_crypto_session_create(odp_crypto_session_param_t *param, return -1; }

+ /* Copy parameters */ + session->p = *param; + + /* Copy IV data */ + if (session->p.iv.data) { + if (session->p.iv.length > MAX_IV_LEN) { + ODP_DBG("Maximum IV length exceeded\n"); + return -1; + } + + memcpy(session->cipher.iv_data, session->p.iv.data, + session->p.iv.length); + } + /* Derive order */ if (ODP_CRYPTO_OP_ENCODE == param->op) session->do_cipher_first = param->auth_cipher_text; else session->do_cipher_first = !param->auth_cipher_text;

- /* Copy stuff over */ - session->op = param->op; - session->compl_queue = param->compl_queue; - session->cipher.alg = param->cipher_alg; - session->cipher.iv.data = param->iv.data; - session->cipher.iv.len = param->iv.length; - session->auth.alg = param->auth_alg; - session->output_pool = param->output_pool; - /* Process based on cipher */ switch (param->cipher_alg) { case ODP_CIPHER_ALG_NULL: @@ -772,19 +774,23 @@ odp_crypto_session_create(odp_crypto_session_param_t *param, break; case ODP_CIPHER_ALG_DES: case ODP_CIPHER_ALG_3DES_CBC: - rc = process_des_param(session, param); + rc = process_des_param(session); break; + case ODP_CIPHER_ALG_AES_CBC: + /* deprecated */ case ODP_CIPHER_ALG_AES128_CBC: - rc = process_aes_param(session, param); + rc = process_aes_param(session); break; + case ODP_CIPHER_ALG_AES_GCM: + /* deprecated */ case ODP_CIPHER_ALG_AES128_GCM: /* AES-GCM requires to do both auth and * cipher at the same time */ - if (param->auth_alg != ODP_AUTH_ALG_AES128_GCM) { + if (param->auth_alg == ODP_AUTH_ALG_AES_GCM || + param->auth_alg == ODP_AUTH_ALG_AES128_GCM) + rc = process_aes_gcm_param(session); + else rc = -1; - break; - } - rc = process_aes_gcm_param(session, param); break; default: rc = -1; @@ -802,21 +808,28 @@ odp_crypto_session_create(odp_crypto_session_param_t *param, session->auth.func = null_crypto_routine; rc = 0; break; + case ODP_AUTH_ALG_MD5_HMAC: + /* deprecated */ case ODP_AUTH_ALG_MD5_96: - rc = process_md5_param(session, param, 96); + rc = process_md5_param(session, 96); break; + case ODP_AUTH_ALG_SHA256_HMAC: + /* deprecated */ case ODP_AUTH_ALG_SHA256_128: - rc = process_sha256_param(session, param, 128); + rc = process_sha256_param(session, 128); break; + case ODP_AUTH_ALG_AES_GCM: + /* deprecated */ case ODP_AUTH_ALG_AES128_GCM: /* AES-GCM requires to do both auth and * cipher at the same time */ - if (param->cipher_alg != ODP_CIPHER_ALG_AES128_GCM) { + if (param->cipher_alg == ODP_CIPHER_ALG_AES_GCM || + param->cipher_alg == ODP_CIPHER_ALG_AES128_GCM) { + session->auth.func = null_crypto_routine; + rc = 0; + } else { rc = -1; - break; } - session->auth.func = null_crypto_routine; - rc = 0; break; default: rc = -1; @@ -838,7 +851,8 @@ int odp_crypto_session_destroy(odp_crypto_session_t session) odp_crypto_generic_session_t *generic;

generic = (odp_crypto_generic_session_t *)(intptr_t)session; - if (generic->cipher.alg == ODP_CIPHER_ALG_AES128_GCM) + if (generic->p.cipher_alg == ODP_CIPHER_ALG_AES128_GCM || + generic->p.cipher_alg == ODP_CIPHER_ALG_AES_GCM) EVP_CIPHER_CTX_free(generic->cipher.data.aes_gcm.ctx); memset(generic, 0, sizeof(*generic)); free_session(generic); @@ -859,8 +873,8 @@ odp_crypto_operation(odp_crypto_op_param_t *param,

/* Resolve output buffer */ if (ODP_PACKET_INVALID == param->out_pkt && - ODP_POOL_INVALID != session->output_pool) - param->out_pkt = odp_packet_alloc(session->output_pool, + ODP_POOL_INVALID != session->p.output_pool) + param->out_pkt = odp_packet_alloc(session->p.output_pool, odp_packet_len(param->pkt));

if (odp_unlikely(ODP_PACKET_INVALID == param->out_pkt)) { @@ -900,7 +914,7 @@ odp_crypto_operation(odp_crypto_op_param_t *param, (rc_auth == ODP_CRYPTO_ALG_ERR_NONE);

/* If specified during creation post event to completion queue */ - if (ODP_QUEUE_INVALID != session->compl_queue) { + if (ODP_QUEUE_INVALID != session->p.compl_queue) { odp_event_t completion_event; odp_crypto_generic_op_result_t *op_result;

@@ -913,7 +927,7 @@ odp_crypto_operation(odp_crypto_op_param_t *param, op_result = get_op_result_from_event(completion_event); op_result->magic = OP_RESULT_MAGIC; op_result->result = local_result; - if (odp_queue_enq(session->compl_queue, completion_event)) { + if (odp_queue_enq(session->p.compl_queue, completion_event)) { odp_event_free(completion_event); return -1; }

commit 804c1667832794c43a1a4867b778b31e7a104e9a Author: Petri Savolainen petri.savolainen@nokia.com Date: Thu Dec 8 16:25:58 2016 +0200

linux-gen: crypto: add algo capability functions

Implemented cipher and authentication algorithm capability functions.

Signed-off-by: Petri Savolainen petri.savolainen@nokia.com Reviewed-by: Bill Fischofer bill.fischofer@linaro.org Reviewed-by: Balasubramanian Manoharan bala.manoharan@linaro.org Reviewed-by: Nikhil Agarwal nikhil.agarwal@linaro.org Signed-off-by: Maxim Uvarov maxim.uvarov@linaro.org

diff --git a/platform/linux-generic/odp_crypto.c b/platform/linux-generic/odp_crypto.c index 70d3a97..44b8e06 100644 --- a/platform/linux-generic/odp_crypto.c +++ b/platform/linux-generic/odp_crypto.c @@ -27,6 +27,37 @@

#define MAX_SESSIONS 32

+/* + * Cipher algorithm capabilities + * + * Keep sorted: first by key length, then by IV length + */ +static const odp_crypto_cipher_capability_t cipher_capa_des[] = { +{.key_len = 24, .iv_len = 8} }; + +static const odp_crypto_cipher_capability_t cipher_capa_trides_cbc[] = { +{.key_len = 24, .iv_len = 8} }; + +static const odp_crypto_cipher_capability_t cipher_capa_aes_cbc[] = { +{.key_len = 16, .iv_len = 16} }; + +static const odp_crypto_cipher_capability_t cipher_capa_aes_gcm[] = { +{.key_len = 16, .iv_len = 12} }; + +/* + * Authentication algorithm capabilities + * + * Keep sorted: first by digest length, then by key length + */ +static const odp_crypto_auth_capability_t auth_capa_md5_hmac[] = { +{.digest_len = 12, .key_len = 16, .aad_len = {.min = 0, .max = 0, .inc = 0} } }; + +static const odp_crypto_auth_capability_t auth_capa_sha256_hmac[] = { +{.digest_len = 16, .key_len = 32, .aad_len = {.min = 0, .max = 0, .inc = 0} } }; + +static const odp_crypto_auth_capability_t auth_capa_aes_gcm[] = { +{.digest_len = 16, .key_len = 0, .aad_len = {.min = 8, .max = 12, .inc = 4} } }; + typedef struct odp_crypto_global_s odp_crypto_global_t;

struct odp_crypto_global_s { @@ -623,6 +654,83 @@ int odp_crypto_capability(odp_crypto_capability_t *capa) return 0; }

+int odp_crypto_cipher_capability(odp_cipher_alg_t cipher, + odp_crypto_cipher_capability_t dst[], + int num_copy) +{ + const odp_crypto_cipher_capability_t *src; + int num; + int size = sizeof(odp_crypto_cipher_capability_t); + + switch (cipher) { + case ODP_CIPHER_ALG_NULL: + src = NULL; + num = 0; + break; + case ODP_CIPHER_ALG_DES: + src = cipher_capa_des; + num = sizeof(cipher_capa_des) / size; + break; + case ODP_CIPHER_ALG_3DES_CBC: + src = cipher_capa_trides_cbc; + num = sizeof(cipher_capa_trides_cbc) / size; + break; + case ODP_CIPHER_ALG_AES_CBC: + src = cipher_capa_aes_cbc; + num = sizeof(cipher_capa_aes_cbc) / size; + break; + case ODP_CIPHER_ALG_AES_GCM: + src = cipher_capa_aes_gcm; + num = sizeof(cipher_capa_aes_gcm) / size; + break; + default: + return -1; + } + + if (num < num_copy) + num_copy = num; + + memcpy(dst, src, num_copy * size); + + return num; +} + +int odp_crypto_auth_capability(odp_auth_alg_t auth, + odp_crypto_auth_capability_t dst[], int num_copy) +{ + const odp_crypto_auth_capability_t *src; + int num; + int size = sizeof(odp_crypto_auth_capability_t); + + switch (auth) { + case ODP_AUTH_ALG_NULL: + src = NULL; + num = 0; + break; + case ODP_AUTH_ALG_MD5_HMAC: + src = auth_capa_md5_hmac; + num = sizeof(auth_capa_md5_hmac) / size; + break; + case ODP_AUTH_ALG_SHA256_HMAC: + src = auth_capa_sha256_hmac; + num = sizeof(auth_capa_sha256_hmac) / size; + break; + case ODP_AUTH_ALG_AES_GCM: + src = auth_capa_aes_gcm; + num = sizeof(auth_capa_aes_gcm) / size; + break; + default: + return -1; + } + + if (num < num_copy) + num_copy = num; + + memcpy(dst, src, num_copy * size); + + return num; +} + int odp_crypto_session_create(odp_crypto_session_param_t *param, odp_crypto_session_t *session_out,

commit 9e4bca41f1df50b1cf1f324118bb669f9bee0369 Author: Petri Savolainen petri.savolainen@nokia.com Date: Thu Dec 8 16:05:24 2016 +0200

api: crypto: decouple key length from algorithm enumeration

Enumerations for cipher and authentication algorithms grow fast if key and digest lengths are included into the enum. Decoupled lengths from algorithm names, only exception is SHA-2 family of authentication algorithms which has established naming convention with digest lengths (SHA-224, SHA-256, ...). Old enumerations are still functional but deprecated.

Algotrithm level capability functions provide a flexible way to handle all possible key/digest/iv length combinations.

Signed-off-by: Petri Savolainen petri.savolainen@nokia.com Reviewed-by: Bill Fischofer bill.fischofer@linaro.org Reviewed-by: Balasubramanian Manoharan bala.manoharan@linaro.org Reviewed-by: Nikhil Agarwal nikhil.agarwal@linaro.org Signed-off-by: Maxim Uvarov maxim.uvarov@linaro.org

diff --git a/include/odp/api/spec/crypto.h b/include/odp/api/spec/crypto.h index f24f527..4b94824 100644 --- a/include/odp/api/spec/crypto.h +++ b/include/odp/api/spec/crypto.h @@ -65,14 +65,28 @@ typedef enum { typedef enum { /** No cipher algorithm specified */ ODP_CIPHER_ALG_NULL, + /** DES */ ODP_CIPHER_ALG_DES, + /** Triple DES with cipher block chaining */ ODP_CIPHER_ALG_3DES_CBC, - /** AES128 with cipher block chaining */ + + /** AES with cipher block chaining */ + ODP_CIPHER_ALG_AES_CBC, + + /** AES in Galois/Counter Mode + * + * @note Must be paired with cipher ODP_AUTH_ALG_AES_GCM + */ + ODP_CIPHER_ALG_AES_GCM, + + /** @deprecated Use ODP_CIPHER_ALG_AES_CBC instead */ ODP_CIPHER_ALG_AES128_CBC, - /** AES128 in Galois/Counter Mode */ - ODP_CIPHER_ALG_AES128_GCM, + + /** @deprecated Use ODP_CIPHER_ALG_AES_GCM instead */ + ODP_CIPHER_ALG_AES128_GCM + } odp_cipher_alg_t;

/** @@ -81,12 +95,33 @@ typedef enum { typedef enum { /** No authentication algorithm specified */ ODP_AUTH_ALG_NULL, - /** HMAC-MD5 with 96 bit key */ + + /** HMAC-MD5 + * + * MD5 algorithm in HMAC mode + */ + ODP_AUTH_ALG_MD5_HMAC, + + /** HMAC-SHA-256 + * + * SHA-256 algorithm in HMAC mode + */ + ODP_AUTH_ALG_SHA256_HMAC, + + /** AES in Galois/Counter Mode + * + * @note Must be paired with cipher ODP_CIPHER_ALG_AES_GCM + */ + ODP_AUTH_ALG_AES_GCM, + + /** @deprecated Use ODP_AUTH_ALG_MD5_HMAC instead */ ODP_AUTH_ALG_MD5_96, - /** SHA256 with 128 bit key */ + + /** @deprecated Use ODP_AUTH_ALG_SHA256_HMAC instead */ ODP_AUTH_ALG_SHA256_128, - /** AES128 in Galois/Counter Mode */ - ODP_AUTH_ALG_AES128_GCM, + + /** @deprecated Use ODP_AUTH_ALG_AES_GCM instead */ + ODP_AUTH_ALG_AES128_GCM } odp_auth_alg_t;

/** @@ -96,19 +131,25 @@ typedef union odp_crypto_cipher_algos_t { /** Cipher algorithms */ struct { /** ODP_CIPHER_ALG_NULL */ - uint32_t null : 1; + uint32_t null : 1;

/** ODP_CIPHER_ALG_DES */ - uint32_t des : 1; + uint32_t des : 1;

/** ODP_CIPHER_ALG_3DES_CBC */ - uint32_t trides_cbc : 1; + uint32_t trides_cbc : 1; + + /** ODP_CIPHER_ALG_AES_CBC */ + uint32_t aes_cbc : 1;

- /** ODP_CIPHER_ALG_AES128_CBC */ - uint32_t aes128_cbc : 1; + /** ODP_CIPHER_ALG_AES_GCM */ + uint32_t aes_gcm : 1;

- /** ODP_CIPHER_ALG_AES128_GCM */ - uint32_t aes128_gcm : 1; + /** @deprecated Use aes_cbc instead */ + uint32_t aes128_cbc : 1; + + /** @deprecated Use aes_gcm instead */ + uint32_t aes128_gcm : 1; } bit;

/** All bits of the bit field structure @@ -125,16 +166,25 @@ typedef union odp_crypto_auth_algos_t { /** Authentication algorithms */ struct { /** ODP_AUTH_ALG_NULL */ - uint32_t null : 1; + uint32_t null : 1; + + /** ODP_AUTH_ALG_MD5_HMAC */ + uint32_t md5_hmac : 1; + + /** ODP_AUTH_ALG_SHA256_HMAC */ + uint32_t sha256_hmac : 1;

- /** ODP_AUTH_ALG_MD5_96 */ - uint32_t md5_96 : 1; + /** ODP_AUTH_ALG_AES_GCM */ + uint32_t aes_gcm : 1;

- /** ODP_AUTH_ALG_SHA256_128 */ - uint32_t sha256_128 : 1; + /** @deprecated Use md5_hmac instead */ + uint32_t md5_96 : 1;

- /** ODP_AUTH_ALG_AES128_GCM */ - uint32_t aes128_gcm : 1; + /** @deprecated Use sha256_hmac instead */ + uint32_t sha256_128 : 1; + + /** @deprecated Use aes_gcm instead */ + uint32_t aes128_gcm : 1; } bit;

/** All bits of the bit field structure @@ -341,6 +391,43 @@ typedef struct odp_crypto_capability_t { } odp_crypto_capability_t;

/** + * Cipher algorithm capabilities + */ +typedef struct odp_crypto_cipher_capability_t { + /** Key length in bytes */ + uint32_t key_len; + + /** IV length in bytes */ + uint32_t iv_len; + +} odp_crypto_cipher_capability_t; + +/** + * Authentication algorithm capabilities + */ +typedef struct odp_crypto_auth_capability_t { + /** Digest length in bytes */ + uint32_t digest_len; + + /** Key length in bytes */ + uint32_t key_len; + + /** Additional Authenticated Data (AAD) lengths */ + struct { + /** Minimum AAD length in bytes */ + uint32_t min; + + /** Maximum AAD length in bytes */ + uint32_t max; + + /** Increment of supported lengths between min and max + * (in bytes) */ + uint32_t inc; + } aad_len; + +} odp_crypto_auth_capability_t; + +/** * Query crypto capabilities * * Outputs crypto capabilities on success. @@ -353,6 +440,45 @@ typedef struct odp_crypto_capability_t { int odp_crypto_capability(odp_crypto_capability_t *capa);

/** + * Query supported cipher algorithm capabilities + * + * Outputs all supported configuration options for the algorithm. Output is + * sorted (from the smallest to the largest) first by key length, then by IV + * length. + * + * @param cipher Cipher algorithm + * @param[out] capa Array of capability structures for output + * @param num Maximum number of capability structures to output + * + * @return Number of capability structures for the algorithm. If this is larger + * than 'num', only 'num' first structures were output and application + * may call the function again with a larger value of 'num'. + * @retval <0 on failure + */ +int odp_crypto_cipher_capability(odp_cipher_alg_t cipher, + odp_crypto_cipher_capability_t capa[], + int num); + +/** + * Query supported authentication algorithm capabilities + * + * Outputs all supported configuration options for the algorithm. Output is + * sorted (from the smallest to the largest) first by digest length, then by key + * length. + * + * @param auth Authentication algorithm + * @param[out] capa Array of capability structures for output + * @param num Maximum number of capability structures to output + * + * @return Number of capability structures for the algorithm. If this is larger + * than 'num', only 'num' first structures were output and application + * may call the function again with a larger value of 'num'. + * @retval <0 on failure + */ +int odp_crypto_auth_capability(odp_auth_alg_t auth, + odp_crypto_auth_capability_t capa[], int num); + +/** * Crypto session creation (synchronous) * * @param param Session parameters

commit f27d3d3bdd745a000b58a7ce95d77707b957c903 Author: Petri Savolainen petri.savolainen@nokia.com Date: Thu Dec 8 16:05:23 2016 +0200

linux-gen: crypto: rename params to param

Use new _param_t type names instead of _params_t.

Signed-off-by: Petri Savolainen petri.savolainen@nokia.com Reviewed-by: Bill Fischofer bill.fischofer@linaro.org Reviewed-by: Balasubramanian Manoharan bala.manoharan@linaro.org Reviewed-by: Nikhil Agarwal nikhil.agarwal@linaro.org Signed-off-by: Maxim Uvarov maxim.uvarov@linaro.org

diff --git a/platform/linux-generic/include/odp_crypto_internal.h b/platform/linux-generic/include/odp_crypto_internal.h index 7b104af..7b4eb61 100644 --- a/platform/linux-generic/include/odp_crypto_internal.h +++ b/platform/linux-generic/include/odp_crypto_internal.h @@ -23,7 +23,7 @@ typedef struct odp_crypto_generic_session odp_crypto_generic_session_t; * Algorithm handler function prototype */ typedef -odp_crypto_alg_err_t (*crypto_func_t)(odp_crypto_op_params_t *params, +odp_crypto_alg_err_t (*crypto_func_t)(odp_crypto_op_param_t *param, odp_crypto_generic_session_t *session);

/** diff --git a/platform/linux-generic/odp_crypto.c b/platform/linux-generic/odp_crypto.c index 7e686ff..70d3a97 100644 --- a/platform/linux-generic/odp_crypto.c +++ b/platform/linux-generic/odp_crypto.c @@ -69,24 +69,24 @@ void free_session(odp_crypto_generic_session_t *session) }

static odp_crypto_alg_err_t -null_crypto_routine(odp_crypto_op_params_t *params ODP_UNUSED, +null_crypto_routine(odp_crypto_op_param_t *param ODP_UNUSED, odp_crypto_generic_session_t *session ODP_UNUSED) { return ODP_CRYPTO_ALG_ERR_NONE; }

static -odp_crypto_alg_err_t md5_gen(odp_crypto_op_params_t *params, +odp_crypto_alg_err_t md5_gen(odp_crypto_op_param_t *param, odp_crypto_generic_session_t *session) { - uint8_t *data = odp_packet_data(params->out_pkt); + uint8_t *data = odp_packet_data(param->out_pkt); uint8_t *icv = data; - uint32_t len = params->auth_range.length; + uint32_t len = param->auth_range.length; uint8_t hash[EVP_MAX_MD_SIZE];

/* Adjust pointer for beginning of area to auth */ - data += params->auth_range.offset; - icv += params->hash_result_offset; + data += param->auth_range.offset; + icv += param->hash_result_offset;

/* Hash it */ HMAC(EVP_md5(), @@ -104,19 +104,19 @@ odp_crypto_alg_err_t md5_gen(odp_crypto_op_params_t *params, }

static -odp_crypto_alg_err_t md5_check(odp_crypto_op_params_t *params, +odp_crypto_alg_err_t md5_check(odp_crypto_op_param_t *param, odp_crypto_generic_session_t *session) { - uint8_t *data = odp_packet_data(params->out_pkt); + uint8_t *data = odp_packet_data(param->out_pkt); uint8_t *icv = data; - uint32_t len = params->auth_range.length; + uint32_t len = param->auth_range.length; uint32_t bytes = session->auth.data.md5.bytes; uint8_t hash_in[EVP_MAX_MD_SIZE]; uint8_t hash_out[EVP_MAX_MD_SIZE];

/* Adjust pointer for beginning of area to auth */ - data += params->auth_range.offset; - icv += params->hash_result_offset; + data += param->auth_range.offset; + icv += param->hash_result_offset;

/* Copy current value out and clear it before authentication */ memset(hash_in, 0, sizeof(hash_in)); @@ -142,17 +142,17 @@ odp_crypto_alg_err_t md5_check(odp_crypto_op_params_t *params, }

static -odp_crypto_alg_err_t sha256_gen(odp_crypto_op_params_t *params, +odp_crypto_alg_err_t sha256_gen(odp_crypto_op_param_t *param, odp_crypto_generic_session_t *session) { - uint8_t *data = odp_packet_data(params->out_pkt); + uint8_t *data = odp_packet_data(param->out_pkt); uint8_t *icv = data; - uint32_t len = params->auth_range.length; + uint32_t len = param->auth_range.length; uint8_t hash[EVP_MAX_MD_SIZE];

/* Adjust pointer for beginning of area to auth */ - data += params->auth_range.offset; - icv += params->hash_result_offset; + data += param->auth_range.offset; + icv += param->hash_result_offset;

/* Hash it */ HMAC(EVP_sha256(), @@ -170,19 +170,19 @@ odp_crypto_alg_err_t sha256_gen(odp_crypto_op_params_t *params, }

static -odp_crypto_alg_err_t sha256_check(odp_crypto_op_params_t *params, +odp_crypto_alg_err_t sha256_check(odp_crypto_op_param_t *param, odp_crypto_generic_session_t *session) { - uint8_t *data = odp_packet_data(params->out_pkt); + uint8_t *data = odp_packet_data(param->out_pkt); uint8_t *icv = data; - uint32_t len = params->auth_range.length; + uint32_t len = param->auth_range.length; uint32_t bytes = session->auth.data.sha256.bytes; uint8_t hash_in[EVP_MAX_MD_SIZE]; uint8_t hash_out[EVP_MAX_MD_SIZE];

/* Adjust pointer for beginning of area to auth */ - data += params->auth_range.offset; - icv += params->hash_result_offset; + data += param->auth_range.offset; + icv += param->hash_result_offset;

/* Copy current value out and clear it before authentication */ memset(hash_in, 0, sizeof(hash_in)); @@ -208,16 +208,16 @@ odp_crypto_alg_err_t sha256_check(odp_crypto_op_params_t *params, }

static -odp_crypto_alg_err_t aes_encrypt(odp_crypto_op_params_t *params, +odp_crypto_alg_err_t aes_encrypt(odp_crypto_op_param_t *param, odp_crypto_generic_session_t *session) { - uint8_t *data = odp_packet_data(params->out_pkt); - uint32_t len = params->cipher_range.length; + uint8_t *data = odp_packet_data(param->out_pkt); + uint32_t len = param->cipher_range.length; unsigned char iv_enc[AES_BLOCK_SIZE]; void *iv_ptr;

- if (params->override_iv_ptr) - iv_ptr = params->override_iv_ptr; + if (param->override_iv_ptr) + iv_ptr = param->override_iv_ptr; else if (session->cipher.iv.data) iv_ptr = session->cipher.iv.data; else @@ -231,7 +231,7 @@ odp_crypto_alg_err_t aes_encrypt(odp_crypto_op_params_t *params, memcpy(iv_enc, iv_ptr, AES_BLOCK_SIZE);

/* Adjust pointer for beginning of area to cipher */ - data += params->cipher_range.offset; + data += param->cipher_range.offset; /* Encrypt it */ AES_cbc_encrypt(data, data, len, &session->cipher.data.aes.key, iv_enc, AES_ENCRYPT); @@ -240,16 +240,16 @@ odp_crypto_alg_err_t aes_encrypt(odp_crypto_op_params_t *params, }

static -odp_crypto_alg_err_t aes_decrypt(odp_crypto_op_params_t *params, +odp_crypto_alg_err_t aes_decrypt(odp_crypto_op_param_t *param, odp_crypto_generic_session_t *session) { - uint8_t *data = odp_packet_data(params->out_pkt); - uint32_t len = params->cipher_range.length; + uint8_t *data = odp_packet_data(param->out_pkt); + uint32_t len = param->cipher_range.length; unsigned char iv_enc[AES_BLOCK_SIZE]; void *iv_ptr;

- if (params->override_iv_ptr) - iv_ptr = params->override_iv_ptr; + if (param->override_iv_ptr) + iv_ptr = param->override_iv_ptr; else if (session->cipher.iv.data) iv_ptr = session->cipher.iv.data; else @@ -263,7 +263,7 @@ odp_crypto_alg_err_t aes_decrypt(odp_crypto_op_params_t *params, memcpy(iv_enc, iv_ptr, AES_BLOCK_SIZE);

/* Adjust pointer for beginning of area to cipher */ - data += params->cipher_range.offset; + data += param->cipher_range.offset; /* Encrypt it */ AES_cbc_encrypt(data, data, len, &session->cipher.data.aes.key, iv_enc, AES_DECRYPT); @@ -272,21 +272,21 @@ odp_crypto_alg_err_t aes_decrypt(odp_crypto_op_params_t *params, }

static -int process_aes_params(odp_crypto_generic_session_t *session, - odp_crypto_session_params_t *params) +int process_aes_param(odp_crypto_generic_session_t *session, + odp_crypto_session_param_t *param) { /* Verify IV len is either 0 or 16 */ - if (!((0 == params->iv.length) || (16 == params->iv.length))) + if (!((0 == param->iv.length) || (16 == param->iv.length))) return -1;

/* Set function */ - if (ODP_CRYPTO_OP_ENCODE == params->op) { + if (ODP_CRYPTO_OP_ENCODE == param->op) { session->cipher.func = aes_encrypt; - AES_set_encrypt_key(params->cipher_key.data, 128, + AES_set_encrypt_key(param->cipher_key.data, 128, &session->cipher.data.aes.key); } else { session->cipher.func = aes_decrypt; - AES_set_decrypt_key(params->cipher_key.data, 128, + AES_set_decrypt_key(param->cipher_key.data, 128, &session->cipher.data.aes.key); }

@@ -294,30 +294,30 @@ int process_aes_params(odp_crypto_generic_session_t *session, }

static -odp_crypto_alg_err_t aes_gcm_encrypt(odp_crypto_op_params_t *params, +odp_crypto_alg_err_t aes_gcm_encrypt(odp_crypto_op_param_t *param, odp_crypto_generic_session_t *session) { - uint8_t *data = odp_packet_data(params->out_pkt); - uint32_t plain_len = params->cipher_range.length; - uint8_t *aad_head = data + params->auth_range.offset; - uint8_t *aad_tail = data + params->cipher_range.offset + - params->cipher_range.length; - uint32_t auth_len = params->auth_range.length; + uint8_t *data = odp_packet_data(param->out_pkt); + uint32_t plain_len = param->cipher_range.length; + uint8_t *aad_head = data + param->auth_range.offset; + uint8_t *aad_tail = data + param->cipher_range.offset + + param->cipher_range.length; + uint32_t auth_len = param->auth_range.length; unsigned char iv_enc[AES_BLOCK_SIZE]; void *iv_ptr; - uint8_t *tag = data + params->hash_result_offset; + uint8_t *tag = data + param->hash_result_offset;

- if (params->override_iv_ptr) - iv_ptr = params->override_iv_ptr; + if (param->override_iv_ptr) + iv_ptr = param->override_iv_ptr; else if (session->cipher.iv.data) iv_ptr = session->cipher.iv.data; else return ODP_CRYPTO_ALG_ERR_IV_INVALID;

/* All cipher data must be part of the authentication */ - if (params->auth_range.offset > params->cipher_range.offset || - params->auth_range.offset + auth_len < - params->cipher_range.offset + plain_len) + if (param->auth_range.offset > param->cipher_range.offset || + param->auth_range.offset + auth_len < + param->cipher_range.offset + plain_len) return ODP_CRYPTO_ALG_ERR_DATA_SIZE;

/* @@ -328,7 +328,7 @@ odp_crypto_alg_err_t aes_gcm_encrypt(odp_crypto_op_params_t *params, memcpy(iv_enc, iv_ptr, AES_BLOCK_SIZE);

/* Adjust pointer for beginning of area to cipher/auth */ - uint8_t *plaindata = data + params->cipher_range.offset; + uint8_t *plaindata = data + param->cipher_range.offset;

/* Encrypt it */ EVP_CIPHER_CTX *ctx = session->cipher.data.aes_gcm.ctx; @@ -359,30 +359,30 @@ odp_crypto_alg_err_t aes_gcm_encrypt(odp_crypto_op_params_t *params, }

static -odp_crypto_alg_err_t aes_gcm_decrypt(odp_crypto_op_params_t *params, +odp_crypto_alg_err_t aes_gcm_decrypt(odp_crypto_op_param_t *param, odp_crypto_generic_session_t *session) { - uint8_t *data = odp_packet_data(params->out_pkt); - uint32_t cipher_len = params->cipher_range.length; - uint8_t *aad_head = data + params->auth_range.offset; - uint8_t *aad_tail = data + params->cipher_range.offset + - params->cipher_range.length; - uint32_t auth_len = params->auth_range.length; + uint8_t *data = odp_packet_data(param->out_pkt); + uint32_t cipher_len = param->cipher_range.length; + uint8_t *aad_head = data + param->auth_range.offset; + uint8_t *aad_tail = data + param->cipher_range.offset + + param->cipher_range.length; + uint32_t auth_len = param->auth_range.length; unsigned char iv_enc[AES_BLOCK_SIZE]; void *iv_ptr; - uint8_t *tag = data + params->hash_result_offset; + uint8_t *tag = data + param->hash_result_offset;

- if (params->override_iv_ptr) - iv_ptr = params->override_iv_ptr; + if (param->override_iv_ptr) + iv_ptr = param->override_iv_ptr; else if (session->cipher.iv.data) iv_ptr = session->cipher.iv.data; else return ODP_CRYPTO_ALG_ERR_IV_INVALID;

/* All cipher data must be part of the authentication */ - if (params->auth_range.offset > params->cipher_range.offset || - params->auth_range.offset + auth_len < - params->cipher_range.offset + cipher_len) + if (param->auth_range.offset > param->cipher_range.offset || + param->auth_range.offset + auth_len < + param->cipher_range.offset + cipher_len) return ODP_CRYPTO_ALG_ERR_DATA_SIZE;

/* @@ -393,7 +393,7 @@ odp_crypto_alg_err_t aes_gcm_decrypt(odp_crypto_op_params_t *params, memcpy(iv_enc, iv_ptr, AES_BLOCK_SIZE);

/* Adjust pointer for beginning of area to cipher/auth */ - uint8_t *cipherdata = data + params->cipher_range.offset; + uint8_t *cipherdata = data + param->cipher_range.offset; /* Encrypt it */ EVP_CIPHER_CTX *ctx = session->cipher.data.aes_gcm.ctx; int plain_len = 0; @@ -425,18 +425,18 @@ odp_crypto_alg_err_t aes_gcm_decrypt(odp_crypto_op_params_t *params, }

static -int process_aes_gcm_params(odp_crypto_generic_session_t *session, - odp_crypto_session_params_t *params) +int process_aes_gcm_param(odp_crypto_generic_session_t *session, + odp_crypto_session_param_t *param) { /* Verify Key len is 16 */ - if (params->cipher_key.length != 16) + if (param->cipher_key.length != 16) return -1;

/* Set function */ EVP_CIPHER_CTX *ctx = session->cipher.data.aes_gcm.ctx = EVP_CIPHER_CTX_new();

- if (ODP_CRYPTO_OP_ENCODE == params->op) { + if (ODP_CRYPTO_OP_ENCODE == param->op) { session->cipher.func = aes_gcm_encrypt; EVP_EncryptInit_ex(ctx, EVP_aes_128_gcm(), NULL, NULL, NULL); } else { @@ -445,29 +445,29 @@ int process_aes_gcm_params(odp_crypto_generic_session_t *session, }

EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_IVLEN, - params->iv.length, NULL); - if (ODP_CRYPTO_OP_ENCODE == params->op) { + param->iv.length, NULL); + if (ODP_CRYPTO_OP_ENCODE == param->op) { EVP_EncryptInit_ex(ctx, NULL, NULL, - params->cipher_key.data, NULL); + param->cipher_key.data, NULL); } else { EVP_DecryptInit_ex(ctx, NULL, NULL, - params->cipher_key.data, NULL); + param->cipher_key.data, NULL); }

return 0; }

static -odp_crypto_alg_err_t des_encrypt(odp_crypto_op_params_t *params, +odp_crypto_alg_err_t des_encrypt(odp_crypto_op_param_t *param, odp_crypto_generic_session_t *session) { - uint8_t *data = odp_packet_data(params->out_pkt); - uint32_t len = params->cipher_range.length; + uint8_t *data = odp_packet_data(param->out_pkt); + uint32_t len = param->cipher_range.length; DES_cblock iv; void *iv_ptr;

- if (params->override_iv_ptr) - iv_ptr = params->override_iv_ptr; + if (param->override_iv_ptr) + iv_ptr = param->override_iv_ptr; else if (session->cipher.iv.data) iv_ptr = session->cipher.iv.data; else @@ -481,7 +481,7 @@ odp_crypto_alg_err_t des_encrypt(odp_crypto_op_params_t *params, memcpy(iv, iv_ptr, sizeof(iv));

/* Adjust pointer for beginning of area to cipher */ - data += params->cipher_range.offset; + data += param->cipher_range.offset; /* Encrypt it */ DES_ede3_cbc_encrypt(data, data, @@ -496,16 +496,16 @@ odp_crypto_alg_err_t des_encrypt(odp_crypto_op_params_t *params, }

static -odp_crypto_alg_err_t des_decrypt(odp_crypto_op_params_t *params, +odp_crypto_alg_err_t des_decrypt(odp_crypto_op_param_t *param, odp_crypto_generic_session_t *session) { - uint8_t *data = odp_packet_data(params->out_pkt); - uint32_t len = params->cipher_range.length; + uint8_t *data = odp_packet_data(param->out_pkt); + uint32_t len = param->cipher_range.length; DES_cblock iv; void *iv_ptr;

- if (params->override_iv_ptr) - iv_ptr = params->override_iv_ptr; + if (param->override_iv_ptr) + iv_ptr = param->override_iv_ptr; else if (session->cipher.iv.data) iv_ptr = session->cipher.iv.data; else @@ -519,7 +519,7 @@ odp_crypto_alg_err_t des_decrypt(odp_crypto_op_params_t *params, memcpy(iv, iv_ptr, sizeof(iv));

/* Adjust pointer for beginning of area to cipher */ - data += params->cipher_range.offset; + data += param->cipher_range.offset;

/* Decrypt it */ DES_ede3_cbc_encrypt(data, @@ -535,37 +535,37 @@ odp_crypto_alg_err_t des_decrypt(odp_crypto_op_params_t *params, }

static -int process_des_params(odp_crypto_generic_session_t *session, - odp_crypto_session_params_t *params) +int process_des_param(odp_crypto_generic_session_t *session, + odp_crypto_session_param_t *param) { /* Verify IV len is either 0 or 8 */ - if (!((0 == params->iv.length) || (8 == params->iv.length))) + if (!((0 == param->iv.length) || (8 == param->iv.length))) return -1;

/* Set function */ - if (ODP_CRYPTO_OP_ENCODE == params->op) + if (ODP_CRYPTO_OP_ENCODE == param->op) session->cipher.func = des_encrypt; else session->cipher.func = des_decrypt;

/* Convert keys */ - DES_set_key((DES_cblock *)&params->cipher_key.data[0], + DES_set_key((DES_cblock *)&param->cipher_key.data[0], &session->cipher.data.des.ks1); - DES_set_key((DES_cblock *)&params->cipher_key.data[8], + DES_set_key((DES_cblock *)&param->cipher_key.data[8], &session->cipher.data.des.ks2); - DES_set_key((DES_cblock *)&params->cipher_key.data[16], + DES_set_key((DES_cblock *)&param->cipher_key.data[16], &session->cipher.data.des.ks3);

return 0; }

static -int process_md5_params(odp_crypto_generic_session_t *session, - odp_crypto_session_params_t *params, - uint32_t bits) +int process_md5_param(odp_crypto_generic_session_t *session, + odp_crypto_session_param_t *param, + uint32_t bits) { /* Set function */ - if (ODP_CRYPTO_OP_ENCODE == params->op) + if (ODP_CRYPTO_OP_ENCODE == param->op) session->auth.func = md5_gen; else session->auth.func = md5_check; @@ -574,18 +574,18 @@ int process_md5_params(odp_crypto_generic_session_t *session, session->auth.data.md5.bytes = bits / 8;

/* Convert keys */ - memcpy(session->auth.data.md5.key, params->auth_key.data, 16); + memcpy(session->auth.data.md5.key, param->auth_key.data, 16);

return 0; }

static -int process_sha256_params(odp_crypto_generic_session_t *session, - odp_crypto_session_params_t *params, - uint32_t bits) +int process_sha256_param(odp_crypto_generic_session_t *session, + odp_crypto_session_param_t *param, + uint32_t bits) { /* Set function */ - if (ODP_CRYPTO_OP_ENCODE == params->op) + if (ODP_CRYPTO_OP_ENCODE == param->op) session->auth.func = sha256_gen; else session->auth.func = sha256_check; @@ -594,7 +594,7 @@ int process_sha256_params(odp_crypto_generic_session_t *session, session->auth.data.sha256.bytes = bits / 8;

/* Convert keys */ - memcpy(session->auth.data.sha256.key, params->auth_key.data, 32); + memcpy(session->auth.data.sha256.key, param->auth_key.data, 32);

return 0; } @@ -624,7 +624,7 @@ int odp_crypto_capability(odp_crypto_capability_t *capa) }

int -odp_crypto_session_create(odp_crypto_session_params_t *params, +odp_crypto_session_create(odp_crypto_session_param_t *param, odp_crypto_session_t *session_out, odp_crypto_ses_create_err_t *status) { @@ -642,41 +642,41 @@ odp_crypto_session_create(odp_crypto_session_params_t *params, }

/* Derive order */ - if (ODP_CRYPTO_OP_ENCODE == params->op) - session->do_cipher_first = params->auth_cipher_text; + if (ODP_CRYPTO_OP_ENCODE == param->op) + session->do_cipher_first = param->auth_cipher_text; else - session->do_cipher_first = !params->auth_cipher_text; + session->do_cipher_first = !param->auth_cipher_text;

/* Copy stuff over */ - session->op = params->op; - session->compl_queue = params->compl_queue; - session->cipher.alg = params->cipher_alg; - session->cipher.iv.data = params->iv.data; - session->cipher.iv.len = params->iv.length; - session->auth.alg = params->auth_alg; - session->output_pool = params->output_pool; + session->op = param->op; + session->compl_queue = param->compl_queue; + session->cipher.alg = param->cipher_alg; + session->cipher.iv.data = param->iv.data; + session->cipher.iv.len = param->iv.length; + session->auth.alg = param->auth_alg; + session->output_pool = param->output_pool;

/* Process based on cipher */ - switch (params->cipher_alg) { + switch (param->cipher_alg) { case ODP_CIPHER_ALG_NULL: session->cipher.func = null_crypto_routine; rc = 0; break; case ODP_CIPHER_ALG_DES: case ODP_CIPHER_ALG_3DES_CBC: - rc = process_des_params(session, params); + rc = process_des_param(session, param); break; case ODP_CIPHER_ALG_AES128_CBC: - rc = process_aes_params(session, params); + rc = process_aes_param(session, param); break; case ODP_CIPHER_ALG_AES128_GCM: /* AES-GCM requires to do both auth and * cipher at the same time */ - if (params->auth_alg != ODP_AUTH_ALG_AES128_GCM) { + if (param->auth_alg != ODP_AUTH_ALG_AES128_GCM) { rc = -1; break; } - rc = process_aes_gcm_params(session, params); + rc = process_aes_gcm_param(session, param); break; default: rc = -1; @@ -689,21 +689,21 @@ odp_crypto_session_create(odp_crypto_session_params_t *params, }

/* Process based on auth */ - switch (params->auth_alg) { + switch (param->auth_alg) { case ODP_AUTH_ALG_NULL: session->auth.func = null_crypto_routine; rc = 0; break; case ODP_AUTH_ALG_MD5_96: - rc = process_md5_params(session, params, 96); + rc = process_md5_param(session, param, 96); break; case ODP_AUTH_ALG_SHA256_128: - rc = process_sha256_params(session, params, 128); + rc = process_sha256_param(session, param, 128); break; case ODP_AUTH_ALG_AES128_GCM: /* AES-GCM requires to do both auth and * cipher at the same time */ - if (params->cipher_alg != ODP_CIPHER_ALG_AES128_GCM) { + if (param->cipher_alg != ODP_CIPHER_ALG_AES128_GCM) { rc = -1; break; } @@ -738,7 +738,7 @@ int odp_crypto_session_destroy(odp_crypto_session_t session) }

int -odp_crypto_operation(odp_crypto_op_params_t *params, +odp_crypto_operation(odp_crypto_op_param_t *param, odp_bool_t *posted, odp_crypto_op_result_t *result) { @@ -747,42 +747,42 @@ odp_crypto_operation(odp_crypto_op_params_t *params, odp_crypto_generic_session_t *session; odp_crypto_op_result_t local_result;

- session = (odp_crypto_generic_session_t *)(intptr_t)params->session; + session = (odp_crypto_generic_session_t *)(intptr_t)param->session;

/* Resolve output buffer */ - if (ODP_PACKET_INVALID == params->out_pkt && + if (ODP_PACKET_INVALID == param->out_pkt && ODP_POOL_INVALID != session->output_pool) - params->out_pkt = odp_packet_alloc(session->output_pool, - odp_packet_len(params->pkt)); + param->out_pkt = odp_packet_alloc(session->output_pool, + odp_packet_len(param->pkt));

- if (odp_unlikely(ODP_PACKET_INVALID == params->out_pkt)) { + if (odp_unlikely(ODP_PACKET_INVALID == param->out_pkt)) { ODP_DBG("Alloc failed.\n"); return -1; }

- if (params->pkt != params->out_pkt) { - (void)odp_packet_copy_from_pkt(params->out_pkt, + if (param->pkt != param->out_pkt) { + (void)odp_packet_copy_from_pkt(param->out_pkt, 0, - params->pkt, + param->pkt, 0, - odp_packet_len(params->pkt)); - _odp_packet_copy_md_to_packet(params->pkt, params->out_pkt); - odp_packet_free(params->pkt); - params->pkt = ODP_PACKET_INVALID; + odp_packet_len(param->pkt)); + _odp_packet_copy_md_to_packet(param->pkt, param->out_pkt); + odp_packet_free(param->pkt); + param->pkt = ODP_PACKET_INVALID; }

/* Invoke the functions */ if (session->do_cipher_first) { - rc_cipher = session->cipher.func(params, session); - rc_auth = session->auth.func(params, session); + rc_cipher = session->cipher.func(param, session); + rc_auth = session->auth.func(param, session); } else { - rc_auth = session->auth.func(params, session); - rc_cipher = session->cipher.func(params, session); + rc_auth = session->auth.func(param, session); + rc_cipher = session->cipher.func(param, session); }

/* Fill in result */ - local_result.ctx = params->ctx; - local_result.pkt = params->out_pkt; + local_result.ctx = param->ctx; + local_result.pkt = param->out_pkt; local_result.cipher_status.alg_err = rc_cipher; local_result.cipher_status.hw_err = ODP_CRYPTO_HW_ERR_NONE; local_result.auth_status.alg_err = rc_auth; @@ -797,7 +797,7 @@ odp_crypto_operation(odp_crypto_op_params_t *params, odp_crypto_generic_op_result_t *op_result;

/* Linux generic will always use packet for completion event */ - completion_event = odp_packet_to_event(params->out_pkt); + completion_event = odp_packet_to_event(param->out_pkt); _odp_buffer_event_type_set( odp_buffer_from_event(completion_event), ODP_EVENT_CRYPTO_COMPL);

commit 55c4f518ecd510d2d812e3fc766d6690f65871c3 Author: Petri Savolainen petri.savolainen@nokia.com Date: Thu Dec 8 16:25:32 2016 +0200

api: crypto: rename _params_t to _param_t

The common naming convention for parameter types is _param_t (without 's'). Old type names remain for backwards compatibility, but are deprecated.

Signed-off-by: Petri Savolainen petri.savolainen@nokia.com Reviewed-by: Bill Fischofer bill.fischofer@linaro.org Reviewed-by: Balasubramanian Manoharan bala.manoharan@linaro.org Reviewed-by: Nikhil Agarwal nikhil.agarwal@linaro.org Signed-off-by: Maxim Uvarov maxim.uvarov@linaro.org

diff --git a/include/odp/api/spec/crypto.h b/include/odp/api/spec/crypto.h index 0cb8814..f24f527 100644 --- a/include/odp/api/spec/crypto.h +++ b/include/odp/api/spec/crypto.h @@ -171,7 +171,7 @@ typedef struct odp_crypto_data_range { /** * Crypto API session creation parameters */ -typedef struct odp_crypto_session_params { +typedef struct odp_crypto_session_param_t { odp_crypto_op_t op; /**< Encode versus decode */ odp_bool_t auth_cipher_text; /**< Authenticate/cipher ordering */ odp_crypto_op_mode_t pref_mode; /**< Preferred sync vs async */ @@ -182,7 +182,10 @@ typedef struct odp_crypto_session_params { odp_crypto_key_t auth_key; /**< Authentication key */ odp_queue_t compl_queue; /**< Async mode completion event queue */ odp_pool_t output_pool; /**< Output buffer pool */ -} odp_crypto_session_params_t; +} odp_crypto_session_param_t; + +/** @deprecated Use odp_crypto_session_param_t instead */ +typedef odp_crypto_session_param_t odp_crypto_session_params_t;

/** * @var odp_crypto_session_params_t::auth_cipher_text @@ -209,7 +212,7 @@ typedef struct odp_crypto_session_params { /** * Crypto API per packet operation parameters */ -typedef struct odp_crypto_op_params { +typedef struct odp_crypto_op_param_t { odp_crypto_session_t session; /**< Session handle from creation */ void *ctx; /**< User context */ odp_packet_t pkt; /**< Input packet buffer */ @@ -218,7 +221,10 @@ typedef struct odp_crypto_op_params { uint32_t hash_result_offset; /**< Offset from start of packet buffer for hash result */ odp_crypto_data_range_t cipher_range; /**< Data range to apply cipher */ odp_crypto_data_range_t auth_range; /**< Data range to authenticate */ -} odp_crypto_op_params_t; +} odp_crypto_op_param_t; + +/** @deprecated Use odp_crypto_op_param_t instead */ +typedef odp_crypto_op_param_t odp_crypto_op_params_t;

/** * @var odp_crypto_op_params_t::pkt @@ -349,14 +355,14 @@ int odp_crypto_capability(odp_crypto_capability_t *capa); /** * Crypto session creation (synchronous) * - * @param params Session parameters + * @param param Session parameters * @param session Created session else ODP_CRYPTO_SESSION_INVALID * @param status Failure code if unsuccessful * * @retval 0 on success * @retval <0 on failure */ -int odp_crypto_session_create(odp_crypto_session_params_t *params, +int odp_crypto_session_create(odp_crypto_session_param_t *param, odp_crypto_session_t *session, odp_crypto_ses_create_err_t *status);

@@ -410,14 +416,14 @@ void odp_crypto_compl_free(odp_crypto_compl_t completion_event); * If "posted" returns TRUE the result will be delivered via the completion * queue specified when the session was created. * - * @param params Operation parameters + * @param param Operation parameters * @param posted Pointer to return posted, TRUE for async operation * @param result Results of operation (when posted returns FALSE) * * @retval 0 on success * @retval <0 on failure */ -int odp_crypto_operation(odp_crypto_op_params_t *params, +int odp_crypto_operation(odp_crypto_op_param_t *param, odp_bool_t *posted, odp_crypto_op_result_t *result);

commit 2deb9c98440a06f0b2e6733693f58d081059dc0d Author: Balasubramanian Manoharan bala.manoharan@linaro.org Date: Thu Oct 27 18:30:55 2016 +0530

validation: classification: fix TCP/UDP checksum update

Fixes https://bugs.linaro.org/show_bug.cgi?id=2512

Signed-off-by: Balasubramanian Manoharan bala.manoharan@linaro.org Reviewed-by: Nikhil Agarwal nikhil.agarwal@linaro.org Signed-off-by: Maxim Uvarov maxim.uvarov@linaro.org

diff --git a/test/common_plat/validation/api/classification/odp_classification_common.c b/test/common_plat/validation/api/classification/odp_classification_common.c index 7a42ac7..2923a71 100644 --- a/test/common_plat/validation/api/classification/odp_classification_common.c +++ b/test/common_plat/validation/api/classification/odp_classification_common.c @@ -11,6 +11,7 @@ #include <odp/helper/ip.h> #include <odp/helper/udp.h> #include <odp/helper/tcp.h> +#include "test_debug.h"

typedef struct cls_test_packet { odp_u32be_t magic; @@ -291,6 +292,8 @@ odp_packet_t create_packet_len(odp_pool_t pool, bool vlan, parse_ipv4_string(CLS_DEFAULT_SADDR, &addr, &mask); ip->src_addr = odp_cpu_to_be_32(addr); ip->ver_ihl = ODPH_IPV4 << 4 | ODPH_IPV4HDR_IHL_MIN; + odp_packet_has_ipv4_set(pkt, 1); + if (flag_udp) ip->tot_len = odp_cpu_to_be_16(ODPH_UDPHDR_LEN + payload_len + ODPH_IPV4HDR_LEN); @@ -318,14 +321,23 @@ odp_packet_t create_packet_len(odp_pool_t pool, bool vlan, udp->dst_port = odp_cpu_to_be_16(CLS_DEFAULT_DPORT); udp->length = odp_cpu_to_be_16(payload_len + ODPH_UDPHDR_LEN); udp->chksum = 0; + odp_packet_has_udp_set(pkt, 1); + if (odph_udp_tcp_chksum(pkt, ODPH_CHKSUM_GENERATE, NULL) != 0) { + LOG_ERR("odph_udp_tcp_chksum failed\n"); + return ODP_PACKET_INVALID; + } } else { odp_packet_l4_offset_set(pkt, offset); tcp = (odph_tcphdr_t *)odp_packet_l4_ptr(pkt, NULL); tcp->src_port = odp_cpu_to_be_16(CLS_DEFAULT_SPORT); tcp->dst_port = odp_cpu_to_be_16(CLS_DEFAULT_DPORT); tcp->hl = ODPH_TCPHDR_LEN / 4; - /* TODO: checksum field has to be updated */ tcp->cksm = 0; + odp_packet_has_tcp_set(pkt, 1); + if (odph_udp_tcp_chksum(pkt, ODPH_CHKSUM_GENERATE, NULL) != 0) { + LOG_ERR("odph_udp_tcp_chksum failed\n"); + return ODP_PACKET_INVALID; + } }

/* set pkt sequence number */ diff --git a/test/common_plat/validation/api/classification/odp_classification_test_pmr.c b/test/common_plat/validation/api/classification/odp_classification_test_pmr.c index c8bbf50..88fbf8f 100644 --- a/test/common_plat/validation/api/classification/odp_classification_test_pmr.c +++ b/test/common_plat/validation/api/classification/odp_classification_test_pmr.c @@ -717,6 +717,7 @@ void classification_test_pmr_term_dmac(void)

/* Other packets delivered to default queue */ pkt = create_packet(pkt_pool, false, &seq, false); + CU_ASSERT_FATAL(pkt != ODP_PACKET_INVALID); eth = (odph_ethhdr_t *)odp_packet_l2_ptr(pkt, NULL); memset(eth->dst.addr, 0, ODPH_ETHADDR_LEN); CU_ASSERT_FATAL(pkt != ODP_PACKET_INVALID); @@ -1102,6 +1103,7 @@ static void classification_test_pmr_term_daddr(void) /* packet with dst ip address matching PMR rule to be received in the CoS queue*/ pkt = create_packet(pkt_pool, false, &seq, false); + CU_ASSERT_FATAL(pkt != ODP_PACKET_INVALID); eth = (odph_ethhdr_t *)odp_packet_l2_ptr(pkt, NULL); odp_pktio_mac_addr(pktio, eth->src.addr, ODPH_ETHADDR_LEN); odp_pktio_mac_addr(pktio, eth->dst.addr, ODPH_ETHADDR_LEN); @@ -1122,6 +1124,7 @@ static void classification_test_pmr_term_daddr(void)

/* Other packets delivered to default queue */ pkt = create_packet(pkt_pool, false, &seq, false); + CU_ASSERT_FATAL(pkt != ODP_PACKET_INVALID); seqno = cls_pkt_get_seq(pkt); CU_ASSERT(seqno != TEST_SEQ_INVALID); eth = (odph_ethhdr_t *)odp_packet_l2_ptr(pkt, NULL);

commit cbc97ae9237f1a206d1f4e25ae7e98ad99d430cb Author: Xuelin Shi forrest.shi@linaro.org Date: Fri Dec 2 15:42:32 2016 +0800

linux-generic: only enable pktout when egress kind is pktio

Signed-off-by: Xuelin Shi forrest.shi@linaro.org Reviewed-by: Balasubramanian Manoharan bala.manoharan@linaro.org Signed-off-by: Maxim Uvarov maxim.uvarov@linaro.org

diff --git a/platform/linux-generic/odp_traffic_mngr.c b/platform/linux-generic/odp_traffic_mngr.c index 4a09da1..9dc3a86 100644 --- a/platform/linux-generic/odp_traffic_mngr.c +++ b/platform/linux-generic/odp_traffic_mngr.c @@ -2855,10 +2855,9 @@ odp_tm_t odp_tm_create(const char *name, return ODP_TM_INVALID; }

- if (odp_pktout_queue(egress->pktio, &pktout, 1) != 1) - return ODP_TM_INVALID; + if (egress->egress_kind == ODP_TM_EGRESS_PKT_IO) + tm_system->pktout = pktout;

- tm_system->pktout = pktout; tm_system->name_tbl_id = name_tbl_id; max_tm_queues = requirements->max_tm_queues; memcpy(&tm_system->egress, egress, sizeof(odp_tm_egress_t));

-----------------------------------------------------------------------

Summary of changes: include/odp/api/spec/event.h | 2 +- include/odp/api/spec/ipsec.h | 883 --------------------- include/odp_api.h | 1 - platform/Makefile.inc | 1 - platform/linux-generic/Makefile.am | 2 - platform/linux-generic/include/odp/api/ipsec.h | 36 - .../include/odp/api/plat/event_types.h | 1 - .../include/odp/api/plat/ipsec_types.h | 39 - 8 files changed, 1 insertion(+), 964 deletions(-) delete mode 100644 include/odp/api/spec/ipsec.h delete mode 100644 platform/linux-generic/include/odp/api/ipsec.h delete mode 100644 platform/linux-generic/include/odp/api/plat/ipsec_types.h

hooks/post-receive