- Linux-kselftest-mirror - lists.linaro.org

[PATCH v3] selftests/mm: Fix ARM related issue with fork after pthread_create

by Edward Liaw

Following issue was observed while running the uffd-unit-tests selftest on ARM devices. On x86_64 no issues were detected: pthread_create followed by fork caused deadlock in certain cases wherein fork required some work to be completed by the created thread. Used synchronization to ensure that created thread's start function has started before invoking fork. Signed-off-by: Lokesh Gidra <lokeshgidra(a)google.com> [edliaw: Refactored to use atomic_bool] Signed-off-by: Edward Liaw <edliaw(a)google.com> --- v2: restored accidentally removed uffd_test_case_ops when merging v3: fixed commit subject to use selftests/mm prefix tools/testing/selftests/mm/uffd-common.c | 3 +++ tools/testing/selftests/mm/uffd-common.h | 2 ++ tools/testing/selftests/mm/uffd-unit-tests.c | 10 ++++++++++ 3 files changed, 15 insertions(+) diff --git a/tools/testing/selftests/mm/uffd-common.c b/tools/testing/selftests/mm/uffd-common.c index b0ac0ec2356d..7ad6ba660c7d 100644 --- a/tools/testing/selftests/mm/uffd-common.c +++ b/tools/testing/selftests/mm/uffd-common.c @@ -18,6 +18,7 @@ bool test_uffdio_wp = true; unsigned long long *count_verify; uffd_test_ops_t *uffd_test_ops; uffd_test_case_ops_t *uffd_test_case_ops; +atomic_bool ready_for_fork; static int uffd_mem_fd_create(off_t mem_size, bool hugetlb) { @@ -518,6 +519,8 @@ void *uffd_poll_thread(void *arg) pollfd[1].fd = pipefd[cpu*2]; pollfd[1].events = POLLIN; + ready_for_fork = true; + for (;;) { ret = poll(pollfd, 2, -1); if (ret <= 0) { diff --git a/tools/testing/selftests/mm/uffd-common.h b/tools/testing/selftests/mm/uffd-common.h index cb055282c89c..cc5629c3d2aa 100644 --- a/tools/testing/selftests/mm/uffd-common.h +++ b/tools/testing/selftests/mm/uffd-common.h @@ -32,6 +32,7 @@ #include <inttypes.h> #include <stdint.h> #include <sys/random.h> +#include <stdatomic.h> #include "../kselftest.h" #include "vm_util.h" @@ -103,6 +104,7 @@ extern bool map_shared; extern bool test_uffdio_wp; extern unsigned long long *count_verify; extern volatile bool test_uffdio_copy_eexist; +extern atomic_bool ready_for_fork; extern uffd_test_ops_t anon_uffd_test_ops; extern uffd_test_ops_t shmem_uffd_test_ops; diff --git a/tools/testing/selftests/mm/uffd-unit-tests.c b/tools/testing/selftests/mm/uffd-unit-tests.c index 2b9f8cc52639..4a48dc617c6b 100644 --- a/tools/testing/selftests/mm/uffd-unit-tests.c +++ b/tools/testing/selftests/mm/uffd-unit-tests.c @@ -775,6 +775,8 @@ static void uffd_sigbus_test_common(bool wp) char c; struct uffd_args args = { 0 }; + ready_for_fork = false; + fcntl(uffd, F_SETFL, uffd_flags | O_NONBLOCK); if (uffd_register(uffd, area_dst, nr_pages * page_size, @@ -790,6 +792,9 @@ static void uffd_sigbus_test_common(bool wp) if (pthread_create(&uffd_mon, NULL, uffd_poll_thread, &args)) err("uffd_poll_thread create"); + while (!ready_for_fork) + ; /* Wait for the poll_thread to start executing before forking */ + pid = fork(); if (pid < 0) err("fork"); @@ -829,6 +834,8 @@ static void uffd_events_test_common(bool wp) char c; struct uffd_args args = { 0 }; + ready_for_fork = false; + fcntl(uffd, F_SETFL, uffd_flags | O_NONBLOCK); if (uffd_register(uffd, area_dst, nr_pages * page_size, true, wp, false)) @@ -838,6 +845,9 @@ static void uffd_events_test_common(bool wp) if (pthread_create(&uffd_mon, NULL, uffd_poll_thread, &args)) err("uffd_poll_thread create"); + while (!ready_for_fork) + ; /* Wait for the poll_thread to start executing before forking */ + pid = fork(); if (pid < 0) err("fork"); -- 2.44.0.396.g6e790dbe36-goog

1 year, 3 months

4
3
0 0

[PATCH v2] selftests/mm: include strings.h for ffsl

by Edward Liaw

Got a compilation error on Android for ffsl after 91b80cc5b39f ("selftests: mm: fix map_hugetlb failure on 64K page size systems") included vm_util.h. Fixes: af605d26a8f2 ("selftests/mm: merge util.h into vm_util.h") Signed-off-by: Edward Liaw <edliaw(a)google.com> --- v2: s/import/include and clarified that compilation error was on Android tools/testing/selftests/mm/vm_util.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/testing/selftests/mm/vm_util.h b/tools/testing/selftests/mm/vm_util.h index c02990bbd56f..9007c420d52c 100644 --- a/tools/testing/selftests/mm/vm_util.h +++ b/tools/testing/selftests/mm/vm_util.h @@ -3,7 +3,7 @@ #include <stdbool.h> #include <sys/mman.h> #include <err.h> -#include <string.h> /* ffsl() */ +#include <strings.h> /* ffsl() */ #include <unistd.h> /* _SC_PAGESIZE */ #define BIT_ULL(nr) (1ULL << (nr)) -- 2.44.0.478.gd926399ef9-goog

1 year, 3 months

1
0
0 0

[PATCH v2 00/14] Add support for suppressing warning backtraces

by Guenter Roeck

Some unit tests intentionally trigger warning backtraces by passing bad parameters to kernel API functions. Such unit tests typically check the return value from such calls, not the existence of the warning backtrace. Such intentionally generated warning backtraces are neither desirable nor useful for a number of reasons. - They can result in overlooked real problems. - A warning that suddenly starts to show up in unit tests needs to be investigated and has to be marked to be ignored, for example by adjusting filter scripts. Such filters are ad-hoc because there is no real standard format for warnings. On top of that, such filter scripts would require constant maintenance. One option to address problem would be to add messages such as "expected warning backtraces start / end here" to the kernel log. However, that would again require filter scripts, it might result in missing real problematic warning backtraces triggered while the test is running, and the irrelevant backtrace(s) would still clog the kernel log. Solve the problem by providing a means to identify and suppress specific warning backtraces while executing test code. Support suppressing multiple backtraces while at the same time limiting changes to generic code to the absolute minimum. Architecture specific changes are kept at minimum by retaining function names only if both CONFIG_DEBUG_BUGVERBOSE and CONFIG_KUNIT are enabled. The first patch of the series introduces the necessary infrastructure. The second patch introduces support for counting suppressed backtraces. This capability is used in patch three to implement unit tests. Patch four documents the new API. The next two patches add support for suppressing backtraces in drm_rect and dev_addr_lists unit tests. These patches are intended to serve as examples for the use of the functionality introduced with this series. The remaining patches implement the necessary changes for all architectures with GENERIC_BUG support. With CONFIG_KUNIT enabled, image size increase with this series applied is approximately 1%. The image size increase (and with it the functionality introduced by this series) can be avoided by disabling CONFIG_KUNIT_SUPPRESS_BACKTRACE. This series is based on the RFC patch and subsequent discussion at https://patchwork.kernel.org/project/linux-kselftest/patch/02546e59-1afe-4b… and offers a more comprehensive solution of the problem discussed there. Design note: Function pointers are only added to the __bug_table section if both CONFIG_KUNIT_SUPPRESS_BACKTRACE and CONFIG_DEBUG_BUGVERBOSE are enabled to avoid image size increases if CONFIG_KUNIT is disabled. There would be some benefits to adding those pointers all the time (reduced complexity, ability to display function names in BUG/WARNING messages). That change, if desired, can be made later. Checkpatch note: Remaining checkpatch errors and warnings were deliberately ignored. Some are triggered by matching coding style or by comments interpreted as code, others by assembler macros which are disliked by checkpatch. Suggestions for improvements are welcome. Changes since RFC: - Introduced CONFIG_KUNIT_SUPPRESS_BACKTRACE - Minor cleanups and bug fixes - Added support for all affected architectures - Added support for counting suppressed warnings - Added unit tests using those counters - Added patch to suppress warning backtraces in dev_addr_lists tests Changes since v1: - Rebased to v6.9-rc1 - Added Tested-by:, Acked-by:, and Reviewed-by: tags [I retained those tags since there have been no functional changes] - Introduced KUNIT_SUPPRESS_BACKTRACE configuration option, enabled by default.

1 year, 3 months

3
22
0 0

pidfd_setns_test.c:428:nsfd_incremental_setns:Too many users - Failed to setns to time namespace of 31577 via nsfd 45

by Paul Menzel

Dear Linux folks, Running the selftests in a QEMU q35 VM, they hang. ``` $ make kselftest […] ok 6 selftests: pidfd: pidfd_getfd_test # timeout set to 45 # selftests: pidfd: pidfd_setns_test # TAP version 13 # 1..7 # # Starting 7 tests from 2 test cases. # # RUN global.setns_einval ... # # OK global.setns_einval # ok 1 global.setns_einval # # RUN current_nsset.invalid_flags ... # # pidfd_setns_test.c:161:invalid_flags:Expected self->child_pid_exited (0) > 0 (0) # # OK current_nsset.invalid_flags # ok 2 current_nsset.invalid_flags # # RUN current_nsset.pidfd_exited_child ... # # pidfd_setns_test.c:161:pidfd_exited_child:Expected self->child_pid_exited (0) > 0 (0) # # OK current_nsset.pidfd_exited_child # ok 3 current_nsset.pidfd_exited_child # # RUN current_nsset.pidfd_incremental_setns ... # # pidfd_setns_test.c:161:pidfd_incremental_setns:Expected self->child_pid_exited (0) > 0 (0) # # pidfd_setns_test.c:408:pidfd_incremental_setns:Managed to correctly setns to user namespace of 31570 via pidfd 20 # # pidfd_setns_test.c:408:pidfd_incremental_setns:Managed to correctly setns to mnt namespace of 31570 via pidfd 20 # # pidfd_setns_test.c:408:pidfd_incremental_setns:Managed to correctly setns to pid namespace of 31570 via pidfd 20 # # pidfd_setns_test.c:408:pidfd_incremental_setns:Managed to correctly setns to uts namespace of 31570 via pidfd 20 # # pidfd_setns_test.c:408:pidfd_incremental_setns:Managed to correctly setns to ipc namespace of 31570 via pidfd 20 # # pidfd_setns_test.c:408:pidfd_incremental_setns:Managed to correctly setns to net namespace of 31570 via pidfd 20 # # pidfd_setns_test.c:408:pidfd_incremental_setns:Managed to correctly setns to cgroup namespace of 31570 via pidfd 20 # # pidfd_setns_test.c:408:pidfd_incremental_setns:Managed to correctly setns to pid_for_children namespace of 31570 via pidfd 20 # # pidfd_setns_test.c:391:pidfd_incremental_setns:Expected setns(self->child_pidfd1, info->flag) (-1) == 0 (0) # # pidfd_setns_test.c:392:pidfd_incremental_setns:Too many users - Failed to setns to time namespace of 31570 via pidfd 20 # # pidfd_incremental_setns: Test terminated by timeout # # FAIL current_nsset.pidfd_incremental_setns # not ok 4 current_nsset.pidfd_incremental_setns # # RUN current_nsset.nsfd_incremental_setns ... # # pidfd_setns_test.c:161:nsfd_incremental_setns:Expected self->child_pid_exited (0) > 0 (0) # # pidfd_setns_test.c:444:nsfd_incremental_setns:Managed to correctly setns to user namespace of 31577 via nsfd 19 # # pidfd_setns_test.c:444:nsfd_incremental_setns:Managed to correctly setns to mnt namespace of 31577 via nsfd 24 # # pidfd_setns_test.c:444:nsfd_incremental_setns:Managed to correctly setns to pid namespace of 31577 via nsfd 27 # # pidfd_setns_test.c:444:nsfd_incremental_setns:Managed to correctly setns to uts namespace of 31577 via nsfd 30 # # pidfd_setns_test.c:444:nsfd_incremental_setns:Managed to correctly setns to ipc namespace of 31577 via nsfd 33 # # pidfd_setns_test.c:444:nsfd_incremental_setns:Managed to correctly setns to net namespace of 31577 via nsfd 36 # # pidfd_setns_test.c:444:nsfd_incremental_setns:Managed to correctly setns to cgroup namespace of 31577 via nsfd 39 # # pidfd_setns_test.c:444:nsfd_incremental_setns:Managed to correctly setns to pid_for_children namespace of 31577 via nsfd 42 # # pidfd_setns_test.c:427:nsfd_incremental_setns:Expected setns(self->child_nsfds1[i], info->flag) (-1) == 0 (0) # # pidfd_setns_test.c:428:nsfd_incremental_setns:Too many users - Failed to setns to time namespace of 31577 via nsfd 45 ``` Ctrl + c gets me back to the shell prompt. ``` $ ps auxf […] build 31574 0.0 0.0 2528 1024 pts/0 D 09:57 0:00 ./pidfd_setns_test build 31575 99.9 0.0 2528 1024 pts/0 R 09:57 37:37 \_ ./pidfd_setns_test build 31576 0.0 0.0 0 0 pts/0 Z 09:57 0:00 \_ [pidfd_setns_tes] <defunct> build 31577 0.0 0.0 0 0 pts/0 Z 09:57 0:00 \_ [pidfd_setns_tes] <defunct> build 31578 0.0 0.0 2528 512 pts/0 S 09:57 0:00 \_ ./pidfd_setns_test ``` During shutdown, the system waits for that process: [42733.704347] systemd-shutdown[1]: Waiting for process: 31578 (pidfd_setns_tes) This is commit 8d025e2092e2 with one patch on top: $ git log --no-decorate --oneline -2 a2ce022afcbb a2ce022afcbb [PATCH] kbuild: Disable KCSAN for autogenerated *.mod.c intermediaries 8d025e2092e2 Merge tag 'erofs-for-6.9-rc2-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs KCSAN is not enabled though. Kind regards, Paul

1 year, 3 months

1
0
0 0

[PATCH net-next 00/14] selftests: Fixes for kernel CI

by Petr Machata

As discussed on the bi-weekly call on Jan 30, and in mailing around kernel CI effort, some changes are desirable in the suite of forwarding selftests the better to work with the CI tooling. Namely: - The forwarding selftests use a configuration file where names of interfaces are defined and various variables can be overridden. There is also forwarding.config.sample that users can use as a template to refer to when creating the config file. What happens a fair bit is that users either do not know about this at all, or simply forget, and are confused by cryptic failures about interfaces that cannot be created. In patches #1 - #3 have lib.sh just be the single source of truth with regards to which variables exist. That includes the topology variables which were previously only in the sample file, and any "tweak variables", such as what tools to use, sleep times, etc. forwarding.config.sample then becomes just a placeholder with a couple examples. Unless specific HW should be exercised, or specific tools used, the defaults are usually just fine. - Several net/forwarding/ selftests (and one net/ one) cannot be run on veth pairs, they need an actual HW interface to run on. They are generic in the sense that any capable HW should pass them, which is why they have been put to net/forwarding/ as opposed to drivers/net/, but they do not generalize to veth. The fact that these tests are in net/forwarding/, but still complaining when run, is confusing. In patches #4 - #6 move these tests to a new directory drivers/net/hw. - The following patches extend the codebase to handle well test results other than pass and fail. Patch #7 is preparatory. It converts several log_test_skip to XFAIL, so that tests do not spuriously end up returning non-0 when they are not supposed to. In patches #8 - #10, introduce some missing ksft constants, then support having those constants in RET, and then finally in EXIT_STATUS. - The traffic scheduler tests generate a large amount of network traffic to test the behavior of the scheduler. This demands a relatively high-performance computer. On slow machines, such as with a debugging kernel, the test would spuriously fail. It can still be useful to "go through the motions" though, to possibly catch bugs in setup of the scheduler graph and passing packets around. Thus we still want to run the tests, just with lowered demands. To that end, in patches #11 - #12, introduce an environment variable KSFT_MACHINE_SLOW, with obvious meaning. Tests can then make checks more lenient, such as mark failures as XFAIL. A helper, xfail_on_slow, is provided to mark performance-sensitive parts of the selftest. - In patch #13, use a similar mechanism to mark a NH group stats selftest to XFAIL HW stats tests when run on VETH pairs. - All these changes complicate the hitherto straightforward logging and checking logic, so in patch #14, add a selftest that checks this functionality in lib.sh. v1 (vs. an RFC circulated through linux-kselftest): - Patch #9: - Clarify intended usage by s/set_ret/ret_set_ksft_status/, s/nret/ksft_status/ Petr Machata (14): selftests: net: libs: Change variable fallback syntax selftests: forwarding.config.sample: Move overrides to lib.sh selftests: forwarding: README: Document customization selftests: forwarding: ipip_lib: Do not import lib.sh selftests: forwarding: Move several selftests selftests: forwarding: Ditch skip_on_veth() selftests: forwarding: Change inappropriate log_test_skip() calls selftests: lib: Define more kselftest exit codes selftests: forwarding: Have RET track kselftest framework constants selftests: forwarding: Convert log_test() to recognize RET values selftests: forwarding: Support for performance sensitive tests selftests: forwarding: Mark performance-sensitive tests selftests: forwarding: router_mpath_nh_lib: Don't skip, xfail on veth selftests: forwarding: Add a test for testing lib.sh functionality .../testing/selftests/drivers/net/hw/Makefile | 25 ++ .../net/hw}/devlink_port_split.py | 0 .../forwarding => drivers/net/hw}/ethtool.sh | 5 +- .../net/hw}/ethtool_extended_state.sh | 5 +- .../net/hw}/ethtool_lib.sh | 0 .../net/hw}/ethtool_mm.sh | 3 +- .../net/hw}/ethtool_rmon.sh | 7 +- .../net/hw}/hw_stats_l3.sh | 19 +- .../net/hw}/hw_stats_l3_gre.sh | 7 +- .../forwarding => drivers/net/hw}/loopback.sh | 5 +- .../testing/selftests/drivers/net/hw/settings | 1 + .../selftests/drivers/net/mlxsw/mlxsw_lib.sh | 2 +- .../net/mlxsw/spectrum-2/resource_scale.sh | 1 - .../net/mlxsw/spectrum/resource_scale.sh | 1 - tools/testing/selftests/net/Makefile | 1 - .../testing/selftests/net/forwarding/Makefile | 9 +- tools/testing/selftests/net/forwarding/README | 33 +++ .../net/forwarding/forwarding.config.sample | 53 ++-- .../selftests/net/forwarding/ipip_lib.sh | 1 - tools/testing/selftests/net/forwarding/lib.sh | 255 +++++++++++++----- .../selftests/net/forwarding/lib_sh_test.sh | 208 ++++++++++++++ .../net/forwarding/router_mpath_nh_lib.sh | 12 +- .../selftests/net/forwarding/sch_ets_tests.sh | 19 +- .../selftests/net/forwarding/sch_red.sh | 10 +- .../selftests/net/forwarding/sch_tbf_core.sh | 2 +- .../selftests/net/forwarding/tc_common.sh | 2 +- .../selftests/net/forwarding/tc_tunnel_key.sh | 2 - tools/testing/selftests/net/lib.sh | 48 +++- 28 files changed, 565 insertions(+), 171 deletions(-) create mode 100644 tools/testing/selftests/drivers/net/hw/Makefile rename tools/testing/selftests/{net => drivers/net/hw}/devlink_port_split.py (100%) rename tools/testing/selftests/{net/forwarding => drivers/net/hw}/ethtool.sh (98%) rename tools/testing/selftests/{net/forwarding => drivers/net/hw}/ethtool_extended_state.sh (96%) rename tools/testing/selftests/{net/forwarding => drivers/net/hw}/ethtool_lib.sh (100%) rename tools/testing/selftests/{net/forwarding => drivers/net/hw}/ethtool_mm.sh (99%) rename tools/testing/selftests/{net/forwarding => drivers/net/hw}/ethtool_rmon.sh (92%) rename tools/testing/selftests/{net/forwarding => drivers/net/hw}/hw_stats_l3.sh (96%) rename tools/testing/selftests/{net/forwarding => drivers/net/hw}/hw_stats_l3_gre.sh (92%) rename tools/testing/selftests/{net/forwarding => drivers/net/hw}/loopback.sh (92%) create mode 100644 tools/testing/selftests/drivers/net/hw/settings create mode 100755 tools/testing/selftests/net/forwarding/lib_sh_test.sh -- 2.43.0

1 year, 3 months

2
15
0 0

kunit alltests runs broken in mainline

by Mark Brown

Hi, Commit 28b3df1fe6ba2cb4 ("kunit: add wireless unit tests") which I can't seem to find on lore breaks full kunit runs on non-UML builds and is now present in mainline. If I run: ./tools/testing/kunit/kunit.py run --alltests --cross_compile x86_64-linux-gnu- --arch x86_64 on a clean tree then I get: [15:09:20] Configuring KUnit Kernel ... Generating .config ... Populating config with: $ make ARCH=x86_64 O=.kunit olddefconfig CROSS_COMPILE=x86_64-linux-gnu- ERROR:root:Not all Kconfig options selected in kunitconfig were in the generated .config. This is probably due to unsatisfied dependencies. Missing: CONFIG_IWLWIFI=y, CONFIG_WLAN_VENDOR_INTEL=y UML works fine, but other real architectures (eg, arm64) seem similarly broken. I've not looked properly yet, I'm a bit confused given that there's not even any dependencies for WLAN_VENDOR_INTEL and it's not mentoned in the defconfig. Thanks, Mark

1 year, 3 months

5
10
0 0

[RFC PATCH net-next v7 00/14] Device Memory TCP

by Mina Almasry

RFC v7: ======= Major Changes: -------------- This revision largely rebases on top of net-next and addresses the feedback RFCv6 received from folks, namely Jakub, Yunsheng, Arnd, David, & Pavel. The series remains in RFC because the queue-API ndos defined in this series are not yet implemented. I have a GVE implementation I carry out of tree for my testing. A upstreamable GVE implementation is in the works. Aside from that, in my estimation all the patches are ready for review/merge. Please do take a look. As usual the full devmem TCP changes including the full GVE driver implementation is here: https://github.com/mina/linux/commits/tcpdevmem-v7/ Detailed changelog: - Use admin-perm in netlink API. - Addressed feedback from Jakub with regards to netlink API implementation. - Renamed devmem.c functions to something more appropriate for that file. - Improve the performance seen through the page_pool benchmark. - Fix the value definition of all the SO_DEVMEM_* uapi. - Various fixes to documentation. Perf - page-pool benchmark: --------------------------- Improved performance of bench_page_pool_simple.ko tests compared to v6: https://pastebin.com/raw/v5dYRg8L net-next base: 8 cycle fast path. RFC v6: 10 cycle fast path. RFC v7: 9 cycle fast path. RFC v7 with CONFIG_DMA_SHARED_BUFFER disabled: 8 cycle fast path, same as baseline. Perf - Devmem TCP benchmark: --------------------- Perf is about the same regardless of the changes in v7, namely the removal of the static_branch_unlikely to improve the page_pool benchmark performance: 189/200gbps bi-directional throughput with RX devmem TCP and regular TCP TX i.e. ~95% line rate. RFC v6: ======= Major Changes: -------------- This revision largely rebases on top of net-next and addresses the little feedback RFCv5 received. The series remains in RFC because the queue-API ndos defined in this series are not yet implemented. I have a GVE implementation I carry out of tree for my testing. A upstreamable GVE implementation is in the works. Aside from that, in my estimation all the patches are ready for review/merge. Please do take a look. As usual the full devmem TCP changes including the full GVE driver implementation is here: https://github.com/mina/linux/commits/tcpdevmem-v6/ This version also comes with some performance data recorded in the cover letter (see below changelog). Detailed changelog: - Rebased on top of the merged netmem_ref changes. - Converted skb->dmabuf to skb->readable (Pavel). Pavel's original suggestion was to remove the skb->dmabuf flag entirely, but when I looked into it closely, I found the issue that if we remove the flag we have to dereference the shinfo(skb) pointer to obtain the first frag to tell whether an skb is readable or not. This can cause a performance regression if it dirties the cache line when the shinfo(skb) was not really needed. Instead, I converted the skb->dmabuf flag into a generic skb->readable flag which can be re-used by io_uring 0-copy RX. - Squashed a few locking optimizations from Eric Dumazet in the RX path and the DEVMEM_DONTNEED setsockopt. - Expanded the tests a bit. Added validation for invalid scenarios and added some more coverage. Perf - page-pool benchmark: --------------------------- bench_page_pool_simple.ko tests with and without these changes: https://pastebin.com/raw/ncHDwAbn AFAIK the number that really matters in the perf tests is the 'tasklet_page_pool01_fast_path Per elem'. This one measures at about 8 cycles without the changes but there is some 1 cycle noise in some results. With the patches this regresses to 9 cycles with the changes but there is 1 cycle noise occasionally running this test repeatedly. Lastly I tried disable the static_branch_unlikely() in netmem_is_net_iov() check. To my surprise disabling the static_branch_unlikely() check reduces the fast path back to 8 cycles, but the 1 cycle noise remains. Perf - Devmem TCP benchmark: --------------------- 189/200gbps bi-directional throughput with RX devmem TCP and regular TCP TX i.e. ~95% line rate. Major changes in RFC v5: ======================== 1. Rebased on top of 'Abstract page from net stack' series and used the new netmem type to refer to LSB set pointers instead of re-using struct page. 2. Downgraded this series back to RFC and called it RFC v5. This is because this series is now dependent on 'Abstract page from net stack'[1] and the queue API. Both are removed from the series to reduce the patch # and those bits are fairly independent or pre-requisite work. 3. Reworked the page_pool devmem support to use netmem and for some more unified handling. 4. Reworked the reference counting of net_iov (renamed from page_pool_iov) to use pp_ref_count for refcounting. The full changes including the dependent series and GVE page pool support is here: https://github.com/mina/linux/commits/tcpdevmem-rfcv5/ [1] https://patchwork.kernel.org/project/netdevbpf/list/?series=810774 Major changes in v1: ==================== 1. Implemented MVP queue API ndos to remove the userspace-visible driver reset. 2. Fixed issues in the napi_pp_put_page() devmem frag unref path. 3. Removed RFC tag. Many smaller addressed comments across all the patches (patches have individual change log). Full tree including the rest of the GVE driver changes: https://github.com/mina/linux/commits/tcpdevmem-v1 Changes in RFC v3: ================== 1. Pulled in the memory-provider dependency from Jakub's RFC[1] to make the series reviewable and mergeable. 2. Implemented multi-rx-queue binding which was a todo in v2. 3. Fix to cmsg handling. The sticking point in RFC v2[2] was the device reset required to refill the device rx-queues after the dmabuf bind/unbind. The solution suggested as I understand is a subset of the per-queue management ops Jakub suggested or similar: https://lore.kernel.org/netdev/20230815171638.4c057dcd@kernel.org/ This is not addressed in this revision, because: 1. This point was discussed at netconf & netdev and there is openness to using the current approach of requiring a device reset. 2. Implementing individual queue resetting seems to be difficult for my test bed with GVE. My prototype to test this ran into issues with the rx-queues not coming back up properly if reset individually. At the moment I'm unsure if it's a mistake in the POC or a genuine issue in the virtualization stack behind GVE, which currently doesn't test individual rx-queue restart. 3. Our usecases are not bothered by requiring a device reset to refill the buffer queues, and we'd like to support NICs that run into this limitation with resetting individual queues. My thought is that drivers that have trouble with per-queue configs can use the support in this series, while drivers that support new netdev ops to reset individual queues can automatically reset the queue as part of the dma-buf bind/unbind. The same approach with device resets is presented again for consideration with other sticking points addressed. This proposal includes the rx devmem path only proposed for merge. For a snapshot of my entire tree which includes the GVE POC page pool support & device memory support: https://github.com/torvalds/linux/compare/master...mina:linux:tcpdevmem-v3 [1] https://lore.kernel.org/netdev/f8270765-a27b-6ccf-33ea-cda097168d79@redhat.… [2] https://lore.kernel.org/netdev/CAHS8izOVJGJH5WF68OsRWFKJid1_huzzUK+hpKbLcL4… Changes in RFC v2: ================== The sticking point in RFC v1[1] was the dma-buf pages approach we used to deliver the device memory to the TCP stack. RFC v2 is a proof-of-concept that attempts to resolve this by implementing scatterlist support in the networking stack, such that we can import the dma-buf scatterlist directly. This is the approach proposed at a high level here[2]. Detailed changes: 1. Replaced dma-buf pages approach with importing scatterlist into the page pool. 2. Replace the dma-buf pages centric API with a netlink API. 3. Removed the TX path implementation - there is no issue with implementing the TX path with scatterlist approach, but leaving out the TX path makes it easier to review. 4. Functionality is tested with this proposal, but I have not conducted perf testing yet. I'm not sure there are regressions, but I removed perf claims from the cover letter until they can be re-confirmed. 5. Added Signed-off-by: contributors to the implementation. 6. Fixed some bugs with the RX path since RFC v1. Any feedback welcome, but specifically the biggest pending questions needing feedback IMO are: 1. Feedback on the scatterlist-based approach in general. 2. Netlink API (Patch 1 & 2). 3. Approach to handle all the drivers that expect to receive pages from the page pool (Patch 6). [1] https://lore.kernel.org/netdev/dfe4bae7-13a0-3c5d-d671-f61b375cb0b4@gmail.c… [2] https://lore.kernel.org/netdev/CAHS8izPm6XRS54LdCDZVd0C75tA1zHSu6jLVO8nzTLX… ================== * TL;DR: Device memory TCP (devmem TCP) is a proposal for transferring data to and/or from device memory efficiently, without bouncing the data to a host memory buffer. * Problem: A large amount of data transfers have device memory as the source and/or destination. Accelerators drastically increased the volume of such transfers. Some examples include: - ML accelerators transferring large amounts of training data from storage into GPU/TPU memory. In some cases ML training setup time can be as long as 50% of TPU compute time, improving data transfer throughput & efficiency can help improving GPU/TPU utilization. - Distributed training, where ML accelerators, such as GPUs on different hosts, exchange data among them. - Distributed raw block storage applications transfer large amounts of data with remote SSDs, much of this data does not require host processing. Today, the majority of the Device-to-Device data transfers the network are implemented as the following low level operations: Device-to-Host copy, Host-to-Host network transfer, and Host-to-Device copy. The implementation is suboptimal, especially for bulk data transfers, and can put significant strains on system resources, such as host memory bandwidth, PCIe bandwidth, etc. One important reason behind the current state is the kernel’s lack of semantics to express device to network transfers. * Proposal: In this patch series we attempt to optimize this use case by implementing socket APIs that enable the user to: 1. send device memory across the network directly, and 2. receive incoming network packets directly into device memory. Packet _payloads_ go directly from the NIC to device memory for receive and from device memory to NIC for transmit. Packet _headers_ go to/from host memory and are processed by the TCP/IP stack normally. The NIC _must_ support header split to achieve this. Advantages: - Alleviate host memory bandwidth pressure, compared to existing network-transfer + device-copy semantics. - Alleviate PCIe BW pressure, by limiting data transfer to the lowest level of the PCIe tree, compared to traditional path which sends data through the root complex. * Patch overview: ** Part 1: netlink API Gives user ability to bind dma-buf to an RX queue. ** Part 2: scatterlist support Currently the standard for device memory sharing is DMABUF, which doesn't generate struct pages. On the other hand, networking stack (skbs, drivers, and page pool) operate on pages. We have 2 options: 1. Generate struct pages for dmabuf device memory, or, 2. Modify the networking stack to process scatterlist. Approach #1 was attempted in RFC v1. RFC v2 implements approach #2. ** part 3: page pool support We piggy back on page pool memory providers proposal: https://github.com/kuba-moo/linux/tree/pp-providers It allows the page pool to define a memory provider that provides the page allocation and freeing. It helps abstract most of the device memory TCP changes from the driver. ** part 4: support for unreadable skb frags Page pool iovs are not accessible by the host; we implement changes throughput the networking stack to correctly handle skbs with unreadable frags. ** Part 5: recvmsg() APIs We define user APIs for the user to send and receive device memory. Not included with this series is the GVE devmem TCP support, just to simplify the review. Code available here if desired: https://github.com/mina/linux/tree/tcpdevmem This series is built on top of net-next with Jakub's pp-providers changes cherry-picked. * NIC dependencies: 1. (strict) Devmem TCP require the NIC to support header split, i.e. the capability to split incoming packets into a header + payload and to put each into a separate buffer. Devmem TCP works by using device memory for the packet payload, and host memory for the packet headers. 2. (optional) Devmem TCP works better with flow steering support & RSS support, i.e. the NIC's ability to steer flows into certain rx queues. This allows the sysadmin to enable devmem TCP on a subset of the rx queues, and steer devmem TCP traffic onto these queues and non devmem TCP elsewhere. The NIC I have access to with these properties is the GVE with DQO support running in Google Cloud, but any NIC that supports these features would suffice. I may be able to help reviewers bring up devmem TCP on their NICs. * Testing: The series includes a udmabuf kselftest that show a simple use case of devmem TCP and validates the entire data path end to end without a dependency on a specific dmabuf provider. ** Test Setup Kernel: net-next with this series and memory provider API cherry-picked locally. Hardware: Google Cloud A3 VMs. NIC: GVE with header split & RSS & flow steering support. Cc: Pavel Begunkov <asml.silence(a)gmail.com> Cc: David Wei <dw(a)davidwei.uk> Cc: Jason Gunthorpe <jgg(a)ziepe.ca> Cc: Yunsheng Lin <linyunsheng(a)huawei.com> Cc: Shailend Chand <shailend(a)google.com> Cc: Harshitha Ramamurthy <hramamurthy(a)google.com> Cc: Shakeel Butt <shakeel.butt(a)linux.dev> Cc: Jeroen de Borst <jeroendb(a)google.com> Cc: Praveen Kaligineedi <pkaligineedi(a)google.com> Jakub Kicinski (1): net: page_pool: create hooks for custom page providers Mina Almasry (13): queue_api: define queue api net: netdev netlink api to bind dma-buf to a net device netdev: support binding dma-buf to netdevice netdev: netdevice devmem allocator page_pool: convert to use netmem page_pool: devmem support memory-provider: dmabuf devmem memory provider net: support non paged skb frags net: add support for skbs with unreadable frags tcp: RX path for devmem TCP net: add SO_DEVMEM_DONTNEED setsockopt to release RX frags net: add devmem TCP documentation selftests: add ncdevmem, netcat for devmem TCP Documentation/netlink/specs/netdev.yaml | 57 +++ Documentation/networking/devmem.rst | 256 +++++++++++ Documentation/networking/index.rst | 1 + arch/alpha/include/uapi/asm/socket.h | 6 + arch/mips/include/uapi/asm/socket.h | 6 + arch/parisc/include/uapi/asm/socket.h | 6 + arch/sparc/include/uapi/asm/socket.h | 6 + include/linux/netdevice.h | 3 + include/linux/skbuff.h | 73 +++- include/linux/socket.h | 1 + include/net/devmem.h | 124 ++++++ include/net/netdev_queues.h | 27 ++ include/net/netdev_rx_queue.h | 2 + include/net/netmem.h | 234 +++++++++- include/net/page_pool/helpers.h | 155 +++++-- include/net/page_pool/types.h | 33 +- include/net/sock.h | 2 + include/net/tcp.h | 5 +- include/trace/events/page_pool.h | 29 +- include/uapi/asm-generic/socket.h | 6 + include/uapi/linux/netdev.h | 19 + include/uapi/linux/uio.h | 17 + net/bpf/test_run.c | 5 +- net/core/Makefile | 2 +- net/core/datagram.c | 6 + net/core/dev.c | 6 +- net/core/devmem.c | 425 ++++++++++++++++++ net/core/gro.c | 8 +- net/core/netdev-genl-gen.c | 23 + net/core/netdev-genl-gen.h | 6 + net/core/netdev-genl.c | 107 +++++ net/core/page_pool.c | 364 +++++++++------- net/core/skbuff.c | 110 ++++- net/core/sock.c | 61 +++ net/ipv4/esp4.c | 2 +- net/ipv4/tcp.c | 254 ++++++++++- net/ipv4/tcp_input.c | 13 +- net/ipv4/tcp_ipv4.c | 9 + net/ipv4/tcp_minisocks.c | 2 + net/ipv4/tcp_output.c | 5 +- net/ipv6/esp6.c | 2 +- net/packet/af_packet.c | 4 +- tools/include/uapi/linux/netdev.h | 19 + tools/testing/selftests/net/.gitignore | 1 + tools/testing/selftests/net/Makefile | 5 + tools/testing/selftests/net/ncdevmem.c | 546 ++++++++++++++++++++++++ 46 files changed, 2776 insertions(+), 277 deletions(-) create mode 100644 Documentation/networking/devmem.rst create mode 100644 include/net/devmem.h create mode 100644 net/core/devmem.c create mode 100644 tools/testing/selftests/net/ncdevmem.c -- 2.44.0.396.g6e790dbe36-goog

1 year, 3 months

3
20
0 0

[PATCH v5 0/3] selftests/resctrl: Simplify test cleanup functions

by Maciej Wieczor-Retman

Cleaning up after tests is implemented separately for individual tests and called at the end of each test execution. Since these functions are very similar and a more generalized test framework was introduced a function pointer in the resctrl_test struct can be used to reduce the amount of function calls. These functions are also all called in the ctrl-c handler because the handler isn't aware which test is currently running. Since the handler is implemented with a sigaction no function parameters can be passed there but information about what test is currently running can be passed with a global variable. Series applies cleanly on top of kselftests/next. Changelog v5: - Rebase onto kselftests/next. - Add Reinette's reviewed-by tag. Changelog v4: - Check current_test pointer and reset it in signal unregistering. - Move cleanup call to test_cleanup function. Changelog v3: - Make current_test static. - Add callback NULL check to the ctrl-c handler. Changelog v2: - Make current_test a const pointer limited in scope to resctrl_val file. - Remove tests_cleanup from resctrl.h. - Cleanup 'goto out' path and labels in individual test functions. Older versions of this series: [v1] https://lore.kernel.org/all/cover.1708434017.git.maciej.wieczor-retman@inte… [v2] https://lore.kernel.org/all/cover.1708596015.git.maciej.wieczor-retman@inte… [v3] https://lore.kernel.org/all/cover.1708599491.git.maciej.wieczor-retman@inte… [v4] https://lore.kernel.org/all/cover.1708949785.git.maciej.wieczor-retman@inte… Maciej Wieczor-Retman (3): selftests/resctrl: Add cleanup function to test framework selftests/resctrl: Simplify cleanup in ctrl-c handler selftests/resctrl: Move cleanups out of individual tests tools/testing/selftests/resctrl/cat_test.c | 8 +++----- tools/testing/selftests/resctrl/cmt_test.c | 4 ++-- tools/testing/selftests/resctrl/mba_test.c | 8 +++----- tools/testing/selftests/resctrl/mbm_test.c | 8 +++----- tools/testing/selftests/resctrl/resctrl.h | 9 +++------ .../testing/selftests/resctrl/resctrl_tests.c | 20 +++++++------------ tools/testing/selftests/resctrl/resctrl_val.c | 8 ++++++-- 7 files changed, 27 insertions(+), 38 deletions(-) -- 2.43.2

1 year, 3 months

4
8
0 0

[PATCH v2 1/3] selftests: x86: test_vsyscall: reorder code to reduce #ifdef blocks

by Muhammad Usama Anjum

There are multiple #ifdef blocks inside functions where they return just 0 if #ifdef is false. This makes number of tests counting difficult. Move those functions inside one #ifdef block and move all of them together. This is preparatory patch for next patch to convert this into TAP format. So in this patch, we are just moving functions around without any changes. With and without this patch, the output of this patch is same. Signed-off-by: Muhammad Usama Anjum <usama.anjum(a)collabora.com> --- tools/testing/selftests/x86/test_vsyscall.c | 176 +++++++++----------- 1 file changed, 83 insertions(+), 93 deletions(-) diff --git a/tools/testing/selftests/x86/test_vsyscall.c b/tools/testing/selftests/x86/test_vsyscall.c index 47cab972807c4..c78b0648aa524 100644 --- a/tools/testing/selftests/x86/test_vsyscall.c +++ b/tools/testing/selftests/x86/test_vsyscall.c @@ -21,6 +21,7 @@ #include <sys/uio.h> #include "helpers.h" +#include "../kselftest.h" #ifdef __x86_64__ # define VSYS(x) (x) @@ -39,18 +40,6 @@ /* max length of lines in /proc/self/maps - anything longer is skipped here */ #define MAPS_LINE_LEN 128 -static void sethandler(int sig, void (*handler)(int, siginfo_t *, void *), - int flags) -{ - struct sigaction sa; - memset(&sa, 0, sizeof(sa)); - sa.sa_sigaction = handler; - sa.sa_flags = SA_SIGINFO | flags; - sigemptyset(&sa.sa_mask); - if (sigaction(sig, &sa, 0)) - err(1, "sigaction"); -} - /* vsyscalls and vDSO */ bool vsyscall_map_r = false, vsyscall_map_x = false; @@ -96,64 +85,6 @@ static void init_vdso(void) printf("[WARN]\tfailed to find getcpu in vDSO\n"); } -static int init_vsys(void) -{ -#ifdef __x86_64__ - int nerrs = 0; - FILE *maps; - char line[MAPS_LINE_LEN]; - bool found = false; - - maps = fopen("/proc/self/maps", "r"); - if (!maps) { - printf("[WARN]\tCould not open /proc/self/maps -- assuming vsyscall is r-x\n"); - vsyscall_map_r = true; - return 0; - } - - while (fgets(line, MAPS_LINE_LEN, maps)) { - char r, x; - void *start, *end; - char name[MAPS_LINE_LEN]; - - /* sscanf() is safe here as strlen(name) >= strlen(line) */ - if (sscanf(line, "%p-%p %c-%cp %*x %*x:%*x %*u %s", - &start, &end, &r, &x, name) != 5) - continue; - - if (strcmp(name, "[vsyscall]")) - continue; - - printf("\tvsyscall map: %s", line); - - if (start != (void *)0xffffffffff600000 || - end != (void *)0xffffffffff601000) { - printf("[FAIL]\taddress range is nonsense\n"); - nerrs++; - } - - printf("\tvsyscall permissions are %c-%c\n", r, x); - vsyscall_map_r = (r == 'r'); - vsyscall_map_x = (x == 'x'); - - found = true; - break; - } - - fclose(maps); - - if (!found) { - printf("\tno vsyscall map in /proc/self/maps\n"); - vsyscall_map_r = false; - vsyscall_map_x = false; - } - - return nerrs; -#else - return 0; -#endif -} - /* syscalls */ static inline long sys_gtod(struct timeval *tv, struct timezone *tz) { @@ -176,17 +107,6 @@ static inline long sys_getcpu(unsigned * cpu, unsigned * node, return syscall(SYS_getcpu, cpu, node, cache); } -static jmp_buf jmpbuf; -static volatile unsigned long segv_err; - -static void sigsegv(int sig, siginfo_t *info, void *ctx_void) -{ - ucontext_t *ctx = (ucontext_t *)ctx_void; - - segv_err = ctx->uc_mcontext.gregs[REG_ERR]; - siglongjmp(jmpbuf, 1); -} - static double tv_diff(const struct timeval *a, const struct timeval *b) { return (double)(a->tv_sec - b->tv_sec) + @@ -396,9 +316,33 @@ static int test_getcpu(int cpu) return nerrs; } +#ifdef __x86_64__ + +static jmp_buf jmpbuf; +static volatile unsigned long segv_err; + +static void sethandler(int sig, void (*handler)(int, siginfo_t *, void *), + int flags) +{ + struct sigaction sa; + memset(&sa, 0, sizeof(sa)); + sa.sa_sigaction = handler; + sa.sa_flags = SA_SIGINFO | flags; + sigemptyset(&sa.sa_mask); + if (sigaction(sig, &sa, 0)) + err(1, "sigaction"); +} + +static void sigsegv(int sig, siginfo_t *info, void *ctx_void) +{ + ucontext_t *ctx = (ucontext_t *)ctx_void; + + segv_err = ctx->uc_mcontext.gregs[REG_ERR]; + siglongjmp(jmpbuf, 1); +} + static int test_vsys_r(void) { -#ifdef __x86_64__ printf("[RUN]\tChecking read access to the vsyscall page\n"); bool can_read; if (sigsetjmp(jmpbuf, 1) == 0) { @@ -420,14 +364,12 @@ static int test_vsys_r(void) printf("[OK]\tWe do not have read access: #PF(0x%lx)\n", segv_err); } -#endif return 0; } static int test_vsys_x(void) { -#ifdef __x86_64__ if (vsyscall_map_x) { /* We already tested this adequately. */ return 0; @@ -454,8 +396,6 @@ static int test_vsys_x(void) segv_err); return 1; } -#endif - return 0; } @@ -472,7 +412,6 @@ static int test_vsys_x(void) */ static int test_process_vm_readv(void) { -#ifdef __x86_64__ char buf[4096]; struct iovec local, remote; int ret; @@ -504,12 +443,63 @@ static int test_process_vm_readv(void) printf("[FAIL]\tprocess_rm_readv() succeeded, but it should have failed in this configuration\n"); return 1; } -#endif - return 0; } -#ifdef __x86_64__ +static int init_vsys(void) +{ + int nerrs = 0; + FILE *maps; + char line[MAPS_LINE_LEN]; + bool found = false; + + maps = fopen("/proc/self/maps", "r"); + if (!maps) { + printf("[WARN]\tCould not open /proc/self/maps -- assuming vsyscall is r-x\n"); + vsyscall_map_r = true; + return 0; + } + + while (fgets(line, MAPS_LINE_LEN, maps)) { + char r, x; + void *start, *end; + char name[MAPS_LINE_LEN]; + + /* sscanf() is safe here as strlen(name) >= strlen(line) */ + if (sscanf(line, "%p-%p %c-%cp %*x %*x:%*x %*u %s", + &start, &end, &r, &x, name) != 5) + continue; + + if (strcmp(name, "[vsyscall]")) + continue; + + printf("\tvsyscall map: %s", line); + + if (start != (void *)0xffffffffff600000 || + end != (void *)0xffffffffff601000) { + printf("[FAIL]\taddress range is nonsense\n"); + nerrs++; + } + + printf("\tvsyscall permissions are %c-%c\n", r, x); + vsyscall_map_r = (r == 'r'); + vsyscall_map_x = (x == 'x'); + + found = true; + break; + } + + fclose(maps); + + if (!found) { + printf("\tno vsyscall map in /proc/self/maps\n"); + vsyscall_map_r = false; + vsyscall_map_x = false; + } + + return nerrs; +} + static volatile sig_atomic_t num_vsyscall_traps; static void sigtrap(int sig, siginfo_t *info, void *ctx_void) @@ -559,20 +549,20 @@ int main(int argc, char **argv) int nerrs = 0; init_vdso(); +#ifdef __x86_64__ nerrs += init_vsys(); +#endif nerrs += test_gtod(); nerrs += test_time(); nerrs += test_getcpu(0); nerrs += test_getcpu(1); +#ifdef __x86_64__ sethandler(SIGSEGV, sigsegv, 0); nerrs += test_vsys_r(); nerrs += test_vsys_x(); - nerrs += test_process_vm_readv(); - -#ifdef __x86_64__ nerrs += test_emulation(); #endif -- 2.39.2

1 year, 3 months

2
3
0 0

[PATCH] kunit: configs: Enable CONFIG_DAMON_DBGFS_DEPRECATED for --alltests

by David Gow

This is required, as CONFIG_DAMON_DEBUGFS is enabled, and --alltests UML builds will fail due to the missing config option otherwise. Fixes: f4cba4bf6777 ("mm/damon: rename CONFIG_DAMON_DBGFS to DAMON_DBGFS_DEPRECATED") Signed-off-by: David Gow <davidgow(a)google.com> --- This is breaking all UML alltests builds, so we'd like to fix it sooner rather than later. SeongJae, would you rather take this yourself, or can we push it alongside any other KUnit fixes? Johannes: Does this conflict with the CONFIG_NETDEVICES / CONFIG_WLAN fixes to all_tests.config? I'd assume not, but I'm happy to take them via KUnit if you'd prefer anyway. Thanks, -- David --- tools/testing/kunit/configs/all_tests.config | 1 + 1 file changed, 1 insertion(+) diff --git a/tools/testing/kunit/configs/all_tests.config b/tools/testing/kunit/configs/all_tests.config index aa5ec149f96c..f388742cf266 100644 --- a/tools/testing/kunit/configs/all_tests.config +++ b/tools/testing/kunit/configs/all_tests.config @@ -38,6 +38,7 @@ CONFIG_DAMON_VADDR=y CONFIG_DAMON_PADDR=y CONFIG_DEBUG_FS=y CONFIG_DAMON_DBGFS=y +CONFIG_DAMON_DBGFS_DEPRECATED=y CONFIG_REGMAP_BUILD=y -- 2.44.0.396.g6e790dbe36-goog

1 year, 3 months

4
3
0 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

Linux-kselftest-mirror