May 2023 - Linux-kselftest-mirror

[PATCH] selftests/mincore: fix skip condition for check_huge_pages test

by Ricardo Cañuelo

The check_huge_pages test was failing instead of skipping on qemu-armv7 because the skip condition wasn't handled properly. Add an additional check to fix it. Signed-off-by: Ricardo Cañuelo <ricardo.canuelo(a)collabora.com> Reported-by: Naresh Kamboju <naresh.kamboju(a)linaro.org> Reported-by: Linux Kernel Functional Testing <lkft(a)linaro.org> Closes: https://lore.kernel.org/all/CA+G9fYuoB8Ug8PcTU-YGmemL7_eeEksXFihvxWF6OikD7s… --- tools/testing/selftests/mincore/mincore_selftest.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/mincore/mincore_selftest.c b/tools/testing/selftests/mincore/mincore_selftest.c index 4c88238fc8f0..6fb3eea5b6ee 100644 --- a/tools/testing/selftests/mincore/mincore_selftest.c +++ b/tools/testing/selftests/mincore/mincore_selftest.c @@ -150,8 +149,8 @@ TEST(check_huge_pages) MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB, -1, 0); if (addr == MAP_FAILED) { - if (errno == ENOMEM) - SKIP(return, "No huge pages available."); + if (errno == ENOMEM || errno == EINVAL) + SKIP(return, "No huge pages available or CONFIG_HUGETLB_PAGE disabled."); else TH_LOG("mmap error: %s", strerror(errno)); } -- 2.25.1

1 year, 6 months

3
3
0 0

[PATCH 0/4] selftests/nolibc: add user-space 'efault' handler

by Zhangjin Wu

Hi, Willy, Thomas This is not really for merge, but only let it work as a demo code to test whether it is possible to restore the next test when there is a bad pointer access in user-space [1]. Besides, a new 'run' command is added to 'NOLIBC_TEST' environment variable or arguments to control the running iterations, this may be used to test the reentrancy issues, but no failures found currently ;-) With glibc, it works as following: $ ./nolibc-test run:2,syscall:28-30,stdlib:1 Running iteration(s): 2 Current iteration: 1 Running test 'syscall', from 28 to 30 28 dup3_m1 = -1 EBADF [OK] 29 efault_handler ! 11 SIGSEGV [OK] 30 execve_root = -1 EACCES [OK] Errors during this test: 0 Running test 'stdlib' 1 getenv_blah = <(null)> [OK] Errors during this test: 0 Total number of errors in the 1 iteration(s): 0 Current iteration: 2 Running test 'syscall' 28 dup3_m1 = -1 EBADF [OK] 29 efault_handler ! 11 SIGSEGV [OK] 30 execve_root = -1 EACCES [OK] Errors during this test: 0 Running test 'stdlib' 1 getenv_blah = <(null)> [OK] Errors during this test: 0 Total number of errors in the 2 iteration(s): 0 With nolibc, it will be skipped (run:2,syscall:28-30,stdlib:10): Running iteration(s): 2 Current iteration: 1 Running test 'syscall', from 28 to 30 28 dup3_m1 = -1 EBADF [OK] 29 efault_handler [SKIPPED] 30 execve_root = -1 EACCES [OK] Errors during this test: 0 Running test 'stdlib', from 10 to 10 10 strrchr_foobar_o = <obar> [OK] Errors during this test: 0 Total number of errors in the 1 iteration(s): 0 Current iteration: 2 Running test 'syscall', from 28 to 30 28 dup3_m1 = -1 EBADF [OK] 29 efault_handler [SKIPPED] 30 execve_root = -1 EACCES [OK] Errors during this test: 0 Running test 'stdlib', from 10 to 10 10 strrchr_foobar_o = <obar> [OK] Errors during this test: 0 Total number of errors in the 2 iteration(s): 0 Best regards, Zhangjin --- [1]: https://lore.kernel.org/linux-riscv/20230529113143.GB2762@1wt.eu/ Zhangjin Wu (4): selftests/nolibc: allow rerun with the same settings selftests/nolibc: add rerun support selftests/nolibc: add user space efault handler selftests/nolibc: add user-space efault restore test case tools/testing/selftests/nolibc/nolibc-test.c | 247 +++++++++++++++++-- 1 file changed, 221 insertions(+), 26 deletions(-) -- 2.25.1

1 year, 6 months

3
8
0 0

[PATCH] tracing/selftests: Update synthetic event selftest to use common_stacktrace

by Steven Rostedt

From: "Steven Rostedt (Google)" <rostedt(a)goodmis.org> With the rename of the stacktrace field to common_stacktrace, update the selftests to reflect this change. Copy the current selftest to test the backward compatibility "stacktrace" keyword. Also the "requires" of that test was incorrect, so it would never actually ran before. That is fixed now. Signed-off-by: Steven Rostedt (Google) <rostedt(a)goodmis.org> --- .../trigger-synthetic-event-stack-legacy.tc | 24 +++++++++++++++++++ .../trigger-synthetic-event-stack.tc | 5 ++-- 2 files changed, 26 insertions(+), 3 deletions(-) create mode 100644 tools/testing/selftests/ftrace/test.d/trigger/inter-event/trigger-synthetic-event-stack-legacy.tc diff --git a/tools/testing/selftests/ftrace/test.d/trigger/inter-event/trigger-synthetic-event-stack-legacy.tc b/tools/testing/selftests/ftrace/test.d/trigger/inter-event/trigger-synthetic-event-stack-legacy.tc new file mode 100644 index 000000000000..d0cd91a93069 --- /dev/null +++ b/tools/testing/selftests/ftrace/test.d/trigger/inter-event/trigger-synthetic-event-stack-legacy.tc @@ -0,0 +1,24 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0 +# description: event trigger - test inter-event histogram trigger trace action with dynamic string param (legacy stack) +# requires: set_event synthetic_events events/sched/sched_process_exec/hist "long[] stack' >> synthetic_events":README + +fail() { #msg + echo $1 + exit_fail +} + +echo "Test create synthetic event with stack" + +# Test the old stacktrace keyword (for backward compatibility) +echo 's:wake_lat pid_t pid; u64 delta; unsigned long[] stack;' > dynamic_events +echo 'hist:keys=next_pid:ts=common_timestamp.usecs,st=stacktrace if prev_state == 1||prev_state == 2' >> events/sched/sched_switch/trigger +echo 'hist:keys=prev_pid:delta=common_timestamp.usecs-$ts,s=$st:onmax($delta).trace(wake_lat,prev_pid,$delta,$s)' >> events/sched/sched_switch/trigger +echo 1 > events/synthetic/wake_lat/enable +sleep 1 + +if ! grep -q "=>.*sched" trace; then + fail "Failed to create synthetic event with stack" +fi + +exit 0 diff --git a/tools/testing/selftests/ftrace/test.d/trigger/inter-event/trigger-synthetic-event-stack.tc b/tools/testing/selftests/ftrace/test.d/trigger/inter-event/trigger-synthetic-event-stack.tc index 755dbe94ccf4..8f1cc9a86a06 100644 --- a/tools/testing/selftests/ftrace/test.d/trigger/inter-event/trigger-synthetic-event-stack.tc +++ b/tools/testing/selftests/ftrace/test.d/trigger/inter-event/trigger-synthetic-event-stack.tc @@ -1,7 +1,7 @@ #!/bin/sh # SPDX-License-Identifier: GPL-2.0 # description: event trigger - test inter-event histogram trigger trace action with dynamic string param -# requires: set_event synthetic_events events/sched/sched_process_exec/hist "long[]' >> synthetic_events":README +# requires: set_event synthetic_events events/sched/sched_process_exec/hist "can be any field, or the special string 'common_stacktrace'":README fail() { #msg echo $1 @@ -10,9 +10,8 @@ fail() { #msg echo "Test create synthetic event with stack" - echo 's:wake_lat pid_t pid; u64 delta; unsigned long[] stack;' > dynamic_events -echo 'hist:keys=next_pid:ts=common_timestamp.usecs,st=stacktrace if prev_state == 1||prev_state == 2' >> events/sched/sched_switch/trigger +echo 'hist:keys=next_pid:ts=common_timestamp.usecs,st=common_stacktrace if prev_state == 1||prev_state == 2' >> events/sched/sched_switch/trigger echo 'hist:keys=prev_pid:delta=common_timestamp.usecs-$ts,s=$st:onmax($delta).trace(wake_lat,prev_pid,$delta,$s)' >> events/sched/sched_switch/trigger echo 1 > events/synthetic/wake_lat/enable sleep 1 -- 2.39.2

1 year, 6 months

3
2
0 0

Re: [Linux-kernel-mentorship] [PATCH] selftest: pidfd: Omit long and repeating outputs

by Christian Brauner

On Tue, May 23, 2023 at 11:22:07PM +0000, Ziqi Zhao wrote: > An output message: > > > # # waitpid WEXITSTATUS=0 > > will be printed for 30,000+ times in the `pidfd_test` selftest, which > does not seem ideal. This patch removes the print logic in the > `wait_for_pid` function, so each call to this function does not output > a line by default. Any existing call sites where the extra line might > be beneficial have been modified to include extra print statements > outside of the function calls. > > Signed-off-by: Ziqi Zhao <astrajoan(a)yahoo.com> > --- Fine by me, Reviewed-by: Christian Brauner <brauner(a)kernel.org>

1 year, 6 months

2
1
0 0

[PATCH] selftests: allow runners to override the timeout

by Luis Chamberlain

The default timeout for selftests tests is 45 seconds. Although we already have 13 settings for tests of about 96 sefltests which use a timeout greater than this, we want to try to avoid encouraging more tests to forcing a higher test timeout as selftests strives to run all tests quickly. Selftests also uses the timeout as a non-fatal error. Only tests runners which have control over a system would know if to treat a timeout as fatal or not. To help with all this: o Enhance documentation to avoid future increases of insane timeouts o Add the option to allow overriding the default timeout with test runners with a command line option Suggested-by: Shuah Khan <skhan(a)linuxfoundation.org> Signed-off-by: Luis Chamberlain <mcgrof(a)kernel.org> --- Documentation/dev-tools/kselftest.rst | 22 +++++++++++++++++++++ tools/testing/selftests/kselftest/runner.sh | 11 ++++++++++- tools/testing/selftests/run_kselftest.sh | 5 +++++ 3 files changed, 37 insertions(+), 1 deletion(-) diff --git a/Documentation/dev-tools/kselftest.rst b/Documentation/dev-tools/kselftest.rst index 12b575b76b20..dd214af7b7ff 100644 --- a/Documentation/dev-tools/kselftest.rst +++ b/Documentation/dev-tools/kselftest.rst @@ -168,6 +168,28 @@ the `-t` option for specific single tests. Either can be used multiple times:: For other features see the script usage output, seen with the `-h` option. +Timeout for selftests +===================== + +Selftests are designed to be quick and so a default timeout is used of 45 +seconds for each test. Tests can override the default timeout by adding +a settings file in their directory and set a timeout variable there to the +configured a desired upper timeout for the test. Only a few tests override +the timeout with a value higher than 45 seconds, selftests strives to keep +it that way. Timeouts in selftests are not considered fatal because the +system under which a test runs may change and this can also modify the +expected time it takes to run a test. If you have control over the systems +which will run the tests you can configure a test runner on those systems to +use a greater or lower timeout on the command line as with the `-o` or +the `--override-timeout` argument. For example to use 165 seconds instead +one would use: + + $ ./run_kselftest.sh --override-timeout 165 + +You can look at the TAP output to see if you ran into the timeout. Test +runners which know a test must run under a specific time can then optionally +treat these timeouts then as fatal. + Packaging selftests =================== diff --git a/tools/testing/selftests/kselftest/runner.sh b/tools/testing/selftests/kselftest/runner.sh index 294619ade49f..1c952d1401d4 100644 --- a/tools/testing/selftests/kselftest/runner.sh +++ b/tools/testing/selftests/kselftest/runner.sh @@ -8,7 +8,8 @@ export logfile=/dev/stdout export per_test_logging= # Defaults for "settings" file fields: -# "timeout" how many seconds to let each test run before failing. +# "timeout" how many seconds to let each test run before running +# over our soft timeout limit. export kselftest_default_timeout=45 # There isn't a shell-agnostic way to find the path of a sourced file, @@ -90,6 +91,14 @@ run_one() done < "$settings" fi + # Command line timeout overrides the settings file + if [ -n "$kselftest_override_timeout" ]; then + kselftest_timeout="$kselftest_override_timeout" + echo "# overriding timeout to $kselftest_timeout" >> "$logfile" + else + echo "# timeout set to $kselftest_timeout" >> "$logfile" + fi + TEST_HDR_MSG="selftests: $DIR: $BASENAME_TEST" echo "# $TEST_HDR_MSG" if [ ! -e "$TEST" ]; then diff --git a/tools/testing/selftests/run_kselftest.sh b/tools/testing/selftests/run_kselftest.sh index 97165a83df63..9a981b36bd7f 100755 --- a/tools/testing/selftests/run_kselftest.sh +++ b/tools/testing/selftests/run_kselftest.sh @@ -26,6 +26,7 @@ Usage: $0 [OPTIONS] -l | --list List the available collection:test entries -d | --dry-run Don't actually run any tests -h | --help Show this usage info + -o | --override-timeout Number of seconds after which we timeout EOF exit $1 } @@ -33,6 +34,7 @@ EOF COLLECTIONS="" TESTS="" dryrun="" +kselftest_override_timeout="" while true; do case "$1" in -s | --summary) @@ -51,6 +53,9 @@ while true; do -d | --dry-run) dryrun="echo" shift ;; + -o | --override-timeout) + kselftest_override_timeout="$2" + shift 2 ;; -h | --help) usage 0 ;; "") -- 2.39.2

1 year, 6 months

3
4
0 0

[PATCH v1 1/1] selftests: alsa: pcm-test: Fix compiler warnings about the format

by Mirsad Goran Todorovac

GCC 11.3.0 issues warnings in this module about wrong sizes of format specifiers: pcm-test.c: In function ‘test_pcm_time’: pcm-test.c:384:68: warning: format ‘%ld’ expects argument of type ‘long int’, but argument 5 \ has type ‘unsigned int’ [-Wformat=] 384 | snprintf(msg, sizeof(msg), "rate mismatch %ld != %ld", rate, rrate); pcm-test.c:455:53: warning: format ‘%d’ expects argument of type ‘int’, but argument 4 has \ type ‘long int’ [-Wformat=] 455 | "expected %d, wrote %li", rate, frames); pcm-test.c:462:53: warning: format ‘%d’ expects argument of type ‘int’, but argument 4 has \ type ‘long int’ [-Wformat=] 462 | "expected %d, wrote %li", rate, frames); pcm-test.c:467:53: warning: format ‘%d’ expects argument of type ‘int’, but argument 4 has \ type ‘long int’ [-Wformat=] 467 | "expected %d, wrote %li", rate, frames); Simple fix according to compiler's suggestion removed the warnings. Signed-off-by: Mirsad Goran Todorovac <mirsad.todorovac(a)alu.unizg.hr> --- tools/testing/selftests/alsa/pcm-test.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/tools/testing/selftests/alsa/pcm-test.c b/tools/testing/selftests/alsa/pcm-test.c index 3e390fe67eb9..b7eef32addb4 100644 --- a/tools/testing/selftests/alsa/pcm-test.c +++ b/tools/testing/selftests/alsa/pcm-test.c @@ -381,7 +381,7 @@ static void test_pcm_time(struct pcm_data *data, enum test_class class, goto __close; } if (rrate != rate) { - snprintf(msg, sizeof(msg), "rate mismatch %ld != %ld", rate, rrate); + snprintf(msg, sizeof(msg), "rate mismatch %ld != %d", rate, rrate); goto __close; } rperiod_size = period_size; @@ -447,24 +447,24 @@ static void test_pcm_time(struct pcm_data *data, enum test_class class, frames = snd_pcm_writei(handle, samples, rate); if (frames < 0) { snprintf(msg, sizeof(msg), - "Write failed: expected %d, wrote %li", rate, frames); + "Write failed: expected %ld, wrote %li", rate, frames); goto __close; } if (frames < rate) { snprintf(msg, sizeof(msg), - "expected %d, wrote %li", rate, frames); + "expected %ld, wrote %li", rate, frames); goto __close; } } else { frames = snd_pcm_readi(handle, samples, rate); if (frames < 0) { snprintf(msg, sizeof(msg), - "expected %d, wrote %li", rate, frames); + "expected %ld, wrote %li", rate, frames); goto __close; } if (frames < rate) { snprintf(msg, sizeof(msg), - "expected %d, wrote %li", rate, frames); + "expected %ld, wrote %li", rate, frames); goto __close; } } -- 2.34.1

1 year, 6 months

3
3
0 0

[PATCH] tools/nolibc: ensure fast64 integer types have 64 bits

by Thomas Weißschuh

On 32bit platforms size_t is not enough to represent [u]int_fast64_t. Fixes: 3e9fd4e9a1d5 ("tools/nolibc: add integer types and integer limit macros") Signed-off-by: Thomas Weißschuh <linux(a)weissschuh.net> --- Cc: Vincent Dagonneau <v(a)vda.io> Note: We could also fall back to compiler-provided data like: __UINT_FAST{8,16,32,64}_{TYPE,MIN,MAX}__ --- tools/include/nolibc/stdint.h | 4 ++-- tools/testing/selftests/nolibc/nolibc-test.c | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/tools/include/nolibc/stdint.h b/tools/include/nolibc/stdint.h index c1ce4f5e0603..3fc418cfc3d7 100644 --- a/tools/include/nolibc/stdint.h +++ b/tools/include/nolibc/stdint.h @@ -36,8 +36,8 @@ typedef ssize_t int_fast16_t; typedef size_t uint_fast16_t; typedef ssize_t int_fast32_t; typedef size_t uint_fast32_t; -typedef ssize_t int_fast64_t; -typedef size_t uint_fast64_t; +typedef int64_t int_fast64_t; +typedef uint64_t uint_fast64_t; typedef int64_t intmax_t; typedef uint64_t uintmax_t; diff --git a/tools/testing/selftests/nolibc/nolibc-test.c b/tools/testing/selftests/nolibc/nolibc-test.c index 7de46305f419..65be0317d184 100644 --- a/tools/testing/selftests/nolibc/nolibc-test.c +++ b/tools/testing/selftests/nolibc/nolibc-test.c @@ -696,9 +696,9 @@ int run_stdlib(int min, int max) CASE_TEST(limit_int_fast32_min); EXPECT_EQ(1, INT_FAST32_MIN, (int_fast32_t) INTPTR_MIN); break; CASE_TEST(limit_int_fast32_max); EXPECT_EQ(1, INT_FAST32_MAX, (int_fast32_t) INTPTR_MAX); break; CASE_TEST(limit_uint_fast32_max); EXPECT_EQ(1, UINT_FAST32_MAX, (uint_fast32_t) UINTPTR_MAX); break; - CASE_TEST(limit_int_fast64_min); EXPECT_EQ(1, INT_FAST64_MIN, (int_fast64_t) INTPTR_MIN); break; - CASE_TEST(limit_int_fast64_max); EXPECT_EQ(1, INT_FAST64_MAX, (int_fast64_t) INTPTR_MAX); break; - CASE_TEST(limit_uint_fast64_max); EXPECT_EQ(1, UINT_FAST64_MAX, (uint_fast64_t) UINTPTR_MAX); break; + CASE_TEST(limit_int_fast64_min); EXPECT_EQ(1, INT_FAST64_MIN, (int_fast64_t) INT64_MIN); break; + CASE_TEST(limit_int_fast64_max); EXPECT_EQ(1, INT_FAST64_MAX, (int_fast64_t) INT64_MAX); break; + CASE_TEST(limit_uint_fast64_max); EXPECT_EQ(1, UINT_FAST64_MAX, (uint_fast64_t) UINT64_MAX); break; #if __SIZEOF_LONG__ == 8 CASE_TEST(limit_intptr_min); EXPECT_EQ(1, INTPTR_MIN, (intptr_t) 0x8000000000000000LL); break; CASE_TEST(limit_intptr_max); EXPECT_EQ(1, INTPTR_MAX, (intptr_t) 0x7fffffffffffffffLL); break; --- base-commit: 5b21219d67d3483144d10332709d0c04f733ab93 change-id: 20230530-nolibc-fast64-8777cdbc7e01 Best regards, -- Thomas Weißschuh <linux(a)weissschuh.net>

1 year, 6 months

3
3
0 0

[PATCH] selftests/nolibc: test_fork: fix up duplicated print

by Zhangjin Wu

running nolibc-test with glibc on x86_64 got such print issue: 29 execve_root = -1 EACCES [OK] 30 fork30 fork = 0 [OK] 31 getdents64_root = 712 [OK] The fork test case has three printf calls: (1) llen += printf("%d %s", test, #name); (2) llen += printf(" = %d %s ", expr, errorname(errno)); (3) llen += pad_spc(llen, 64, "[FAIL]\n"); --> vfprintf() In the following scene, the above issue happens: (a) The parent calls (1) (b) The parent calls fork() (c) The child runs and shares the print buffer of (1) (d) The child exits, flushs the print buffer and closes its own stdout/stderr * "30 fork" is printed at the first time. (e) The parent calls (2) and (3), with "\n" in (3), it flushs the whole buffer * "30 fork = 0 ..." is printed Therefore, there are two "30 fork" in the stdout. Between (a) and (b), if flush the stdout (and the sterr), the child in stage (c) will not be able to 'see' the print buffer. Signed-off-by: Zhangjin Wu <falcon(a)tinylab.org> --- tools/testing/selftests/nolibc/nolibc-test.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/nolibc/nolibc-test.c b/tools/testing/selftests/nolibc/nolibc-test.c index 7de46305f419..88323a60aa4a 100644 --- a/tools/testing/selftests/nolibc/nolibc-test.c +++ b/tools/testing/selftests/nolibc/nolibc-test.c @@ -486,7 +486,13 @@ static int test_getpagesize(void) static int test_fork(void) { int status; - pid_t pid = fork(); + pid_t pid; + + /* flush the printf buffer to avoid child flush it */ + fflush(stdout); + fflush(stderr); + + pid = fork(); switch (pid) { case -1: -- 2.25.1

1 year, 6 months

3
3
0 0

[PATCH 0/2] drivers: base: Add tests showing devm handling inconsistencies

by Maxime Ripard

Hi, This follows the discussion here: https://lore.kernel.org/linux-kselftest/20230324123157.bbwvfq4gsxnlnfwb@hou… This shows a couple of inconsistencies with regard to how device-managed resources are cleaned up. Basically, devm resources will only be cleaned up if the device is attached to a bus and bound to a driver. Failing any of these cases, a call to device_unregister will not end up in the devm resources being released. We had to work around it in DRM to provide helpers to create a device for kunit tests, but the current discussion around creating similar, generic, helpers for kunit resumed interest in fixing this. This can be tested using the command: ./tools/testing/kunit/kunit.py run --kunitconfig=drivers/base/test/ Let me know what you think, Maxime Signed-off-by: Maxime Ripard <maxime(a)cerno.tech> --- Maxime Ripard (2): drivers: base: Add basic devm tests for root devices drivers: base: Add basic devm tests for platform devices drivers/base/test/.kunitconfig | 2 + drivers/base/test/Kconfig | 4 + drivers/base/test/Makefile | 3 + drivers/base/test/platform-device-test.c | 278 +++++++++++++++++++++++++++++++ drivers/base/test/root-device-test.c | 120 +++++++++++++ 5 files changed, 407 insertions(+) --- base-commit: a6faf7ea9fcb7267d06116d4188947f26e00e57e change-id: 20230329-kunit-devm-inconsistencies-test-5e5a7d01e60d Best regards, -- Maxime Ripard <maxime(a)cerno.tech>

1 year, 6 months

3
8
0 0

[PATCH v16 0/5] Implement IOCTL to get and optionally clear info about PTEs

by Muhammad Usama Anjum

*Changes in v16* - Fix a corner case - Add exclusive PM_SCAN_OP_WP back *Changes in v15* - Build fix (Add missed build fix in RESEND) *Changes in v14* - Fix build error caused by #ifdef added at last minute in some configs *Changes in v13* - Rebase on top of next-20230414 - Give-up on using uffd_wp_range() and write new helpers, flush tlb only once *Changes in v12* - Update and other memory types to UFFD_FEATURE_WP_ASYNC - Rebaase on top of next-20230406 - Review updates *Changes in v11* - Rebase on top of next-20230307 - Base patches on UFFD_FEATURE_WP_UNPOPULATED - Do a lot of cosmetic changes and review updates - Remove ENGAGE_WP + !GET operation as it can be performed with UFFDIO_WRITEPROTECT *Changes in v10* - Add specific condition to return error if hugetlb is used with wp async - Move changes in tools/include/uapi/linux/fs.h to separate patch - Add documentation *Changes in v9:* - Correct fault resolution for userfaultfd wp async - Fix build warnings and errors which were happening on some configs - Simplify pagemap ioctl's code *Changes in v8:* - Update uffd async wp implementation - Improve PAGEMAP_IOCTL implementation *Changes in v7:* - Add uffd wp async - Update the IOCTL to use uffd under the hood instead of soft-dirty flags *Motivation* The real motivation for adding PAGEMAP_SCAN IOCTL is to emulate Windows GetWriteWatch() syscall [1]. The GetWriteWatch{} retrieves the addresses of the pages that are written to in a region of virtual memory. This syscall is used in Windows applications and games etc. This syscall is being emulated in pretty slow manner in userspace. Our purpose is to enhance the kernel such that we translate it efficiently in a better way. Currently some out of tree hack patches are being used to efficiently emulate it in some kernels. We intend to replace those with these patches. So the whole gaming on Linux can effectively get benefit from this. It means there would be tons of users of this code. CRIU use case [2] was mentioned by Andrei and Danylo: > Use cases for migrating sparse VMAs are binaries sanitized with ASAN, > MSAN or TSAN [3]. All of these sanitizers produce sparse mappings of > shadow memory [4]. Being able to migrate such binaries allows to highly > reduce the amount of work needed to identify and fix post-migration > crashes, which happen constantly. Andrei's defines the following uses of this code: * it is more granular and allows us to track changed pages more effectively. The current interface can clear dirty bits for the entire process only. In addition, reading info about pages is a separate operation. It means we must freeze the process to read information about all its pages, reset dirty bits, only then we can start dumping pages. The information about pages becomes more and more outdated, while we are processing pages. The new interface solves both these downsides. First, it allows us to read pte bits and clear the soft-dirty bit atomically. It means that CRIU will not need to freeze processes to pre-dump their memory. Second, it clears soft-dirty bits for a specified region of memory. It means CRIU will have actual info about pages to the moment of dumping them. * The new interface has to be much faster because basic page filtering is happening in the kernel. With the old interface, we have to read pagemap for each page. *Implementation Evolution (Short Summary)* From the definition of GetWriteWatch(), we feel like kernel's soft-dirty feature can be used under the hood with some additions like: * reset soft-dirty flag for only a specific region of memory instead of clearing the flag for the entire process * get and clear soft-dirty flag for a specific region atomically So we decided to use ioctl on pagemap file to read or/and reset soft-dirty flag. But using soft-dirty flag, sometimes we get extra pages which weren't even written. They had become soft-dirty because of VMA merging and VM_SOFTDIRTY flag. This breaks the definition of GetWriteWatch(). We were able to by-pass this short coming by ignoring VM_SOFTDIRTY until David reported that mprotect etc messes up the soft-dirty flag while ignoring VM_SOFTDIRTY [5]. This wasn't happening until [6] got introduced. We discussed if we can revert these patches. But we could not reach to any conclusion. So at this point, I made couple of tries to solve this whole VM_SOFTDIRTY issue by correcting the soft-dirty implementation: * [7] Correct the bug fixed wrongly back in 2014. It had potential to cause regression. We left it behind. * [8] Keep a list of soft-dirty part of a VMA across splits and merges. I got the reply don't increase the size of the VMA by 8 bytes. At this point, we left soft-dirty considering it is too much delicate and userfaultfd [9] seemed like the only way forward. From there onward, we have been basing soft-dirty emulation on userfaultfd wp feature where kernel resolves the faults itself when WP_ASYNC feature is used. It was straight forward to add WP_ASYNC feature in userfautlfd. Now we get only those pages dirty or written-to which are really written in reality. (PS There is another WP_UNPOPULATED userfautfd feature is required which is needed to avoid pre-faulting memory before write-protecting [9].) All the different masks were added on the request of CRIU devs to create interface more generic and better. [1] https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-… [2] https://lore.kernel.org/all/20221014134802.1361436-1-mdanylo@google.com [3] https://github.com/google/sanitizers [4] https://github.com/google/sanitizers/wiki/AddressSanitizerAlgorithm#64-bit [5] https://lore.kernel.org/all/bfcae708-db21-04b4-0bbe-712badd03071@redhat.com [6] https://lore.kernel.org/all/20220725142048.30450-1-peterx@redhat.com/ [7] https://lore.kernel.org/all/20221122115007.2787017-1-usama.anjum@collabora.… [8] https://lore.kernel.org/all/20221220162606.1595355-1-usama.anjum@collabora.… [9] https://lore.kernel.org/all/20230306213925.617814-1-peterx@redhat.com [10] https://lore.kernel.org/all/20230125144529.1630917-1-mdanylo@google.com * Original Cover letter from v8* Hello, Note: Soft-dirty pages and pages which have been written-to are synonyms. As kernel already has soft-dirty feature inside which we have given up to use, we are using written-to terminology while using UFFD async WP under the hood. This IOCTL, PAGEMAP_SCAN on pagemap file can be used to get and/or clear the info about page table entries. The following operations are supported in this ioctl: - Get the information if the pages have been written-to (PAGE_IS_WRITTEN), file mapped (PAGE_IS_FILE), present (PAGE_IS_PRESENT) or swapped (PAGE_IS_SWAPPED). - Write-protect the pages (PAGEMAP_WP_ENGAGE) to start finding which pages have been written-to. - Find pages which have been written-to and write protect the pages (atomic PAGE_IS_WRITTEN + PAGEMAP_WP_ENGAGE) It is possible to find and clear soft-dirty pages entirely in userspace. But it isn't efficient: - The mprotect and SIGSEGV handler for bookkeeping - The userfaultfd wp (synchronous) with the handler for bookkeeping Some benchmarks can be seen here[1]. This series adds features that weren't present earlier: - There is no atomic get soft-dirty/Written-to status and clear present in the kernel. - The pages which have been written-to can not be found in accurate way. (Kernel's soft-dirty PTE bit + sof_dirty VMA bit shows more soft-dirty pages than there actually are.) Historically, soft-dirty PTE bit tracking has been used in the CRIU project. The procfs interface is enough for finding the soft-dirty bit status and clearing the soft-dirty bit of all the pages of a process. We have the use case where we need to track the soft-dirty PTE bit for only specific pages on-demand. We need this tracking and clear mechanism of a region of memory while the process is running to emulate the getWriteWatch() syscall of Windows. *(Moved to using UFFD instead of soft-dirtyi feature to find pages which have been written-to from v7 patch series)*: Stop using the soft-dirty flags for finding which pages have been written to. It is too delicate and wrong as it shows more soft-dirty pages than the actual soft-dirty pages. There is no interest in correcting it [2][3] as this is how the feature was written years ago. It shouldn't be updated to changed behaviour. Peter Xu has suggested using the async version of the UFFD WP [4] as it is based inherently on the PTEs. So in this patch series, I've added a new mode to the UFFD which is asynchronous version of the write protect. When this variant of the UFFD WP is used, the page faults are resolved automatically by the kernel. The pages which have been written-to can be found by reading pagemap file (!PM_UFFD_WP). This feature can be used successfully to find which pages have been written to from the time the pages were write protected. This works just like the soft-dirty flag without showing any extra pages which aren't soft-dirty in reality. The information related to pages if the page is file mapped, present and swapped is required for the CRIU project [5][6]. The addition of the required mask, any mask, excluded mask and return masks are also required for the CRIU project [5]. The IOCTL returns the addresses of the pages which match the specific masks. The page addresses are returned in struct page_region in a compact form. The max_pages is needed to support a use case where user only wants to get a specific number of pages. So there is no need to find all the pages of interest in the range when max_pages is specified. The IOCTL returns when the maximum number of the pages are found. The max_pages is optional. If max_pages is specified, it must be equal or greater than the vec_size. This restriction is needed to handle worse case when one page_region only contains info of one page and it cannot be compacted. This is needed to emulate the Windows getWriteWatch() syscall. The patch series include the detailed selftest which can be used as an example for the uffd async wp test and PAGEMAP_IOCTL. It shows the interface usages as well. [1] https://lore.kernel.org/lkml/54d4c322-cd6e-eefd-b161-2af2b56aae24@collabora… [2] https://lore.kernel.org/all/20221220162606.1595355-1-usama.anjum@collabora.… [3] https://lore.kernel.org/all/20221122115007.2787017-1-usama.anjum@collabora.… [4] https://lore.kernel.org/all/Y6Hc2d+7eTKs7AiH@x1n [5] https://lore.kernel.org/all/YyiDg79flhWoMDZB@gmail.com/ [6] https://lore.kernel.org/all/20221014134802.1361436-1-mdanylo@google.com/ Regards, Muhammad Usama Anjum Muhammad Usama Anjum (4): fs/proc/task_mmu: Implement IOCTL to get and optionally clear info about PTEs tools headers UAPI: Update linux/fs.h with the kernel sources mm/pagemap: add documentation of PAGEMAP_SCAN IOCTL selftests: mm: add pagemap ioctl tests Peter Xu (1): userfaultfd: UFFD_FEATURE_WP_ASYNC Documentation/admin-guide/mm/pagemap.rst | 58 + Documentation/admin-guide/mm/userfaultfd.rst | 35 + fs/proc/task_mmu.c | 503 ++++++ fs/userfaultfd.c | 26 +- include/linux/userfaultfd_k.h | 21 +- include/uapi/linux/fs.h | 53 + include/uapi/linux/userfaultfd.h | 9 +- mm/hugetlb.c | 32 +- mm/memory.c | 27 +- tools/include/uapi/linux/fs.h | 53 + tools/testing/selftests/mm/.gitignore | 1 + tools/testing/selftests/mm/Makefile | 3 +- tools/testing/selftests/mm/config | 1 + tools/testing/selftests/mm/pagemap_ioctl.c | 1459 ++++++++++++++++++ tools/testing/selftests/mm/run_vmtests.sh | 4 + 15 files changed, 2262 insertions(+), 23 deletions(-) create mode 100644 tools/testing/selftests/mm/pagemap_ioctl.c mode change 100644 => 100755 tools/testing/selftests/mm/run_vmtests.sh -- 2.39.2

1 year, 6 months

2
13
0 0

2024

2023

2022

2021

2020

2019

2018

2017

Linux-kselftest-mirror May 2023