- Linux-kselftest-mirror - lists.linaro.org

[PATCH v6 0/6] RISC-V Hardware Probing User Interface

by Evan Green

There's been a bunch of off-list discussions about this, including at Plumbers. The original plan was to do something involving providing an ISA string to userspace, but ISA strings just aren't sufficient for a stable ABI any more: in order to parse an ISA string users need the version of the specifications that the string is written to, the version of each extension (sometimes at a finer granularity than the RISC-V releases/versions encode), and the expected use case for the ISA string (ie, is it a U-mode or M-mode string). That's a lot of complexity to try and keep ABI compatible and it's probably going to continue to grow, as even if there's no more complexity in the specifications we'll have to deal with the various ISA string parsing oddities that end up all over userspace. Instead this patch set takes a very different approach and provides a set of key/value pairs that encode various bits about the system. The big advantage here is that we can clearly define what these mean so we can ensure ABI stability, but it also allows us to encode information that's unlikely to ever appear in an ISA string (see the misaligned access performance, for example). The resulting interface looks a lot like what arm64 and x86 do, and will hopefully fit well into something like ACPI in the future. The actual user interface is a syscall, with a vDSO function in front of it. The vDSO function can answer some queries without a syscall at all, and falls back to the syscall for cases it doesn't have answers to. Currently we prepopulate it with an array of answers for all keys and a CPU set of "all CPUs". This can be adjusted as necessary to provide fast answers to the most common queries. An example series in glibc exposing this syscall and using it in an ifunc selector for memcpy can be found at [1]. I was asked about the performance delta between this and something like sysfs. I created a small test program [2] and ran it on a Nezha D1 Allwinner board. Doing each operation 100000 times and dividing, these operations take the following amount of time: - open()+read()+close() of /sys/kernel/cpu_byteorder: 3.8us - access("/sys/kernel/cpu_byteorder", R_OK): 1.3us - riscv_hwprobe() vDSO and syscall: .0094us - riscv_hwprobe() vDSO with no syscall: 0.0091us These numbers get farther apart if we query multiple keys, as sysfs will scale linearly with the number of keys, where the dedicated syscall stays the same. To frame these numbers, I also did a tight fork/exec/wait loop, which I measured as 4.8ms. So doing 4 open/read/close operations is a delta of about 0.3%, versus a single vDSO call is a delta of essentially zero. [1] https://patchwork.ozlabs.org/project/glibc/list/?series=343050 [2] https://pastebin.com/x84NEKaS Changes in v6: - Remove spurious blank line (Conorbot) - Update copyrights (Paul) - Update copyrights (Paul) - Wrap init_hwprobe_vdso_data() in CONFIG_MMU to fix nommu build break (Conorbot) - Update copyrights (Paul) Changes in v5: - Added tags - Fixed misuse of ISA_EXT_c as bitmap, changed to use riscv_isa_extension_available() (Heiko, Conor) - Document the alternatives approach in the commit message (Conor and Heiko). - Fix __init call warnings by making probe_vendor_features() and thead_feature_probe_func() __init_or_module. - Fixed compat vdso compilation failure (lkp). Changes in v4: - Used real types in syscall prototypes (Arnd) - Fixed static line break in do_riscv_hwprobe() (Conor) - Added newlines between documentation lists (Conor) - Crispen up size types to size_t, and cpu indices to int (Joe) - Fix copy_from_user() return logic bug (found via kselftests!) - Add __user to SYSCALL_DEFINE() to fix warning - More newlines in BASE_BEHAVIOR_IMA documentation (Conor) - Add newlines to CPUPERF_0 documentation (Conor) - Add UNSUPPORTED value (Conor) - Switched from DT to alternatives-based probing (Rob) - Crispen up cpu index type to always be int (Conor) - Fixed selftests commit description, no more tiny libc (Mark Brown) - Fixed selftest syscall prototype types to match v4. - Added a prototype to fix -Wmissing-prototype warning (lkp(a)intel.com) - Fixed rv32 build failure (lkp(a)intel.com) - Make vdso prototype match syscall types update Changes in v3: - Updated copyright date in cpufeature.h - Fixed typo in cpufeature.h comment (Conor) - Refactored functions so that kernel mode can query too, in preparation for the vDSO data population. - Changed the vendor/arch/imp IDs to return a value of -1 on mismatch rather than failing the whole call. - Const cpumask pointer in hwprobe_mid() - Embellished documentation WRT cpu_set and the returned values. - Renamed hwprobe_mid() to hwprobe_arch_id() (Conor) - Fixed machine ID doc warnings, changed elements to c:macro:. - Completed dangling unistd.h comment (Conor) - Fixed line breaks and minor logic optimization (Conor). - Use riscv_cached_mxxxid() (Conor) - Refactored base ISA behavior probe to allow kernel probing as well, in prep for vDSO data initialization. - Fixed doc warnings in IMA text list, use :c:macro:. - Have hwprobe_misaligned return int instead of long. - Constify cpumask pointer in hwprobe_misaligned() - Fix warnings in _PERF_O list documentation, use :c:macro:. - Move include cpufeature.h to misaligned patch. - Fix documentation mismatch for RISCV_HWPROBE_KEY_CPUPERF_0 (Conor) - Use for_each_possible_cpu() instead of NR_CPUS (Conor) - Break early in misaligned access iteration (Conor) - Increase MISALIGNED_MASK from 2 bits to 3 for possible UNSUPPORTED future value (Conor) - Introduced vDSO function Changes in v2: - Factored the move of struct riscv_cpuinfo to its own header - Changed the interface to look more like poll(). Rather than supplying key_offset and getting back an array of values with numerically contiguous keys, have the user pre-fill the key members of the array, and the kernel will fill in the corresponding values. For any key it doesn't recognize, it will set the key of that element to -1. This allows usermode to quickly ask for exactly the elements it cares about, and not get bogged down in a back and forth about newer keys that older kernels might not recognize. In other words, the kernel can communicate that it doesn't recognize some of the keys while still providing the data for the keys it does know. - Added a shortcut to the cpuset parameters that if a size of 0 and NULL is provided for the CPU set, the kernel will use a cpu mask of all online CPUs. This is convenient because I suspect most callers will only want to act on a feature if it's supported on all CPUs, and it's a headache to dynamically allocate an array of all 1s, not to mention a waste to have the kernel loop over all of the offline bits. - Fixed logic error in if(of_property_read_string...) that caused crash - Include cpufeature.h in cpufeature.h to avoid undeclared variable warning. - Added a _MASK define - Fix random checkpatch complaints - Updated the selftests to the new API and added some more. - Fixed indentation, comments in .S, and general checkpatch complaints. Evan Green (6): RISC-V: Move struct riscv_cpuinfo to new header RISC-V: Add a syscall for HW probing RISC-V: hwprobe: Add support for RISCV_HWPROBE_BASE_BEHAVIOR_IMA RISC-V: hwprobe: Support probing of misaligned access performance selftests: Test the new RISC-V hwprobe interface RISC-V: Add hwprobe vDSO function and data Documentation/riscv/hwprobe.rst | 86 +++++++ Documentation/riscv/index.rst | 1 + arch/riscv/Kconfig | 1 + arch/riscv/errata/thead/errata.c | 10 + arch/riscv/include/asm/alternative.h | 5 + arch/riscv/include/asm/cpufeature.h | 23 ++ arch/riscv/include/asm/hwprobe.h | 13 + arch/riscv/include/asm/syscall.h | 4 + arch/riscv/include/asm/vdso/data.h | 17 ++ arch/riscv/include/asm/vdso/gettimeofday.h | 8 + arch/riscv/include/uapi/asm/hwprobe.h | 37 +++ arch/riscv/include/uapi/asm/unistd.h | 9 + arch/riscv/kernel/alternative.c | 19 ++ arch/riscv/kernel/compat_vdso/Makefile | 2 +- arch/riscv/kernel/cpu.c | 8 +- arch/riscv/kernel/cpufeature.c | 3 + arch/riscv/kernel/smpboot.c | 1 + arch/riscv/kernel/sys_riscv.c | 228 +++++++++++++++++- arch/riscv/kernel/vdso.c | 6 - arch/riscv/kernel/vdso/Makefile | 4 + arch/riscv/kernel/vdso/hwprobe.c | 52 ++++ arch/riscv/kernel/vdso/sys_hwprobe.S | 15 ++ arch/riscv/kernel/vdso/vdso.lds.S | 3 + tools/testing/selftests/Makefile | 1 + tools/testing/selftests/riscv/Makefile | 58 +++++ .../testing/selftests/riscv/hwprobe/Makefile | 10 + .../testing/selftests/riscv/hwprobe/hwprobe.c | 90 +++++++ .../selftests/riscv/hwprobe/sys_hwprobe.S | 12 + 28 files changed, 712 insertions(+), 14 deletions(-) create mode 100644 Documentation/riscv/hwprobe.rst create mode 100644 arch/riscv/include/asm/cpufeature.h create mode 100644 arch/riscv/include/asm/hwprobe.h create mode 100644 arch/riscv/include/asm/vdso/data.h create mode 100644 arch/riscv/include/uapi/asm/hwprobe.h create mode 100644 arch/riscv/kernel/vdso/hwprobe.c create mode 100644 arch/riscv/kernel/vdso/sys_hwprobe.S create mode 100644 tools/testing/selftests/riscv/Makefile create mode 100644 tools/testing/selftests/riscv/hwprobe/Makefile create mode 100644 tools/testing/selftests/riscv/hwprobe/hwprobe.c create mode 100644 tools/testing/selftests/riscv/hwprobe/sys_hwprobe.S -- 2.25.1

2 years, 2 months

3
3
0 0

[PATCH v3 00/17] Add iommufd physical device operations for replace and alloc hwpt

by Jason Gunthorpe

This is the basic functionality for iommufd to support iommufd_device_replace() and IOMMU_HWPT_ALLOC for physical devices. iommufd_device_replace() allows changing the HWPT associated with the device to a new IOAS or HWPT. Replace does this in way that failure leaves things unchanged, and utilizes the iommu iommu_group_replace_domain() API to allow the iommu driver to perform an optional non-disruptive change. IOMMU_HWPT_ALLOC allows HWPTs to be explicitly allocated by the user and used by attach or replace. At this point it isn't very useful since the HWPT is the same as the automatically managed HWPT from the IOAS. However a following series will allow userspace to customize the created HWPT. The implementation is complicated because we have to introduce some per-iommu_group memory in iommufd and redo how we think about multi-device groups to be more explicit. This solves all the locking problems in the prior attempts. This series is infrastructure work for the following series which: - Add replace for attach - Expose replace through VFIO APIs - Implement driver parameters for HWPT creation (nesting) Once review of this is complete I will keep it on a side branch and accumulate the following series when they are ready so we can have a stable base and make more incremental progress. When we have all the parts together to get a full implementation it can go to Linus. I have this on github: https://github.com/jgunthorpe/linux/commits/iommufd_hwpt v3: - Refine comments and commit messages - Adjust the flow in iommufd_device_auto_get_domain() so pt_id is only set on success - Reject replace on non-attached devices - Add missing __reserved check for IOMMU_HWPT_ALLOC v2: https://lore.kernel.org/r/0-v2-51b9896e7862+8a8c-iommufd_alloc_jgg@nvidia.c… - Use WARN_ON for the igroup->group test and move that logic to a function iommufd_group_try_get() - Change igroup->devices to igroup->device list Replace will need to iterate over all attached idevs - Rename to iommufd_group_setup_msi() - New patch to export iommu_get_resv_regions() - New patch to use per-device reserved regions instead of per-group regions - Split out the reorganizing of iommufd_device_change_pt() from the replace patch - Replace uses the per-dev reserved regions - Use stdev_id in a few more places in the selftest - Fix error handling in IOMMU_HWPT_ALLOC - Clarify comments - Rebase on v6.3-rc1 v1: https://lore.kernel.org/all/0-v1-7612f88c19f5+2f21-iommufd_alloc_jgg@nvidia… Jason Gunthorpe (15): iommufd: Move isolated msi enforcement to iommufd_device_bind() iommufd: Add iommufd_group iommufd: Replace the hwpt->devices list with iommufd_group iommu: Export iommu_get_resv_regions() iommufd: Keep track of each device's reserved regions instead of groups iommufd: Use the iommufd_group to avoid duplicate MSI setup iommufd: Make sw_msi_start a group global iommufd: Move putting a hwpt to a helper function iommufd: Add enforced_cache_coherency to iommufd_hw_pagetable_alloc() iommufd: Reorganize iommufd_device_attach into iommufd_device_change_pt iommufd: Add iommufd_device_replace() iommufd: Make destroy_rwsem use a lock class per object type iommufd: Add IOMMU_HWPT_ALLOC iommufd/selftest: Return the real idev id from selftest mock_domain iommufd/selftest: Add a selftest for IOMMU_HWPT_ALLOC Nicolin Chen (2): iommu: Introduce a new iommu_group_replace_domain() API iommufd/selftest: Test iommufd_device_replace() drivers/iommu/iommu-priv.h | 10 + drivers/iommu/iommu.c | 41 +- drivers/iommu/iommufd/device.c | 512 +++++++++++++----- drivers/iommu/iommufd/hw_pagetable.c | 96 +++- drivers/iommu/iommufd/io_pagetable.c | 27 +- drivers/iommu/iommufd/iommufd_private.h | 51 +- drivers/iommu/iommufd/iommufd_test.h | 6 + drivers/iommu/iommufd/main.c | 17 +- drivers/iommu/iommufd/selftest.c | 40 ++ include/linux/iommufd.h | 1 + include/uapi/linux/iommufd.h | 26 + tools/testing/selftests/iommu/iommufd.c | 64 ++- .../selftests/iommu/iommufd_fail_nth.c | 52 +- tools/testing/selftests/iommu/iommufd_utils.h | 61 ++- 14 files changed, 804 insertions(+), 200 deletions(-) create mode 100644 drivers/iommu/iommu-priv.h base-commit: fd8c1a4aee973e87d890a5861e106625a33b2c4e -- 2.40.0

2 years, 2 months

2
45
0 0

[PATCH RESEND] .gitignore: Do not ignore .kunitconfig files

by Chuck Lever

From: Chuck Lever <chuck.lever(a)oracle.com> Circumvent the .gitignore wildcard to avoid warnings about ignored .kunitconfig files. As far as I can tell, the warnings are harmless and these files are not actually ignored. Reported-by: kernel test robot <lkp(a)intel.com> Link: https://lore.kernel.org/oe-kbuild-all/202304142337.jc4oUrov-lkp@intel.com/ Signed-off-by: Chuck Lever <chuck.lever(a)oracle.com> --- .gitignore | 1 + 1 file changed, 1 insertion(+) Resending... It was not clear to me if this file has a specific maintainer. I chose to send it to the most recent committer. diff --git a/.gitignore b/.gitignore index 70ec6037fa7a..51117ba29c88 100644 --- a/.gitignore +++ b/.gitignore @@ -105,6 +105,7 @@ modules.order !.gitignore !.mailmap !.rustfmt.toml +!.kunitconfig # # Generated include files

2 years, 2 months

3
2
0 0

[PATCH bpf 0/4] Socket lookup BPF API from tc/xdp ingress does not respect VRF bindings.

by Gilad Sever

When calling socket lookup from L2 (tc, xdp), VRF boundaries aren't respected. This patchset fixes this by regarding the incoming device's VRF attachment when performing the socket lookups from tc/xdp. The first two patches are coding changes which facilitate this fix by factoring out the tc helper's logic which was shared with cg/sk_skb (which operate correctly). The third patch contains the actual bugfix. The fourth patch adds bpf tests for these lookup functions. Gilad Sever (4): bpf: factor out socket lookup functions for the TC hookpoint. bpf: Call __bpf_sk_lookup()/__bpf_skc_lookup() directly via TC hookpoint bpf: fix bpf socket lookup from tc/xdp to respect socket VRF bindings selftests/bpf: Add tc_socket_lookup tests net/core/filter.c | 132 +++++-- .../bpf/prog_tests/tc_socket_lookup.c | 341 ++++++++++++++++++ .../selftests/bpf/progs/tc_socket_lookup.c | 73 ++++ 3 files changed, 525 insertions(+), 21 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/tc_socket_lookup.c create mode 100644 tools/testing/selftests/bpf/progs/tc_socket_lookup.c -- 2.34.1

2 years, 2 months

1
4
0 0

[PATCH AUTOSEL 5.15 6/7] selftests mount: Fix mount_setattr_test builds failed

by Sasha Levin

From: Anh Tuan Phan <tuananhlfc(a)gmail.com> [ Upstream commit f1594bc676579133a3cd906d7d27733289edfb86 ] When compiling selftests with target mount_setattr I encountered some errors with the below messages: mount_setattr_test.c: In function ‘mount_setattr_thread’: mount_setattr_test.c:343:16: error: variable ‘attr’ has initializer but incomplete type 343 | struct mount_attr attr = { | ^~~~~~~~~~ These errors might be because of linux/mount.h is not included. This patch resolves that issue. Signed-off-by: Anh Tuan Phan <tuananhlfc(a)gmail.com> Acked-by: Christian Brauner <brauner(a)kernel.org> Signed-off-by: Shuah Khan <skhan(a)linuxfoundation.org> Signed-off-by: Sasha Levin <sashal(a)kernel.org> --- tools/testing/selftests/mount_setattr/mount_setattr_test.c | 1 + 1 file changed, 1 insertion(+) diff --git a/tools/testing/selftests/mount_setattr/mount_setattr_test.c b/tools/testing/selftests/mount_setattr/mount_setattr_test.c index 8c5fea68ae677..969647228817b 100644 --- a/tools/testing/selftests/mount_setattr/mount_setattr_test.c +++ b/tools/testing/selftests/mount_setattr/mount_setattr_test.c @@ -18,6 +18,7 @@ #include <grp.h> #include <stdbool.h> #include <stdarg.h> +#include <linux/mount.h> #include "../kselftest_harness.h" -- 2.39.2

2 years, 2 months

1
0
0 0

[PATCH AUTOSEL 6.1 10/15] selftests mount: Fix mount_setattr_test builds failed

by Sasha Levin

From: Anh Tuan Phan <tuananhlfc(a)gmail.com> [ Upstream commit f1594bc676579133a3cd906d7d27733289edfb86 ] When compiling selftests with target mount_setattr I encountered some errors with the below messages: mount_setattr_test.c: In function ‘mount_setattr_thread’: mount_setattr_test.c:343:16: error: variable ‘attr’ has initializer but incomplete type 343 | struct mount_attr attr = { | ^~~~~~~~~~ These errors might be because of linux/mount.h is not included. This patch resolves that issue. Signed-off-by: Anh Tuan Phan <tuananhlfc(a)gmail.com> Acked-by: Christian Brauner <brauner(a)kernel.org> Signed-off-by: Shuah Khan <skhan(a)linuxfoundation.org> Signed-off-by: Sasha Levin <sashal(a)kernel.org> --- tools/testing/selftests/mount_setattr/mount_setattr_test.c | 1 + 1 file changed, 1 insertion(+) diff --git a/tools/testing/selftests/mount_setattr/mount_setattr_test.c b/tools/testing/selftests/mount_setattr/mount_setattr_test.c index 8c5fea68ae677..969647228817b 100644 --- a/tools/testing/selftests/mount_setattr/mount_setattr_test.c +++ b/tools/testing/selftests/mount_setattr/mount_setattr_test.c @@ -18,6 +18,7 @@ #include <grp.h> #include <stdbool.h> #include <stdarg.h> +#include <linux/mount.h> #include "../kselftest_harness.h" -- 2.39.2

2 years, 2 months

1
0
0 0

[PATCH AUTOSEL 6.2 12/17] selftests mount: Fix mount_setattr_test builds failed

by Sasha Levin

From: Anh Tuan Phan <tuananhlfc(a)gmail.com> [ Upstream commit f1594bc676579133a3cd906d7d27733289edfb86 ] When compiling selftests with target mount_setattr I encountered some errors with the below messages: mount_setattr_test.c: In function ‘mount_setattr_thread’: mount_setattr_test.c:343:16: error: variable ‘attr’ has initializer but incomplete type 343 | struct mount_attr attr = { | ^~~~~~~~~~ These errors might be because of linux/mount.h is not included. This patch resolves that issue. Signed-off-by: Anh Tuan Phan <tuananhlfc(a)gmail.com> Acked-by: Christian Brauner <brauner(a)kernel.org> Signed-off-by: Shuah Khan <skhan(a)linuxfoundation.org> Signed-off-by: Sasha Levin <sashal(a)kernel.org> --- tools/testing/selftests/mount_setattr/mount_setattr_test.c | 1 + 1 file changed, 1 insertion(+) diff --git a/tools/testing/selftests/mount_setattr/mount_setattr_test.c b/tools/testing/selftests/mount_setattr/mount_setattr_test.c index 8c5fea68ae677..969647228817b 100644 --- a/tools/testing/selftests/mount_setattr/mount_setattr_test.c +++ b/tools/testing/selftests/mount_setattr/mount_setattr_test.c @@ -18,6 +18,7 @@ #include <grp.h> #include <stdbool.h> #include <stdarg.h> +#include <linux/mount.h> #include "../kselftest_harness.h" -- 2.39.2

2 years, 2 months

1
0
0 0

[PATCH v2 1/3] kunit: Always run cleanup from a test kthread

by David Gow

KUnit tests run in a kthread, with the current->kunit_test pointer set to the test's context. This allows the kunit_get_current_test() and kunit_fail_current_test() macros to work. Normally, this pointer is still valid during test shutdown (i.e., the suite->exit function, and any resource cleanup). However, if the test has exited early (e.g., due to a failed assertion), the cleanup is done in the parent KUnit thread, which does not have an active context. Instead, in the event test terminates early, run the test exit and cleanup from a new 'cleanup' kthread, which sets current->kunit_test, and better isolates the rest of KUnit from issues which arise in test cleanup. If a test cleanup function itself aborts (e.g., due to an assertion failing), there will be no further attempts to clean up: an error will be logged and the test failed. This should also make it easier to get access to the KUnit context, particularly from within resource cleanup functions, which may, for example, need access to data in test->priv. Signed-off-by: David Gow <davidgow(a)google.com> --- This is an updated version of / replacement of "kunit: Set the current KUnit context when cleaning up", which instead creates a new kthread for cleanup tasks if the original test kthread is aborted. This protects us from failed assertions during cleanup, if the test exited early. Changes since v1: https://lore.kernel.org/linux-kselftest/20230415091401.681395-1-davidgow@go… - Move cleanup execution to another kthread - (Thanks, Benjamin, for pointing out the assertion issues) --- lib/kunit/test.c | 54 ++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 52 insertions(+), 2 deletions(-) diff --git a/lib/kunit/test.c b/lib/kunit/test.c index e2910b261112..caeae0dfd82b 100644 --- a/lib/kunit/test.c +++ b/lib/kunit/test.c @@ -423,8 +423,51 @@ static void kunit_try_run_case(void *data) kunit_run_case_cleanup(test, suite); } +static void kunit_try_run_case_cleanup(void *data) +{ + struct kunit_try_catch_context *ctx = data; + struct kunit *test = ctx->test; + struct kunit_suite *suite = ctx->suite; + + current->kunit_test = test; + + kunit_run_case_cleanup(test, suite); +} + +static void kunit_catch_run_case_cleanup(void *data) +{ + struct kunit_try_catch_context *ctx = data; + struct kunit *test = ctx->test; + int try_exit_code = kunit_try_catch_get_result(&test->try_catch); + + /* It is always a failure if cleanup aborts. */ + kunit_set_failure(test); + + if (try_exit_code) { + /* + * Test case could not finish, we have no idea what state it is + * in, so don't do clean up. + */ + if (try_exit_code == -ETIMEDOUT) { + kunit_err(test, "test case cleanup timed out\n"); + /* + * Unknown internal error occurred preventing test case from + * running, so there is nothing to clean up. + */ + } else { + kunit_err(test, "internal error occurred during test case cleanup: %d\n", + try_exit_code); + } + return; + } + + kunit_err(test, "test aborted during cleanup. continuing without cleaning up\n"); +} + + static void kunit_catch_run_case(void *data) { + struct kunit_try_catch cleanup; struct kunit_try_catch_context *ctx = data; struct kunit *test = ctx->test; struct kunit_suite *suite = ctx->suite; @@ -451,9 +494,16 @@ static void kunit_catch_run_case(void *data) /* * Test case was run, but aborted. It is the test case's business as to - * whether it failed or not, we just need to clean up. + * whether it failed or not, we just need to clean up. Do this in a new + * try / catch context, in case it asserts, too. */ - kunit_run_case_cleanup(test, suite); + kunit_try_catch_init(&cleanup, + test, + kunit_try_run_case_cleanup, + kunit_catch_run_case_cleanup); + ctx->test = test; + ctx->suite = suite; + kunit_try_catch_run(&cleanup, ctx); } /* -- 2.40.0.634.g4ca3ef3211-goog

2 years, 2 months

2
3
0 0

[PATCH bpf-next v2 0/2] Access variable length array relaxed for integer type

by Feng zhou

From: Feng Zhou <zhoufeng.zf(a)bytedance.com> Add support for integer type of accessing variable length array. Add a selftest to check it. Feng Zhou (2): bpf: support access variable length array of integer type selftests/bpf: Add test to access integer type of variable array Changelog: v1->v2: Addressed comments from Alexei Starovoitov - Add one more use case. Details in here: https://lore.kernel.org/bpf/20230417080749.39074-1-zhoufeng.zf@bytedance.co… kernel/bpf/btf.c | 8 +++++--- .../selftests/bpf/bpf_testmod/bpf_testmod.c | 20 +++++++++++++++++++ .../bpf/prog_tests/access_variable_array.c | 16 +++++++++++++++ .../selftests/bpf/prog_tests/tracing_struct.c | 2 ++ .../bpf/progs/test_access_variable_array.c | 19 ++++++++++++++++++ .../selftests/bpf/progs/tracing_struct.c | 13 ++++++++++++ 6 files changed, 75 insertions(+), 3 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/access_variable_array.c create mode 100644 tools/testing/selftests/bpf/progs/test_access_variable_array.c -- 2.20.1

2 years, 2 months

2
3
0 0

[PATCH v15 0/5] Implement IOCTL to get and optionally clear info about PTEs

by Muhammad Usama Anjum

*Changes in v15* - Build fix *Changes in v14* - Fix build error caused by #ifdef added at last minute in some configs *Changes in v13* - Rebase on top of next-20230414 - Give-up on using uffd_wp_range() and write new helpers, flush tlb only once *Changes in v12* - Update and other memory types to UFFD_FEATURE_WP_ASYNC - Rebaase on top of next-20230406 - Review updates *Changes in v11* - Rebase on top of next-20230307 - Base patches on UFFD_FEATURE_WP_UNPOPULATED - Do a lot of cosmetic changes and review updates - Remove ENGAGE_WP + !GET operation as it can be performed with UFFDIO_WRITEPROTECT *Changes in v10* - Add specific condition to return error if hugetlb is used with wp async - Move changes in tools/include/uapi/linux/fs.h to separate patch - Add documentation *Changes in v9:* - Correct fault resolution for userfaultfd wp async - Fix build warnings and errors which were happening on some configs - Simplify pagemap ioctl's code *Changes in v8:* - Update uffd async wp implementation - Improve PAGEMAP_IOCTL implementation *Changes in v7:* - Add uffd wp async - Update the IOCTL to use uffd under the hood instead of soft-dirty flags *Motivation* The real motivation for adding PAGEMAP_SCAN IOCTL is to emulate Windows GetWriteWatch() syscall [1]. The GetWriteWatch{} retrieves the addresses of the pages that are written to in a region of virtual memory. This syscall is used in Windows applications and games etc. This syscall is being emulated in pretty slow manner in userspace. Our purpose is to enhance the kernel such that we translate it efficiently in a better way. Currently some out of tree hack patches are being used to efficiently emulate it in some kernels. We intend to replace those with these patches. So the whole gaming on Linux can effectively get benefit from this. It means there would be tons of users of this code. CRIU use case [2] was mentioned by Andrei and Danylo: > Use cases for migrating sparse VMAs are binaries sanitized with ASAN, > MSAN or TSAN [3]. All of these sanitizers produce sparse mappings of > shadow memory [4]. Being able to migrate such binaries allows to highly > reduce the amount of work needed to identify and fix post-migration > crashes, which happen constantly. Andrei's defines the following uses of this code: * it is more granular and allows us to track changed pages more effectively. The current interface can clear dirty bits for the entire process only. In addition, reading info about pages is a separate operation. It means we must freeze the process to read information about all its pages, reset dirty bits, only then we can start dumping pages. The information about pages becomes more and more outdated, while we are processing pages. The new interface solves both these downsides. First, it allows us to read pte bits and clear the soft-dirty bit atomically. It means that CRIU will not need to freeze processes to pre-dump their memory. Second, it clears soft-dirty bits for a specified region of memory. It means CRIU will have actual info about pages to the moment of dumping them. * The new interface has to be much faster because basic page filtering is happening in the kernel. With the old interface, we have to read pagemap for each page. *Implementation Evolution (Short Summary)* From the definition of GetWriteWatch(), we feel like kernel's soft-dirty feature can be used under the hood with some additions like: * reset soft-dirty flag for only a specific region of memory instead of clearing the flag for the entire process * get and clear soft-dirty flag for a specific region atomically So we decided to use ioctl on pagemap file to read or/and reset soft-dirty flag. But using soft-dirty flag, sometimes we get extra pages which weren't even written. They had become soft-dirty because of VMA merging and VM_SOFTDIRTY flag. This breaks the definition of GetWriteWatch(). We were able to by-pass this short coming by ignoring VM_SOFTDIRTY until David reported that mprotect etc messes up the soft-dirty flag while ignoring VM_SOFTDIRTY [5]. This wasn't happening until [6] got introduced. We discussed if we can revert these patches. But we could not reach to any conclusion. So at this point, I made couple of tries to solve this whole VM_SOFTDIRTY issue by correcting the soft-dirty implementation: * [7] Correct the bug fixed wrongly back in 2014. It had potential to cause regression. We left it behind. * [8] Keep a list of soft-dirty part of a VMA across splits and merges. I got the reply don't increase the size of the VMA by 8 bytes. At this point, we left soft-dirty considering it is too much delicate and userfaultfd [9] seemed like the only way forward. From there onward, we have been basing soft-dirty emulation on userfaultfd wp feature where kernel resolves the faults itself when WP_ASYNC feature is used. It was straight forward to add WP_ASYNC feature in userfautlfd. Now we get only those pages dirty or written-to which are really written in reality. (PS There is another WP_UNPOPULATED userfautfd feature is required which is needed to avoid pre-faulting memory before write-protecting [9].) All the different masks were added on the request of CRIU devs to create interface more generic and better. [1] https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-… [2] https://lore.kernel.org/all/20221014134802.1361436-1-mdanylo@google.com [3] https://github.com/google/sanitizers [4] https://github.com/google/sanitizers/wiki/AddressSanitizerAlgorithm#64-bit [5] https://lore.kernel.org/all/bfcae708-db21-04b4-0bbe-712badd03071@redhat.com [6] https://lore.kernel.org/all/20220725142048.30450-1-peterx@redhat.com/ [7] https://lore.kernel.org/all/20221122115007.2787017-1-usama.anjum@collabora.… [8] https://lore.kernel.org/all/20221220162606.1595355-1-usama.anjum@collabora.… [9] https://lore.kernel.org/all/20230306213925.617814-1-peterx@redhat.com [10] https://lore.kernel.org/all/20230125144529.1630917-1-mdanylo@google.com * Original Cover letter from v8* Hello, Note: Soft-dirty pages and pages which have been written-to are synonyms. As kernel already has soft-dirty feature inside which we have given up to use, we are using written-to terminology while using UFFD async WP under the hood. This IOCTL, PAGEMAP_SCAN on pagemap file can be used to get and/or clear the info about page table entries. The following operations are supported in this ioctl: - Get the information if the pages have been written-to (PAGE_IS_WRITTEN), file mapped (PAGE_IS_FILE), present (PAGE_IS_PRESENT) or swapped (PAGE_IS_SWAPPED). - Write-protect the pages (PAGEMAP_WP_ENGAGE) to start finding which pages have been written-to. - Find pages which have been written-to and write protect the pages (atomic PAGE_IS_WRITTEN + PAGEMAP_WP_ENGAGE) It is possible to find and clear soft-dirty pages entirely in userspace. But it isn't efficient: - The mprotect and SIGSEGV handler for bookkeeping - The userfaultfd wp (synchronous) with the handler for bookkeeping Some benchmarks can be seen here[1]. This series adds features that weren't present earlier: - There is no atomic get soft-dirty/Written-to status and clear present in the kernel. - The pages which have been written-to can not be found in accurate way. (Kernel's soft-dirty PTE bit + sof_dirty VMA bit shows more soft-dirty pages than there actually are.) Historically, soft-dirty PTE bit tracking has been used in the CRIU project. The procfs interface is enough for finding the soft-dirty bit status and clearing the soft-dirty bit of all the pages of a process. We have the use case where we need to track the soft-dirty PTE bit for only specific pages on-demand. We need this tracking and clear mechanism of a region of memory while the process is running to emulate the getWriteWatch() syscall of Windows. *(Moved to using UFFD instead of soft-dirtyi feature to find pages which have been written-to from v7 patch series)*: Stop using the soft-dirty flags for finding which pages have been written to. It is too delicate and wrong as it shows more soft-dirty pages than the actual soft-dirty pages. There is no interest in correcting it [2][3] as this is how the feature was written years ago. It shouldn't be updated to changed behaviour. Peter Xu has suggested using the async version of the UFFD WP [4] as it is based inherently on the PTEs. So in this patch series, I've added a new mode to the UFFD which is asynchronous version of the write protect. When this variant of the UFFD WP is used, the page faults are resolved automatically by the kernel. The pages which have been written-to can be found by reading pagemap file (!PM_UFFD_WP). This feature can be used successfully to find which pages have been written to from the time the pages were write protected. This works just like the soft-dirty flag without showing any extra pages which aren't soft-dirty in reality. The information related to pages if the page is file mapped, present and swapped is required for the CRIU project [5][6]. The addition of the required mask, any mask, excluded mask and return masks are also required for the CRIU project [5]. The IOCTL returns the addresses of the pages which match the specific masks. The page addresses are returned in struct page_region in a compact form. The max_pages is needed to support a use case where user only wants to get a specific number of pages. So there is no need to find all the pages of interest in the range when max_pages is specified. The IOCTL returns when the maximum number of the pages are found. The max_pages is optional. If max_pages is specified, it must be equal or greater than the vec_size. This restriction is needed to handle worse case when one page_region only contains info of one page and it cannot be compacted. This is needed to emulate the Windows getWriteWatch() syscall. The patch series include the detailed selftest which can be used as an example for the uffd async wp test and PAGEMAP_IOCTL. It shows the interface usages as well. [1] https://lore.kernel.org/lkml/54d4c322-cd6e-eefd-b161-2af2b56aae24@collabora… [2] https://lore.kernel.org/all/20221220162606.1595355-1-usama.anjum@collabora.… [3] https://lore.kernel.org/all/20221122115007.2787017-1-usama.anjum@collabora.… [4] https://lore.kernel.org/all/Y6Hc2d+7eTKs7AiH@x1n [5] https://lore.kernel.org/all/YyiDg79flhWoMDZB@gmail.com/ [6] https://lore.kernel.org/all/20221014134802.1361436-1-mdanylo@google.com/ Regards, Muhammad Usama Anjum Muhammad Usama Anjum (4): fs/proc/task_mmu: Implement IOCTL to get and optionally clear info about PTEs tools headers UAPI: Update linux/fs.h with the kernel sources mm/pagemap: add documentation of PAGEMAP_SCAN IOCTL selftests: mm: add pagemap ioctl tests Peter Xu (1): userfaultfd: UFFD_FEATURE_WP_ASYNC Documentation/admin-guide/mm/pagemap.rst | 56 + Documentation/admin-guide/mm/userfaultfd.rst | 35 + fs/proc/task_mmu.c | 481 +++++++ fs/userfaultfd.c | 26 +- include/linux/userfaultfd_k.h | 21 +- include/uapi/linux/fs.h | 53 + include/uapi/linux/userfaultfd.h | 9 +- mm/hugetlb.c | 32 +- mm/memory.c | 27 +- tools/include/uapi/linux/fs.h | 53 + tools/testing/selftests/mm/.gitignore | 1 + tools/testing/selftests/mm/Makefile | 3 +- tools/testing/selftests/mm/config | 1 + tools/testing/selftests/mm/pagemap_ioctl.c | 1326 ++++++++++++++++++ tools/testing/selftests/mm/run_vmtests.sh | 4 + 15 files changed, 2105 insertions(+), 23 deletions(-) create mode 100644 tools/testing/selftests/mm/pagemap_ioctl.c mode change 100644 => 100755 tools/testing/selftests/mm/run_vmtests.sh -- 2.39.2

2 years, 2 months

2
7
0 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

Linux-kselftest-mirror