July 2019 - Linux-kselftest-mirror

[PATCH v7 00/25] Unify vDSOs across more architectures

by Vincenzo Frascino

vDSO (virtual dynamic shared object) is a mechanism that the Linux kernel provides as an alternative to system calls to reduce where possible the costs in terms of cycles. This is possible because certain syscalls like gettimeofday() do not write any data and return one or more values that are stored in the kernel, which makes relatively safe calling them directly as a library function. Even if the mechanism is pretty much standard, every architecture in the last few years ended up implementing their own vDSO library in the architectural code. The purpose of this patch-set is to identify the commonalities in between the architectures and try to consolidate the common code paths, starting with gettimeofday(). This implementation contains the following design choices: * Every architecture defines the arch specific code in an header in "asm/vdso/". * The generic implementation includes the arch specific one and lives in "lib/vdso". * The arch specific code for gettimeofday lives in "<arch path>/vdso/gettimeofday.c" and includes the generic code only. * The generic implementation of update_vsyscall and update_vsyscall_tz lives in kernel/vdso and provide the bindings that can be implemented by each architecture. * Each architecture provides its implementation of the bindings in "asm/vdso/vsyscall.h". * This approach allows to consolidate the common code in a single place with the benefit of avoiding code duplication. This implementation contains the portings to the common library for: arm64, compat mode for arm64, arm, mips, x86_64, x32, compat mode for x86_64 and i386. The mips porting has been tested on qemu for mips32el. A configuration to repeat the tests can be found at [4]. The x86_64 porting has been tested on an Intel Xeon 5120T based machine running Ubuntu 18.04 and using the Ubuntu provided defconfig. The i386 porting has been tested on qemu using the i386_defconfig configuration. Last but not least from this porting arm64, compat arm64, arm and mips gain the support for: * CLOCK_BOOTTIME that can be useful in certain scenarios since it keeps track of the time during sleep as well. * CLOCK_TAI that is like CLOCK_REALTIME, but uses the International Atomic Time (TAI) reference instead of UTC to avoid jumping on leap second updates. for both clock_gettime and clock_getres. The porting has been validated using the vdsotest test-suite [1] extended to cover all the clock ids [2]. A new test has been added to the linux kselftest in order to validate the newly added library. The porting has been benchmarked and the performance results are provided as part of this cover letter. To simplify the testing, a copy of the patchset on top of a recent linux tree can be found at [3] and [4]. The v7 of this patchseries has been rebased on [5]. [1] https://github.com/nathanlynch/vdsotest [2] https://github.com/fvincenzo/vdsotest [3] git://linux-arm.org/linux-vf.git vdso/v7 [4] git://linux-arm.org/linux-vf.git vdso-mips/v7 [5] git://git.kernel.org/pub/scm/linux/kernel/git/hyperv/linux.git hyperv-next Changes: -------- v7: - Rebased on [5] (5.2-rc3). - Added performance numbers for arm64 provided by Shijith Thotton. - Aimed at 1:1 replacement for pre-exisiting vDSO libraries. - Provided separate patches for newly added API. - Addressed review comments. v6: - Rebased on 5.2-rc2. - Added performance numbers. - Removed vdso_types.h. - Unified update_vsyscall and update_vsyscall_tz. - Reworked the kselftest included in this patchset. - Addressed review comments. v5: - Rebased on 5.0-rc7. - Added x86_64, compat mode for x86_64 and i386 portings. - Extended vDSO kselftest. - Addressed review comments. v4: - Rebased on 5.0-rc2. - Addressed review comments. - Disabled compat vdso on arm64 when the kernel is compiled with clang. v3: - Ported the latest fixes and optimizations done on the x86 architecture to the generic library. - Addressed review comments. - Improved the documentation of the interfaces. - Changed the HAVE_ARCH_TIMER config option to a more generic HAVE_HW_COUNTER. v2: - Added -ffixed-x18 to arm64 - Repleced occurrences of timeval and timespec - Modified datapage.h to be compliant with y2038 on all the architectures - Removed __u_vdso type Cc: Catalin Marinas <catalin.marinas(a)arm.com> Cc: Will Deacon <will.deacon(a)arm.com> Cc: Arnd Bergmann <arnd(a)arndb.de> Cc: Russell King <linux(a)armlinux.org.uk> Cc: Ralf Baechle <ralf(a)linux-mips.org> Cc: Paul Burton <paul.burton(a)mips.com> Cc: Daniel Lezcano <daniel.lezcano(a)linaro.org> Cc: Thomas Gleixner <tglx(a)linutronix.de> Cc: Mark Salyzyn <salyzyn(a)android.com> Cc: Peter Collingbourne <pcc(a)google.com> Cc: Shuah Khan <shuah(a)kernel.org> Cc: Dmitry Safonov <0x7f454c46(a)gmail.com> Cc: Rasmus Villemoes <linux(a)rasmusvillemoes.dk> Cc: Huw Davies <huw(a)codeweavers.com> Cc: Shijith Thotton <sthotton(a)marvell.com> Cc: Andre Przywara <andre.przywara(a)arm.com> Signed-off-by: Vincenzo Frascino <vincenzo.frascino(a)arm.com> Tested-by: Shijith Thotton <sthotton(a)marvell.com> Tested-by: Andre Przywara <andre.przywara(a)arm.com> Performance Numbers: Linux 5.2.0-rc2 - Xeon Gold 5120T ====================================================== Unified vDSO: ------------- clock-gettime-monotonic: syscall: 342 nsec/call clock-gettime-monotonic: libc: 25 nsec/call clock-gettime-monotonic: vdso: 24 nsec/call clock-getres-monotonic: syscall: 296 nsec/call clock-getres-monotonic: libc: 296 nsec/call clock-getres-monotonic: vdso: 3 nsec/call clock-gettime-monotonic-coarse: syscall: 294 nsec/call clock-gettime-monotonic-coarse: libc: 5 nsec/call clock-gettime-monotonic-coarse: vdso: 5 nsec/call clock-getres-monotonic-coarse: syscall: 295 nsec/call clock-getres-monotonic-coarse: libc: 292 nsec/call clock-getres-monotonic-coarse: vdso: 5 nsec/call clock-gettime-monotonic-raw: syscall: 343 nsec/call clock-gettime-monotonic-raw: libc: 25 nsec/call clock-gettime-monotonic-raw: vdso: 23 nsec/call clock-getres-monotonic-raw: syscall: 290 nsec/call clock-getres-monotonic-raw: libc: 290 nsec/call clock-getres-monotonic-raw: vdso: 4 nsec/call clock-gettime-tai: syscall: 332 nsec/call clock-gettime-tai: libc: 24 nsec/call clock-gettime-tai: vdso: 23 nsec/call clock-getres-tai: syscall: 288 nsec/call clock-getres-tai: libc: 288 nsec/call clock-getres-tai: vdso: 3 nsec/call clock-gettime-boottime: syscall: 342 nsec/call clock-gettime-boottime: libc: 24 nsec/call clock-gettime-boottime: vdso: 23 nsec/call clock-getres-boottime: syscall: 284 nsec/call clock-getres-boottime: libc: 291 nsec/call clock-getres-boottime: vdso: 3 nsec/call clock-gettime-realtime: syscall: 337 nsec/call clock-gettime-realtime: libc: 24 nsec/call clock-gettime-realtime: vdso: 23 nsec/call clock-getres-realtime: syscall: 287 nsec/call clock-getres-realtime: libc: 284 nsec/call clock-getres-realtime: vdso: 3 nsec/call clock-gettime-realtime-coarse: syscall: 307 nsec/call clock-gettime-realtime-coarse: libc: 4 nsec/call clock-gettime-realtime-coarse: vdso: 4 nsec/call clock-getres-realtime-coarse: syscall: 294 nsec/call clock-getres-realtime-coarse: libc: 291 nsec/call clock-getres-realtime-coarse: vdso: 4 nsec/call getcpu: syscall: 246 nsec/call getcpu: libc: 14 nsec/call getcpu: vdso: 11 nsec/call gettimeofday: syscall: 293 nsec/call gettimeofday: libc: 26 nsec/call gettimeofday: vdso: 25 nsec/call Stock Kernel: ------------- clock-gettime-monotonic: syscall: 338 nsec/call clock-gettime-monotonic: libc: 24 nsec/call clock-gettime-monotonic: vdso: 23 nsec/call clock-getres-monotonic: syscall: 291 nsec/call clock-getres-monotonic: libc: 304 nsec/call clock-getres-monotonic: vdso: not tested Note: vDSO version of clock_getres not found clock-gettime-monotonic-coarse: syscall: 297 nsec/call clock-gettime-monotonic-coarse: libc: 5 nsec/call clock-gettime-monotonic-coarse: vdso: 4 nsec/call clock-getres-monotonic-coarse: syscall: 281 nsec/call clock-getres-monotonic-coarse: libc: 286 nsec/call clock-getres-monotonic-coarse: vdso: not tested Note: vDSO version of clock_getres not found clock-gettime-monotonic-raw: syscall: 336 nsec/call clock-gettime-monotonic-raw: libc: 340 nsec/call clock-gettime-monotonic-raw: vdso: 346 nsec/call clock-getres-monotonic-raw: syscall: 297 nsec/call clock-getres-monotonic-raw: libc: 301 nsec/call clock-getres-monotonic-raw: vdso: not tested Note: vDSO version of clock_getres not found clock-gettime-tai: syscall: 351 nsec/call clock-gettime-tai: libc: 24 nsec/call clock-gettime-tai: vdso: 23 nsec/call clock-getres-tai: syscall: 298 nsec/call clock-getres-tai: libc: 290 nsec/call clock-getres-tai: vdso: not tested Note: vDSO version of clock_getres not found clock-gettime-boottime: syscall: 342 nsec/call clock-gettime-boottime: libc: 347 nsec/call clock-gettime-boottime: vdso: 355 nsec/call clock-getres-boottime: syscall: 296 nsec/call clock-getres-boottime: libc: 295 nsec/call clock-getres-boottime: vdso: not tested Note: vDSO version of clock_getres not found clock-gettime-realtime: syscall: 346 nsec/call clock-gettime-realtime: libc: 24 nsec/call clock-gettime-realtime: vdso: 22 nsec/call clock-getres-realtime: syscall: 295 nsec/call clock-getres-realtime: libc: 291 nsec/call clock-getres-realtime: vdso: not tested Note: vDSO version of clock_getres not found clock-gettime-realtime-coarse: syscall: 292 nsec/call clock-gettime-realtime-coarse: libc: 5 nsec/call clock-gettime-realtime-coarse: vdso: 4 nsec/call clock-getres-realtime-coarse: syscall: 300 nsec/call clock-getres-realtime-coarse: libc: 301 nsec/call clock-getres-realtime-coarse: vdso: not tested Note: vDSO version of clock_getres not found getcpu: syscall: 252 nsec/call getcpu: libc: 14 nsec/call getcpu: vdso: 11 nsec/call gettimeofday: syscall: 293 nsec/call gettimeofday: libc: 24 nsec/call gettimeofday: vdso: 25 nsec/call Performance Numbers: Linux 5.2.0-rc2 - ThunderX2 (arm64) ======================================================== Provided by: Shijith Thotton <sthotton(a)marvell.com> Unified vDSO: ------------- clock-gettime-monotonic: syscall: 346 nsec/call clock-gettime-monotonic: libc: 38 nsec/call clock-gettime-monotonic: vdso: 36 nsec/call clock-getres-monotonic: syscall: 262 nsec/call clock-getres-monotonic: libc: 6 nsec/call clock-getres-monotonic: vdso: 5 nsec/call clock-gettime-monotonic-coarse: syscall: 296 nsec/call clock-gettime-monotonic-coarse: libc: 39 nsec/call clock-gettime-monotonic-coarse: vdso: 38 nsec/call clock-getres-monotonic-coarse: syscall: 260 nsec/call clock-getres-monotonic-coarse: libc: 8 nsec/call clock-getres-monotonic-coarse: vdso: 5 nsec/call clock-gettime-monotonic-raw: syscall: 345 nsec/call clock-gettime-monotonic-raw: libc: 35 nsec/call clock-gettime-monotonic-raw: vdso: 34 nsec/call clock-getres-monotonic-raw: syscall: 261 nsec/call clock-getres-monotonic-raw: libc: 7 nsec/call clock-getres-monotonic-raw: vdso: 5 nsec/call clock-gettime-tai: syscall: 357 nsec/call clock-gettime-tai: libc: 38 nsec/call clock-gettime-tai: vdso: 36 nsec/call clock-getres-tai: syscall: 257 nsec/call clock-getres-tai: libc: 7 nsec/call clock-getres-tai: vdso: 5 nsec/call clock-gettime-boottime: syscall: 356 nsec/call clock-gettime-boottime: libc: 38 nsec/call clock-gettime-boottime: vdso: 36 nsec/call clock-getres-boottime: syscall: 257 nsec/call clock-getres-boottime: libc: 6 nsec/call clock-getres-boottime: vdso: 5 nsec/call clock-gettime-realtime: syscall: 345 nsec/call clock-gettime-realtime: libc: 38 nsec/call clock-gettime-realtime: vdso: 36 nsec/call clock-getres-realtime: syscall: 257 nsec/call clock-getres-realtime: libc: 7 nsec/call clock-getres-realtime: vdso: 5 nsec/call clock-gettime-realtime-coarse: syscall: 295 nsec/call clock-gettime-realtime-coarse: libc: 39 nsec/call clock-gettime-realtime-coarse: vdso: 38 nsec/call clock-getres-realtime-coarse: syscall: 260 nsec/call clock-getres-realtime-coarse: libc: 8 nsec/call clock-getres-realtime-coarse: vdso: 5 nsec/call getcpu: syscall: 244 nsec/call getcpu: libc: 247 nsec/call getcpu: vdso: not tested Note: vDSO version of getcpu not found gettimeofday: syscall: 383 nsec/call gettimeofday: libc: 39 nsec/call gettimeofday: vdso: 35 nsec/call Stock Kernel: ------------- clock-gettime-monotonic: syscall: 344 nsec/call clock-gettime-monotonic: libc: 74 nsec/call clock-gettime-monotonic: vdso: 73 nsec/call clock-getres-monotonic: syscall: 258 nsec/call clock-getres-monotonic: libc: 6 nsec/call clock-getres-monotonic: vdso: 4 nsec/call clock-gettime-monotonic-coarse: syscall: 300 nsec/call clock-gettime-monotonic-coarse: libc: 36 nsec/call clock-gettime-monotonic-coarse: vdso: 34 nsec/call clock-getres-monotonic-coarse: syscall: 261 nsec/call clock-getres-monotonic-coarse: libc: 6 nsec/call clock-getres-monotonic-coarse: vdso: 4 nsec/call clock-gettime-monotonic-raw: syscall: 346 nsec/call clock-gettime-monotonic-raw: libc: 74 nsec/call clock-gettime-monotonic-raw: vdso: 72 nsec/call clock-getres-monotonic-raw: syscall: 254 nsec/call clock-getres-monotonic-raw: libc: 6 nsec/call clock-getres-monotonic-raw: vdso: 4 nsec/call clock-gettime-tai: syscall: 345 nsec/call clock-gettime-tai: libc: 361 nsec/call clock-gettime-tai: vdso: 359 nsec/call clock-getres-tai: syscall: 259 nsec/call clock-getres-tai: libc: 262 nsec/call clock-getres-tai: vdso: 258 nsec/call clock-gettime-boottime: syscall: 353 nsec/call clock-gettime-boottime: libc: 365 nsec/call clock-gettime-boottime: vdso: 362 nsec/call clock-getres-boottime: syscall: 260 nsec/call clock-getres-boottime: libc: 267 nsec/call clock-getres-boottime: vdso: 259 nsec/call clock-gettime-realtime: syscall: 344 nsec/call clock-gettime-realtime: libc: 73 nsec/call clock-gettime-realtime: vdso: 72 nsec/call clock-getres-realtime: syscall: 255 nsec/call clock-getres-realtime: libc: 7 nsec/call clock-getres-realtime: vdso: 4 nsec/call clock-gettime-realtime-coarse: syscall: 296 nsec/call clock-gettime-realtime-coarse: libc: 35 nsec/call clock-gettime-realtime-coarse: vdso: 33 nsec/call clock-getres-realtime-coarse: syscall: 258 nsec/call clock-getres-realtime-coarse: libc: 6 nsec/call clock-getres-realtime-coarse: vdso: 4 nsec/call getcpu: syscall: 237 nsec/call getcpu: libc: 242 nsec/call getcpu: vdso: not tested Note: vDSO version of getcpu not found gettimeofday: syscall: 378 nsec/call gettimeofday: libc: 73 nsec/call gettimeofday: vdso: 70 nsec/call Peter Collingbourne (1): arm64: Build vDSO with -ffixed-x18 Vincenzo Frascino (24): kernel: Standardize vdso_datapage kernel: Define gettimeofday vdso common code kernel: Unify update_vsyscall implementation arm64: Substitute gettimeofday with C implementation arm64: compat: Add missing syscall numbers arm64: compat: Expose signal related structures arm64: compat: Generate asm offsets for signals lib: vdso: Add compat support arm64: compat: Add vDSO arm64: Refactor vDSO code arm64: compat: vDSO setup for compat layer arm64: elf: vDSO code page discovery arm64: compat: Get sigreturn trampolines from vDSO arm64: Add vDSO compat support arm: Add support for generic vDSO arm: Add clock_getres entry point arm: Add clock_gettime64 entry point mips: Add support for generic vDSO mips: Add clock_getres entry point mips: Add clock_gettime64 entry point x86: Add support for generic vDSO x86: Add clock_getres entry point x86: Add clock_gettime64 entry point kselftest: Extend vDSO selftest arch/arm/Kconfig | 3 + arch/arm/include/asm/vdso/gettimeofday.h | 94 +++++ arch/arm/include/asm/vdso/vsyscall.h | 71 ++++ arch/arm/include/asm/vdso_datapage.h | 29 +- arch/arm/kernel/vdso.c | 87 +---- arch/arm/vdso/Makefile | 13 +- arch/arm/vdso/note.c | 15 + arch/arm/vdso/vdso.lds.S | 2 + arch/arm/vdso/vgettimeofday.c | 256 +------------ arch/arm64/Kconfig | 3 + arch/arm64/Makefile | 23 +- arch/arm64/include/asm/elf.h | 14 + arch/arm64/include/asm/signal32.h | 46 +++ arch/arm64/include/asm/unistd.h | 5 + arch/arm64/include/asm/vdso.h | 3 + arch/arm64/include/asm/vdso/compat_barrier.h | 51 +++ .../include/asm/vdso/compat_gettimeofday.h | 110 ++++++ arch/arm64/include/asm/vdso/gettimeofday.h | 86 +++++ arch/arm64/include/asm/vdso/vsyscall.h | 53 +++ arch/arm64/include/asm/vdso_datapage.h | 48 --- arch/arm64/kernel/Makefile | 6 +- arch/arm64/kernel/asm-offsets.c | 39 +- arch/arm64/kernel/signal32.c | 72 ++-- arch/arm64/kernel/vdso.c | 356 ++++++++++++------ arch/arm64/kernel/vdso/Makefile | 34 +- arch/arm64/kernel/vdso/gettimeofday.S | 334 ---------------- arch/arm64/kernel/vdso/vgettimeofday.c | 28 ++ arch/arm64/kernel/vdso32/.gitignore | 2 + arch/arm64/kernel/vdso32/Makefile | 186 +++++++++ arch/arm64/kernel/vdso32/note.c | 15 + arch/arm64/kernel/vdso32/sigreturn.S | 62 +++ arch/arm64/kernel/vdso32/vdso.S | 19 + arch/arm64/kernel/vdso32/vdso.lds.S | 82 ++++ arch/arm64/kernel/vdso32/vgettimeofday.c | 59 +++ arch/mips/Kconfig | 2 + arch/mips/include/asm/vdso.h | 78 +--- arch/mips/include/asm/vdso/gettimeofday.h | 177 +++++++++ arch/mips/{ => include/asm}/vdso/vdso.h | 6 +- arch/mips/include/asm/vdso/vsyscall.h | 43 +++ arch/mips/kernel/vdso.c | 37 +- arch/mips/vdso/Makefile | 27 +- arch/mips/vdso/elf.S | 2 +- arch/mips/vdso/sigreturn.S | 2 +- arch/mips/vdso/vdso.lds.S | 4 + arch/mips/vdso/vgettimeofday.c | 58 +++ arch/x86/Kconfig | 3 + arch/x86/entry/vdso/Makefile | 9 + arch/x86/entry/vdso/vclock_gettime.c | 249 +++--------- arch/x86/entry/vdso/vdso.lds.S | 2 + arch/x86/entry/vdso/vdso32/vdso32.lds.S | 2 + arch/x86/entry/vdso/vdsox32.lds.S | 1 + arch/x86/entry/vsyscall/Makefile | 2 - arch/x86/entry/vsyscall/vsyscall_gtod.c | 83 ---- arch/x86/include/asm/pvclock.h | 2 +- arch/x86/include/asm/vdso/gettimeofday.h | 205 ++++++++++ arch/x86/include/asm/vdso/vsyscall.h | 44 +++ arch/x86/include/asm/vgtod.h | 75 +--- arch/x86/include/asm/vvar.h | 7 +- arch/x86/kernel/pvclock.c | 1 + include/asm-generic/vdso/vsyscall.h | 56 +++ include/linux/hrtimer.h | 15 +- include/linux/hrtimer_defs.h | 25 ++ include/linux/timekeeper_internal.h | 9 + include/vdso/datapage.h | 93 +++++ include/vdso/helpers.h | 56 +++ include/vdso/vsyscall.h | 11 + kernel/Makefile | 1 + kernel/vdso/Makefile | 2 + kernel/vdso/vsyscall.c | 131 +++++++ lib/Kconfig | 5 + lib/vdso/Kconfig | 36 ++ lib/vdso/Makefile | 22 ++ lib/vdso/gettimeofday.c | 231 ++++++++++++ tools/testing/selftests/vDSO/Makefile | 2 + tools/testing/selftests/vDSO/vdso_config.h | 90 +++++ tools/testing/selftests/vDSO/vdso_full_test.c | 244 ++++++++++++ 76 files changed, 3044 insertions(+), 1412 deletions(-) create mode 100644 arch/arm/include/asm/vdso/gettimeofday.h create mode 100644 arch/arm/include/asm/vdso/vsyscall.h create mode 100644 arch/arm/vdso/note.c create mode 100644 arch/arm64/include/asm/vdso/compat_barrier.h create mode 100644 arch/arm64/include/asm/vdso/compat_gettimeofday.h create mode 100644 arch/arm64/include/asm/vdso/gettimeofday.h create mode 100644 arch/arm64/include/asm/vdso/vsyscall.h delete mode 100644 arch/arm64/include/asm/vdso_datapage.h delete mode 100644 arch/arm64/kernel/vdso/gettimeofday.S create mode 100644 arch/arm64/kernel/vdso/vgettimeofday.c create mode 100644 arch/arm64/kernel/vdso32/.gitignore create mode 100644 arch/arm64/kernel/vdso32/Makefile create mode 100644 arch/arm64/kernel/vdso32/note.c create mode 100644 arch/arm64/kernel/vdso32/sigreturn.S create mode 100644 arch/arm64/kernel/vdso32/vdso.S create mode 100644 arch/arm64/kernel/vdso32/vdso.lds.S create mode 100644 arch/arm64/kernel/vdso32/vgettimeofday.c create mode 100644 arch/mips/include/asm/vdso/gettimeofday.h rename arch/mips/{ => include/asm}/vdso/vdso.h (89%) create mode 100644 arch/mips/include/asm/vdso/vsyscall.h create mode 100644 arch/mips/vdso/vgettimeofday.c delete mode 100644 arch/x86/entry/vsyscall/vsyscall_gtod.c create mode 100644 arch/x86/include/asm/vdso/gettimeofday.h create mode 100644 arch/x86/include/asm/vdso/vsyscall.h create mode 100644 include/asm-generic/vdso/vsyscall.h create mode 100644 include/linux/hrtimer_defs.h create mode 100644 include/vdso/datapage.h create mode 100644 include/vdso/helpers.h create mode 100644 include/vdso/vsyscall.h create mode 100644 kernel/vdso/Makefile create mode 100644 kernel/vdso/vsyscall.c create mode 100644 lib/vdso/Kconfig create mode 100644 lib/vdso/Makefile create mode 100644 lib/vdso/gettimeofday.c create mode 100644 tools/testing/selftests/vDSO/vdso_config.h create mode 100644 tools/testing/selftests/vDSO/vdso_full_test.c -- 2.21.0

5 years

18
107
0 0

[PATCH v4 0/6] lib/string: Add strscpy_pad() function

by Tobin C. Harding

Hi Shua, Here is the set with cleanup as suggested by Kees on v3. Configured, built, and tested all modules loaded by tools/testing/selftests/lib/*.sh >From previous cover letters ... While doing the testing for strscpy_pad() it was noticed that there is duplication in how test modules are being fed to kselftest and also in the test modules themselves. This set makes an attempt at adding a framework to kselftest for writing kernel test modules. It also adds a script for use in creating script test runners for kselftest. My macro-foo is not great, all criticism and suggestions very much appreciated. The design is based on test modules lib/test_printf.c, lib/test_bitmap.c, lib/test_xarray.c. Changes since last version: - Remove dependency on Bash (thanks Kees) - Use oneliner to implement kselftest test runners (thanks Kees) - Squash patch that adds kselftest script creator script with patch that uses it. - Fix typos (thanks Randy) - Add Kees' Acked-by tags to all patches thanks, Tobin. Tobin C. Harding (6): lib/test_printf: Add empty module_exit function kselftest: Add test runner creation script kselftest: Add test module framework header lib: Use new kselftest header lib/string: Add strscpy_pad() function lib: Add test module for strscpy_pad Documentation/dev-tools/kselftest.rst | 94 +++++++++++- include/linux/string.h | 4 + lib/Kconfig.debug | 3 + lib/Makefile | 1 + lib/string.c | 47 +++++- lib/test_bitmap.c | 20 +-- lib/test_printf.c | 17 +-- lib/test_strscpy.c | 150 +++++++++++++++++++ tools/testing/selftests/kselftest_module.h | 48 ++++++ tools/testing/selftests/kselftest_module.sh | 84 +++++++++++ tools/testing/selftests/lib/Makefile | 2 +- tools/testing/selftests/lib/bitmap.sh | 18 +-- tools/testing/selftests/lib/config | 1 + tools/testing/selftests/lib/prime_numbers.sh | 17 +-- tools/testing/selftests/lib/printf.sh | 19 +-- tools/testing/selftests/lib/strscpy.sh | 3 + 16 files changed, 440 insertions(+), 88 deletions(-) create mode 100644 lib/test_strscpy.c create mode 100644 tools/testing/selftests/kselftest_module.h create mode 100755 tools/testing/selftests/kselftest_module.sh create mode 100755 tools/testing/selftests/lib/strscpy.sh -- 2.21.0

5 years, 1 month

5
11
0 0

[RFC v3 00/19] kunit: introduce KUnit, the Linux kernel unit testing framework

by Brendan Higgins

This patch set proposes KUnit, a lightweight unit testing and mocking framework for the Linux kernel. Unlike Autotest and kselftest, KUnit is a true unit testing framework; it does not require installing the kernel on a test machine or in a VM and does not require tests to be written in userspace running on a host kernel. Additionally, KUnit is fast: From invocation to completion KUnit can run several dozen tests in under a second. Currently, the entire KUnit test suite for KUnit runs in under a second from the initial invocation (build time excluded). KUnit is heavily inspired by JUnit, Python's unittest.mock, and Googletest/Googlemock for C++. KUnit provides facilities for defining unit test cases, grouping related test cases into test suites, providing common infrastructure for running tests, mocking, spying, and much more. ## What's so special about unit testing? A unit test is supposed to test a single unit of code in isolation, hence the name. There should be no dependencies outside the control of the test; this means no external dependencies, which makes tests orders of magnitudes faster. Likewise, since there are no external dependencies, there are no hoops to jump through to run the tests. Additionally, this makes unit tests deterministic: a failing unit test always indicates a problem. Finally, because unit tests necessarily have finer granularity, they are able to test all code paths easily solving the classic problem of difficulty in exercising error handling code. ## Is KUnit trying to replace other testing frameworks for the kernel? No. Most existing tests for the Linux kernel are end-to-end tests, which have their place. A well tested system has lots of unit tests, a reasonable number of integration tests, and some end-to-end tests. KUnit is just trying to address the unit test space which is currently not being addressed. ## More information on KUnit There is a bunch of documentation near the end of this patch set that describes how to use KUnit and best practices for writing unit tests. For convenience I am hosting the compiled docs here: https://google.github.io/kunit-docs/third_party/kernel/docs/ Additionally for convenience, I have applied these patches to a branch: https://kunit.googlesource.com/linux/+/kunit/rfc/4.19/v3 The repo may be cloned with: git clone https://kunit.googlesource.com/linux This patchset is on the kunit/rfc/4.19/v3 branch. ## Changes Since Last Version - Changed namespace prefix from `test_*` to `kunit_*` as requested by Shuah. - Started converting/cleaning up the device tree unittest to use KUnit. - Started adding KUnit expectations with custom messages. -- 2.20.0.rc0.387.gc7a69e6b6c-goog

5 years, 3 months

11
105
0 0

[PATCH v18 00/15] arm64: untag user pointers passed to the kernel

by Andrey Konovalov

=== Overview arm64 has a feature called Top Byte Ignore, which allows to embed pointer tags into the top byte of each pointer. Userspace programs (such as HWASan, a memory debugging tool [1]) might use this feature and pass tagged user pointers to the kernel through syscalls or other interfaces. Right now the kernel is already able to handle user faults with tagged pointers, due to these patches: 1. 81cddd65 ("arm64: traps: fix userspace cache maintenance emulation on a tagged pointer") 2. 7dcd9dd8 ("arm64: hw_breakpoint: fix watchpoint matching for tagged pointers") 3. 276e9327 ("arm64: entry: improve data abort handling of tagged pointers") This patchset extends tagged pointer support to syscall arguments. As per the proposed ABI change [3], tagged pointers are only allowed to be passed to syscalls when they point to memory ranges obtained by anonymous mmap() or sbrk() (see the patchset [3] for more details). For non-memory syscalls this is done by untaging user pointers when the kernel performs pointer checking to find out whether the pointer comes from userspace (most notably in access_ok). The untagging is done only when the pointer is being checked, the tag is preserved as the pointer makes its way through the kernel and stays tagged when the kernel dereferences the pointer when perfoming user memory accesses. The mmap and mremap (only new_addr) syscalls do not currently accept tagged addresses. Architectures may interpret the tag as a background colour for the corresponding vma. Other memory syscalls (mprotect, etc.) don't do user memory accesses but rather deal with memory ranges, and untagged pointers are better suited to describe memory ranges internally. Thus for memory syscalls we untag pointers completely when they enter the kernel. === Other approaches One of the alternative approaches to untagging that was considered is to completely strip the pointer tag as the pointer enters the kernel with some kind of a syscall wrapper, but that won't work with the countless number of different ioctl calls. With this approach we would need a custom wrapper for each ioctl variation, which doesn't seem practical. An alternative approach to untagging pointers in memory syscalls prologues is to inspead allow tagged pointers to be passed to find_vma() (and other vma related functions) and untag them there. Unfortunately, a lot of find_vma() callers then compare or subtract the returned vma start and end fields against the pointer that was being searched. Thus this approach would still require changing all find_vma() callers. === Testing The following testing approaches has been taken to find potential issues with user pointer untagging: 1. Static testing (with sparse [2] and separately with a custom static analyzer based on Clang) to track casts of __user pointers to integer types to find places where untagging needs to be done. 2. Static testing with grep to find parts of the kernel that call find_vma() (and other similar functions) or directly compare against vm_start/vm_end fields of vma. 3. Static testing with grep to find parts of the kernel that compare user pointers with TASK_SIZE or other similar consts and macros. 4. Dynamic testing: adding BUG_ON(has_tag(addr)) to find_vma() and running a modified syzkaller version that passes tagged pointers to the kernel. Based on the results of the testing the requried patches have been added to the patchset. === Notes This patchset is meant to be merged together with "arm64 relaxed ABI" [3]. This patchset is a prerequisite for ARM's memory tagging hardware feature support [4]. This patchset has been merged into the Pixel 2 & 3 kernel trees and is now being used to enable testing of Pixel phones with HWASan. Thanks! [1] http://clang.llvm.org/docs/HardwareAssistedAddressSanitizerDesign.html [2] https://github.com/lucvoo/sparse-dev/commit/5f960cb10f56ec2017c128ef9d16060… [3] https://lkml.org/lkml/2019/6/12/745 [4] https://community.arm.com/processors/b/blog/posts/arm-a-profile-architectur… === History Changes in v18: - Reverted the selftest back to not using the LD_PRELOAD approach. - Added prctl(PR_SET_TAGGED_ADDR_CTRL) call to the selftest. - Reworded the patch descriptions to make them less oriented on arm64 only. - Catalin's patch: "I added a Kconfig option and dropped the prctl args zero check. There is some minor clean-up as well". Changes in v17: - The "uaccess: add noop untagged_addr definition" patch is dropped, as it was merged into upstream named as "uaccess: add noop untagged_addr definition". - Merged "mm, arm64: untag user pointers in do_pages_move" into "mm, arm64: untag user pointers passed to memory syscalls". - Added "arm64: Introduce prctl() options to control the tagged user addresses ABI" patch from Catalin. - Add tags_lib.so to tools/testing/selftests/arm64/.gitignore. - Added a comment clarifying untagged in mremap. - Moved untagging back into mlx4_get_umem_mr() for the IB patch. Changes in v16: - Moved untagging for memory syscalls from arm64 wrappers back to generic code. - Dropped untagging for the following memory syscalls: brk, mmap, munmap; mremap (only dropped for new_address); mmap_pgoff (not used on arm64); remap_file_pages (deprecated); shmat, shmdt (work on shared memory). - Changed kselftest to LD_PRELOAD a shared library that overrides malloc to return tagged pointers. - Rebased onto 5.2-rc3. Changes in v15: - Removed unnecessary untagging from radeon_ttm_tt_set_userptr(). - Removed unnecessary untagging from amdgpu_ttm_tt_set_userptr(). - Moved untagging to validate_range() in userfaultfd code. - Moved untagging to ib_uverbs_(re)reg_mr() from mlx4_get_umem_mr(). - Rebased onto 5.1. Changes in v14: - Moved untagging for most memory syscalls to an arm64 specific implementation, instead of doing that in the common code. - Dropped "net, arm64: untag user pointers in tcp_zerocopy_receive", since the provided user pointers don't come from an anonymous map and thus are not covered by this ABI relaxation. - Dropped "kernel, arm64: untag user pointers in prctl_set_mm*". - Moved untagging from __check_mem_type() to tee_shm_register(). - Updated untagging for the amdgpu and radeon drivers to cover the MMU notifier, as suggested by Felix. - Since this ABI relaxation doesn't actually allow tagged instruction pointers, dropped the following patches: - Dropped "tracing, arm64: untag user pointers in seq_print_user_ip". - Dropped "uprobes, arm64: untag user pointers in find_active_uprobe". - Dropped "bpf, arm64: untag user pointers in stack_map_get_build_id_offset". - Rebased onto 5.1-rc7 (37624b58). Changes in v13: - Simplified untagging in tcp_zerocopy_receive(). - Looked at find_vma() callers in drivers/, which allowed to identify a few other places where untagging is needed. - Added patch "mm, arm64: untag user pointers in get_vaddr_frames". - Added patch "drm/amdgpu, arm64: untag user pointers in amdgpu_ttm_tt_get_user_pages". - Added patch "drm/radeon, arm64: untag user pointers in radeon_ttm_tt_pin_userptr". - Added patch "IB/mlx4, arm64: untag user pointers in mlx4_get_umem_mr". - Added patch "media/v4l2-core, arm64: untag user pointers in videobuf_dma_contig_user_get". - Added patch "tee/optee, arm64: untag user pointers in check_mem_type". - Added patch "vfio/type1, arm64: untag user pointers". Changes in v12: - Changed untagging in tcp_zerocopy_receive() to also untag zc->address. - Fixed untagging in prctl_set_mm* to only untag pointers for vma lookups and validity checks, but leave them as is for actual user space accesses. - Updated the link to the v2 of the "arm64 relaxed ABI" patchset [3]. - Dropped the documentation patch, as the "arm64 relaxed ABI" patchset [3] handles that. Changes in v11: - Added "uprobes, arm64: untag user pointers in find_active_uprobe" patch. - Added "bpf, arm64: untag user pointers in stack_map_get_build_id_offset" patch. - Fixed "tracing, arm64: untag user pointers in seq_print_user_ip" to correctly perform subtration with a tagged addr. - Moved untagged_addr() from SYSCALL_DEFINE3(mprotect) and SYSCALL_DEFINE4(pkey_mprotect) to do_mprotect_pkey(). - Moved untagged_addr() definition for other arches from include/linux/memory.h to include/linux/mm.h. - Changed untagging in strn*_user() to perform userspace accesses through tagged pointers. - Updated the documentation to mention that passing tagged pointers to memory syscalls is allowed. - Updated the test to use malloc'ed memory instead of stack memory. Changes in v10: - Added "mm, arm64: untag user pointers passed to memory syscalls" back. - New patch "fs, arm64: untag user pointers in fs/userfaultfd.c". - New patch "net, arm64: untag user pointers in tcp_zerocopy_receive". - New patch "kernel, arm64: untag user pointers in prctl_set_mm*". - New patch "tracing, arm64: untag user pointers in seq_print_user_ip". Changes in v9: - Rebased onto 4.20-rc6. - Used u64 instead of __u64 in type casts in the untagged_addr macro for arm64. - Added braces around (addr) in the untagged_addr macro for other arches. Changes in v8: - Rebased onto 65102238 (4.20-rc1). - Added a note to the cover letter on why syscall wrappers/shims that untag user pointers won't work. - Added a note to the cover letter that this patchset has been merged into the Pixel 2 kernel tree. - Documentation fixes, in particular added a list of syscalls that don't support tagged user pointers. Changes in v7: - Rebased onto 17b57b18 (4.19-rc6). - Dropped the "arm64: untag user address in __do_user_fault" patch, since the existing patches already handle user faults properly. - Dropped the "usb, arm64: untag user addresses in devio" patch, since the passed pointer must come from a vma and therefore be untagged. - Dropped the "arm64: annotate user pointers casts detected by sparse" patch (see the discussion to the replies of the v6 of this patchset). - Added more context to the cover letter. - Updated Documentation/arm64/tagged-pointers.txt. Changes in v6: - Added annotations for user pointer casts found by sparse. - Rebased onto 050cdc6c (4.19-rc1+). Changes in v5: - Added 3 new patches that add untagging to places found with static analysis. - Rebased onto 44c929e1 (4.18-rc8). Changes in v4: - Added a selftest for checking that passing tagged pointers to the kernel succeeds. - Rebased onto 81e97f013 (4.18-rc1+). Changes in v3: - Rebased onto e5c51f30 (4.17-rc6+). - Added linux-arch@ to the list of recipients. Changes in v2: - Rebased onto 2d618bdf (4.17-rc3+). - Removed excessive untagging in gup.c. - Removed untagging pointers returned from __uaccess_mask_ptr. Changes in v1: - Rebased onto 4.17-rc1. Changes in RFC v2: - Added "#ifndef untagged_addr..." fallback in linux/uaccess.h instead of defining it for each arch individually. - Updated Documentation/arm64/tagged-pointers.txt. - Dropped "mm, arm64: untag user addresses in memory syscalls". - Rebased onto 3eb2ce82 (4.16-rc7). Signed-off-by: Andrey Konovalov <andreyknvl(a)google.com> Andrey Konovalov (14): arm64: untag user pointers in access_ok and __uaccess_mask_ptr lib: untag user pointers in strn*_user mm: untag user pointers passed to memory syscalls mm: untag user pointers in mm/gup.c mm: untag user pointers in get_vaddr_frames fs/namespace: untag user pointers in copy_mount_options userfaultfd: untag user pointers drm/amdgpu: untag user pointers drm/radeon: untag user pointers in radeon_gem_userptr_ioctl IB/mlx4: untag user pointers in mlx4_get_umem_mr media/v4l2-core: untag user pointers in videobuf_dma_contig_user_get tee/shm: untag user pointers in tee_shm_register vfio/type1: untag user pointers in vaddr_get_pfn selftests, arm64: add a selftest for passing tagged pointers to kernel Catalin Marinas (1): arm64: Introduce prctl() options to control the tagged user addresses ABI arch/arm64/Kconfig | 9 +++ arch/arm64/include/asm/processor.h | 8 +++ arch/arm64/include/asm/thread_info.h | 1 + arch/arm64/include/asm/uaccess.h | 12 +++- arch/arm64/kernel/process.c | 72 +++++++++++++++++++ .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 2 + drivers/gpu/drm/radeon/radeon_gem.c | 2 + drivers/infiniband/hw/mlx4/mr.c | 7 +- drivers/media/v4l2-core/videobuf-dma-contig.c | 9 +-- drivers/tee/tee_shm.c | 1 + drivers/vfio/vfio_iommu_type1.c | 2 + fs/namespace.c | 2 +- fs/userfaultfd.c | 22 +++--- include/uapi/linux/prctl.h | 5 ++ kernel/sys.c | 12 ++++ lib/strncpy_from_user.c | 3 +- lib/strnlen_user.c | 3 +- mm/frame_vector.c | 2 + mm/gup.c | 4 ++ mm/madvise.c | 2 + mm/mempolicy.c | 3 + mm/migrate.c | 2 +- mm/mincore.c | 2 + mm/mlock.c | 4 ++ mm/mprotect.c | 2 + mm/mremap.c | 7 ++ mm/msync.c | 2 + tools/testing/selftests/arm64/.gitignore | 1 + tools/testing/selftests/arm64/Makefile | 11 +++ .../testing/selftests/arm64/run_tags_test.sh | 12 ++++ tools/testing/selftests/arm64/tags_test.c | 29 ++++++++ 32 files changed, 232 insertions(+), 25 deletions(-) create mode 100644 tools/testing/selftests/arm64/.gitignore create mode 100644 tools/testing/selftests/arm64/Makefile create mode 100755 tools/testing/selftests/arm64/run_tags_test.sh create mode 100644 tools/testing/selftests/arm64/tags_test.c -- 2.22.0.410.gd8fdbe21b5-goog

5 years, 3 months

7
46
0 0

general protection fault in tls_setsockopt

by syzbot

Hello, syzbot found the following crash on: HEAD commit: a131c2bf Merge tag 'acpi-5.3-rc1-2' of git://git.kernel.or.. git tree: net-next console output: https://syzkaller.appspot.com/x/log.txt?x=1603e9c0600000 kernel config: https://syzkaller.appspot.com/x/.config?x=8bff73c5ba9e876 dashboard link: https://syzkaller.appspot.com/bug?extid=23d9570edec63669d890 compiler: gcc (GCC) 9.0.0 20181231 (experimental) syz repro: https://syzkaller.appspot.com/x/repro.syz?x=13560870600000 The bug was bisected to: commit 7c85c448e7d74c4ddd759440a2141eab663567cf Author: Prashant Bhole <bhole_prashant_q7(a)lab.ntt.co.jp> Date: Tue Oct 9 01:04:54 2018 +0000 selftests/bpf: test_verifier, check bpf_map_lookup_elem access in bpf prog bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=17000114600000 final crash: https://syzkaller.appspot.com/x/report.txt?x=14800114600000 console output: https://syzkaller.appspot.com/x/log.txt?x=10800114600000 IMPORTANT: if you fix the bug, please add the following tag to the commit: Reported-by: syzbot+23d9570edec63669d890(a)syzkaller.appspotmail.com Fixes: 7c85c448e7d7 ("selftests/bpf: test_verifier, check bpf_map_lookup_elem access in bpf prog") kasan: CONFIG_KASAN_INLINE enabled kasan: GPF could be caused by NULL-ptr deref or user memory access general protection fault: 0000 [#1] PREEMPT SMP KASAN CPU: 0 PID: 9098 Comm: syz-executor.2 Not tainted 5.2.0+ #65 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 RIP: 0010:tls_setsockopt+0x87/0x8c0 net/tls/tls_main.c:592 Code: 80 3c 02 00 0f 85 a5 07 00 00 49 8b 84 24 b0 06 00 00 48 ba 00 00 00 00 00 fc ff df 48 8d b8 88 00 00 00 48 89 f9 48 c1 e9 03 <80> 3c 11 00 0f 85 85 07 00 00 41 89 d8 4c 89 f1 44 89 ea 44 89 fe RSP: 0018:ffff88809b2e7d30 EFLAGS: 00010206 RAX: 0000000000000000 RBX: 0000000000000004 RCX: 0000000000000011 RDX: dffffc0000000000 RSI: ffffffff86186a84 RDI: 0000000000000088 RBP: ffff88809b2e7d80 R08: ffff888098896480 R09: ffff888098896d08 R10: ffffed1015d06c83 R11: ffff8880ae83641b R12: ffff888099d86d00 R13: 000000000000001f R14: 0000000020000340 R15: 0000000000000006 FS: 00007f29b1f6b700(0000) GS:ffff8880ae800000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007ffb5492d518 CR3: 00000000a49ea000 CR4: 00000000001406f0 Call Trace: sock_common_setsockopt+0x94/0xd0 net/core/sock.c:3130 __sys_setsockopt+0x253/0x4b0 net/socket.c:2080 __do_sys_setsockopt net/socket.c:2096 [inline] __se_sys_setsockopt net/socket.c:2093 [inline] __x64_sys_setsockopt+0xbe/0x150 net/socket.c:2093 do_syscall_64+0xfd/0x680 arch/x86/entry/common.c:296 entry_SYSCALL_64_after_hwframe+0x49/0xbe RIP: 0033:0x459819 Code: fd b7 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 cb b7 fb ff c3 66 2e 0f 1f 84 00 00 00 00 RSP: 002b:00007f29b1f6ac78 EFLAGS: 00000246 ORIG_RAX: 0000000000000036 RAX: ffffffffffffffda RBX: 0000000000000005 RCX: 0000000000459819 RDX: 000000000000001f RSI: 0000000000000006 RDI: 0000000000000005 RBP: 000000000075bf20 R08: 0000000000000004 R09: 0000000000000000 R10: 0000000020000340 R11: 0000000000000246 R12: 00007f29b1f6b6d4 R13: 00000000004c7cb9 R14: 00000000004dd878 R15: 00000000ffffffff Modules linked in: ---[ end trace 1e30f09ab6b57d8c ]--- RIP: 0010:tls_setsockopt+0x87/0x8c0 net/tls/tls_main.c:592 Code: 80 3c 02 00 0f 85 a5 07 00 00 49 8b 84 24 b0 06 00 00 48 ba 00 00 00 00 00 fc ff df 48 8d b8 88 00 00 00 48 89 f9 48 c1 e9 03 <80> 3c 11 00 0f 85 85 07 00 00 41 89 d8 4c 89 f1 44 89 ea 44 89 fe RSP: 0018:ffff88809b2e7d30 EFLAGS: 00010206 RAX: 0000000000000000 RBX: 0000000000000004 RCX: 0000000000000011 RDX: dffffc0000000000 RSI: ffffffff86186a84 RDI: 0000000000000088 RBP: ffff88809b2e7d80 R08: ffff888098896480 R09: ffff888098896d08 R10: ffffed1015d06c83 R11: ffff8880ae83641b R12: ffff888099d86d00 R13: 000000000000001f R14: 0000000020000340 R15: 0000000000000006 FS: 00007f29b1f6b700(0000) GS:ffff8880ae800000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000007126b4 CR3: 00000000a49ea000 CR4: 00000000001406f0 --- This bug is generated by a bot. It may contain errors. See https://goo.gl/tpsmEJ for more information about syzbot. syzbot engineers can be reached at syzkaller(a)googlegroups.com. syzbot will keep track of this bug report. See: https://goo.gl/tpsmEJ#status for how to communicate with syzbot. For information about bisection process see: https://goo.gl/tpsmEJ#bisection syzbot can test patches for this bug, for details see: https://goo.gl/tpsmEJ#testing-patches

5 years, 4 months

2
1
0 0

[PATCH v9 00/18] kunit: introduce KUnit, the Linux kernel unit testing framework

by Brendan Higgins

## TL;DR This new patch set only contains a very minor change to address a sparse warning in the PROC SYSCTL KUnit test. Otherwise this patchset is identical to the previous. As I mentioned in the previous patchset, all patches now have acks and reviews. ## Background This patch set proposes KUnit, a lightweight unit testing and mocking framework for the Linux kernel. Unlike Autotest and kselftest, KUnit is a true unit testing framework; it does not require installing the kernel on a test machine or in a VM (however, KUnit still allows you to run tests on test machines or in VMs if you want[1]) and does not require tests to be written in userspace running on a host kernel. Additionally, KUnit is fast: From invocation to completion KUnit can run several dozen tests in about a second. Currently, the entire KUnit test suite for KUnit runs in under a second from the initial invocation (build time excluded). KUnit is heavily inspired by JUnit, Python's unittest.mock, and Googletest/Googlemock for C++. KUnit provides facilities for defining unit test cases, grouping related test cases into test suites, providing common infrastructure for running tests, mocking, spying, and much more. ### What's so special about unit testing? A unit test is supposed to test a single unit of code in isolation, hence the name. There should be no dependencies outside the control of the test; this means no external dependencies, which makes tests orders of magnitudes faster. Likewise, since there are no external dependencies, there are no hoops to jump through to run the tests. Additionally, this makes unit tests deterministic: a failing unit test always indicates a problem. Finally, because unit tests necessarily have finer granularity, they are able to test all code paths easily solving the classic problem of difficulty in exercising error handling code. ### Is KUnit trying to replace other testing frameworks for the kernel? No. Most existing tests for the Linux kernel are end-to-end tests, which have their place. A well tested system has lots of unit tests, a reasonable number of integration tests, and some end-to-end tests. KUnit is just trying to address the unit test space which is currently not being addressed. ### More information on KUnit There is a bunch of documentation near the end of this patch set that describes how to use KUnit and best practices for writing unit tests. For convenience I am hosting the compiled docs here[2]. Additionally for convenience, I have applied these patches to a branch[3]. The repo may be cloned with: git clone https://kunit.googlesource.com/linux This patchset is on the kunit/rfc/v5.2/v9 branch. ## Changes Since Last Version Like I said in the TL;DR, there is only one minor change since the previous revision. That change only affects patch 17/18; it addresses a sparse warning in the PROC SYSCTL unit test. Thanks to Masahiro for applying previous patches to a branch in his kbuild tree and running sparse and other static analysis tools against my patches. [1] https://google.github.io/kunit-docs/third_party/kernel/docs/usage.html#kuni… [2] https://google.github.io/kunit-docs/third_party/kernel/docs/ [3] https://kunit.googlesource.com/linux/+/kunit/rfc/v5.2/v9 -- 2.22.0.410.gd8fdbe21b5-goog

5 years, 4 months

4
56
0 0

[PATCH v19 00/15] arm64: untag user pointers passed to the kernel

by Andrey Konovalov

=== Overview arm64 has a feature called Top Byte Ignore, which allows to embed pointer tags into the top byte of each pointer. Userspace programs (such as HWASan, a memory debugging tool [1]) might use this feature and pass tagged user pointers to the kernel through syscalls or other interfaces. Right now the kernel is already able to handle user faults with tagged pointers, due to these patches: 1. 81cddd65 ("arm64: traps: fix userspace cache maintenance emulation on a tagged pointer") 2. 7dcd9dd8 ("arm64: hw_breakpoint: fix watchpoint matching for tagged pointers") 3. 276e9327 ("arm64: entry: improve data abort handling of tagged pointers") This patchset extends tagged pointer support to syscall arguments. As per the proposed ABI change [3], tagged pointers are only allowed to be passed to syscalls when they point to memory ranges obtained by anonymous mmap() or sbrk() (see the patchset [3] for more details). For non-memory syscalls this is done by untaging user pointers when the kernel performs pointer checking to find out whether the pointer comes from userspace (most notably in access_ok). The untagging is done only when the pointer is being checked, the tag is preserved as the pointer makes its way through the kernel and stays tagged when the kernel dereferences the pointer when perfoming user memory accesses. The mmap and mremap (only new_addr) syscalls do not currently accept tagged addresses. Architectures may interpret the tag as a background colour for the corresponding vma. Other memory syscalls (mprotect, etc.) don't do user memory accesses but rather deal with memory ranges, and untagged pointers are better suited to describe memory ranges internally. Thus for memory syscalls we untag pointers completely when they enter the kernel. === Other approaches One of the alternative approaches to untagging that was considered is to completely strip the pointer tag as the pointer enters the kernel with some kind of a syscall wrapper, but that won't work with the countless number of different ioctl calls. With this approach we would need a custom wrapper for each ioctl variation, which doesn't seem practical. An alternative approach to untagging pointers in memory syscalls prologues is to inspead allow tagged pointers to be passed to find_vma() (and other vma related functions) and untag them there. Unfortunately, a lot of find_vma() callers then compare or subtract the returned vma start and end fields against the pointer that was being searched. Thus this approach would still require changing all find_vma() callers. === Testing The following testing approaches has been taken to find potential issues with user pointer untagging: 1. Static testing (with sparse [2] and separately with a custom static analyzer based on Clang) to track casts of __user pointers to integer types to find places where untagging needs to be done. 2. Static testing with grep to find parts of the kernel that call find_vma() (and other similar functions) or directly compare against vm_start/vm_end fields of vma. 3. Static testing with grep to find parts of the kernel that compare user pointers with TASK_SIZE or other similar consts and macros. 4. Dynamic testing: adding BUG_ON(has_tag(addr)) to find_vma() and running a modified syzkaller version that passes tagged pointers to the kernel. Based on the results of the testing the requried patches have been added to the patchset. === Notes This patchset is meant to be merged together with "arm64 relaxed ABI" [3]. This patchset is a prerequisite for ARM's memory tagging hardware feature support [4]. This patchset has been merged into the Pixel 2 & 3 kernel trees and is now being used to enable testing of Pixel phones with HWASan. Thanks! [1] http://clang.llvm.org/docs/HardwareAssistedAddressSanitizerDesign.html [2] https://github.com/lucvoo/sparse-dev/commit/5f960cb10f56ec2017c128ef9d16060… [3] https://lkml.org/lkml/2019/6/12/745 [4] https://community.arm.com/processors/b/blog/posts/arm-a-profile-architectur… === History Changes in v19: - Rebased onto 7b5cf701 (5.3-rc1+). Changes in v18: - Reverted the selftest back to not using the LD_PRELOAD approach. - Added prctl(PR_SET_TAGGED_ADDR_CTRL) call to the selftest. - Reworded the patch descriptions to make them less oriented on arm64 only. - Catalin's patch: "I added a Kconfig option and dropped the prctl args zero check. There is some minor clean-up as well". Changes in v17: - The "uaccess: add noop untagged_addr definition" patch is dropped, as it was merged into upstream named as "uaccess: add noop untagged_addr definition". - Merged "mm, arm64: untag user pointers in do_pages_move" into "mm, arm64: untag user pointers passed to memory syscalls". - Added "arm64: Introduce prctl() options to control the tagged user addresses ABI" patch from Catalin. - Add tags_lib.so to tools/testing/selftests/arm64/.gitignore. - Added a comment clarifying untagged in mremap. - Moved untagging back into mlx4_get_umem_mr() for the IB patch. Changes in v16: - Moved untagging for memory syscalls from arm64 wrappers back to generic code. - Dropped untagging for the following memory syscalls: brk, mmap, munmap; mremap (only dropped for new_address); mmap_pgoff (not used on arm64); remap_file_pages (deprecated); shmat, shmdt (work on shared memory). - Changed kselftest to LD_PRELOAD a shared library that overrides malloc to return tagged pointers. - Rebased onto 5.2-rc3. Changes in v15: - Removed unnecessary untagging from radeon_ttm_tt_set_userptr(). - Removed unnecessary untagging from amdgpu_ttm_tt_set_userptr(). - Moved untagging to validate_range() in userfaultfd code. - Moved untagging to ib_uverbs_(re)reg_mr() from mlx4_get_umem_mr(). - Rebased onto 5.1. Changes in v14: - Moved untagging for most memory syscalls to an arm64 specific implementation, instead of doing that in the common code. - Dropped "net, arm64: untag user pointers in tcp_zerocopy_receive", since the provided user pointers don't come from an anonymous map and thus are not covered by this ABI relaxation. - Dropped "kernel, arm64: untag user pointers in prctl_set_mm*". - Moved untagging from __check_mem_type() to tee_shm_register(). - Updated untagging for the amdgpu and radeon drivers to cover the MMU notifier, as suggested by Felix. - Since this ABI relaxation doesn't actually allow tagged instruction pointers, dropped the following patches: - Dropped "tracing, arm64: untag user pointers in seq_print_user_ip". - Dropped "uprobes, arm64: untag user pointers in find_active_uprobe". - Dropped "bpf, arm64: untag user pointers in stack_map_get_build_id_offset". - Rebased onto 5.1-rc7 (37624b58). Changes in v13: - Simplified untagging in tcp_zerocopy_receive(). - Looked at find_vma() callers in drivers/, which allowed to identify a few other places where untagging is needed. - Added patch "mm, arm64: untag user pointers in get_vaddr_frames". - Added patch "drm/amdgpu, arm64: untag user pointers in amdgpu_ttm_tt_get_user_pages". - Added patch "drm/radeon, arm64: untag user pointers in radeon_ttm_tt_pin_userptr". - Added patch "IB/mlx4, arm64: untag user pointers in mlx4_get_umem_mr". - Added patch "media/v4l2-core, arm64: untag user pointers in videobuf_dma_contig_user_get". - Added patch "tee/optee, arm64: untag user pointers in check_mem_type". - Added patch "vfio/type1, arm64: untag user pointers". Changes in v12: - Changed untagging in tcp_zerocopy_receive() to also untag zc->address. - Fixed untagging in prctl_set_mm* to only untag pointers for vma lookups and validity checks, but leave them as is for actual user space accesses. - Updated the link to the v2 of the "arm64 relaxed ABI" patchset [3]. - Dropped the documentation patch, as the "arm64 relaxed ABI" patchset [3] handles that. Changes in v11: - Added "uprobes, arm64: untag user pointers in find_active_uprobe" patch. - Added "bpf, arm64: untag user pointers in stack_map_get_build_id_offset" patch. - Fixed "tracing, arm64: untag user pointers in seq_print_user_ip" to correctly perform subtration with a tagged addr. - Moved untagged_addr() from SYSCALL_DEFINE3(mprotect) and SYSCALL_DEFINE4(pkey_mprotect) to do_mprotect_pkey(). - Moved untagged_addr() definition for other arches from include/linux/memory.h to include/linux/mm.h. - Changed untagging in strn*_user() to perform userspace accesses through tagged pointers. - Updated the documentation to mention that passing tagged pointers to memory syscalls is allowed. - Updated the test to use malloc'ed memory instead of stack memory. Changes in v10: - Added "mm, arm64: untag user pointers passed to memory syscalls" back. - New patch "fs, arm64: untag user pointers in fs/userfaultfd.c". - New patch "net, arm64: untag user pointers in tcp_zerocopy_receive". - New patch "kernel, arm64: untag user pointers in prctl_set_mm*". - New patch "tracing, arm64: untag user pointers in seq_print_user_ip". Changes in v9: - Rebased onto 4.20-rc6. - Used u64 instead of __u64 in type casts in the untagged_addr macro for arm64. - Added braces around (addr) in the untagged_addr macro for other arches. Changes in v8: - Rebased onto 65102238 (4.20-rc1). - Added a note to the cover letter on why syscall wrappers/shims that untag user pointers won't work. - Added a note to the cover letter that this patchset has been merged into the Pixel 2 kernel tree. - Documentation fixes, in particular added a list of syscalls that don't support tagged user pointers. Changes in v7: - Rebased onto 17b57b18 (4.19-rc6). - Dropped the "arm64: untag user address in __do_user_fault" patch, since the existing patches already handle user faults properly. - Dropped the "usb, arm64: untag user addresses in devio" patch, since the passed pointer must come from a vma and therefore be untagged. - Dropped the "arm64: annotate user pointers casts detected by sparse" patch (see the discussion to the replies of the v6 of this patchset). - Added more context to the cover letter. - Updated Documentation/arm64/tagged-pointers.txt. Changes in v6: - Added annotations for user pointer casts found by sparse. - Rebased onto 050cdc6c (4.19-rc1+). Changes in v5: - Added 3 new patches that add untagging to places found with static analysis. - Rebased onto 44c929e1 (4.18-rc8). Changes in v4: - Added a selftest for checking that passing tagged pointers to the kernel succeeds. - Rebased onto 81e97f013 (4.18-rc1+). Changes in v3: - Rebased onto e5c51f30 (4.17-rc6+). - Added linux-arch@ to the list of recipients. Changes in v2: - Rebased onto 2d618bdf (4.17-rc3+). - Removed excessive untagging in gup.c. - Removed untagging pointers returned from __uaccess_mask_ptr. Changes in v1: - Rebased onto 4.17-rc1. Changes in RFC v2: - Added "#ifndef untagged_addr..." fallback in linux/uaccess.h instead of defining it for each arch individually. - Updated Documentation/arm64/tagged-pointers.txt. - Dropped "mm, arm64: untag user addresses in memory syscalls". - Rebased onto 3eb2ce82 (4.16-rc7). Signed-off-by: Andrey Konovalov <andreyknvl(a)google.com> Andrey Konovalov (14): arm64: untag user pointers in access_ok and __uaccess_mask_ptr lib: untag user pointers in strn*_user mm: untag user pointers passed to memory syscalls mm: untag user pointers in mm/gup.c mm: untag user pointers in get_vaddr_frames fs/namespace: untag user pointers in copy_mount_options userfaultfd: untag user pointers drm/amdgpu: untag user pointers drm/radeon: untag user pointers in radeon_gem_userptr_ioctl IB/mlx4: untag user pointers in mlx4_get_umem_mr media/v4l2-core: untag user pointers in videobuf_dma_contig_user_get tee/shm: untag user pointers in tee_shm_register vfio/type1: untag user pointers in vaddr_get_pfn selftests, arm64: add a selftest for passing tagged pointers to kernel Catalin Marinas (1): arm64: Introduce prctl() options to control the tagged user addresses ABI arch/arm64/Kconfig | 9 +++ arch/arm64/include/asm/processor.h | 8 ++ arch/arm64/include/asm/thread_info.h | 1 + arch/arm64/include/asm/uaccess.h | 12 ++- arch/arm64/kernel/process.c | 73 +++++++++++++++++++ .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 2 + drivers/gpu/drm/radeon/radeon_gem.c | 2 + drivers/infiniband/hw/mlx4/mr.c | 7 +- drivers/media/v4l2-core/videobuf-dma-contig.c | 9 ++- drivers/tee/tee_shm.c | 1 + drivers/vfio/vfio_iommu_type1.c | 2 + fs/namespace.c | 2 +- fs/userfaultfd.c | 22 +++--- include/uapi/linux/prctl.h | 5 ++ kernel/sys.c | 12 +++ lib/strncpy_from_user.c | 3 +- lib/strnlen_user.c | 3 +- mm/frame_vector.c | 2 + mm/gup.c | 4 + mm/madvise.c | 2 + mm/mempolicy.c | 3 + mm/migrate.c | 2 +- mm/mincore.c | 2 + mm/mlock.c | 4 + mm/mprotect.c | 2 + mm/mremap.c | 7 ++ mm/msync.c | 2 + tools/testing/selftests/arm64/.gitignore | 1 + tools/testing/selftests/arm64/Makefile | 11 +++ .../testing/selftests/arm64/run_tags_test.sh | 12 +++ tools/testing/selftests/arm64/tags_test.c | 29 ++++++++ 32 files changed, 233 insertions(+), 25 deletions(-) create mode 100644 tools/testing/selftests/arm64/.gitignore create mode 100644 tools/testing/selftests/arm64/Makefile create mode 100755 tools/testing/selftests/arm64/run_tags_test.sh create mode 100644 tools/testing/selftests/arm64/tags_test.c -- 2.22.0.709.g102302147b-goog

5 years, 4 months

11
50
0 0

[PATCH v3 1/2] tests: move common definitions and functions into pidfd.h

by Suren Baghdasaryan

Move definitions and functions used across different pidfd tests into pidfd.h header. Signed-off-by: Suren Baghdasaryan <surenb(a)google.com> Reviewed-by: Christian Brauner <christian(a)brauner.io> --- Changes since v1 (https://lore.kernel.org/linux-kselftest/20190723173907.196488-1-surenb@goog…) - Move common test definitions and functions into pidfd.h as per Christian Brauner - Code cleanup and additional error handling as per Christian Brauner Changes since v2 (https://lore.kernel.org/linux-kselftest/20190725002204.185225-2-surenb@goog…) - Added waitpid to prevent zombies as per Yann Droneaud Patches are based on v5.3-rc1 in Linus' tree. --- tools/testing/selftests/pidfd/pidfd.h | 18 ++++++++++++++++++ .../testing/selftests/pidfd/pidfd_open_test.c | 5 ----- tools/testing/selftests/pidfd/pidfd_test.c | 10 ---------- 3 files changed, 18 insertions(+), 15 deletions(-) diff --git a/tools/testing/selftests/pidfd/pidfd.h b/tools/testing/selftests/pidfd/pidfd.h index 8452e910463f..db4377af6be7 100644 --- a/tools/testing/selftests/pidfd/pidfd.h +++ b/tools/testing/selftests/pidfd/pidfd.h @@ -16,6 +16,14 @@ #include "../kselftest.h" +#ifndef __NR_pidfd_open +#define __NR_pidfd_open -1 +#endif + +#ifndef __NR_pidfd_send_signal +#define __NR_pidfd_send_signal -1 +#endif + /* * The kernel reserves 300 pids via RESERVED_PIDS in kernel/pid.c * That means, when it wraps around any pid < 300 will be skipped. @@ -53,5 +61,15 @@ int wait_for_pid(pid_t pid) return WEXITSTATUS(status); } +static inline int sys_pidfd_open(pid_t pid, unsigned int flags) +{ + return syscall(__NR_pidfd_open, pid, flags); +} + +static inline int sys_pidfd_send_signal(int pidfd, int sig, siginfo_t *info, + unsigned int flags) +{ + return syscall(__NR_pidfd_send_signal, pidfd, sig, info, flags); +} #endif /* __PIDFD_H */ diff --git a/tools/testing/selftests/pidfd/pidfd_open_test.c b/tools/testing/selftests/pidfd/pidfd_open_test.c index 0377133dd6dc..b9fe75fc3e51 100644 --- a/tools/testing/selftests/pidfd/pidfd_open_test.c +++ b/tools/testing/selftests/pidfd/pidfd_open_test.c @@ -22,11 +22,6 @@ #include "pidfd.h" #include "../kselftest.h" -static inline int sys_pidfd_open(pid_t pid, unsigned int flags) -{ - return syscall(__NR_pidfd_open, pid, flags); -} - static int safe_int(const char *numstr, int *converted) { char *err = NULL; diff --git a/tools/testing/selftests/pidfd/pidfd_test.c b/tools/testing/selftests/pidfd/pidfd_test.c index 7eaa8a3de262..17b2fd621726 100644 --- a/tools/testing/selftests/pidfd/pidfd_test.c +++ b/tools/testing/selftests/pidfd/pidfd_test.c @@ -21,10 +21,6 @@ #include "pidfd.h" #include "../kselftest.h" -#ifndef __NR_pidfd_send_signal -#define __NR_pidfd_send_signal -1 -#endif - #define str(s) _str(s) #define _str(s) #s #define CHILD_THREAD_MIN_WAIT 3 /* seconds */ @@ -47,12 +43,6 @@ static pid_t pidfd_clone(int flags, int *pidfd, int (*fn)(void *)) #endif } -static inline int sys_pidfd_send_signal(int pidfd, int sig, siginfo_t *info, - unsigned int flags) -{ - return syscall(__NR_pidfd_send_signal, pidfd, sig, info, flags); -} - static int signal_received; static void set_signal_received_on_sigusr1(int sig) -- 2.22.0.709.g102302147b-goog

5 years, 4 months

4
8
0 0

[PATCH] KVM: selftests: Update gitignore file for latest changes

by Thomas Huth

The kvm_create_max_vcpus test has been moved to the main directory, and sync_regs_test is now available on s390x, too. Signed-off-by: Thomas Huth <thuth(a)redhat.com> --- tools/testing/selftests/kvm/.gitignore | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/kvm/.gitignore b/tools/testing/selftests/kvm/.gitignore index 41266af0d3dc..b35da375530a 100644 --- a/tools/testing/selftests/kvm/.gitignore +++ b/tools/testing/selftests/kvm/.gitignore @@ -1,7 +1,7 @@ +/s390x/sync_regs_test /x86_64/cr4_cpuid_sync_test /x86_64/evmcs_test /x86_64/hyperv_cpuid -/x86_64/kvm_create_max_vcpus /x86_64/mmio_warning_test /x86_64/platform_info_test /x86_64/set_sregs_test @@ -13,3 +13,4 @@ /x86_64/vmx_tsc_adjust_test /clear_dirty_log_test /dirty_log_test +/kvm_create_max_vcpus -- 2.21.0

5 years, 4 months

3
2
0 0

[PATCH v3 0/3] KVM: selftests: Enable ucall and dirty_log_test on s390x

by Thomas Huth

Implement the ucall() interface on s390x to be able to use the dirty_log_test KVM selftest on s390x, too. v3: - Fix compilation issue on aarch64 (thanks to Andrew for testing it!) - Added Reviewed-bys v2: - Split up ucall.c into architecture specific files - Removed some #ifdef __s390x__ in the dirty_log patch Thomas Huth (3): KVM: selftests: Split ucall.c into architecture specific files KVM: selftests: Implement ucall() for s390x KVM: selftests: Enable dirty_log_test on s390x tools/testing/selftests/kvm/Makefile | 9 +- tools/testing/selftests/kvm/dirty_log_test.c | 61 ++++++- .../testing/selftests/kvm/include/kvm_util.h | 8 +- .../testing/selftests/kvm/lib/aarch64/ucall.c | 112 +++++++++++++ tools/testing/selftests/kvm/lib/s390x/ucall.c | 56 +++++++ tools/testing/selftests/kvm/lib/ucall.c | 157 ------------------ .../testing/selftests/kvm/lib/x86_64/ucall.c | 56 +++++++ .../selftests/kvm/s390x/sync_regs_test.c | 6 +- 8 files changed, 287 insertions(+), 178 deletions(-) create mode 100644 tools/testing/selftests/kvm/lib/aarch64/ucall.c create mode 100644 tools/testing/selftests/kvm/lib/s390x/ucall.c delete mode 100644 tools/testing/selftests/kvm/lib/ucall.c create mode 100644 tools/testing/selftests/kvm/lib/x86_64/ucall.c -- 2.21.0

5 years, 4 months

3
5
0 0

2024

2023

2022

2021

2020

2019

2018

2017

Linux-kselftest-mirror July 2019