- Linux-kselftest-mirror - lists.linaro.org

[REGRESSION] lkft kselftest for next-20211011

by lkft＠linaro.org

## Build * kernel: 5.15.0-rc4 * git: https://gitlab.com/Linaro/lkft/mirrors/next/linux-next * git branch: master * git commit: d3134eb5de8546a214c028fb7195e764b89da7d4 * git describe: next-20211011 * test details: https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20211011 ## Regressions (compared to next-20211006) * hi6220-hikey, kselftest-timers - timers.nsleep-lat - timers.set-timer-lat * qemu_i386, kselftest-pidfd - pidfd.pidfd_test * qemu_x86_64, kselftest-pidfd - pidfd.pidfd_test Reported-by: Linux Kernel Functional Testing <lkft(a)linaro.org> ## Fixes (compared to next-20211006) * dragonboard-410c, kselftest-gpio - gpio.gpio-mockup.sh * qemu_i386, kselftest-gpio - gpio.gpio-mockup.sh * qemu_i386, kselftest-seccomp - seccomp.seccomp_benchmark * qemu_x86_64, kselftest-seccomp - seccomp.seccomp_benchmark * x86, kselftest-lkdtm - lkdtm.SLAB_FREE_DOUBLE.sh ## Test result summary total: 2802, pass: 1736, fail: 237, skip: 829, xfail: 0 ## Build Summary ## Test suites summary * kselftest-android * kselftest-bpf * kselftest-breakpoints * kselftest-capabilities * kselftest-cgroup * kselftest-clone3 * kselftest-core * kselftest-cpu-hotplug * kselftest-cpufreq * kselftest-drivers * kselftest-efivarfs * kselftest-filesystems * kselftest-firmware * kselftest-fpu * kselftest-futex * kselftest-gpio * kselftest-intel_pstate * kselftest-ipc * kselftest-ir * kselftest-kcmp * kselftest-kexec * kselftest-kvm * kselftest-lib * kselftest-livepatch * kselftest-lkdtm * kselftest-membarrier * kselftest-net * kselftest-netfilter * kselftest-nsfs * kselftest-openat2 * kselftest-pid_namespace * kselftest-pidfd * kselftest-proc * kselftest-pstore * kselftest-ptrace * kselftest-rseq * kselftest-rtc * kselftest-seccomp * kselftest-sigaltstack * kselftest-size * kselftest-splice * kselftest-static_keys * kselftest-sync * kselftest-sysctl * kselftest-tc-testing * kselftest-timens * kselftest-timers * kselftest-tmpfs * kselftest-tpm2 * kselftest-user * kselftest-vm * kselftest-x86 * kselftest-zram -- Linaro LKFT https://lkft.linaro.org

3 years, 7 months

1
0
0 0

[REGRESSION] lkft kselftest for next-20211008

by lkft＠linaro.org

## Build * kernel: 5.15.0-rc4 * git: https://gitlab.com/Linaro/lkft/mirrors/next/linux-next * git branch: master * git commit: 683f29b781aeaab6bf302eeb2ef08a5e5f9d8a27 * git describe: next-20211008 * test details: https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20211008 ## Regressions (compared to next-20211006) * qemu_x86_64, kselftest-pidfd - pidfd.pidfd_test * x86, kselftest-kvm - kvm.hyperv_clock Reported-by: Linux Kernel Functional Testing <lkft(a)linaro.org> ## Fixes (compared to next-20211006) * dragonboard-410c, kselftest-gpio - gpio.gpio-mockup.sh * hi6220-hikey, kselftest-proc - proc.proc-uptime-001 * qemu_x86_64, kselftest-seccomp - seccomp.seccomp_benchmark * x86, kselftest-lkdtm - lkdtm.SLAB_FREE_DOUBLE.sh ## Test result summary total: 1652, pass: 1005, fail: 105, skip: 542, xfail: 0 ## Build Summary ## Test suites summary * kselftest-android * kselftest-arm64 * kselftest-arm64/arm64.btitest.bti_c_func * kselftest-arm64/arm64.btitest.bti_j_func * kselftest-arm64/arm64.btitest.bti_jc_func * kselftest-arm64/arm64.btitest.bti_none_func * kselftest-arm64/arm64.btitest.nohint_func * kselftest-arm64/arm64.btitest.paciasp_func * kselftest-arm64/arm64.nobtitest.bti_c_func * kselftest-arm64/arm64.nobtitest.bti_j_func * kselftest-arm64/arm64.nobtitest.bti_jc_func * kselftest-arm64/arm64.nobtitest.bti_none_func * kselftest-arm64/arm64.nobtitest.nohint_func * kselftest-arm64/arm64.nobtitest.paciasp_func * kselftest-bpf * kselftest-breakpoints * kselftest-capabilities * kselftest-cgroup * kselftest-clone3 * kselftest-core * kselftest-cpu-hotplug * kselftest-cpufreq * kselftest-drivers * kselftest-efivarfs * kselftest-filesystems * kselftest-firmware * kselftest-fpu * kselftest-futex * kselftest-gpio * kselftest-intel_pstate * kselftest-ipc * kselftest-ir * kselftest-kcmp * kselftest-kexec * kselftest-kvm * kselftest-lib * kselftest-livepatch * kselftest-lkdtm * kselftest-membarrier * kselftest-net * kselftest-netfilter * kselftest-nsfs * kselftest-openat2 * kselftest-pid_namespace * kselftest-pidfd * kselftest-proc * kselftest-pstore * kselftest-ptrace * kselftest-rseq * kselftest-rtc * kselftest-seccomp * kselftest-sigaltstack * kselftest-size * kselftest-splice * kselftest-static_keys * kselftest-sync * kselftest-sysctl * kselftest-tc-testing * kselftest-timens * kselftest-timers * kselftest-tmpfs * kselftest-tpm2 * kselftest-user * kselftest-vm * kselftest-x86 * kselftest-zram -- Linaro LKFT https://lkft.linaro.org

3 years, 7 months

1
0
0 0

[RFC][PATCH] selftests/vm/transhuge-stress: fix ram size thinko

by George G. Davis

From: "George G. Davis" <davis.george(a)siemens.com> When executing transhuge-stress with an argument to specify the virtual memory size for testing, the ram size is reported as 0, e.g. transhuge-stress 384 thp-mmap: allocate 192 transhuge pages, using 384 MiB virtual memory and 0 MiB of ram thp-mmap: 0.184 s/loop, 0.957 ms/page, 2090.265 MiB/s 192 succeed, 0 failed This appears to be due to a thinko in commit 0085d61fe05e ("selftests/vm/transhuge-stress: stress test for memory compaction"), where, at a guess, the intent was to base "xyz MiB of ram" on `ram` size. Here are results after using `ram` size: thp-mmap: allocate 192 transhuge pages, using 384 MiB virtual memory and 14 MiB of ram Fixes: 0085d61fe05e ("selftests/vm/transhuge-stress: stress test for memory compaction") Signed-off-by: George G. Davis <davis.george(a)siemens.com> --- tools/testing/selftests/vm/transhuge-stress.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/testing/selftests/vm/transhuge-stress.c b/tools/testing/selftests/vm/transhuge-stress.c index fd7f1b4a96f9..5e4c036f6ad3 100644 --- a/tools/testing/selftests/vm/transhuge-stress.c +++ b/tools/testing/selftests/vm/transhuge-stress.c @@ -79,7 +79,7 @@ int main(int argc, char **argv) warnx("allocate %zd transhuge pages, using %zd MiB virtual memory" " and %zd MiB of ram", len >> HPAGE_SHIFT, len >> 20, - len >> (20 + HPAGE_SHIFT - PAGE_SHIFT - 1)); + ram >> (20 + HPAGE_SHIFT - PAGE_SHIFT - 1)); pagemap_fd = open("/proc/self/pagemap", O_RDONLY); if (pagemap_fd < 0) -- 2.17.1

3 years, 7 months

3
3
0 0

[PATCH v1] selftests/vm: make MADV_POPULATE_(READ|WRITE) use in-tree headers

by David Hildenbrand

The madv_populate selftest currently builds with a warning when the local installed headers (via the distribution) don't include MADV_POPULATE_READ and MADV_POPULATE_WRITE. The warning is correct, because the test cannot locate the necessary header. Reason is that the in-tree installed headers (usr/include) have a "linux" instead of a "sys" subdirectory. Including "linux/mman.h" instead of "sys/mman.h" doesn't work (e.g., mmap() and madvise() are not defined that way). The only thing that seems to work is including "linux/mman.h" in addition to "sys/mman.h". We can get rid of our availability check and simplify. Reported-by: Shuah Khan <skhan(a)linuxfoundation.org> Cc: Andrew Morton <akpm(a)linux-foundation.org> Cc: linux-mm(a)kvack.org Cc: linux-kselftest(a)vger.kernel.org Signed-off-by: David Hildenbrand <david(a)redhat.com> --- tools/testing/selftests/vm/madv_populate.c | 15 +-------------- 1 file changed, 1 insertion(+), 14 deletions(-) diff --git a/tools/testing/selftests/vm/madv_populate.c b/tools/testing/selftests/vm/madv_populate.c index b959e4ebdad4..3ee0e8275600 100644 --- a/tools/testing/selftests/vm/madv_populate.c +++ b/tools/testing/selftests/vm/madv_populate.c @@ -14,12 +14,11 @@ #include <unistd.h> #include <errno.h> #include <fcntl.h> +#include <linux/mman.h> #include <sys/mman.h> #include "../kselftest.h" -#if defined(MADV_POPULATE_READ) && defined(MADV_POPULATE_WRITE) - /* * For now, we're using 2 MiB of private anonymous memory for all tests. */ @@ -328,15 +327,3 @@ int main(int argc, char **argv) err, ksft_test_num()); return ksft_exit_pass(); } - -#else /* defined(MADV_POPULATE_READ) && defined(MADV_POPULATE_WRITE) */ - -#warning "missing MADV_POPULATE_READ or MADV_POPULATE_WRITE definition" - -int main(int argc, char **argv) -{ - ksft_print_header(); - ksft_exit_skip("MADV_POPULATE_READ or MADV_POPULATE_WRITE not defined\n"); -} - -#endif /* defined(MADV_POPULATE_READ) && defined(MADV_POPULATE_WRITE) */ -- 2.31.1

3 years, 7 months

1
0
0 0

selftests/vm madv_populate.c test

by Shuah Khan

Hi David, I am running into the following warning when try to build this test: madv_populate.c:334:2: warning: #warning "missing MADV_POPULATE_READ or MADV_POPULATE_WRITE definition" [-Wcpp] 334 | #warning "missing MADV_POPULATE_READ or MADV_POPULATE_WRITE definition" | ^~~~~~~ I see that the following handling is in place. However there is no other information to explain why the check is necessary. #if defined(MADV_POPULATE_READ) && defined(MADV_POPULATE_WRITE) #else /* defined(MADV_POPULATE_READ) && defined(MADV_POPULATE_WRITE) */ #warning "missing MADV_POPULATE_READ or MADV_POPULATE_WRITE definition" I do see these defined in: include/uapi/asm-generic/mman-common.h:#define MADV_POPULATE_READ 22 include/uapi/asm-generic/mman-common.h:#define MADV_POPULATE_WRITE 23 Is this the case of missing include from madv_populate.c? thanks, -- Shuah

3 years, 7 months

2
12
0 0

[PATCH v1 00/38] arm64/sme: Initial support for the Scalable Matrix Extension

by Mark Brown

This series provides initial support for the ARMv9 Scalable Matrix Extension (SME). SME takes the approach used for vectors in SVE and extends this to provide architectural support for matrix operations. A more detailed overview can be found in [1]. For the kernel SME can be thought of as a series of features which are intended to be used together by applications but operate mostly orthogonally: - The ZA matrix register. - Streaming mode, in which ZA can be accessed and a subset of SVE features are available. - A second vector length, used for streaming mode SVE and ZA and controlled using a similar interface to that for SVE. - TPIDR2, a new userspace controllable system register intended for use by the C library for storing context related to the ZA ABI. A substantial part of the series is dedicated to refactoring the existing SVE support so that we don't need to duplicate code for handling vector lengths and the SVE registers, this involves creating an array of vector types and making the users take the vector type as a parameter. I'm not 100% happy with this but wasn't able to come up with anything better, duplicating code definitely felt like a bad idea so this felt like the least bad thing. If this approach makes sense to people it might make sense to split this off into a separate series and/or merge it while the rest is pending review to try to make things a little more digestable, the series is very large so it'd probably make things easier to digest if some of the preparatory refactoring could be merged before the rest is ready. One feature of the architecture of particular note is that switching to and from streaming mode may change the size of and invalidate the contents of the SVE registers, and when in streaming mode the FFR is not accessible. This complicates aspects of the ABI like signal handling and ptrace. This initial implementation is mainly intended to get the ABI in place, there are several areas which will be worked on going forwards - some of these will be blockers, others could be handled in followup serieses: - KVM is not currently supported and we depend on !KVM, in hopefully the next version I will add support for coexisting with KVM and then in a subsequent series implement real support for KVM guests. - It is likely some build configurations have issues, I've not fully checked this yet. In general testing is still ongoing, I anticipate finding and fixing some issues in the implementation. - No support is currently provided for scheduler control of SME or SME applications, given the size of the SME register state the context switch overhead may be noticable so this may be needed especially for real time applications. Similar concerns already exist for larger SVE vector lengths but are amplified for SME, particularly as the vector length increases. - There has been no work on optimising the performance of anything the kernel does. It is not expected that any systems will be encountered that support SME but not SVE, SME is an ARMv9 feature and SVE is mandatory for ARMv9. The code attempts to handle any such systems that are encountered but this hasn't been tested extensively. Due to dependencies on kselftest changes already upstreamed this series is based on for-next/kselftest in the arm64 tree. [1] https://community.arm.com/developer/ip-products/processors/b/processors-ip-… Mark Brown (38): arm64/fp: Reindent fpsimd_save() arm64/sve: Remove sve_load_from_fpsimd_state() arm64/sve: Make access to FFR optional arm64/sve: Rename find_supported_vector_length() arm64/sve: Use accessor functions for vector lengths in thread_struct arm64/sve: Put system wide vector length information into structs arm64/sve: Explicitly load vector length when restoring SVE state arm64/sve: Track vector lengths for tasks in an array arm64/sve: Make sysctl interface for SVE reusable by SME arm64/sve: Generalise vector length configuration prctl() for SME selftests: arm64: Parameterise ptrace vector length information arm64/sme: Provide ABI documentation for SME arm64/sme: System register and exception syndrome definitions arm64/sme: Define macros for manually encoding SME instructions arm64/sme: Early CPU setup for SME arm64/sme: Basic enumeration support arm64/sme: Identify supported SME vector lengths at boot arm64/sme: Implement sysctl to set the default vector length arm64/sme: Implement vector length configuration prctl()s arm64/sme: Implement support for TPIDR2 arm64/sme: Implement SVCR context switching arm64/sme: Implement streaming SVE context switching arm64/sme: Implement ZA context switching arm64/sme: Implement traps and syscall handling for SME arm64/sme: Implement streaming SVE signal handling arm64/sme: Implement ZA signal handling arm64/sme: Implement ptrace support for streaming mode SVE registers arm64/sme: Add ptrace support for ZA arm64/sme: Disable streaming mode and ZA when flushing CPU state arm64/sme: Save and restore streaming mode over EFI runtime calls arm64/sme: Provide Kconfig for SME kselftest/arm64: Add tests for TPIDR2 kselftest/arm64: Extend vector configuration API tests to cover SME kselftest/arm64: sme: Provide streaming mode SVE stress test kselftest/arm64: Add stress test for SME ZA context switching kselftest/arm64: signal: Add SME signal handling tests selftests: arm64: Add streaming SVE to SVE ptrace tests selftests: arm64: Add coverage for the ZA ptrace interface Documentation/arm64/elf_hwcaps.rst | 29 + Documentation/arm64/index.rst | 1 + Documentation/arm64/sme.rst | 427 +++++++++ Documentation/arm64/sve.rst | 62 +- arch/arm64/Kconfig | 11 + arch/arm64/include/asm/cpu.h | 4 + arch/arm64/include/asm/cpufeature.h | 18 + arch/arm64/include/asm/el2_setup.h | 36 + arch/arm64/include/asm/esr.h | 3 +- arch/arm64/include/asm/exception.h | 1 + arch/arm64/include/asm/fpsimd.h | 178 +++- arch/arm64/include/asm/fpsimdmacros.h | 94 +- arch/arm64/include/asm/hwcap.h | 7 + arch/arm64/include/asm/kvm_arm.h | 1 + arch/arm64/include/asm/processor.h | 67 +- arch/arm64/include/asm/sysreg.h | 53 ++ arch/arm64/include/asm/thread_info.h | 4 +- arch/arm64/include/uapi/asm/hwcap.h | 7 + arch/arm64/include/uapi/asm/ptrace.h | 69 +- arch/arm64/include/uapi/asm/sigcontext.h | 55 +- arch/arm64/kernel/cpufeature.c | 96 +- arch/arm64/kernel/cpuinfo.c | 12 + arch/arm64/kernel/entry-common.c | 10 + arch/arm64/kernel/entry-fpsimd.S | 63 +- arch/arm64/kernel/fpsimd.c | 900 ++++++++++++++---- arch/arm64/kernel/process.c | 19 +- arch/arm64/kernel/ptrace.c | 358 ++++++- arch/arm64/kernel/signal.c | 189 +++- arch/arm64/kernel/syscall.c | 49 +- arch/arm64/kernel/traps.c | 1 + arch/arm64/kvm/fpsimd.c | 3 +- arch/arm64/kvm/hyp/fpsimd.S | 6 +- arch/arm64/kvm/reset.c | 14 +- arch/arm64/tools/cpucaps | 1 + include/uapi/linux/elf.h | 2 + include/uapi/linux/prctl.h | 9 + kernel/sys.c | 6 + tools/testing/selftests/arm64/Makefile | 2 +- tools/testing/selftests/arm64/abi/.gitignore | 1 + tools/testing/selftests/arm64/abi/Makefile | 13 + tools/testing/selftests/arm64/abi/tpidr2.c | 204 ++++ tools/testing/selftests/arm64/fp/.gitignore | 4 + tools/testing/selftests/arm64/fp/Makefile | 12 +- tools/testing/selftests/arm64/fp/rdvl-sme.c | 14 + tools/testing/selftests/arm64/fp/rdvl.S | 16 + tools/testing/selftests/arm64/fp/rdvl.h | 1 + tools/testing/selftests/arm64/fp/ssve-stress | 59 ++ tools/testing/selftests/arm64/fp/sve-ptrace.c | 203 ++-- tools/testing/selftests/arm64/fp/sve-test.S | 30 + tools/testing/selftests/arm64/fp/vec-syscfg.c | 10 + tools/testing/selftests/arm64/fp/za-ptrace.c | 353 +++++++ tools/testing/selftests/arm64/fp/za-stress | 59 ++ tools/testing/selftests/arm64/fp/za-test.S | 545 +++++++++++ .../testing/selftests/arm64/signal/.gitignore | 2 + .../selftests/arm64/signal/test_signals.h | 2 + .../arm64/signal/test_signals_utils.c | 3 + .../testcases/fake_sigreturn_sme_change_vl.c | 92 ++ .../selftests/arm64/signal/testcases/sme_vl.c | 70 ++ .../arm64/signal/testcases/ssve_regs.c | 129 +++ 59 files changed, 4293 insertions(+), 396 deletions(-) create mode 100644 Documentation/arm64/sme.rst create mode 100644 tools/testing/selftests/arm64/abi/.gitignore create mode 100644 tools/testing/selftests/arm64/abi/Makefile create mode 100644 tools/testing/selftests/arm64/abi/tpidr2.c create mode 100644 tools/testing/selftests/arm64/fp/rdvl-sme.c create mode 100644 tools/testing/selftests/arm64/fp/ssve-stress create mode 100644 tools/testing/selftests/arm64/fp/za-ptrace.c create mode 100644 tools/testing/selftests/arm64/fp/za-stress create mode 100644 tools/testing/selftests/arm64/fp/za-test.S create mode 100644 tools/testing/selftests/arm64/signal/testcases/fake_sigreturn_sme_change_vl.c create mode 100644 tools/testing/selftests/arm64/signal/testcases/sme_vl.c create mode 100644 tools/testing/selftests/arm64/signal/testcases/ssve_regs.c base-commit: 8694e5e6388695195a32bd5746635ca166a8df56 -- 2.20.1

3 years, 7 months

6
68
0 0

[PATCH] selftests/ftrace: Stop tracing before searching non-exist event entry

by Masami Hiramatsu

Stop tracing before searching the pattern which is expected to not exist on the trace buffer. In some case, it will take too long and may not come back eternally because while searching the tracing data will be increased by the searching activity. I found this with enabling kernel debug options, like kmemleak, lockdep etc. and run it on qemu with 2 CPUs. It did not come back in 20 minutes and finally I need to interrupt it to stop. Signed-off-by: Masami Hiramatsu <mhiramat(a)kernel.org> --- .../ftrace/test.d/ftrace/func_profiler.tc | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tools/testing/selftests/ftrace/test.d/ftrace/func_profiler.tc b/tools/testing/selftests/ftrace/test.d/ftrace/func_profiler.tc index 1dbd766c0cd2..440f4d87aa4b 100644 --- a/tools/testing/selftests/ftrace/test.d/ftrace/func_profiler.tc +++ b/tools/testing/selftests/ftrace/test.d/ftrace/func_profiler.tc @@ -56,6 +56,9 @@ clear_trace sleep 1 echo "make sure something other than scheduler is being traced" + +echo 0 > tracing_on + if ! grep -v -e '^#' -e 'schedule' trace > /dev/null; then cat trace fail "no other functions besides schedule was found"

3 years, 7 months

2
1
0 0

Re: [PATCH v7 5/6] cgroup/cpuset: Update description of cpuset.cpus.partition in cgroup-v2.rst

by Michal Koutný

On Wed, Oct 06, 2021 at 02:21:03PM -0400, Waiman Long <llong(a)redhat.com> wrote: > Sorry for not following up with this patchset sooner as I was busy on other > tasks. Thanks for continuing with this. > 1) The "cpuset.cpus" is not empty and the list of CPUs are > exclusive, i.e. they are not shared by any of its siblings. > 2) The parent cgroup is a partition root. > 3) The "cpuset.cpus" is a subset of the union of parent's > "cpuset.cpus.effective" and offlined CPUs in parent's > "cpuset.cpus". > 4) There is no child cgroups with cpuset enabled. This avoids > cpu migrations of multiple cgroups simultaneously which can > be problematic. > > A partition, when enabled, can be in an invalid state. An example > is when its parent is also an invalid partition. You say: "it can only be enabled in a cgroup if all the following conditions are met.", "2) The parent cgroup is a partition root." and then the example: "A partition, when enabled, can be in an invalid state. An example is when its parent is also an invalid partition." But the first two statements imply you can't have enabled the partition in such a case. I think there is still mixup of partition validity conditions and transition conditions, yours would roughly divide into (not precisely, just to share my understanding): Validity conditions 1) The "cpuset.cpus" is not empty and the list of CPUs are exclusive, i.e. they are not shared by any of its siblings. 2) The parent cgroup is a partition root. Transition conditions: 3) The "cpuset.cpus" is a subset of the union of parent's "cpuset.cpus.effective" and offlined CPUs in parent's "cpuset.cpus". 4) There is no child cgroups with cpuset enabled. This avoids cpu migrations of multiple cgroups simultaneously which can be problematic. (I've put no. 3 into transition conditions because _after_ the transition parent's cpuset.cpus.effective are subtracted the new root's cpuset.cpus but I'd like to have something similar as a validity condition but I haven't come up with that yet.) I consider the following situation: r // all cpus 0-7 `- part1 cpus=0-3 root >partition ` subpart1 cpus=0-1 root >partition ` subpart2 cpus=2-3 root >partition `- other cpus=4-7 // member by default Both subpart1 and subpart2 are valid partition roots. Look at actions listed below (as alternatives, not a sequence): a) hotplug offlines cpu 3 - would part1 still be considered a valid root? - perhaps not - would subpart1 still be considered a valid root? - it could be, but its parent is invalid so no? - would subpart2 still be considered a valid root? - perhaps not b) administrative change writes 0-2 into part1 cpus - would part1 still be considered a valid root? - yes - would subpart1 still be considered a valid root? - yes - would subpart2 still be considered a valid root? - perhaps not c) administrative change writes 3-7 into `other` cpus - should this fail or invalidate a root partition part1? - perhaps fail since the same "owner" manages all siblings and should reduce part1 first The answers above are just my "natural" responses, the ideal may be different. The issue I want to illustrate is that if all the conditions are formed as transition conditions only, they can't be used to reason about hotplug or config changes (except for cpuset.cpus.partitions writes). What would help me with the understanding -- the invalid root partition is defined as 1) such a cgroup where no cpus are granted from the top (and thus has to fall back to ancestors) or 2) such a cgroup where cpus requested in cpuset.cpus can't be fulfilled (i.e. any missing invalidates)? Furthermore, another example (motivated by the patch 4/6) r // all cpus 0-7 `- part1 cpus=0-4 root >partition ` subpart1 cpus=0-1 root >partition ` subpart2 cpus=2-3 root >partition ` task `- other cpus=5-7 // member by default It's a valid and achievable state (even on v2 since cpuset is a threaded controller). a) cpu 4 is offlined - this should invalidate part1 (and propagate invalidation into subpart1 and subpart2). b) administrative write 0-3 into part1 cpus - should this invalidate part1 or be rejected? In conclusion, it'd be good to have validity conditions separate from transition conditions (since hotplug transition can't be rejected) and perhaps treat administrative changes from an ancestor equally as a hotplug. Thanks, Michal

3 years, 7 months

2
2
0 0

[PATCH v5 0/7] kernel.h further split

by Andy Shevchenko

v4: https://lore.kernel.org/linux-media/20211007154407.29746-1-andriy.shevchenk… v3: https://lore.kernel.org/linux-media/20211007150339.28910-1-andriy.shevchenk… v2: https://lore.kernel.org/linux-media/20211007095129.22037-1-andriy.shevchenk… The kernel.h is a set of something which is not related to each other and often used in non-crossed compilation units, especially when drivers need only one or two macro definitions from it. Here is the split of container_of(). The goals are the following: - untwist the dependency hell a bit - drop kernel.h inclusion where it's only used for container_of() In v5: - dropped code duplication (Miguel) - added necessary includes into container_of.h (Joe) - dropped other header shuffling in list.h (Jonathan) - added tag (Sakari) In v4: - dropped kobject.h change (Greg) - Cc'ed more people (as per v1) In v3: - split patch 2 to more patches (Greg) - excluded C changes (Herbert, Greg) - measured with kcbench, see below (Greg) Andy Shevchenko (7): kernel.h: Drop unneeded <linux/kernel.h> inclusion from other headers kernel.h: Split out container_of() and typeof_member() macros kunit: Replace kernel.h with the necessary inclusions list: Replace kernel.h with the necessary inclusions llist: Replace kernel.h with the necessary inclusions plist: Replace kernel.h with the necessary inclusions media: entity: Replace kernel.h with the necessary inclusions include/kunit/test.h | 13 ++++++++++-- include/linux/container_of.h | 40 ++++++++++++++++++++++++++++++++++++ include/linux/kernel.h | 33 +---------------------------- include/linux/list.h | 4 +++- include/linux/llist.h | 4 +++- include/linux/plist.h | 5 ++++- include/linux/rwsem.h | 1 - include/linux/smp.h | 1 - include/linux/spinlock.h | 1 - include/media/media-entity.h | 3 ++- 10 files changed, 64 insertions(+), 41 deletions(-) create mode 100644 include/linux/container_of.h -- 2.33.0

3 years, 7 months

2
9
0 0

DRM KUnit hackathon

by Nícolas F. R. A. Prado

Hello, We belong to a student group, LKCAMP [1], which is focused on sharing kernel and free software development knowledge and mentoring newcomers to become contributors to these projects. As part of our efforts, we'll be organizing a hackathon to convert the drm selftests in drivers/gpu/drm/selftests/ (and possibly the ones in drivers/dma-buf too) to the KUnit framework. It will take place on October 30. So please expect to receive some patches from our mentees on that date. It probably won't be a big volume (experience tells it'll be around half a dozen patches). We'll also make sure to do an internal review beforehand to catch common first-timer mistakes and teach the basics. We're already working on making sure that the converted KUnit tests can still be run by IGT. Please let us know if there's any issue with this date. Otherwise we look forward to helping a few newcomers get their patches in the kernel on the 30th :). Thanks! [1] - https://lkcamp.dev/

3 years, 7 months

2
1
0 0

[PATCH v2 0/4] kernel.h further split

by Andy Shevchenko

The kernel.h is a set of something which is not related to each other and often used in non-crossed compilation units, especially when drivers need only one or two macro definitions from it. Here is the split of container_of(). The goals are the following: - untwist the dependency hell a bit - drop kernel.h inclusion where it's only used for container_of() - speed up C preprocessing. People, like Greg KH and Miguel Ojeda, were asking about the latter. Read below the methodology and test setup with outcome numbers. The methodology =============== The question here is how to measure in the more or less clean way the C preprocessing time when building a project like Linux kernel. To answer it, let's look around and see what tools do we have that may help. Aha, here is ccache tool that seems quite plausible to be used. Its core idea is to preprocess C file, count hash (MD4) and compare to ones that are in the cache. If found, return the object file, avoiding compilation stage. Taking into account the property of the ccache, configure and use it in the below steps: 1. Configure kernel with allyesconfig 2. Make it with `make` to be sure that the cache is filled with the latest data. I.o.w. warm up the cache. 3. Run `make -s` (silent mode to reduce the influence of the unrelated things, like console output) 10 times and measure 'real' time spent. 4. Repeat 1-3 for each patch or patch set to get data sets before and after. When we get the raw data, calculating median will show us the number. Comparing them before and after we will see the difference. The setup ========= I have used the Intel x86_64 server platform (see partial output of `lscpu` below): $ lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 88 On-line CPU(s) list: 0-87 Vendor ID: GenuineIntel Model name: Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz CPU family: 6 Model: 79 Thread(s) per core: 2 Core(s) per socket: 22 Socket(s): 2 Stepping: 1 CPU max MHz: 3600.0000 CPU min MHz: 1200.0000 ... Caches (sum of all): L1d: 1.4 MiB (44 instances) L1i: 1.4 MiB (44 instances) L2: 11 MiB (44 instances) L3: 110 MiB (2 instances) NUMA: NUMA node(s): 2 NUMA node0 CPU(s): 0-21,44-65 NUMA node1 CPU(s): 22-43,66-87 Vulnerabilities: Itlb multihit: KVM: Mitigation: Split huge pages L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable Mds: Mitigation; Clear CPU buffers; SMT vulnerable Meltdown: Mitigation; PTI Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Spectre v2: Mitigation; Full generic retpoline, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable With the following GCC: $ gcc --version gcc (Debian 10.3.0-11) 10.3.0 The commands I have run during the measurement were: rm -rf $O make O=$O allyesconfig time make O=$O -s -j64 # this step has been measured The raw data and median ======================= Before patch 2 (yes, I have measured the only patch 2 effect) in the series (the data is sorted by time): real 2m8.794s real 2m11.183s real 2m11.235s real 2m11.639s real 2m11.960s real 2m12.014s real 2m12.609s real 2m13.177s real 2m13.462s real 2m19.132s After patch 2 has been applied: real 2m8.536s real 2m8.776s real 2m9.071s real 2m9.459s real 2m9.531s real 2m9.610s real 2m10.356s real 2m10.430s real 2m11.117s real 2m11.885s Median values are: 131.987s before 129.571s after We see the steady speedup as of 1.83%. Andy Shevchenko (4): kernel.h: Drop unneeded <linux/kernel.h> inclusion from other headers kernel.h: Split out container_of() and typeof_member() macros lib/rhashtable: Replace kernel.h with the necessary inclusions kunit: Replace kernel.h with the necessary inclusions include/kunit/test.h | 14 ++++++++++++-- include/linux/container_of.h | 37 ++++++++++++++++++++++++++++++++++++ include/linux/kernel.h | 31 +----------------------------- include/linux/kobject.h | 1 + include/linux/list.h | 6 ++++-- include/linux/llist.h | 4 +++- include/linux/plist.h | 5 ++++- include/linux/rwsem.h | 1 - include/linux/spinlock.h | 1 - include/media/media-entity.h | 3 ++- lib/radix-tree.c | 6 +++++- lib/rhashtable.c | 7 ++++++- 12 files changed, 75 insertions(+), 41 deletions(-) create mode 100644 include/linux/container_of.h -- 2.33.0

3 years, 7 months

5
16
0 0

[PATCH v2] selftests: vm: add KSM huge pages merging time test

by Pedro Demarchi Gomes

Add test case of KSM merging time using mostly huge pages Signed-off-by: Pedro Demarchi Gomes <pedrodemargomes(a)gmail.com> --- tools/testing/selftests/vm/ksm_tests.c | 125 ++++++++++++++++++++++++- 1 file changed, 124 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/vm/ksm_tests.c b/tools/testing/selftests/vm/ksm_tests.c index b61dcdb44c5b..61f3360a19df 100644 --- a/tools/testing/selftests/vm/ksm_tests.c +++ b/tools/testing/selftests/vm/ksm_tests.c @@ -5,6 +5,10 @@ #include <time.h> #include <string.h> #include <numa.h> +#include <unistd.h> +#include <fcntl.h> +#include <stdint.h> +#include <err.h> #include "../kselftest.h" #include "../../../../include/vdso/time64.h" @@ -18,6 +22,15 @@ #define KSM_MERGE_ACROSS_NODES_DEFAULT true #define MB (1ul << 20) +#define PAGE_SHIFT 12 +#define HPAGE_SHIFT 21 + +#define PAGE_SIZE (1 << PAGE_SHIFT) +#define HPAGE_SIZE (1 << HPAGE_SHIFT) + +#define PAGEMAP_PRESENT(ent) (((ent) & (1ull << 63)) != 0) +#define PAGEMAP_PFN(ent) ((ent) & ((1ull << 55) - 1)) + struct ksm_sysfs { unsigned long max_page_sharing; unsigned long merge_across_nodes; @@ -34,6 +47,7 @@ enum ksm_test_name { CHECK_KSM_ZERO_PAGE_MERGE, CHECK_KSM_NUMA_MERGE, KSM_MERGE_TIME, + KSM_MERGE_TIME_HUGE_PAGES, KSM_COW_TIME }; @@ -99,6 +113,9 @@ static void print_help(void) " -U (page unmerging)\n" " -P evaluate merging time and speed.\n" " For this test, the size of duplicated memory area (in MiB)\n" + " must be provided using -s option\n" + " -H evaluate merging time and speed of area allocated mostly with huge pages\n" + " For this test, the size of duplicated memory area (in MiB)\n" " must be provided using -s option\n" " -C evaluate the time required to break COW of merged pages.\n\n"); @@ -416,6 +433,101 @@ static int check_ksm_numa_merge(int mapping, int prot, int timeout, bool merge_a return KSFT_FAIL; } +int64_t allocate_transhuge(void *ptr, int pagemap_fd) +{ + uint64_t ent[2]; + + /* drop pmd */ + if (mmap(ptr, HPAGE_SIZE, PROT_READ | PROT_WRITE, + MAP_FIXED | MAP_ANONYMOUS | + MAP_NORESERVE | MAP_PRIVATE, -1, 0) != ptr) + errx(2, "mmap transhuge"); + + if (madvise(ptr, HPAGE_SIZE, MADV_HUGEPAGE)) + err(2, "MADV_HUGEPAGE"); + + /* allocate transparent huge page */ + *(volatile void **)ptr = ptr; + + if (pread(pagemap_fd, ent, sizeof(ent), + (uintptr_t)ptr >> (PAGE_SHIFT - 3)) != sizeof(ent)) + err(2, "read pagemap"); + + if (PAGEMAP_PRESENT(ent[0]) && PAGEMAP_PRESENT(ent[1]) && + PAGEMAP_PFN(ent[0]) + 1 == PAGEMAP_PFN(ent[1]) && + !(PAGEMAP_PFN(ent[0]) & ((1 << (HPAGE_SHIFT - PAGE_SHIFT)) - 1))) + return PAGEMAP_PFN(ent[0]); + + return -1; +} + +static int ksm_merge_hugepages_time(int mapping, int prot, int timeout, size_t map_size) +{ + void *map_ptr, *map_ptr_orig; + struct timespec start_time, end_time; + unsigned long scan_time_ns; + int pagemap_fd, n_normal_pages, n_huge_pages; + + map_size *= MB; + size_t len = map_size; + + len -= len % HPAGE_SIZE; + map_ptr_orig = mmap(NULL, len + HPAGE_SIZE, PROT_READ | PROT_WRITE, + MAP_ANONYMOUS | MAP_NORESERVE | MAP_PRIVATE, -1, 0); + map_ptr = map_ptr_orig + HPAGE_SIZE - (uintptr_t)map_ptr_orig % HPAGE_SIZE; + + if (map_ptr_orig == MAP_FAILED) + err(2, "initial mmap"); + + if (madvise(map_ptr, len + HPAGE_SIZE, MADV_HUGEPAGE)) + err(2, "MADV_HUGEPAGE"); + + pagemap_fd = open("/proc/self/pagemap", O_RDONLY); + if (pagemap_fd < 0) + err(2, "open pagemap"); + + n_normal_pages = 0; + n_huge_pages = 0; + for (void *p = map_ptr; p < map_ptr + len; p += HPAGE_SIZE) { + if (allocate_transhuge(p, pagemap_fd) < 0) + n_normal_pages++; + else + n_huge_pages++; + } + printf("Number of normal pages: %d\n", n_normal_pages); + printf("Number of huge pages: %d\n", n_huge_pages); + + memset(map_ptr, '*', len); + + if (clock_gettime(CLOCK_MONOTONIC_RAW, &start_time)) { + perror("clock_gettime"); + goto err_out; + } + if (ksm_merge_pages(map_ptr, map_size, start_time, timeout)) + goto err_out; + if (clock_gettime(CLOCK_MONOTONIC_RAW, &end_time)) { + perror("clock_gettime"); + goto err_out; + } + + scan_time_ns = (end_time.tv_sec - start_time.tv_sec) * NSEC_PER_SEC + + (end_time.tv_nsec - start_time.tv_nsec); + + printf("Total size: %lu MiB\n", map_size / MB); + printf("Total time: %ld.%09ld s\n", scan_time_ns / NSEC_PER_SEC, + scan_time_ns % NSEC_PER_SEC); + printf("Average speed: %.3f MiB/s\n", (map_size / MB) / + ((double)scan_time_ns / NSEC_PER_SEC)); + + munmap(map_ptr_orig, len + HPAGE_SIZE); + return KSFT_PASS; + +err_out: + printf("Not OK\n"); + munmap(map_ptr_orig, len + HPAGE_SIZE); + return KSFT_FAIL; +} + static int ksm_merge_time(int mapping, int prot, int timeout, size_t map_size) { void *map_ptr; @@ -541,7 +653,7 @@ int main(int argc, char *argv[]) bool merge_across_nodes = KSM_MERGE_ACROSS_NODES_DEFAULT; long size_MB = 0; - while ((opt = getopt(argc, argv, "ha:p:l:z:m:s:MUZNPC")) != -1) { + while ((opt = getopt(argc, argv, "ha:p:l:z:m:s:MUZNPCH")) != -1) { switch (opt) { case 'a': prot = str_to_prot(optarg); @@ -595,6 +707,9 @@ int main(int argc, char *argv[]) case 'P': test_name = KSM_MERGE_TIME; break; + case 'H': + test_name = KSM_MERGE_TIME_HUGE_PAGES; + break; case 'C': test_name = KSM_COW_TIME; break; @@ -647,6 +762,14 @@ int main(int argc, char *argv[]) ret = ksm_merge_time(MAP_PRIVATE | MAP_ANONYMOUS, prot, ksm_scan_limit_sec, size_MB); break; + case KSM_MERGE_TIME_HUGE_PAGES: + if (size_MB == 0) { + printf("Option '-s' is required.\n"); + return KSFT_FAIL; + } + ret = ksm_merge_hugepages_time(MAP_PRIVATE | MAP_ANONYMOUS, prot, + ksm_scan_limit_sec, size_MB); + break; case KSM_COW_TIME: ret = ksm_cow_time(MAP_PRIVATE | MAP_ANONYMOUS, prot, ksm_scan_limit_sec, page_size); -- 2.25.1

3 years, 7 months

1
0
0 0

[PATCH v6 0/1] lib: Convert UUID runtime test to KUnit

by André Almeida

Hi, This patch converts existing UUID runtime test to use KUnit framework. Below, there's a comparison between the old output format and the new one. Keep in mind that even if KUnit seems very verbose, this is the corner case where _every_ test has failed. * This is how the current output looks like in success: test_uuid: all 18 tests passed * And when it fails: test_uuid: conversion test #1 failed on LE data: 'c33f4995-3701-450e-9fbf-206a2e98e576' test_uuid: cmp test #2 failed on LE data: 'c33f4995-3701-450e-9fbf-206a2e98e576' test_uuid: cmp test #2 actual data: 'c33f4995-3701-450e-9fbf-206a2e98e576' test_uuid: conversion test #3 failed on BE data: 'c33f4995-3701-450e-9fbf-206a2e98e576' test_uuid: cmp test #4 failed on BE data: 'c33f4995-3701-450e-9fbf-206a2e98e576' test_uuid: cmp test #4 actual data: 'c33f4995-3701-450e-9fbf-206a2e98e576' test_uuid: conversion test #5 failed on LE data: '64b4371c-77c1-48f9-8221-29f054fc023b' test_uuid: cmp test #6 failed on LE data: '64b4371c-77c1-48f9-8221-29f054fc023b' test_uuid: cmp test #6 actual data: '64b4371c-77c1-48f9-8221-29f054fc023b' test_uuid: conversion test #7 failed on BE data: '64b4371c-77c1-48f9-8221-29f054fc023b' test_uuid: cmp test #8 failed on BE data: '64b4371c-77c1-48f9-8221-29f054fc023b' test_uuid: cmp test #8 actual data: '64b4371c-77c1-48f9-8221-29f054fc023b' test_uuid: conversion test #9 failed on LE data: '0cb4ddff-a545-4401-9d06-688af53e7f84' test_uuid: cmp test #10 failed on LE data: '0cb4ddff-a545-4401-9d06-688af53e7f84' test_uuid: cmp test #10 actual data: '0cb4ddff-a545-4401-9d06-688af53e7f84' test_uuid: conversion test #11 failed on BE data: '0cb4ddff-a545-4401-9d06-688af53e7f84' test_uuid: cmp test #12 failed on BE data: '0cb4ddff-a545-4401-9d06-688af53e7f84' test_uuid: cmp test #12 actual data: '0cb4ddff-a545-4401-9d06-688af53e7f84' test_uuid: negative test #13 passed on wrong LE data: 'c33f4995-3701-450e-9fbf206a2e98e576 ' test_uuid: negative test #14 passed on wrong BE data: 'c33f4995-3701-450e-9fbf206a2e98e576 ' test_uuid: negative test #15 passed on wrong LE data: '64b4371c-77c1-48f9-8221-29f054XX023b' test_uuid: negative test #16 passed on wrong BE data: '64b4371c-77c1-48f9-8221-29f054XX023b' test_uuid: negative test #17 passed on wrong LE data: '0cb4ddff-a545-4401-9d06-688af53e' test_uuid: negative test #18 passed on wrong BE data: '0cb4ddff-a545-4401-9d06-688af53e' test_uuid: failed 18 out of 18 tests * Now, here's how it looks like with KUnit: ======== [PASSED] uuid ======== [PASSED] uuid_correct_be [PASSED] uuid_correct_le [PASSED] uuid_wrong_be [PASSED] uuid_wrong_le * And if every test fail with KUnit: ======== [FAILED] uuid ======== [FAILED] uuid_correct_be # uuid_correct_be: ASSERTION FAILED at lib/test_uuid.c:57 Expected uuid_parse(data->uuid, &be) == 1, but uuid_parse(data->uuid, &be) == 0 failed to parse 'c33f4995-3701-450e-9fbf-206a2e98e576' # uuid_correct_be: not ok 1 - c33f4995-3701-450e-9fbf-206a2e98e576 # uuid_correct_be: ASSERTION FAILED at lib/test_uuid.c:57 Expected uuid_parse(data->uuid, &be) == 1, but uuid_parse(data->uuid, &be) == 0 failed to parse '64b4371c-77c1-48f9-8221-29f054fc023b' # uuid_correct_be: not ok 2 - 64b4371c-77c1-48f9-8221-29f054fc023b # uuid_correct_be: ASSERTION FAILED at lib/test_uuid.c:57 Expected uuid_parse(data->uuid, &be) == 1, but uuid_parse(data->uuid, &be) == 0 failed to parse '0cb4ddff-a545-4401-9d06-688af53e7f84' # uuid_correct_be: not ok 3 - 0cb4ddff-a545-4401-9d06-688af53e7f84 not ok 1 - uuid_correct_be [FAILED] uuid_correct_le # uuid_correct_le: ASSERTION FAILED at lib/test_uuid.c:46 Expected guid_parse(data->uuid, &le) == 1, but guid_parse(data->uuid, &le) == 0 failed to parse 'c33f4995-3701-450e-9fbf-206a2e98e576' # uuid_correct_le: not ok 1 - c33f4995-3701-450e-9fbf-206a2e98e576 # uuid_correct_le: ASSERTION FAILED at lib/test_uuid.c:46 Expected guid_parse(data->uuid, &le) == 1, but guid_parse(data->uuid, &le) == 0 failed to parse '64b4371c-77c1-48f9-8221-29f054fc023b' # uuid_correct_le: not ok 2 - 64b4371c-77c1-48f9-8221-29f054fc023b # uuid_correct_le: ASSERTION FAILED at lib/test_uuid.c:46 Expected guid_parse(data->uuid, &le) == 1, but guid_parse(data->uuid, &le) == 0 failed to parse '0cb4ddff-a545-4401-9d06-688af53e7f84' # uuid_correct_le: not ok 3 - 0cb4ddff-a545-4401-9d06-688af53e7f84 not ok 2 - uuid_correct_le [FAILED] uuid_wrong_be # uuid_wrong_be: ASSERTION FAILED at lib/test_uuid.c:77 Expected uuid_parse(*data, &be) == 0, but uuid_parse(*data, &be) == -22 parsing of 'c33f4995-3701-450e-9fbf206a2e98e576 ' should've failed # uuid_wrong_be: not ok 1 - c33f4995-3701-450e-9fbf206a2e98e576 # uuid_wrong_be: ASSERTION FAILED at lib/test_uuid.c:77 Expected uuid_parse(*data, &be) == 0, but uuid_parse(*data, &be) == -22 parsing of '64b4371c-77c1-48f9-8221-29f054XX023b' should've failed # uuid_wrong_be: not ok 2 - 64b4371c-77c1-48f9-8221-29f054XX023b # uuid_wrong_be: ASSERTION FAILED at lib/test_uuid.c:77 Expected uuid_parse(*data, &be) == 0, but uuid_parse(*data, &be) == -22 parsing of '0cb4ddff-a545-4401-9d06-688af53e' should've failed # uuid_wrong_be: not ok 3 - 0cb4ddff-a545-4401-9d06-688af53e not ok 3 - uuid_wrong_be [FAILED] uuid_wrong_le # uuid_wrong_le: ASSERTION FAILED at lib/test_uuid.c:68 Expected guid_parse(*data, &le) == 0, but guid_parse(*data, &le) == -22 parsing of 'c33f4995-3701-450e-9fbf206a2e98e576 ' should've failed # uuid_wrong_le: not ok 1 - c33f4995-3701-450e-9fbf206a2e98e576 # uuid_wrong_le: ASSERTION FAILED at lib/test_uuid.c:68 Expected guid_parse(*data, &le) == 0, but guid_parse(*data, &le) == -22 parsing of '64b4371c-77c1-48f9-8221-29f054XX023b' should've failed # uuid_wrong_le: not ok 2 - 64b4371c-77c1-48f9-8221-29f054XX023b # uuid_wrong_le: ASSERTION FAILED at lib/test_uuid.c:68 Expected guid_parse(*data, &le) == 0, but guid_parse(*data, &le) == -22 parsing of '0cb4ddff-a545-4401-9d06-688af53e' should've failed # uuid_wrong_le: not ok 3 - 0cb4ddff-a545-4401-9d06-688af53e not ok 4 - uuid_wrong_le Changes from v5: - Add reviewed-by Brendan - Rebased on top of 5.15-rc4 Changes from v4: - Add reviewed-by v4: https://lore.kernel.org/lkml/20210621133148.9226-1-andrealmeid@collabora.co… Changes from v3: - Drop unnecessary casts and braces. - Simplify Kconfig entry v3: https://lore.kernel.org/lkml/20210610163959.71634-1-andrealmeid@collabora.c… Changes from v2: - Clarify in commit message the new test cases setup v2: https://lore.kernel.org/lkml/20210609233730.164082-1-andrealmeid@collabora.… Changes from v1: - Test suite name: uuid_test -> uuid - Config name: TEST_UUID -> UUID_KUNIT_TEST - Config entry in the Kconfig file left where it is - Converted tests to use _MSG variant v1: https://lore.kernel.org/lkml/20210605215215.171165-1-andrealmeid@collabora.… André Almeida (1): lib: Convert UUID runtime test to KUnit lib/Kconfig.debug | 8 ++- lib/Makefile | 2 +- lib/test_uuid.c | 137 +++++++++++++++++++--------------------------- 3 files changed, 64 insertions(+), 83 deletions(-) -- 2.33.0

3 years, 7 months

3
3
0 0

[PATCH 00/14] nSVM fixes and optional features

by Maxim Levitsky

Those are few patches I was working on lately, all somewhat related to the two CVEs that I found recently. First 7 patches fix various minor bugs that relate to these CVEs. The rest of the patches implement various optional SVM features, some of which the guest could enable anyway due to incorrect checking of virt_ext field. Last patch is somewhat an RFC, I would like to hear your opinion on that. I also implemented nested TSC scaling while at it. As for other optional SVM features here is my summary of few features I took a look at: X86_FEATURE_DECODEASSISTS: this feature should make it easier for the L1 to emulate an instruction on MMIO access, by not needing to read the guest memory but rather using the instruction bytes that the CPU already fetched. The challenge of implementing this is that we sometimes inject #PF and #NPT syntenically and in those cases we must be sure we set the correct instruction bytes. Also this feature adds assists for MOV CR/DR, INTn, and INVLPG, which aren't that interesting but must be supported as well to expose this feature to the nested guest. X86_FEATURE_VGIF Might allow the L2 to run the L3 a bit faster, but due to crazy complex logic we already have around int_ctl and vgif probably not worth it. X86_FEATURE_VMCBCLEAN Should just be enabled, because otherwise L1 doesn't even attempt to set the clean bits. But we need to know if we can take an advantage of these bits first. X86_FEATURE_FLUSHBYASID X86_FEATURE_AVIC These two features would be very good to enable, but that would require lots of work, and will be done eventually. There are few more nested SVM features that I didn't yet had a chance to take a look at. Best regards, Maxim Levitsky Maxim Levitsky (14): KVM: x86: nSVM: restore int_vector in svm_clear_vintr KVM: x86: selftests: test simultaneous uses of V_IRQ from L1 and L0 KVM: x86: nSVM: test eax for 4K alignment for GP errata workaround KVM: x86: nSVM: don't copy pause related settings KVM: x86: nSVM: don't copy virt_ext from vmcb12 KVM: x86: SVM: don't set VMLOAD/VMSAVE intercepts on vCPU reset KVM: x86: SVM: add warning for CVE-2021-3656 KVM: x86: SVM: add module param to control LBR virtualization KVM: x86: nSVM: correctly virtualize LBR msrs when L2 is running KVM: x86: nSVM: implement nested LBR virtualization KVM: x86: nSVM: implement nested VMLOAD/VMSAVE KVM: x86: SVM: add module param to control TSC scaling KVM: x86: nSVM: implement nested TSC scaling KVM: x86: nSVM: support PAUSE filter threshold and count arch/x86/kvm/svm/nested.c | 105 +++++++-- arch/x86/kvm/svm/svm.c | 218 +++++++++++++++--- arch/x86/kvm/svm/svm.h | 20 +- arch/x86/kvm/vmx/vmx.c | 1 + arch/x86/kvm/x86.c | 1 + tools/testing/selftests/kvm/.gitignore | 1 + tools/testing/selftests/kvm/Makefile | 1 + .../selftests/kvm/x86_64/svm_int_ctl_test.c | 128 ++++++++++ 8 files changed, 427 insertions(+), 48 deletions(-) create mode 100644 tools/testing/selftests/kvm/x86_64/svm_int_ctl_test.c -- 2.26.3

3 years, 7 months

4
24
0 0

[PATCH] Fix application of sizeof to pointer

by davidcomponentone＠gmail.com

From: David Yang <davidcomponentone(a)gmail.com> The coccinelle check report: "./tools/testing/selftests/vm/split_huge_page_test.c:344:36-42: ERROR: application of sizeof to pointer" Using the "strlen" to fix it. Reported-by: Zeal Robot <zealci(a)zte.com.cn> Signed-off-by: David Yang <davidcomponentone(a)gmail.com> --- tools/testing/selftests/vm/split_huge_page_test.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/testing/selftests/vm/split_huge_page_test.c b/tools/testing/selftests/vm/split_huge_page_test.c index 1af16d2c2a0a..52497b7b9f1d 100644 --- a/tools/testing/selftests/vm/split_huge_page_test.c +++ b/tools/testing/selftests/vm/split_huge_page_test.c @@ -341,7 +341,7 @@ void split_file_backed_thp(void) } /* write something to the file, so a file-backed THP can be allocated */ - num_written = write(fd, tmpfs_loc, sizeof(tmpfs_loc)); + num_written = write(fd, tmpfs_loc, strlen(tmpfs_loc) + 1); close(fd); if (num_written < 1) { -- 2.30.2

3 years, 7 months

2
1
0 0

[PATCH v8] kunit: tool: improve compatibility of kunit_parser with KTAP specification

by Daniel Latypov

From: Rae Moar <rmoar(a)google.com> Update to kunit_parser to improve compatibility with KTAP specification including arbitrarily nested tests. Patch accomplishes three major changes: - Use a general Test object to represent all tests rather than TestCase and TestSuite objects. This allows for easier implementation of arbitrary levels of nested tests and promotes the idea that both test suites and test cases are tests. - Print errors incrementally rather than all at once after the parsing finishes to maximize information given to the user in the case of the parser given invalid input and to increase the helpfulness of the timestamps given during printing. Note that kunit.py parse does not print incrementally yet. However, this fix brings us closer to this feature. - Increase compatibility for different formats of input. Arbitrary levels of nested tests supported. Also, test cases and test suites are now supported to be present on the same level of testing. This patch now implements the draft KTAP specification here: https://lore.kernel.org/linux-kselftest/CA+GJov6tdjvY9x12JsJT14qn6c7NViJxqa… We'll update the parser as the spec evolves. This patch adjusts the kunit_tool_test.py file to check for the correct outputs from the new parser and adds a new test to check the parsing for a KTAP result log with correct format for multiple nested subtests (test_is_test_passed-all_passed_nested.log). This patch also alters the kunit_json.py file to allow for arbitrarily nested tests. Signed-off-by: Rae Moar <rmoar(a)google.com> Reviewed-by: Brendan Higgins <brendanhiggins(a)google.com> Signed-off-by: Daniel Latypov <dlatypov(a)google.com> Reviewed-by: David Gow <davidgow(a)google.com> --- Change log from v7: https://lore.kernel.org/linux-kselftest/20211007210324.707912-1-dlatypov@go… - don't have 1 "passed" test when no tests run - don't treat BUG as a kernel crash (KASAN tests print this out normally) Change log from v6: https://lore.kernel.org/linux-kselftest/20211006170049.106852-1-dlatypov@go… - Rebase onto shuah/linux-kselftest/kunit - fix one new unit test failure (s/suites/test.subtests) Change log from v5: https://lore.kernel.org/linux-kselftest/20211006001447.20919-1-dlatypov@goo… - Tweak commit message to reflect the KTAP spec is a draft - Add missing Signed-off-by - Tweak docstrings Change log from v3,4: https://lore.kernel.org/linux-kselftest/20210901190623.315736-1-rmoar@googl… - Move test_kselftest_nested from LinuxSourceTreeTest => KUnitParserTest. - Resolve conflict with hermetic testing patches. - max_status is no longer defined, so we need to use the TestCounts type now. And to keep --raw_output working, we need to set this to SUCCESS to avoid the default assumption that the kernel crashed. Ignore v4, was accidentally based on v2. Change log from v2: https://lore.kernel.org/linux-kselftest/20210826195505.3066755-1-rmoar@goog… - Fixes bug of type disagreement in kunit_json.py for build_dir - Removes raw_output() - Changes docstrings in kunit_parser.py (class docstring, LineStream docstrings, add_error(), total(), get_status(), all parsing methods) - Fixes bug of not printing diagnostic log in the case of end of lines - Sets default status of all tests to TEST_CRASHED - Adds and prints empty tests with crashed status in case of missing tests - Prints 'subtest' in instance of 1 subtest instead of 'subtests' - Includes checking for 'BUG:' message in search of crash messages in log (note that parse_crash_in_log method could be removed but would require deleting tests in kunit_tool_test.py that include the crash message that is no longer used. If removed, parser would still print log in cases of test crashed or failure, which would now include missing subtests) - Fixes bug of including directives (other than SKIP) in test name when matching name in result line for subtests Change log from v1: https://lore.kernel.org/linux-kselftest/20210820200032.2178134-1-rmoar@goog… - Rebase onto kselftest/kunit branch - Add tests to kunit_tool_test.py to check parser is correctly stripping hyphen, producing correct json objects with nested tests, correctly passing kselftest TAP output, and correctly deals with missing test plan. - Fix bug to correctly match test name in instance of a missing test plan. - Fix bug in kunit_tool_test.py pointed out by Daniel where it was not correctly checking for a proper match to the '0 tests run!' error message. Reverts changes back to original. - A few minor changes to commit message using Daniel's comments. - Change docstrings using Daniel's comments to reduce: - Shortens some docstrings to be one-line or just description if it is self explanatory. - Remove explicit respecification of types of parameters and returns because this is already specified in the function annoations. However, some descriptions of the parameters and returns remain and some contain the type for context. Additionally, the types of public attributes of classes remain. - Remove any documentation of 'Return: None' - Remove docstrings of helper methods within other methods --- tools/testing/kunit/kunit.py | 19 +- tools/testing/kunit/kunit_json.py | 56 +- tools/testing/kunit/kunit_parser.py | 1015 ++++++++++++----- tools/testing/kunit/kunit_tool_test.py | 136 ++- .../test_is_test_passed-all_passed_nested.log | 34 + .../test_is_test_passed-kselftest.log | 14 + .../test_is_test_passed-missing_plan.log | 31 + .../kunit/test_data/test_strip_hyphen.log | 16 + 8 files changed, 938 insertions(+), 383 deletions(-) create mode 100644 tools/testing/kunit/test_data/test_is_test_passed-all_passed_nested.log create mode 100644 tools/testing/kunit/test_data/test_is_test_passed-kselftest.log create mode 100644 tools/testing/kunit/test_data/test_is_test_passed-missing_plan.log create mode 100644 tools/testing/kunit/test_data/test_strip_hyphen.log diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py index 7197e5fb8342..9c9ed4071e9e 100755 --- a/tools/testing/kunit/kunit.py +++ b/tools/testing/kunit/kunit.py @@ -135,7 +135,7 @@ def exec_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest, test_glob = request.filter_glob.split('.', maxsplit=2)[1] filter_globs = [g + '.'+ test_glob for g in filter_globs] - overall_status = kunit_parser.TestStatus.SUCCESS + test_counts = kunit_parser.TestCounts() exec_time = 0.0 for i, filter_glob in enumerate(filter_globs): kunit_parser.print_with_timestamp('Starting KUnit Kernel ({}/{})...'.format(i+1, len(filter_globs))) @@ -154,18 +154,29 @@ def exec_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest, test_end = time.time() exec_time += test_end - test_start - overall_status = kunit_parser.max_status(overall_status, result.status) + test_counts.add_subtest_counts(result.result.test.counts) - return KunitResult(status=result.status, result=result.result, elapsed_time=exec_time) + kunit_status = _map_to_overall_status(test_counts.get_status()) + return KunitResult(status=kunit_status, result=result.result, elapsed_time=exec_time) + +def _map_to_overall_status(test_status: kunit_parser.TestStatus) -> KunitStatus: + if test_status in (kunit_parser.TestStatus.SUCCESS, kunit_parser.TestStatus.SKIPPED): + return KunitStatus.SUCCESS + else: + return KunitStatus.TEST_FAILURE def parse_tests(request: KunitParseRequest, input_data: Iterable[str]) -> KunitResult: parse_start = time.time() test_result = kunit_parser.TestResult(kunit_parser.TestStatus.SUCCESS, - [], + kunit_parser.Test(), 'Tests not Parsed.') if request.raw_output: + # Treat unparsed results as one passing test. + test_result.test.status = kunit_parser.TestStatus.SUCCESS + test_result.test.counts.passed = 1 + output: Iterable[str] = input_data if request.raw_output == 'all': pass diff --git a/tools/testing/kunit/kunit_json.py b/tools/testing/kunit/kunit_json.py index f5cca5c38cac..746bec72b9ac 100644 --- a/tools/testing/kunit/kunit_json.py +++ b/tools/testing/kunit/kunit_json.py @@ -11,47 +11,47 @@ import os import kunit_parser -from kunit_parser import TestStatus - -def get_json_result(test_result, def_config, build_dir, json_path) -> str: - sub_groups = [] - - # Each test suite is mapped to a KernelCI sub_group - for test_suite in test_result.suites: - sub_group = { - "name": test_suite.name, - "arch": "UM", - "defconfig": def_config, - "build_environment": build_dir, - "test_cases": [], - "lab_name": None, - "kernel": None, - "job": None, - "git_branch": "kselftest", - } - test_cases = [] - # TODO: Add attachments attribute in test_case with detailed - # failure message, see https://api.kernelci.org/schema-test-case.html#get - for case in test_suite.cases: - test_case = {"name": case.name, "status": "FAIL"} - if case.status == TestStatus.SUCCESS: +from kunit_parser import Test, TestResult, TestStatus +from typing import Any, Dict, Optional + +JsonObj = Dict[str, Any] + +def _get_group_json(test: Test, def_config: str, + build_dir: Optional[str]) -> JsonObj: + sub_groups = [] # List[JsonObj] + test_cases = [] # List[JsonObj] + + for subtest in test.subtests: + if len(subtest.subtests): + sub_group = _get_group_json(subtest, def_config, + build_dir) + sub_groups.append(sub_group) + else: + test_case = {"name": subtest.name, "status": "FAIL"} + if subtest.status == TestStatus.SUCCESS: test_case["status"] = "PASS" - elif case.status == TestStatus.TEST_CRASHED: + elif subtest.status == TestStatus.TEST_CRASHED: test_case["status"] = "ERROR" test_cases.append(test_case) - sub_group["test_cases"] = test_cases - sub_groups.append(sub_group) + test_group = { - "name": "KUnit Test Group", + "name": test.name, "arch": "UM", "defconfig": def_config, "build_environment": build_dir, "sub_groups": sub_groups, + "test_cases": test_cases, "lab_name": None, "kernel": None, "job": None, "git_branch": "kselftest", } + return test_group + +def get_json_result(test_result: TestResult, def_config: str, + build_dir: Optional[str], json_path: str) -> str: + test_group = _get_group_json(test_result.test, def_config, build_dir) + test_group["name"] = "KUnit Test Group" json_obj = json.dumps(test_group, indent=4) if json_path != 'stdout': with open(json_path, 'w') as result_path: diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py index 6310a641b151..3355196d0515 100644 --- a/tools/testing/kunit/kunit_parser.py +++ b/tools/testing/kunit/kunit_parser.py @@ -1,11 +1,15 @@ # SPDX-License-Identifier: GPL-2.0 # -# Parses test results from a kernel dmesg log. +# Parses KTAP test results from a kernel dmesg log and incrementally prints +# results with reader-friendly format. Stores and returns test results in a +# Test object. # # Copyright (C) 2019, Google LLC. # Author: Felix Guo <felixguoxiuping(a)gmail.com> # Author: Brendan Higgins <brendanhiggins(a)google.com> +# Author: Rae Moar <rmoar(a)google.com> +from __future__ import annotations import re from collections import namedtuple @@ -14,33 +18,52 @@ from enum import Enum, auto from functools import reduce from typing import Iterable, Iterator, List, Optional, Tuple -TestResult = namedtuple('TestResult', ['status','suites','log']) - -class TestSuite(object): +TestResult = namedtuple('TestResult', ['status','test','log']) + +class Test(object): + """ + A class to represent a test parsed from KTAP results. All KTAP + results within a test log are stored in a main Test object as + subtests. + + Attributes: + status : TestStatus - status of the test + name : str - name of the test + expected_count : int - expected number of subtests (0 if single + test case and None if unknown expected number of subtests) + subtests : List[Test] - list of subtests + log : List[str] - log of KTAP lines that correspond to the test + counts : TestCounts - counts of the test statuses and errors of + subtests or of the test itself if the test is a single + test case. + """ def __init__(self) -> None: - self.status = TestStatus.SUCCESS - self.name = '' - self.cases = [] # type: List[TestCase] - - def __str__(self) -> str: - return 'TestSuite(' + str(self.status) + ',' + self.name + ',' + str(self.cases) + ')' - - def __repr__(self) -> str: - return str(self) - -class TestCase(object): - def __init__(self) -> None: - self.status = TestStatus.SUCCESS + """Creates Test object with default attributes.""" + self.status = TestStatus.TEST_CRASHED self.name = '' + self.expected_count = 0 # type: Optional[int] + self.subtests = [] # type: List[Test] self.log = [] # type: List[str] + self.counts = TestCounts() def __str__(self) -> str: - return 'TestCase(' + str(self.status) + ',' + self.name + ',' + str(self.log) + ')' + """Returns string representation of a Test class object.""" + return ('Test(' + str(self.status) + ', ' + self.name + + ', ' + str(self.expected_count) + ', ' + + str(self.subtests) + ', ' + str(self.log) + ', ' + + str(self.counts) + ')') def __repr__(self) -> str: + """Returns string representation of a Test class object.""" return str(self) + def add_error(self, error_message: str) -> None: + """Records an error that occurred while parsing this test.""" + self.counts.errors += 1 + print_error('Test ' + self.name + ': ' + error_message) + class TestStatus(Enum): + """An enumeration class to represent the status of a test.""" SUCCESS = auto() FAILURE = auto() SKIPPED = auto() @@ -48,381 +71,747 @@ class TestStatus(Enum): NO_TESTS = auto() FAILURE_TO_PARSE_TESTS = auto() +class TestCounts: + """ + Tracks the counts of statuses of all test cases and any errors within + a Test. + + Attributes: + passed : int - the number of tests that have passed + failed : int - the number of tests that have failed + crashed : int - the number of tests that have crashed + skipped : int - the number of tests that have skipped + errors : int - the number of errors in the test and subtests + """ + def __init__(self): + """Creates TestCounts object with counts of all test + statuses and test errors set to 0. + """ + self.passed = 0 + self.failed = 0 + self.crashed = 0 + self.skipped = 0 + self.errors = 0 + + def __str__(self) -> str: + """Returns the string representation of a TestCounts object. + """ + return ('Passed: ' + str(self.passed) + + ', Failed: ' + str(self.failed) + + ', Crashed: ' + str(self.crashed) + + ', Skipped: ' + str(self.skipped) + + ', Errors: ' + str(self.errors)) + + def total(self) -> int: + """Returns the total number of test cases within a test + object, where a test case is a test with no subtests. + """ + return (self.passed + self.failed + self.crashed + + self.skipped) + + def add_subtest_counts(self, counts: TestCounts) -> None: + """ + Adds the counts of another TestCounts object to the current + TestCounts object. Used to add the counts of a subtest to the + parent test. + + Parameters: + counts - a different TestCounts object whose counts + will be added to the counts of the TestCounts object + """ + self.passed += counts.passed + self.failed += counts.failed + self.crashed += counts.crashed + self.skipped += counts.skipped + self.errors += counts.errors + + def get_status(self) -> TestStatus: + """Returns the aggregated status of a Test using test + counts. + """ + if self.total() == 0: + return TestStatus.NO_TESTS + elif self.crashed: + # If one of the subtests crash, the expected status + # of the Test is crashed. + return TestStatus.TEST_CRASHED + elif self.failed: + # Otherwise if one of the subtests fail, the + # expected status of the Test is failed. + return TestStatus.FAILURE + elif self.passed: + # Otherwise if one of the subtests pass, the + # expected status of the Test is passed. + return TestStatus.SUCCESS + else: + # Finally, if none of the subtests have failed, + # crashed, or passed, the expected status of the + # Test is skipped. + return TestStatus.SKIPPED + + def add_status(self, status: TestStatus) -> None: + """ + Increments count of inputted status. + + Parameters: + status - status to be added to the TestCounts object + """ + if status == TestStatus.SUCCESS: + self.passed += 1 + elif status == TestStatus.FAILURE: + self.failed += 1 + elif status == TestStatus.SKIPPED: + self.skipped += 1 + elif status != TestStatus.NO_TESTS: + self.crashed += 1 + class LineStream: - """Provides a peek()/pop() interface over an iterator of (line#, text).""" + """ + A class to represent the lines of kernel output. + Provides a peek()/pop() interface over an iterator of + (line#, text). + """ _lines: Iterator[Tuple[int, str]] _next: Tuple[int, str] _done: bool def __init__(self, lines: Iterator[Tuple[int, str]]): + """Creates a new LineStream that wraps the given iterator.""" self._lines = lines self._done = False self._next = (0, '') self._get_next() def _get_next(self) -> None: + """Advances the LineSteam to the next line.""" try: self._next = next(self._lines) except StopIteration: self._done = True def peek(self) -> str: + """Returns the current line, without advancing the LineStream. + """ return self._next[1] def pop(self) -> str: + """Returns the current line and advances the LineStream to + the next line. + """ n = self._next self._get_next() return n[1] def __bool__(self) -> bool: + """Returns True if stream has more lines.""" return not self._done # Only used by kunit_tool_test.py. def __iter__(self) -> Iterator[str]: + """Empties all lines stored in LineStream object into + Iterator object and returns the Iterator object. + """ while bool(self): yield self.pop() def line_number(self) -> int: + """Returns the line number of the current line.""" return self._next[0] -kunit_start_re = re.compile(r'TAP version [0-9]+$') -kunit_end_re = re.compile('(List of all partitions:|' - 'Kernel panic - not syncing: VFS:|reboot: System halted)') +# Parsing helper methods: + +KTAP_START = re.compile(r'KTAP version ([0-9]+)$') +TAP_START = re.compile(r'TAP version ([0-9]+)$') +KTAP_END = re.compile('(List of all partitions:|' + 'Kernel panic - not syncing: VFS:|reboot: System halted)') def extract_tap_lines(kernel_output: Iterable[str]) -> LineStream: - def isolate_kunit_output(kernel_output: Iterable[str]) -> Iterator[Tuple[int, str]]: + """Extracts KTAP lines from the kernel output.""" + def isolate_ktap_output(kernel_output: Iterable[str]) \ + -> Iterator[Tuple[int, str]]: line_num = 0 started = False for line in kernel_output: line_num += 1 - line = line.rstrip() # line always has a trailing \n - if kunit_start_re.search(line): + line = line.rstrip() # remove trailing \n + if not started and KTAP_START.search(line): + # start extracting KTAP lines and set prefix + # to number of characters before version line + prefix_len = len( + line.split('KTAP version')[0]) + started = True + yield line_num, line[prefix_len:] + elif not started and TAP_START.search(line): + # start extracting KTAP lines and set prefix + # to number of characters before version line prefix_len = len(line.split('TAP version')[0]) started = True yield line_num, line[prefix_len:] - elif kunit_end_re.search(line): + elif started and KTAP_END.search(line): + # stop extracting KTAP lines break elif started: - yield line_num, line[prefix_len:] - return LineStream(lines=isolate_kunit_output(kernel_output)) - -DIVIDER = '=' * 60 - -RESET = '\033[0;0m' - -def red(text) -> str: - return '\033[1;31m' + text + RESET - -def yellow(text) -> str: - return '\033[1;33m' + text + RESET - -def green(text) -> str: - return '\033[1;32m' + text + RESET - -def print_with_timestamp(message) -> None: - print('[%s] %s' % (datetime.now().strftime('%H:%M:%S'), message)) - -def format_suite_divider(message) -> str: - return '======== ' + message + ' ========' + # remove prefix and any indention and yield + # line with line number + line = line[prefix_len:].lstrip() + yield line_num, line + return LineStream(lines=isolate_ktap_output(kernel_output)) + +KTAP_VERSIONS = [1] +TAP_VERSIONS = [13, 14] + +def check_version(version_num: int, accepted_versions: List[int], + version_type: str, test: Test) -> None: + """ + Adds error to test object if version number is too high or too + low. + + Parameters: + version_num - The inputted version number from the parsed KTAP or TAP + header line + accepted_version - List of accepted KTAP or TAP versions + version_type - 'KTAP' or 'TAP' depending on the type of + version line. + test - Test object for current test being parsed + """ + if version_num < min(accepted_versions): + test.add_error(version_type + + ' version lower than expected!') + elif version_num > max(accepted_versions): + test.add_error( + version_type + ' version higher than expected!') + +def parse_ktap_header(lines: LineStream, test: Test) -> bool: + """ + Parses KTAP/TAP header line and checks version number. + Returns False if fails to parse KTAP/TAP header line. + + Accepted formats: + - 'KTAP version [version number]' + - 'TAP version [version number]' + + Parameters: + lines - LineStream of KTAP output to parse + test - Test object for current test being parsed + + Return: + True if successfully parsed KTAP/TAP header line + """ + ktap_match = KTAP_START.match(lines.peek()) + tap_match = TAP_START.match(lines.peek()) + if ktap_match: + version_num = int(ktap_match.group(1)) + check_version(version_num, KTAP_VERSIONS, 'KTAP', test) + elif tap_match: + version_num = int(tap_match.group(1)) + check_version(version_num, TAP_VERSIONS, 'TAP', test) + else: + return False + test.log.append(lines.pop()) + return True -def print_suite_divider(message) -> None: - print_with_timestamp(DIVIDER) - print_with_timestamp(format_suite_divider(message)) +TEST_HEADER = re.compile(r'^# Subtest: (.*)$') -def print_log(log) -> None: - for m in log: - print_with_timestamp(m) +def parse_test_header(lines: LineStream, test: Test) -> bool: + """ + Parses test header and stores test name in test object. + Returns False if fails to parse test header line. -TAP_ENTRIES = re.compile(r'^(TAP|[\s]*ok|[\s]*not ok|[\s]*[0-9]+\.\.[0-9]+|[\s]*# (Subtest:|.*: kunit test case crashed!)).*$') + Accepted format: + - '# Subtest: [test name]' -def consume_non_diagnostic(lines: LineStream) -> None: - while lines and not TAP_ENTRIES.match(lines.peek()): - lines.pop() + Parameters: + lines - LineStream of KTAP output to parse + test - Test object for current test being parsed -def save_non_diagnostic(lines: LineStream, test_case: TestCase) -> None: - while lines and not TAP_ENTRIES.match(lines.peek()): - test_case.log.append(lines.peek()) - lines.pop() + Return: + True if successfully parsed test header line + """ + match = TEST_HEADER.match(lines.peek()) + if not match: + return False + test.log.append(lines.pop()) + test.name = match.group(1) + return True -OkNotOkResult = namedtuple('OkNotOkResult', ['is_ok','description', 'text']) +TEST_PLAN = re.compile(r'1\.\.([0-9]+)') -OK_NOT_OK_SKIP = re.compile(r'^[\s]*(ok|not ok) [0-9]+ - (.*) # SKIP(.*)$') +def parse_test_plan(lines: LineStream, test: Test) -> bool: + """ + Parses test plan line and stores the expected number of subtests in + test object. Reports an error if expected count is 0. + Returns False and reports missing test plan error if fails to parse + test plan. -OK_NOT_OK_SUBTEST = re.compile(r'^[\s]+(ok|not ok) [0-9]+ - (.*)$') + Accepted format: + - '1..[number of subtests]' -OK_NOT_OK_MODULE = re.compile(r'^(ok|not ok) ([0-9]+) - (.*)$') + Parameters: + lines - LineStream of KTAP output to parse + test - Test object for current test being parsed -def parse_ok_not_ok_test_case(lines: LineStream, test_case: TestCase) -> bool: - save_non_diagnostic(lines, test_case) - if not lines: - test_case.status = TestStatus.TEST_CRASHED - return True - line = lines.peek() - match = OK_NOT_OK_SUBTEST.match(line) - while not match and lines: - line = lines.pop() - match = OK_NOT_OK_SUBTEST.match(line) - if match: - test_case.log.append(lines.pop()) - test_case.name = match.group(2) - skip_match = OK_NOT_OK_SKIP.match(line) - if skip_match: - test_case.status = TestStatus.SKIPPED - return True - if test_case.status == TestStatus.TEST_CRASHED: - return True - if match.group(1) == 'ok': - test_case.status = TestStatus.SUCCESS - else: - test_case.status = TestStatus.FAILURE - return True - else: + Return: + True if successfully parsed test plan line + """ + match = TEST_PLAN.match(lines.peek()) + if not match: + test.expected_count = None + test.add_error('missing plan line!') return False - -SUBTEST_DIAGNOSTIC = re.compile(r'^[\s]+# (.*)$') -DIAGNOSTIC_CRASH_MESSAGE = re.compile(r'^[\s]+# .*?: kunit test case crashed!$') - -def parse_diagnostic(lines: LineStream, test_case: TestCase) -> bool: - save_non_diagnostic(lines, test_case) - if not lines: + test.log.append(lines.pop()) + expected_count = int(match.group(1)) + test.expected_count = expected_count + if expected_count == 0: + test.status = TestStatus.NO_TESTS + test.add_error('0 tests run!') + return True + +TEST_RESULT = re.compile(r'^(ok|not ok) ([0-9]+) (- )?([^#]*)( # .*)?$') + +TEST_RESULT_SKIP = re.compile(r'^(ok|not ok) ([0-9]+) (- )?(.*) # SKIP(.*)$') + +def peek_test_name_match(lines: LineStream, test: Test) -> bool: + """ + Matches current line with the format of a test result line and checks + if the name matches the name of the current test. + Returns False if fails to match format or name. + + Accepted format: + - '[ok|not ok] [test number] [-] [test name] [optional skip + directive]' + + Parameters: + lines - LineStream of KTAP output to parse + test - Test object for current test being parsed + + Return: + True if matched a test result line and the name matching the + expected test name + """ + line = lines.peek() + match = TEST_RESULT.match(line) + if not match: return False + name = match.group(4) + return (name == test.name) + +def parse_test_result(lines: LineStream, test: Test, + expected_num: int) -> bool: + """ + Parses test result line and stores the status and name in the test + object. Reports an error if the test number does not match expected + test number. + Returns False if fails to parse test result line. + + Note that the SKIP directive is the only direction that causes a + change in status. + + Accepted format: + - '[ok|not ok] [test number] [-] [test name] [optional skip + directive]' + + Parameters: + lines - LineStream of KTAP output to parse + test - Test object for current test being parsed + expected_num - expected test number for current test + + Return: + True if successfully parsed a test result line. + """ line = lines.peek() - match = SUBTEST_DIAGNOSTIC.match(line) - if match: - test_case.log.append(lines.pop()) - crash_match = DIAGNOSTIC_CRASH_MESSAGE.match(line) - if crash_match: - test_case.status = TestStatus.TEST_CRASHED - return True - else: + match = TEST_RESULT.match(line) + skip_match = TEST_RESULT_SKIP.match(line) + + # Check if line matches test result line format + if not match: return False + test.log.append(lines.pop()) -def parse_test_case(lines: LineStream) -> Optional[TestCase]: - test_case = TestCase() - save_non_diagnostic(lines, test_case) - while parse_diagnostic(lines, test_case): - pass - if parse_ok_not_ok_test_case(lines, test_case): - return test_case + # Set name of test object + if skip_match: + test.name = skip_match.group(4) else: - return None - -SUBTEST_HEADER = re.compile(r'^[\s]+# Subtest: (.*)$') - -def parse_subtest_header(lines: LineStream) -> Optional[str]: - consume_non_diagnostic(lines) - if not lines: - return None - match = SUBTEST_HEADER.match(lines.peek()) - if match: - lines.pop() - return match.group(1) + test.name = match.group(4) + + # Check test num + num = int(match.group(2)) + if num != expected_num: + test.add_error('Expected test number ' + + str(expected_num) + ' but found ' + str(num)) + + # Set status of test object + status = match.group(1) + if skip_match: + test.status = TestStatus.SKIPPED + elif status == 'ok': + test.status = TestStatus.SUCCESS else: - return None + test.status = TestStatus.FAILURE + return True + +def parse_diagnostic(lines: LineStream) -> List[str]: + """ + Parse lines that do not match the format of a test result line or + test header line and returns them in list. + + Line formats that are not parsed: + - '# Subtest: [test name]' + - '[ok|not ok] [test number] [-] [test name] [optional skip + directive]' + + Parameters: + lines - LineStream of KTAP output to parse + + Return: + Log of diagnostic lines + """ + log = [] # type: List[str] + while lines and not TEST_RESULT.match(lines.peek()) and not \ + TEST_HEADER.match(lines.peek()): + log.append(lines.pop()) + return log + +DIAGNOSTIC_CRASH_MESSAGE = re.compile(r'^# .*?: kunit test case crashed!$') + +def parse_crash_in_log(test: Test) -> bool: + """ + Iterate through the lines of the log to parse for crash message. + If crash message found, set status to crashed and return True. + Otherwise return False. + + Parameters: + test - Test object for current test being parsed + + Return: + True if crash message found in log + """ + for line in test.log: + if DIAGNOSTIC_CRASH_MESSAGE.match(line): + test.status = TestStatus.TEST_CRASHED + return True + return False -SUBTEST_PLAN = re.compile(r'[\s]+[0-9]+\.\.([0-9]+)') -def parse_subtest_plan(lines: LineStream) -> Optional[int]: - consume_non_diagnostic(lines) - match = SUBTEST_PLAN.match(lines.peek()) - if match: - lines.pop() - return int(match.group(1)) - else: - return None - -def max_status(left: TestStatus, right: TestStatus) -> TestStatus: - if left == right: - return left - elif left == TestStatus.TEST_CRASHED or right == TestStatus.TEST_CRASHED: - return TestStatus.TEST_CRASHED - elif left == TestStatus.FAILURE or right == TestStatus.FAILURE: - return TestStatus.FAILURE - elif left == TestStatus.SKIPPED: - return right - else: - return left +# Printing helper methods: -def parse_ok_not_ok_test_suite(lines: LineStream, - test_suite: TestSuite, - expected_suite_index: int) -> bool: - consume_non_diagnostic(lines) - if not lines: - test_suite.status = TestStatus.TEST_CRASHED - return False - line = lines.peek() - match = OK_NOT_OK_MODULE.match(line) - if match: - lines.pop() - if match.group(1) == 'ok': - test_suite.status = TestStatus.SUCCESS - else: - test_suite.status = TestStatus.FAILURE - skip_match = OK_NOT_OK_SKIP.match(line) - if skip_match: - test_suite.status = TestStatus.SKIPPED - suite_index = int(match.group(2)) - if suite_index != expected_suite_index: - print_with_timestamp( - red('[ERROR] ') + 'expected_suite_index ' + - str(expected_suite_index) + ', but got ' + - str(suite_index)) - return True - else: - return False +DIVIDER = '=' * 60 -def bubble_up_errors(status_list: Iterable[TestStatus]) -> TestStatus: - return reduce(max_status, status_list, TestStatus.SKIPPED) +RESET = '\033[0;0m' -def bubble_up_test_case_errors(test_suite: TestSuite) -> TestStatus: - max_test_case_status = bubble_up_errors(x.status for x in test_suite.cases) - return max_status(max_test_case_status, test_suite.status) +def red(text: str) -> str: + """Returns inputted string with red color code.""" + return '\033[1;31m' + text + RESET -def parse_test_suite(lines: LineStream, expected_suite_index: int) -> Optional[TestSuite]: - if not lines: - return None - consume_non_diagnostic(lines) - test_suite = TestSuite() - test_suite.status = TestStatus.SUCCESS - name = parse_subtest_header(lines) - if not name: - return None - test_suite.name = name - expected_test_case_num = parse_subtest_plan(lines) - if expected_test_case_num is None: - return None - while expected_test_case_num > 0: - test_case = parse_test_case(lines) - if not test_case: - break - test_suite.cases.append(test_case) - expected_test_case_num -= 1 - if parse_ok_not_ok_test_suite(lines, test_suite, expected_suite_index): - test_suite.status = bubble_up_test_case_errors(test_suite) - return test_suite - elif not lines: - print_with_timestamp(red('[ERROR] ') + 'ran out of lines before end token') - return test_suite - else: - print(f'failed to parse end of suite "{name}", at line {lines.line_number()}: {lines.peek()}') - return None +def yellow(text: str) -> str: + """Returns inputted string with yellow color code.""" + return '\033[1;33m' + text + RESET -TAP_HEADER = re.compile(r'^TAP version 14$') +def green(text: str) -> str: + """Returns inputted string with green color code.""" + return '\033[1;32m' + text + RESET -def parse_tap_header(lines: LineStream) -> bool: - consume_non_diagnostic(lines) - if TAP_HEADER.match(lines.peek()): - lines.pop() - return True - else: - return False +ANSI_LEN = len(red('')) -TEST_PLAN = re.compile(r'[0-9]+\.\.([0-9]+)') +def print_with_timestamp(message: str) -> None: + """Prints message with timestamp at beginning.""" + print('[%s] %s' % (datetime.now().strftime('%H:%M:%S'), message)) -def parse_test_plan(lines: LineStream) -> Optional[int]: - consume_non_diagnostic(lines) - match = TEST_PLAN.match(lines.peek()) - if match: - lines.pop() - return int(match.group(1)) - else: - return None - -def bubble_up_suite_errors(test_suites: Iterable[TestSuite]) -> TestStatus: - return bubble_up_errors(x.status for x in test_suites) - -def parse_test_result(lines: LineStream) -> TestResult: - consume_non_diagnostic(lines) - if not lines or not parse_tap_header(lines): - return TestResult(TestStatus.FAILURE_TO_PARSE_TESTS, [], lines) - expected_test_suite_num = parse_test_plan(lines) - if expected_test_suite_num == 0: - return TestResult(TestStatus.NO_TESTS, [], lines) - elif expected_test_suite_num is None: - return TestResult(TestStatus.FAILURE_TO_PARSE_TESTS, [], lines) - test_suites = [] - for i in range(1, expected_test_suite_num + 1): - test_suite = parse_test_suite(lines, i) - if test_suite: - test_suites.append(test_suite) +def format_test_divider(message: str, len_message: int) -> str: + """ + Returns string with message centered in fixed width divider. + + Example: + '===================== message example =====================' + + Parameters: + message - message to be centered in divider line + len_message - length of the message to be printed such that + any characters of the color codes are not counted + + Return: + String containing message centered in fixed width divider + """ + default_count = 3 # default number of dashes + len_1 = default_count + len_2 = default_count + difference = len(DIVIDER) - len_message - 2 # 2 spaces added + if difference > 0: + # calculate number of dashes for each side of the divider + len_1 = int(difference / 2) + len_2 = difference - len_1 + return ('=' * len_1) + ' ' + message + ' ' + ('=' * len_2) + +def print_test_header(test: Test) -> None: + """ + Prints test header with test name and optionally the expected number + of subtests. + + Example: + '=================== example (2 subtests) ===================' + + Parameters: + test - Test object representing current test being printed + """ + message = test.name + if test.expected_count: + if test.expected_count == 1: + message += (' (' + str(test.expected_count) + + ' subtest)') else: - print_with_timestamp( - red('[ERROR] ') + ' expected ' + - str(expected_test_suite_num) + - ' test suites, but got ' + str(i - 2)) - break - test_suite = parse_test_suite(lines, -1) - if test_suite: - print_with_timestamp(red('[ERROR] ') + - 'got unexpected test suite: ' + test_suite.name) - if test_suites: - return TestResult(bubble_up_suite_errors(test_suites), test_suites, lines) - else: - return TestResult(TestStatus.NO_TESTS, [], lines) + message += (' (' + str(test.expected_count) + + ' subtests)') + print_with_timestamp(format_test_divider(message, len(message))) -class TestCounts: - passed: int - failed: int - crashed: int - skipped: int +def print_log(log: Iterable[str]) -> None: + """ + Prints all strings in saved log for test in yellow. - def __init__(self): - self.passed = 0 - self.failed = 0 - self.crashed = 0 - self.skipped = 0 - - def total(self) -> int: - return self.passed + self.failed + self.crashed + self.skipped - -def print_and_count_results(test_result: TestResult) -> TestCounts: - counts = TestCounts() - for test_suite in test_result.suites: - if test_suite.status == TestStatus.SUCCESS: - print_suite_divider(green('[PASSED] ') + test_suite.name) - elif test_suite.status == TestStatus.SKIPPED: - print_suite_divider(yellow('[SKIPPED] ') + test_suite.name) - elif test_suite.status == TestStatus.TEST_CRASHED: - print_suite_divider(red('[CRASHED] ' + test_suite.name)) - else: - print_suite_divider(red('[FAILED] ') + test_suite.name) - for test_case in test_suite.cases: - if test_case.status == TestStatus.SUCCESS: - counts.passed += 1 - print_with_timestamp(green('[PASSED] ') + test_case.name) - elif test_case.status == TestStatus.SKIPPED: - counts.skipped += 1 - print_with_timestamp(yellow('[SKIPPED] ') + test_case.name) - elif test_case.status == TestStatus.TEST_CRASHED: - counts.crashed += 1 - print_with_timestamp(red('[CRASHED] ' + test_case.name)) - print_log(map(yellow, test_case.log)) - print_with_timestamp('') + Parameters: + log - Iterable object with all strings saved in log for test + """ + for m in log: + print_with_timestamp(yellow(m)) + +def format_test_result(test: Test) -> str: + """ + Returns string with formatted test result with colored status and test + name. + + Example: + '[PASSED] example' + + Parameters: + test - Test object representing current test being printed + + Return: + String containing formatted test result + """ + if test.status == TestStatus.SUCCESS: + return (green('[PASSED] ') + test.name) + elif test.status == TestStatus.SKIPPED: + return (yellow('[SKIPPED] ') + test.name) + elif test.status == TestStatus.TEST_CRASHED: + print_log(test.log) + return (red('[CRASHED] ') + test.name) + else: + print_log(test.log) + return (red('[FAILED] ') + test.name) + +def print_test_result(test: Test) -> None: + """ + Prints result line with status of test. + + Example: + '[PASSED] example' + + Parameters: + test - Test object representing current test being printed + """ + print_with_timestamp(format_test_result(test)) + +def print_test_footer(test: Test) -> None: + """ + Prints test footer with status of test. + + Example: + '===================== [PASSED] example =====================' + + Parameters: + test - Test object representing current test being printed + """ + message = format_test_result(test) + print_with_timestamp(format_test_divider(message, + len(message) - ANSI_LEN)) + +def print_summary_line(test: Test) -> None: + """ + Prints summary line of test object. Color of line is dependent on + status of test. Color is green if test passes, yellow if test is + skipped, and red if the test fails or crashes. Summary line contains + counts of the statuses of the tests subtests or the test itself if it + has no subtests. + + Example: + "Testing complete. Passed: 2, Failed: 0, Crashed: 0, Skipped: 0, + Errors: 0" + + test - Test object representing current test being printed + """ + if test.status == TestStatus.SUCCESS: + color = green + elif test.status == TestStatus.SKIPPED or test.status == TestStatus.NO_TESTS: + color = yellow + else: + color = red + counts = test.counts + print_with_timestamp(color('Testing complete. ' + str(counts))) + +def print_error(error_message: str) -> None: + """ + Prints error message with error format. + + Example: + "[ERROR] Test example: missing test plan!" + + Parameters: + error_message - message describing error + """ + print_with_timestamp(red('[ERROR] ') + error_message) + +# Other methods: + +def bubble_up_test_results(test: Test) -> None: + """ + If the test has subtests, add the test counts of the subtests to the + test and check if any of the tests crashed and if so set the test + status to crashed. Otherwise if the test has no subtests add the + status of the test to the test counts. + + Parameters: + test - Test object for current test being parsed + """ + parse_crash_in_log(test) + subtests = test.subtests + counts = test.counts + status = test.status + for t in subtests: + counts.add_subtest_counts(t.counts) + if counts.total() == 0: + counts.add_status(status) + elif test.counts.get_status() == TestStatus.TEST_CRASHED: + test.status = TestStatus.TEST_CRASHED + +def parse_test(lines: LineStream, expected_num: int, log: List[str]) -> Test: + """ + Finds next test to parse in LineStream, creates new Test object, + parses any subtests of the test, populates Test object with all + information (status, name) about the test and the Test objects for + any subtests, and then returns the Test object. The method accepts + three formats of tests: + + Accepted test formats: + + - Main KTAP/TAP header + + Example: + + KTAP version 1 + 1..4 + [subtests] + + - Subtest header line + + Example: + + # Subtest: name + 1..3 + [subtests] + ok 1 name + + - Test result line + + Example: + + ok 1 - test + + Parameters: + lines - LineStream of KTAP output to parse + expected_num - expected test number for test to be parsed + log - list of strings containing any preceding diagnostic lines + corresponding to the current test + + Return: + Test object populated with characteristics and any subtests + """ + test = Test() + test.log.extend(log) + parent_test = False + main = parse_ktap_header(lines, test) + if main: + # If KTAP/TAP header is found, attempt to parse + # test plan + test.name = "main" + parse_test_plan(lines, test) + else: + # If KTAP/TAP header is not found, test must be subtest + # header or test result line so parse attempt to parser + # subtest header + parent_test = parse_test_header(lines, test) + if parent_test: + # If subtest header is found, attempt to parse + # test plan and print header + parse_test_plan(lines, test) + print_test_header(test) + expected_count = test.expected_count + subtests = [] + test_num = 1 + while expected_count is None or test_num <= expected_count: + # Loop to parse any subtests. + # Break after parsing expected number of tests or + # if expected number of tests is unknown break when test + # result line with matching name to subtest header is found + # or no more lines in stream. + sub_log = parse_diagnostic(lines) + sub_test = Test() + if not lines or (peek_test_name_match(lines, test) and + not main): + if expected_count and test_num <= expected_count: + # If parser reaches end of test before + # parsing expected number of subtests, print + # crashed subtest and record error + test.add_error('missing expected subtest!') + sub_test.log.extend(sub_log) + test.counts.add_status( + TestStatus.TEST_CRASHED) + print_test_result(sub_test) else: - counts.failed += 1 - print_with_timestamp(red('[FAILED] ') + test_case.name) - print_log(map(yellow, test_case.log)) - print_with_timestamp('') - return counts + test.log.extend(sub_log) + break + else: + sub_test = parse_test(lines, test_num, sub_log) + subtests.append(sub_test) + test_num += 1 + test.subtests = subtests + if not main: + # If not main test, look for test result line + test.log.extend(parse_diagnostic(lines)) + if (parent_test and peek_test_name_match(lines, test)) or \ + not parent_test: + parse_test_result(lines, test, expected_num) + else: + test.add_error('missing subtest result line!') + # Add statuses to TestCounts attribute in Test object + bubble_up_test_results(test) + if parent_test: + # If test has subtests and is not the main test object, print + # footer. + print_test_footer(test) + elif not main: + print_test_result(test) + return test def parse_run_tests(kernel_output: Iterable[str]) -> TestResult: - counts = TestCounts() + """ + Using kernel output, extract KTAP lines, parse the lines for test + results and print condensed test results and summary line . + + Parameters: + kernel_output - Iterable object contains lines of kernel output + + Return: + TestResult - Tuple containg status of main test object, main test + object with all subtests, and log of all KTAP lines. + """ + print_with_timestamp(DIVIDER) lines = extract_tap_lines(kernel_output) - test_result = parse_test_result(lines) - if test_result.status == TestStatus.NO_TESTS: - print(red('[ERROR] ') + yellow('no tests run!')) - elif test_result.status == TestStatus.FAILURE_TO_PARSE_TESTS: - print(red('[ERROR] ') + yellow('could not parse test results!')) + test = Test() + if not lines: + test.add_error('invalid KTAP input!') + test.status = TestStatus.FAILURE_TO_PARSE_TESTS else: - counts = print_and_count_results(test_result) + test = parse_test(lines, 0, []) + if test.status != TestStatus.NO_TESTS: + test.status = test.counts.get_status() print_with_timestamp(DIVIDER) - if test_result.status == TestStatus.SUCCESS: - fmt = green - elif test_result.status == TestStatus.SKIPPED: - fmt = yellow - else: - fmt =red - print_with_timestamp( - fmt('Testing complete. %d tests run. %d failed. %d crashed. %d skipped.' % - (counts.total(), counts.failed, counts.crashed, counts.skipped))) - return test_result + print_summary_line(test) + return TestResult(test.status, test, lines) diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index 6e01140cb60e..0c7483d35f53 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -107,10 +107,10 @@ class KUnitParserTest(unittest.TestCase): with open(log_path) as file: result = kunit_parser.extract_tap_lines(file.readlines()) self.assertContains('TAP version 14', result) - self.assertContains(' # Subtest: example', result) - self.assertContains(' 1..2', result) - self.assertContains(' ok 1 - example_simple_test', result) - self.assertContains(' ok 2 - example_mock_test', result) + self.assertContains('# Subtest: example', result) + self.assertContains('1..2', result) + self.assertContains('ok 1 - example_simple_test', result) + self.assertContains('ok 2 - example_mock_test', result) self.assertContains('ok 1 - example', result) def test_output_with_prefix_isolated_correctly(self): @@ -118,28 +118,28 @@ class KUnitParserTest(unittest.TestCase): with open(log_path) as file: result = kunit_parser.extract_tap_lines(file.readlines()) self.assertContains('TAP version 14', result) - self.assertContains(' # Subtest: kunit-resource-test', result) - self.assertContains(' 1..5', result) - self.assertContains(' ok 1 - kunit_resource_test_init_resources', result) - self.assertContains(' ok 2 - kunit_resource_test_alloc_resource', result) - self.assertContains(' ok 3 - kunit_resource_test_destroy_resource', result) - self.assertContains(' foo bar #', result) - self.assertContains(' ok 4 - kunit_resource_test_cleanup_resources', result) - self.assertContains(' ok 5 - kunit_resource_test_proper_free_ordering', result) + self.assertContains('# Subtest: kunit-resource-test', result) + self.assertContains('1..5', result) + self.assertContains('ok 1 - kunit_resource_test_init_resources', result) + self.assertContains('ok 2 - kunit_resource_test_alloc_resource', result) + self.assertContains('ok 3 - kunit_resource_test_destroy_resource', result) + self.assertContains('foo bar #', result) + self.assertContains('ok 4 - kunit_resource_test_cleanup_resources', result) + self.assertContains('ok 5 - kunit_resource_test_proper_free_ordering', result) self.assertContains('ok 1 - kunit-resource-test', result) - self.assertContains(' foo bar # non-kunit output', result) - self.assertContains(' # Subtest: kunit-try-catch-test', result) - self.assertContains(' 1..2', result) - self.assertContains(' ok 1 - kunit_test_try_catch_successful_try_no_catch', + self.assertContains('foo bar # non-kunit output', result) + self.assertContains('# Subtest: kunit-try-catch-test', result) + self.assertContains('1..2', result) + self.assertContains('ok 1 - kunit_test_try_catch_successful_try_no_catch', result) - self.assertContains(' ok 2 - kunit_test_try_catch_unsuccessful_try_does_catch', + self.assertContains('ok 2 - kunit_test_try_catch_unsuccessful_try_does_catch', result) self.assertContains('ok 2 - kunit-try-catch-test', result) - self.assertContains(' # Subtest: string-stream-test', result) - self.assertContains(' 1..3', result) - self.assertContains(' ok 1 - string_stream_test_empty_on_creation', result) - self.assertContains(' ok 2 - string_stream_test_not_empty_after_add', result) - self.assertContains(' ok 3 - string_stream_test_get_string', result) + self.assertContains('# Subtest: string-stream-test', result) + self.assertContains('1..3', result) + self.assertContains('ok 1 - string_stream_test_empty_on_creation', result) + self.assertContains('ok 2 - string_stream_test_not_empty_after_add', result) + self.assertContains('ok 3 - string_stream_test_get_string', result) self.assertContains('ok 3 - string-stream-test', result) def test_parse_successful_test_log(self): @@ -150,6 +150,22 @@ class KUnitParserTest(unittest.TestCase): kunit_parser.TestStatus.SUCCESS, result.status) + def test_parse_successful_nested_tests_log(self): + all_passed_log = test_data_path('test_is_test_passed-all_passed_nested.log') + with open(all_passed_log) as file: + result = kunit_parser.parse_run_tests(file.readlines()) + self.assertEqual( + kunit_parser.TestStatus.SUCCESS, + result.status) + + def test_kselftest_nested(self): + kselftest_log = test_data_path('test_is_test_passed-kselftest.log') + with open(kselftest_log) as file: + result = kunit_parser.parse_run_tests(file.readlines()) + self.assertEqual( + kunit_parser.TestStatus.SUCCESS, + result.status) + def test_parse_failed_test_log(self): failed_log = test_data_path('test_is_test_passed-failure.log') with open(failed_log) as file: @@ -163,17 +179,29 @@ class KUnitParserTest(unittest.TestCase): with open(empty_log) as file: result = kunit_parser.parse_run_tests( kunit_parser.extract_tap_lines(file.readlines())) - self.assertEqual(0, len(result.suites)) + self.assertEqual(0, len(result.test.subtests)) self.assertEqual( kunit_parser.TestStatus.FAILURE_TO_PARSE_TESTS, result.status) + def test_missing_test_plan(self): + missing_plan_log = test_data_path('test_is_test_passed-' + 'missing_plan.log') + with open(missing_plan_log) as file: + result = kunit_parser.parse_run_tests( + kunit_parser.extract_tap_lines( + file.readlines())) + self.assertEqual(2, result.test.counts.errors) + self.assertEqual( + kunit_parser.TestStatus.SUCCESS, + result.status) + def test_no_tests(self): - empty_log = test_data_path('test_is_test_passed-no_tests_run_with_header.log') - with open(empty_log) as file: + header_log = test_data_path('test_is_test_passed-no_tests_run_with_header.log') + with open(header_log) as file: result = kunit_parser.parse_run_tests( kunit_parser.extract_tap_lines(file.readlines())) - self.assertEqual(0, len(result.suites)) + self.assertEqual(0, len(result.test.subtests)) self.assertEqual( kunit_parser.TestStatus.NO_TESTS, result.status) @@ -184,14 +212,15 @@ class KUnitParserTest(unittest.TestCase): with open(crash_log) as file: result = kunit_parser.parse_run_tests( kunit_parser.extract_tap_lines(file.readlines())) - print_mock.assert_any_call(StrContains('could not parse test results!')) + print_mock.assert_any_call(StrContains('invalid KTAP input!')) print_mock.stop() - self.assertEqual(0, len(result.suites)) + self.assertEqual(0, len(result.test.subtests)) def test_crashed_test(self): crashed_log = test_data_path('test_is_test_passed-crash.log') with open(crashed_log) as file: - result = kunit_parser.parse_run_tests(file.readlines()) + result = kunit_parser.parse_run_tests( + file.readlines()) self.assertEqual( kunit_parser.TestStatus.TEST_CRASHED, result.status) @@ -215,6 +244,23 @@ class KUnitParserTest(unittest.TestCase): kunit_parser.TestStatus.SKIPPED, result.status) + def test_ignores_hyphen(self): + hyphen_log = test_data_path('test_strip_hyphen.log') + file = open(hyphen_log) + result = kunit_parser.parse_run_tests(file.readlines()) + + # A skipped test does not fail the whole suite. + self.assertEqual( + kunit_parser.TestStatus.SUCCESS, + result.status) + self.assertEqual( + "sysctl_test", + result.test.subtests[0].name) + self.assertEqual( + "example", + result.test.subtests[1].name) + file.close() + def test_ignores_prefix_printk_time(self): prefix_log = test_data_path('test_config_printk_time.log') @@ -223,7 +269,7 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual( kunit_parser.TestStatus.SUCCESS, result.status) - self.assertEqual('kunit-resource-test', result.suites[0].name) + self.assertEqual('kunit-resource-test', result.test.subtests[0].name) def test_ignores_multiple_prefixes(self): prefix_log = test_data_path('test_multiple_prefixes.log') @@ -232,7 +278,7 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual( kunit_parser.TestStatus.SUCCESS, result.status) - self.assertEqual('kunit-resource-test', result.suites[0].name) + self.assertEqual('kunit-resource-test', result.test.subtests[0].name) def test_prefix_mixed_kernel_output(self): mixed_prefix_log = test_data_path('test_interrupted_tap_output.log') @@ -241,7 +287,7 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual( kunit_parser.TestStatus.SUCCESS, result.status) - self.assertEqual('kunit-resource-test', result.suites[0].name) + self.assertEqual('kunit-resource-test', result.test.subtests[0].name) def test_prefix_poundsign(self): pound_log = test_data_path('test_pound_sign.log') @@ -250,7 +296,7 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual( kunit_parser.TestStatus.SUCCESS, result.status) - self.assertEqual('kunit-resource-test', result.suites[0].name) + self.assertEqual('kunit-resource-test', result.test.subtests[0].name) def test_kernel_panic_end(self): panic_log = test_data_path('test_kernel_panic_interrupt.log') @@ -259,7 +305,7 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual( kunit_parser.TestStatus.TEST_CRASHED, result.status) - self.assertEqual('kunit-resource-test', result.suites[0].name) + self.assertEqual('kunit-resource-test', result.test.subtests[0].name) def test_pound_no_prefix(self): pound_log = test_data_path('test_pound_no_prefix.log') @@ -268,7 +314,7 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual( kunit_parser.TestStatus.SUCCESS, result.status) - self.assertEqual('kunit-resource-test', result.suites[0].name) + self.assertEqual('kunit-resource-test', result.test.subtests[0].name) class LinuxSourceTreeTest(unittest.TestCase): @@ -341,6 +387,12 @@ class KUnitJsonTest(unittest.TestCase): result = self._json_for('test_is_test_passed-no_tests_run_with_header.log') self.assertEqual(0, len(result['sub_groups'])) + def test_nested_json(self): + result = self._json_for('test_is_test_passed-all_passed_nested.log') + self.assertEqual( + {'name': 'example_simple_test', 'status': 'PASS'}, + result["sub_groups"][0]["sub_groups"][0]["test_cases"][0]) + class StrContains(str): def __eq__(self, other): return self in other @@ -399,7 +451,15 @@ class KUnitMainTest(unittest.TestCase): self.assertEqual(e.exception.code, 1) self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 1) self.assertEqual(self.linux_source_mock.run_kernel.call_count, 1) - self.print_mock.assert_any_call(StrContains(' 0 tests run')) + self.print_mock.assert_any_call(StrContains('invalid KTAP input!')) + + def test_exec_no_tests(self): + self.linux_source_mock.run_kernel = mock.Mock(return_value=['TAP version 14', '1..0']) + with self.assertRaises(SystemExit) as e: + kunit.main(['run'], self.linux_source_mock) + self.linux_source_mock.run_kernel.assert_called_once_with( + args=None, build_dir='.kunit', filter_glob='', timeout=300) + self.print_mock.assert_any_call(StrContains(' 0 tests run!')) def test_exec_raw_output(self): self.linux_source_mock.run_kernel = mock.Mock(return_value=[]) @@ -407,7 +467,7 @@ class KUnitMainTest(unittest.TestCase): self.assertEqual(self.linux_source_mock.run_kernel.call_count, 1) for call in self.print_mock.call_args_list: self.assertNotEqual(call, mock.call(StrContains('Testing complete.'))) - self.assertNotEqual(call, mock.call(StrContains(' 0 tests run'))) + self.assertNotEqual(call, mock.call(StrContains(' 0 tests run!'))) def test_run_raw_output(self): self.linux_source_mock.run_kernel = mock.Mock(return_value=[]) @@ -416,7 +476,7 @@ class KUnitMainTest(unittest.TestCase): self.assertEqual(self.linux_source_mock.run_kernel.call_count, 1) for call in self.print_mock.call_args_list: self.assertNotEqual(call, mock.call(StrContains('Testing complete.'))) - self.assertNotEqual(call, mock.call(StrContains(' 0 tests run'))) + self.assertNotEqual(call, mock.call(StrContains(' 0 tests run!'))) def test_run_raw_output_kunit(self): self.linux_source_mock.run_kernel = mock.Mock(return_value=[]) diff --git a/tools/testing/kunit/test_data/test_is_test_passed-all_passed_nested.log b/tools/testing/kunit/test_data/test_is_test_passed-all_passed_nested.log new file mode 100644 index 000000000000..9d5b04fe43a6 --- /dev/null +++ b/tools/testing/kunit/test_data/test_is_test_passed-all_passed_nested.log @@ -0,0 +1,34 @@ +TAP version 14 +1..2 + # Subtest: sysctl_test + 1..4 + # sysctl_test_dointvec_null_tbl_data: sysctl_test_dointvec_null_tbl_data passed + ok 1 - sysctl_test_dointvec_null_tbl_data + # Subtest: example + 1..2 + init_suite + # example_simple_test: initializing + # example_simple_test: example_simple_test passed + ok 1 - example_simple_test + # example_mock_test: initializing + # example_mock_test: example_mock_test passed + ok 2 - example_mock_test + kunit example: all tests passed + ok 2 - example + # sysctl_test_dointvec_table_len_is_zero: sysctl_test_dointvec_table_len_is_zero passed + ok 3 - sysctl_test_dointvec_table_len_is_zero + # sysctl_test_dointvec_table_read_but_position_set: sysctl_test_dointvec_table_read_but_position_set passed + ok 4 - sysctl_test_dointvec_table_read_but_position_set +kunit sysctl_test: all tests passed +ok 1 - sysctl_test + # Subtest: example + 1..2 +init_suite + # example_simple_test: initializing + # example_simple_test: example_simple_test passed + ok 1 - example_simple_test + # example_mock_test: initializing + # example_mock_test: example_mock_test passed + ok 2 - example_mock_test +kunit example: all tests passed +ok 2 - example diff --git a/tools/testing/kunit/test_data/test_is_test_passed-kselftest.log b/tools/testing/kunit/test_data/test_is_test_passed-kselftest.log new file mode 100644 index 000000000000..65d3f27feaf2 --- /dev/null +++ b/tools/testing/kunit/test_data/test_is_test_passed-kselftest.log @@ -0,0 +1,14 @@ +TAP version 13 +1..2 +# selftests: membarrier: membarrier_test_single_thread +# TAP version 13 +# 1..2 +# ok 1 sys_membarrier available +# ok 2 sys membarrier invalid command test: command = -1, flags = 0, errno = 22. Failed as expected +ok 1 selftests: membarrier: membarrier_test_single_thread +# selftests: membarrier: membarrier_test_multi_thread +# TAP version 13 +# 1..2 +# ok 1 sys_membarrier available +# ok 2 sys membarrier invalid command test: command = -1, flags = 0, errno = 22. Failed as expected +ok 2 selftests: membarrier: membarrier_test_multi_thread diff --git a/tools/testing/kunit/test_data/test_is_test_passed-missing_plan.log b/tools/testing/kunit/test_data/test_is_test_passed-missing_plan.log new file mode 100644 index 000000000000..5cd17b7f818a --- /dev/null +++ b/tools/testing/kunit/test_data/test_is_test_passed-missing_plan.log @@ -0,0 +1,31 @@ +KTAP version 1 + # Subtest: sysctl_test + # sysctl_test_dointvec_null_tbl_data: sysctl_test_dointvec_null_tbl_data passed + ok 1 - sysctl_test_dointvec_null_tbl_data + # sysctl_test_dointvec_table_maxlen_unset: sysctl_test_dointvec_table_maxlen_unset passed + ok 2 - sysctl_test_dointvec_table_maxlen_unset + # sysctl_test_dointvec_table_len_is_zero: sysctl_test_dointvec_table_len_is_zero passed + ok 3 - sysctl_test_dointvec_table_len_is_zero + # sysctl_test_dointvec_table_read_but_position_set: sysctl_test_dointvec_table_read_but_position_set passed + ok 4 - sysctl_test_dointvec_table_read_but_position_set + # sysctl_test_dointvec_happy_single_positive: sysctl_test_dointvec_happy_single_positive passed + ok 5 - sysctl_test_dointvec_happy_single_positive + # sysctl_test_dointvec_happy_single_negative: sysctl_test_dointvec_happy_single_negative passed + ok 6 - sysctl_test_dointvec_happy_single_negative + # sysctl_test_dointvec_single_less_int_min: sysctl_test_dointvec_single_less_int_min passed + ok 7 - sysctl_test_dointvec_single_less_int_min + # sysctl_test_dointvec_single_greater_int_max: sysctl_test_dointvec_single_greater_int_max passed + ok 8 - sysctl_test_dointvec_single_greater_int_max +kunit sysctl_test: all tests passed +ok 1 - sysctl_test + # Subtest: example + 1..2 +init_suite + # example_simple_test: initializing + # example_simple_test: example_simple_test passed + ok 1 - example_simple_test + # example_mock_test: initializing + # example_mock_test: example_mock_test passed + ok 2 - example_mock_test +kunit example: all tests passed +ok 2 - example diff --git a/tools/testing/kunit/test_data/test_strip_hyphen.log b/tools/testing/kunit/test_data/test_strip_hyphen.log new file mode 100644 index 000000000000..92ac7c24b374 --- /dev/null +++ b/tools/testing/kunit/test_data/test_strip_hyphen.log @@ -0,0 +1,16 @@ +KTAP version 1 +1..2 + # Subtest: sysctl_test + 1..1 + # sysctl_test_dointvec_null_tbl_data: sysctl_test_dointvec_null_tbl_data passed + ok 1 - sysctl_test_dointvec_null_tbl_data +kunit sysctl_test: all tests passed +ok 1 - sysctl_test + # Subtest: example + 1..1 +init_suite + # example_simple_test: initializing + # example_simple_test: example_simple_test passed + ok 1 example_simple_test +kunit example: all tests passed +ok 2 example base-commit: e7198adb84dcad671ad4f0e90aaa7e9fabf258dc -- 2.33.0.882.g93a45727a2-goog

3 years, 7 months

2
1
0 0

Re: [PATCH] selftests: bpf: Remove duplicated include in cgroup_helpers

by Jiabing Wan

On 2021/10/11 22:30, Daniel Borkmann wrote: > On 10/11/21 1:19 PM, Wan Jiabing wrote: >> Fix following checkincludes.pl warning: >> ./tools/testing/selftests/bpf/cgroup_helpers.c >> 12 #include <unistd.h> >> 14 #include <unistd.h> > > What does the 12 vs 14 mean here? Please provide a proper commit > description, e.g. if > you used checkincludes.pl, maybe include the full command invocation > and the relevant > output, so that this is more obvious and in a better shape. Thanks! Sorry for my fuzzy description. 12 and 14 mean the line of includes file. The script checkincludes.pl can only show which file is included duplicated, like ./scripts/checkincludes.pl tools/testing/selftests/bpf/cgroup_helpers.c tools/testing/selftests/bpf/cgroup_helpers.c: unistd.h is included more than once. So I upgrade the script so it can tell me the position of the duplicated includes. I'll fix the description in v2. >> Signed-off-by: Wan Jiabing <wanjiabing(a)vivo.com> >> --- >> tools/testing/selftests/bpf/cgroup_helpers.c | 1 - >> 1 file changed, 1 deletion(-) >> >> diff --git a/tools/testing/selftests/bpf/cgroup_helpers.c >> b/tools/testing/selftests/bpf/cgroup_helpers.c >> index 8fcd44841bb2..9d59c3990ca8 100644 >> --- a/tools/testing/selftests/bpf/cgroup_helpers.c >> +++ b/tools/testing/selftests/bpf/cgroup_helpers.c >> @@ -11,7 +11,6 @@ >> #include <fcntl.h> >> #include <unistd.h> >> #include <ftw.h> >> -#include <unistd.h> >> #include "cgroup_helpers.h" >> >

3 years, 7 months

1
0
0 0

[GIT PULL] KUnit fixes update for Linux 5.15-rc6

by Shuah Khan

Hi Linus, Please pull the following KUnit fixes update for Linux 5.15-rc6. This KUnit fixes update for Linux 5.15-rc6 consists of: - Fixes to address the structleak plugin causing the stack frame size to grow immensely when used with KUnit. Fixes include adding a new makefile to disable structleak and using it from KUnit iio, device property, thunderbolt, and bitfield tests to disable it. - KUnit framework reference count leak in kfree_at_end - KUnit tool fix to resolve conflict between --json and --raw_output and generate correct test output in either case. - kernel-doc warnings due to mismatched arg names diff is attached. thanks, -- Shuah ---------------------------------------------------------------- The following changes since commit 6880fa6c56601bb8ed59df6c30fd390cc5f6dd8f: Linux 5.15-rc1 (2021-09-12 16:28:37 -0700) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest tags/linux-kselftest-kunit-fixes-5.15-rc6 for you to fetch changes up to 361b57df62de249dc0b2acbf48823662a5001bcd: kunit: fix kernel-doc warnings due to mismatched arg names (2021-10-06 17:54:07 -0600) ---------------------------------------------------------------- linux-kselftest-kunit-fixes-5.15-rc6 This KUnit fixes update for Linux 5.15-rc6 consists of: - Fixes to address the structleak plugin causing the stack frame size to grow immensely when used with KUnit. Fixes include adding a new makefile to disable structleak and using it from KUnit iio, device property, thunderbolt, and bitfield tests to disable it. - KUnit framework reference count leak in kfree_at_end - KUnit tool fix to resolve conflict between --json and --raw_output and generate correct test output in either case. - kernel-doc warnings due to mismatched arg names ---------------------------------------------------------------- Arnd Bergmann (1): bitfield: build kunit tests without structleak plugin Brendan Higgins (4): gcc-plugins/structleak: add makefile var for disabling structleak iio/test-format: build kunit tests without structleak plugin device property: build kunit tests without structleak plugin thunderbolt: build kunit tests without structleak plugin Daniel Latypov (2): kunit: tool: better handling of quasi-bool args (--json, --raw_output) kunit: fix kernel-doc warnings due to mismatched arg names Xiyu Yang (1): kunit: fix reference count leak in kfree_at_end drivers/base/test/Makefile | 2 +- drivers/iio/test/Makefile | 1 + drivers/thunderbolt/Makefile | 1 + include/kunit/test.h | 6 +++--- lib/Makefile | 2 +- lib/kunit/executor_test.c | 4 ++-- scripts/Makefile.gcc-plugins | 4 ++++ tools/testing/kunit/kunit.py | 24 ++++++++++++++++++++++-- tools/testing/kunit/kunit_tool_test.py | 8 ++++++++ 9 files changed, 43 insertions(+), 9 deletions(-) ----------------------------------------------------------------

3 years, 7 months

2
1
0 0

[REGRESSION] lkft kselftest for next-20211007

by lkft＠linaro.org

## Build * kernel: 5.15.0-rc4 * git: https://gitlab.com/Linaro/lkft/mirrors/next/linux-next * git branch: master * git commit: f8dc23b3dc0cc5b32dfd0c446e59377736d073a7 * git describe: next-20211007 * test details: https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20211007 ## Regressions (compared to next-20211006) * qemu_x86_64, kselftest-pidfd - pidfd.pidfd_test Reported-by: Linux Kernel Functional Testing <lkft(a)linaro.org> ## Fixes (compared to next-20211006) * dragonboard-410c, kselftest-gpio - gpio.gpio-mockup.sh * dragonboard-410c, kselftest-rtc - rtc.rtctest.rtc.uie_read * hi6220-hikey, kselftest-proc - proc.proc-uptime-001 * qemu_x86_64, kselftest-seccomp - seccomp.seccomp_benchmark * x86, kselftest-lkdtm - lkdtm.SLAB_FREE_DOUBLE.sh ## Test result summary total: 1517, pass: 973, fail: 98, skip: 446, xfail: 0 ## Build Summary ## Test suites summary * kselftest-android * kselftest-arm64 * kselftest-arm64/arm64.btitest.bti_c_func * kselftest-arm64/arm64.btitest.bti_j_func * kselftest-arm64/arm64.btitest.bti_jc_func * kselftest-arm64/arm64.btitest.bti_none_func * kselftest-arm64/arm64.btitest.nohint_func * kselftest-arm64/arm64.btitest.paciasp_func * kselftest-arm64/arm64.nobtitest.bti_c_func * kselftest-arm64/arm64.nobtitest.bti_j_func * kselftest-arm64/arm64.nobtitest.bti_jc_func * kselftest-arm64/arm64.nobtitest.bti_none_func * kselftest-arm64/arm64.nobtitest.nohint_func * kselftest-arm64/arm64.nobtitest.paciasp_func * kselftest-bpf * kselftest-breakpoints * kselftest-capabilities * kselftest-cgroup * kselftest-clone3 * kselftest-core * kselftest-cpu-hotplug * kselftest-cpufreq * kselftest-drivers * kselftest-efivarfs * kselftest-filesystems * kselftest-firmware * kselftest-fpu * kselftest-futex * kselftest-gpio * kselftest-intel_pstate * kselftest-ipc * kselftest-ir * kselftest-kcmp * kselftest-kexec * kselftest-kvm * kselftest-lib * kselftest-livepatch * kselftest-lkdtm * kselftest-membarrier * kselftest-net * kselftest-netfilter * kselftest-nsfs * kselftest-openat2 * kselftest-pid_namespace * kselftest-pidfd * kselftest-proc * kselftest-pstore * kselftest-ptrace * kselftest-rseq * kselftest-rtc * kselftest-seccomp * kselftest-sigaltstack * kselftest-size * kselftest-splice * kselftest-static_keys * kselftest-sync * kselftest-sysctl * kselftest-tc-testing * kselftest-timens * kselftest-timers * kselftest-tmpfs * kselftest-tpm2 * kselftest-user * kselftest-vm * kselftest-x86 * kselftest-zram -- Linaro LKFT https://lkft.linaro.org

3 years, 7 months

1
0
0 0

[PATCH v7] kunit: tool: improve compatibility of kunit_parser with KTAP specification

by Daniel Latypov

From: Rae Moar <rmoar(a)google.com> Update to kunit_parser to improve compatibility with KTAP specification including arbitrarily nested tests. Patch accomplishes three major changes: - Use a general Test object to represent all tests rather than TestCase and TestSuite objects. This allows for easier implementation of arbitrary levels of nested tests and promotes the idea that both test suites and test cases are tests. - Print errors incrementally rather than all at once after the parsing finishes to maximize information given to the user in the case of the parser given invalid input and to increase the helpfulness of the timestamps given during printing. Note that kunit.py parse does not print incrementally yet. However, this fix brings us closer to this feature. - Increase compatibility for different formats of input. Arbitrary levels of nested tests supported. Also, test cases and test suites are now supported to be present on the same level of testing. This patch now implements the draft KTAP specification here: https://lore.kernel.org/linux-kselftest/CA+GJov6tdjvY9x12JsJT14qn6c7NViJxqa… We'll update the parser as the spec evolves. This patch adjusts the kunit_tool_test.py file to check for the correct outputs from the new parser and adds a new test to check the parsing for a KTAP result log with correct format for multiple nested subtests (test_is_test_passed-all_passed_nested.log). This patch also alters the kunit_json.py file to allow for arbitrarily nested tests. Signed-off-by: Rae Moar <rmoar(a)google.com> Reviewed-by: Brendan Higgins <brendanhiggins(a)google.com> Signed-off-by: Daniel Latypov <dlatypov(a)google.com> Reviewed-by: David Gow <davidgow(a)google.com> --- Change log from v6: https://lore.kernel.org/linux-kselftest/20211006170049.106852-1-dlatypov@go… - Rebase onto shuah/linux-kselftest/kunit - fix one new unit test failure (s/suites/test.subtests) Change log from v5: https://lore.kernel.org/linux-kselftest/20211006001447.20919-1-dlatypov@goo… - Tweak commit message to reflect the KTAP spec is a draft - Add missing Signed-off-by - Tweak docstrings Change log from v3,4: https://lore.kernel.org/linux-kselftest/20210901190623.315736-1-rmoar@googl… - Move test_kselftest_nested from LinuxSourceTreeTest => KUnitParserTest. - Resolve conflict with hermetic testing patches. - max_status is no longer defined, so we need to use the TestCounts type now. And to keep --raw_output working, we need to set this to SUCCESS to avoid the default assumption that the kernel crashed. Ignore v4, was accidentally based on v2. Change log from v2: https://lore.kernel.org/linux-kselftest/20210826195505.3066755-1-rmoar@goog… - Fixes bug of type disagreement in kunit_json.py for build_dir - Removes raw_output() - Changes docstrings in kunit_parser.py (class docstring, LineStream docstrings, add_error(), total(), get_status(), all parsing methods) - Fixes bug of not printing diagnostic log in the case of end of lines - Sets default status of all tests to TEST_CRASHED - Adds and prints empty tests with crashed status in case of missing tests - Prints 'subtest' in instance of 1 subtest instead of 'subtests' - Includes checking for 'BUG:' message in search of crash messages in log (note that parse_crash_in_log method could be removed but would require deleting tests in kunit_tool_test.py that include the crash message that is no longer used. If removed, parser would still print log in cases of test crashed or failure, which would now include missing subtests) - Fixes bug of including directives (other than SKIP) in test name when matching name in result line for subtests Change log from v1: https://lore.kernel.org/linux-kselftest/20210820200032.2178134-1-rmoar@goog… - Rebase onto kselftest/kunit branch - Add tests to kunit_tool_test.py to check parser is correctly stripping hyphen, producing correct json objects with nested tests, correctly passing kselftest TAP output, and correctly deals with missing test plan. - Fix bug to correctly match test name in instance of a missing test plan. - Fix bug in kunit_tool_test.py pointed out by Daniel where it was not correctly checking for a proper match to the '0 tests run!' error message. Reverts changes back to original. - A few minor changes to commit message using Daniel's comments. - Change docstrings using Daniel's comments to reduce: - Shortens some docstrings to be one-line or just description if it is self explanatory. - Remove explicit respecification of types of parameters and returns because this is already specified in the function annoations. However, some descriptions of the parameters and returns remain and some contain the type for context. Additionally, the types of public attributes of classes remain. - Remove any documentation of 'Return: None' - Remove docstrings of helper methods within other methods --- --- tools/testing/kunit/kunit.py | 19 +- tools/testing/kunit/kunit_json.py | 56 +- tools/testing/kunit/kunit_parser.py | 1021 ++++++++++++----- tools/testing/kunit/kunit_tool_test.py | 135 ++- .../test_is_test_passed-all_passed_nested.log | 34 + .../test_is_test_passed-kselftest.log | 14 + .../test_is_test_passed-missing_plan.log | 31 + .../kunit/test_data/test_strip_hyphen.log | 16 + 8 files changed, 941 insertions(+), 385 deletions(-) create mode 100644 tools/testing/kunit/test_data/test_is_test_passed-all_passed_nested.log create mode 100644 tools/testing/kunit/test_data/test_is_test_passed-kselftest.log create mode 100644 tools/testing/kunit/test_data/test_is_test_passed-missing_plan.log create mode 100644 tools/testing/kunit/test_data/test_strip_hyphen.log diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py index 7197e5fb8342..9c9ed4071e9e 100755 --- a/tools/testing/kunit/kunit.py +++ b/tools/testing/kunit/kunit.py @@ -135,7 +135,7 @@ def exec_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest, test_glob = request.filter_glob.split('.', maxsplit=2)[1] filter_globs = [g + '.'+ test_glob for g in filter_globs] - overall_status = kunit_parser.TestStatus.SUCCESS + test_counts = kunit_parser.TestCounts() exec_time = 0.0 for i, filter_glob in enumerate(filter_globs): kunit_parser.print_with_timestamp('Starting KUnit Kernel ({}/{})...'.format(i+1, len(filter_globs))) @@ -154,18 +154,29 @@ def exec_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest, test_end = time.time() exec_time += test_end - test_start - overall_status = kunit_parser.max_status(overall_status, result.status) + test_counts.add_subtest_counts(result.result.test.counts) - return KunitResult(status=result.status, result=result.result, elapsed_time=exec_time) + kunit_status = _map_to_overall_status(test_counts.get_status()) + return KunitResult(status=kunit_status, result=result.result, elapsed_time=exec_time) + +def _map_to_overall_status(test_status: kunit_parser.TestStatus) -> KunitStatus: + if test_status in (kunit_parser.TestStatus.SUCCESS, kunit_parser.TestStatus.SKIPPED): + return KunitStatus.SUCCESS + else: + return KunitStatus.TEST_FAILURE def parse_tests(request: KunitParseRequest, input_data: Iterable[str]) -> KunitResult: parse_start = time.time() test_result = kunit_parser.TestResult(kunit_parser.TestStatus.SUCCESS, - [], + kunit_parser.Test(), 'Tests not Parsed.') if request.raw_output: + # Treat unparsed results as one passing test. + test_result.test.status = kunit_parser.TestStatus.SUCCESS + test_result.test.counts.passed = 1 + output: Iterable[str] = input_data if request.raw_output == 'all': pass diff --git a/tools/testing/kunit/kunit_json.py b/tools/testing/kunit/kunit_json.py index f5cca5c38cac..746bec72b9ac 100644 --- a/tools/testing/kunit/kunit_json.py +++ b/tools/testing/kunit/kunit_json.py @@ -11,47 +11,47 @@ import os import kunit_parser -from kunit_parser import TestStatus - -def get_json_result(test_result, def_config, build_dir, json_path) -> str: - sub_groups = [] - - # Each test suite is mapped to a KernelCI sub_group - for test_suite in test_result.suites: - sub_group = { - "name": test_suite.name, - "arch": "UM", - "defconfig": def_config, - "build_environment": build_dir, - "test_cases": [], - "lab_name": None, - "kernel": None, - "job": None, - "git_branch": "kselftest", - } - test_cases = [] - # TODO: Add attachments attribute in test_case with detailed - # failure message, see https://api.kernelci.org/schema-test-case.html#get - for case in test_suite.cases: - test_case = {"name": case.name, "status": "FAIL"} - if case.status == TestStatus.SUCCESS: +from kunit_parser import Test, TestResult, TestStatus +from typing import Any, Dict, Optional + +JsonObj = Dict[str, Any] + +def _get_group_json(test: Test, def_config: str, + build_dir: Optional[str]) -> JsonObj: + sub_groups = [] # List[JsonObj] + test_cases = [] # List[JsonObj] + + for subtest in test.subtests: + if len(subtest.subtests): + sub_group = _get_group_json(subtest, def_config, + build_dir) + sub_groups.append(sub_group) + else: + test_case = {"name": subtest.name, "status": "FAIL"} + if subtest.status == TestStatus.SUCCESS: test_case["status"] = "PASS" - elif case.status == TestStatus.TEST_CRASHED: + elif subtest.status == TestStatus.TEST_CRASHED: test_case["status"] = "ERROR" test_cases.append(test_case) - sub_group["test_cases"] = test_cases - sub_groups.append(sub_group) + test_group = { - "name": "KUnit Test Group", + "name": test.name, "arch": "UM", "defconfig": def_config, "build_environment": build_dir, "sub_groups": sub_groups, + "test_cases": test_cases, "lab_name": None, "kernel": None, "job": None, "git_branch": "kselftest", } + return test_group + +def get_json_result(test_result: TestResult, def_config: str, + build_dir: Optional[str], json_path: str) -> str: + test_group = _get_group_json(test_result.test, def_config, build_dir) + test_group["name"] = "KUnit Test Group" json_obj = json.dumps(test_group, indent=4) if json_path != 'stdout': with open(json_path, 'w') as result_path: diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py index 6310a641b151..f01fd565f978 100644 --- a/tools/testing/kunit/kunit_parser.py +++ b/tools/testing/kunit/kunit_parser.py @@ -1,11 +1,15 @@ # SPDX-License-Identifier: GPL-2.0 # -# Parses test results from a kernel dmesg log. +# Parses KTAP test results from a kernel dmesg log and incrementally prints +# results with reader-friendly format. Stores and returns test results in a +# Test object. # # Copyright (C) 2019, Google LLC. # Author: Felix Guo <felixguoxiuping(a)gmail.com> # Author: Brendan Higgins <brendanhiggins(a)google.com> +# Author: Rae Moar <rmoar(a)google.com> +from __future__ import annotations import re from collections import namedtuple @@ -14,33 +18,52 @@ from enum import Enum, auto from functools import reduce from typing import Iterable, Iterator, List, Optional, Tuple -TestResult = namedtuple('TestResult', ['status','suites','log']) - -class TestSuite(object): +TestResult = namedtuple('TestResult', ['status','test','log']) + +class Test(object): + """ + A class to represent a test parsed from KTAP results. All KTAP + results within a test log are stored in a main Test object as + subtests. + + Attributes: + status : TestStatus - status of the test + name : str - name of the test + expected_count : int - expected number of subtests (0 if single + test case and None if unknown expected number of subtests) + subtests : List[Test] - list of subtests + log : List[str] - log of KTAP lines that correspond to the test + counts : TestCounts - counts of the test statuses and errors of + subtests or of the test itself if the test is a single + test case. + """ def __init__(self) -> None: - self.status = TestStatus.SUCCESS - self.name = '' - self.cases = [] # type: List[TestCase] - - def __str__(self) -> str: - return 'TestSuite(' + str(self.status) + ',' + self.name + ',' + str(self.cases) + ')' - - def __repr__(self) -> str: - return str(self) - -class TestCase(object): - def __init__(self) -> None: - self.status = TestStatus.SUCCESS + """Creates Test object with default attributes.""" + self.status = TestStatus.TEST_CRASHED self.name = '' + self.expected_count = 0 # type: Optional[int] + self.subtests = [] # type: List[Test] self.log = [] # type: List[str] + self.counts = TestCounts() def __str__(self) -> str: - return 'TestCase(' + str(self.status) + ',' + self.name + ',' + str(self.log) + ')' + """Returns string representation of a Test class object.""" + return ('Test(' + str(self.status) + ', ' + self.name + + ', ' + str(self.expected_count) + ', ' + + str(self.subtests) + ', ' + str(self.log) + ', ' + + str(self.counts) + ')') def __repr__(self) -> str: + """Returns string representation of a Test class object.""" return str(self) + def add_error(self, error_message: str) -> None: + """Records an error that occurred while parsing this test.""" + self.counts.errors += 1 + print_error('Test ' + self.name + ': ' + error_message) + class TestStatus(Enum): + """An enumeration class to represent the status of a test.""" SUCCESS = auto() FAILURE = auto() SKIPPED = auto() @@ -48,381 +71,753 @@ class TestStatus(Enum): NO_TESTS = auto() FAILURE_TO_PARSE_TESTS = auto() +class TestCounts: + """ + Tracks the counts of statuses of all test cases and any errors within + a Test. + + Attributes: + passed : int - the number of tests that have passed + failed : int - the number of tests that have failed + crashed : int - the number of tests that have crashed + skipped : int - the number of tests that have skipped + errors : int - the number of errors in the test and subtests + """ + def __init__(self): + """Creates TestCounts object with counts of all test + statuses and test errors set to 0. + """ + self.passed = 0 + self.failed = 0 + self.crashed = 0 + self.skipped = 0 + self.errors = 0 + + def __str__(self) -> str: + """Returns the string representation of a TestCounts object. + """ + return ('Passed: ' + str(self.passed) + + ', Failed: ' + str(self.failed) + + ', Crashed: ' + str(self.crashed) + + ', Skipped: ' + str(self.skipped) + + ', Errors: ' + str(self.errors)) + + def total(self) -> int: + """Returns the total number of test cases within a test + object, where a test case is a test with no subtests. + """ + return (self.passed + self.failed + self.crashed + + self.skipped) + + def add_subtest_counts(self, counts: TestCounts) -> None: + """ + Adds the counts of another TestCounts object to the current + TestCounts object. Used to add the counts of a subtest to the + parent test. + + Parameters: + counts - a different TestCounts object whose counts + will be added to the counts of the TestCounts object + """ + self.passed += counts.passed + self.failed += counts.failed + self.crashed += counts.crashed + self.skipped += counts.skipped + self.errors += counts.errors + + def get_status(self) -> TestStatus: + """Returns the aggregated status of a Test using test + counts. + """ + if self.total() == 0: + return TestStatus.NO_TESTS + elif self.crashed: + # If one of the subtests crash, the expected status + # of the Test is crashed. + return TestStatus.TEST_CRASHED + elif self.failed: + # Otherwise if one of the subtests fail, the + # expected status of the Test is failed. + return TestStatus.FAILURE + elif self.passed: + # Otherwise if one of the subtests pass, the + # expected status of the Test is passed. + return TestStatus.SUCCESS + else: + # Finally, if none of the subtests have failed, + # crashed, or passed, the expected status of the + # Test is skipped. + return TestStatus.SKIPPED + + def add_status(self, status: TestStatus) -> None: + """ + Increments count of inputted status. + + Parameters: + status - status to be added to the TestCounts object + """ + if status == TestStatus.SUCCESS or \ + status == TestStatus.NO_TESTS: + # if status is NO_TESTS the most appropriate + # attribute to increment is passed because + # the test did not fail, crash or get skipped. + self.passed += 1 + elif status == TestStatus.FAILURE: + self.failed += 1 + elif status == TestStatus.SKIPPED: + self.skipped += 1 + else: + self.crashed += 1 + class LineStream: - """Provides a peek()/pop() interface over an iterator of (line#, text).""" + """ + A class to represent the lines of kernel output. + Provides a peek()/pop() interface over an iterator of + (line#, text). + """ _lines: Iterator[Tuple[int, str]] _next: Tuple[int, str] _done: bool def __init__(self, lines: Iterator[Tuple[int, str]]): + """Creates a new LineStream that wraps the given iterator.""" self._lines = lines self._done = False self._next = (0, '') self._get_next() def _get_next(self) -> None: + """Advances the LineSteam to the next line.""" try: self._next = next(self._lines) except StopIteration: self._done = True def peek(self) -> str: + """Returns the current line, without advancing the LineStream. + """ return self._next[1] def pop(self) -> str: + """Returns the current line and advances the LineStream to + the next line. + """ n = self._next self._get_next() return n[1] def __bool__(self) -> bool: + """Returns True if stream has more lines.""" return not self._done # Only used by kunit_tool_test.py. def __iter__(self) -> Iterator[str]: + """Empties all lines stored in LineStream object into + Iterator object and returns the Iterator object. + """ while bool(self): yield self.pop() def line_number(self) -> int: + """Returns the line number of the current line.""" return self._next[0] -kunit_start_re = re.compile(r'TAP version [0-9]+$') -kunit_end_re = re.compile('(List of all partitions:|' - 'Kernel panic - not syncing: VFS:|reboot: System halted)') +# Parsing helper methods: + +KTAP_START = re.compile(r'KTAP version ([0-9]+)$') +TAP_START = re.compile(r'TAP version ([0-9]+)$') +KTAP_END = re.compile('(List of all partitions:|' + 'Kernel panic - not syncing: VFS:|reboot: System halted)') def extract_tap_lines(kernel_output: Iterable[str]) -> LineStream: - def isolate_kunit_output(kernel_output: Iterable[str]) -> Iterator[Tuple[int, str]]: + """Extracts KTAP lines from the kernel output.""" + def isolate_ktap_output(kernel_output: Iterable[str]) \ + -> Iterator[Tuple[int, str]]: line_num = 0 started = False for line in kernel_output: line_num += 1 - line = line.rstrip() # line always has a trailing \n - if kunit_start_re.search(line): + line = line.rstrip() # remove trailing \n + if not started and KTAP_START.search(line): + # start extracting KTAP lines and set prefix + # to number of characters before version line + prefix_len = len( + line.split('KTAP version')[0]) + started = True + yield line_num, line[prefix_len:] + elif not started and TAP_START.search(line): + # start extracting KTAP lines and set prefix + # to number of characters before version line prefix_len = len(line.split('TAP version')[0]) started = True yield line_num, line[prefix_len:] - elif kunit_end_re.search(line): + elif started and KTAP_END.search(line): + # stop extracting KTAP lines break elif started: - yield line_num, line[prefix_len:] - return LineStream(lines=isolate_kunit_output(kernel_output)) - -DIVIDER = '=' * 60 - -RESET = '\033[0;0m' - -def red(text) -> str: - return '\033[1;31m' + text + RESET - -def yellow(text) -> str: - return '\033[1;33m' + text + RESET - -def green(text) -> str: - return '\033[1;32m' + text + RESET - -def print_with_timestamp(message) -> None: - print('[%s] %s' % (datetime.now().strftime('%H:%M:%S'), message)) + # remove prefix and any indention and yield + # line with line number + line = line[prefix_len:].lstrip() + yield line_num, line + return LineStream(lines=isolate_ktap_output(kernel_output)) + +KTAP_VERSIONS = [1] +TAP_VERSIONS = [13, 14] + +def check_version(version_num: int, accepted_versions: List[int], + version_type: str, test: Test) -> None: + """ + Adds error to test object if version number is too high or too + low. + + Parameters: + version_num - The inputted version number from the parsed KTAP or TAP + header line + accepted_version - List of accepted KTAP or TAP versions + version_type - 'KTAP' or 'TAP' depending on the type of + version line. + test - Test object for current test being parsed + """ + if version_num < min(accepted_versions): + test.add_error(version_type + + ' version lower than expected!') + elif version_num > max(accepted_versions): + test.add_error( + version_type + ' version higher than expected!') + +def parse_ktap_header(lines: LineStream, test: Test) -> bool: + """ + Parses KTAP/TAP header line and checks version number. + Returns False if fails to parse KTAP/TAP header line. + + Accepted formats: + - 'KTAP version [version number]' + - 'TAP version [version number]' + + Parameters: + lines - LineStream of KTAP output to parse + test - Test object for current test being parsed + + Return: + True if successfully parsed KTAP/TAP header line + """ + ktap_match = KTAP_START.match(lines.peek()) + tap_match = TAP_START.match(lines.peek()) + if ktap_match: + version_num = int(ktap_match.group(1)) + check_version(version_num, KTAP_VERSIONS, 'KTAP', test) + elif tap_match: + version_num = int(tap_match.group(1)) + check_version(version_num, TAP_VERSIONS, 'TAP', test) + else: + return False + test.log.append(lines.pop()) + return True -def format_suite_divider(message) -> str: - return '======== ' + message + ' ========' +TEST_HEADER = re.compile(r'^# Subtest: (.*)$') -def print_suite_divider(message) -> None: - print_with_timestamp(DIVIDER) - print_with_timestamp(format_suite_divider(message)) +def parse_test_header(lines: LineStream, test: Test) -> bool: + """ + Parses test header and stores test name in test object. + Returns False if fails to parse test header line. -def print_log(log) -> None: - for m in log: - print_with_timestamp(m) + Accepted format: + - '# Subtest: [test name]' -TAP_ENTRIES = re.compile(r'^(TAP|[\s]*ok|[\s]*not ok|[\s]*[0-9]+\.\.[0-9]+|[\s]*# (Subtest:|.*: kunit test case crashed!)).*$') + Parameters: + lines - LineStream of KTAP output to parse + test - Test object for current test being parsed -def consume_non_diagnostic(lines: LineStream) -> None: - while lines and not TAP_ENTRIES.match(lines.peek()): - lines.pop() - -def save_non_diagnostic(lines: LineStream, test_case: TestCase) -> None: - while lines and not TAP_ENTRIES.match(lines.peek()): - test_case.log.append(lines.peek()) - lines.pop() + Return: + True if successfully parsed test header line + """ + match = TEST_HEADER.match(lines.peek()) + if not match: + return False + test.log.append(lines.pop()) + test.name = match.group(1) + return True -OkNotOkResult = namedtuple('OkNotOkResult', ['is_ok','description', 'text']) +TEST_PLAN = re.compile(r'1\.\.([0-9]+)') -OK_NOT_OK_SKIP = re.compile(r'^[\s]*(ok|not ok) [0-9]+ - (.*) # SKIP(.*)$') +def parse_test_plan(lines: LineStream, test: Test) -> bool: + """ + Parses test plan line and stores the expected number of subtests in + test object. Reports an error if expected count is 0. + Returns False and reports missing test plan error if fails to parse + test plan. -OK_NOT_OK_SUBTEST = re.compile(r'^[\s]+(ok|not ok) [0-9]+ - (.*)$') + Accepted format: + - '1..[number of subtests]' -OK_NOT_OK_MODULE = re.compile(r'^(ok|not ok) ([0-9]+) - (.*)$') + Parameters: + lines - LineStream of KTAP output to parse + test - Test object for current test being parsed -def parse_ok_not_ok_test_case(lines: LineStream, test_case: TestCase) -> bool: - save_non_diagnostic(lines, test_case) - if not lines: - test_case.status = TestStatus.TEST_CRASHED - return True - line = lines.peek() - match = OK_NOT_OK_SUBTEST.match(line) - while not match and lines: - line = lines.pop() - match = OK_NOT_OK_SUBTEST.match(line) - if match: - test_case.log.append(lines.pop()) - test_case.name = match.group(2) - skip_match = OK_NOT_OK_SKIP.match(line) - if skip_match: - test_case.status = TestStatus.SKIPPED - return True - if test_case.status == TestStatus.TEST_CRASHED: - return True - if match.group(1) == 'ok': - test_case.status = TestStatus.SUCCESS - else: - test_case.status = TestStatus.FAILURE - return True - else: + Return: + True if successfully parsed test plan line + """ + match = TEST_PLAN.match(lines.peek()) + if not match: + test.expected_count = None + test.add_error('missing plan line!') return False - -SUBTEST_DIAGNOSTIC = re.compile(r'^[\s]+# (.*)$') -DIAGNOSTIC_CRASH_MESSAGE = re.compile(r'^[\s]+# .*?: kunit test case crashed!$') - -def parse_diagnostic(lines: LineStream, test_case: TestCase) -> bool: - save_non_diagnostic(lines, test_case) - if not lines: + test.log.append(lines.pop()) + expected_count = int(match.group(1)) + test.expected_count = expected_count + if expected_count == 0: + test.status = TestStatus.NO_TESTS + test.add_error('0 tests run!') + return True + +TEST_RESULT = re.compile(r'^(ok|not ok) ([0-9]+) (- )?([^#]*)( # .*)?$') + +TEST_RESULT_SKIP = re.compile(r'^(ok|not ok) ([0-9]+) (- )?(.*) # SKIP(.*)$') + +def peek_test_name_match(lines: LineStream, test: Test) -> bool: + """ + Matches current line with the format of a test result line and checks + if the name matches the name of the current test. + Returns False if fails to match format or name. + + Accepted format: + - '[ok|not ok] [test number] [-] [test name] [optional skip + directive]' + + Parameters: + lines - LineStream of KTAP output to parse + test - Test object for current test being parsed + + Return: + True if matched a test result line and the name matching the + expected test name + """ + line = lines.peek() + match = TEST_RESULT.match(line) + if not match: return False + name = match.group(4) + return (name == test.name) + +def parse_test_result(lines: LineStream, test: Test, + expected_num: int) -> bool: + """ + Parses test result line and stores the status and name in the test + object. Reports an error if the test number does not match expected + test number. + Returns False if fails to parse test result line. + + Note that the SKIP directive is the only direction that causes a + change in status. + + Accepted format: + - '[ok|not ok] [test number] [-] [test name] [optional skip + directive]' + + Parameters: + lines - LineStream of KTAP output to parse + test - Test object for current test being parsed + expected_num - expected test number for current test + + Return: + True if successfully parsed a test result line. + """ line = lines.peek() - match = SUBTEST_DIAGNOSTIC.match(line) - if match: - test_case.log.append(lines.pop()) - crash_match = DIAGNOSTIC_CRASH_MESSAGE.match(line) - if crash_match: - test_case.status = TestStatus.TEST_CRASHED - return True - else: + match = TEST_RESULT.match(line) + skip_match = TEST_RESULT_SKIP.match(line) + + # Check if line matches test result line format + if not match: return False + test.log.append(lines.pop()) -def parse_test_case(lines: LineStream) -> Optional[TestCase]: - test_case = TestCase() - save_non_diagnostic(lines, test_case) - while parse_diagnostic(lines, test_case): - pass - if parse_ok_not_ok_test_case(lines, test_case): - return test_case + # Set name of test object + if skip_match: + test.name = skip_match.group(4) else: - return None - -SUBTEST_HEADER = re.compile(r'^[\s]+# Subtest: (.*)$') - -def parse_subtest_header(lines: LineStream) -> Optional[str]: - consume_non_diagnostic(lines) - if not lines: - return None - match = SUBTEST_HEADER.match(lines.peek()) - if match: - lines.pop() - return match.group(1) + test.name = match.group(4) + + # Check test num + num = int(match.group(2)) + if num != expected_num: + test.add_error('Expected test number ' + + str(expected_num) + ' but found ' + str(num)) + + # Set status of test object + status = match.group(1) + if skip_match: + test.status = TestStatus.SKIPPED + elif status == 'ok': + test.status = TestStatus.SUCCESS else: - return None + test.status = TestStatus.FAILURE + return True + +def parse_diagnostic(lines: LineStream) -> List[str]: + """ + Parse lines that do not match the format of a test result line or + test header line and returns them in list. + + Line formats that are not parsed: + - '# Subtest: [test name]' + - '[ok|not ok] [test number] [-] [test name] [optional skip + directive]' + + Parameters: + lines - LineStream of KTAP output to parse + + Return: + Log of diagnostic lines + """ + log = [] # type: List[str] + while lines and not TEST_RESULT.match(lines.peek()) and not \ + TEST_HEADER.match(lines.peek()): + log.append(lines.pop()) + return log + +DIAGNOSTIC_CRASH_MESSAGE = re.compile( + r'^(BUG:|# .*?: kunit test case crashed!$)') + +def parse_crash_in_log(test: Test) -> bool: + """ + Iterate through the lines of the log to parse for crash message. + If crash message found, set status to crashed and return True. + Otherwise return False. + + Parameters: + test - Test object for current test being parsed + + Return: + True if crash message found in log + """ + for line in test.log: + if DIAGNOSTIC_CRASH_MESSAGE.match(line): + test.status = TestStatus.TEST_CRASHED + return True + return False -SUBTEST_PLAN = re.compile(r'[\s]+[0-9]+\.\.([0-9]+)') -def parse_subtest_plan(lines: LineStream) -> Optional[int]: - consume_non_diagnostic(lines) - match = SUBTEST_PLAN.match(lines.peek()) - if match: - lines.pop() - return int(match.group(1)) - else: - return None - -def max_status(left: TestStatus, right: TestStatus) -> TestStatus: - if left == right: - return left - elif left == TestStatus.TEST_CRASHED or right == TestStatus.TEST_CRASHED: - return TestStatus.TEST_CRASHED - elif left == TestStatus.FAILURE or right == TestStatus.FAILURE: - return TestStatus.FAILURE - elif left == TestStatus.SKIPPED: - return right - else: - return left +# Printing helper methods: -def parse_ok_not_ok_test_suite(lines: LineStream, - test_suite: TestSuite, - expected_suite_index: int) -> bool: - consume_non_diagnostic(lines) - if not lines: - test_suite.status = TestStatus.TEST_CRASHED - return False - line = lines.peek() - match = OK_NOT_OK_MODULE.match(line) - if match: - lines.pop() - if match.group(1) == 'ok': - test_suite.status = TestStatus.SUCCESS - else: - test_suite.status = TestStatus.FAILURE - skip_match = OK_NOT_OK_SKIP.match(line) - if skip_match: - test_suite.status = TestStatus.SKIPPED - suite_index = int(match.group(2)) - if suite_index != expected_suite_index: - print_with_timestamp( - red('[ERROR] ') + 'expected_suite_index ' + - str(expected_suite_index) + ', but got ' + - str(suite_index)) - return True - else: - return False +DIVIDER = '=' * 60 -def bubble_up_errors(status_list: Iterable[TestStatus]) -> TestStatus: - return reduce(max_status, status_list, TestStatus.SKIPPED) +RESET = '\033[0;0m' -def bubble_up_test_case_errors(test_suite: TestSuite) -> TestStatus: - max_test_case_status = bubble_up_errors(x.status for x in test_suite.cases) - return max_status(max_test_case_status, test_suite.status) +def red(text: str) -> str: + """Returns inputted string with red color code.""" + return '\033[1;31m' + text + RESET -def parse_test_suite(lines: LineStream, expected_suite_index: int) -> Optional[TestSuite]: - if not lines: - return None - consume_non_diagnostic(lines) - test_suite = TestSuite() - test_suite.status = TestStatus.SUCCESS - name = parse_subtest_header(lines) - if not name: - return None - test_suite.name = name - expected_test_case_num = parse_subtest_plan(lines) - if expected_test_case_num is None: - return None - while expected_test_case_num > 0: - test_case = parse_test_case(lines) - if not test_case: - break - test_suite.cases.append(test_case) - expected_test_case_num -= 1 - if parse_ok_not_ok_test_suite(lines, test_suite, expected_suite_index): - test_suite.status = bubble_up_test_case_errors(test_suite) - return test_suite - elif not lines: - print_with_timestamp(red('[ERROR] ') + 'ran out of lines before end token') - return test_suite - else: - print(f'failed to parse end of suite "{name}", at line {lines.line_number()}: {lines.peek()}') - return None +def yellow(text: str) -> str: + """Returns inputted string with yellow color code.""" + return '\033[1;33m' + text + RESET -TAP_HEADER = re.compile(r'^TAP version 14$') +def green(text: str) -> str: + """Returns inputted string with green color code.""" + return '\033[1;32m' + text + RESET -def parse_tap_header(lines: LineStream) -> bool: - consume_non_diagnostic(lines) - if TAP_HEADER.match(lines.peek()): - lines.pop() - return True - else: - return False +ANSI_LEN = len(red('')) -TEST_PLAN = re.compile(r'[0-9]+\.\.([0-9]+)') +def print_with_timestamp(message: str) -> None: + """Prints message with timestamp at beginning.""" + print('[%s] %s' % (datetime.now().strftime('%H:%M:%S'), message)) -def parse_test_plan(lines: LineStream) -> Optional[int]: - consume_non_diagnostic(lines) - match = TEST_PLAN.match(lines.peek()) - if match: - lines.pop() - return int(match.group(1)) - else: - return None - -def bubble_up_suite_errors(test_suites: Iterable[TestSuite]) -> TestStatus: - return bubble_up_errors(x.status for x in test_suites) - -def parse_test_result(lines: LineStream) -> TestResult: - consume_non_diagnostic(lines) - if not lines or not parse_tap_header(lines): - return TestResult(TestStatus.FAILURE_TO_PARSE_TESTS, [], lines) - expected_test_suite_num = parse_test_plan(lines) - if expected_test_suite_num == 0: - return TestResult(TestStatus.NO_TESTS, [], lines) - elif expected_test_suite_num is None: - return TestResult(TestStatus.FAILURE_TO_PARSE_TESTS, [], lines) - test_suites = [] - for i in range(1, expected_test_suite_num + 1): - test_suite = parse_test_suite(lines, i) - if test_suite: - test_suites.append(test_suite) +def format_test_divider(message: str, len_message: int) -> str: + """ + Returns string with message centered in fixed width divider. + + Example: + '===================== message example =====================' + + Parameters: + message - message to be centered in divider line + len_message - length of the message to be printed such that + any characters of the color codes are not counted + + Return: + String containing message centered in fixed width divider + """ + default_count = 3 # default number of dashes + len_1 = default_count + len_2 = default_count + difference = len(DIVIDER) - len_message - 2 # 2 spaces added + if difference > 0: + # calculate number of dashes for each side of the divider + len_1 = int(difference / 2) + len_2 = difference - len_1 + return ('=' * len_1) + ' ' + message + ' ' + ('=' * len_2) + +def print_test_header(test: Test) -> None: + """ + Prints test header with test name and optionally the expected number + of subtests. + + Example: + '=================== example (2 subtests) ===================' + + Parameters: + test - Test object representing current test being printed + """ + message = test.name + if test.expected_count: + if test.expected_count == 1: + message += (' (' + str(test.expected_count) + + ' subtest)') else: - print_with_timestamp( - red('[ERROR] ') + ' expected ' + - str(expected_test_suite_num) + - ' test suites, but got ' + str(i - 2)) - break - test_suite = parse_test_suite(lines, -1) - if test_suite: - print_with_timestamp(red('[ERROR] ') + - 'got unexpected test suite: ' + test_suite.name) - if test_suites: - return TestResult(bubble_up_suite_errors(test_suites), test_suites, lines) - else: - return TestResult(TestStatus.NO_TESTS, [], lines) + message += (' (' + str(test.expected_count) + + ' subtests)') + print_with_timestamp(format_test_divider(message, len(message))) -class TestCounts: - passed: int - failed: int - crashed: int - skipped: int +def print_log(log: Iterable[str]) -> None: + """ + Prints all strings in saved log for test in yellow. - def __init__(self): - self.passed = 0 - self.failed = 0 - self.crashed = 0 - self.skipped = 0 - - def total(self) -> int: - return self.passed + self.failed + self.crashed + self.skipped - -def print_and_count_results(test_result: TestResult) -> TestCounts: - counts = TestCounts() - for test_suite in test_result.suites: - if test_suite.status == TestStatus.SUCCESS: - print_suite_divider(green('[PASSED] ') + test_suite.name) - elif test_suite.status == TestStatus.SKIPPED: - print_suite_divider(yellow('[SKIPPED] ') + test_suite.name) - elif test_suite.status == TestStatus.TEST_CRASHED: - print_suite_divider(red('[CRASHED] ' + test_suite.name)) - else: - print_suite_divider(red('[FAILED] ') + test_suite.name) - for test_case in test_suite.cases: - if test_case.status == TestStatus.SUCCESS: - counts.passed += 1 - print_with_timestamp(green('[PASSED] ') + test_case.name) - elif test_case.status == TestStatus.SKIPPED: - counts.skipped += 1 - print_with_timestamp(yellow('[SKIPPED] ') + test_case.name) - elif test_case.status == TestStatus.TEST_CRASHED: - counts.crashed += 1 - print_with_timestamp(red('[CRASHED] ' + test_case.name)) - print_log(map(yellow, test_case.log)) - print_with_timestamp('') + Parameters: + log - Iterable object with all strings saved in log for test + """ + for m in log: + print_with_timestamp(yellow(m)) + +def format_test_result(test: Test) -> str: + """ + Returns string with formatted test result with colored status and test + name. + + Example: + '[PASSED] example' + + Parameters: + test - Test object representing current test being printed + + Return: + String containing formatted test result + """ + if test.status == TestStatus.SUCCESS: + return (green('[PASSED] ') + test.name) + elif test.status == TestStatus.SKIPPED: + return (yellow('[SKIPPED] ') + test.name) + elif test.status == TestStatus.TEST_CRASHED: + print_log(test.log) + return (red('[CRASHED] ') + test.name) + else: + print_log(test.log) + return (red('[FAILED] ') + test.name) + +def print_test_result(test: Test) -> None: + """ + Prints result line with status of test. + + Example: + '[PASSED] example' + + Parameters: + test - Test object representing current test being printed + """ + print_with_timestamp(format_test_result(test)) + +def print_test_footer(test: Test) -> None: + """ + Prints test footer with status of test. + + Example: + '===================== [PASSED] example =====================' + + Parameters: + test - Test object representing current test being printed + """ + message = format_test_result(test) + print_with_timestamp(format_test_divider(message, + len(message) - ANSI_LEN)) + +def print_summary_line(test: Test) -> None: + """ + Prints summary line of test object. Color of line is dependent on + status of test. Color is green if test passes, yellow if test is + skipped, and red if the test fails or crashes. Summary line contains + counts of the statuses of the tests subtests or the test itself if it + has no subtests. + + Example: + "Testing complete. Passed: 2, Failed: 0, Crashed: 0, Skipped: 0, + Errors: 0" + + test - Test object representing current test being printed + """ + if test.status == TestStatus.SUCCESS or \ + test.status == TestStatus.NO_TESTS: + color = green + elif test.status == TestStatus.SKIPPED: + color = yellow + else: + color = red + counts = test.counts + print_with_timestamp(color('Testing complete. ' + str(counts))) + +def print_error(error_message: str) -> None: + """ + Prints error message with error format. + + Example: + "[ERROR] Test example: missing test plan!" + + Parameters: + error_message - message describing error + """ + print_with_timestamp(red('[ERROR] ') + error_message) + +# Other methods: + +def bubble_up_test_results(test: Test) -> None: + """ + If the test has subtests, add the test counts of the subtests to the + test and check if any of the tests crashed and if so set the test + status to crashed. Otherwise if the test has no subtests add the + status of the test to the test counts. + + Parameters: + test - Test object for current test being parsed + """ + parse_crash_in_log(test) + subtests = test.subtests + counts = test.counts + status = test.status + for t in subtests: + counts.add_subtest_counts(t.counts) + if counts.total() == 0: + counts.add_status(status) + elif test.counts.get_status() == TestStatus.TEST_CRASHED: + test.status = TestStatus.TEST_CRASHED + +def parse_test(lines: LineStream, expected_num: int, log: List[str]) -> Test: + """ + Finds next test to parse in LineStream, creates new Test object, + parses any subtests of the test, populates Test object with all + information (status, name) about the test and the Test objects for + any subtests, and then returns the Test object. The method accepts + three formats of tests: + + Accepted test formats: + + - Main KTAP/TAP header + + Example: + + KTAP version 1 + 1..4 + [subtests] + + - Subtest header line + + Example: + + # Subtest: name + 1..3 + [subtests] + ok 1 name + + - Test result line + + Example: + + ok 1 - test + + Parameters: + lines - LineStream of KTAP output to parse + expected_num - expected test number for test to be parsed + log - list of strings containing any preceding diagnostic lines + corresponding to the current test + + Return: + Test object populated with characteristics and any subtests + """ + test = Test() + test.log.extend(log) + parent_test = False + main = parse_ktap_header(lines, test) + if main: + # If KTAP/TAP header is found, attempt to parse + # test plan + test.name = "main" + parse_test_plan(lines, test) + else: + # If KTAP/TAP header is not found, test must be subtest + # header or test result line so parse attempt to parser + # subtest header + parent_test = parse_test_header(lines, test) + if parent_test: + # If subtest header is found, attempt to parse + # test plan and print header + parse_test_plan(lines, test) + print_test_header(test) + expected_count = test.expected_count + subtests = [] + test_num = 1 + while expected_count is None or test_num <= expected_count: + # Loop to parse any subtests. + # Break after parsing expected number of tests or + # if expected number of tests is unknown break when test + # result line with matching name to subtest header is found + # or no more lines in stream. + sub_log = parse_diagnostic(lines) + sub_test = Test() + if not lines or (peek_test_name_match(lines, test) and + not main): + if expected_count and test_num <= expected_count: + # If parser reaches end of test before + # parsing expected number of subtests, print + # crashed subtest and record error + test.add_error('missing expected subtest!') + sub_test.log.extend(sub_log) + test.counts.add_status( + TestStatus.TEST_CRASHED) + print_test_result(sub_test) else: - counts.failed += 1 - print_with_timestamp(red('[FAILED] ') + test_case.name) - print_log(map(yellow, test_case.log)) - print_with_timestamp('') - return counts + test.log.extend(sub_log) + break + else: + sub_test = parse_test(lines, test_num, sub_log) + subtests.append(sub_test) + test_num += 1 + test.subtests = subtests + if not main: + # If not main test, look for test result line + test.log.extend(parse_diagnostic(lines)) + if (parent_test and peek_test_name_match(lines, test)) or \ + not parent_test: + parse_test_result(lines, test, expected_num) + else: + test.add_error('missing subtest result line!') + # Add statuses to TestCounts attribute in Test object + bubble_up_test_results(test) + if parent_test: + # If test has subtests and is not the main test object, print + # footer. + print_test_footer(test) + elif not main: + print_test_result(test) + return test def parse_run_tests(kernel_output: Iterable[str]) -> TestResult: - counts = TestCounts() + """ + Using kernel output, extract KTAP lines, parse the lines for test + results and print condensed test results and summary line . + + Parameters: + kernel_output - Iterable object contains lines of kernel output + + Return: + TestResult - Tuple containg status of main test object, main test + object with all subtests, and log of all KTAP lines. + """ + print_with_timestamp(DIVIDER) lines = extract_tap_lines(kernel_output) - test_result = parse_test_result(lines) - if test_result.status == TestStatus.NO_TESTS: - print(red('[ERROR] ') + yellow('no tests run!')) - elif test_result.status == TestStatus.FAILURE_TO_PARSE_TESTS: - print(red('[ERROR] ') + yellow('could not parse test results!')) + test = Test() + if not lines: + test.add_error('invalid KTAP input!') + test.status = TestStatus.FAILURE_TO_PARSE_TESTS else: - counts = print_and_count_results(test_result) + test = parse_test(lines, 0, []) + if test.status != TestStatus.NO_TESTS: + test.status = test.counts.get_status() print_with_timestamp(DIVIDER) - if test_result.status == TestStatus.SUCCESS: - fmt = green - elif test_result.status == TestStatus.SKIPPED: - fmt = yellow - else: - fmt =red - print_with_timestamp( - fmt('Testing complete. %d tests run. %d failed. %d crashed. %d skipped.' % - (counts.total(), counts.failed, counts.crashed, counts.skipped))) - return test_result + print_summary_line(test) + return TestResult(test.status, test, lines) diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index 6e01140cb60e..6648de1f9ceb 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -107,10 +107,10 @@ class KUnitParserTest(unittest.TestCase): with open(log_path) as file: result = kunit_parser.extract_tap_lines(file.readlines()) self.assertContains('TAP version 14', result) - self.assertContains(' # Subtest: example', result) - self.assertContains(' 1..2', result) - self.assertContains(' ok 1 - example_simple_test', result) - self.assertContains(' ok 2 - example_mock_test', result) + self.assertContains('# Subtest: example', result) + self.assertContains('1..2', result) + self.assertContains('ok 1 - example_simple_test', result) + self.assertContains('ok 2 - example_mock_test', result) self.assertContains('ok 1 - example', result) def test_output_with_prefix_isolated_correctly(self): @@ -118,28 +118,28 @@ class KUnitParserTest(unittest.TestCase): with open(log_path) as file: result = kunit_parser.extract_tap_lines(file.readlines()) self.assertContains('TAP version 14', result) - self.assertContains(' # Subtest: kunit-resource-test', result) - self.assertContains(' 1..5', result) - self.assertContains(' ok 1 - kunit_resource_test_init_resources', result) - self.assertContains(' ok 2 - kunit_resource_test_alloc_resource', result) - self.assertContains(' ok 3 - kunit_resource_test_destroy_resource', result) - self.assertContains(' foo bar #', result) - self.assertContains(' ok 4 - kunit_resource_test_cleanup_resources', result) - self.assertContains(' ok 5 - kunit_resource_test_proper_free_ordering', result) + self.assertContains('# Subtest: kunit-resource-test', result) + self.assertContains('1..5', result) + self.assertContains('ok 1 - kunit_resource_test_init_resources', result) + self.assertContains('ok 2 - kunit_resource_test_alloc_resource', result) + self.assertContains('ok 3 - kunit_resource_test_destroy_resource', result) + self.assertContains('foo bar #', result) + self.assertContains('ok 4 - kunit_resource_test_cleanup_resources', result) + self.assertContains('ok 5 - kunit_resource_test_proper_free_ordering', result) self.assertContains('ok 1 - kunit-resource-test', result) - self.assertContains(' foo bar # non-kunit output', result) - self.assertContains(' # Subtest: kunit-try-catch-test', result) - self.assertContains(' 1..2', result) - self.assertContains(' ok 1 - kunit_test_try_catch_successful_try_no_catch', + self.assertContains('foo bar # non-kunit output', result) + self.assertContains('# Subtest: kunit-try-catch-test', result) + self.assertContains('1..2', result) + self.assertContains('ok 1 - kunit_test_try_catch_successful_try_no_catch', result) - self.assertContains(' ok 2 - kunit_test_try_catch_unsuccessful_try_does_catch', + self.assertContains('ok 2 - kunit_test_try_catch_unsuccessful_try_does_catch', result) self.assertContains('ok 2 - kunit-try-catch-test', result) - self.assertContains(' # Subtest: string-stream-test', result) - self.assertContains(' 1..3', result) - self.assertContains(' ok 1 - string_stream_test_empty_on_creation', result) - self.assertContains(' ok 2 - string_stream_test_not_empty_after_add', result) - self.assertContains(' ok 3 - string_stream_test_get_string', result) + self.assertContains('# Subtest: string-stream-test', result) + self.assertContains('1..3', result) + self.assertContains('ok 1 - string_stream_test_empty_on_creation', result) + self.assertContains('ok 2 - string_stream_test_not_empty_after_add', result) + self.assertContains('ok 3 - string_stream_test_get_string', result) self.assertContains('ok 3 - string-stream-test', result) def test_parse_successful_test_log(self): @@ -150,6 +150,22 @@ class KUnitParserTest(unittest.TestCase): kunit_parser.TestStatus.SUCCESS, result.status) + def test_parse_successful_nested_tests_log(self): + all_passed_log = test_data_path('test_is_test_passed-all_passed_nested.log') + with open(all_passed_log) as file: + result = kunit_parser.parse_run_tests(file.readlines()) + self.assertEqual( + kunit_parser.TestStatus.SUCCESS, + result.status) + + def test_kselftest_nested(self): + kselftest_log = test_data_path('test_is_test_passed-kselftest.log') + with open(kselftest_log) as file: + result = kunit_parser.parse_run_tests(file.readlines()) + self.assertEqual( + kunit_parser.TestStatus.SUCCESS, + result.status) + def test_parse_failed_test_log(self): failed_log = test_data_path('test_is_test_passed-failure.log') with open(failed_log) as file: @@ -163,17 +179,31 @@ class KUnitParserTest(unittest.TestCase): with open(empty_log) as file: result = kunit_parser.parse_run_tests( kunit_parser.extract_tap_lines(file.readlines())) - self.assertEqual(0, len(result.suites)) + self.assertEqual(0, len(result.test.subtests)) self.assertEqual( kunit_parser.TestStatus.FAILURE_TO_PARSE_TESTS, result.status) + def test_missing_test_plan(self): + missing_plan_log = test_data_path('test_is_test_passed-' + 'missing_plan.log') + with open(missing_plan_log) as file: + result = kunit_parser.parse_run_tests( + kunit_parser.extract_tap_lines( + file.readlines())) + self.assertEqual(2, result.test.counts.errors) + self.assertEqual( + kunit_parser.TestStatus.SUCCESS, + result.status) + def test_no_tests(self): - empty_log = test_data_path('test_is_test_passed-no_tests_run_with_header.log') - with open(empty_log) as file: + header_log = test_data_path('test_is_test_passed-' + 'no_tests_run_with_header.log') + with open(header_log) as file: result = kunit_parser.parse_run_tests( - kunit_parser.extract_tap_lines(file.readlines())) - self.assertEqual(0, len(result.suites)) + kunit_parser.extract_tap_lines( + file.readlines())) + self.assertEqual(0, len(result.test.subtests)) self.assertEqual( kunit_parser.TestStatus.NO_TESTS, result.status) @@ -183,15 +213,17 @@ class KUnitParserTest(unittest.TestCase): print_mock = mock.patch('builtins.print').start() with open(crash_log) as file: result = kunit_parser.parse_run_tests( - kunit_parser.extract_tap_lines(file.readlines())) - print_mock.assert_any_call(StrContains('could not parse test results!')) + kunit_parser.extract_tap_lines( + file.readlines())) + print_mock.assert_any_call(StrContains('invalid KTAP input!')) print_mock.stop() - self.assertEqual(0, len(result.suites)) + self.assertEqual(0, len(result.test.subtests)) def test_crashed_test(self): crashed_log = test_data_path('test_is_test_passed-crash.log') with open(crashed_log) as file: - result = kunit_parser.parse_run_tests(file.readlines()) + result = kunit_parser.parse_run_tests( + file.readlines()) self.assertEqual( kunit_parser.TestStatus.TEST_CRASHED, result.status) @@ -215,6 +247,23 @@ class KUnitParserTest(unittest.TestCase): kunit_parser.TestStatus.SKIPPED, result.status) + def test_ignores_hyphen(self): + hyphen_log = test_data_path('test_strip_hyphen.log') + file = open(hyphen_log) + result = kunit_parser.parse_run_tests(file.readlines()) + + # A skipped test does not fail the whole suite. + self.assertEqual( + kunit_parser.TestStatus.SUCCESS, + result.status) + self.assertEqual( + "sysctl_test", + result.test.subtests[0].name) + self.assertEqual( + "example", + result.test.subtests[1].name) + file.close() + def test_ignores_prefix_printk_time(self): prefix_log = test_data_path('test_config_printk_time.log') @@ -223,7 +272,7 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual( kunit_parser.TestStatus.SUCCESS, result.status) - self.assertEqual('kunit-resource-test', result.suites[0].name) + self.assertEqual('kunit-resource-test', result.test.subtests[0].name) def test_ignores_multiple_prefixes(self): prefix_log = test_data_path('test_multiple_prefixes.log') @@ -232,7 +281,7 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual( kunit_parser.TestStatus.SUCCESS, result.status) - self.assertEqual('kunit-resource-test', result.suites[0].name) + self.assertEqual('kunit-resource-test', result.test.subtests[0].name) def test_prefix_mixed_kernel_output(self): mixed_prefix_log = test_data_path('test_interrupted_tap_output.log') @@ -241,7 +290,7 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual( kunit_parser.TestStatus.SUCCESS, result.status) - self.assertEqual('kunit-resource-test', result.suites[0].name) + self.assertEqual('kunit-resource-test', result.test.subtests[0].name) def test_prefix_poundsign(self): pound_log = test_data_path('test_pound_sign.log') @@ -250,7 +299,7 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual( kunit_parser.TestStatus.SUCCESS, result.status) - self.assertEqual('kunit-resource-test', result.suites[0].name) + self.assertEqual('kunit-resource-test', result.test.subtests[0].name) def test_kernel_panic_end(self): panic_log = test_data_path('test_kernel_panic_interrupt.log') @@ -259,7 +308,7 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual( kunit_parser.TestStatus.TEST_CRASHED, result.status) - self.assertEqual('kunit-resource-test', result.suites[0].name) + self.assertEqual('kunit-resource-test', result.test.subtests[0].name) def test_pound_no_prefix(self): pound_log = test_data_path('test_pound_no_prefix.log') @@ -268,7 +317,7 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual( kunit_parser.TestStatus.SUCCESS, result.status) - self.assertEqual('kunit-resource-test', result.suites[0].name) + self.assertEqual('kunit-resource-test', result.test.subtests[0].name) class LinuxSourceTreeTest(unittest.TestCase): @@ -341,6 +390,12 @@ class KUnitJsonTest(unittest.TestCase): result = self._json_for('test_is_test_passed-no_tests_run_with_header.log') self.assertEqual(0, len(result['sub_groups'])) + def test_nested_json(self): + result = self._json_for('test_is_test_passed-all_passed_nested.log') + self.assertEqual( + {'name': 'example_simple_test', 'status': 'PASS'}, + result["sub_groups"][0]["sub_groups"][0]["test_cases"][0]) + class StrContains(str): def __eq__(self, other): return self in other @@ -399,7 +454,7 @@ class KUnitMainTest(unittest.TestCase): self.assertEqual(e.exception.code, 1) self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 1) self.assertEqual(self.linux_source_mock.run_kernel.call_count, 1) - self.print_mock.assert_any_call(StrContains(' 0 tests run')) + self.print_mock.assert_any_call(StrContains('invalid KTAP input!')) def test_exec_raw_output(self): self.linux_source_mock.run_kernel = mock.Mock(return_value=[]) @@ -407,7 +462,7 @@ class KUnitMainTest(unittest.TestCase): self.assertEqual(self.linux_source_mock.run_kernel.call_count, 1) for call in self.print_mock.call_args_list: self.assertNotEqual(call, mock.call(StrContains('Testing complete.'))) - self.assertNotEqual(call, mock.call(StrContains(' 0 tests run'))) + self.assertNotEqual(call, mock.call(StrContains(' 0 tests run!'))) def test_run_raw_output(self): self.linux_source_mock.run_kernel = mock.Mock(return_value=[]) @@ -416,7 +471,7 @@ class KUnitMainTest(unittest.TestCase): self.assertEqual(self.linux_source_mock.run_kernel.call_count, 1) for call in self.print_mock.call_args_list: self.assertNotEqual(call, mock.call(StrContains('Testing complete.'))) - self.assertNotEqual(call, mock.call(StrContains(' 0 tests run'))) + self.assertNotEqual(call, mock.call(StrContains(' 0 tests run!'))) def test_run_raw_output_kunit(self): self.linux_source_mock.run_kernel = mock.Mock(return_value=[]) diff --git a/tools/testing/kunit/test_data/test_is_test_passed-all_passed_nested.log b/tools/testing/kunit/test_data/test_is_test_passed-all_passed_nested.log new file mode 100644 index 000000000000..9d5b04fe43a6 --- /dev/null +++ b/tools/testing/kunit/test_data/test_is_test_passed-all_passed_nested.log @@ -0,0 +1,34 @@ +TAP version 14 +1..2 + # Subtest: sysctl_test + 1..4 + # sysctl_test_dointvec_null_tbl_data: sysctl_test_dointvec_null_tbl_data passed + ok 1 - sysctl_test_dointvec_null_tbl_data + # Subtest: example + 1..2 + init_suite + # example_simple_test: initializing + # example_simple_test: example_simple_test passed + ok 1 - example_simple_test + # example_mock_test: initializing + # example_mock_test: example_mock_test passed + ok 2 - example_mock_test + kunit example: all tests passed + ok 2 - example + # sysctl_test_dointvec_table_len_is_zero: sysctl_test_dointvec_table_len_is_zero passed + ok 3 - sysctl_test_dointvec_table_len_is_zero + # sysctl_test_dointvec_table_read_but_position_set: sysctl_test_dointvec_table_read_but_position_set passed + ok 4 - sysctl_test_dointvec_table_read_but_position_set +kunit sysctl_test: all tests passed +ok 1 - sysctl_test + # Subtest: example + 1..2 +init_suite + # example_simple_test: initializing + # example_simple_test: example_simple_test passed + ok 1 - example_simple_test + # example_mock_test: initializing + # example_mock_test: example_mock_test passed + ok 2 - example_mock_test +kunit example: all tests passed +ok 2 - example diff --git a/tools/testing/kunit/test_data/test_is_test_passed-kselftest.log b/tools/testing/kunit/test_data/test_is_test_passed-kselftest.log new file mode 100644 index 000000000000..65d3f27feaf2 --- /dev/null +++ b/tools/testing/kunit/test_data/test_is_test_passed-kselftest.log @@ -0,0 +1,14 @@ +TAP version 13 +1..2 +# selftests: membarrier: membarrier_test_single_thread +# TAP version 13 +# 1..2 +# ok 1 sys_membarrier available +# ok 2 sys membarrier invalid command test: command = -1, flags = 0, errno = 22. Failed as expected +ok 1 selftests: membarrier: membarrier_test_single_thread +# selftests: membarrier: membarrier_test_multi_thread +# TAP version 13 +# 1..2 +# ok 1 sys_membarrier available +# ok 2 sys membarrier invalid command test: command = -1, flags = 0, errno = 22. Failed as expected +ok 2 selftests: membarrier: membarrier_test_multi_thread diff --git a/tools/testing/kunit/test_data/test_is_test_passed-missing_plan.log b/tools/testing/kunit/test_data/test_is_test_passed-missing_plan.log new file mode 100644 index 000000000000..5cd17b7f818a --- /dev/null +++ b/tools/testing/kunit/test_data/test_is_test_passed-missing_plan.log @@ -0,0 +1,31 @@ +KTAP version 1 + # Subtest: sysctl_test + # sysctl_test_dointvec_null_tbl_data: sysctl_test_dointvec_null_tbl_data passed + ok 1 - sysctl_test_dointvec_null_tbl_data + # sysctl_test_dointvec_table_maxlen_unset: sysctl_test_dointvec_table_maxlen_unset passed + ok 2 - sysctl_test_dointvec_table_maxlen_unset + # sysctl_test_dointvec_table_len_is_zero: sysctl_test_dointvec_table_len_is_zero passed + ok 3 - sysctl_test_dointvec_table_len_is_zero + # sysctl_test_dointvec_table_read_but_position_set: sysctl_test_dointvec_table_read_but_position_set passed + ok 4 - sysctl_test_dointvec_table_read_but_position_set + # sysctl_test_dointvec_happy_single_positive: sysctl_test_dointvec_happy_single_positive passed + ok 5 - sysctl_test_dointvec_happy_single_positive + # sysctl_test_dointvec_happy_single_negative: sysctl_test_dointvec_happy_single_negative passed + ok 6 - sysctl_test_dointvec_happy_single_negative + # sysctl_test_dointvec_single_less_int_min: sysctl_test_dointvec_single_less_int_min passed + ok 7 - sysctl_test_dointvec_single_less_int_min + # sysctl_test_dointvec_single_greater_int_max: sysctl_test_dointvec_single_greater_int_max passed + ok 8 - sysctl_test_dointvec_single_greater_int_max +kunit sysctl_test: all tests passed +ok 1 - sysctl_test + # Subtest: example + 1..2 +init_suite + # example_simple_test: initializing + # example_simple_test: example_simple_test passed + ok 1 - example_simple_test + # example_mock_test: initializing + # example_mock_test: example_mock_test passed + ok 2 - example_mock_test +kunit example: all tests passed +ok 2 - example diff --git a/tools/testing/kunit/test_data/test_strip_hyphen.log b/tools/testing/kunit/test_data/test_strip_hyphen.log new file mode 100644 index 000000000000..92ac7c24b374 --- /dev/null +++ b/tools/testing/kunit/test_data/test_strip_hyphen.log @@ -0,0 +1,16 @@ +KTAP version 1 +1..2 + # Subtest: sysctl_test + 1..1 + # sysctl_test_dointvec_null_tbl_data: sysctl_test_dointvec_null_tbl_data passed + ok 1 - sysctl_test_dointvec_null_tbl_data +kunit sysctl_test: all tests passed +ok 1 - sysctl_test + # Subtest: example + 1..1 +init_suite + # example_simple_test: initializing + # example_simple_test: example_simple_test passed + ok 1 example_simple_test +kunit example: all tests passed +ok 2 example base-commit: e7198adb84dcad671ad4f0e90aaa7e9fabf258dc -- 2.33.0.882.g93a45727a2-goog

3 years, 7 months

2
4
0 0

Re: [PATCH v8 03/12] selftests: add tests_sysfs module

by Luis Chamberlain

On Tue, Oct 05, 2021 at 09:30:10AM -0700, Kees Cook wrote: > On Mon, Sep 27, 2021 at 09:37:56AM -0700, Luis Chamberlain wrote: > > --- /dev/null > > +++ b/lib/test_sysfs.c > > @@ -0,0 +1,921 @@ > > +// SPDX-License-Identifier: GPL-2.0-or-later OR copyleft-next-0.3.1 > > +/* > > + * sysfs test driver > > + * > > + * Copyright (C) 2021 Luis Chamberlain <mcgrof(a)kernel.org> > > + * > > + * This program is free software; you can redistribute it and/or modify it > > + * under the terms of the GNU General Public License as published by the Free > > + * Software Foundation; either version 2 of the License, or at your option any > > + * later version; or, when distributed separately from the Linux kernel or > > + * when incorporated into other software packages, subject to the following > > + * license: > > + * > > + * This program is free software; you can redistribute it and/or modify it > > + * under the terms of copyleft-next (version 0.3.1 or later) as published > > + * at http://copyleft-next.org/. > > As Greg suggested, please drop the boilerplate here. Sure, sorry for missing that fixed. > > +static ssize_t config_show(struct device *dev, > > + struct device_attribute *attr, > > + char *buf) > > +{ > > + struct sysfs_test_device *test_dev = dev_to_test_dev(dev); > > + struct test_config *config = &test_dev->config; > > + int len = 0; > > + > > + test_dev_config_lock(test_dev); > > + > > + len += snprintf(buf, PAGE_SIZE, > > + "Configuration for: %s\n", > > + dev_name(dev)); > > Please use sysfs_emit() instead of snprintf(). Oh nice, done and fixed also in the other places. > > +static int sysfs_test_dev_alloc_blockdev(struct sysfs_test_device *test_dev) > > +{ > > + int ret = -ENOMEM; > > + > > + test_dev->disk = blk_alloc_disk(NUMA_NO_NODE); > > + if (!test_dev->disk) { > > + pr_err("Error allocating disk structure for device %d\n", > > + test_dev->dev_idx); > > + goto out; > > + } > > + > > + test_dev->disk->major = sysfs_test_major; > > + test_dev->disk->first_minor = test_dev->dev_idx + 1; > > + test_dev->disk->fops = &sysfs_testdev_ops; > > + test_dev->disk->private_data = test_dev; > > + snprintf(test_dev->disk->disk_name, 16, "test_sysfs%d", > > + test_dev->dev_idx); > > Prefer sizeof(test_dev->disk->disk_name) over open-coded "16". Sure. > > +static ssize_t read_reset_first_test_dev(struct file *file, > > + char __user *user_buf, > > + size_t count, loff_t *ppos) > > +{ > > + ssize_t len; > > + char buf[32]; > > + > > + reset_first_test_dev++; > > + len = sprintf(buf, "%d\n", reset_first_test_dev); > > Even though it's safe as-is, I was going to suggest scnprintf() here > (i.e. explicit bounds and a bounds-checked "len"). However, scnprintf() > returns ssize_t, and there's no bounds checking in > simple_read_from_buffer. That needs fixing (I'll send a patch). OK we can later change it to scnprintf() once your patch gets merged. > > --- /dev/null > > +++ b/tools/testing/selftests/sysfs/sysfs.sh > > @@ -0,0 +1,1208 @@ > > +#!/bin/bash > > +# SPDX-License-Identifier: GPL-2.0-or-later > > +# Copyright (C) 2021 Luis Chamberlain <mcgrof(a)kernel.org> > > +# > > +# This program is free software; you can redistribute it and/or modify it > > +# under the terms of the GNU General Public License as published by the Free > > +# Software Foundation; either version 2 of the License, or at your option any > > +# later version; or, when distributed separately from the Linux kernel or > > +# when incorporated into other software packages, subject to the following > > +# license: > > +# > > +# This program is free software; you can redistribute it and/or modify it > > +# under the terms of copyleft-next (version 0.3.1 or later) as published > > +# at http://copyleft-next.org/. > > + > > +# This performs a series tests against the sysfs filesystem. > > -boilerplate Nuked. > > +check_dmesg() > > +{ > > + # filter out intentional WARNINGs or Oopses > > + local filter=${1:-_check_dmesg_filter} > > + > > + _dmesg_since_test_start | $filter >$seqres.dmesg > > + egrep -q -e "kernel BUG at" \ > > + -e "WARNING:" \ > > + -e "\bBUG:" \ > > + -e "Oops:" \ > > + -e "possible recursive locking detected" \ > > + -e "Internal error" \ > > + -e "(INFO|ERR): suspicious RCU usage" \ > > + -e "INFO: possible circular locking dependency detected" \ > > + -e "general protection fault:" \ > > + -e "BUG .* remaining" \ > > + -e "UBSAN:" \ > > + $seqres.dmesg > > Is just looking for "call trace" sufficient here? So far from my testing yes. This strategy is also borrowed from fstests and that's what is done there, and so quite a lot of testing has been done with that. If we are to consider an enhancement here we should then also consider an enhancement welcome for fstests. Luis

3 years, 7 months

1
0
0 0

Re: [PATCH v8 01/12] LICENSES: Add the copyleft-next-0.3.1 license

by Kees Cook

On Mon, Oct 11, 2021 at 10:37:42AM -0700, Luis Chamberlain wrote: > On Tue, Oct 05, 2021 at 09:08:59AM -0700, Kees Cook wrote: > > On Mon, Sep 27, 2021 at 09:37:54AM -0700, Luis Chamberlain wrote: > > I can confirm that LICENSES/dual/copyleft-next-0.3.1 matches > > https://github.com/copyleft-next/copyleft-next/blob/master/Releases/copylef… > > > > Reviewed-by: Kees Cook <keescook(a)chromium.org> > > > > > + If the Derived Work includes material licensed under the GPL, You may > > > + instead license the Derived Work under the GPL. > > > + > > > > nit: needless whitespace, though technically the original license > > includes this too. :) > > Indeed, I decided to leave the white space as the original had it too. > Should I really get rid of the space or keep it? Probably keep it for 0 diff with original. :) -- Kees Cook

3 years, 7 months

1
0
0 0

[PATCH] selftests: bpf: Remove dumplicated include in cgroup_helpers

by Wan Jiabing

Fix following checkincludes.pl warning: ./tools/testing/selftests/bpf/cgroup_helpers.c 12 #include <unistd.h> 14 #include <unistd.h> Signed-off-by: Wan Jiabing <wanjiabing(a)vivo.com> --- tools/testing/selftests/bpf/cgroup_helpers.c | 1 - 1 file changed, 1 deletion(-) diff --git a/tools/testing/selftests/bpf/cgroup_helpers.c b/tools/testing/selftests/bpf/cgroup_helpers.c index 8fcd44841bb2..9d59c3990ca8 100644 --- a/tools/testing/selftests/bpf/cgroup_helpers.c +++ b/tools/testing/selftests/bpf/cgroup_helpers.c @@ -11,7 +11,6 @@ #include <fcntl.h> #include <unistd.h> #include <ftw.h> -#include <unistd.h> #include "cgroup_helpers.h" -- 2.30.2

3 years, 7 months

2
1
0 0

[PATCH v4 0/7] kernel.h further split

by Andy Shevchenko

v3: https://lore.kernel.org/linux-media/20211007150339.28910-1-andriy.shevchenk… v2: https://lore.kernel.org/linux-media/20211007095129.22037-1-andriy.shevchenk… The kernel.h is a set of something which is not related to each other and often used in non-crossed compilation units, especially when drivers need only one or two macro definitions from it. Here is the split of container_of(). The goals are the following: - untwist the dependency hell a bit - drop kernel.h inclusion where it's only used for container_of() - speed up C preprocessing. The build speedup is 1.83% (ccache approach, see v2 cover letter for the details) 0.5% (kcbench approach, see v3 cover letter for the details) In v4: - dropped kobject.h change (Greg) - Cc'ed more people (as per v1) In v3: - split patch 2 to more patches (Greg) - excluded C changes (Herbert, Greg) - measured with kcbench, see below (Greg) Andy Shevchenko (7): kernel.h: Drop unneeded <linux/kernel.h> inclusion from other headers kernel.h: Split out container_of() and typeof_member() macros kunit: Replace kernel.h with the necessary inclusions list.h: Replace kernel.h with the necessary inclusions llist: Replace kernel.h with the necessary inclusions plist: Replace kernel.h with the necessary inclusions media: entity: Replace kernel.h with the necessary inclusions include/kunit/test.h | 14 ++++++++++++-- include/linux/container_of.h | 37 ++++++++++++++++++++++++++++++++++++ include/linux/kernel.h | 31 +----------------------------- include/linux/list.h | 6 ++++-- include/linux/llist.h | 4 +++- include/linux/plist.h | 5 ++++- include/linux/rwsem.h | 1 - include/linux/spinlock.h | 1 - include/media/media-entity.h | 3 ++- 9 files changed, 63 insertions(+), 39 deletions(-) create mode 100644 include/linux/container_of.h -- 2.33.0

3 years, 7 months

7
24
0 0

[PATCH] selftests: vm: add KSM huge pages merging time test

by Pedro Demarchi Gomes

Signed-off-by: Pedro Demarchi Gomes <pedrodemargomes(a)gmail.com> --- tools/testing/selftests/vm/ksm_tests.c | 40 +++++++++++++++++++------- 1 file changed, 30 insertions(+), 10 deletions(-) diff --git a/tools/testing/selftests/vm/ksm_tests.c b/tools/testing/selftests/vm/ksm_tests.c index b61dcdb44c5b..92b716565d9c 100644 --- a/tools/testing/selftests/vm/ksm_tests.c +++ b/tools/testing/selftests/vm/ksm_tests.c @@ -5,6 +5,7 @@ #include <time.h> #include <string.h> #include <numa.h> +#include <err.h> #include "../kselftest.h" #include "../../../../include/vdso/time64.h" @@ -34,7 +35,8 @@ enum ksm_test_name { CHECK_KSM_ZERO_PAGE_MERGE, CHECK_KSM_NUMA_MERGE, KSM_MERGE_TIME, - KSM_COW_TIME + KSM_COW_TIME, + KSM_MERGE_TIME_HUGE_PAGES }; static int ksm_write_sysfs(const char *file_path, unsigned long val) @@ -99,6 +101,9 @@ static void print_help(void) " -U (page unmerging)\n" " -P evaluate merging time and speed.\n" " For this test, the size of duplicated memory area (in MiB)\n" + " must be provided using -s option\n" + " -H evaluate merging time and speed of huge pages.\n" + " For this test, the size of duplicated memory area (in MiB)\n" " must be provided using -s option\n" " -C evaluate the time required to break COW of merged pages.\n\n"); @@ -118,10 +123,14 @@ static void print_help(void) exit(0); } -static void *allocate_memory(void *ptr, int prot, int mapping, char data, size_t map_size) +static void *allocate_memory(void *ptr, int prot, int mapping, char data, size_t map_size, + bool huge_page) { void *map_ptr = mmap(ptr, map_size, PROT_WRITE, mapping, -1, 0); + if (huge_page && madvise(map_ptr, map_size, MADV_HUGEPAGE)) + err(2, "MADV_HUGEPAGE"); + if (!map_ptr) { perror("mmap"); return NULL; @@ -250,7 +259,7 @@ static int check_ksm_merge(int mapping, int prot, long page_count, int timeout, } /* fill pages with the same data and merge them */ - map_ptr = allocate_memory(NULL, prot, mapping, '*', page_size * page_count); + map_ptr = allocate_memory(NULL, prot, mapping, '*', page_size * page_count, false); if (!map_ptr) return KSFT_FAIL; @@ -282,7 +291,7 @@ static int check_ksm_unmerge(int mapping, int prot, int timeout, size_t page_siz } /* fill pages with the same data and merge them */ - map_ptr = allocate_memory(NULL, prot, mapping, '*', page_size * page_count); + map_ptr = allocate_memory(NULL, prot, mapping, '*', page_size * page_count, false); if (!map_ptr) return KSFT_FAIL; @@ -325,7 +334,7 @@ static int check_ksm_zero_page_merge(int mapping, int prot, long page_count, int return KSFT_FAIL; /* fill pages with zero and try to merge them */ - map_ptr = allocate_memory(NULL, prot, mapping, 0, page_size * page_count); + map_ptr = allocate_memory(NULL, prot, mapping, 0, page_size * page_count, false); if (!map_ptr) return KSFT_FAIL; @@ -416,7 +425,7 @@ static int check_ksm_numa_merge(int mapping, int prot, int timeout, bool merge_a return KSFT_FAIL; } -static int ksm_merge_time(int mapping, int prot, int timeout, size_t map_size) +static int ksm_merge_time(int mapping, int prot, int timeout, size_t map_size, bool huge_page) { void *map_ptr; struct timespec start_time, end_time; @@ -424,7 +433,7 @@ static int ksm_merge_time(int mapping, int prot, int timeout, size_t map_size) map_size *= MB; - map_ptr = allocate_memory(NULL, prot, mapping, '*', map_size); + map_ptr = allocate_memory(NULL, prot, mapping, '*', map_size, huge_page); if (!map_ptr) return KSFT_FAIL; @@ -466,7 +475,7 @@ static int ksm_cow_time(int mapping, int prot, int timeout, size_t page_size) /* page_count must be less than 2*page_size */ size_t page_count = 4000; - map_ptr = allocate_memory(NULL, prot, mapping, '*', page_size * page_count); + map_ptr = allocate_memory(NULL, prot, mapping, '*', page_size * page_count, false); if (!map_ptr) return KSFT_FAIL; @@ -541,7 +550,7 @@ int main(int argc, char *argv[]) bool merge_across_nodes = KSM_MERGE_ACROSS_NODES_DEFAULT; long size_MB = 0; - while ((opt = getopt(argc, argv, "ha:p:l:z:m:s:MUZNPC")) != -1) { + while ((opt = getopt(argc, argv, "ha:p:l:z:m:s:MUZNPCH")) != -1) { switch (opt) { case 'a': prot = str_to_prot(optarg); @@ -598,6 +607,9 @@ int main(int argc, char *argv[]) case 'C': test_name = KSM_COW_TIME; break; + case 'H': + test_name = KSM_MERGE_TIME_HUGE_PAGES; + break; default: return KSFT_FAIL; } @@ -645,12 +657,20 @@ int main(int argc, char *argv[]) return KSFT_FAIL; } ret = ksm_merge_time(MAP_PRIVATE | MAP_ANONYMOUS, prot, ksm_scan_limit_sec, - size_MB); + size_MB, false); break; case KSM_COW_TIME: ret = ksm_cow_time(MAP_PRIVATE | MAP_ANONYMOUS, prot, ksm_scan_limit_sec, page_size); break; + case KSM_MERGE_TIME_HUGE_PAGES: + if (size_MB == 0) { + printf("Option '-s' is required.\n"); + return KSFT_FAIL; + } + ret = ksm_merge_time(MAP_PRIVATE | MAP_ANONYMOUS, prot, ksm_scan_limit_sec, + size_MB, true); + break; } if (ksm_restore(&ksm_sysfs_old)) { -- 2.25.1

3 years, 7 months

1
0
0 0

[REGRESSION] lkft kselftest for next-20211008

by lkft＠linaro.org

## Build * kernel: 5.15.0-rc4 * git: https://gitlab.com/Linaro/lkft/mirrors/next/linux-next * git branch: master * git commit: 683f29b781aeaab6bf302eeb2ef08a5e5f9d8a27 * git describe: next-20211008 * test details: https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20211008 ## Regressions (compared to next-20211006) * x86, kselftest-kvm - kvm.hyperv_clock Reported-by: Linux Kernel Functional Testing <lkft(a)linaro.org> ## Fixes (compared to next-20211006) * x86, kselftest-lkdtm - lkdtm.SLAB_FREE_DOUBLE.sh ## Test result summary total: 734, pass: 504, fail: 45, skip: 185, xfail: 0 ## Build Summary ## Test suites summary * kselftest-android * kselftest-bpf * kselftest-capabilities * kselftest-cgroup * kselftest-clone3 * kselftest-core * kselftest-cpu-hotplug * kselftest-cpufreq * kselftest-efivarfs * kselftest-filesystems * kselftest-firmware * kselftest-fpu * kselftest-futex * kselftest-gpio * kselftest-intel_pstate * kselftest-ipc * kselftest-ir * kselftest-kcmp * kselftest-kexec * kselftest-kvm * kselftest-lib * kselftest-livepatch * kselftest-lkdtm * kselftest-membarrier * kselftest-net * kselftest-netfilter * kselftest-nsfs * kselftest-ptrace * kselftest-rseq * kselftest-rtc * kselftest-seccomp * kselftest-sigaltstack * kselftest-size * kselftest-splice * kselftest-static_keys * kselftest-sync * kselftest-sysctl * kselftest-tc-testing * kselftest-timens * kselftest-timers * kselftest-tmpfs * kselftest-tpm2 * kselftest-user * kselftest-vm * kselftest-x86 -- Linaro LKFT https://lkft.linaro.org

3 years, 7 months

1
0
0 0

[PATCH v1 1/3] kernel.h: Don't pollute header with single user macros

by Andy Shevchenko

The COUNT_ARGS() and CONCATENATE() macros are used by a single user. Let move them to it. Signed-off-by: Andy Shevchenko <andriy.shevchenko(a)linux.intel.com> --- include/linux/kernel.h | 7 ------- include/trace/bpf_probe.h | 7 +++++++ 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/include/linux/kernel.h b/include/linux/kernel.h index 1b2f0a7e00d6..743d3c9a3227 100644 --- a/include/linux/kernel.h +++ b/include/linux/kernel.h @@ -476,13 +476,6 @@ ftrace_vprintk(const char *fmt, va_list ap) static inline void ftrace_dump(enum ftrace_dump_mode oops_dump_mode) { } #endif /* CONFIG_TRACING */ -/* This counts to 12. Any more, it will return 13th argument. */ -#define __COUNT_ARGS(_0, _1, _2, _3, _4, _5, _6, _7, _8, _9, _10, _11, _12, _n, X...) _n -#define COUNT_ARGS(X...) __COUNT_ARGS(, ##X, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0) - -#define __CONCAT(a, b) a ## b -#define CONCATENATE(a, b) __CONCAT(a, b) - /** * container_of - cast a member of a structure out to the containing structure * @ptr: the pointer to the member. diff --git a/include/trace/bpf_probe.h b/include/trace/bpf_probe.h index a23be89119aa..6f57c96f7dc3 100644 --- a/include/trace/bpf_probe.h +++ b/include/trace/bpf_probe.h @@ -27,6 +27,13 @@ #undef __perf_task #define __perf_task(t) (t) +/* This counts to 12. Any more, it will return 13th argument. */ +#define __COUNT_ARGS(_0, _1, _2, _3, _4, _5, _6, _7, _8, _9, _10, _11, _12, _n, X...) _n +#define COUNT_ARGS(X...) __COUNT_ARGS(, ##X, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0) + +#define __CONCAT(a, b) a ## b +#define CONCATENATE(a, b) __CONCAT(a, b) + /* cast any integer, pointer, or small struct to u64 */ #define UINTTYPE(size) \ __typeof__(__builtin_choose_expr(size == 1, (u8)1, \ -- 2.30.2

3 years, 7 months

7
15
0 0

[PATCH v3 0/7] kernel.h further split

by Andy Shevchenko

v2: https://lore.kernel.org/linux-media/20211007095129.22037-1-andriy.shevchenk… The kernel.h is a set of something which is not related to each other and often used in non-crossed compilation units, especially when drivers need only one or two macro definitions from it. Here is the split of container_of(). The goals are the following: - untwist the dependency hell a bit - drop kernel.h inclusion where it's only used for container_of() - speed up C preprocessing. In v3: - split patch 2 to more patches (Greg) - exclude C changes (Herbert, Greg) - measure with kcbench, see below (Greg) Cc: Thorsten Leemhuis <regressions(a)leemhuis.info> People, like Greg KH and Miguel Ojeda, were asking about the latter. My methodology an testing has been provided in cover letter for v2 (see above) and here is what Greg KH insisted to have which is speedup of the kernel build. $ kcbench -i 3 -j 64 -o $O -s $PWD --no-download -m Processor: Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz [88 CPUs] Cpufreq; Memory: powersave [intel_pstate]; 128823 MiB Linux running: 5.6.0-2-amd64 [x86_64] Compiler: gcc (Debian 10.3.0-11) 10.3.0 Linux compiled: 5.15.0-rc4 Config; Environment: allmodconfig; CCACHE_DISABLE="1" Build command: make vmlinux modules Filling caches: This might take a while... Done Run 1 (-j 64): 464.07 seconds / 7.76 kernels/hour [P:6001%] Run 2 (-j 64): 464.64 seconds / 7.75 kernels/hour [P:6000%] Run 3 (-j 64): 486.41 seconds / 7.40 kernels/hour [P:5727%] $ kcbench -i 3 -j 64 -o $O -s $PWD --no-download -m Processor: Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz [88 CPUs] Cpufreq; Memory: powersave [intel_pstate]; 128823 MiB Linux running: 5.6.0-2-amd64 [x86_64] Compiler: gcc (Debian 10.3.0-11) 10.3.0 Linux compiled: 5.15.0-rc4 Config; Environment: allmodconfig; CCACHE_DISABLE="1" Build command: make vmlinux modules Filling caches: This might take a while... Done Run 1 (-j 64): 462.32 seconds / 7.79 kernels/hour [P:6009%] Run 2 (-j 64): 462.33 seconds / 7.79 kernels/hour [P:6006%] Run 3 (-j 64): 465.45 seconds / 7.73 kernels/hour [P:5999%] Median values 464.64 before 462.33 after Speedup: +0.5% This supports and in align with my own approach, but shows lower numbers due to additional big take in the measurements (compilation without ccache). Andy Shevchenko (7): kernel.h: Drop unneeded <linux/kernel.h> inclusion from other headers kernel.h: Split out container_of() and typeof_member() macros kunit: Replace kernel.h with the necessary inclusions list.h: Replace kernel.h with the necessary inclusions llist: Replace kernel.h with the necessary inclusions plist: Replace kernel.h with the necessary inclusions media: entity: Replace kernel.h with the necessary inclusions include/kunit/test.h | 14 ++++++++++++-- include/linux/container_of.h | 37 ++++++++++++++++++++++++++++++++++++ include/linux/kernel.h | 31 +----------------------------- include/linux/kobject.h | 1 + include/linux/list.h | 6 ++++-- include/linux/llist.h | 4 +++- include/linux/plist.h | 5 ++++- include/linux/rwsem.h | 1 - include/linux/spinlock.h | 1 - include/media/media-entity.h | 3 ++- 10 files changed, 64 insertions(+), 39 deletions(-) create mode 100644 include/linux/container_of.h -- 2.33.0

3 years, 7 months

2
9
0 0

[PATCH] selftests: arm64: Add coverage of ptrace flags for SVE VL inheritance

by Mark Brown

Add a test that covers enabling and disabling of SVE vector length inheritance via the ptrace interface. Signed-off-by: Mark Brown <broonie(a)kernel.org> --- tools/testing/selftests/arm64/fp/sve-ptrace.c | 55 ++++++++++++++++++- 1 file changed, 54 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/arm64/fp/sve-ptrace.c b/tools/testing/selftests/arm64/fp/sve-ptrace.c index ac0629f05365..c4417bc48d4f 100644 --- a/tools/testing/selftests/arm64/fp/sve-ptrace.c +++ b/tools/testing/selftests/arm64/fp/sve-ptrace.c @@ -22,7 +22,7 @@ #include "../../kselftest.h" #define VL_TESTS (((SVE_VQ_MAX - SVE_VQ_MIN) + 1) * 3) -#define FPSIMD_TESTS 3 +#define FPSIMD_TESTS 5 #define EXPECTED_TESTS (VL_TESTS + FPSIMD_TESTS) @@ -105,6 +105,56 @@ static int set_sve(pid_t pid, const struct user_sve_header *sve) return ptrace(PTRACE_SETREGSET, pid, NT_ARM_SVE, &iov); } +/* Validate setting and getting the inherit flag */ +static void ptrace_set_get_inherit(pid_t child) +{ + struct user_sve_header sve; + struct user_sve_header *new_sve = NULL; + size_t new_sve_size = 0; + int ret; + + /* First set the flag */ + memset(&sve, 0, sizeof(sve)); + sve.size = sizeof(sve); + sve.vl = sve_vl_from_vq(SVE_VQ_MIN); + sve.flags = SVE_PT_VL_INHERIT; + ret = set_sve(child, &sve); + if (ret != 0) { + ksft_test_result_fail("Failed to set SVE_PT_VL_INHERIT\n"); + return; + } + + /* + * Read back the new register state and verify that we have + * set the flags we expected. + */ + if (!get_sve(child, (void **)&new_sve, &new_sve_size)) { + ksft_test_result_fail("Failed to read SVE flags\n"); + return; + } + + ksft_test_result(new_sve->flags & SVE_PT_VL_INHERIT, + "SVE_PT_VL_INHERIT set\n"); + + /* Now clear */ + sve.flags &= ~SVE_PT_VL_INHERIT; + ret = set_sve(child, &sve); + if (ret != 0) { + ksft_test_result_fail("Failed to clear SVE_PT_VL_INHERIT\n"); + return; + } + + if (!get_sve(child, (void **)&new_sve, &new_sve_size)) { + ksft_test_result_fail("Failed to read SVE flags\n"); + return; + } + + ksft_test_result(!(new_sve->flags & SVE_PT_VL_INHERIT), + "SVE_PT_VL_INHERIT cleared\n"); + + free(new_sve); +} + /* Validate attempting to set the specfied VL via ptrace */ static void ptrace_set_get_vl(pid_t child, unsigned int vl, bool *supported) { @@ -452,6 +502,9 @@ static int do_parent(pid_t child) /* FPSIMD via SVE regset */ ptrace_sve_fpsimd(child); + /* prctl() flags */ + ptrace_set_get_inherit(child); + /* Step through every possible VQ */ for (vq = SVE_VQ_MIN; vq <= SVE_VQ_MAX; vq++) { vl = sve_vl_from_vq(vq); -- 2.30.2

3 years, 7 months

3
2
0 0

[PATCH v1 0/6] mm/memory_hotplug: Kconfig and 32 bit cleanups

by David Hildenbrand

Some cleanups around CONFIG_MEMORY_HOTPLUG, including removing 32 bit leftovers of memory hotplug support. Compile-tested on various architectures, quickly tested memory hotplug on x86-64. Cc: Andrew Morton <akpm(a)linux-foundation.org> Cc: Jonathan Corbet <corbet(a)lwn.net> Cc: Alex Shi <alexs(a)kernel.org> Cc: Michael Ellerman <mpe(a)ellerman.id.au> Cc: Benjamin Herrenschmidt <benh(a)kernel.crashing.org> Cc: Paul Mackerras <paulus(a)samba.org> Cc: Thomas Gleixner <tglx(a)linutronix.de> Cc: Ingo Molnar <mingo(a)redhat.com> Cc: Borislav Petkov <bp(a)alien8.de> Cc: "H. Peter Anvin" <hpa(a)zytor.com> Cc: Dave Hansen <dave.hansen(a)linux.intel.com> Cc: Andy Lutomirski <luto(a)kernel.org> Cc: Peter Zijlstra <peterz(a)infradead.org> Cc: Greg Kroah-Hartman <gregkh(a)linuxfoundation.org> Cc: "Rafael J. Wysocki" <rafael(a)kernel.org> Cc: "Michael S. Tsirkin" <mst(a)redhat.com> Cc: Jason Wang <jasowang(a)redhat.com> Cc: Shuah Khan <shuah(a)kernel.org> Cc: Michal Hocko <mhocko(a)suse.com> Cc: Oscar Salvador <osalvador(a)suse.de> Cc: Mike Rapoport <rppt(a)kernel.org> Cc: x86(a)kernel.org Cc: linux-mm(a)kvack.org Cc: linux-kselftest(a)vger.kernel.org Cc: linux-doc(a)vger.kernel.org Cc: linuxppc-dev(a)lists.ozlabs.org Cc: virtualization(a)lists.linux-foundation.org David Hildenbrand (6): mm/memory_hotplug: remove CONFIG_X86_64_ACPI_NUMA dependency from CONFIG_MEMORY_HOTPLUG mm/memory_hotplug: remove CONFIG_MEMORY_HOTPLUG_SPARSE mm/memory_hotplug: restrict CONFIG_MEMORY_HOTPLUG to 64 bit mm/memory_hotplug: remove HIGHMEM leftovers mm/memory_hotplug: remove stale function declarations x86: remove memory hotplug support on X86_32 Documentation/core-api/memory-hotplug.rst | 3 -- .../zh_CN/core-api/memory-hotplug.rst | 4 -- arch/powerpc/include/asm/machdep.h | 2 +- arch/powerpc/kernel/setup_64.c | 2 +- arch/powerpc/platforms/powernv/setup.c | 4 +- arch/powerpc/platforms/pseries/setup.c | 2 +- arch/x86/Kconfig | 6 +-- arch/x86/mm/init_32.c | 31 --------------- drivers/base/Makefile | 2 +- drivers/base/node.c | 9 ++--- drivers/virtio/Kconfig | 2 +- include/linux/memory.h | 19 ++++------ include/linux/memory_hotplug.h | 3 -- include/linux/node.h | 4 +- lib/Kconfig.debug | 2 +- mm/Kconfig | 8 +--- mm/memory_hotplug.c | 38 +------------------ tools/testing/selftests/memory-hotplug/config | 1 - 18 files changed, 28 insertions(+), 114 deletions(-) base-commit: 5816b3e6577eaa676ceb00a848f0fd65fe2adc29 -- 2.31.1

3 years, 7 months

4
15
0 0

[PATCH] kunit: tool: print parsed test results fully incrementally

by Daniel Latypov

With the parser rework [1] and run_kernel() rework [2], this allows the parser to print out test results incrementally. Currently, that's held up by the fact that the LineStream eagerly pre-fetches the next line when you call pop(). This blocks parse_test_result() from returning until the line *after* the "ok 1 - test name" line is also printed. One can see this with the following example: $ (echo -e 'TAP version 14\n1..3\nok 1 - fake test'; sleep 2; echo -e 'ok 2 - fake test 2'; sleep 3; echo -e 'ok 3 - fake test 3') | ./tools/testing/kunit/kunit.py parse Before this patch [1]: there's a pause before 'fake test' is printed. After this patch: 'fake test' is printed out immediately. This patch also adds * a unit test to verify LineStream's behavior directly * a test case to ensure that it's lazily calling the generator * an explicit exception for when users go beyond EOF [1] https://lore.kernel.org/linux-kselftest/20211006170049.106852-1-dlatypov@go… [2] https://lore.kernel.org/linux-kselftest/20211005011340.2826268-1-dlatypov@g… Signed-off-by: Daniel Latypov <dlatypov(a)google.com> --- tools/testing/kunit/kunit_parser.py | 22 ++++++++++---- tools/testing/kunit/kunit_tool_test.py | 42 +++++++++++++++++++++++++- 2 files changed, 57 insertions(+), 7 deletions(-) diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py index f01fd565f978..82900a5f9ad6 100644 --- a/tools/testing/kunit/kunit_parser.py +++ b/tools/testing/kunit/kunit_parser.py @@ -172,42 +172,51 @@ class TestCounts: class LineStream: """ A class to represent the lines of kernel output. - Provides a peek()/pop() interface over an iterator of + Provides a lazy peek()/pop() interface over an iterator of (line#, text). """ _lines: Iterator[Tuple[int, str]] _next: Tuple[int, str] + _need_next: bool _done: bool def __init__(self, lines: Iterator[Tuple[int, str]]): """Creates a new LineStream that wraps the given iterator.""" self._lines = lines self._done = False + self._need_next = True self._next = (0, '') - self._get_next() def _get_next(self) -> None: - """Advances the LineSteam to the next line.""" + """Advances the LineSteam to the next line, if necessary.""" + if not self._need_next: + return try: self._next = next(self._lines) except StopIteration: self._done = True + finally: + self._need_next = False def peek(self) -> str: """Returns the current line, without advancing the LineStream. """ + self._get_next() return self._next[1] def pop(self) -> str: """Returns the current line and advances the LineStream to the next line. """ - n = self._next - self._get_next() - return n[1] + s = self.peek() + if self._done: + raise ValueError(f'LineStream: going past EOF, last line was {s}') + self._need_next = True + return s def __bool__(self) -> bool: """Returns True if stream has more lines.""" + self._get_next() return not self._done # Only used by kunit_tool_test.py. @@ -220,6 +229,7 @@ class LineStream: def line_number(self) -> int: """Returns the line number of the current line.""" + self._get_next() return self._next[0] # Parsing helper methods: diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index c309ed76aef5..3cb02827c941 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -13,8 +13,9 @@ import tempfile, shutil # Handling test_tmpdir import itertools import json -import signal import os +import signal +from typing import Iterable import kunit_config import kunit_parser @@ -320,6 +321,45 @@ class KUnitParserTest(unittest.TestCase): result.status) self.assertEqual('kunit-resource-test', result.test.subtests[0].name) +def line_stream_from_strs(strs: Iterable[str]) -> kunit_parser.LineStream: + return kunit_parser.LineStream(enumerate(strs, start=1)) + +class LineStreamTest(unittest.TestCase): + + def test_basic(self): + stream = line_stream_from_strs(['hello', 'world']) + + self.assertTrue(stream, msg='Should be more input') + self.assertEqual(stream.line_number(), 1) + self.assertEqual(stream.peek(), 'hello') + self.assertEqual(stream.pop(), 'hello') + + self.assertTrue(stream, msg='Should be more input') + self.assertEqual(stream.line_number(), 2) + self.assertEqual(stream.peek(), 'world') + self.assertEqual(stream.pop(), 'world') + + self.assertFalse(stream, msg='Should be no more input') + with self.assertRaisesRegex(ValueError, 'LineStream: going past EOF'): + stream.pop() + + def test_is_lazy(self): + called_times = 0 + def generator(): + nonlocal called_times + for i in range(1,5): + called_times += 1 + yield called_times, str(called_times) + + stream = kunit_parser.LineStream(generator()) + self.assertEqual(called_times, 0) + + self.assertEqual(stream.pop(), '1') + self.assertEqual(called_times, 1) + + self.assertEqual(stream.pop(), '2') + self.assertEqual(called_times, 2) + class LinuxSourceTreeTest(unittest.TestCase): def setUp(self): base-commit: 9b409050eaf2da929408fa60fbf535745d828e67 -- 2.33.0.882.g93a45727a2-goog

3 years, 7 months

2
1
0 0

[PATCH v6] kunit: tool: improve compatibility of kunit_parser with KTAP specification

by Daniel Latypov

From: Rae Moar <rmoar(a)google.com> Update to kunit_parser to improve compatibility with KTAP specification including arbitrarily nested tests. Patch accomplishes three major changes: - Use a general Test object to represent all tests rather than TestCase and TestSuite objects. This allows for easier implementation of arbitrary levels of nested tests and promotes the idea that both test suites and test cases are tests. - Print errors incrementally rather than all at once after the parsing finishes to maximize information given to the user in the case of the parser given invalid input and to increase the helpfulness of the timestamps given during printing. Note that kunit.py parse does not print incrementally yet. However, this fix brings us closer to this feature. - Increase compatibility for different formats of input. Arbitrary levels of nested tests supported. Also, test cases and test suites are now supported to be present on the same level of testing. This patch now implements the draft KTAP specification here: https://lore.kernel.org/linux-kselftest/CA+GJov6tdjvY9x12JsJT14qn6c7NViJxqa… We'll update the parser as the spec evolves. This patch adjusts the kunit_tool_test.py file to check for the correct outputs from the new parser and adds a new test to check the parsing for a KTAP result log with correct format for multiple nested subtests (test_is_test_passed-all_passed_nested.log). This patch also alters the kunit_json.py file to allow for arbitrarily nested tests. Signed-off-by: Rae Moar <rmoar(a)google.com> Reviewed-by: Brendan Higgins <brendanhiggins(a)google.com> Signed-off-by: Daniel Latypov <dlatypov(a)google.com> Reviewed-by: David Gow <davidgow(a)google.com> --- NOTE: this patch is now applied on top of https://lore.kernel.org/linux-kselftest/20210930222048.1692635-5-dlatypov@g… to resolve a conflict. Change log from v5: https://lore.kernel.org/linux-kselftest/20211006001447.20919-1-dlatypov@goo… - Tweak commit message to reflect the KTAP spec is a draft - Add missing Signed-off-by - Tweak docstrings Change log from v3,4: https://lore.kernel.org/linux-kselftest/20210901190623.315736-1-rmoar@googl… - Move test_kselftest_nested from LinuxSourceTreeTest => KUnitParserTest. - Resolve conflict with hermetic testing patches. - max_status is no longer defined, so we need to use the TestCounts type now. And to keep --raw_output working, we need to set this to SUCCESS to avoid the default assumption that the kernel crashed. Ignore v4, was accidentally based on v2. Change log from v2: https://lore.kernel.org/linux-kselftest/20210826195505.3066755-1-rmoar@goog… - Fixes bug of type disagreement in kunit_json.py for build_dir - Removes raw_output() - Changes docstrings in kunit_parser.py (class docstring, LineStream docstrings, add_error(), total(), get_status(), all parsing methods) - Fixes bug of not printing diagnostic log in the case of end of lines - Sets default status of all tests to TEST_CRASHED - Adds and prints empty tests with crashed status in case of missing tests - Prints 'subtest' in instance of 1 subtest instead of 'subtests' - Includes checking for 'BUG:' message in search of crash messages in log (note that parse_crash_in_log method could be removed but would require deleting tests in kunit_tool_test.py that include the crash message that is no longer used. If removed, parser would still print log in cases of test crashed or failure, which would now include missing subtests) - Fixes bug of including directives (other than SKIP) in test name when matching name in result line for subtests Change log from v1: https://lore.kernel.org/linux-kselftest/20210820200032.2178134-1-rmoar@goog… - Rebase onto kselftest/kunit branch - Add tests to kunit_tool_test.py to check parser is correctly stripping hyphen, producing correct json objects with nested tests, correctly passing kselftest TAP output, and correctly deals with missing test plan. - Fix bug to correctly match test name in instance of a missing test plan. - Fix bug in kunit_tool_test.py pointed out by Daniel where it was not correctly checking for a proper match to the '0 tests run!' error message. Reverts changes back to original. - A few minor changes to commit message using Daniel's comments. - Change docstrings using Daniel's comments to reduce: - Shortens some docstrings to be one-line or just description if it is self explanatory. - Remove explicit respecification of types of parameters and returns because this is already specified in the function annoations. However, some descriptions of the parameters and returns remain and some contain the type for context. Additionally, the types of public attributes of classes remain. - Remove any documentation of 'Return: None' - Remove docstrings of helper methods within other methods --- tools/testing/kunit/kunit.py | 19 +- tools/testing/kunit/kunit_json.py | 56 +- tools/testing/kunit/kunit_parser.py | 1021 ++++++++++++----- tools/testing/kunit/kunit_tool_test.py | 133 ++- .../test_is_test_passed-all_passed_nested.log | 34 + .../test_is_test_passed-kselftest.log | 14 + .../test_is_test_passed-missing_plan.log | 31 + .../kunit/test_data/test_strip_hyphen.log | 16 + 8 files changed, 940 insertions(+), 384 deletions(-) create mode 100644 tools/testing/kunit/test_data/test_is_test_passed-all_passed_nested.log create mode 100644 tools/testing/kunit/test_data/test_is_test_passed-kselftest.log create mode 100644 tools/testing/kunit/test_data/test_is_test_passed-missing_plan.log create mode 100644 tools/testing/kunit/test_data/test_strip_hyphen.log diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py index 8c7e8c7b2c97..6aef16478efb 100755 --- a/tools/testing/kunit/kunit.py +++ b/tools/testing/kunit/kunit.py @@ -136,7 +136,7 @@ def exec_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest, test_glob = request.filter_glob.split('.', maxsplit=2)[1] filter_globs = [g + '.'+ test_glob for g in filter_globs] - overall_status = kunit_parser.TestStatus.SUCCESS + test_counts = kunit_parser.TestCounts() exec_time = 0.0 for i, filter_glob in enumerate(filter_globs): kunit_parser.print_with_timestamp('Starting KUnit Kernel ({}/{})...'.format(i+1, len(filter_globs))) @@ -155,18 +155,29 @@ def exec_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest, test_end = time.time() exec_time += test_end - test_start - overall_status = kunit_parser.max_status(overall_status, result.status) + test_counts.add_subtest_counts(result.result.test.counts) - return KunitResult(status=result.status, result=result.result, elapsed_time=exec_time) + kunit_status = _map_to_overall_status(test_counts.get_status()) + return KunitResult(status=kunit_status, result=result.result, elapsed_time=exec_time) + +def _map_to_overall_status(test_status: kunit_parser.TestStatus) -> KunitStatus: + if test_status in (kunit_parser.TestStatus.SUCCESS, kunit_parser.TestStatus.SKIPPED): + return KunitStatus.SUCCESS + else: + return KunitStatus.TEST_FAILURE def parse_tests(request: KunitParseRequest, input_data: Iterable[str]) -> KunitResult: parse_start = time.time() test_result = kunit_parser.TestResult(kunit_parser.TestStatus.SUCCESS, - [], + kunit_parser.Test(), 'Tests not Parsed.') if request.raw_output: + # Treat unparsed results as one passing test. + test_result.test.status = kunit_parser.TestStatus.SUCCESS + test_result.test.counts.passed = 1 + output: Iterable[str] = input_data if request.raw_output == 'all': pass diff --git a/tools/testing/kunit/kunit_json.py b/tools/testing/kunit/kunit_json.py index f5cca5c38cac..746bec72b9ac 100644 --- a/tools/testing/kunit/kunit_json.py +++ b/tools/testing/kunit/kunit_json.py @@ -11,47 +11,47 @@ import os import kunit_parser -from kunit_parser import TestStatus - -def get_json_result(test_result, def_config, build_dir, json_path) -> str: - sub_groups = [] - - # Each test suite is mapped to a KernelCI sub_group - for test_suite in test_result.suites: - sub_group = { - "name": test_suite.name, - "arch": "UM", - "defconfig": def_config, - "build_environment": build_dir, - "test_cases": [], - "lab_name": None, - "kernel": None, - "job": None, - "git_branch": "kselftest", - } - test_cases = [] - # TODO: Add attachments attribute in test_case with detailed - # failure message, see https://api.kernelci.org/schema-test-case.html#get - for case in test_suite.cases: - test_case = {"name": case.name, "status": "FAIL"} - if case.status == TestStatus.SUCCESS: +from kunit_parser import Test, TestResult, TestStatus +from typing import Any, Dict, Optional + +JsonObj = Dict[str, Any] + +def _get_group_json(test: Test, def_config: str, + build_dir: Optional[str]) -> JsonObj: + sub_groups = [] # List[JsonObj] + test_cases = [] # List[JsonObj] + + for subtest in test.subtests: + if len(subtest.subtests): + sub_group = _get_group_json(subtest, def_config, + build_dir) + sub_groups.append(sub_group) + else: + test_case = {"name": subtest.name, "status": "FAIL"} + if subtest.status == TestStatus.SUCCESS: test_case["status"] = "PASS" - elif case.status == TestStatus.TEST_CRASHED: + elif subtest.status == TestStatus.TEST_CRASHED: test_case["status"] = "ERROR" test_cases.append(test_case) - sub_group["test_cases"] = test_cases - sub_groups.append(sub_group) + test_group = { - "name": "KUnit Test Group", + "name": test.name, "arch": "UM", "defconfig": def_config, "build_environment": build_dir, "sub_groups": sub_groups, + "test_cases": test_cases, "lab_name": None, "kernel": None, "job": None, "git_branch": "kselftest", } + return test_group + +def get_json_result(test_result: TestResult, def_config: str, + build_dir: Optional[str], json_path: str) -> str: + test_group = _get_group_json(test_result.test, def_config, build_dir) + test_group["name"] = "KUnit Test Group" json_obj = json.dumps(test_group, indent=4) if json_path != 'stdout': with open(json_path, 'w') as result_path: diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py index 6310a641b151..f01fd565f978 100644 --- a/tools/testing/kunit/kunit_parser.py +++ b/tools/testing/kunit/kunit_parser.py @@ -1,11 +1,15 @@ # SPDX-License-Identifier: GPL-2.0 # -# Parses test results from a kernel dmesg log. +# Parses KTAP test results from a kernel dmesg log and incrementally prints +# results with reader-friendly format. Stores and returns test results in a +# Test object. # # Copyright (C) 2019, Google LLC. # Author: Felix Guo <felixguoxiuping(a)gmail.com> # Author: Brendan Higgins <brendanhiggins(a)google.com> +# Author: Rae Moar <rmoar(a)google.com> +from __future__ import annotations import re from collections import namedtuple @@ -14,33 +18,52 @@ from enum import Enum, auto from functools import reduce from typing import Iterable, Iterator, List, Optional, Tuple -TestResult = namedtuple('TestResult', ['status','suites','log']) - -class TestSuite(object): +TestResult = namedtuple('TestResult', ['status','test','log']) + +class Test(object): + """ + A class to represent a test parsed from KTAP results. All KTAP + results within a test log are stored in a main Test object as + subtests. + + Attributes: + status : TestStatus - status of the test + name : str - name of the test + expected_count : int - expected number of subtests (0 if single + test case and None if unknown expected number of subtests) + subtests : List[Test] - list of subtests + log : List[str] - log of KTAP lines that correspond to the test + counts : TestCounts - counts of the test statuses and errors of + subtests or of the test itself if the test is a single + test case. + """ def __init__(self) -> None: - self.status = TestStatus.SUCCESS - self.name = '' - self.cases = [] # type: List[TestCase] - - def __str__(self) -> str: - return 'TestSuite(' + str(self.status) + ',' + self.name + ',' + str(self.cases) + ')' - - def __repr__(self) -> str: - return str(self) - -class TestCase(object): - def __init__(self) -> None: - self.status = TestStatus.SUCCESS + """Creates Test object with default attributes.""" + self.status = TestStatus.TEST_CRASHED self.name = '' + self.expected_count = 0 # type: Optional[int] + self.subtests = [] # type: List[Test] self.log = [] # type: List[str] + self.counts = TestCounts() def __str__(self) -> str: - return 'TestCase(' + str(self.status) + ',' + self.name + ',' + str(self.log) + ')' + """Returns string representation of a Test class object.""" + return ('Test(' + str(self.status) + ', ' + self.name + + ', ' + str(self.expected_count) + ', ' + + str(self.subtests) + ', ' + str(self.log) + ', ' + + str(self.counts) + ')') def __repr__(self) -> str: + """Returns string representation of a Test class object.""" return str(self) + def add_error(self, error_message: str) -> None: + """Records an error that occurred while parsing this test.""" + self.counts.errors += 1 + print_error('Test ' + self.name + ': ' + error_message) + class TestStatus(Enum): + """An enumeration class to represent the status of a test.""" SUCCESS = auto() FAILURE = auto() SKIPPED = auto() @@ -48,381 +71,753 @@ class TestStatus(Enum): NO_TESTS = auto() FAILURE_TO_PARSE_TESTS = auto() +class TestCounts: + """ + Tracks the counts of statuses of all test cases and any errors within + a Test. + + Attributes: + passed : int - the number of tests that have passed + failed : int - the number of tests that have failed + crashed : int - the number of tests that have crashed + skipped : int - the number of tests that have skipped + errors : int - the number of errors in the test and subtests + """ + def __init__(self): + """Creates TestCounts object with counts of all test + statuses and test errors set to 0. + """ + self.passed = 0 + self.failed = 0 + self.crashed = 0 + self.skipped = 0 + self.errors = 0 + + def __str__(self) -> str: + """Returns the string representation of a TestCounts object. + """ + return ('Passed: ' + str(self.passed) + + ', Failed: ' + str(self.failed) + + ', Crashed: ' + str(self.crashed) + + ', Skipped: ' + str(self.skipped) + + ', Errors: ' + str(self.errors)) + + def total(self) -> int: + """Returns the total number of test cases within a test + object, where a test case is a test with no subtests. + """ + return (self.passed + self.failed + self.crashed + + self.skipped) + + def add_subtest_counts(self, counts: TestCounts) -> None: + """ + Adds the counts of another TestCounts object to the current + TestCounts object. Used to add the counts of a subtest to the + parent test. + + Parameters: + counts - a different TestCounts object whose counts + will be added to the counts of the TestCounts object + """ + self.passed += counts.passed + self.failed += counts.failed + self.crashed += counts.crashed + self.skipped += counts.skipped + self.errors += counts.errors + + def get_status(self) -> TestStatus: + """Returns the aggregated status of a Test using test + counts. + """ + if self.total() == 0: + return TestStatus.NO_TESTS + elif self.crashed: + # If one of the subtests crash, the expected status + # of the Test is crashed. + return TestStatus.TEST_CRASHED + elif self.failed: + # Otherwise if one of the subtests fail, the + # expected status of the Test is failed. + return TestStatus.FAILURE + elif self.passed: + # Otherwise if one of the subtests pass, the + # expected status of the Test is passed. + return TestStatus.SUCCESS + else: + # Finally, if none of the subtests have failed, + # crashed, or passed, the expected status of the + # Test is skipped. + return TestStatus.SKIPPED + + def add_status(self, status: TestStatus) -> None: + """ + Increments count of inputted status. + + Parameters: + status - status to be added to the TestCounts object + """ + if status == TestStatus.SUCCESS or \ + status == TestStatus.NO_TESTS: + # if status is NO_TESTS the most appropriate + # attribute to increment is passed because + # the test did not fail, crash or get skipped. + self.passed += 1 + elif status == TestStatus.FAILURE: + self.failed += 1 + elif status == TestStatus.SKIPPED: + self.skipped += 1 + else: + self.crashed += 1 + class LineStream: - """Provides a peek()/pop() interface over an iterator of (line#, text).""" + """ + A class to represent the lines of kernel output. + Provides a peek()/pop() interface over an iterator of + (line#, text). + """ _lines: Iterator[Tuple[int, str]] _next: Tuple[int, str] _done: bool def __init__(self, lines: Iterator[Tuple[int, str]]): + """Creates a new LineStream that wraps the given iterator.""" self._lines = lines self._done = False self._next = (0, '') self._get_next() def _get_next(self) -> None: + """Advances the LineSteam to the next line.""" try: self._next = next(self._lines) except StopIteration: self._done = True def peek(self) -> str: + """Returns the current line, without advancing the LineStream. + """ return self._next[1] def pop(self) -> str: + """Returns the current line and advances the LineStream to + the next line. + """ n = self._next self._get_next() return n[1] def __bool__(self) -> bool: + """Returns True if stream has more lines.""" return not self._done # Only used by kunit_tool_test.py. def __iter__(self) -> Iterator[str]: + """Empties all lines stored in LineStream object into + Iterator object and returns the Iterator object. + """ while bool(self): yield self.pop() def line_number(self) -> int: + """Returns the line number of the current line.""" return self._next[0] -kunit_start_re = re.compile(r'TAP version [0-9]+$') -kunit_end_re = re.compile('(List of all partitions:|' - 'Kernel panic - not syncing: VFS:|reboot: System halted)') +# Parsing helper methods: + +KTAP_START = re.compile(r'KTAP version ([0-9]+)$') +TAP_START = re.compile(r'TAP version ([0-9]+)$') +KTAP_END = re.compile('(List of all partitions:|' + 'Kernel panic - not syncing: VFS:|reboot: System halted)') def extract_tap_lines(kernel_output: Iterable[str]) -> LineStream: - def isolate_kunit_output(kernel_output: Iterable[str]) -> Iterator[Tuple[int, str]]: + """Extracts KTAP lines from the kernel output.""" + def isolate_ktap_output(kernel_output: Iterable[str]) \ + -> Iterator[Tuple[int, str]]: line_num = 0 started = False for line in kernel_output: line_num += 1 - line = line.rstrip() # line always has a trailing \n - if kunit_start_re.search(line): + line = line.rstrip() # remove trailing \n + if not started and KTAP_START.search(line): + # start extracting KTAP lines and set prefix + # to number of characters before version line + prefix_len = len( + line.split('KTAP version')[0]) + started = True + yield line_num, line[prefix_len:] + elif not started and TAP_START.search(line): + # start extracting KTAP lines and set prefix + # to number of characters before version line prefix_len = len(line.split('TAP version')[0]) started = True yield line_num, line[prefix_len:] - elif kunit_end_re.search(line): + elif started and KTAP_END.search(line): + # stop extracting KTAP lines break elif started: - yield line_num, line[prefix_len:] - return LineStream(lines=isolate_kunit_output(kernel_output)) - -DIVIDER = '=' * 60 - -RESET = '\033[0;0m' - -def red(text) -> str: - return '\033[1;31m' + text + RESET - -def yellow(text) -> str: - return '\033[1;33m' + text + RESET - -def green(text) -> str: - return '\033[1;32m' + text + RESET - -def print_with_timestamp(message) -> None: - print('[%s] %s' % (datetime.now().strftime('%H:%M:%S'), message)) + # remove prefix and any indention and yield + # line with line number + line = line[prefix_len:].lstrip() + yield line_num, line + return LineStream(lines=isolate_ktap_output(kernel_output)) + +KTAP_VERSIONS = [1] +TAP_VERSIONS = [13, 14] + +def check_version(version_num: int, accepted_versions: List[int], + version_type: str, test: Test) -> None: + """ + Adds error to test object if version number is too high or too + low. + + Parameters: + version_num - The inputted version number from the parsed KTAP or TAP + header line + accepted_version - List of accepted KTAP or TAP versions + version_type - 'KTAP' or 'TAP' depending on the type of + version line. + test - Test object for current test being parsed + """ + if version_num < min(accepted_versions): + test.add_error(version_type + + ' version lower than expected!') + elif version_num > max(accepted_versions): + test.add_error( + version_type + ' version higher than expected!') + +def parse_ktap_header(lines: LineStream, test: Test) -> bool: + """ + Parses KTAP/TAP header line and checks version number. + Returns False if fails to parse KTAP/TAP header line. + + Accepted formats: + - 'KTAP version [version number]' + - 'TAP version [version number]' + + Parameters: + lines - LineStream of KTAP output to parse + test - Test object for current test being parsed + + Return: + True if successfully parsed KTAP/TAP header line + """ + ktap_match = KTAP_START.match(lines.peek()) + tap_match = TAP_START.match(lines.peek()) + if ktap_match: + version_num = int(ktap_match.group(1)) + check_version(version_num, KTAP_VERSIONS, 'KTAP', test) + elif tap_match: + version_num = int(tap_match.group(1)) + check_version(version_num, TAP_VERSIONS, 'TAP', test) + else: + return False + test.log.append(lines.pop()) + return True -def format_suite_divider(message) -> str: - return '======== ' + message + ' ========' +TEST_HEADER = re.compile(r'^# Subtest: (.*)$') -def print_suite_divider(message) -> None: - print_with_timestamp(DIVIDER) - print_with_timestamp(format_suite_divider(message)) +def parse_test_header(lines: LineStream, test: Test) -> bool: + """ + Parses test header and stores test name in test object. + Returns False if fails to parse test header line. -def print_log(log) -> None: - for m in log: - print_with_timestamp(m) + Accepted format: + - '# Subtest: [test name]' -TAP_ENTRIES = re.compile(r'^(TAP|[\s]*ok|[\s]*not ok|[\s]*[0-9]+\.\.[0-9]+|[\s]*# (Subtest:|.*: kunit test case crashed!)).*$') + Parameters: + lines - LineStream of KTAP output to parse + test - Test object for current test being parsed -def consume_non_diagnostic(lines: LineStream) -> None: - while lines and not TAP_ENTRIES.match(lines.peek()): - lines.pop() - -def save_non_diagnostic(lines: LineStream, test_case: TestCase) -> None: - while lines and not TAP_ENTRIES.match(lines.peek()): - test_case.log.append(lines.peek()) - lines.pop() + Return: + True if successfully parsed test header line + """ + match = TEST_HEADER.match(lines.peek()) + if not match: + return False + test.log.append(lines.pop()) + test.name = match.group(1) + return True -OkNotOkResult = namedtuple('OkNotOkResult', ['is_ok','description', 'text']) +TEST_PLAN = re.compile(r'1\.\.([0-9]+)') -OK_NOT_OK_SKIP = re.compile(r'^[\s]*(ok|not ok) [0-9]+ - (.*) # SKIP(.*)$') +def parse_test_plan(lines: LineStream, test: Test) -> bool: + """ + Parses test plan line and stores the expected number of subtests in + test object. Reports an error if expected count is 0. + Returns False and reports missing test plan error if fails to parse + test plan. -OK_NOT_OK_SUBTEST = re.compile(r'^[\s]+(ok|not ok) [0-9]+ - (.*)$') + Accepted format: + - '1..[number of subtests]' -OK_NOT_OK_MODULE = re.compile(r'^(ok|not ok) ([0-9]+) - (.*)$') + Parameters: + lines - LineStream of KTAP output to parse + test - Test object for current test being parsed -def parse_ok_not_ok_test_case(lines: LineStream, test_case: TestCase) -> bool: - save_non_diagnostic(lines, test_case) - if not lines: - test_case.status = TestStatus.TEST_CRASHED - return True - line = lines.peek() - match = OK_NOT_OK_SUBTEST.match(line) - while not match and lines: - line = lines.pop() - match = OK_NOT_OK_SUBTEST.match(line) - if match: - test_case.log.append(lines.pop()) - test_case.name = match.group(2) - skip_match = OK_NOT_OK_SKIP.match(line) - if skip_match: - test_case.status = TestStatus.SKIPPED - return True - if test_case.status == TestStatus.TEST_CRASHED: - return True - if match.group(1) == 'ok': - test_case.status = TestStatus.SUCCESS - else: - test_case.status = TestStatus.FAILURE - return True - else: + Return: + True if successfully parsed test plan line + """ + match = TEST_PLAN.match(lines.peek()) + if not match: + test.expected_count = None + test.add_error('missing plan line!') return False - -SUBTEST_DIAGNOSTIC = re.compile(r'^[\s]+# (.*)$') -DIAGNOSTIC_CRASH_MESSAGE = re.compile(r'^[\s]+# .*?: kunit test case crashed!$') - -def parse_diagnostic(lines: LineStream, test_case: TestCase) -> bool: - save_non_diagnostic(lines, test_case) - if not lines: + test.log.append(lines.pop()) + expected_count = int(match.group(1)) + test.expected_count = expected_count + if expected_count == 0: + test.status = TestStatus.NO_TESTS + test.add_error('0 tests run!') + return True + +TEST_RESULT = re.compile(r'^(ok|not ok) ([0-9]+) (- )?([^#]*)( # .*)?$') + +TEST_RESULT_SKIP = re.compile(r'^(ok|not ok) ([0-9]+) (- )?(.*) # SKIP(.*)$') + +def peek_test_name_match(lines: LineStream, test: Test) -> bool: + """ + Matches current line with the format of a test result line and checks + if the name matches the name of the current test. + Returns False if fails to match format or name. + + Accepted format: + - '[ok|not ok] [test number] [-] [test name] [optional skip + directive]' + + Parameters: + lines - LineStream of KTAP output to parse + test - Test object for current test being parsed + + Return: + True if matched a test result line and the name matching the + expected test name + """ + line = lines.peek() + match = TEST_RESULT.match(line) + if not match: return False + name = match.group(4) + return (name == test.name) + +def parse_test_result(lines: LineStream, test: Test, + expected_num: int) -> bool: + """ + Parses test result line and stores the status and name in the test + object. Reports an error if the test number does not match expected + test number. + Returns False if fails to parse test result line. + + Note that the SKIP directive is the only direction that causes a + change in status. + + Accepted format: + - '[ok|not ok] [test number] [-] [test name] [optional skip + directive]' + + Parameters: + lines - LineStream of KTAP output to parse + test - Test object for current test being parsed + expected_num - expected test number for current test + + Return: + True if successfully parsed a test result line. + """ line = lines.peek() - match = SUBTEST_DIAGNOSTIC.match(line) - if match: - test_case.log.append(lines.pop()) - crash_match = DIAGNOSTIC_CRASH_MESSAGE.match(line) - if crash_match: - test_case.status = TestStatus.TEST_CRASHED - return True - else: + match = TEST_RESULT.match(line) + skip_match = TEST_RESULT_SKIP.match(line) + + # Check if line matches test result line format + if not match: return False + test.log.append(lines.pop()) -def parse_test_case(lines: LineStream) -> Optional[TestCase]: - test_case = TestCase() - save_non_diagnostic(lines, test_case) - while parse_diagnostic(lines, test_case): - pass - if parse_ok_not_ok_test_case(lines, test_case): - return test_case + # Set name of test object + if skip_match: + test.name = skip_match.group(4) else: - return None - -SUBTEST_HEADER = re.compile(r'^[\s]+# Subtest: (.*)$') - -def parse_subtest_header(lines: LineStream) -> Optional[str]: - consume_non_diagnostic(lines) - if not lines: - return None - match = SUBTEST_HEADER.match(lines.peek()) - if match: - lines.pop() - return match.group(1) + test.name = match.group(4) + + # Check test num + num = int(match.group(2)) + if num != expected_num: + test.add_error('Expected test number ' + + str(expected_num) + ' but found ' + str(num)) + + # Set status of test object + status = match.group(1) + if skip_match: + test.status = TestStatus.SKIPPED + elif status == 'ok': + test.status = TestStatus.SUCCESS else: - return None + test.status = TestStatus.FAILURE + return True + +def parse_diagnostic(lines: LineStream) -> List[str]: + """ + Parse lines that do not match the format of a test result line or + test header line and returns them in list. + + Line formats that are not parsed: + - '# Subtest: [test name]' + - '[ok|not ok] [test number] [-] [test name] [optional skip + directive]' + + Parameters: + lines - LineStream of KTAP output to parse + + Return: + Log of diagnostic lines + """ + log = [] # type: List[str] + while lines and not TEST_RESULT.match(lines.peek()) and not \ + TEST_HEADER.match(lines.peek()): + log.append(lines.pop()) + return log + +DIAGNOSTIC_CRASH_MESSAGE = re.compile( + r'^(BUG:|# .*?: kunit test case crashed!$)') + +def parse_crash_in_log(test: Test) -> bool: + """ + Iterate through the lines of the log to parse for crash message. + If crash message found, set status to crashed and return True. + Otherwise return False. + + Parameters: + test - Test object for current test being parsed + + Return: + True if crash message found in log + """ + for line in test.log: + if DIAGNOSTIC_CRASH_MESSAGE.match(line): + test.status = TestStatus.TEST_CRASHED + return True + return False -SUBTEST_PLAN = re.compile(r'[\s]+[0-9]+\.\.([0-9]+)') -def parse_subtest_plan(lines: LineStream) -> Optional[int]: - consume_non_diagnostic(lines) - match = SUBTEST_PLAN.match(lines.peek()) - if match: - lines.pop() - return int(match.group(1)) - else: - return None - -def max_status(left: TestStatus, right: TestStatus) -> TestStatus: - if left == right: - return left - elif left == TestStatus.TEST_CRASHED or right == TestStatus.TEST_CRASHED: - return TestStatus.TEST_CRASHED - elif left == TestStatus.FAILURE or right == TestStatus.FAILURE: - return TestStatus.FAILURE - elif left == TestStatus.SKIPPED: - return right - else: - return left +# Printing helper methods: -def parse_ok_not_ok_test_suite(lines: LineStream, - test_suite: TestSuite, - expected_suite_index: int) -> bool: - consume_non_diagnostic(lines) - if not lines: - test_suite.status = TestStatus.TEST_CRASHED - return False - line = lines.peek() - match = OK_NOT_OK_MODULE.match(line) - if match: - lines.pop() - if match.group(1) == 'ok': - test_suite.status = TestStatus.SUCCESS - else: - test_suite.status = TestStatus.FAILURE - skip_match = OK_NOT_OK_SKIP.match(line) - if skip_match: - test_suite.status = TestStatus.SKIPPED - suite_index = int(match.group(2)) - if suite_index != expected_suite_index: - print_with_timestamp( - red('[ERROR] ') + 'expected_suite_index ' + - str(expected_suite_index) + ', but got ' + - str(suite_index)) - return True - else: - return False +DIVIDER = '=' * 60 -def bubble_up_errors(status_list: Iterable[TestStatus]) -> TestStatus: - return reduce(max_status, status_list, TestStatus.SKIPPED) +RESET = '\033[0;0m' -def bubble_up_test_case_errors(test_suite: TestSuite) -> TestStatus: - max_test_case_status = bubble_up_errors(x.status for x in test_suite.cases) - return max_status(max_test_case_status, test_suite.status) +def red(text: str) -> str: + """Returns inputted string with red color code.""" + return '\033[1;31m' + text + RESET -def parse_test_suite(lines: LineStream, expected_suite_index: int) -> Optional[TestSuite]: - if not lines: - return None - consume_non_diagnostic(lines) - test_suite = TestSuite() - test_suite.status = TestStatus.SUCCESS - name = parse_subtest_header(lines) - if not name: - return None - test_suite.name = name - expected_test_case_num = parse_subtest_plan(lines) - if expected_test_case_num is None: - return None - while expected_test_case_num > 0: - test_case = parse_test_case(lines) - if not test_case: - break - test_suite.cases.append(test_case) - expected_test_case_num -= 1 - if parse_ok_not_ok_test_suite(lines, test_suite, expected_suite_index): - test_suite.status = bubble_up_test_case_errors(test_suite) - return test_suite - elif not lines: - print_with_timestamp(red('[ERROR] ') + 'ran out of lines before end token') - return test_suite - else: - print(f'failed to parse end of suite "{name}", at line {lines.line_number()}: {lines.peek()}') - return None +def yellow(text: str) -> str: + """Returns inputted string with yellow color code.""" + return '\033[1;33m' + text + RESET -TAP_HEADER = re.compile(r'^TAP version 14$') +def green(text: str) -> str: + """Returns inputted string with green color code.""" + return '\033[1;32m' + text + RESET -def parse_tap_header(lines: LineStream) -> bool: - consume_non_diagnostic(lines) - if TAP_HEADER.match(lines.peek()): - lines.pop() - return True - else: - return False +ANSI_LEN = len(red('')) -TEST_PLAN = re.compile(r'[0-9]+\.\.([0-9]+)') +def print_with_timestamp(message: str) -> None: + """Prints message with timestamp at beginning.""" + print('[%s] %s' % (datetime.now().strftime('%H:%M:%S'), message)) -def parse_test_plan(lines: LineStream) -> Optional[int]: - consume_non_diagnostic(lines) - match = TEST_PLAN.match(lines.peek()) - if match: - lines.pop() - return int(match.group(1)) - else: - return None - -def bubble_up_suite_errors(test_suites: Iterable[TestSuite]) -> TestStatus: - return bubble_up_errors(x.status for x in test_suites) - -def parse_test_result(lines: LineStream) -> TestResult: - consume_non_diagnostic(lines) - if not lines or not parse_tap_header(lines): - return TestResult(TestStatus.FAILURE_TO_PARSE_TESTS, [], lines) - expected_test_suite_num = parse_test_plan(lines) - if expected_test_suite_num == 0: - return TestResult(TestStatus.NO_TESTS, [], lines) - elif expected_test_suite_num is None: - return TestResult(TestStatus.FAILURE_TO_PARSE_TESTS, [], lines) - test_suites = [] - for i in range(1, expected_test_suite_num + 1): - test_suite = parse_test_suite(lines, i) - if test_suite: - test_suites.append(test_suite) +def format_test_divider(message: str, len_message: int) -> str: + """ + Returns string with message centered in fixed width divider. + + Example: + '===================== message example =====================' + + Parameters: + message - message to be centered in divider line + len_message - length of the message to be printed such that + any characters of the color codes are not counted + + Return: + String containing message centered in fixed width divider + """ + default_count = 3 # default number of dashes + len_1 = default_count + len_2 = default_count + difference = len(DIVIDER) - len_message - 2 # 2 spaces added + if difference > 0: + # calculate number of dashes for each side of the divider + len_1 = int(difference / 2) + len_2 = difference - len_1 + return ('=' * len_1) + ' ' + message + ' ' + ('=' * len_2) + +def print_test_header(test: Test) -> None: + """ + Prints test header with test name and optionally the expected number + of subtests. + + Example: + '=================== example (2 subtests) ===================' + + Parameters: + test - Test object representing current test being printed + """ + message = test.name + if test.expected_count: + if test.expected_count == 1: + message += (' (' + str(test.expected_count) + + ' subtest)') else: - print_with_timestamp( - red('[ERROR] ') + ' expected ' + - str(expected_test_suite_num) + - ' test suites, but got ' + str(i - 2)) - break - test_suite = parse_test_suite(lines, -1) - if test_suite: - print_with_timestamp(red('[ERROR] ') + - 'got unexpected test suite: ' + test_suite.name) - if test_suites: - return TestResult(bubble_up_suite_errors(test_suites), test_suites, lines) - else: - return TestResult(TestStatus.NO_TESTS, [], lines) + message += (' (' + str(test.expected_count) + + ' subtests)') + print_with_timestamp(format_test_divider(message, len(message))) -class TestCounts: - passed: int - failed: int - crashed: int - skipped: int +def print_log(log: Iterable[str]) -> None: + """ + Prints all strings in saved log for test in yellow. - def __init__(self): - self.passed = 0 - self.failed = 0 - self.crashed = 0 - self.skipped = 0 - - def total(self) -> int: - return self.passed + self.failed + self.crashed + self.skipped - -def print_and_count_results(test_result: TestResult) -> TestCounts: - counts = TestCounts() - for test_suite in test_result.suites: - if test_suite.status == TestStatus.SUCCESS: - print_suite_divider(green('[PASSED] ') + test_suite.name) - elif test_suite.status == TestStatus.SKIPPED: - print_suite_divider(yellow('[SKIPPED] ') + test_suite.name) - elif test_suite.status == TestStatus.TEST_CRASHED: - print_suite_divider(red('[CRASHED] ' + test_suite.name)) - else: - print_suite_divider(red('[FAILED] ') + test_suite.name) - for test_case in test_suite.cases: - if test_case.status == TestStatus.SUCCESS: - counts.passed += 1 - print_with_timestamp(green('[PASSED] ') + test_case.name) - elif test_case.status == TestStatus.SKIPPED: - counts.skipped += 1 - print_with_timestamp(yellow('[SKIPPED] ') + test_case.name) - elif test_case.status == TestStatus.TEST_CRASHED: - counts.crashed += 1 - print_with_timestamp(red('[CRASHED] ' + test_case.name)) - print_log(map(yellow, test_case.log)) - print_with_timestamp('') + Parameters: + log - Iterable object with all strings saved in log for test + """ + for m in log: + print_with_timestamp(yellow(m)) + +def format_test_result(test: Test) -> str: + """ + Returns string with formatted test result with colored status and test + name. + + Example: + '[PASSED] example' + + Parameters: + test - Test object representing current test being printed + + Return: + String containing formatted test result + """ + if test.status == TestStatus.SUCCESS: + return (green('[PASSED] ') + test.name) + elif test.status == TestStatus.SKIPPED: + return (yellow('[SKIPPED] ') + test.name) + elif test.status == TestStatus.TEST_CRASHED: + print_log(test.log) + return (red('[CRASHED] ') + test.name) + else: + print_log(test.log) + return (red('[FAILED] ') + test.name) + +def print_test_result(test: Test) -> None: + """ + Prints result line with status of test. + + Example: + '[PASSED] example' + + Parameters: + test - Test object representing current test being printed + """ + print_with_timestamp(format_test_result(test)) + +def print_test_footer(test: Test) -> None: + """ + Prints test footer with status of test. + + Example: + '===================== [PASSED] example =====================' + + Parameters: + test - Test object representing current test being printed + """ + message = format_test_result(test) + print_with_timestamp(format_test_divider(message, + len(message) - ANSI_LEN)) + +def print_summary_line(test: Test) -> None: + """ + Prints summary line of test object. Color of line is dependent on + status of test. Color is green if test passes, yellow if test is + skipped, and red if the test fails or crashes. Summary line contains + counts of the statuses of the tests subtests or the test itself if it + has no subtests. + + Example: + "Testing complete. Passed: 2, Failed: 0, Crashed: 0, Skipped: 0, + Errors: 0" + + test - Test object representing current test being printed + """ + if test.status == TestStatus.SUCCESS or \ + test.status == TestStatus.NO_TESTS: + color = green + elif test.status == TestStatus.SKIPPED: + color = yellow + else: + color = red + counts = test.counts + print_with_timestamp(color('Testing complete. ' + str(counts))) + +def print_error(error_message: str) -> None: + """ + Prints error message with error format. + + Example: + "[ERROR] Test example: missing test plan!" + + Parameters: + error_message - message describing error + """ + print_with_timestamp(red('[ERROR] ') + error_message) + +# Other methods: + +def bubble_up_test_results(test: Test) -> None: + """ + If the test has subtests, add the test counts of the subtests to the + test and check if any of the tests crashed and if so set the test + status to crashed. Otherwise if the test has no subtests add the + status of the test to the test counts. + + Parameters: + test - Test object for current test being parsed + """ + parse_crash_in_log(test) + subtests = test.subtests + counts = test.counts + status = test.status + for t in subtests: + counts.add_subtest_counts(t.counts) + if counts.total() == 0: + counts.add_status(status) + elif test.counts.get_status() == TestStatus.TEST_CRASHED: + test.status = TestStatus.TEST_CRASHED + +def parse_test(lines: LineStream, expected_num: int, log: List[str]) -> Test: + """ + Finds next test to parse in LineStream, creates new Test object, + parses any subtests of the test, populates Test object with all + information (status, name) about the test and the Test objects for + any subtests, and then returns the Test object. The method accepts + three formats of tests: + + Accepted test formats: + + - Main KTAP/TAP header + + Example: + + KTAP version 1 + 1..4 + [subtests] + + - Subtest header line + + Example: + + # Subtest: name + 1..3 + [subtests] + ok 1 name + + - Test result line + + Example: + + ok 1 - test + + Parameters: + lines - LineStream of KTAP output to parse + expected_num - expected test number for test to be parsed + log - list of strings containing any preceding diagnostic lines + corresponding to the current test + + Return: + Test object populated with characteristics and any subtests + """ + test = Test() + test.log.extend(log) + parent_test = False + main = parse_ktap_header(lines, test) + if main: + # If KTAP/TAP header is found, attempt to parse + # test plan + test.name = "main" + parse_test_plan(lines, test) + else: + # If KTAP/TAP header is not found, test must be subtest + # header or test result line so parse attempt to parser + # subtest header + parent_test = parse_test_header(lines, test) + if parent_test: + # If subtest header is found, attempt to parse + # test plan and print header + parse_test_plan(lines, test) + print_test_header(test) + expected_count = test.expected_count + subtests = [] + test_num = 1 + while expected_count is None or test_num <= expected_count: + # Loop to parse any subtests. + # Break after parsing expected number of tests or + # if expected number of tests is unknown break when test + # result line with matching name to subtest header is found + # or no more lines in stream. + sub_log = parse_diagnostic(lines) + sub_test = Test() + if not lines or (peek_test_name_match(lines, test) and + not main): + if expected_count and test_num <= expected_count: + # If parser reaches end of test before + # parsing expected number of subtests, print + # crashed subtest and record error + test.add_error('missing expected subtest!') + sub_test.log.extend(sub_log) + test.counts.add_status( + TestStatus.TEST_CRASHED) + print_test_result(sub_test) else: - counts.failed += 1 - print_with_timestamp(red('[FAILED] ') + test_case.name) - print_log(map(yellow, test_case.log)) - print_with_timestamp('') - return counts + test.log.extend(sub_log) + break + else: + sub_test = parse_test(lines, test_num, sub_log) + subtests.append(sub_test) + test_num += 1 + test.subtests = subtests + if not main: + # If not main test, look for test result line + test.log.extend(parse_diagnostic(lines)) + if (parent_test and peek_test_name_match(lines, test)) or \ + not parent_test: + parse_test_result(lines, test, expected_num) + else: + test.add_error('missing subtest result line!') + # Add statuses to TestCounts attribute in Test object + bubble_up_test_results(test) + if parent_test: + # If test has subtests and is not the main test object, print + # footer. + print_test_footer(test) + elif not main: + print_test_result(test) + return test def parse_run_tests(kernel_output: Iterable[str]) -> TestResult: - counts = TestCounts() + """ + Using kernel output, extract KTAP lines, parse the lines for test + results and print condensed test results and summary line . + + Parameters: + kernel_output - Iterable object contains lines of kernel output + + Return: + TestResult - Tuple containg status of main test object, main test + object with all subtests, and log of all KTAP lines. + """ + print_with_timestamp(DIVIDER) lines = extract_tap_lines(kernel_output) - test_result = parse_test_result(lines) - if test_result.status == TestStatus.NO_TESTS: - print(red('[ERROR] ') + yellow('no tests run!')) - elif test_result.status == TestStatus.FAILURE_TO_PARSE_TESTS: - print(red('[ERROR] ') + yellow('could not parse test results!')) + test = Test() + if not lines: + test.add_error('invalid KTAP input!') + test.status = TestStatus.FAILURE_TO_PARSE_TESTS else: - counts = print_and_count_results(test_result) + test = parse_test(lines, 0, []) + if test.status != TestStatus.NO_TESTS: + test.status = test.counts.get_status() print_with_timestamp(DIVIDER) - if test_result.status == TestStatus.SUCCESS: - fmt = green - elif test_result.status == TestStatus.SKIPPED: - fmt = yellow - else: - fmt =red - print_with_timestamp( - fmt('Testing complete. %d tests run. %d failed. %d crashed. %d skipped.' % - (counts.total(), counts.failed, counts.crashed, counts.skipped))) - return test_result + print_summary_line(test) + return TestResult(test.status, test, lines) diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index 3251cef25b76..c309ed76aef5 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -106,10 +106,10 @@ class KUnitParserTest(unittest.TestCase): with open(log_path) as file: result = kunit_parser.extract_tap_lines(file.readlines()) self.assertContains('TAP version 14', result) - self.assertContains(' # Subtest: example', result) - self.assertContains(' 1..2', result) - self.assertContains(' ok 1 - example_simple_test', result) - self.assertContains(' ok 2 - example_mock_test', result) + self.assertContains('# Subtest: example', result) + self.assertContains('1..2', result) + self.assertContains('ok 1 - example_simple_test', result) + self.assertContains('ok 2 - example_mock_test', result) self.assertContains('ok 1 - example', result) def test_output_with_prefix_isolated_correctly(self): @@ -117,28 +117,28 @@ class KUnitParserTest(unittest.TestCase): with open(log_path) as file: result = kunit_parser.extract_tap_lines(file.readlines()) self.assertContains('TAP version 14', result) - self.assertContains(' # Subtest: kunit-resource-test', result) - self.assertContains(' 1..5', result) - self.assertContains(' ok 1 - kunit_resource_test_init_resources', result) - self.assertContains(' ok 2 - kunit_resource_test_alloc_resource', result) - self.assertContains(' ok 3 - kunit_resource_test_destroy_resource', result) - self.assertContains(' foo bar #', result) - self.assertContains(' ok 4 - kunit_resource_test_cleanup_resources', result) - self.assertContains(' ok 5 - kunit_resource_test_proper_free_ordering', result) + self.assertContains('# Subtest: kunit-resource-test', result) + self.assertContains('1..5', result) + self.assertContains('ok 1 - kunit_resource_test_init_resources', result) + self.assertContains('ok 2 - kunit_resource_test_alloc_resource', result) + self.assertContains('ok 3 - kunit_resource_test_destroy_resource', result) + self.assertContains('foo bar #', result) + self.assertContains('ok 4 - kunit_resource_test_cleanup_resources', result) + self.assertContains('ok 5 - kunit_resource_test_proper_free_ordering', result) self.assertContains('ok 1 - kunit-resource-test', result) - self.assertContains(' foo bar # non-kunit output', result) - self.assertContains(' # Subtest: kunit-try-catch-test', result) - self.assertContains(' 1..2', result) - self.assertContains(' ok 1 - kunit_test_try_catch_successful_try_no_catch', + self.assertContains('foo bar # non-kunit output', result) + self.assertContains('# Subtest: kunit-try-catch-test', result) + self.assertContains('1..2', result) + self.assertContains('ok 1 - kunit_test_try_catch_successful_try_no_catch', result) - self.assertContains(' ok 2 - kunit_test_try_catch_unsuccessful_try_does_catch', + self.assertContains('ok 2 - kunit_test_try_catch_unsuccessful_try_does_catch', result) self.assertContains('ok 2 - kunit-try-catch-test', result) - self.assertContains(' # Subtest: string-stream-test', result) - self.assertContains(' 1..3', result) - self.assertContains(' ok 1 - string_stream_test_empty_on_creation', result) - self.assertContains(' ok 2 - string_stream_test_not_empty_after_add', result) - self.assertContains(' ok 3 - string_stream_test_get_string', result) + self.assertContains('# Subtest: string-stream-test', result) + self.assertContains('1..3', result) + self.assertContains('ok 1 - string_stream_test_empty_on_creation', result) + self.assertContains('ok 2 - string_stream_test_not_empty_after_add', result) + self.assertContains('ok 3 - string_stream_test_get_string', result) self.assertContains('ok 3 - string-stream-test', result) def test_parse_successful_test_log(self): @@ -149,6 +149,22 @@ class KUnitParserTest(unittest.TestCase): kunit_parser.TestStatus.SUCCESS, result.status) + def test_parse_successful_nested_tests_log(self): + all_passed_log = test_data_path('test_is_test_passed-all_passed_nested.log') + with open(all_passed_log) as file: + result = kunit_parser.parse_run_tests(file.readlines()) + self.assertEqual( + kunit_parser.TestStatus.SUCCESS, + result.status) + + def test_kselftest_nested(self): + kselftest_log = test_data_path('test_is_test_passed-kselftest.log') + with open(kselftest_log) as file: + result = kunit_parser.parse_run_tests(file.readlines()) + self.assertEqual( + kunit_parser.TestStatus.SUCCESS, + result.status) + def test_parse_failed_test_log(self): failed_log = test_data_path('test_is_test_passed-failure.log') with open(failed_log) as file: @@ -162,17 +178,31 @@ class KUnitParserTest(unittest.TestCase): with open(empty_log) as file: result = kunit_parser.parse_run_tests( kunit_parser.extract_tap_lines(file.readlines())) - self.assertEqual(0, len(result.suites)) + self.assertEqual(0, len(result.test.subtests)) self.assertEqual( kunit_parser.TestStatus.FAILURE_TO_PARSE_TESTS, result.status) + def test_missing_test_plan(self): + missing_plan_log = test_data_path('test_is_test_passed-' + 'missing_plan.log') + with open(missing_plan_log) as file: + result = kunit_parser.parse_run_tests( + kunit_parser.extract_tap_lines( + file.readlines())) + self.assertEqual(2, result.test.counts.errors) + self.assertEqual( + kunit_parser.TestStatus.SUCCESS, + result.status) + def test_no_tests(self): - empty_log = test_data_path('test_is_test_passed-no_tests_run_with_header.log') - with open(empty_log) as file: + header_log = test_data_path('test_is_test_passed-' + 'no_tests_run_with_header.log') + with open(header_log) as file: result = kunit_parser.parse_run_tests( - kunit_parser.extract_tap_lines(file.readlines())) - self.assertEqual(0, len(result.suites)) + kunit_parser.extract_tap_lines( + file.readlines())) + self.assertEqual(0, len(result.test.subtests)) self.assertEqual( kunit_parser.TestStatus.NO_TESTS, result.status) @@ -182,15 +212,17 @@ class KUnitParserTest(unittest.TestCase): print_mock = mock.patch('builtins.print').start() with open(crash_log) as file: result = kunit_parser.parse_run_tests( - kunit_parser.extract_tap_lines(file.readlines())) - print_mock.assert_any_call(StrContains('could not parse test results!')) + kunit_parser.extract_tap_lines( + file.readlines())) + print_mock.assert_any_call(StrContains('invalid KTAP input!')) print_mock.stop() file.close() def test_crashed_test(self): crashed_log = test_data_path('test_is_test_passed-crash.log') with open(crashed_log) as file: - result = kunit_parser.parse_run_tests(file.readlines()) + result = kunit_parser.parse_run_tests( + file.readlines()) self.assertEqual( kunit_parser.TestStatus.TEST_CRASHED, result.status) @@ -216,6 +248,23 @@ class KUnitParserTest(unittest.TestCase): result.status) file.close() + def test_ignores_hyphen(self): + hyphen_log = test_data_path('test_strip_hyphen.log') + file = open(hyphen_log) + result = kunit_parser.parse_run_tests(file.readlines()) + + # A skipped test does not fail the whole suite. + self.assertEqual( + kunit_parser.TestStatus.SUCCESS, + result.status) + self.assertEqual( + "sysctl_test", + result.test.subtests[0].name) + self.assertEqual( + "example", + result.test.subtests[1].name) + file.close() + def test_ignores_prefix_printk_time(self): prefix_log = test_data_path('test_config_printk_time.log') @@ -224,7 +273,7 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual( kunit_parser.TestStatus.SUCCESS, result.status) - self.assertEqual('kunit-resource-test', result.suites[0].name) + self.assertEqual('kunit-resource-test', result.test.subtests[0].name) def test_ignores_multiple_prefixes(self): prefix_log = test_data_path('test_multiple_prefixes.log') @@ -233,7 +282,7 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual( kunit_parser.TestStatus.SUCCESS, result.status) - self.assertEqual('kunit-resource-test', result.suites[0].name) + self.assertEqual('kunit-resource-test', result.test.subtests[0].name) def test_prefix_mixed_kernel_output(self): mixed_prefix_log = test_data_path('test_interrupted_tap_output.log') @@ -242,7 +291,7 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual( kunit_parser.TestStatus.SUCCESS, result.status) - self.assertEqual('kunit-resource-test', result.suites[0].name) + self.assertEqual('kunit-resource-test', result.test.subtests[0].name) def test_prefix_poundsign(self): pound_log = test_data_path('test_pound_sign.log') @@ -251,7 +300,7 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual( kunit_parser.TestStatus.SUCCESS, result.status) - self.assertEqual('kunit-resource-test', result.suites[0].name) + self.assertEqual('kunit-resource-test', result.test.subtests[0].name) def test_kernel_panic_end(self): panic_log = test_data_path('test_kernel_panic_interrupt.log') @@ -260,7 +309,7 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual( kunit_parser.TestStatus.TEST_CRASHED, result.status) - self.assertEqual('kunit-resource-test', result.suites[0].name) + self.assertEqual('kunit-resource-test', result.test.subtests[0].name) def test_pound_no_prefix(self): pound_log = test_data_path('test_pound_no_prefix.log') @@ -269,7 +318,7 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual( kunit_parser.TestStatus.SUCCESS, result.status) - self.assertEqual('kunit-resource-test', result.suites[0].name) + self.assertEqual('kunit-resource-test', result.test.subtests[0].name) class LinuxSourceTreeTest(unittest.TestCase): @@ -322,6 +371,12 @@ class KUnitJsonTest(unittest.TestCase): result = self._json_for('test_is_test_passed-no_tests_run_with_header.log') self.assertEqual(0, len(result['sub_groups'])) + def test_nested_json(self): + result = self._json_for('test_is_test_passed-all_passed_nested.log') + self.assertEqual( + {'name': 'example_simple_test', 'status': 'PASS'}, + result["sub_groups"][0]["sub_groups"][0]["test_cases"][0]) + class StrContains(str): def __eq__(self, other): return self in other @@ -380,7 +435,7 @@ class KUnitMainTest(unittest.TestCase): self.assertEqual(e.exception.code, 1) self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 1) self.assertEqual(self.linux_source_mock.run_kernel.call_count, 1) - self.print_mock.assert_any_call(StrContains(' 0 tests run')) + self.print_mock.assert_any_call(StrContains('invalid KTAP input!')) def test_exec_raw_output(self): self.linux_source_mock.run_kernel = mock.Mock(return_value=[]) @@ -388,7 +443,7 @@ class KUnitMainTest(unittest.TestCase): self.assertEqual(self.linux_source_mock.run_kernel.call_count, 1) for call in self.print_mock.call_args_list: self.assertNotEqual(call, mock.call(StrContains('Testing complete.'))) - self.assertNotEqual(call, mock.call(StrContains(' 0 tests run'))) + self.assertNotEqual(call, mock.call(StrContains(' 0 tests run!'))) def test_run_raw_output(self): self.linux_source_mock.run_kernel = mock.Mock(return_value=[]) @@ -397,7 +452,7 @@ class KUnitMainTest(unittest.TestCase): self.assertEqual(self.linux_source_mock.run_kernel.call_count, 1) for call in self.print_mock.call_args_list: self.assertNotEqual(call, mock.call(StrContains('Testing complete.'))) - self.assertNotEqual(call, mock.call(StrContains(' 0 tests run'))) + self.assertNotEqual(call, mock.call(StrContains(' 0 tests run!'))) def test_run_raw_output_kunit(self): self.linux_source_mock.run_kernel = mock.Mock(return_value=[]) diff --git a/tools/testing/kunit/test_data/test_is_test_passed-all_passed_nested.log b/tools/testing/kunit/test_data/test_is_test_passed-all_passed_nested.log new file mode 100644 index 000000000000..9d5b04fe43a6 --- /dev/null +++ b/tools/testing/kunit/test_data/test_is_test_passed-all_passed_nested.log @@ -0,0 +1,34 @@ +TAP version 14 +1..2 + # Subtest: sysctl_test + 1..4 + # sysctl_test_dointvec_null_tbl_data: sysctl_test_dointvec_null_tbl_data passed + ok 1 - sysctl_test_dointvec_null_tbl_data + # Subtest: example + 1..2 + init_suite + # example_simple_test: initializing + # example_simple_test: example_simple_test passed + ok 1 - example_simple_test + # example_mock_test: initializing + # example_mock_test: example_mock_test passed + ok 2 - example_mock_test + kunit example: all tests passed + ok 2 - example + # sysctl_test_dointvec_table_len_is_zero: sysctl_test_dointvec_table_len_is_zero passed + ok 3 - sysctl_test_dointvec_table_len_is_zero + # sysctl_test_dointvec_table_read_but_position_set: sysctl_test_dointvec_table_read_but_position_set passed + ok 4 - sysctl_test_dointvec_table_read_but_position_set +kunit sysctl_test: all tests passed +ok 1 - sysctl_test + # Subtest: example + 1..2 +init_suite + # example_simple_test: initializing + # example_simple_test: example_simple_test passed + ok 1 - example_simple_test + # example_mock_test: initializing + # example_mock_test: example_mock_test passed + ok 2 - example_mock_test +kunit example: all tests passed +ok 2 - example diff --git a/tools/testing/kunit/test_data/test_is_test_passed-kselftest.log b/tools/testing/kunit/test_data/test_is_test_passed-kselftest.log new file mode 100644 index 000000000000..65d3f27feaf2 --- /dev/null +++ b/tools/testing/kunit/test_data/test_is_test_passed-kselftest.log @@ -0,0 +1,14 @@ +TAP version 13 +1..2 +# selftests: membarrier: membarrier_test_single_thread +# TAP version 13 +# 1..2 +# ok 1 sys_membarrier available +# ok 2 sys membarrier invalid command test: command = -1, flags = 0, errno = 22. Failed as expected +ok 1 selftests: membarrier: membarrier_test_single_thread +# selftests: membarrier: membarrier_test_multi_thread +# TAP version 13 +# 1..2 +# ok 1 sys_membarrier available +# ok 2 sys membarrier invalid command test: command = -1, flags = 0, errno = 22. Failed as expected +ok 2 selftests: membarrier: membarrier_test_multi_thread diff --git a/tools/testing/kunit/test_data/test_is_test_passed-missing_plan.log b/tools/testing/kunit/test_data/test_is_test_passed-missing_plan.log new file mode 100644 index 000000000000..5cd17b7f818a --- /dev/null +++ b/tools/testing/kunit/test_data/test_is_test_passed-missing_plan.log @@ -0,0 +1,31 @@ +KTAP version 1 + # Subtest: sysctl_test + # sysctl_test_dointvec_null_tbl_data: sysctl_test_dointvec_null_tbl_data passed + ok 1 - sysctl_test_dointvec_null_tbl_data + # sysctl_test_dointvec_table_maxlen_unset: sysctl_test_dointvec_table_maxlen_unset passed + ok 2 - sysctl_test_dointvec_table_maxlen_unset + # sysctl_test_dointvec_table_len_is_zero: sysctl_test_dointvec_table_len_is_zero passed + ok 3 - sysctl_test_dointvec_table_len_is_zero + # sysctl_test_dointvec_table_read_but_position_set: sysctl_test_dointvec_table_read_but_position_set passed + ok 4 - sysctl_test_dointvec_table_read_but_position_set + # sysctl_test_dointvec_happy_single_positive: sysctl_test_dointvec_happy_single_positive passed + ok 5 - sysctl_test_dointvec_happy_single_positive + # sysctl_test_dointvec_happy_single_negative: sysctl_test_dointvec_happy_single_negative passed + ok 6 - sysctl_test_dointvec_happy_single_negative + # sysctl_test_dointvec_single_less_int_min: sysctl_test_dointvec_single_less_int_min passed + ok 7 - sysctl_test_dointvec_single_less_int_min + # sysctl_test_dointvec_single_greater_int_max: sysctl_test_dointvec_single_greater_int_max passed + ok 8 - sysctl_test_dointvec_single_greater_int_max +kunit sysctl_test: all tests passed +ok 1 - sysctl_test + # Subtest: example + 1..2 +init_suite + # example_simple_test: initializing + # example_simple_test: example_simple_test passed + ok 1 - example_simple_test + # example_mock_test: initializing + # example_mock_test: example_mock_test passed + ok 2 - example_mock_test +kunit example: all tests passed +ok 2 - example diff --git a/tools/testing/kunit/test_data/test_strip_hyphen.log b/tools/testing/kunit/test_data/test_strip_hyphen.log new file mode 100644 index 000000000000..92ac7c24b374 --- /dev/null +++ b/tools/testing/kunit/test_data/test_strip_hyphen.log @@ -0,0 +1,16 @@ +KTAP version 1 +1..2 + # Subtest: sysctl_test + 1..1 + # sysctl_test_dointvec_null_tbl_data: sysctl_test_dointvec_null_tbl_data passed + ok 1 - sysctl_test_dointvec_null_tbl_data +kunit sysctl_test: all tests passed +ok 1 - sysctl_test + # Subtest: example + 1..1 +init_suite + # example_simple_test: initializing + # example_simple_test: example_simple_test passed + ok 1 example_simple_test +kunit example: all tests passed +ok 2 example base-commit: 28a8b037484044d79f3b8d409a076a925ac75f8d -- 2.33.0.882.g93a45727a2-goog

3 years, 7 months

2
1
0 0

[REGRESSION] lkft kselftest for next-20211006

by lkft＠linaro.org

## Build * kernel: 5.15.0-rc4 * git: https://gitlab.com/Linaro/lkft/mirrors/next/linux-next * git branch: master * git commit: 0dda5ee37364eb58a960d7a031aebd7fa3e26384 * git describe: next-20211006 * test details: https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20211006 ## Regressions (compared to next-20211001) * dragonboard-410c, kselftest-net - net.ip_defrag.sh * dragonboard-410c, kselftest-rseq - rseq.basic_percpu_ops_test - rseq.basic_test - rseq.param_test - rseq.param_test_benchmark - rseq.param_test_compare_twice * qemu-arm64-gic-version3, kselftest-timers - timers.rtcpie * qemu_arm64, kselftest-timers - timers.rtcpie * qemu_i386, kselftest-mqueue - mqueue.mq_perf_tests * qemu_i386, kselftest-rseq - rseq.basic_percpu_ops_test - rseq.basic_test - rseq.param_test - rseq.param_test_benchmark - rseq.param_test_compare_twice - rseq.run_param_test.sh * qemu_x86_64, kselftest-kvm - kvm.steal_time * qemu_x86_64, kselftest-mqueue - mqueue.mq_perf_tests * qemu_x86_64, kselftest-rseq - rseq.basic_percpu_ops_test - rseq.basic_test - rseq.param_test - rseq.param_test_benchmark - rseq.param_test_compare_twice - rseq.run_param_test.sh * qemu_x86_64, kselftest-rtc - rtc.rtctest * x86, kselftest-lkdtm - lkdtm.SLAB_FREE_DOUBLE.sh Reported-by: Linux Kernel Functional Testing <lkft(a)linaro.org> ## Fixes (compared to next-20211001) * qemu-arm64-mte, kselftest-timers - timers.rtcpie ## Test result summary total: 4010, pass: 2387, fail: 331, skip: 1292, xfail: 0 ## Build Summary ## Test suites summary * kselftest-android * kselftest-arm64 * kselftest-arm64/arm64.btitest.bti_c_func * kselftest-arm64/arm64.btitest.bti_j_func * kselftest-arm64/arm64.btitest.bti_jc_func * kselftest-arm64/arm64.btitest.bti_none_func * kselftest-arm64/arm64.btitest.nohint_func * kselftest-arm64/arm64.btitest.paciasp_func * kselftest-arm64/arm64.nobtitest.bti_c_func * kselftest-arm64/arm64.nobtitest.bti_j_func * kselftest-arm64/arm64.nobtitest.bti_jc_func * kselftest-arm64/arm64.nobtitest.bti_none_func * kselftest-arm64/arm64.nobtitest.nohint_func * kselftest-arm64/arm64.nobtitest.paciasp_func * kselftest-bpf * kselftest-breakpoints * kselftest-capabilities * kselftest-cgroup * kselftest-clone3 * kselftest-core * kselftest-cpu-hotplug * kselftest-cpufreq * kselftest-drivers * kselftest-efivarfs * kselftest-filesystems * kselftest-firmware * kselftest-fpu * kselftest-futex * kselftest-gpio * kselftest-intel_pstate * kselftest-ipc * kselftest-ir * kselftest-kcmp * kselftest-kexec * kselftest-kvm * kselftest-lib * kselftest-livepatch * kselftest-lkdtm * kselftest-membarrier * kselftest-memfd * kselftest-memory-hotplug * kselftest-mincore * kselftest-mount * kselftest-mqueue * kselftest-net * kselftest-netfilter * kselftest-nsfs * kselftest-openat2 * kselftest-pid_namespace * kselftest-pidfd * kselftest-proc * kselftest-pstore * kselftest-ptrace * kselftest-rseq * kselftest-rtc * kselftest-seccomp * kselftest-sigaltstack * kselftest-size * kselftest-splice * kselftest-static_keys * kselftest-sync * kselftest-sysctl * kselftest-tc-testing * kselftest-timens * kselftest-timers * kselftest-tmpfs * kselftest-tpm2 * kselftest-user * kselftest-vm * kselftest-x86 * kselftest-zram -- Linaro LKFT https://lkft.linaro.org

3 years, 7 months

1
0
0 0

[PATCH] kunit: Report test parameter results as (K)TAP subtests

by David Gow

Currently, the results for individial parameters in a parameterised test are simply output as (K)TAP diagnostic lines. However, the plan was always[1] to make these (K)TAP subtests when kunit_tool supported them. With [2], these are now supported. (v5 will print out an error about the missing plan line, but this can safely be ignored, and will hopefully be changed). As a result, individual test parameter results are parsed, displayed in the formatted results, and counted for test statistics. [1]: https://lore.kernel.org/linux-kselftest/CABVgOSnJAgWvTTABaF082LuYjAoAWzrBsy… [2]: https://lore.kernel.org/linux-kselftest/20211006001447.20919-1-dlatypov@goo… Signed-off-by: David Gow <davidgow(a)google.com> --- lib/kunit/test.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/lib/kunit/test.c b/lib/kunit/test.c index f246b847024e..02a9fdadcae2 100644 --- a/lib/kunit/test.c +++ b/lib/kunit/test.c @@ -508,6 +508,8 @@ int kunit_run_tests(struct kunit_suite *suite) /* Get initial param. */ param_desc[0] = '\0'; test.param_value = test_case->generate_params(NULL, param_desc); + kunit_log(KERN_INFO, &test, KUNIT_SUBTEST_INDENT KUNIT_SUBTEST_INDENT + "# Subtest: %s", test_case->name); } do { @@ -520,9 +522,8 @@ int kunit_run_tests(struct kunit_suite *suite) } kunit_log(KERN_INFO, &test, - KUNIT_SUBTEST_INDENT - "# %s: %s %d - %s", - test_case->name, + KUNIT_SUBTEST_INDENT KUNIT_SUBTEST_INDENT + "%s %d - %s", kunit_status_to_ok_not_ok(test.status), test.param_index + 1, param_desc); -- 2.33.0.800.g4c38ced690-goog

3 years, 7 months

3
2
0 0

[PATCH] kunit: Reset suite count after running tests

by David Gow

There are some KUnit tests (KFENCE, Thunderbolt) which, for various reasons, do not use the kunit_test_suite() macro and end up running before the KUnit executor runs its tests. This means that their results are printed separately, and they aren't included in the suite count used by the executor. This causes the executor output to be invalid TAP, however, as the suite numbers used are no-longer 1-based, and don't match the test plan. kunit_tool, therefore, prints a large number of warnings. While it'd be nice to fix the tests to run in the executor, in the meantime, reset the suite counter to 1 in __kunit_test_suites_exit. Not only does this fix the executor, it means that if there are multiple calls to __kunit_test_suites_init() across different tests, they'll each get their own numbering. kunit_tool likes this better: even if it's lacking the results for those tests which don't use the executor (due to the lack of TAP header), the output for the other tests is valid. Signed-off-by: David Gow <davidgow(a)google.com> --- lib/kunit/test.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/lib/kunit/test.c b/lib/kunit/test.c index f246b847024e..3bd741e50a2d 100644 --- a/lib/kunit/test.c +++ b/lib/kunit/test.c @@ -190,10 +190,10 @@ enum kunit_status kunit_suite_has_succeeded(struct kunit_suite *suite) } EXPORT_SYMBOL_GPL(kunit_suite_has_succeeded); +static size_t kunit_suite_counter = 1; + static void kunit_print_subtest_end(struct kunit_suite *suite) { - static size_t kunit_suite_counter = 1; - kunit_print_ok_not_ok((void *)suite, false, kunit_suite_has_succeeded(suite), kunit_suite_counter++, @@ -583,6 +583,8 @@ void __kunit_test_suites_exit(struct kunit_suite **suites) for (i = 0; suites[i] != NULL; i++) kunit_exit_suite(suites[i]); + + kunit_suite_counter = 1; } EXPORT_SYMBOL_GPL(__kunit_test_suites_exit); -- 2.33.0.800.g4c38ced690-goog

3 years, 7 months

2
1
0 0

[REGRESSION] lkft kselftest for next-20211005

by lkft＠linaro.org

## Build * kernel: 5.15.0-rc4 * git: ['https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git', 'https://gitlab.com/Linaro/lkft/mirrors/next/linux-next'] * git branch: master * git commit: 29616f67fcbd6b26242440372a3b0f0a085109c6 * git describe: next-20211005 * test details: https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20211005 ## Regressions (compared to next-20211001) * dragonboard-410c, kselftest-net - net.udpgso_bench.sh * qemu_arm, kselftest-rtc - rtc.rtctest * qemu_i386, kselftest-mqueue - mqueue.mq_perf_tests Reported-by: Linux Kernel Functional Testing <lkft(a)linaro.org> ## Fixes (compared to next-20211001) * qemu-arm64-mte, kselftest-timers - timers.rtcpie * x86, kselftest-rtc - rtc.rtctest ## Test result summary total: 4329, pass: 2399, fail: 349, skip: 1581, xfail: 0 ## Build Summary ## Test suites summary * kselftest-android * kselftest-arm64 * kselftest-arm64/arm64.btitest.bti_c_func * kselftest-arm64/arm64.btitest.bti_j_func * kselftest-arm64/arm64.btitest.bti_jc_func * kselftest-arm64/arm64.btitest.bti_none_func * kselftest-arm64/arm64.btitest.nohint_func * kselftest-arm64/arm64.btitest.paciasp_func * kselftest-arm64/arm64.nobtitest.bti_c_func * kselftest-arm64/arm64.nobtitest.bti_j_func * kselftest-arm64/arm64.nobtitest.bti_jc_func * kselftest-arm64/arm64.nobtitest.bti_none_func * kselftest-arm64/arm64.nobtitest.nohint_func * kselftest-arm64/arm64.nobtitest.paciasp_func * kselftest-bpf * kselftest-breakpoints * kselftest-capabilities * kselftest-cgroup * kselftest-clone3 * kselftest-core * kselftest-cpu-hotplug * kselftest-cpufreq * kselftest-drivers * kselftest-efivarfs * kselftest-filesystems * kselftest-firmware * kselftest-fpu * kselftest-futex * kselftest-gpio * kselftest-intel_pstate * kselftest-ipc * kselftest-ir * kselftest-kcmp * kselftest-kexec * kselftest-kvm * kselftest-lib * kselftest-livepatch * kselftest-lkdtm * kselftest-membarrier * kselftest-memfd * kselftest-memory-hotplug * kselftest-mincore * kselftest-mount * kselftest-mqueue * kselftest-net * kselftest-netfilter * kselftest-nsfs * kselftest-openat2 * kselftest-pid_namespace * kselftest-pidfd * kselftest-proc * kselftest-pstore * kselftest-ptrace * kselftest-rseq * kselftest-rtc * kselftest-seccomp * kselftest-sigaltstack * kselftest-size * kselftest-splice * kselftest-static_keys * kselftest-sync * kselftest-sysctl * kselftest-tc-testing * kselftest-timens * kselftest-timers * kselftest-tmpfs * kselftest-tpm2 * kselftest-user * kselftest-vm * kselftest-x86 * kselftest-zram -- Linaro LKFT https://lkft.linaro.org

3 years, 7 months

1
0
0 0

[PATCH] kunit: fix kernel-doc warnings due to mismatched arg names

by Daniel Latypov

Commit 7122debb4367 ("kunit: introduce kunit_kmalloc_array/kunit_kcalloc() helpers") added new functions but called last arg `flags`, unlike the existing code that used `gfp`. This only is an issue in test.h, test.c still used `gfp`. But the documentation was copy-pasted with the old names, leading to kernel-doc warnings. Do s/flags/gfp to make the names consistent and fix the warnings. Fixes: 7122debb4367 ("kunit: introduce kunit_kmalloc_array/kunit_kcalloc() helpers") Reported-by: Randy Dunlap <rdunlap(a)infradead.org> Signed-off-by: Daniel Latypov <dlatypov(a)google.com> --- include/kunit/test.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/include/kunit/test.h b/include/kunit/test.h index 24b40e5c160b..018e776a34b9 100644 --- a/include/kunit/test.h +++ b/include/kunit/test.h @@ -613,7 +613,7 @@ void kunit_remove_resource(struct kunit *test, struct kunit_resource *res); * and is automatically cleaned up after the test case concludes. See &struct * kunit_resource for more information. */ -void *kunit_kmalloc_array(struct kunit *test, size_t n, size_t size, gfp_t flags); +void *kunit_kmalloc_array(struct kunit *test, size_t n, size_t size, gfp_t gfp); /** * kunit_kmalloc() - Like kmalloc() except the allocation is *test managed*. @@ -657,9 +657,9 @@ static inline void *kunit_kzalloc(struct kunit *test, size_t size, gfp_t gfp) * * See kcalloc() and kunit_kmalloc_array() for more information. */ -static inline void *kunit_kcalloc(struct kunit *test, size_t n, size_t size, gfp_t flags) +static inline void *kunit_kcalloc(struct kunit *test, size_t n, size_t size, gfp_t gfp) { - return kunit_kmalloc_array(test, n, size, flags | __GFP_ZERO); + return kunit_kmalloc_array(test, n, size, gfp | __GFP_ZERO); } void kunit_cleanup(struct kunit *test); base-commit: 3b29021ddd10cfb6b2565c623595bd3b02036f33 -- 2.33.0.882.g93a45727a2-goog

3 years, 7 months

3
2
0 0

[PATCH v2] kunit: tool: yield output from run_kernel in real time

by Daniel Latypov

Currently, `run_kernel()` dumps all the kernel output to a file (.kunit/test.log) and then opens the file and yields it to callers. This made it easier to respect the requested timeout, if any. But it means that we can't yield the results in real time, either to the parser or to stdout (if --raw_output is set). This change spins up a background thread to enforce the timeout, which allows us to yield the kernel output in real time, while also copying it to the .kunit/test.log file. It's also careful to ensure that the .kunit/test.log file is complete, even in the kunit_parser throws an exception/otherwise doesn't consume every line, see the new `finally` block and unit test. For example: $ ./tools/testing/kunit/kunit.py run --arch=x86_64 --raw_output <configure + build steps> ... <can now see output from QEMU in real time> This does not currently have a visible effect when --raw_output is not passed, as kunit_parser.py currently only outputs everything at the end. But that could change, and this patch is a necessary step towards showing parsed test results in real time. Signed-off-by: Daniel Latypov <dlatypov(a)google.com> --- v1 -> v2: restore stdin=subprocess.PIPE to avoid a UML (?) bug. https://lore.kernel.org/linux-kselftest/CABVgOSnXBtCHEMDYYqrbXWvKcrSbY9BXP4… --- tools/testing/kunit/kunit_kernel.py | 75 +++++++++++++++----------- tools/testing/kunit/kunit_tool_test.py | 17 ++++++ 2 files changed, 62 insertions(+), 30 deletions(-) diff --git a/tools/testing/kunit/kunit_kernel.py b/tools/testing/kunit/kunit_kernel.py index 2c6f916ccbaf..450f1c386e01 100644 --- a/tools/testing/kunit/kunit_kernel.py +++ b/tools/testing/kunit/kunit_kernel.py @@ -12,7 +12,8 @@ import subprocess import os import shutil import signal -from typing import Iterator, Optional, Tuple +import threading +from typing import Iterator, List, Optional, Tuple from contextlib import ExitStack @@ -103,8 +104,8 @@ class LinuxSourceTreeOperations(object): if stderr: # likely only due to build warnings print(stderr.decode()) - def run(self, params, timeout, build_dir, outfile) -> None: - pass + def start(self, params: List[str], build_dir: str) -> subprocess.Popen: + raise RuntimeError('not implemented!') class LinuxSourceTreeOperationsQemu(LinuxSourceTreeOperations): @@ -123,7 +124,7 @@ class LinuxSourceTreeOperationsQemu(LinuxSourceTreeOperations): kconfig.parse_from_string(self._kconfig) base_kunitconfig.merge_in_entries(kconfig) - def run(self, params, timeout, build_dir, outfile): + def start(self, params: List[str], build_dir: str) -> subprocess.Popen: kernel_path = os.path.join(build_dir, self._kernel_path) qemu_command = ['qemu-system-' + self._qemu_arch, '-nodefaults', @@ -134,18 +135,11 @@ class LinuxSourceTreeOperationsQemu(LinuxSourceTreeOperations): '-nographic', '-serial stdio'] + self._extra_qemu_params print('Running tests with:\n$', ' '.join(qemu_command)) - with open(outfile, 'w') as output: - process = subprocess.Popen(' '.join(qemu_command), - stdin=subprocess.PIPE, - stdout=output, - stderr=subprocess.STDOUT, - text=True, shell=True) - try: - process.wait(timeout=timeout) - except Exception as e: - print(e) - process.terminate() - return process + return subprocess.Popen(' '.join(qemu_command), + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + text=True, shell=True) class LinuxSourceTreeOperationsUml(LinuxSourceTreeOperations): """An abstraction over command line operations performed on a source tree.""" @@ -175,17 +169,14 @@ class LinuxSourceTreeOperationsUml(LinuxSourceTreeOperations): kunit_parser.print_with_timestamp( 'Starting Kernel with all configs takes a few minutes...') - def run(self, params, timeout, build_dir, outfile): + def start(self, params: List[str], build_dir: str) -> subprocess.Popen: """Runs the Linux UML binary. Must be named 'linux'.""" linux_bin = get_file_path(build_dir, 'linux') - outfile = get_outfile_path(build_dir) - with open(outfile, 'w') as output: - process = subprocess.Popen([linux_bin] + params, - stdin=subprocess.PIPE, - stdout=output, - stderr=subprocess.STDOUT, - text=True) - process.wait(timeout) + return subprocess.Popen([linux_bin] + params, + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + text=True) def get_kconfig_path(build_dir) -> str: return get_file_path(build_dir, KCONFIG_PATH) @@ -330,12 +321,36 @@ class LinuxSourceTree(object): args.extend(['mem=1G', 'console=tty', 'kunit_shutdown=halt']) if filter_glob: args.append('kunit.filter_glob='+filter_glob) - outfile = get_outfile_path(build_dir) - self._ops.run(args, timeout, build_dir, outfile) - subprocess.call(['stty', 'sane']) - with open(outfile, 'r') as file: - for line in file: + + process = self._ops.start(args, build_dir) + assert process.stdout is not None # tell mypy it's set + + # Enforce the timeout in a background thread. + def _wait_proc(): + try: + process.wait(timeout=timeout) + except Exception as e: + print(e) + process.terminate() + process.wait() + waiter = threading.Thread(target=_wait_proc) + waiter.start() + + output = open(get_outfile_path(build_dir), 'w') + try: + # Tee the output to the file and to our caller in real time. + for line in process.stdout: + output.write(line) yield line + # This runs even if our caller doesn't consume every line. + finally: + # Flush any leftover output to the file + output.write(process.stdout.read()) + output.close() + process.stdout.close() + + waiter.join() + subprocess.call(['stty', 'sane']) def signal_handler(self, sig, frame) -> None: logging.error('Build interruption occurred. Cleaning console.') diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index 619c4554cbff..f9a7398a9584 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -14,6 +14,7 @@ import tempfile, shutil # Handling test_tmpdir import itertools import json import signal +import subprocess import os import kunit_config @@ -291,6 +292,22 @@ class LinuxSourceTreeTest(unittest.TestCase): pass tree = kunit_kernel.LinuxSourceTree('', kunitconfig_path=dir) + def test_run_kernel_hits_exception(self): + def fake_start(unused_args, unused_build_dir): + return subprocess.Popen(['echo "hi\nbye"'], shell=True, text=True, stdout=subprocess.PIPE) + + with tempfile.TemporaryDirectory('') as build_dir: + tree = kunit_kernel.LinuxSourceTree(build_dir, load_config=False) + mock.patch.object(tree._ops, 'start', side_effect=fake_start).start() + + with self.assertRaises(ValueError): + for line in tree.run_kernel(build_dir=build_dir): + self.assertEqual(line, 'hi\n') + raise ValueError('uh oh, did not read all output') + + with open(kunit_kernel.get_outfile_path(build_dir), 'rt') as outfile: + self.assertEqual(outfile.read(), 'hi\nbye\n', msg='Missing some output') + # TODO: add more test cases. base-commit: 3b29021ddd10cfb6b2565c623595bd3b02036f33 -- 2.33.0.800.g4c38ced690-goog

3 years, 7 months

3
2
0 0

[PATCH v4 0/3] kunit: allow running test suites/cases individually

by Daniel Latypov

Allow running each suite or each test case alone per kernel boot. The motivation for this is to debug "test hermeticity" issues. This new --run_isolated flag would be a good first step to try and narrow down root causes. Context: sometimes tests pass/fail depending on what ran before them. Memory corruption errors in particular might only cause noticeable issues later on. But you can also have the opposite, where "fixing" one test causes another to start failing. Usage: $ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit --run_isolated=suite $ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit --run_isolated=test $ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit --run_isolated=test example The last one would provide output like ======== [PASSED] example ======== [PASSED] example_simple_test ============================================================ Testing complete. 1 tests run. 0 failed. 0 crashed. 0 skipped. Starting KUnit Kernel (2/3)... ============================================================ ======== [SKIPPED] example ======== [SKIPPED] example_skip_test # SKIP this test should be skipped ============================================================ Testing complete. 1 tests run. 0 failed. 0 crashed. 1 skipped. Starting KUnit Kernel (3/3)... ============================================================ ======== [SKIPPED] example ======== [SKIPPED] example_mark_skipped_test # SKIP this test should be skipped ============================================================ Testing complete. 1 tests run. 0 failed. 0 crashed. 1 skipped. See the last patch's description for a bit more detail. Meta: The first patch is from another series with just a reworded commit message, https://lore.kernel.org/linux-kselftest/20210805235145.2528054-2-dlatypov@g… This patch series is based on Shuah's kunit branch: https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git/?… Changes: v1 -> v2: rebase onto Shuah's kunit branch, fix missing code in patch 1. v2 -> v3: fix mypy errors, drop test plan from output, fix pre-existing bug where kunit was not actually tracking test execution time (new patch 3). v3 -> v4: attempt to filter out non-KUnit dmesg output when getting list of test names, using this regex: ^[^\s.]+\.[^\s.]+$ Daniel Latypov (4): kunit: add 'kunit.action' param to allow listing out tests kunit: tool: factor exec + parse steps into a function kunit: tool: actually track how long it took to run tests kunit: tool: support running each suite/test separately lib/kunit/executor.c | 45 ++++++++- tools/testing/kunit/kunit.py | 134 +++++++++++++++++-------- tools/testing/kunit/kunit_tool_test.py | 40 ++++++++ 3 files changed, 173 insertions(+), 46 deletions(-) base-commit: 3b29021ddd10cfb6b2565c623595bd3b02036f33 -- 2.33.0.800.g4c38ced690-goog

3 years, 7 months

3
11
0 0

[PATCH v5] kunit: tool: improve compatibility of kunit_parser with KTAP specification

by Daniel Latypov

From: Rae Moar <rmoar(a)google.com> Update to kunit_parser to improve compatibility with KTAP specification including arbitrarily nested tests. Patch accomplishes three major changes: - Use a general Test object to represent all tests rather than TestCase and TestSuite objects. This allows for easier implementation of arbitrary levels of nested tests and promotes the idea that both test suites and test cases are tests. - Print errors incrementally rather than all at once after the parsing finishes to maximize information given to the user in the case of the parser given invalid input and to increase the helpfulness of the timestamps given during printing. Note that kunit.py parse does not print incrementally yet. However, this fix brings us closer to this feature. - Increase compatibility for different formats of input. Arbitrary levels of nested tests supported. Also, test cases and test suites are now supported to be present on the same level of testing. This patch now implements the KTAP specification as described here: https://lore.kernel.org/linux-kselftest/CA+GJov6tdjvY9x12JsJT14qn6c7NViJxqa…. This patch adjusts the kunit_tool_test.py file to check for the correct outputs from the new parser and adds a new test to check the parsing for a KTAP result log with correct format for multiple nested subtests (test_is_test_passed-all_passed_nested.log). This patch also alters the kunit_json.py file to allow for arbitrarily nested tests. Signed-off-by: Rae Moar <rmoar(a)google.com> Reviewed-by: Brendan Higgins <brendanhiggins(a)google.com> --- NOTE: this patch is now applied on top of https://lore.kernel.org/linux-kselftest/20210930222048.1692635-5-dlatypov@g… to resolve a conflict. Change log from v3,4: https://lore.kernel.org/linux-kselftest/20210901190623.315736-1-rmoar@googl… - Move test_kselftest_nested from LinuxSourceTreeTest => KUnitParserTest. - Resolve conflict with hermetic testing patches. - max_status is no longer defined, so we need to use the TestCounts type now. And to keep --raw_output working, we need to set this to SUCCESS to avoid the default assumption that the kernel crashed. Ignore v4, was accidentally based on v2. Change log from v2: https://lore.kernel.org/linux-kselftest/20210826195505.3066755-1-rmoar@goog… - Fixes bug of type disagreement in kunit_json.py for build_dir - Removes raw_output() - Changes docstrings in kunit_parser.py (class docstring, LineStream docstrings, add_error(), total(), get_status(), all parsing methods) - Fixes bug of not printing diagnostic log in the case of end of lines - Sets default status of all tests to TEST_CRASHED - Adds and prints empty tests with crashed status in case of missing tests - Prints 'subtest' in instance of 1 subtest instead of 'subtests' - Includes checking for 'BUG:' message in search of crash messages in log (note that parse_crash_in_log method could be removed but would require deleting tests in kunit_tool_test.py that include the crash message that is no longer used. If removed, parser would still print log in cases of test crashed or failure, which would now include missing subtests) - Fixes bug of including directives (other than SKIP) in test name when matching name in result line for subtests Change log from v1: https://lore.kernel.org/linux-kselftest/20210820200032.2178134-1-rmoar@goog… - Rebase onto kselftest/kunit branch - Add tests to kunit_tool_test.py to check parser is correctly stripping hyphen, producing correct json objects with nested tests, correctly passing kselftest TAP output, and correctly deals with missing test plan. - Fix bug to correctly match test name in instance of a missing test plan. - Fix bug in kunit_tool_test.py pointed out by Daniel where it was not correctly checking for a proper match to the '0 tests run!' error message. Reverts changes back to original. - A few minor changes to commit message using Daniel's comments. - Change docstrings using Daniel's comments to reduce: - Shortens some docstrings to be one-line or just description if it is self explanatory. - Remove explicit respecification of types of parameters and returns because this is already specified in the function annoations. However, some descriptions of the parameters and returns remain and some contain the type for context. Additionally, the types of public attributes of classes remain. - Remove any documentation of 'Return: None' - Remove docstrings of helper methods within other methods --- tools/testing/kunit/kunit.py | 19 +- tools/testing/kunit/kunit_json.py | 56 +- tools/testing/kunit/kunit_parser.py | 1024 ++++++++++++----- tools/testing/kunit/kunit_tool_test.py | 133 ++- .../test_is_test_passed-all_passed_nested.log | 34 + .../test_is_test_passed-kselftest.log | 14 + .../test_is_test_passed-missing_plan.log | 31 + .../kunit/test_data/test_strip_hyphen.log | 16 + 8 files changed, 943 insertions(+), 384 deletions(-) create mode 100644 tools/testing/kunit/test_data/test_is_test_passed-all_passed_nested.log create mode 100644 tools/testing/kunit/test_data/test_is_test_passed-kselftest.log create mode 100644 tools/testing/kunit/test_data/test_is_test_passed-missing_plan.log create mode 100644 tools/testing/kunit/test_data/test_strip_hyphen.log diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py index 8c7e8c7b2c97..6aef16478efb 100755 --- a/tools/testing/kunit/kunit.py +++ b/tools/testing/kunit/kunit.py @@ -136,7 +136,7 @@ def exec_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest, test_glob = request.filter_glob.split('.', maxsplit=2)[1] filter_globs = [g + '.'+ test_glob for g in filter_globs] - overall_status = kunit_parser.TestStatus.SUCCESS + test_counts = kunit_parser.TestCounts() exec_time = 0.0 for i, filter_glob in enumerate(filter_globs): kunit_parser.print_with_timestamp('Starting KUnit Kernel ({}/{})...'.format(i+1, len(filter_globs))) @@ -155,18 +155,29 @@ def exec_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest, test_end = time.time() exec_time += test_end - test_start - overall_status = kunit_parser.max_status(overall_status, result.status) + test_counts.add_subtest_counts(result.result.test.counts) - return KunitResult(status=result.status, result=result.result, elapsed_time=exec_time) + kunit_status = _map_to_overall_status(test_counts.get_status()) + return KunitResult(status=kunit_status, result=result.result, elapsed_time=exec_time) + +def _map_to_overall_status(test_status: kunit_parser.TestStatus) -> KunitStatus: + if test_status in (kunit_parser.TestStatus.SUCCESS, kunit_parser.TestStatus.SKIPPED): + return KunitStatus.SUCCESS + else: + return KunitStatus.TEST_FAILURE def parse_tests(request: KunitParseRequest, input_data: Iterable[str]) -> KunitResult: parse_start = time.time() test_result = kunit_parser.TestResult(kunit_parser.TestStatus.SUCCESS, - [], + kunit_parser.Test(), 'Tests not Parsed.') if request.raw_output: + # Treat unparsed results as one passing test. + test_result.test.status = kunit_parser.TestStatus.SUCCESS + test_result.test.counts.passed = 1 + output: Iterable[str] = input_data if request.raw_output == 'all': pass diff --git a/tools/testing/kunit/kunit_json.py b/tools/testing/kunit/kunit_json.py index f5cca5c38cac..746bec72b9ac 100644 --- a/tools/testing/kunit/kunit_json.py +++ b/tools/testing/kunit/kunit_json.py @@ -11,47 +11,47 @@ import os import kunit_parser -from kunit_parser import TestStatus - -def get_json_result(test_result, def_config, build_dir, json_path) -> str: - sub_groups = [] - - # Each test suite is mapped to a KernelCI sub_group - for test_suite in test_result.suites: - sub_group = { - "name": test_suite.name, - "arch": "UM", - "defconfig": def_config, - "build_environment": build_dir, - "test_cases": [], - "lab_name": None, - "kernel": None, - "job": None, - "git_branch": "kselftest", - } - test_cases = [] - # TODO: Add attachments attribute in test_case with detailed - # failure message, see https://api.kernelci.org/schema-test-case.html#get - for case in test_suite.cases: - test_case = {"name": case.name, "status": "FAIL"} - if case.status == TestStatus.SUCCESS: +from kunit_parser import Test, TestResult, TestStatus +from typing import Any, Dict, Optional + +JsonObj = Dict[str, Any] + +def _get_group_json(test: Test, def_config: str, + build_dir: Optional[str]) -> JsonObj: + sub_groups = [] # List[JsonObj] + test_cases = [] # List[JsonObj] + + for subtest in test.subtests: + if len(subtest.subtests): + sub_group = _get_group_json(subtest, def_config, + build_dir) + sub_groups.append(sub_group) + else: + test_case = {"name": subtest.name, "status": "FAIL"} + if subtest.status == TestStatus.SUCCESS: test_case["status"] = "PASS" - elif case.status == TestStatus.TEST_CRASHED: + elif subtest.status == TestStatus.TEST_CRASHED: test_case["status"] = "ERROR" test_cases.append(test_case) - sub_group["test_cases"] = test_cases - sub_groups.append(sub_group) + test_group = { - "name": "KUnit Test Group", + "name": test.name, "arch": "UM", "defconfig": def_config, "build_environment": build_dir, "sub_groups": sub_groups, + "test_cases": test_cases, "lab_name": None, "kernel": None, "job": None, "git_branch": "kselftest", } + return test_group + +def get_json_result(test_result: TestResult, def_config: str, + build_dir: Optional[str], json_path: str) -> str: + test_group = _get_group_json(test_result.test, def_config, build_dir) + test_group["name"] = "KUnit Test Group" json_obj = json.dumps(test_group, indent=4) if json_path != 'stdout': with open(json_path, 'w') as result_path: diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py index 6310a641b151..db796580dc5f 100644 --- a/tools/testing/kunit/kunit_parser.py +++ b/tools/testing/kunit/kunit_parser.py @@ -1,11 +1,15 @@ # SPDX-License-Identifier: GPL-2.0 # -# Parses test results from a kernel dmesg log. +# Parses KTAP test results from a kernel dmesg log and incrementally prints +# results with reader-friendly format. Stores and returns test results in a +# Test object. # # Copyright (C) 2019, Google LLC. # Author: Felix Guo <felixguoxiuping(a)gmail.com> # Author: Brendan Higgins <brendanhiggins(a)google.com> +# Author: Rae Moar <rmoar(a)google.com> +from __future__ import annotations import re from collections import namedtuple @@ -14,33 +18,52 @@ from enum import Enum, auto from functools import reduce from typing import Iterable, Iterator, List, Optional, Tuple -TestResult = namedtuple('TestResult', ['status','suites','log']) - -class TestSuite(object): +TestResult = namedtuple('TestResult', ['status','test','log']) + +class Test(object): + """ + A class to represent a test parsed from KTAP results. All KTAP + results within a test log are stored in a main Test object as + subtests. + + Attributes: + status : TestStatus - status of the test + name : str - name of the test + expected_count : int - expected number of subtests (0 if single + test case and None if unknown expected number of subtests) + subtests : List[Test] - list of subtests + log : List[str] - log of KTAP lines that correspond to the test + counts : TestCounts - counts of the test statuses and errors of + subtests or of the test itself if the test is a single + test case. + """ def __init__(self) -> None: - self.status = TestStatus.SUCCESS - self.name = '' - self.cases = [] # type: List[TestCase] - - def __str__(self) -> str: - return 'TestSuite(' + str(self.status) + ',' + self.name + ',' + str(self.cases) + ')' - - def __repr__(self) -> str: - return str(self) - -class TestCase(object): - def __init__(self) -> None: - self.status = TestStatus.SUCCESS + """Creates Test object with default attributes.""" + self.status = TestStatus.TEST_CRASHED self.name = '' + self.expected_count = 0 # type: Optional[int] + self.subtests = [] # type: List[Test] self.log = [] # type: List[str] + self.counts = TestCounts() def __str__(self) -> str: - return 'TestCase(' + str(self.status) + ',' + self.name + ',' + str(self.log) + ')' + """Returns string representation of a Test class object.""" + return ('Test(' + str(self.status) + ', ' + self.name + + ', ' + str(self.expected_count) + ', ' + + str(self.subtests) + ', ' + str(self.log) + ', ' + + str(self.counts) + ')') def __repr__(self) -> str: + """Returns string representation of a Test class object.""" return str(self) + def add_error(self, error_message: str) -> None: + """Records an error that occurred while parsing this test.""" + self.counts.errors += 1 + print_error('Test ' + self.name + ': ' + error_message) + class TestStatus(Enum): + """An enumeration class to represent the status of a test.""" SUCCESS = auto() FAILURE = auto() SKIPPED = auto() @@ -48,381 +71,756 @@ class TestStatus(Enum): NO_TESTS = auto() FAILURE_TO_PARSE_TESTS = auto() +class TestCounts: + """ + Tracks the counts of statuses of all test cases and any errors within + a Test. + + Attributes: + passed : int - the number of tests that have passed + failed : int - the number of tests that have failed + crashed : int - the number of tests that have crashed + skipped : int - the number of tests that have skipped + errors : int - the number of errors in the test and subtests + """ + def __init__(self): + """Creates TestCounts object with counts of all test + statuses and test errors set to 0. + """ + self.passed = 0 + self.failed = 0 + self.crashed = 0 + self.skipped = 0 + self.errors = 0 + + def __str__(self) -> str: + """Returns the string representation of a TestCounts object. + """ + return ('Passed: ' + str(self.passed) + + ', Failed: ' + str(self.failed) + + ', Crashed: ' + str(self.crashed) + + ', Skipped: ' + str(self.skipped) + + ', Errors: ' + str(self.errors)) + + def total(self) -> int: + """Returns the total number of test cases within a test + object, where a test case is a test with no subtests. + """ + return (self.passed + self.failed + self.crashed + + self.skipped) + + def add_subtest_counts(self, counts: TestCounts) -> None: + """ + Adds the counts of another TestCounts object to the current + TestCounts object. Used to add the counts of a subtest to the + parent test. + + Parameters: + counts - a different TestCounts object whose counts + will be added to the counts of the TestCounts object + """ + self.passed += counts.passed + self.failed += counts.failed + self.crashed += counts.crashed + self.skipped += counts.skipped + self.errors += counts.errors + + def get_status(self) -> TestStatus: + """Returns the aggregated status of a Test using test + counts. + """ + if self.total() == 0: + return TestStatus.NO_TESTS + elif self.crashed: + # If one of the subtests crash, the expected status + # of the Test is crashed. + return TestStatus.TEST_CRASHED + elif self.failed: + # Otherwise if one of the subtests fail, the + # expected status of the Test is failed. + return TestStatus.FAILURE + elif self.passed: + # Otherwise if one of the subtests pass, the + # expected status of the Test is passed. + return TestStatus.SUCCESS + else: + # Finally, if none of the subtests have failed, + # crashed, or passed, the expected status of the + # Test is skipped. + return TestStatus.SKIPPED + + def add_status(self, status: TestStatus) -> None: + """ + Increments count of inputted status. + + Parameters: + status - status to be added to the TestCounts object + """ + if status == TestStatus.SUCCESS or \ + status == TestStatus.NO_TESTS: + # if status is NO_TESTS the most appropriate + # attribute to increment is passed because + # the test did not fail, crash or get skipped. + self.passed += 1 + elif status == TestStatus.FAILURE: + self.failed += 1 + elif status == TestStatus.SKIPPED: + self.skipped += 1 + else: + self.crashed += 1 + class LineStream: - """Provides a peek()/pop() interface over an iterator of (line#, text).""" + """ + A class to represent the lines of kernel output. + Provides a peek()/pop() interface over an iterator of + (line#, text). + """ _lines: Iterator[Tuple[int, str]] _next: Tuple[int, str] _done: bool def __init__(self, lines: Iterator[Tuple[int, str]]): + """Creates a new LineStream that wraps the given iterator.""" self._lines = lines self._done = False self._next = (0, '') self._get_next() def _get_next(self) -> None: + """Advances the LineSteam to the next line or sets the _done + attribute if the LineStream has reached the end of the lines. + """ try: self._next = next(self._lines) except StopIteration: self._done = True def peek(self) -> str: + """Returns the current line, without advancing the LineStream. + """ return self._next[1] def pop(self) -> str: + """Returns the current line and advances the LineStream to + the next line. + """ n = self._next self._get_next() return n[1] def __bool__(self) -> bool: + """Returns True if stream has more lines.""" return not self._done # Only used by kunit_tool_test.py. def __iter__(self) -> Iterator[str]: + """Empties all lines stored in LineStream object into + Iterator object and returns the Iterator object. + """ while bool(self): yield self.pop() def line_number(self) -> int: + """Returns the line number of the current line.""" return self._next[0] -kunit_start_re = re.compile(r'TAP version [0-9]+$') -kunit_end_re = re.compile('(List of all partitions:|' - 'Kernel panic - not syncing: VFS:|reboot: System halted)') +# Parsing helper methods: + +KTAP_START = re.compile(r'KTAP version ([0-9]+)$') +TAP_START = re.compile(r'TAP version ([0-9]+)$') +KTAP_END = re.compile('(List of all partitions:|' + 'Kernel panic - not syncing: VFS:|reboot: System halted)') def extract_tap_lines(kernel_output: Iterable[str]) -> LineStream: - def isolate_kunit_output(kernel_output: Iterable[str]) -> Iterator[Tuple[int, str]]: + """Extracts KTAP lines from inputted kernel output in LineStream + object.""" + def isolate_ktap_output(kernel_output: Iterable[str]) \ + -> Iterator[Tuple[int, str]]: line_num = 0 started = False for line in kernel_output: line_num += 1 - line = line.rstrip() # line always has a trailing \n - if kunit_start_re.search(line): + line = line.rstrip() # remove trailing \n + if not started and KTAP_START.search(line): + # start extracting KTAP lines and set prefix + # to number of characters before version line + prefix_len = len( + line.split('KTAP version')[0]) + started = True + yield line_num, line[prefix_len:] + elif not started and TAP_START.search(line): + # start extracting KTAP lines and set prefix + # to number of characters before version line prefix_len = len(line.split('TAP version')[0]) started = True yield line_num, line[prefix_len:] - elif kunit_end_re.search(line): + elif started and KTAP_END.search(line): + # stop extracting KTAP lines break elif started: - yield line_num, line[prefix_len:] - return LineStream(lines=isolate_kunit_output(kernel_output)) - -DIVIDER = '=' * 60 - -RESET = '\033[0;0m' - -def red(text) -> str: - return '\033[1;31m' + text + RESET - -def yellow(text) -> str: - return '\033[1;33m' + text + RESET - -def green(text) -> str: - return '\033[1;32m' + text + RESET - -def print_with_timestamp(message) -> None: - print('[%s] %s' % (datetime.now().strftime('%H:%M:%S'), message)) + # remove prefix and any indention and yield + # line with line number + line = line[prefix_len:].lstrip() + yield line_num, line + return LineStream(lines=isolate_ktap_output(kernel_output)) + +KTAP_VERSIONS = [1] +TAP_VERSIONS = [13, 14] + +def check_version(version_num: int, accepted_versions: List[int], + version_type: str, test: Test) -> None: + """ + Adds error to test object if version number is too high or too + low. + + Parameters: + version_num - The inputted version number from the parsed KTAP or TAP + header line + accepted_version - List of accepted KTAP or TAP versions + version_type - 'KTAP' or 'TAP' depending on the type of + version line. + test - Test object for current test being parsed + """ + if version_num < min(accepted_versions): + test.add_error(version_type + + ' version lower than expected!') + elif version_num > max(accepted_versions): + test.add_error( + version_type + ' version higher than expected!') + +def parse_ktap_header(lines: LineStream, test: Test) -> bool: + """ + Parses KTAP/TAP header line and checks version number. + Returns False if fails to parse KTAP/TAP header line. + + Accepted formats: + - 'KTAP version [version number]' + - 'TAP version [version number]' + + Parameters: + lines - LineStream of KTAP output to parse + test - Test object for current test being parsed + + Return: + True if successfully parsed KTAP/TAP header line + """ + ktap_match = KTAP_START.match(lines.peek()) + tap_match = TAP_START.match(lines.peek()) + if ktap_match: + version_num = int(ktap_match.group(1)) + check_version(version_num, KTAP_VERSIONS, 'KTAP', test) + elif tap_match: + version_num = int(tap_match.group(1)) + check_version(version_num, TAP_VERSIONS, 'TAP', test) + else: + return False + test.log.append(lines.pop()) + return True -def format_suite_divider(message) -> str: - return '======== ' + message + ' ========' +TEST_HEADER = re.compile(r'^# Subtest: (.*)$') -def print_suite_divider(message) -> None: - print_with_timestamp(DIVIDER) - print_with_timestamp(format_suite_divider(message)) +def parse_test_header(lines: LineStream, test: Test) -> bool: + """ + Parses test header and stores test name in test object. + Returns False if fails to parse test header line. -def print_log(log) -> None: - for m in log: - print_with_timestamp(m) + Accepted format: + - '# Subtest: [test name]' -TAP_ENTRIES = re.compile(r'^(TAP|[\s]*ok|[\s]*not ok|[\s]*[0-9]+\.\.[0-9]+|[\s]*# (Subtest:|.*: kunit test case crashed!)).*$') + Parameters: + lines - LineStream of ktap output to parse + test - Test object for current test being parsed -def consume_non_diagnostic(lines: LineStream) -> None: - while lines and not TAP_ENTRIES.match(lines.peek()): - lines.pop() - -def save_non_diagnostic(lines: LineStream, test_case: TestCase) -> None: - while lines and not TAP_ENTRIES.match(lines.peek()): - test_case.log.append(lines.peek()) - lines.pop() + Return: + True if successfully parsed test header line + """ + match = TEST_HEADER.match(lines.peek()) + if not match: + return False + test.log.append(lines.pop()) + test.name = match.group(1) + return True -OkNotOkResult = namedtuple('OkNotOkResult', ['is_ok','description', 'text']) +TEST_PLAN = re.compile(r'1\.\.([0-9]+)') -OK_NOT_OK_SKIP = re.compile(r'^[\s]*(ok|not ok) [0-9]+ - (.*) # SKIP(.*)$') +def parse_test_plan(lines: LineStream, test: Test) -> bool: + """ + Parses test plan line and stores the expected number of subtests in + test object. Reports an error if expected count is 0. + Returns False and reports missing test plan error if fails to parse + test plan. -OK_NOT_OK_SUBTEST = re.compile(r'^[\s]+(ok|not ok) [0-9]+ - (.*)$') + Accepted format: + - '1..[number of subtests]' -OK_NOT_OK_MODULE = re.compile(r'^(ok|not ok) ([0-9]+) - (.*)$') + Parameters: + lines - LineStream of ktap output to parse + test - Test object for current test being parsed -def parse_ok_not_ok_test_case(lines: LineStream, test_case: TestCase) -> bool: - save_non_diagnostic(lines, test_case) - if not lines: - test_case.status = TestStatus.TEST_CRASHED - return True - line = lines.peek() - match = OK_NOT_OK_SUBTEST.match(line) - while not match and lines: - line = lines.pop() - match = OK_NOT_OK_SUBTEST.match(line) - if match: - test_case.log.append(lines.pop()) - test_case.name = match.group(2) - skip_match = OK_NOT_OK_SKIP.match(line) - if skip_match: - test_case.status = TestStatus.SKIPPED - return True - if test_case.status == TestStatus.TEST_CRASHED: - return True - if match.group(1) == 'ok': - test_case.status = TestStatus.SUCCESS - else: - test_case.status = TestStatus.FAILURE - return True - else: + Return: + True if successfully parsed test plan line + """ + match = TEST_PLAN.match(lines.peek()) + if not match: + test.expected_count = None + test.add_error('missing plan line!') return False - -SUBTEST_DIAGNOSTIC = re.compile(r'^[\s]+# (.*)$') -DIAGNOSTIC_CRASH_MESSAGE = re.compile(r'^[\s]+# .*?: kunit test case crashed!$') - -def parse_diagnostic(lines: LineStream, test_case: TestCase) -> bool: - save_non_diagnostic(lines, test_case) - if not lines: + test.log.append(lines.pop()) + expected_count = int(match.group(1)) + test.expected_count = expected_count + if expected_count == 0: + test.status = TestStatus.NO_TESTS + test.add_error('0 tests run!') + return True + +TEST_RESULT = re.compile(r'^(ok|not ok) ([0-9]+) (- )?([^#]*)( # .*)?$') + +TEST_RESULT_SKIP = re.compile(r'^(ok|not ok) ([0-9]+) (- )?(.*) # SKIP(.*)$') + +def peek_test_name_match(lines: LineStream, test: Test) -> bool: + """ + Matches current line with the format of a test result line and checks + if the name matches the name of the current test. + Returns False if fails to match format or name. + + Accepted format: + - '[ok|not ok] [test number] [-] [test name] [optional skip + directive]' + + Parameters: + lines - LineStream of KTAP output to parse + test - Test object for current test being parsed + + Return: + True if matched a test result line and the name matching the + expected test name + """ + line = lines.peek() + match = TEST_RESULT.match(line) + if not match: return False + name = match.group(4) + return (name == test.name) + +def parse_test_result(lines: LineStream, test: Test, + expected_num: int) -> bool: + """ + Parses test result line and stores the status and name in the test + object. Reports an error if the test number does not match expected + test number. + Returns False if fails to parse test result line. + + Note that the SKIP directive is the only direction that causes a + change in status. + + Accepted format: + - '[ok|not ok] [test number] [-] [test name] [optional skip + directive]' + + Parameters: + lines - LineStream of KTAP output to parse + test - Test object for current test being parsed + expected_num - expected test number for current test + + Return: + True if successfully parsed a test result line. + """ line = lines.peek() - match = SUBTEST_DIAGNOSTIC.match(line) - if match: - test_case.log.append(lines.pop()) - crash_match = DIAGNOSTIC_CRASH_MESSAGE.match(line) - if crash_match: - test_case.status = TestStatus.TEST_CRASHED - return True - else: + match = TEST_RESULT.match(line) + skip_match = TEST_RESULT_SKIP.match(line) + + # Check if line matches test result line format + if not match: return False + test.log.append(lines.pop()) -def parse_test_case(lines: LineStream) -> Optional[TestCase]: - test_case = TestCase() - save_non_diagnostic(lines, test_case) - while parse_diagnostic(lines, test_case): - pass - if parse_ok_not_ok_test_case(lines, test_case): - return test_case + # Set name of test object + if skip_match: + test.name = skip_match.group(4) else: - return None - -SUBTEST_HEADER = re.compile(r'^[\s]+# Subtest: (.*)$') - -def parse_subtest_header(lines: LineStream) -> Optional[str]: - consume_non_diagnostic(lines) - if not lines: - return None - match = SUBTEST_HEADER.match(lines.peek()) - if match: - lines.pop() - return match.group(1) + test.name = match.group(4) + + # Check test num + num = int(match.group(2)) + if num != expected_num: + test.add_error('Expected test number ' + + str(expected_num) + ' but found ' + str(num)) + + # Set status of test object + status = match.group(1) + if skip_match: + test.status = TestStatus.SKIPPED + elif status == 'ok': + test.status = TestStatus.SUCCESS else: - return None + test.status = TestStatus.FAILURE + return True + +def parse_diagnostic(lines: LineStream) -> List[str]: + """ + Parse lines that do not match the format of a test result line or + test header line and returns them in list. + + Line formats that are not parsed: + - '# Subtest: [test name]' + - '[ok|not ok] [test number] [-] [test name] [optional skip + directive]' + + Parameters: + lines - LineStream of KTAP output to parse + + Return: + Log of diagnostic lines + """ + log = [] # type: List[str] + while lines and not TEST_RESULT.match(lines.peek()) and not \ + TEST_HEADER.match(lines.peek()): + log.append(lines.pop()) + return log + +DIAGNOSTIC_CRASH_MESSAGE = re.compile( + r'^(BUG:|# .*?: kunit test case crashed!$)') + +def parse_crash_in_log(test: Test) -> bool: + """ + Iterate through the lines of the log to parse for crash message. + If crash message found, set status to crashed and return True. + Otherwise return False. + + Parameters: + test - Test object for current test being parsed + + Return: + True if crash message found in log + """ + for line in test.log: + if DIAGNOSTIC_CRASH_MESSAGE.match(line): + test.status = TestStatus.TEST_CRASHED + return True + return False -SUBTEST_PLAN = re.compile(r'[\s]+[0-9]+\.\.([0-9]+)') -def parse_subtest_plan(lines: LineStream) -> Optional[int]: - consume_non_diagnostic(lines) - match = SUBTEST_PLAN.match(lines.peek()) - if match: - lines.pop() - return int(match.group(1)) - else: - return None - -def max_status(left: TestStatus, right: TestStatus) -> TestStatus: - if left == right: - return left - elif left == TestStatus.TEST_CRASHED or right == TestStatus.TEST_CRASHED: - return TestStatus.TEST_CRASHED - elif left == TestStatus.FAILURE or right == TestStatus.FAILURE: - return TestStatus.FAILURE - elif left == TestStatus.SKIPPED: - return right - else: - return left +# Printing helper methods: -def parse_ok_not_ok_test_suite(lines: LineStream, - test_suite: TestSuite, - expected_suite_index: int) -> bool: - consume_non_diagnostic(lines) - if not lines: - test_suite.status = TestStatus.TEST_CRASHED - return False - line = lines.peek() - match = OK_NOT_OK_MODULE.match(line) - if match: - lines.pop() - if match.group(1) == 'ok': - test_suite.status = TestStatus.SUCCESS - else: - test_suite.status = TestStatus.FAILURE - skip_match = OK_NOT_OK_SKIP.match(line) - if skip_match: - test_suite.status = TestStatus.SKIPPED - suite_index = int(match.group(2)) - if suite_index != expected_suite_index: - print_with_timestamp( - red('[ERROR] ') + 'expected_suite_index ' + - str(expected_suite_index) + ', but got ' + - str(suite_index)) - return True - else: - return False +DIVIDER = '=' * 60 -def bubble_up_errors(status_list: Iterable[TestStatus]) -> TestStatus: - return reduce(max_status, status_list, TestStatus.SKIPPED) +RESET = '\033[0;0m' -def bubble_up_test_case_errors(test_suite: TestSuite) -> TestStatus: - max_test_case_status = bubble_up_errors(x.status for x in test_suite.cases) - return max_status(max_test_case_status, test_suite.status) +def red(text: str) -> str: + """Returns inputted string with red color code.""" + return '\033[1;31m' + text + RESET -def parse_test_suite(lines: LineStream, expected_suite_index: int) -> Optional[TestSuite]: - if not lines: - return None - consume_non_diagnostic(lines) - test_suite = TestSuite() - test_suite.status = TestStatus.SUCCESS - name = parse_subtest_header(lines) - if not name: - return None - test_suite.name = name - expected_test_case_num = parse_subtest_plan(lines) - if expected_test_case_num is None: - return None - while expected_test_case_num > 0: - test_case = parse_test_case(lines) - if not test_case: - break - test_suite.cases.append(test_case) - expected_test_case_num -= 1 - if parse_ok_not_ok_test_suite(lines, test_suite, expected_suite_index): - test_suite.status = bubble_up_test_case_errors(test_suite) - return test_suite - elif not lines: - print_with_timestamp(red('[ERROR] ') + 'ran out of lines before end token') - return test_suite - else: - print(f'failed to parse end of suite "{name}", at line {lines.line_number()}: {lines.peek()}') - return None +def yellow(text: str) -> str: + """Returns inputted string with yellow color code.""" + return '\033[1;33m' + text + RESET -TAP_HEADER = re.compile(r'^TAP version 14$') +def green(text: str) -> str: + """Returns inputted string with green color code.""" + return '\033[1;32m' + text + RESET -def parse_tap_header(lines: LineStream) -> bool: - consume_non_diagnostic(lines) - if TAP_HEADER.match(lines.peek()): - lines.pop() - return True - else: - return False +ANSI_LEN = len(red('')) -TEST_PLAN = re.compile(r'[0-9]+\.\.([0-9]+)') +def print_with_timestamp(message: str) -> None: + """Prints message with timestamp at beginning.""" + print('[%s] %s' % (datetime.now().strftime('%H:%M:%S'), message)) -def parse_test_plan(lines: LineStream) -> Optional[int]: - consume_non_diagnostic(lines) - match = TEST_PLAN.match(lines.peek()) - if match: - lines.pop() - return int(match.group(1)) - else: - return None - -def bubble_up_suite_errors(test_suites: Iterable[TestSuite]) -> TestStatus: - return bubble_up_errors(x.status for x in test_suites) - -def parse_test_result(lines: LineStream) -> TestResult: - consume_non_diagnostic(lines) - if not lines or not parse_tap_header(lines): - return TestResult(TestStatus.FAILURE_TO_PARSE_TESTS, [], lines) - expected_test_suite_num = parse_test_plan(lines) - if expected_test_suite_num == 0: - return TestResult(TestStatus.NO_TESTS, [], lines) - elif expected_test_suite_num is None: - return TestResult(TestStatus.FAILURE_TO_PARSE_TESTS, [], lines) - test_suites = [] - for i in range(1, expected_test_suite_num + 1): - test_suite = parse_test_suite(lines, i) - if test_suite: - test_suites.append(test_suite) +def format_test_divider(message: str, len_message: int) -> str: + """ + Returns string with message centered in fixed width divider. + + Example: + '===================== message example =====================' + + Parameters: + message - message to be centered in divider line + len_message - length of the message to be printed such that + any characters of the color codes are not counted + + Return: + String containing message centered in fixed width divider + """ + default_count = 3 # default number of dashes + len_1 = default_count + len_2 = default_count + difference = len(DIVIDER) - len_message - 2 # 2 spaces added + if difference > 0: + # calculate number of dashes for each side of the divider + len_1 = int(difference / 2) + len_2 = difference - len_1 + return ('=' * len_1) + ' ' + message + ' ' + ('=' * len_2) + +def print_test_header(test: Test) -> None: + """ + Prints test header with test name and optionally the expected number + of subtests. + + Example: + '=================== example (2 subtests) ===================' + + Parameters: + test - Test object representing current test being printed + """ + message = test.name + if test.expected_count: + if test.expected_count == 1: + message += (' (' + str(test.expected_count) + + ' subtest)') else: - print_with_timestamp( - red('[ERROR] ') + ' expected ' + - str(expected_test_suite_num) + - ' test suites, but got ' + str(i - 2)) - break - test_suite = parse_test_suite(lines, -1) - if test_suite: - print_with_timestamp(red('[ERROR] ') + - 'got unexpected test suite: ' + test_suite.name) - if test_suites: - return TestResult(bubble_up_suite_errors(test_suites), test_suites, lines) - else: - return TestResult(TestStatus.NO_TESTS, [], lines) + message += (' (' + str(test.expected_count) + + ' subtests)') + print_with_timestamp(format_test_divider(message, len(message))) -class TestCounts: - passed: int - failed: int - crashed: int - skipped: int +def print_log(log: Iterable[str]) -> None: + """ + Prints all strings in saved log for test in yellow. - def __init__(self): - self.passed = 0 - self.failed = 0 - self.crashed = 0 - self.skipped = 0 - - def total(self) -> int: - return self.passed + self.failed + self.crashed + self.skipped - -def print_and_count_results(test_result: TestResult) -> TestCounts: - counts = TestCounts() - for test_suite in test_result.suites: - if test_suite.status == TestStatus.SUCCESS: - print_suite_divider(green('[PASSED] ') + test_suite.name) - elif test_suite.status == TestStatus.SKIPPED: - print_suite_divider(yellow('[SKIPPED] ') + test_suite.name) - elif test_suite.status == TestStatus.TEST_CRASHED: - print_suite_divider(red('[CRASHED] ' + test_suite.name)) - else: - print_suite_divider(red('[FAILED] ') + test_suite.name) - for test_case in test_suite.cases: - if test_case.status == TestStatus.SUCCESS: - counts.passed += 1 - print_with_timestamp(green('[PASSED] ') + test_case.name) - elif test_case.status == TestStatus.SKIPPED: - counts.skipped += 1 - print_with_timestamp(yellow('[SKIPPED] ') + test_case.name) - elif test_case.status == TestStatus.TEST_CRASHED: - counts.crashed += 1 - print_with_timestamp(red('[CRASHED] ' + test_case.name)) - print_log(map(yellow, test_case.log)) - print_with_timestamp('') + Parameters: + log - Iterable object with all strings saved in log for test + """ + for m in log: + print_with_timestamp(yellow(m)) + +def format_test_result(test: Test) -> str: + """ + Returns string with formatted test result with colored status and test + name. + + Example: + '[PASSED] example' + + Parameters: + test - Test object representing current test being printed + + Return: + String containing formatted test result + """ + if test.status == TestStatus.SUCCESS: + return (green('[PASSED] ') + test.name) + elif test.status == TestStatus.SKIPPED: + return (yellow('[SKIPPED] ') + test.name) + elif test.status == TestStatus.TEST_CRASHED: + print_log(test.log) + return (red('[CRASHED] ') + test.name) + else: + print_log(test.log) + return (red('[FAILED] ') + test.name) + +def print_test_result(test: Test) -> None: + """ + Prints result line with status of test. + + Example: + '[PASSED] example' + + Parameters: + test - Test object representing current test being printed + """ + print_with_timestamp(format_test_result(test)) + +def print_test_footer(test: Test) -> None: + """ + Prints test footer with status of test. + + Example: + '===================== [PASSED] example =====================' + + Parameters: + test - Test object representing current test being printed + """ + message = format_test_result(test) + print_with_timestamp(format_test_divider(message, + len(message) - ANSI_LEN)) + +def print_summary_line(test: Test) -> None: + """ + Prints summary line of test object. Color of line is dependent on + status of test. Color is green if test passes, yellow if test is + skipped, and red if the test fails or crashes. Summary line contains + counts of the statuses of the tests subtests or the test itself if it + has no subtests. + + Example: + "Testing complete. Passed: 2, Failed: 0, Crashed: 0, Skipped: 0, + Errors: 0" + + test - Test object representing current test being printed + """ + if test.status == TestStatus.SUCCESS or \ + test.status == TestStatus.NO_TESTS: + color = green + elif test.status == TestStatus.SKIPPED: + color = yellow + else: + color = red + counts = test.counts + print_with_timestamp(color('Testing complete. ' + str(counts))) + +def print_error(error_message: str) -> None: + """ + Prints error message with error format. + + Example: + "[ERROR] Test example: missing test plan!" + + Parameters: + error_message - message describing error + """ + print_with_timestamp(red('[ERROR] ') + error_message) + +# Other methods: + +def bubble_up_test_results(test: Test) -> None: + """ + If the test has subtests, add the test counts of the subtests to the + test and check if any of the tests crashed and if so set the test + status to crashed. Otherwise if the test has no subtests add the + status of the test to the test counts. + + Parameters: + test - Test object for current test being parsed + """ + parse_crash_in_log(test) + subtests = test.subtests + counts = test.counts + status = test.status + for t in subtests: + counts.add_subtest_counts(t.counts) + if counts.total() == 0: + counts.add_status(status) + elif test.counts.get_status() == TestStatus.TEST_CRASHED: + test.status = TestStatus.TEST_CRASHED + +def parse_test(lines: LineStream, expected_num: int, log: List[str]) -> Test: + """ + Finds next test to parse in LineStream, creates new Test object, + parses any subtests of the test, populates Test object with all + information (status, name) about the test and the Test objects for + any subtests, and then returns the Test object. The method accepts + three formats of tests: + + Accepted test formats: + + - Main KTAP/TAP header + + Example: + + KTAP version 1 + 1..4 + [subtests] + + - Subtest header line + + Example: + + # Subtest: name + 1..3 + [subtests] + ok 1 name + + - Test result line + + Example: + + ok 1 - test + + Parameters: + lines - LineStream of KTAP output to parse + expected_num - expected test number for test to be parsed + log - list of strings containing any preceding diagnostic lines + corresponding to the current test + + Return: + Test object populated with characteristics and any subtests + """ + test = Test() + test.log.extend(log) + parent_test = False + main = parse_ktap_header(lines, test) + if main: + # If KTAP/TAP header is found, attempt to parse + # test plan + test.name = "main" + parse_test_plan(lines, test) + else: + # If KTAP/TAP header is not found, test must be subtest + # header or test result line so parse attempt to parser + # subtest header + parent_test = parse_test_header(lines, test) + if parent_test: + # If subtest header is found, attempt to parse + # test plan and print header + parse_test_plan(lines, test) + print_test_header(test) + expected_count = test.expected_count + subtests = [] + test_num = 1 + while expected_count is None or test_num <= expected_count: + # Loop to parse any subtests. + # Break after parsing expected number of tests or + # if expected number of tests is unknown break when test + # result line with matching name to subtest header is found + # or no more lines in stream. + sub_log = parse_diagnostic(lines) + sub_test = Test() + if not lines or (peek_test_name_match(lines, test) and + not main): + if expected_count and test_num <= expected_count: + # If parser reaches end of test before + # parsing expected number of subtests, print + # crashed subtest and record error + test.add_error('missing expected subtest!') + sub_test.log.extend(sub_log) + test.counts.add_status( + TestStatus.TEST_CRASHED) + print_test_result(sub_test) else: - counts.failed += 1 - print_with_timestamp(red('[FAILED] ') + test_case.name) - print_log(map(yellow, test_case.log)) - print_with_timestamp('') - return counts + test.log.extend(sub_log) + break + else: + sub_test = parse_test(lines, test_num, sub_log) + subtests.append(sub_test) + test_num += 1 + test.subtests = subtests + if not main: + # If not main test, look for test result line + test.log.extend(parse_diagnostic(lines)) + if (parent_test and peek_test_name_match(lines, test)) or \ + not parent_test: + parse_test_result(lines, test, expected_num) + else: + test.add_error('missing subtest result line!') + # Add statuses to TestCounts attribute in Test object + bubble_up_test_results(test) + if parent_test: + # If test has subtests and is not the main test object, print + # footer. + print_test_footer(test) + elif not main: + print_test_result(test) + return test def parse_run_tests(kernel_output: Iterable[str]) -> TestResult: - counts = TestCounts() + """ + Using kernel output, extract KTAP lines, parse the lines for test + results and print condensed test results and summary line . + + Parameters: + kernel_output - Iterable object contains lines of kernel output + + Return: + TestResult - Tuple containg status of main test object, main test + object with all subtests, and log of all KTAP lines. + """ + print_with_timestamp(DIVIDER) lines = extract_tap_lines(kernel_output) - test_result = parse_test_result(lines) - if test_result.status == TestStatus.NO_TESTS: - print(red('[ERROR] ') + yellow('no tests run!')) - elif test_result.status == TestStatus.FAILURE_TO_PARSE_TESTS: - print(red('[ERROR] ') + yellow('could not parse test results!')) + test = Test() + if not lines: + test.add_error('invalid KTAP input!') + test.status = TestStatus.FAILURE_TO_PARSE_TESTS else: - counts = print_and_count_results(test_result) + test = parse_test(lines, 0, []) + if test.status != TestStatus.NO_TESTS: + test.status = test.counts.get_status() print_with_timestamp(DIVIDER) - if test_result.status == TestStatus.SUCCESS: - fmt = green - elif test_result.status == TestStatus.SKIPPED: - fmt = yellow - else: - fmt =red - print_with_timestamp( - fmt('Testing complete. %d tests run. %d failed. %d crashed. %d skipped.' % - (counts.total(), counts.failed, counts.crashed, counts.skipped))) - return test_result + print_summary_line(test) + return TestResult(test.status, test, lines) diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index 3251cef25b76..c309ed76aef5 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -106,10 +106,10 @@ class KUnitParserTest(unittest.TestCase): with open(log_path) as file: result = kunit_parser.extract_tap_lines(file.readlines()) self.assertContains('TAP version 14', result) - self.assertContains(' # Subtest: example', result) - self.assertContains(' 1..2', result) - self.assertContains(' ok 1 - example_simple_test', result) - self.assertContains(' ok 2 - example_mock_test', result) + self.assertContains('# Subtest: example', result) + self.assertContains('1..2', result) + self.assertContains('ok 1 - example_simple_test', result) + self.assertContains('ok 2 - example_mock_test', result) self.assertContains('ok 1 - example', result) def test_output_with_prefix_isolated_correctly(self): @@ -117,28 +117,28 @@ class KUnitParserTest(unittest.TestCase): with open(log_path) as file: result = kunit_parser.extract_tap_lines(file.readlines()) self.assertContains('TAP version 14', result) - self.assertContains(' # Subtest: kunit-resource-test', result) - self.assertContains(' 1..5', result) - self.assertContains(' ok 1 - kunit_resource_test_init_resources', result) - self.assertContains(' ok 2 - kunit_resource_test_alloc_resource', result) - self.assertContains(' ok 3 - kunit_resource_test_destroy_resource', result) - self.assertContains(' foo bar #', result) - self.assertContains(' ok 4 - kunit_resource_test_cleanup_resources', result) - self.assertContains(' ok 5 - kunit_resource_test_proper_free_ordering', result) + self.assertContains('# Subtest: kunit-resource-test', result) + self.assertContains('1..5', result) + self.assertContains('ok 1 - kunit_resource_test_init_resources', result) + self.assertContains('ok 2 - kunit_resource_test_alloc_resource', result) + self.assertContains('ok 3 - kunit_resource_test_destroy_resource', result) + self.assertContains('foo bar #', result) + self.assertContains('ok 4 - kunit_resource_test_cleanup_resources', result) + self.assertContains('ok 5 - kunit_resource_test_proper_free_ordering', result) self.assertContains('ok 1 - kunit-resource-test', result) - self.assertContains(' foo bar # non-kunit output', result) - self.assertContains(' # Subtest: kunit-try-catch-test', result) - self.assertContains(' 1..2', result) - self.assertContains(' ok 1 - kunit_test_try_catch_successful_try_no_catch', + self.assertContains('foo bar # non-kunit output', result) + self.assertContains('# Subtest: kunit-try-catch-test', result) + self.assertContains('1..2', result) + self.assertContains('ok 1 - kunit_test_try_catch_successful_try_no_catch', result) - self.assertContains(' ok 2 - kunit_test_try_catch_unsuccessful_try_does_catch', + self.assertContains('ok 2 - kunit_test_try_catch_unsuccessful_try_does_catch', result) self.assertContains('ok 2 - kunit-try-catch-test', result) - self.assertContains(' # Subtest: string-stream-test', result) - self.assertContains(' 1..3', result) - self.assertContains(' ok 1 - string_stream_test_empty_on_creation', result) - self.assertContains(' ok 2 - string_stream_test_not_empty_after_add', result) - self.assertContains(' ok 3 - string_stream_test_get_string', result) + self.assertContains('# Subtest: string-stream-test', result) + self.assertContains('1..3', result) + self.assertContains('ok 1 - string_stream_test_empty_on_creation', result) + self.assertContains('ok 2 - string_stream_test_not_empty_after_add', result) + self.assertContains('ok 3 - string_stream_test_get_string', result) self.assertContains('ok 3 - string-stream-test', result) def test_parse_successful_test_log(self): @@ -149,6 +149,22 @@ class KUnitParserTest(unittest.TestCase): kunit_parser.TestStatus.SUCCESS, result.status) + def test_parse_successful_nested_tests_log(self): + all_passed_log = test_data_path('test_is_test_passed-all_passed_nested.log') + with open(all_passed_log) as file: + result = kunit_parser.parse_run_tests(file.readlines()) + self.assertEqual( + kunit_parser.TestStatus.SUCCESS, + result.status) + + def test_kselftest_nested(self): + kselftest_log = test_data_path('test_is_test_passed-kselftest.log') + with open(kselftest_log) as file: + result = kunit_parser.parse_run_tests(file.readlines()) + self.assertEqual( + kunit_parser.TestStatus.SUCCESS, + result.status) + def test_parse_failed_test_log(self): failed_log = test_data_path('test_is_test_passed-failure.log') with open(failed_log) as file: @@ -162,17 +178,31 @@ class KUnitParserTest(unittest.TestCase): with open(empty_log) as file: result = kunit_parser.parse_run_tests( kunit_parser.extract_tap_lines(file.readlines())) - self.assertEqual(0, len(result.suites)) + self.assertEqual(0, len(result.test.subtests)) self.assertEqual( kunit_parser.TestStatus.FAILURE_TO_PARSE_TESTS, result.status) + def test_missing_test_plan(self): + missing_plan_log = test_data_path('test_is_test_passed-' + 'missing_plan.log') + with open(missing_plan_log) as file: + result = kunit_parser.parse_run_tests( + kunit_parser.extract_tap_lines( + file.readlines())) + self.assertEqual(2, result.test.counts.errors) + self.assertEqual( + kunit_parser.TestStatus.SUCCESS, + result.status) + def test_no_tests(self): - empty_log = test_data_path('test_is_test_passed-no_tests_run_with_header.log') - with open(empty_log) as file: + header_log = test_data_path('test_is_test_passed-' + 'no_tests_run_with_header.log') + with open(header_log) as file: result = kunit_parser.parse_run_tests( - kunit_parser.extract_tap_lines(file.readlines())) - self.assertEqual(0, len(result.suites)) + kunit_parser.extract_tap_lines( + file.readlines())) + self.assertEqual(0, len(result.test.subtests)) self.assertEqual( kunit_parser.TestStatus.NO_TESTS, result.status) @@ -182,15 +212,17 @@ class KUnitParserTest(unittest.TestCase): print_mock = mock.patch('builtins.print').start() with open(crash_log) as file: result = kunit_parser.parse_run_tests( - kunit_parser.extract_tap_lines(file.readlines())) - print_mock.assert_any_call(StrContains('could not parse test results!')) + kunit_parser.extract_tap_lines( + file.readlines())) + print_mock.assert_any_call(StrContains('invalid KTAP input!')) print_mock.stop() file.close() def test_crashed_test(self): crashed_log = test_data_path('test_is_test_passed-crash.log') with open(crashed_log) as file: - result = kunit_parser.parse_run_tests(file.readlines()) + result = kunit_parser.parse_run_tests( + file.readlines()) self.assertEqual( kunit_parser.TestStatus.TEST_CRASHED, result.status) @@ -216,6 +248,23 @@ class KUnitParserTest(unittest.TestCase): result.status) file.close() + def test_ignores_hyphen(self): + hyphen_log = test_data_path('test_strip_hyphen.log') + file = open(hyphen_log) + result = kunit_parser.parse_run_tests(file.readlines()) + + # A skipped test does not fail the whole suite. + self.assertEqual( + kunit_parser.TestStatus.SUCCESS, + result.status) + self.assertEqual( + "sysctl_test", + result.test.subtests[0].name) + self.assertEqual( + "example", + result.test.subtests[1].name) + file.close() + def test_ignores_prefix_printk_time(self): prefix_log = test_data_path('test_config_printk_time.log') @@ -224,7 +273,7 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual( kunit_parser.TestStatus.SUCCESS, result.status) - self.assertEqual('kunit-resource-test', result.suites[0].name) + self.assertEqual('kunit-resource-test', result.test.subtests[0].name) def test_ignores_multiple_prefixes(self): prefix_log = test_data_path('test_multiple_prefixes.log') @@ -233,7 +282,7 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual( kunit_parser.TestStatus.SUCCESS, result.status) - self.assertEqual('kunit-resource-test', result.suites[0].name) + self.assertEqual('kunit-resource-test', result.test.subtests[0].name) def test_prefix_mixed_kernel_output(self): mixed_prefix_log = test_data_path('test_interrupted_tap_output.log') @@ -242,7 +291,7 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual( kunit_parser.TestStatus.SUCCESS, result.status) - self.assertEqual('kunit-resource-test', result.suites[0].name) + self.assertEqual('kunit-resource-test', result.test.subtests[0].name) def test_prefix_poundsign(self): pound_log = test_data_path('test_pound_sign.log') @@ -251,7 +300,7 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual( kunit_parser.TestStatus.SUCCESS, result.status) - self.assertEqual('kunit-resource-test', result.suites[0].name) + self.assertEqual('kunit-resource-test', result.test.subtests[0].name) def test_kernel_panic_end(self): panic_log = test_data_path('test_kernel_panic_interrupt.log') @@ -260,7 +309,7 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual( kunit_parser.TestStatus.TEST_CRASHED, result.status) - self.assertEqual('kunit-resource-test', result.suites[0].name) + self.assertEqual('kunit-resource-test', result.test.subtests[0].name) def test_pound_no_prefix(self): pound_log = test_data_path('test_pound_no_prefix.log') @@ -269,7 +318,7 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual( kunit_parser.TestStatus.SUCCESS, result.status) - self.assertEqual('kunit-resource-test', result.suites[0].name) + self.assertEqual('kunit-resource-test', result.test.subtests[0].name) class LinuxSourceTreeTest(unittest.TestCase): @@ -322,6 +371,12 @@ class KUnitJsonTest(unittest.TestCase): result = self._json_for('test_is_test_passed-no_tests_run_with_header.log') self.assertEqual(0, len(result['sub_groups'])) + def test_nested_json(self): + result = self._json_for('test_is_test_passed-all_passed_nested.log') + self.assertEqual( + {'name': 'example_simple_test', 'status': 'PASS'}, + result["sub_groups"][0]["sub_groups"][0]["test_cases"][0]) + class StrContains(str): def __eq__(self, other): return self in other @@ -380,7 +435,7 @@ class KUnitMainTest(unittest.TestCase): self.assertEqual(e.exception.code, 1) self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 1) self.assertEqual(self.linux_source_mock.run_kernel.call_count, 1) - self.print_mock.assert_any_call(StrContains(' 0 tests run')) + self.print_mock.assert_any_call(StrContains('invalid KTAP input!')) def test_exec_raw_output(self): self.linux_source_mock.run_kernel = mock.Mock(return_value=[]) @@ -388,7 +443,7 @@ class KUnitMainTest(unittest.TestCase): self.assertEqual(self.linux_source_mock.run_kernel.call_count, 1) for call in self.print_mock.call_args_list: self.assertNotEqual(call, mock.call(StrContains('Testing complete.'))) - self.assertNotEqual(call, mock.call(StrContains(' 0 tests run'))) + self.assertNotEqual(call, mock.call(StrContains(' 0 tests run!'))) def test_run_raw_output(self): self.linux_source_mock.run_kernel = mock.Mock(return_value=[]) @@ -397,7 +452,7 @@ class KUnitMainTest(unittest.TestCase): self.assertEqual(self.linux_source_mock.run_kernel.call_count, 1) for call in self.print_mock.call_args_list: self.assertNotEqual(call, mock.call(StrContains('Testing complete.'))) - self.assertNotEqual(call, mock.call(StrContains(' 0 tests run'))) + self.assertNotEqual(call, mock.call(StrContains(' 0 tests run!'))) def test_run_raw_output_kunit(self): self.linux_source_mock.run_kernel = mock.Mock(return_value=[]) diff --git a/tools/testing/kunit/test_data/test_is_test_passed-all_passed_nested.log b/tools/testing/kunit/test_data/test_is_test_passed-all_passed_nested.log new file mode 100644 index 000000000000..9d5b04fe43a6 --- /dev/null +++ b/tools/testing/kunit/test_data/test_is_test_passed-all_passed_nested.log @@ -0,0 +1,34 @@ +TAP version 14 +1..2 + # Subtest: sysctl_test + 1..4 + # sysctl_test_dointvec_null_tbl_data: sysctl_test_dointvec_null_tbl_data passed + ok 1 - sysctl_test_dointvec_null_tbl_data + # Subtest: example + 1..2 + init_suite + # example_simple_test: initializing + # example_simple_test: example_simple_test passed + ok 1 - example_simple_test + # example_mock_test: initializing + # example_mock_test: example_mock_test passed + ok 2 - example_mock_test + kunit example: all tests passed + ok 2 - example + # sysctl_test_dointvec_table_len_is_zero: sysctl_test_dointvec_table_len_is_zero passed + ok 3 - sysctl_test_dointvec_table_len_is_zero + # sysctl_test_dointvec_table_read_but_position_set: sysctl_test_dointvec_table_read_but_position_set passed + ok 4 - sysctl_test_dointvec_table_read_but_position_set +kunit sysctl_test: all tests passed +ok 1 - sysctl_test + # Subtest: example + 1..2 +init_suite + # example_simple_test: initializing + # example_simple_test: example_simple_test passed + ok 1 - example_simple_test + # example_mock_test: initializing + # example_mock_test: example_mock_test passed + ok 2 - example_mock_test +kunit example: all tests passed +ok 2 - example diff --git a/tools/testing/kunit/test_data/test_is_test_passed-kselftest.log b/tools/testing/kunit/test_data/test_is_test_passed-kselftest.log new file mode 100644 index 000000000000..65d3f27feaf2 --- /dev/null +++ b/tools/testing/kunit/test_data/test_is_test_passed-kselftest.log @@ -0,0 +1,14 @@ +TAP version 13 +1..2 +# selftests: membarrier: membarrier_test_single_thread +# TAP version 13 +# 1..2 +# ok 1 sys_membarrier available +# ok 2 sys membarrier invalid command test: command = -1, flags = 0, errno = 22. Failed as expected +ok 1 selftests: membarrier: membarrier_test_single_thread +# selftests: membarrier: membarrier_test_multi_thread +# TAP version 13 +# 1..2 +# ok 1 sys_membarrier available +# ok 2 sys membarrier invalid command test: command = -1, flags = 0, errno = 22. Failed as expected +ok 2 selftests: membarrier: membarrier_test_multi_thread diff --git a/tools/testing/kunit/test_data/test_is_test_passed-missing_plan.log b/tools/testing/kunit/test_data/test_is_test_passed-missing_plan.log new file mode 100644 index 000000000000..5cd17b7f818a --- /dev/null +++ b/tools/testing/kunit/test_data/test_is_test_passed-missing_plan.log @@ -0,0 +1,31 @@ +KTAP version 1 + # Subtest: sysctl_test + # sysctl_test_dointvec_null_tbl_data: sysctl_test_dointvec_null_tbl_data passed + ok 1 - sysctl_test_dointvec_null_tbl_data + # sysctl_test_dointvec_table_maxlen_unset: sysctl_test_dointvec_table_maxlen_unset passed + ok 2 - sysctl_test_dointvec_table_maxlen_unset + # sysctl_test_dointvec_table_len_is_zero: sysctl_test_dointvec_table_len_is_zero passed + ok 3 - sysctl_test_dointvec_table_len_is_zero + # sysctl_test_dointvec_table_read_but_position_set: sysctl_test_dointvec_table_read_but_position_set passed + ok 4 - sysctl_test_dointvec_table_read_but_position_set + # sysctl_test_dointvec_happy_single_positive: sysctl_test_dointvec_happy_single_positive passed + ok 5 - sysctl_test_dointvec_happy_single_positive + # sysctl_test_dointvec_happy_single_negative: sysctl_test_dointvec_happy_single_negative passed + ok 6 - sysctl_test_dointvec_happy_single_negative + # sysctl_test_dointvec_single_less_int_min: sysctl_test_dointvec_single_less_int_min passed + ok 7 - sysctl_test_dointvec_single_less_int_min + # sysctl_test_dointvec_single_greater_int_max: sysctl_test_dointvec_single_greater_int_max passed + ok 8 - sysctl_test_dointvec_single_greater_int_max +kunit sysctl_test: all tests passed +ok 1 - sysctl_test + # Subtest: example + 1..2 +init_suite + # example_simple_test: initializing + # example_simple_test: example_simple_test passed + ok 1 - example_simple_test + # example_mock_test: initializing + # example_mock_test: example_mock_test passed + ok 2 - example_mock_test +kunit example: all tests passed +ok 2 - example diff --git a/tools/testing/kunit/test_data/test_strip_hyphen.log b/tools/testing/kunit/test_data/test_strip_hyphen.log new file mode 100644 index 000000000000..92ac7c24b374 --- /dev/null +++ b/tools/testing/kunit/test_data/test_strip_hyphen.log @@ -0,0 +1,16 @@ +KTAP version 1 +1..2 + # Subtest: sysctl_test + 1..1 + # sysctl_test_dointvec_null_tbl_data: sysctl_test_dointvec_null_tbl_data passed + ok 1 - sysctl_test_dointvec_null_tbl_data +kunit sysctl_test: all tests passed +ok 1 - sysctl_test + # Subtest: example + 1..1 +init_suite + # example_simple_test: initializing + # example_simple_test: example_simple_test passed + ok 1 example_simple_test +kunit example: all tests passed +ok 2 example base-commit: 28a8b037484044d79f3b8d409a076a925ac75f8d -- 2.33.0.882.g93a45727a2-goog

3 years, 7 months

2
2
0 0

[PATCH MANUALSEL 5.10 1/7] selftests: KVM: Align SMCCC call with the spec in steal_time

by Sasha Levin

From: Oliver Upton <oupton(a)google.com> [ Upstream commit 01f91acb55be7aac3950b89c458bcea9ef6e4f49 ] The SMC64 calling convention passes a function identifier in w0 and its parameters in x1-x17. Given this, there are two deviations in the SMC64 call performed by the steal_time test: the function identifier is assigned to a 64 bit register and the parameter is only 32 bits wide. Align the call with the SMCCC by using a 32 bit register to handle the function identifier and increasing the parameter width to 64 bits. Suggested-by: Andrew Jones <drjones(a)redhat.com> Signed-off-by: Oliver Upton <oupton(a)google.com> Reviewed-by: Andrew Jones <drjones(a)redhat.com> Message-Id: <20210921171121.2148982-3-oupton(a)google.com> Signed-off-by: Paolo Bonzini <pbonzini(a)redhat.com> Signed-off-by: Sasha Levin <sashal(a)kernel.org> --- tools/testing/selftests/kvm/steal_time.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/kvm/steal_time.c b/tools/testing/selftests/kvm/steal_time.c index fcc840088c91..7daedee3e7ee 100644 --- a/tools/testing/selftests/kvm/steal_time.c +++ b/tools/testing/selftests/kvm/steal_time.c @@ -120,12 +120,12 @@ struct st_time { uint64_t st_time; }; -static int64_t smccc(uint32_t func, uint32_t arg) +static int64_t smccc(uint32_t func, uint64_t arg) { unsigned long ret; asm volatile( - "mov x0, %1\n" + "mov w0, %w1\n" "mov x1, %2\n" "hvc #0\n" "mov %0, x0\n" -- 2.33.0

3 years, 7 months

2
2
0 0

[PATCH MANUALSEL 5.14 1/9] selftests: KVM: Align SMCCC call with the spec in steal_time

by Sasha Levin

From: Oliver Upton <oupton(a)google.com> [ Upstream commit 01f91acb55be7aac3950b89c458bcea9ef6e4f49 ] The SMC64 calling convention passes a function identifier in w0 and its parameters in x1-x17. Given this, there are two deviations in the SMC64 call performed by the steal_time test: the function identifier is assigned to a 64 bit register and the parameter is only 32 bits wide. Align the call with the SMCCC by using a 32 bit register to handle the function identifier and increasing the parameter width to 64 bits. Suggested-by: Andrew Jones <drjones(a)redhat.com> Signed-off-by: Oliver Upton <oupton(a)google.com> Reviewed-by: Andrew Jones <drjones(a)redhat.com> Message-Id: <20210921171121.2148982-3-oupton(a)google.com> Signed-off-by: Paolo Bonzini <pbonzini(a)redhat.com> Signed-off-by: Sasha Levin <sashal(a)kernel.org> --- tools/testing/selftests/kvm/steal_time.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/kvm/steal_time.c b/tools/testing/selftests/kvm/steal_time.c index ecec30865a74..aafaa8e38b7c 100644 --- a/tools/testing/selftests/kvm/steal_time.c +++ b/tools/testing/selftests/kvm/steal_time.c @@ -118,12 +118,12 @@ struct st_time { uint64_t st_time; }; -static int64_t smccc(uint32_t func, uint32_t arg) +static int64_t smccc(uint32_t func, uint64_t arg) { unsigned long ret; asm volatile( - "mov x0, %1\n" + "mov w0, %w1\n" "mov x1, %2\n" "hvc #0\n" "mov %0, x0\n" -- 2.33.0

3 years, 7 months

2
1
0 0

[PATCH v2] selftests/gpio: Fix gpio compiling error

by Li Zhijian

[root@iaas-rpma gpio]# make gcc gpio-mockup-cdev.c -o /home/lizhijian/linux/tools/testing/selftests/gpio/gpio-mockup-cdev gpio-mockup-cdev.c: In function ‘request_line_v2’: gpio-mockup-cdev.c:24:30: error: storage size of ‘req’ isn’t known 24 | struct gpio_v2_line_request req; | ^~~ gpio-mockup-cdev.c:32:14: error: ‘GPIO_V2_LINE_FLAG_OUTPUT’ undeclared (first use in this function); did you mean ‘GPIOLINE_FLAG_IS_OUT’? 32 | if (flags & GPIO_V2_LINE_FLAG_OUTPUT) { | ^~~~~~~~~~~~~~~~~~~~~~~~ gpio-mockup-cdev.c includes <linux/gpio.h> which could be provided by kernel-headers package, and where it's expected to declare GPIO_V2_LINE_FLAG_OUTPUT. However distros or developers will not always install the same kernel-header as we are compiling. So we can tell compiler to search headers from linux tree simply like others, such as sched. CC: Philip Li <philip.li(a)intel.com> Reported-by: kernel test robot <lkp(a)intel.com> Signed-off-by: Li Zhijian <lizhijian(a)cn.fujitsu.com> --- V2: add more details about the fix --- tools/testing/selftests/gpio/Makefile | 1 + 1 file changed, 1 insertion(+) diff --git a/tools/testing/selftests/gpio/Makefile b/tools/testing/selftests/gpio/Makefile index 39f2bbe8dd3d..42ea7d2aa844 100644 --- a/tools/testing/selftests/gpio/Makefile +++ b/tools/testing/selftests/gpio/Makefile @@ -3,5 +3,6 @@ TEST_PROGS := gpio-mockup.sh TEST_FILES := gpio-mockup-sysfs.sh TEST_GEN_PROGS_EXTENDED := gpio-mockup-cdev +CFLAGS += -I../../../../usr/include include ../lib.mk -- 2.31.1

3 years, 7 months

4
3
0 0

[PATCH v2] selftests: Include directories of header files

by Andrew Delgadilo

From: Andrew Delgadillo <adelg(a)google.com> clang complains about using -o when one specifies header files on the commandline. An example can be seen when building the futex selftests: $ make -C tools/testing/selftests TARGETS=futex clang -Wall -g -O2 -Wall -D_GNU_SOURCE -pthread -I../include \ -I../../ -I../../../../../usr/include/ -I/kselftest/usr/include \ futex_wait_timeout.c ../include/futextest.h ../include/atomic.h \ ../include/logging.h -lpthread -lrt -o \ tools/testing/selftests/futex/functional/futex_wait_timeout clang: error: cannot specify -o when generating multiple output files To fix this, instead of specifying the headers explicitliy, we include (with -I) their directories. Note that -I does not work with files, only directories [1,2] There are still errors when building with clang, but this patch does a way with one class of those errors. Tested: No new errors when building with LLVM=0/1. In fact, clang no longer errors out with "cannot specify -o ..." anymore. [1] https://gcc.gnu.org/onlinedocs/gcc-11.2.0/gcc/Directory-Options.html#Direct… [2] https://clang.llvm.org/docs/ClangCommandLineReference.html#cmdoption-clang-… Suggested-by: gthelen(a)google.com Signed-off-by: Andrew Delgadillo <adelg(a)google.com> --- Changes since v1: - Instead of removing the headers from the cmdline altogether, include their directories. - Add more detail to the tested tag. - Add missing Suggested-By tag from original patch. tools/testing/selftests/filesystems/binderfs/Makefile | 4 ++-- tools/testing/selftests/lib.mk | 5 ++++- tools/testing/selftests/x86/Makefile | 4 ++-- 3 files changed, 8 insertions(+), 5 deletions(-) diff --git a/tools/testing/selftests/filesystems/binderfs/Makefile b/tools/testing/selftests/filesystems/binderfs/Makefile index 8af25ae96049..e2f09acb15b4 100644 --- a/tools/testing/selftests/filesystems/binderfs/Makefile +++ b/tools/testing/selftests/filesystems/binderfs/Makefile @@ -1,8 +1,8 @@ # SPDX-License-Identifier: GPL-2.0 -CFLAGS += -I../../../../../usr/include/ -pthread +CFLAGS += -I../../../../../usr/include/ -I$(selfdir) -pthread TEST_GEN_PROGS := binderfs_test -binderfs_test: binderfs_test.c ../../kselftest.h ../../kselftest_harness.h +binderfs_test: binderfs_test.c include ../../lib.mk diff --git a/tools/testing/selftests/lib.mk b/tools/testing/selftests/lib.mk index fa2ac0e56b43..0edf2a421771 100644 --- a/tools/testing/selftests/lib.mk +++ b/tools/testing/selftests/lib.mk @@ -137,12 +137,15 @@ COMPILE.S = $(CC) $(ASFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -c LINK.S = $(CC) $(ASFLAGS) $(CPPFLAGS) $(LDFLAGS) $(TARGET_ARCH) endif + # Selftest makefiles can override those targets by setting # OVERRIDE_TARGETS = 1. ifeq ($(OVERRIDE_TARGETS),) LOCAL_HDRS := $(selfdir)/kselftest_harness.h $(selfdir)/kselftest.h $(OUTPUT)/%:%.c $(LOCAL_HDRS) - $(LINK.c) $(filter-out $(LOCAL_HDRS),$^) $(LDLIBS) -o $@ + $(LINK.c) $(filter-out %.h,$^) \ + $(addprefix -I,$(dir $(filter %.h,$^))) \ + $(LDLIBS) -o $@ $(OUTPUT)/%.o:%.S $(COMPILE.S) $^ -o $@ diff --git a/tools/testing/selftests/x86/Makefile b/tools/testing/selftests/x86/Makefile index b4142cd1c5c2..f794d431f289 100644 --- a/tools/testing/selftests/x86/Makefile +++ b/tools/testing/selftests/x86/Makefile @@ -72,10 +72,10 @@ all_64: $(BINARIES_64) EXTRA_CLEAN := $(BINARIES_32) $(BINARIES_64) $(BINARIES_32): $(OUTPUT)/%_32: %.c helpers.h - $(CC) -m32 -o $@ $(CFLAGS) $(EXTRA_CFLAGS) $^ -lrt -ldl -lm + $(CC) -m32 -o $@ $(CFLAGS) $(EXTRA_CFLAGS) $(filter-out %.h,$^) $(addprefix -I,$(dir $(filter %.h,$^))) -lrt -ldl -lm $(BINARIES_64): $(OUTPUT)/%_64: %.c helpers.h - $(CC) -m64 -o $@ $(CFLAGS) $(EXTRA_CFLAGS) $^ -lrt -ldl + $(CC) -m64 -o $@ $(CFLAGS) $(EXTRA_CFLAGS) $(filter-out %.h,$^) $(addprefix -I,$(dir $(filter %.h,$^))) -lrt -ldl # x86_64 users should be encouraged to install 32-bit libraries ifeq ($(CAN_BUILD_I386)$(CAN_BUILD_X86_64),01) -- 2.33.0.800.g4c38ced690-goog

3 years, 7 months

1
0
0 0

[PATCH] kunit: tool: show list of valid --arch options when invalid

by Daniel Latypov

Consider this attempt to run KUnit in QEMU: $ ./tools/testing/kunit/kunit.py run --arch=x86 Before you'd get this error message: kunit_kernel.ConfigError: x86 is not a valid arch After: kunit_kernel.ConfigError: x86 is not a valid arch, options are ['alpha', 'arm', 'arm64', 'i386', 'powerpc', 'riscv', 's390', 'sparc', 'x86_64'] This should make it a bit easier for people to notice when they make typos, etc. Currently, one would have to dive into the python code to figure out what the valid set is. Signed-off-by: Daniel Latypov <dlatypov(a)google.com> --- tools/testing/kunit/kunit_kernel.py | 5 +++-- tools/testing/kunit/kunit_tool_test.py | 4 ++++ 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/tools/testing/kunit/kunit_kernel.py b/tools/testing/kunit/kunit_kernel.py index 1870e75ff153..a6b3cee3f0d0 100644 --- a/tools/testing/kunit/kunit_kernel.py +++ b/tools/testing/kunit/kunit_kernel.py @@ -198,8 +198,9 @@ def get_source_tree_ops(arch: str, cross_compile: Optional[str]) -> LinuxSourceT return LinuxSourceTreeOperationsUml(cross_compile=cross_compile) elif os.path.isfile(config_path): return get_source_tree_ops_from_qemu_config(config_path, cross_compile)[1] - else: - raise ConfigError(arch + ' is not a valid arch') + + options = [f[:-3] for f in os.listdir(QEMU_CONFIGS_DIR) if f.endswith('.py')] + raise ConfigError(arch + ' is not a valid arch, options are ' + str(sorted(options))) def get_source_tree_ops_from_qemu_config(config_path: str, cross_compile: Optional[str]) -> Tuple[ diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index cad37a98e599..2ae72f04cbe0 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -289,6 +289,10 @@ class LinuxSourceTreeTest(unittest.TestCase): pass kunit_kernel.LinuxSourceTree('', kunitconfig_path=dir) + def test_invalid_arch(self): + with self.assertRaisesRegex(kunit_kernel.ConfigError, 'not a valid arch, options are.*x86_64'): + kunit_kernel.LinuxSourceTree('', arch='invalid') + # TODO: add more test cases. base-commit: 865a0a8025ee0b54d1cc74834c57197d184a441e -- 2.33.0.685.g46640cef36-goog

3 years, 7 months

3
3
0 0

[PATCH] kunit: fix kernel-doc warnings in doc. build

by Randy Dunlap

Fix documentation build warnings in <kunit/test.h>: ../include/kunit/test.h:616: warning: Function parameter or member 'flags' not described in 'kunit_kmalloc_array' ../include/kunit/test.h:616: warning: Excess function parameter 'gfp' description in 'kunit_kmalloc_array' ../include/kunit/test.h:661: warning: Function parameter or member 'flags' not described in 'kunit_kcalloc' ../include/kunit/test.h:661: warning: Excess function parameter 'gfp' description in 'kunit_kcalloc' Fixes: 0a756853586c ("kunit: test: add test resource management API") Fixes: 7122debb4367 ("kunit: introduce kunit_kmalloc_array/kunit_kcalloc() helpers") Signed-off-by: Randy Dunlap <rdunlap(a)infradead.org> Cc: Brendan Higgins <brendanhiggins(a)google.com> Cc: linux-kselftest(a)vger.kernel.org Cc: kunit-dev(a)googlegroups.com Cc: Daniel Latypov <dlatypov(a)google.com> Cc: Shuah Khan <skhan(a)linuxfoundation.org> --- include/kunit/test.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) --- lnx-515-rc4.orig/include/kunit/test.h +++ lnx-515-rc4/include/kunit/test.h @@ -607,7 +607,7 @@ void kunit_remove_resource(struct kunit * @test: The test context object. * @n: number of elements. * @size: The size in bytes of the desired memory. - * @gfp: flags passed to underlying kmalloc(). + * @flags: gfp flags passed to underlying kmalloc(). * * Just like `kmalloc_array(...)`, except the allocation is managed by the test case * and is automatically cleaned up after the test case concludes. See &struct @@ -653,7 +653,7 @@ static inline void *kunit_kzalloc(struct * @test: The test context object. * @n: number of elements. * @size: The size in bytes of the desired memory. - * @gfp: flags passed to underlying kmalloc(). + * @flags: gfp flags passed to underlying kmalloc(). * * See kcalloc() and kunit_kmalloc_array() for more information. */

3 years, 7 months

2
3
0 0

[PATCH] selftests: Remove explicit headers for clang

by Andrew Delgadilo

From: Andrew Delgadillo <adelg(a)google.com> GCC allows paths to header files to be passed on the command line while using -o, but clang does not: $ make -C tools/testing/selftests TARGETS=futex $ make -C tools/testing/selftests TARGETS=futex LLVM=1 clang -Wall -g -O2 -Wall -D_GNU_SOURCE -pthread -I../include \ -I../../ -I../../../../../usr/include/ -I/kselftest/usr/include \ futex_wait_timeout.c ../include/futextest.h ../include/atomic.h \ ../include/logging.h -lpthread -lrt -o \ tools/testing/selftests/futex/functional/futex_wait_timeout clang: error: cannot specify -o when generating multiple output files To fix this, remove explicit paths to headers from the commandline in lib.mk. We must explicitly remove them for x86 and binderfs as they are not filtered out by the change to lib.mk, but the compiler search paths for includes are already setup correctly, so the compiler finds the correct headers. Tested: selftests build with LLVM=1 now. Cc: stable(a)vger.kernel.org Signed-off-by: Andrew Delgadillo <adelg(a)google.com> --- tools/testing/selftests/filesystems/binderfs/Makefile | 2 +- tools/testing/selftests/lib.mk | 2 +- tools/testing/selftests/x86/Makefile | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/tools/testing/selftests/filesystems/binderfs/Makefile b/tools/testing/selftests/filesystems/binderfs/Makefile index 8af25ae96049..58e41bd98200 100644 --- a/tools/testing/selftests/filesystems/binderfs/Makefile +++ b/tools/testing/selftests/filesystems/binderfs/Makefile @@ -3,6 +3,6 @@ CFLAGS += -I../../../../../usr/include/ -pthread TEST_GEN_PROGS := binderfs_test -binderfs_test: binderfs_test.c ../../kselftest.h ../../kselftest_harness.h +binderfs_test: binderfs_test.c include ../../lib.mk diff --git a/tools/testing/selftests/lib.mk b/tools/testing/selftests/lib.mk index fa2ac0e56b43..fb152e20c86a 100644 --- a/tools/testing/selftests/lib.mk +++ b/tools/testing/selftests/lib.mk @@ -142,7 +142,7 @@ endif ifeq ($(OVERRIDE_TARGETS),) LOCAL_HDRS := $(selfdir)/kselftest_harness.h $(selfdir)/kselftest.h $(OUTPUT)/%:%.c $(LOCAL_HDRS) - $(LINK.c) $(filter-out $(LOCAL_HDRS),$^) $(LDLIBS) -o $@ + $(LINK.c) $(filter-out %.h,$^) $(LDLIBS) -o $@ $(OUTPUT)/%.o:%.S $(COMPILE.S) $^ -o $@ diff --git a/tools/testing/selftests/x86/Makefile b/tools/testing/selftests/x86/Makefile index b4142cd1c5c2..68967006b3e9 100644 --- a/tools/testing/selftests/x86/Makefile +++ b/tools/testing/selftests/x86/Makefile @@ -72,10 +72,10 @@ all_64: $(BINARIES_64) EXTRA_CLEAN := $(BINARIES_32) $(BINARIES_64) $(BINARIES_32): $(OUTPUT)/%_32: %.c helpers.h - $(CC) -m32 -o $@ $(CFLAGS) $(EXTRA_CFLAGS) $^ -lrt -ldl -lm + $(CC) -m32 -o $@ $(CFLAGS) $(EXTRA_CFLAGS) $(filter-out %.h,$^) -lrt -ldl -lm $(BINARIES_64): $(OUTPUT)/%_64: %.c helpers.h - $(CC) -m64 -o $@ $(CFLAGS) $(EXTRA_CFLAGS) $^ -lrt -ldl + $(CC) -m64 -o $@ $(CFLAGS) $(EXTRA_CFLAGS) $(filter-out %.h,$^) -lrt -ldl # x86_64 users should be encouraged to install 32-bit libraries ifeq ($(CAN_BUILD_I386)$(CAN_BUILD_X86_64),01) -- 2.33.0.800.g4c38ced690-goog

3 years, 7 months

3
3
0 0

[PATCH v4] kunit: tool: improve compatibility of kunit_parser with KTAP specification

by Daniel Latypov

From: Rae Moar <rmoar(a)google.com> Update to kunit_parser to improve compatibility with KTAP specification including arbitrarily nested tests. Patch accomplishes three major changes: - Use a general Test object to represent all tests rather than TestCase and TestSuite objects. This allows for easier implementation of arbitrary levels of nested tests and promotes the idea that both test suites and test cases are tests. - Print errors incrementally rather than all at once after the parsing finishes to maximize information given to the user in the case of the parser given invalid input and to increase the helpfulness of the timestamps given during printing. Note that kunit.py parse does not print incrementally yet. However, this fix brings us closer to this feature. - Increase compatibility for different formats of input. Arbitrary levels of nested tests supported. Also, test cases and test suites are now supported to be present on the same level of testing. This patch now implements the KTAP specification as described here: https://lore.kernel.org/linux-kselftest/CA+GJov6tdjvY9x12JsJT14qn6c7NViJxqa…. This patch adjusts the kunit_tool_test.py file to check for the correct outputs from the new parser and adds a new test to check the parsing for a KTAP result log with correct format for multiple nested subtests (test_is_test_passed-all_passed_nested.log). This patch also alters the kunit_json.py file to allow for arbitrarily nested tests. Signed-off-by: Rae Moar <rmoar(a)google.com> Reviewed-by: Brendan Higgins <brendanhiggins(a)google.com> Signed-off-by: Daniel Latypov <dlatypov(a)google.com> --- NOTE: this patch is now applied on top of https://lore.kernel.org/linux-kselftest/20210930222048.1692635-5-dlatypov@g… to resolve a conflict. Change log from v3: https://lore.kernel.org/linux-kselftest/20210901190623.315736-1-rmoar@googl… - Resolve conflict with hermetic testing patches - Fix pytype error (str -> Optional[str] in kunit_json.py) Change log from v2: https://lore.kernel.org/linux-kselftest/20210826195505.3066755-1-rmoar@goog… - Fixes bug of type disagreement in kunit_json.py for build_dir - Removes raw_output() - Changes docstrings in kunit_parser.py (class docstring, LineStream docstrings, add_error(), total(), get_status(), all parsing methods) - Fixes bug of not printing diagnostic log in the case of end of lines - Sets default status of all tests to TEST_CRASHED - Adds and prints empty tests with crashed status in case of missing tests - Prints 'subtest' in instance of 1 subtest instead of 'subtests' - Includes checking for 'BUG:' message in search of crash messages in log (note that parse_crash_in_log method could be removed but would require deleting tests in kunit_tool_test.py that include the crash message that is no longer used. If removed, parser would still print log in cases of test crashed or failure, which would now include missing subtests) - Fixes bug of including directives (other than SKIP) in test name when matching name in result line for subtests Change log from v1: https://lore.kernel.org/linux-kselftest/20210820200032.2178134-1-rmoar@goog… - Rebase onto kselftest/kunit branch - Add tests to kunit_tool_test.py to check parser is correctly stripping hyphen, producing correct json objects with nested tests, correctly passing kselftest TAP output, and correctly deals with missing test plan. - Fix bug to correctly match test name in instance of a missing test plan. - Fix bug in kunit_tool_test.py pointed out by Daniel where it was not correctly checking for a proper match to the '0 tests run!' error message. Reverts changes back to original. - A few minor changes to commit message using Daniel's comments. - Change docstrings using Daniel's comments to reduce: - Shortens some docstrings to be one-line or just description if it is self explanatory. - Remove explicit respecification of types of parameters and returns because this is already specified in the function annoations. However, some descriptions of the parameters and returns remain and some contain the type for context. Additionally, the types of public attributes of classes remain. - Remove any documentation of 'Return: None' - Remove docstrings of helper methods within other methods --- tools/testing/kunit/kunit.py | 6 +- tools/testing/kunit/kunit_json.py | 55 +- tools/testing/kunit/kunit_parser.py | 1056 ++++++++++++----- tools/testing/kunit/kunit_tool_test.py | 134 ++- .../test_is_test_passed-all_passed_nested.log | 34 + .../test_is_test_passed-kselftest.log | 14 + .../test_is_test_passed-missing_plan.log | 31 + .../kunit/test_data/test_strip_hyphen.log | 16 + 8 files changed, 954 insertions(+), 392 deletions(-) create mode 100644 tools/testing/kunit/test_data/test_is_test_passed-all_passed_nested.log create mode 100644 tools/testing/kunit/test_data/test_is_test_passed-kselftest.log create mode 100644 tools/testing/kunit/test_data/test_is_test_passed-missing_plan.log create mode 100644 tools/testing/kunit/test_data/test_strip_hyphen.log diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py index 8c7e8c7b2c97..9e2a96e45e3b 100755 --- a/tools/testing/kunit/kunit.py +++ b/tools/testing/kunit/kunit.py @@ -136,7 +136,7 @@ def exec_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest, test_glob = request.filter_glob.split('.', maxsplit=2)[1] filter_globs = [g + '.'+ test_glob for g in filter_globs] - overall_status = kunit_parser.TestStatus.SUCCESS + test_counts = kunit_parser.TestCounts() exec_time = 0.0 for i, filter_glob in enumerate(filter_globs): kunit_parser.print_with_timestamp('Starting KUnit Kernel ({}/{})...'.format(i+1, len(filter_globs))) @@ -155,9 +155,9 @@ def exec_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest, test_end = time.time() exec_time += test_end - test_start - overall_status = kunit_parser.max_status(overall_status, result.status) + test_counts.add_subtest_counts(result.result.test.counts) - return KunitResult(status=result.status, result=result.result, elapsed_time=exec_time) + return KunitResult(status=test_counts.get_status(), result=result.result, elapsed_time=exec_time) def parse_tests(request: KunitParseRequest, input_data: Iterable[str]) -> KunitResult: parse_start = time.time() diff --git a/tools/testing/kunit/kunit_json.py b/tools/testing/kunit/kunit_json.py index f5cca5c38cac..d01581d068ac 100644 --- a/tools/testing/kunit/kunit_json.py +++ b/tools/testing/kunit/kunit_json.py @@ -11,47 +11,46 @@ import os import kunit_parser -from kunit_parser import TestStatus - -def get_json_result(test_result, def_config, build_dir, json_path) -> str: - sub_groups = [] - - # Each test suite is mapped to a KernelCI sub_group - for test_suite in test_result.suites: - sub_group = { - "name": test_suite.name, - "arch": "UM", - "defconfig": def_config, - "build_environment": build_dir, - "test_cases": [], - "lab_name": None, - "kernel": None, - "job": None, - "git_branch": "kselftest", - } - test_cases = [] - # TODO: Add attachments attribute in test_case with detailed - # failure message, see https://api.kernelci.org/schema-test-case.html#get - for case in test_suite.cases: - test_case = {"name": case.name, "status": "FAIL"} - if case.status == TestStatus.SUCCESS: +from kunit_parser import Test, TestResult, TestStatus +from typing import Any, Dict, Optional + +JsonObj = Dict[str, Any] + +def _get_group_json(test: Test, def_config: str, build_dir: Optional[str]) -> JsonObj: + sub_groups = [] # List[JsonObj] + test_cases = [] # List[JsonObj] + + for subtest in test.subtests: + if len(subtest.subtests): + sub_group = _get_group_json(subtest, def_config, + build_dir) + sub_groups.append(sub_group) + else: + test_case = {"name": subtest.name, "status": "FAIL"} + if subtest.status == TestStatus.SUCCESS: test_case["status"] = "PASS" - elif case.status == TestStatus.TEST_CRASHED: + elif subtest.status == TestStatus.TEST_CRASHED: test_case["status"] = "ERROR" test_cases.append(test_case) - sub_group["test_cases"] = test_cases - sub_groups.append(sub_group) + test_group = { - "name": "KUnit Test Group", + "name": test.name, "arch": "UM", "defconfig": def_config, "build_environment": build_dir, "sub_groups": sub_groups, + "test_cases": test_cases, "lab_name": None, "kernel": None, "job": None, "git_branch": "kselftest", } + return test_group + +def get_json_result(test_result: TestResult, def_config: str, build_dir: Optional[str], + json_path: str) -> str: + test_group = _get_group_json(test_result.test, def_config, build_dir) + test_group["name"] = "KUnit Test Group" json_obj = json.dumps(test_group, indent=4) if json_path != 'stdout': with open(json_path, 'w') as result_path: diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py index 6310a641b151..4b6086159c7f 100644 --- a/tools/testing/kunit/kunit_parser.py +++ b/tools/testing/kunit/kunit_parser.py @@ -1,11 +1,15 @@ # SPDX-License-Identifier: GPL-2.0 # -# Parses test results from a kernel dmesg log. +# Parses KTAP test results from a kernel dmesg log and incrementally prints +# results with reader-friendly format. Stores and returns test results in a +# Test object. # # Copyright (C) 2019, Google LLC. # Author: Felix Guo <felixguoxiuping(a)gmail.com> # Author: Brendan Higgins <brendanhiggins(a)google.com> +# Author: Rae Moar <rmoar(a)google.com> +from __future__ import annotations import re from collections import namedtuple @@ -14,33 +18,55 @@ from enum import Enum, auto from functools import reduce from typing import Iterable, Iterator, List, Optional, Tuple -TestResult = namedtuple('TestResult', ['status','suites','log']) - -class TestSuite(object): - def __init__(self) -> None: - self.status = TestStatus.SUCCESS - self.name = '' - self.cases = [] # type: List[TestCase] - - def __str__(self) -> str: - return 'TestSuite(' + str(self.status) + ',' + self.name + ',' + str(self.cases) + ')' - - def __repr__(self) -> str: - return str(self) - -class TestCase(object): +TestResult = namedtuple('TestResult', ['status','test','log']) + +class Test(object): + """ + A class to represent a test parsed from KTAP results. All KTAP + results within a test log are stored in a main Test object as + subtests. + + Attributes: + status : TestStatus - status of the test + name : str - name of the test + expected_count : int - expected number of subtests (0 if single + test case and None if unknown expected number of subtests) + subtests : List[Test] - list of subtests + log : List[str] - log of KTAP lines that correspond to the test + counts : TestCounts - counts of the test statuses and errors of + subtests or of the test itself if the test is a single + test case. + """ def __init__(self) -> None: + """Constructs the default attributes of a Test class object. + """ self.status = TestStatus.SUCCESS self.name = '' + self.expected_count = 0 # type: Optional[int] + self.subtests = [] # type: List[Test] self.log = [] # type: List[str] + self.counts = TestCounts() def __str__(self) -> str: - return 'TestCase(' + str(self.status) + ',' + self.name + ',' + str(self.log) + ')' + """Returns string representation of a Test class object.""" + return ('Test(' + str(self.status) + ', ' + self.name + + ', ' + str(self.expected_count) + ', ' + + str(self.subtests) + ', ' + str(self.log) + ', ' + + str(self.counts) + ')') def __repr__(self) -> str: + """Returns string representation of a Test class object.""" return str(self) + def add_error(self, error_message: str) -> None: + """Adds error to test object by incrementing the error count + and printing the error message. + """ + self.counts.errors += 1 + print_error('Test ' + self.name + ': ' + error_message) + class TestStatus(Enum): + """An enumeration class to represent the status of a test.""" SUCCESS = auto() FAILURE = auto() SKIPPED = auto() @@ -48,381 +74,769 @@ class TestStatus(Enum): NO_TESTS = auto() FAILURE_TO_PARSE_TESTS = auto() +class TestCounts: + """ + A class to represent the counts of statuses and test errors of + subtests or of the test itself if the test is a single test case with + no subtests. Note that the sum of the counts of passed, failed, + crashed, and skipped should sum to the total number of subtests for + the test. + + Attributes: + passed : int - the number of tests that have passed + failed : int - the number of tests that have failed + crashed : int - the number of tests that have crashed + skipped : int - the number of tests that have skipped + errors : int - the number of errors in the test and subtests + """ + def __init__(self): + """Contructs the default attributes of a TestCounts class + object. Sets the counts of all test statuses and test + errors to be 0. + """ + self.passed = 0 + self.failed = 0 + self.crashed = 0 + self.skipped = 0 + self.errors = 0 + + def __str__(self) -> str: + """Returns the string representation of a TestCounts object. + """ + return ('Passed: ' + str(self.passed) + + ', Failed: ' + str(self.failed) + + ', Crashed: ' + str(self.crashed) + + ', Skipped: ' + str(self.skipped) + + ', Errors: ' + str(self.errors)) + + def total(self) -> int: + """Returns total number of subtests or 1 if the test object + has no subtests to represent the test itself. This number is + calculated by the sum of the passed, failed, crashed, and + skipped subtests. + """ + return (self.passed + self.failed + self.crashed + + self.skipped) + + def add_subtest_counts(self, counts: TestCounts) -> None: + """ + Adds the counts of another TestCounts object to the current + TestCounts object. Used to add the counts of a subtest to the + parent test. + + Parameters: + counts - a different TestCounts object whose counts + will be added to the counts of the TestCounts object + """ + self.passed += counts.passed + self.failed += counts.failed + self.crashed += counts.crashed + self.skipped += counts.skipped + self.errors += counts.errors + + def get_status(self) -> TestStatus: + """Returns the expected status of a Test using test counts.""" + if self.crashed: + # If one of the subtests crash, the expected status + # of the Test is crashed. + return TestStatus.TEST_CRASHED + elif self.failed: + # Otherwise if one of the subtests fail, the + # expected status of the Test is failed. + return TestStatus.FAILURE + elif self.passed: + # Otherwise if one of the subtests pass, the + # expected status of the Test is passed. + return TestStatus.SUCCESS + else: + # Finally, if none of the subtests have failed, + # crashed, or passed, the expected status of the + # Test is skipped. + return TestStatus.SKIPPED + + def add_status(self, status: TestStatus) -> None: + """ + Given inputted status, increments corresponding attribute of + TestCounts object. + + Parameters: + status - status to be added to the TestCounts object + """ + if status == TestStatus.SUCCESS or \ + status == TestStatus.NO_TESTS: + # if status is NO_TESTS the most appropriate + # attribute to increment is passed because + # the test did not fail, crash or get skipped. + self.passed += 1 + elif status == TestStatus.FAILURE: + self.failed += 1 + elif status == TestStatus.SKIPPED: + self.skipped += 1 + else: + self.crashed += 1 + class LineStream: - """Provides a peek()/pop() interface over an iterator of (line#, text).""" + """ + A class to represent the lines of kernel output. + Provides a peek()/pop() interface over an iterator of + (line#, text). + """ _lines: Iterator[Tuple[int, str]] _next: Tuple[int, str] _done: bool def __init__(self, lines: Iterator[Tuple[int, str]]): + """Set defaults for LineStream object and sets _lines + attribute to lines parameter. + """ self._lines = lines self._done = False self._next = (0, '') self._get_next() def _get_next(self) -> None: + """Advances the LineSteam to the next line or sets the _done + attribute if the LineStream has reached the end of the lines. + """ try: self._next = next(self._lines) except StopIteration: self._done = True def peek(self) -> str: + """Returns the next line in the LineStream without advancing + the LineStream. + """ return self._next[1] def pop(self) -> str: + """Returns the next line in the LineStream and advances the + LineStream to the next line. + """ n = self._next self._get_next() return n[1] def __bool__(self) -> bool: + """Returns whether the LineStream has reached the end of the + lines. + """ return not self._done # Only used by kunit_tool_test.py. def __iter__(self) -> Iterator[str]: + """Empties all lines stored in LineStream object into + Iterator object and returns the Iterator object. + """ while bool(self): yield self.pop() def line_number(self) -> int: + """Returns the line number of the next line in the + LineStream. + """ return self._next[0] -kunit_start_re = re.compile(r'TAP version [0-9]+$') -kunit_end_re = re.compile('(List of all partitions:|' - 'Kernel panic - not syncing: VFS:|reboot: System halted)') +# Parsing helper methods: + +KTAP_START = re.compile(r'KTAP version ([0-9]+)$') +TAP_START = re.compile(r'TAP version ([0-9]+)$') +KTAP_END = re.compile('(List of all partitions:|' + 'Kernel panic - not syncing: VFS:|reboot: System halted)') def extract_tap_lines(kernel_output: Iterable[str]) -> LineStream: - def isolate_kunit_output(kernel_output: Iterable[str]) -> Iterator[Tuple[int, str]]: + """Extracts KTAP lines from inputted kernel output in LineStream + object.""" + def isolate_ktap_output(kernel_output: Iterable[str]) \ + -> Iterator[Tuple[int, str]]: line_num = 0 started = False for line in kernel_output: line_num += 1 - line = line.rstrip() # line always has a trailing \n - if kunit_start_re.search(line): + line = line.rstrip() # remove trailing \n + if not started and KTAP_START.search(line): + # start extracting KTAP lines and set prefix + # to number of characters before version line + prefix_len = len( + line.split('KTAP version')[0]) + started = True + yield line_num, line[prefix_len:] + elif not started and TAP_START.search(line): + # start extracting KTAP lines and set prefix + # to number of characters before version line prefix_len = len(line.split('TAP version')[0]) started = True yield line_num, line[prefix_len:] - elif kunit_end_re.search(line): + elif started and KTAP_END.search(line): + # stop extracting KTAP lines break elif started: - yield line_num, line[prefix_len:] - return LineStream(lines=isolate_kunit_output(kernel_output)) - -DIVIDER = '=' * 60 - -RESET = '\033[0;0m' - -def red(text) -> str: - return '\033[1;31m' + text + RESET - -def yellow(text) -> str: - return '\033[1;33m' + text + RESET - -def green(text) -> str: - return '\033[1;32m' + text + RESET - -def print_with_timestamp(message) -> None: - print('[%s] %s' % (datetime.now().strftime('%H:%M:%S'), message)) - -def format_suite_divider(message) -> str: - return '======== ' + message + ' ========' - -def print_suite_divider(message) -> None: - print_with_timestamp(DIVIDER) - print_with_timestamp(format_suite_divider(message)) - -def print_log(log) -> None: - for m in log: - print_with_timestamp(m) - -TAP_ENTRIES = re.compile(r'^(TAP|[\s]*ok|[\s]*not ok|[\s]*[0-9]+\.\.[0-9]+|[\s]*# (Subtest:|.*: kunit test case crashed!)).*$') - -def consume_non_diagnostic(lines: LineStream) -> None: - while lines and not TAP_ENTRIES.match(lines.peek()): - lines.pop() - -def save_non_diagnostic(lines: LineStream, test_case: TestCase) -> None: - while lines and not TAP_ENTRIES.match(lines.peek()): - test_case.log.append(lines.peek()) - lines.pop() - -OkNotOkResult = namedtuple('OkNotOkResult', ['is_ok','description', 'text']) - -OK_NOT_OK_SKIP = re.compile(r'^[\s]*(ok|not ok) [0-9]+ - (.*) # SKIP(.*)$') - -OK_NOT_OK_SUBTEST = re.compile(r'^[\s]+(ok|not ok) [0-9]+ - (.*)$') - -OK_NOT_OK_MODULE = re.compile(r'^(ok|not ok) ([0-9]+) - (.*)$') - -def parse_ok_not_ok_test_case(lines: LineStream, test_case: TestCase) -> bool: - save_non_diagnostic(lines, test_case) - if not lines: - test_case.status = TestStatus.TEST_CRASHED - return True - line = lines.peek() - match = OK_NOT_OK_SUBTEST.match(line) - while not match and lines: - line = lines.pop() - match = OK_NOT_OK_SUBTEST.match(line) - if match: - test_case.log.append(lines.pop()) - test_case.name = match.group(2) - skip_match = OK_NOT_OK_SKIP.match(line) - if skip_match: - test_case.status = TestStatus.SKIPPED - return True - if test_case.status == TestStatus.TEST_CRASHED: - return True - if match.group(1) == 'ok': - test_case.status = TestStatus.SUCCESS - else: - test_case.status = TestStatus.FAILURE - return True + # remove prefix and any indention and yield + # line with line number + line = line[prefix_len:].lstrip() + yield line_num, line + return LineStream(lines=isolate_ktap_output(kernel_output)) + +def raw_output(kernel_output: Iterable[str]) -> None: + """Prints all lines of kernel output.""" + for line in kernel_output: + print(line.rstrip()) + +KTAP_VERSIONS = [1] +TAP_VERSIONS = [13, 14] + +def check_version(version_num: int, accepted_versions: List[int], + version_type: str, test: Test) -> None: + """ + Adds error to test object if version number is too high or too + low. + + Parameters: + version_num - The inputted version number from the parsed KTAP or TAP + header line + accepted_version - List of accepted KTAP or TAP versions + version_type - 'KTAP' or 'TAP' depending on the type of + version line. + test - Test object for current test being parsed + """ + if version_num < min(accepted_versions): + test.add_error(version_type + + ' version lower than expected!') + elif version_num > max(accepted_versions): + test.add_error( + version_type + ' version higher than expected!') + +def parse_ktap_header(lines: LineStream, test: Test) -> bool: + """ + If the next line in LineStream matches the format of KTAP or TAP + header line, the version number is checked, the line is popped, + and returns True. Otherwise the method returns False. + + Accepted formats: + - 'KTAP version [version number]' + - 'TAP version [version number]' + + Parameters: + lines - LineStream of KTAP output to parse + test - Test object for current test being parsed + + Return: + Boolean that represents if the next line in the LineStream was parsed + as the KTAP or TAP header line + """ + ktap_match = KTAP_START.match(lines.peek()) + tap_match = TAP_START.match(lines.peek()) + if ktap_match: + version_num = int(ktap_match.group(1)) + check_version(version_num, KTAP_VERSIONS, 'KTAP', test) + elif tap_match: + version_num = int(tap_match.group(1)) + check_version(version_num, TAP_VERSIONS, 'TAP', test) else: return False - -SUBTEST_DIAGNOSTIC = re.compile(r'^[\s]+# (.*)$') -DIAGNOSTIC_CRASH_MESSAGE = re.compile(r'^[\s]+# .*?: kunit test case crashed!$') - -def parse_diagnostic(lines: LineStream, test_case: TestCase) -> bool: - save_non_diagnostic(lines, test_case) - if not lines: + test.log.append(lines.pop()) + return True + +TEST_HEADER = re.compile(r'^# Subtest: (.*)$') + +def parse_test_header(lines: LineStream, test: Test) -> bool: + """ + If the next line in LineStream matches the format of a test + header line, the name of test is set, the line is popped, + and returns True. Otherwise the method returns False. + + Accepted format: + - '# Subtest: [test name]' + + Parameters: + lines - LineStream of ktap output to parse + test - Test object for current test being parsed + + Return: + Boolean that represents if the next line in the LineStream was parsed + as a test header + """ + match = TEST_HEADER.match(lines.peek()) + if not match: return False + test.log.append(lines.pop()) + test.name = match.group(1) + return True + +TEST_PLAN = re.compile(r'1\.\.([0-9]+)') + +def parse_test_plan(lines: LineStream, test: Test) -> bool: + """ + If the next line in LineStream matches the format of a test + plan line, the expected number of subtests is set in test object, an + error is thrown if there are 0 tests, the line is popped, + and returns True. Otherwise the method adds an error that the test + plan is missing to the test object and returns False. + + Accepted format: + - '1..[number of subtests]' + + Parameters: + lines - LineStream of ktap output to parse + test - Test object for current test being parsed + + Return: + Boolean that represents if the next line in the LineStream was parsed + as a test plan + """ + match = TEST_PLAN.match(lines.peek()) + if not match: + test.expected_count = None + test.add_error('missing plan line!') + return False + test.log.append(lines.pop()) + expected_count = int(match.group(1)) + test.expected_count = expected_count + if expected_count == 0: + test.status = TestStatus.NO_TESTS + test.add_error('0 tests run!') + return True + +TEST_RESULT = re.compile(r'^(ok|not ok) ([0-9]+) (- )?(.*)$') + +TEST_RESULT_SKIP = re.compile(r'^(ok|not ok) ([0-9]+) (- )?(.*) # SKIP(.*)$') + +def peek_test_name_match(lines: LineStream, test: Test) -> bool: + """ + If the next line in LineStream matches the format of a test + result line and the name of the result line matches the name of the + current test, the method returns True. Otherwise it returns False. + + Accepted format: + - '[ok|not ok] [test number] [-] [test name] [optional skip + directive]' + + Parameters: + lines - LineStream of KTAP output to parse + test - Test object for current test being parsed + + Return: + Boolean that represents if the next line in the LineStream matched a + test result line and the name matched the expected test name + """ line = lines.peek() - match = SUBTEST_DIAGNOSTIC.match(line) - if match: - test_case.log.append(lines.pop()) - crash_match = DIAGNOSTIC_CRASH_MESSAGE.match(line) - if crash_match: - test_case.status = TestStatus.TEST_CRASHED - return True - else: + match = TEST_RESULT.match(line) + if not match: return False + name = match.group(4) + return (name == test.name) + +def parse_test_result(lines: LineStream, test: Test, + expected_num: int) -> bool: + """ + If the next line in LineStream matches the format of a test + result line, the status in the result line is added to the test + object, the test number is checked to match the expected test number + and if not an error is added to the test object, and returns True. + Otherwise it returns False. + + Note that the skip diirective is the only + directive that causes a change in status and otherwise the directive + is included in the name of the test. + + Accepted format: + - '[ok|not ok] [test number] [-] [test name] [optional skip + directive]' + + Parameters: + lines - LineStream of KTAP output to parse + test - Test object for current test being parsed + expected_num - expected test number for current test + + Return: + Boolean that represents if the next line in the LineStream was parsed + as a test result line. + """ + line = lines.peek() + match = TEST_RESULT.match(line) + skip_match = TEST_RESULT_SKIP.match(line) -def parse_test_case(lines: LineStream) -> Optional[TestCase]: - test_case = TestCase() - save_non_diagnostic(lines, test_case) - while parse_diagnostic(lines, test_case): - pass - if parse_ok_not_ok_test_case(lines, test_case): - return test_case - else: - return None - -SUBTEST_HEADER = re.compile(r'^[\s]+# Subtest: (.*)$') + # Check if line matches test result line format + if not match: + return False + test.log.append(lines.pop()) -def parse_subtest_header(lines: LineStream) -> Optional[str]: - consume_non_diagnostic(lines) - if not lines: - return None - match = SUBTEST_HEADER.match(lines.peek()) - if match: - lines.pop() - return match.group(1) + # Set name of test object + if skip_match: + test.name = skip_match.group(4) else: - return None + test.name = match.group(4) -SUBTEST_PLAN = re.compile(r'[\s]+[0-9]+\.\.([0-9]+)') + # Check test num + num = int(match.group(2)) + if num != expected_num: + test.add_error('Expected test number ' + + str(expected_num) + ' but found ' + str(num)) -def parse_subtest_plan(lines: LineStream) -> Optional[int]: - consume_non_diagnostic(lines) - match = SUBTEST_PLAN.match(lines.peek()) - if match: - lines.pop() - return int(match.group(1)) - else: - return None - -def max_status(left: TestStatus, right: TestStatus) -> TestStatus: - if left == right: - return left - elif left == TestStatus.TEST_CRASHED or right == TestStatus.TEST_CRASHED: - return TestStatus.TEST_CRASHED - elif left == TestStatus.FAILURE or right == TestStatus.FAILURE: - return TestStatus.FAILURE - elif left == TestStatus.SKIPPED: - return right - else: - return left - -def parse_ok_not_ok_test_suite(lines: LineStream, - test_suite: TestSuite, - expected_suite_index: int) -> bool: - consume_non_diagnostic(lines) - if not lines: - test_suite.status = TestStatus.TEST_CRASHED - return False - line = lines.peek() - match = OK_NOT_OK_MODULE.match(line) - if match: - lines.pop() - if match.group(1) == 'ok': - test_suite.status = TestStatus.SUCCESS - else: - test_suite.status = TestStatus.FAILURE - skip_match = OK_NOT_OK_SKIP.match(line) - if skip_match: - test_suite.status = TestStatus.SKIPPED - suite_index = int(match.group(2)) - if suite_index != expected_suite_index: - print_with_timestamp( - red('[ERROR] ') + 'expected_suite_index ' + - str(expected_suite_index) + ', but got ' + - str(suite_index)) + # Set status of test object + status = match.group(1) + if test.status == TestStatus.TEST_CRASHED: return True + elif skip_match: + test.status = TestStatus.SKIPPED + elif status == 'ok': + test.status = TestStatus.SUCCESS else: - return False - -def bubble_up_errors(status_list: Iterable[TestStatus]) -> TestStatus: - return reduce(max_status, status_list, TestStatus.SKIPPED) + test.status = TestStatus.FAILURE + return True + +def parse_diagnostic(lines: LineStream) -> List[str]: + """ + If the next line in LineStream does not match the format of a test + case line or test header line, the line is checked if the test has + crashed and if so adds an error message, pops the line and adds it to + the log. + + Line formats that are not parsed: + - '# Subtest: [test name]' + - '[ok|not ok] [test number] [-] [test name] [optional skip + directive]' + + Parameters: + lines - LineStream of KTAP output to parse + + Return: + Log of diagnostic lines + """ + log = [] # type: List[str] + while lines and not TEST_RESULT.match(lines.peek()) and not \ + TEST_HEADER.match(lines.peek()): + log.append(lines.pop()) + return log + +DIAGNOSTIC_CRASH_MESSAGE = re.compile(r'^# .*?: kunit test case crashed!$') + +def parse_crash_in_log(test: Test) -> bool: + """ + Iterate through the lines of the log to parse for crash message. + If crash message found, set status to crashed and return True. + Otherwise return False. + + Parameters: + test - Test object for current test being parsed + + Return: + Boolean that represents if crash message found in log + """ + for line in test.log: + if DIAGNOSTIC_CRASH_MESSAGE.match(line): + test.status = TestStatus.TEST_CRASHED + return True + return False -def bubble_up_test_case_errors(test_suite: TestSuite) -> TestStatus: - max_test_case_status = bubble_up_errors(x.status for x in test_suite.cases) - return max_status(max_test_case_status, test_suite.status) +# Printing helper methods: -def parse_test_suite(lines: LineStream, expected_suite_index: int) -> Optional[TestSuite]: - if not lines: - return None - consume_non_diagnostic(lines) - test_suite = TestSuite() - test_suite.status = TestStatus.SUCCESS - name = parse_subtest_header(lines) - if not name: - return None - test_suite.name = name - expected_test_case_num = parse_subtest_plan(lines) - if expected_test_case_num is None: - return None - while expected_test_case_num > 0: - test_case = parse_test_case(lines) - if not test_case: - break - test_suite.cases.append(test_case) - expected_test_case_num -= 1 - if parse_ok_not_ok_test_suite(lines, test_suite, expected_suite_index): - test_suite.status = bubble_up_test_case_errors(test_suite) - return test_suite - elif not lines: - print_with_timestamp(red('[ERROR] ') + 'ran out of lines before end token') - return test_suite - else: - print(f'failed to parse end of suite "{name}", at line {lines.line_number()}: {lines.peek()}') - return None +DIVIDER = '=' * 60 -TAP_HEADER = re.compile(r'^TAP version 14$') +RESET = '\033[0;0m' -def parse_tap_header(lines: LineStream) -> bool: - consume_non_diagnostic(lines) - if TAP_HEADER.match(lines.peek()): - lines.pop() - return True - else: - return False +def red(text: str) -> str: + """Returns inputted string with red color code.""" + return '\033[1;31m' + text + RESET -TEST_PLAN = re.compile(r'[0-9]+\.\.([0-9]+)') +def yellow(text: str) -> str: + """Returns inputted string with yellow color code.""" + return '\033[1;33m' + text + RESET -def parse_test_plan(lines: LineStream) -> Optional[int]: - consume_non_diagnostic(lines) - match = TEST_PLAN.match(lines.peek()) - if match: - lines.pop() - return int(match.group(1)) - else: - return None - -def bubble_up_suite_errors(test_suites: Iterable[TestSuite]) -> TestStatus: - return bubble_up_errors(x.status for x in test_suites) - -def parse_test_result(lines: LineStream) -> TestResult: - consume_non_diagnostic(lines) - if not lines or not parse_tap_header(lines): - return TestResult(TestStatus.FAILURE_TO_PARSE_TESTS, [], lines) - expected_test_suite_num = parse_test_plan(lines) - if expected_test_suite_num == 0: - return TestResult(TestStatus.NO_TESTS, [], lines) - elif expected_test_suite_num is None: - return TestResult(TestStatus.FAILURE_TO_PARSE_TESTS, [], lines) - test_suites = [] - for i in range(1, expected_test_suite_num + 1): - test_suite = parse_test_suite(lines, i) - if test_suite: - test_suites.append(test_suite) - else: - print_with_timestamp( - red('[ERROR] ') + ' expected ' + - str(expected_test_suite_num) + - ' test suites, but got ' + str(i - 2)) - break - test_suite = parse_test_suite(lines, -1) - if test_suite: - print_with_timestamp(red('[ERROR] ') + - 'got unexpected test suite: ' + test_suite.name) - if test_suites: - return TestResult(bubble_up_suite_errors(test_suites), test_suites, lines) - else: - return TestResult(TestStatus.NO_TESTS, [], lines) +def green(text: str) -> str: + """Returns inputted string with green color code.""" + return '\033[1;32m' + text + RESET -class TestCounts: - passed: int - failed: int - crashed: int - skipped: int +ANSI_LEN = len(red('')) - def __init__(self): - self.passed = 0 - self.failed = 0 - self.crashed = 0 - self.skipped = 0 +def print_with_timestamp(message: str) -> None: + """Prints message with timestamp at beginning.""" + print('[%s] %s' % (datetime.now().strftime('%H:%M:%S'), message)) - def total(self) -> int: - return self.passed + self.failed + self.crashed + self.skipped - -def print_and_count_results(test_result: TestResult) -> TestCounts: - counts = TestCounts() - for test_suite in test_result.suites: - if test_suite.status == TestStatus.SUCCESS: - print_suite_divider(green('[PASSED] ') + test_suite.name) - elif test_suite.status == TestStatus.SKIPPED: - print_suite_divider(yellow('[SKIPPED] ') + test_suite.name) - elif test_suite.status == TestStatus.TEST_CRASHED: - print_suite_divider(red('[CRASHED] ' + test_suite.name)) +def format_test_divider(message: str, len_message: int) -> str: + """ + Returns string with message centered in fixed width divider. + + Example: + '===================== message example =====================' + + Parameters: + message - message to be centered in divider line + len_message - length of the message to be printed such that + any characters of the color codes are not counted + + Return: + String containing message centered in fixed width divider + """ + default_count = 3 # default number of dashes + len_1 = default_count + len_2 = default_count + difference = len(DIVIDER) - len_message - 2 # 2 spaces added + if difference > 0: + # calculate number of dashes for each side of the divider + len_1 = int(difference / 2) + len_2 = difference - len_1 + return ('=' * len_1) + ' ' + message + ' ' + ('=' * len_2) + +def print_test_header(test: Test) -> None: + """ + Prints test header with test name and optionally the expected number + of subtests. + + Example: + '=================== example (2 subtests) ===================' + + Parameters: + test - Test object representing current test being printed + """ + message = test.name + if test.expected_count: + message += ' (' + str(test.expected_count) + ' subtests)' + print_with_timestamp(format_test_divider(message, len(message))) + +def print_log(log: Iterable[str]) -> None: + """ + Prints all strings in saved log for test in yellow. + + Parameters: + log - Iterable object with all strings saved in log for test + """ + for m in log: + print_with_timestamp(yellow(m)) + print_with_timestamp('') + +def format_test_result(test: Test) -> str: + """ + Returns string with formatted test result with colored status and test + name. + + Example: + '[PASSED] example' + + Parameters: + test - Test object representing current test being printed + + Return: + String containing formatted test result + """ + if test.status == TestStatus.SUCCESS: + return (green('[PASSED] ') + test.name) + elif test.status == TestStatus.SKIPPED: + return (yellow('[SKIPPED] ') + test.name) + elif test.status == TestStatus.TEST_CRASHED: + print_log(test.log) + return (red('[CRASHED] ') + test.name) + else: + print_log(test.log) + return (red('[FAILED] ') + test.name) + +def print_test_result(test: Test) -> None: + """ + Prints result line with status of test. + + Example: + '[PASSED] example' + + Parameters: + test - Test object representing current test being printed + """ + print_with_timestamp(format_test_result(test)) + +def print_test_footer(test: Test) -> None: + """ + Prints test footer with status of test. + + Example: + '===================== [PASSED] example =====================' + + Parameters: + test - Test object representing current test being printed + """ + message = format_test_result(test) + print_with_timestamp(format_test_divider(message, + len(message) - ANSI_LEN)) + +def print_summary_line(test: Test) -> None: + """ + Prints summary line of test object. Color of line is dependent on + status of test. Color is green if test passes, yellow if test is + skipped, and red if the test fails or crashes. Summary line contains + counts of the statuses of the tests subtests or the test itself if it + has no subtests. + + Example: + "Testing complete. Passed: 2, Failed: 0, Crashed: 0, Skipped: 0, + Errors: 0" + + test - Test object representing current test being printed + """ + if test.status == TestStatus.SUCCESS or \ + test.status == TestStatus.NO_TESTS: + color = green + elif test.status == TestStatus.SKIPPED: + color = yellow + else: + color = red + counts = test.counts + print_with_timestamp(color('Testing complete. ' + str(counts))) + +def print_error(error_message: str) -> None: + """ + Prints error message with error format. + + Example: + "[ERROR] Test example: missing test plan!" + + Parameters: + error_message - message describing error + """ + print_with_timestamp(red('[ERROR] ') + error_message) + +# Other methods: + +def bubble_up_test_results(test: Test) -> None: + """ + If the test has subtests, add the test counts of the subtests to the + test and check if any of the tests crashed and if so set the test + status to crashed. Otherwise if the test has no subtests add the + status of the test to the test counts. + + Parameters: + test - Test object for current test being parsed + """ + parse_crash_in_log(test) + subtests = test.subtests + counts = test.counts + status = test.status + for t in subtests: + counts.add_subtest_counts(t.counts) + if counts.total() == 0: + counts.add_status(status) + elif test.counts.get_status() == TestStatus.TEST_CRASHED: + test.status = TestStatus.TEST_CRASHED + +def parse_test(lines: LineStream, expected_num: int, log: List[str]) -> Test: + """ + Finds next test to parse in LineStream, creates new Test object, + parses any subtests of the test, populates Test object with all + information (status, name) about the test and the Test objects for + any subtests, and then returns the Test object. The method accepts + three formats of tests: + + Accepted test formats: + + - Main KTAP/TAP header + + Example: + + KTAP version 1 + 1..4 + [subtests] + + - Subtest header line + + Example: + + # Subtest: name + 1..3 + [subtests] + ok 1 name + + - Test result line + + Example: + + ok 1 - test + + Parameters: + lines - LineStream of KTAP output to parse + expected_num - expected test number for test to be parsed + log - list of strings containing any preceding diagnostic lines + corresponding to the current test + + Return: + Test object populated with characteristics and any subtests + """ + test = Test() + test.log.extend(log) + parent_test = False + main = parse_ktap_header(lines, test) + if main: + # If KTAP/TAP header is found, attempt to parse + # test plan + test.name = "main" + parse_test_plan(lines, test) + else: + # If KTAP/TAP header is not found, test must be subtest + # header or test result line so parse attempt to parser + # subtest header + parent_test = parse_test_header(lines, test) + if parent_test: + # If subtest header is found, attempt to parse + # test plan and print header + parse_test_plan(lines, test) + print_test_header(test) + expected_count = test.expected_count + subtests = [] + test_num = 1 + while main or expected_count is None or test_num <= expected_count: + # Loop to parse any subtests. + # If test is main test, do not break until no lines left. + # Otherwise, break after parsing expected number of tests or + # if expected number of tests is unknown break when found + # test result line with matching name to subtest header. + if not lines: + if expected_count and test_num <= expected_count: + test.add_error('missing expected subtests!') + break + sub_log = parse_diagnostic(lines) + if not expected_count and not main and \ + peek_test_name_match(lines, test): + test.log.extend(sub_log) + break + subtests.append(parse_test(lines, test_num, sub_log)) + test_num += 1 + test.subtests = subtests + if not main: + # If not main test, look for test result line + test.log.extend(parse_diagnostic(lines)) + if (parent_test and peek_test_name_match(lines, test)) or \ + not parent_test: + parse_test_result(lines, test, expected_num) else: - print_suite_divider(red('[FAILED] ') + test_suite.name) - for test_case in test_suite.cases: - if test_case.status == TestStatus.SUCCESS: - counts.passed += 1 - print_with_timestamp(green('[PASSED] ') + test_case.name) - elif test_case.status == TestStatus.SKIPPED: - counts.skipped += 1 - print_with_timestamp(yellow('[SKIPPED] ') + test_case.name) - elif test_case.status == TestStatus.TEST_CRASHED: - counts.crashed += 1 - print_with_timestamp(red('[CRASHED] ' + test_case.name)) - print_log(map(yellow, test_case.log)) - print_with_timestamp('') - else: - counts.failed += 1 - print_with_timestamp(red('[FAILED] ') + test_case.name) - print_log(map(yellow, test_case.log)) - print_with_timestamp('') - return counts + test.add_error('missing subtest result line!') + # Add statuses to TestCounts attribute in Test object + bubble_up_test_results(test) + if parent_test: + # If test has subtests and is not the main test object, print + # footer. + print_test_footer(test) + elif not main: + print_test_result(test) + return test def parse_run_tests(kernel_output: Iterable[str]) -> TestResult: - counts = TestCounts() + """ + Using kernel output, extract KTAP lines, parse the lines for test + results and print condensed test results and summary line . + + Parameters: + kernel_output - Iterable object contains lines of kernel output + + Return: + TestResult - Tuple containg status of main test object, main test + object with all subtests, and log of all KTAP lines. + """ + print_with_timestamp(DIVIDER) lines = extract_tap_lines(kernel_output) - test_result = parse_test_result(lines) - if test_result.status == TestStatus.NO_TESTS: - print(red('[ERROR] ') + yellow('no tests run!')) - elif test_result.status == TestStatus.FAILURE_TO_PARSE_TESTS: - print(red('[ERROR] ') + yellow('could not parse test results!')) + test = Test() + if not lines: + test.add_error('invalid KTAP input!') + test.status = TestStatus.FAILURE_TO_PARSE_TESTS else: - counts = print_and_count_results(test_result) + test = parse_test(lines, 0, []) + if test.status != TestStatus.NO_TESTS: + test.status = test.counts.get_status() print_with_timestamp(DIVIDER) - if test_result.status == TestStatus.SUCCESS: - fmt = green - elif test_result.status == TestStatus.SKIPPED: - fmt = yellow - else: - fmt =red - print_with_timestamp( - fmt('Testing complete. %d tests run. %d failed. %d crashed. %d skipped.' % - (counts.total(), counts.failed, counts.crashed, counts.skipped))) - return test_result + print_summary_line(test) + return TestResult(test.status, test, lines) diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index 3251cef25b76..50c4d3e74115 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -106,10 +106,10 @@ class KUnitParserTest(unittest.TestCase): with open(log_path) as file: result = kunit_parser.extract_tap_lines(file.readlines()) self.assertContains('TAP version 14', result) - self.assertContains(' # Subtest: example', result) - self.assertContains(' 1..2', result) - self.assertContains(' ok 1 - example_simple_test', result) - self.assertContains(' ok 2 - example_mock_test', result) + self.assertContains('# Subtest: example', result) + self.assertContains('1..2', result) + self.assertContains('ok 1 - example_simple_test', result) + self.assertContains('ok 2 - example_mock_test', result) self.assertContains('ok 1 - example', result) def test_output_with_prefix_isolated_correctly(self): @@ -117,28 +117,28 @@ class KUnitParserTest(unittest.TestCase): with open(log_path) as file: result = kunit_parser.extract_tap_lines(file.readlines()) self.assertContains('TAP version 14', result) - self.assertContains(' # Subtest: kunit-resource-test', result) - self.assertContains(' 1..5', result) - self.assertContains(' ok 1 - kunit_resource_test_init_resources', result) - self.assertContains(' ok 2 - kunit_resource_test_alloc_resource', result) - self.assertContains(' ok 3 - kunit_resource_test_destroy_resource', result) - self.assertContains(' foo bar #', result) - self.assertContains(' ok 4 - kunit_resource_test_cleanup_resources', result) - self.assertContains(' ok 5 - kunit_resource_test_proper_free_ordering', result) + self.assertContains('# Subtest: kunit-resource-test', result) + self.assertContains('1..5', result) + self.assertContains('ok 1 - kunit_resource_test_init_resources', result) + self.assertContains('ok 2 - kunit_resource_test_alloc_resource', result) + self.assertContains('ok 3 - kunit_resource_test_destroy_resource', result) + self.assertContains('foo bar #', result) + self.assertContains('ok 4 - kunit_resource_test_cleanup_resources', result) + self.assertContains('ok 5 - kunit_resource_test_proper_free_ordering', result) self.assertContains('ok 1 - kunit-resource-test', result) - self.assertContains(' foo bar # non-kunit output', result) - self.assertContains(' # Subtest: kunit-try-catch-test', result) - self.assertContains(' 1..2', result) - self.assertContains(' ok 1 - kunit_test_try_catch_successful_try_no_catch', + self.assertContains('foo bar # non-kunit output', result) + self.assertContains('# Subtest: kunit-try-catch-test', result) + self.assertContains('1..2', result) + self.assertContains('ok 1 - kunit_test_try_catch_successful_try_no_catch', result) - self.assertContains(' ok 2 - kunit_test_try_catch_unsuccessful_try_does_catch', + self.assertContains('ok 2 - kunit_test_try_catch_unsuccessful_try_does_catch', result) self.assertContains('ok 2 - kunit-try-catch-test', result) - self.assertContains(' # Subtest: string-stream-test', result) - self.assertContains(' 1..3', result) - self.assertContains(' ok 1 - string_stream_test_empty_on_creation', result) - self.assertContains(' ok 2 - string_stream_test_not_empty_after_add', result) - self.assertContains(' ok 3 - string_stream_test_get_string', result) + self.assertContains('# Subtest: string-stream-test', result) + self.assertContains('1..3', result) + self.assertContains('ok 1 - string_stream_test_empty_on_creation', result) + self.assertContains('ok 2 - string_stream_test_not_empty_after_add', result) + self.assertContains('ok 3 - string_stream_test_get_string', result) self.assertContains('ok 3 - string-stream-test', result) def test_parse_successful_test_log(self): @@ -148,6 +148,13 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual( kunit_parser.TestStatus.SUCCESS, result.status) + def test_parse_successful_nested_tests_log(self): + all_passed_log = test_data_path('test_is_test_passed-all_passed_nested.log') + with open(all_passed_log) as file: + result = kunit_parser.parse_run_tests(file.readlines()) + self.assertEqual( + kunit_parser.TestStatus.SUCCESS, + result.status) def test_parse_failed_test_log(self): failed_log = test_data_path('test_is_test_passed-failure.log') @@ -162,17 +169,31 @@ class KUnitParserTest(unittest.TestCase): with open(empty_log) as file: result = kunit_parser.parse_run_tests( kunit_parser.extract_tap_lines(file.readlines())) - self.assertEqual(0, len(result.suites)) + self.assertEqual(0, len(result.test.subtests)) self.assertEqual( kunit_parser.TestStatus.FAILURE_TO_PARSE_TESTS, result.status) + def test_missing_test_plan(self): + missing_plan_log = test_data_path('test_is_test_passed-' + 'missing_plan.log') + with open(missing_plan_log) as file: + result = kunit_parser.parse_run_tests( + kunit_parser.extract_tap_lines( + file.readlines())) + self.assertEqual(2, result.test.counts.errors) + self.assertEqual( + kunit_parser.TestStatus.SUCCESS, + result.status) + def test_no_tests(self): - empty_log = test_data_path('test_is_test_passed-no_tests_run_with_header.log') - with open(empty_log) as file: + header_log = test_data_path('test_is_test_passed-' + 'no_tests_run_with_header.log') + with open(header_log) as file: result = kunit_parser.parse_run_tests( - kunit_parser.extract_tap_lines(file.readlines())) - self.assertEqual(0, len(result.suites)) + kunit_parser.extract_tap_lines( + file.readlines())) + self.assertEqual(0, len(result.test.subtests)) self.assertEqual( kunit_parser.TestStatus.NO_TESTS, result.status) @@ -182,15 +203,17 @@ class KUnitParserTest(unittest.TestCase): print_mock = mock.patch('builtins.print').start() with open(crash_log) as file: result = kunit_parser.parse_run_tests( - kunit_parser.extract_tap_lines(file.readlines())) - print_mock.assert_any_call(StrContains('could not parse test results!')) + kunit_parser.extract_tap_lines( + file.readlines())) + print_mock.assert_any_call(StrContains('invalid KTAP input!')) print_mock.stop() file.close() def test_crashed_test(self): crashed_log = test_data_path('test_is_test_passed-crash.log') with open(crashed_log) as file: - result = kunit_parser.parse_run_tests(file.readlines()) + result = kunit_parser.parse_run_tests( + file.readlines()) self.assertEqual( kunit_parser.TestStatus.TEST_CRASHED, result.status) @@ -216,6 +239,23 @@ class KUnitParserTest(unittest.TestCase): result.status) file.close() + def test_ignores_hyphen(self): + hyphen_log = test_data_path('test_strip_hyphen.log') + file = open(hyphen_log) + result = kunit_parser.parse_run_tests(file.readlines()) + + # A skipped test does not fail the whole suite. + self.assertEqual( + kunit_parser.TestStatus.SUCCESS, + result.status) + self.assertEqual( + "sysctl_test", + result.test.subtests[0].name) + self.assertEqual( + "example", + result.test.subtests[1].name) + file.close() + def test_ignores_prefix_printk_time(self): prefix_log = test_data_path('test_config_printk_time.log') @@ -224,7 +264,7 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual( kunit_parser.TestStatus.SUCCESS, result.status) - self.assertEqual('kunit-resource-test', result.suites[0].name) + self.assertEqual('kunit-resource-test', result.test.subtests[0].name) def test_ignores_multiple_prefixes(self): prefix_log = test_data_path('test_multiple_prefixes.log') @@ -233,7 +273,7 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual( kunit_parser.TestStatus.SUCCESS, result.status) - self.assertEqual('kunit-resource-test', result.suites[0].name) + self.assertEqual('kunit-resource-test', result.test.subtests[0].name) def test_prefix_mixed_kernel_output(self): mixed_prefix_log = test_data_path('test_interrupted_tap_output.log') @@ -242,7 +282,7 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual( kunit_parser.TestStatus.SUCCESS, result.status) - self.assertEqual('kunit-resource-test', result.suites[0].name) + self.assertEqual('kunit-resource-test', result.test.subtests[0].name) def test_prefix_poundsign(self): pound_log = test_data_path('test_pound_sign.log') @@ -251,16 +291,16 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual( kunit_parser.TestStatus.SUCCESS, result.status) - self.assertEqual('kunit-resource-test', result.suites[0].name) + self.assertEqual('kunit-resource-test', result.test.subtests[0].name) def test_kernel_panic_end(self): panic_log = test_data_path('test_kernel_panic_interrupt.log') with open(panic_log) as file: result = kunit_parser.parse_run_tests(file.readlines()) self.assertEqual( - kunit_parser.TestStatus.TEST_CRASHED, + kunit_parser.TestStatus.SUCCESS, result.status) - self.assertEqual('kunit-resource-test', result.suites[0].name) + self.assertEqual('kunit-resource-test', result.test.subtests[0].name) def test_pound_no_prefix(self): pound_log = test_data_path('test_pound_no_prefix.log') @@ -269,7 +309,7 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual( kunit_parser.TestStatus.SUCCESS, result.status) - self.assertEqual('kunit-resource-test', result.suites[0].name) + self.assertEqual('kunit-resource-test', result.test.subtests[0].name) class LinuxSourceTreeTest(unittest.TestCase): @@ -291,6 +331,14 @@ class LinuxSourceTreeTest(unittest.TestCase): pass tree = kunit_kernel.LinuxSourceTree('', kunitconfig_path=dir) + def test_kselftest_nested(self): + kselftest_log = test_data_path('test_is_test_passed-kselftest.log') + with open(kselftest_log) as file: + result = kunit_parser.parse_run_tests(file.readlines()) + self.assertEqual( + kunit_parser.TestStatus.SUCCESS, + result.status) + # TODO: add more test cases. @@ -322,6 +370,12 @@ class KUnitJsonTest(unittest.TestCase): result = self._json_for('test_is_test_passed-no_tests_run_with_header.log') self.assertEqual(0, len(result['sub_groups'])) + def test_nested_json(self): + result = self._json_for('test_is_test_passed-all_passed_nested.log') + self.assertEqual( + {'name': 'example_simple_test', 'status': 'PASS'}, + result["sub_groups"][0]["sub_groups"][0]["test_cases"][0]) + class StrContains(str): def __eq__(self, other): return self in other @@ -380,7 +434,7 @@ class KUnitMainTest(unittest.TestCase): self.assertEqual(e.exception.code, 1) self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 1) self.assertEqual(self.linux_source_mock.run_kernel.call_count, 1) - self.print_mock.assert_any_call(StrContains(' 0 tests run')) + self.print_mock.assert_any_call(StrContains('invalid KTAP input!')) def test_exec_raw_output(self): self.linux_source_mock.run_kernel = mock.Mock(return_value=[]) @@ -388,7 +442,7 @@ class KUnitMainTest(unittest.TestCase): self.assertEqual(self.linux_source_mock.run_kernel.call_count, 1) for call in self.print_mock.call_args_list: self.assertNotEqual(call, mock.call(StrContains('Testing complete.'))) - self.assertNotEqual(call, mock.call(StrContains(' 0 tests run'))) + self.assertNotEqual(call, mock.call(StrContains(' 0 tests run!'))) def test_run_raw_output(self): self.linux_source_mock.run_kernel = mock.Mock(return_value=[]) @@ -397,7 +451,7 @@ class KUnitMainTest(unittest.TestCase): self.assertEqual(self.linux_source_mock.run_kernel.call_count, 1) for call in self.print_mock.call_args_list: self.assertNotEqual(call, mock.call(StrContains('Testing complete.'))) - self.assertNotEqual(call, mock.call(StrContains(' 0 tests run'))) + self.assertNotEqual(call, mock.call(StrContains(' 0 tests run!'))) def test_run_raw_output_kunit(self): self.linux_source_mock.run_kernel = mock.Mock(return_value=[]) diff --git a/tools/testing/kunit/test_data/test_is_test_passed-all_passed_nested.log b/tools/testing/kunit/test_data/test_is_test_passed-all_passed_nested.log new file mode 100644 index 000000000000..9d5b04fe43a6 --- /dev/null +++ b/tools/testing/kunit/test_data/test_is_test_passed-all_passed_nested.log @@ -0,0 +1,34 @@ +TAP version 14 +1..2 + # Subtest: sysctl_test + 1..4 + # sysctl_test_dointvec_null_tbl_data: sysctl_test_dointvec_null_tbl_data passed + ok 1 - sysctl_test_dointvec_null_tbl_data + # Subtest: example + 1..2 + init_suite + # example_simple_test: initializing + # example_simple_test: example_simple_test passed + ok 1 - example_simple_test + # example_mock_test: initializing + # example_mock_test: example_mock_test passed + ok 2 - example_mock_test + kunit example: all tests passed + ok 2 - example + # sysctl_test_dointvec_table_len_is_zero: sysctl_test_dointvec_table_len_is_zero passed + ok 3 - sysctl_test_dointvec_table_len_is_zero + # sysctl_test_dointvec_table_read_but_position_set: sysctl_test_dointvec_table_read_but_position_set passed + ok 4 - sysctl_test_dointvec_table_read_but_position_set +kunit sysctl_test: all tests passed +ok 1 - sysctl_test + # Subtest: example + 1..2 +init_suite + # example_simple_test: initializing + # example_simple_test: example_simple_test passed + ok 1 - example_simple_test + # example_mock_test: initializing + # example_mock_test: example_mock_test passed + ok 2 - example_mock_test +kunit example: all tests passed +ok 2 - example diff --git a/tools/testing/kunit/test_data/test_is_test_passed-kselftest.log b/tools/testing/kunit/test_data/test_is_test_passed-kselftest.log new file mode 100644 index 000000000000..65d3f27feaf2 --- /dev/null +++ b/tools/testing/kunit/test_data/test_is_test_passed-kselftest.log @@ -0,0 +1,14 @@ +TAP version 13 +1..2 +# selftests: membarrier: membarrier_test_single_thread +# TAP version 13 +# 1..2 +# ok 1 sys_membarrier available +# ok 2 sys membarrier invalid command test: command = -1, flags = 0, errno = 22. Failed as expected +ok 1 selftests: membarrier: membarrier_test_single_thread +# selftests: membarrier: membarrier_test_multi_thread +# TAP version 13 +# 1..2 +# ok 1 sys_membarrier available +# ok 2 sys membarrier invalid command test: command = -1, flags = 0, errno = 22. Failed as expected +ok 2 selftests: membarrier: membarrier_test_multi_thread diff --git a/tools/testing/kunit/test_data/test_is_test_passed-missing_plan.log b/tools/testing/kunit/test_data/test_is_test_passed-missing_plan.log new file mode 100644 index 000000000000..5cd17b7f818a --- /dev/null +++ b/tools/testing/kunit/test_data/test_is_test_passed-missing_plan.log @@ -0,0 +1,31 @@ +KTAP version 1 + # Subtest: sysctl_test + # sysctl_test_dointvec_null_tbl_data: sysctl_test_dointvec_null_tbl_data passed + ok 1 - sysctl_test_dointvec_null_tbl_data + # sysctl_test_dointvec_table_maxlen_unset: sysctl_test_dointvec_table_maxlen_unset passed + ok 2 - sysctl_test_dointvec_table_maxlen_unset + # sysctl_test_dointvec_table_len_is_zero: sysctl_test_dointvec_table_len_is_zero passed + ok 3 - sysctl_test_dointvec_table_len_is_zero + # sysctl_test_dointvec_table_read_but_position_set: sysctl_test_dointvec_table_read_but_position_set passed + ok 4 - sysctl_test_dointvec_table_read_but_position_set + # sysctl_test_dointvec_happy_single_positive: sysctl_test_dointvec_happy_single_positive passed + ok 5 - sysctl_test_dointvec_happy_single_positive + # sysctl_test_dointvec_happy_single_negative: sysctl_test_dointvec_happy_single_negative passed + ok 6 - sysctl_test_dointvec_happy_single_negative + # sysctl_test_dointvec_single_less_int_min: sysctl_test_dointvec_single_less_int_min passed + ok 7 - sysctl_test_dointvec_single_less_int_min + # sysctl_test_dointvec_single_greater_int_max: sysctl_test_dointvec_single_greater_int_max passed + ok 8 - sysctl_test_dointvec_single_greater_int_max +kunit sysctl_test: all tests passed +ok 1 - sysctl_test + # Subtest: example + 1..2 +init_suite + # example_simple_test: initializing + # example_simple_test: example_simple_test passed + ok 1 - example_simple_test + # example_mock_test: initializing + # example_mock_test: example_mock_test passed + ok 2 - example_mock_test +kunit example: all tests passed +ok 2 - example diff --git a/tools/testing/kunit/test_data/test_strip_hyphen.log b/tools/testing/kunit/test_data/test_strip_hyphen.log new file mode 100644 index 000000000000..92ac7c24b374 --- /dev/null +++ b/tools/testing/kunit/test_data/test_strip_hyphen.log @@ -0,0 +1,16 @@ +KTAP version 1 +1..2 + # Subtest: sysctl_test + 1..1 + # sysctl_test_dointvec_null_tbl_data: sysctl_test_dointvec_null_tbl_data passed + ok 1 - sysctl_test_dointvec_null_tbl_data +kunit sysctl_test: all tests passed +ok 1 - sysctl_test + # Subtest: example + 1..1 +init_suite + # example_simple_test: initializing + # example_simple_test: example_simple_test passed + ok 1 example_simple_test +kunit example: all tests passed +ok 2 example base-commit: 28a8b037484044d79f3b8d409a076a925ac75f8d -- 2.33.0.800.g4c38ced690-goog

3 years, 7 months

1
1
0 0

[PATCH RESEND 3 00/16] Bitmap patches for 5.15

by Yury Norov

Hi Stephen, Please pull this bitmap series. The git tree is here: https://github.com/norov/linux/tree/bitmap-master-5.15 Thanks, Yury Yury Norov (16): bitops: protect find_first_{,zero}_bit properly bitops: move find_bit_*_le functions from le.h to find.h include: move find.h from asm_generic to linux arch: remove GENERIC_FIND_FIRST_BIT entirely lib: add find_first_and_bit() cpumask: use find_first_and_bit() all: replace find_next{,_zero}_bit with find_first{,_zero}_bit where appropriate tools: sync tools/bitmap with mother linux cpumask: replace cpumask_next_* with cpumask_first_* where appropriate include/linux: move for_each_bit() macros from bitops.h to find.h find: micro-optimize for_each_{set,clear}_bit() Replace for_each_*_bit_from() with for_each_*_bit() where appropriate mm/percpu: micro-optimize pcpu_is_populated() bitmap: unify find_bit operations lib: bitmap: add performance test for bitmap_print_to_pagebuf vsprintf: rework bitmap_list_string MAINTAINERS | 4 +- arch/alpha/include/asm/bitops.h | 2 - arch/arc/Kconfig | 1 - arch/arc/include/asm/bitops.h | 1 - arch/arm/include/asm/bitops.h | 1 - arch/arm64/Kconfig | 1 - arch/arm64/include/asm/bitops.h | 1 - arch/csky/include/asm/bitops.h | 1 - arch/h8300/include/asm/bitops.h | 1 - arch/hexagon/include/asm/bitops.h | 1 - arch/ia64/include/asm/bitops.h | 2 - arch/m68k/include/asm/bitops.h | 2 - arch/mips/Kconfig | 1 - arch/mips/include/asm/bitops.h | 1 - arch/openrisc/include/asm/bitops.h | 1 - arch/parisc/include/asm/bitops.h | 2 - arch/powerpc/include/asm/bitops.h | 2 - arch/powerpc/include/asm/cputhreads.h | 2 +- arch/powerpc/platforms/pasemi/dma_lib.c | 4 +- arch/riscv/include/asm/bitops.h | 1 - arch/s390/Kconfig | 1 - arch/s390/include/asm/bitops.h | 1 - arch/s390/kvm/kvm-s390.c | 2 +- arch/sh/include/asm/bitops.h | 1 - arch/sparc/include/asm/bitops_32.h | 1 - arch/sparc/include/asm/bitops_64.h | 2 - arch/x86/Kconfig | 1 - arch/x86/include/asm/bitops.h | 2 - arch/x86/kernel/apic/vector.c | 4 +- arch/x86/um/Kconfig | 1 - arch/xtensa/include/asm/bitops.h | 1 - block/blk-mq.c | 2 +- drivers/block/rnbd/rnbd-clt.c | 2 +- drivers/dma/ti/edma.c | 2 +- drivers/gpu/drm/etnaviv/etnaviv_gpu.c | 4 +- drivers/hwmon/ltc2992.c | 3 +- drivers/iio/adc/ad7124.c | 2 +- drivers/infiniband/hw/irdma/hw.c | 16 +- drivers/media/cec/core/cec-core.c | 2 +- drivers/media/mc/mc-devnode.c | 2 +- drivers/mmc/host/renesas_sdhi_core.c | 2 +- drivers/net/virtio_net.c | 2 +- drivers/pci/controller/dwc/pci-dra7xx.c | 2 +- drivers/scsi/lpfc/lpfc_sli.c | 10 +- drivers/soc/fsl/qbman/bman_portal.c | 2 +- drivers/soc/fsl/qbman/qman_portal.c | 2 +- drivers/soc/ti/k3-ringacc.c | 4 +- drivers/tty/n_tty.c | 2 +- drivers/virt/acrn/ioreq.c | 3 +- fs/f2fs/segment.c | 8 +- fs/ocfs2/cluster/heartbeat.c | 2 +- fs/ocfs2/dlm/dlmdomain.c | 4 +- fs/ocfs2/dlm/dlmmaster.c | 18 +- fs/ocfs2/dlm/dlmrecovery.c | 2 +- fs/ocfs2/dlm/dlmthread.c | 2 +- include/asm-generic/bitops.h | 1 - include/asm-generic/bitops/le.h | 64 --- include/linux/bitmap.h | 34 +- include/linux/bitops.h | 34 -- include/linux/cpumask.h | 46 ++- include/linux/find.h | 372 ++++++++++++++++++ kernel/time/clocksource.c | 4 +- lib/Kconfig | 3 - lib/find_bit.c | 21 + lib/find_bit_benchmark.c | 21 + lib/genalloc.c | 2 +- lib/test_bitmap.c | 37 ++ lib/vsprintf.c | 24 +- mm/percpu.c | 35 +- net/ncsi/ncsi-manage.c | 4 +- tools/include/asm-generic/bitops.h | 1 - tools/include/asm-generic/bitops/find.h | 145 ------- tools/include/linux/bitmap.h | 7 +- .../bitops => tools/include/linux}/find.h | 54 ++- tools/lib/find_bit.c | 20 + 75 files changed, 637 insertions(+), 441 deletions(-) create mode 100644 include/linux/find.h delete mode 100644 tools/include/asm-generic/bitops/find.h rename {include/asm-generic/bitops => tools/include/linux}/find.h (83%) -- 2.30.2

3 years, 7 months

2
17
0 0

[PATCH] kunit: tool: yield output from run_kernel in real time

by Daniel Latypov

Currently, `run_kernel()` dumps all the kernel output to a file (.kunit/test.log) and then opens the file and yields it to callers. This made it easier to respect the requested timeout, if any. But it means that we can't yield the results in real time, either to the parser or to stdout (if --raw_output is set). This change spins up a background thread to enforce the timeout, which allows us to yield the kernel output in real time, while also copying it to the .kunit/test.log file. It's also careful to ensure that the .kunit/test.log file is complete, even in the kunit_parser throws an exception/otherwise doesn't consume every line, see the new `finally` block and unit test. For example: $ ./tools/testing/kunit/kunit.py run --arch=x86_64 --raw_output <configure + build steps> ... <can now see output from QEMU in real time> This does not currently have a visible effect when --raw_output is not passed, as kunit_parser.py currently only outputs everything at the end. But that could change, and this patch is a necessary step towards showing parsed test results in real time. Signed-off-by: Daniel Latypov <dlatypov(a)google.com> --- tools/testing/kunit/kunit_kernel.py | 73 +++++++++++++++----------- tools/testing/kunit/kunit_tool_test.py | 17 ++++++ 2 files changed, 60 insertions(+), 30 deletions(-) diff --git a/tools/testing/kunit/kunit_kernel.py b/tools/testing/kunit/kunit_kernel.py index 2c6f916ccbaf..b8cba8123aa3 100644 --- a/tools/testing/kunit/kunit_kernel.py +++ b/tools/testing/kunit/kunit_kernel.py @@ -12,7 +12,8 @@ import subprocess import os import shutil import signal -from typing import Iterator, Optional, Tuple +import threading +from typing import Iterator, List, Optional, Tuple from contextlib import ExitStack @@ -103,8 +104,8 @@ class LinuxSourceTreeOperations(object): if stderr: # likely only due to build warnings print(stderr.decode()) - def run(self, params, timeout, build_dir, outfile) -> None: - pass + def start(self, params: List[str], build_dir: str) -> subprocess.Popen: + raise RuntimeError('not implemented!') class LinuxSourceTreeOperationsQemu(LinuxSourceTreeOperations): @@ -123,7 +124,7 @@ class LinuxSourceTreeOperationsQemu(LinuxSourceTreeOperations): kconfig.parse_from_string(self._kconfig) base_kunitconfig.merge_in_entries(kconfig) - def run(self, params, timeout, build_dir, outfile): + def start(self, params: List[str], build_dir: str) -> subprocess.Popen: kernel_path = os.path.join(build_dir, self._kernel_path) qemu_command = ['qemu-system-' + self._qemu_arch, '-nodefaults', @@ -134,18 +135,10 @@ class LinuxSourceTreeOperationsQemu(LinuxSourceTreeOperations): '-nographic', '-serial stdio'] + self._extra_qemu_params print('Running tests with:\n$', ' '.join(qemu_command)) - with open(outfile, 'w') as output: - process = subprocess.Popen(' '.join(qemu_command), - stdin=subprocess.PIPE, - stdout=output, - stderr=subprocess.STDOUT, - text=True, shell=True) - try: - process.wait(timeout=timeout) - except Exception as e: - print(e) - process.terminate() - return process + return subprocess.Popen(' '.join(qemu_command), + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + text=True, shell=True) class LinuxSourceTreeOperationsUml(LinuxSourceTreeOperations): """An abstraction over command line operations performed on a source tree.""" @@ -175,17 +168,13 @@ class LinuxSourceTreeOperationsUml(LinuxSourceTreeOperations): kunit_parser.print_with_timestamp( 'Starting Kernel with all configs takes a few minutes...') - def run(self, params, timeout, build_dir, outfile): + def start(self, params: List[str], build_dir: str) -> subprocess.Popen: """Runs the Linux UML binary. Must be named 'linux'.""" linux_bin = get_file_path(build_dir, 'linux') - outfile = get_outfile_path(build_dir) - with open(outfile, 'w') as output: - process = subprocess.Popen([linux_bin] + params, - stdin=subprocess.PIPE, - stdout=output, - stderr=subprocess.STDOUT, - text=True) - process.wait(timeout) + return subprocess.Popen([linux_bin] + params, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + text=True) def get_kconfig_path(build_dir) -> str: return get_file_path(build_dir, KCONFIG_PATH) @@ -330,12 +319,36 @@ class LinuxSourceTree(object): args.extend(['mem=1G', 'console=tty', 'kunit_shutdown=halt']) if filter_glob: args.append('kunit.filter_glob='+filter_glob) - outfile = get_outfile_path(build_dir) - self._ops.run(args, timeout, build_dir, outfile) - subprocess.call(['stty', 'sane']) - with open(outfile, 'r') as file: - for line in file: + + process = self._ops.start(args, build_dir) + assert process.stdout is not None # tell mypy it's set + + # Enforce the timeout in a background thread. + def _wait_proc(): + try: + process.wait(timeout=timeout) + except Exception as e: + print(e) + process.terminate() + process.wait() + waiter = threading.Thread(target=_wait_proc) + waiter.start() + + output = open(get_outfile_path(build_dir), 'w') + try: + # Tee the output to the file and to our caller in real time. + for line in process.stdout: + output.write(line) yield line + # This runs even if our caller doesn't consume every line. + finally: + # Flush any leftover output to the file + output.write(process.stdout.read()) + output.close() + process.stdout.close() + + waiter.join() + subprocess.call(['stty', 'sane']) def signal_handler(self, sig, frame) -> None: logging.error('Build interruption occurred. Cleaning console.') diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index 619c4554cbff..f9a7398a9584 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -14,6 +14,7 @@ import tempfile, shutil # Handling test_tmpdir import itertools import json import signal +import subprocess import os import kunit_config @@ -291,6 +292,22 @@ class LinuxSourceTreeTest(unittest.TestCase): pass tree = kunit_kernel.LinuxSourceTree('', kunitconfig_path=dir) + def test_run_kernel_hits_exception(self): + def fake_start(unused_args, unused_build_dir): + return subprocess.Popen(['echo "hi\nbye"'], shell=True, text=True, stdout=subprocess.PIPE) + + with tempfile.TemporaryDirectory('') as build_dir: + tree = kunit_kernel.LinuxSourceTree(build_dir, load_config=False) + mock.patch.object(tree._ops, 'start', side_effect=fake_start).start() + + with self.assertRaises(ValueError): + for line in tree.run_kernel(build_dir=build_dir): + self.assertEqual(line, 'hi\n') + raise ValueError('uh oh, did not read all output') + + with open(kunit_kernel.get_outfile_path(build_dir), 'rt') as outfile: + self.assertEqual(outfile.read(), 'hi\nbye\n', msg='Missing some output') + # TODO: add more test cases. base-commit: 3b29021ddd10cfb6b2565c623595bd3b02036f33 -- 2.33.0.800.g4c38ced690-goog

3 years, 7 months

2
6
0 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

Linux-kselftest-mirror