On Mon, 19 Sep 2022 17:25:38 -0700 Daniel Latypov dlatypov@google.com wrote:
On Mon, Sep 19, 2022 at 1:43 PM Isabella Basso isabbasso@riseup.net wrote:
* "(K)TAP version XX" should be the first line on all (sub)tests as per* https://www.kernel.org/doc/html/latest/dev-tools/ktap.html#version-lines* but actually isn't, as it currently depends on whoever writes the* test to print this infoFYI: we're really trying to fix cases of "missing version lines", largely by making the kunit_test_suites() macro work in more circumstances.
So while it may be worth still handling the case where this is missing, I don't think there are any tests in the latest versions of the kernel which should have this missing.
I’m not sure if I totally get how these work. Every time I run a KUnit test I get something like this: https://pastebin.com/7Ff31PMC
As you can see it has been loaded as a module, just like we intend to do it from IGT, and I see no version lines whatsoever. Am I doing something wrong?
You're doing everything right.
The problem is we only print the version line for the *built-in* tests. It never gets printed for tests in loadable modules.
IGT tests require loadable modules, as some tests check if bind/unbind and load/unload works. Also, some tests require modprobing drivers with different parameters.
Unfortunately, kunit.py is currently broken on such cases: its parser simply ignores everything if the subtest doesn't have a TAP version just before it. See:
$ ./tools/testing/kunit/kunit.py parse /var/log/dmesg [12:37:05] ============================================================ [12:37:05] [ERROR] Test: main: 0 tests run! [12:37:05] ============================================================ [12:37:05] Testing complete. Ran 0 tests: errors: 1
Here, I loaded the test module twice. I can force it to parse it by doing
$ (dmesg|grep "TAP version"; dmesg|grep -A9999 intel_i915_mock) >logs
But then, it gets confused with two subtests with the same name:
$ ./tools/testing/kunit/kunit.py parse logs [12:37:34] ============================================================ [12:37:34] ============== intel_i915_mock (18 subtests) =============== [12:37:34] [PASSED] sanitycheck [12:37:34] [PASSED] shmem [12:37:34] [PASSED] fence [12:37:34] [PASSED] scatterlist [12:37:34] [PASSED] syncmap [12:37:34] [PASSED] uncore [12:37:34] [PASSED] ring [12:37:34] [PASSED] engine [12:37:34] [PASSED] timelines [12:37:34] [PASSED] requests [12:37:34] [PASSED] objects [12:37:34] [PASSED] phys [12:37:34] [PASSED] dmabuf [12:37:34] [PASSED] vma [12:37:34] [PASSED] evict [12:37:34] [PASSED] gtt [12:37:34] [PASSED] hugepages [12:37:34] [PASSED] memory_region [12:37:34] ================= [PASSED] intel_i915_mock ================= [12:37:34] ============== intel_i915_mock (18 subtests) =============== [12:37:34] [PASSED] sanitycheck [12:37:34] [PASSED] shmem [12:37:34] [PASSED] fence [12:37:34] [PASSED] scatterlist [12:37:34] [PASSED] syncmap [12:37:34] [PASSED] uncore [12:37:34] [PASSED] ring [12:37:34] [PASSED] engine [12:37:34] [PASSED] timelines [12:37:34] [PASSED] requests [12:37:34] [PASSED] objects [12:37:34] [PASSED] phys [12:37:34] [PASSED] dmabuf [12:37:34] [PASSED] vma [12:37:34] [PASSED] evict [12:37:34] [PASSED] gtt [12:37:34] [PASSED] hugepages [12:37:34] [PASSED] memory_region [12:37:34] [ERROR] Test: intel_i915_mock: Expected test number 2 but found 1 [12:37:34] ================= [PASSED] intel_i915_mock ================= [12:37:34] ============================================================ [12:37:34] Testing complete. Ran 36 tests: passed: 36, errors: 1
Some more details below, if interested. Specifically, it happens in https://elixir.bootlin.com/linux/latest/C/ident/kunit_print_tap_header
What David is referring to is we had tests that weren't in modules, but didn't use the normal built-in code path. Those were also missing TAP version lines. But KUnit needs to do better here in the case of modules.
The goal was that you'd be able to parse the result of module based tests by something like $ cat /sys/kernel/debug/kunit/*/results | ./tools/testing/kunit/kunit.py parse but this doesn't work because of the lack of a version line.
At least here, debug/kunit doesn't exist:
sudo ls -lctra /sys/kernel/debug/|grep kunit
If we add it to each module's test results, then we'll need to update the parse to allow multiple verison lines, I think.
Yeah, the parser is currently broken when used with modules.
Regards, Mauro