On Fri, Aug 6, 2021 at 2:15 AM 'Brendan Higgins' via KUnit Development kunit-dev@googlegroups.com wrote:
On Tue, Aug 3, 2021 at 10:08 PM David Gow davidgow@google.com wrote:
When a number of tests fail, it can be useful to get higher-level statistics of how many tests are failing (or how many parameters are failing in parameterised tests), and in what cases or suites. This is already done by some non-KUnit tests, so add support for automatically generating these for KUnit tests.
This change adds a 'kunit.stats_enabled' switch which has three values:
- 0: No stats are printed (current behaviour)
- 1: Stats are printed only for tests/suites with more than one subtest (new default)
- 2: Always print test statistics
For parameterised tests, the summary line looks as follows: " # inode_test_xtimestamp_decoding: pass:16 fail:0 skip:0 total:16" For test suites, there are two lines looking like this: "# ext4_inode_test: pass:1 fail:0 skip:0 total:1" "# Totals: pass:16 fail:0 skip:0 total:16"
The first line gives the number of direct subtests, the second "Totals" line is the accumulated sum of all tests and test parameters.
This format is based on the one used by kselftest[1].
Signed-off-by: David Gow davidgow@google.com
This looks great, David!
My only suggestion, can you maybe provide a sample of the TAP output with your change running on the thread? I think it looks great, but I imagine that it has the potential of being more interesting to people other than you, Daniel, and myself rather than the actual code change. (To be clear, I think the summary and the code both look good, IMO.)
Sure. The lines in the commit description I think cover the actual change pretty well:
" # inode_test_xtimestamp_decoding: pass:16 fail:0 skip:0 total:16"
"# ext4_inode_test: pass:1 fail:0 skip:0 total:1" "# Totals: pass:16 fail:0 skip:0 total:16"
Nevertheless, here's the complete output of the 'example' and 'rational' suites, which should give a more complete overview. (This is with the default configuration of only printing out lines where the number of subtests is >1)
TAP version 14 1..2 # Subtest: example 1..3 # example_simple_test: initializing ok 1 - example_simple_test # example_skip_test: initializing # example_skip_test: You should not see a line below. ok 2 - example_skip_test # SKIP this test should be skipped # example_mark_skipped_test: initializing # example_mark_skipped_test: You should see a line below. # example_mark_skipped_test: You should see this line. ok 3 - example_mark_skipped_test # SKIP this test should be skipped # example: pass:1 fail:0 skip:2 total:3 # Totals: pass:1 fail:0 skip:2 total:3 ok 1 - example # Subtest: rational 1..1 # rational_test: ok 1 - Exceeds bounds, semi-convergent term > 1/2 last term # rational_test: ok 2 - Exceeds bounds, semi-convergent term < 1/2 last term # rational_test: ok 3 - Closest to zero # rational_test: ok 4 - Closest to smallest non-zero # rational_test: ok 5 - Use convergent # rational_test: ok 6 - Exact answer # rational_test: ok 7 - Semiconvergent, numerator limit # rational_test: ok 8 - Semiconvergent, denominator limit # rational_test: pass:8 fail:0 skip:0 total:8 ok 1 - rational_test # Totals: pass:8 fail:0 skip:0 total:8 ok 2 - rational