It's possible for a test to have a subtest header, but zero valid subtests. We used to error on this if the test plan had no subtests listed, but it's possible to have subtests without a test plan (indeed, this is how parameterised tests work).

Tests with 0 subtests now have the result NO_TESTS, and will report an error (which does not halt test execution, but is printed in a scary red colour and is noted in the results summary).

Signed-off-by: David Gow davidgow@google.com Reviewed-by: Daniel Latypov dlatypov@google.com ---

Changes since v3: https://lore.kernel.org/linux-kselftest/20211028064154.2301049-2-davidgow@go... - Split up and expanded the tests, as suggested by Daniel.

Changes since v2: https://lore.kernel.org/linux-kselftest/20211027013702.2039566-2-davidgow@go... - Report NO_TESTS as '[NO TESTS RUN]' in yellow, instead of '[FAILED]' in red, particularly since it doesn't get counted as a failure.

tools/testing/kunit/kunit_parser.py | 16 +++++++++++----- tools/testing/kunit/kunit_tool_test.py | 12 ++++++++++++ .../test_is_test_passed-no_tests_no_plan.log | 7 +++++++ 3 files changed, 30 insertions(+), 5 deletions(-) create mode 100644 tools/testing/kunit/test_data/test_is_test_passed-no_tests_no_plan.log

diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py index 50ded55c168c..68c847e8ca58 100644 --- a/tools/testing/kunit/kunit_parser.py +++ b/tools/testing/kunit/kunit_parser.py @@ -360,9 +360,6 @@ def parse_test_plan(lines: LineStream, test: Test) -> bool: test.log.append(lines.pop()) expected_count = int(match.group(1)) test.expected_count = expected_count - if expected_count == 0: - test.status = TestStatus.NO_TESTS - test.add_error('0 tests run!') return True

TEST_RESULT = re.compile(r'^(ok|not ok) ([0-9]+) (- )?([^#]*)( # .*)?$') @@ -589,6 +586,8 @@ def format_test_result(test: Test) -> str: return (green('[PASSED] ') + test.name) elif test.status == TestStatus.SKIPPED: return (yellow('[SKIPPED] ') + test.name) + elif test.status == TestStatus.NO_TESTS: + return (yellow('[NO TESTS RUN] ') + test.name) elif test.status == TestStatus.TEST_CRASHED: print_log(test.log) return (red('[CRASHED] ') + test.name) @@ -731,6 +730,7 @@ def parse_test(lines: LineStream, expected_num: int, log: List[str]) -> Test: # test plan test.name = "main" parse_test_plan(lines, test) + parent_test = True else: # If KTAP/TAP header is not found, test must be subtest # header or test result line so parse attempt to parser @@ -744,7 +744,7 @@ def parse_test(lines: LineStream, expected_num: int, log: List[str]) -> Test: expected_count = test.expected_count subtests = [] test_num = 1 - while expected_count is None or test_num <= expected_count: + while parent_test and (expected_count is None or test_num <= expected_count): # Loop to parse any subtests. # Break after parsing expected number of tests or # if expected number of tests is unknown break when test @@ -779,9 +779,15 @@ def parse_test(lines: LineStream, expected_num: int, log: List[str]) -> Test: parse_test_result(lines, test, expected_num) else: test.add_error('missing subtest result line!') + + # Check for there being no tests + if parent_test and len(subtests) == 0: + test.status = TestStatus.NO_TESTS + test.add_error('0 tests run!') + # Add statuses to TestCounts attribute in Test object bubble_up_test_results(test) - if parent_test: + if parent_test and not main: # If test has subtests and is not the main test object, print # footer. print_test_footer(test) diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index bc8793145713..9de2072089e6 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -209,6 +209,18 @@ class KUnitParserTest(unittest.TestCase): kunit_parser.TestStatus.NO_TESTS, result.status)

+ def test_no_tests_no_plan(self): + no_plan_log = test_data_path('test_is_test_passed-no_tests_no_plan.log') + with open(no_plan_log) as file: + result = kunit_parser.parse_run_tests( + kunit_parser.extract_tap_lines(file.readlines())) + self.assertEqual(0, len(result.test.subtests[0].subtests[0].subtests)) + self.assertEqual( + kunit_parser.TestStatus.NO_TESTS, + result.test.subtests[0].subtests[0].status) + self.assertEqual(1, result.test.counts.errors) + + def test_no_kunit_output(self): crash_log = test_data_path('test_insufficient_memory.log') print_mock = mock.patch('builtins.print').start() diff --git a/tools/testing/kunit/test_data/test_is_test_passed-no_tests_no_plan.log b/tools/testing/kunit/test_data/test_is_test_passed-no_tests_no_plan.log new file mode 100644 index 000000000000..dd873c981108 --- /dev/null +++ b/tools/testing/kunit/test_data/test_is_test_passed-no_tests_no_plan.log @@ -0,0 +1,7 @@ +TAP version 14 +1..1 + # Subtest: suite + 1..1 + # Subtest: case + ok 1 - case # SKIP +ok 1 - suite

It's possible that a parameterised test could end up with zero parameters. At the moment, the test function will nevertheless be called with NULL as the parameter. Instead, don't try to run the test code, and just mark the test as SKIPped.

Reported-by: Daniel Latypov dlatypov@google.com Signed-off-by: David Gow davidgow@google.com Reviewed-by: Daniel Latypov dlatypov@google.com ---

Changes since v3: https://lore.kernel.org/linux-kselftest/20211028064154.2301049-3-davidgow@go... - Fix a change which should've been in patch 4. - Reverse the order of the if conditional: handle non-parameterised case first.

Changes since v2: https://lore.kernel.org/linux-kselftest/20211027013702.2039566-3-davidgow@go... - Rework to not share the loop between the parameterised and non-parameterised test cases. - Suggested by Daniel Latypov. - Avoids using a magic non-zero pointer value.

lib/kunit/test.c | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-)

diff --git a/lib/kunit/test.c b/lib/kunit/test.c index 3bd741e50a2d..f96498ede2cc 100644 --- a/lib/kunit/test.c +++ b/lib/kunit/test.c @@ -504,16 +504,18 @@ int kunit_run_tests(struct kunit_suite *suite) struct kunit_result_stats param_stats = { 0 }; test_case->status = KUNIT_SKIPPED;

- if (test_case->generate_params) { + if (!test_case->generate_params) { + /* Non-parameterised test. */ + kunit_run_case_catch_errors(suite, test_case, &test); + kunit_update_stats(&param_stats, test.status); + } else { /* Get initial param. */ param_desc[0] = '\0'; test.param_value = test_case->generate_params(NULL, param_desc); - }

- do { - kunit_run_case_catch_errors(suite, test_case, &test); + while (test.param_value) { + kunit_run_case_catch_errors(suite, test_case, &test);

- if (test_case->generate_params) { if (param_desc[0] == '\0') { snprintf(param_desc, sizeof(param_desc), "param-%d", test.param_index); @@ -530,11 +532,11 @@ int kunit_run_tests(struct kunit_suite *suite) param_desc[0] = '\0'; test.param_value = test_case->generate_params(test.param_value, param_desc); test.param_index++; - }

- kunit_update_stats(&param_stats, test.status); + kunit_update_stats(&param_stats, test.status); + } + }

- } while (test.param_value);

kunit_print_test_stats(&test, param_stats);

Currently, the results for individial parameters in a parameterised test are simply output as (K)TAP diagnostic lines.

As kunit_tool now supports nested subtests, report each parameter as its own subtest.

For example, here's what the output now looks like: # Subtest: inode_test_xtimestamp_decoding ok 1 - 1901-12-13 Lower bound of 32bit < 0 timestamp, no extra bits ok 2 - 1969-12-31 Upper bound of 32bit < 0 timestamp, no extra bits ok 3 - 1970-01-01 Lower bound of 32bit >=0 timestamp, no extra bits ok 4 - 2038-01-19 Upper bound of 32bit >=0 timestamp, no extra bits ok 5 - 2038-01-19 Lower bound of 32bit <0 timestamp, lo extra sec bit on ok 6 - 2106-02-07 Upper bound of 32bit <0 timestamp, lo extra sec bit on ok 7 - 2106-02-07 Lower bound of 32bit >=0 timestamp, lo extra sec bit on ok 8 - 2174-02-25 Upper bound of 32bit >=0 timestamp, lo extra sec bit on ok 9 - 2174-02-25 Lower bound of 32bit <0 timestamp, hi extra sec bit on ok 10 - 2242-03-16 Upper bound of 32bit <0 timestamp, hi extra sec bit on ok 11 - 2242-03-16 Lower bound of 32bit >=0 timestamp, hi extra sec bit on ok 12 - 2310-04-04 Upper bound of 32bit >=0 timestamp, hi extra sec bit on ok 13 - 2310-04-04 Upper bound of 32bit>=0 timestamp, hi extra sec bit 1. 1 ns ok 14 - 2378-04-22 Lower bound of 32bit>= timestamp. Extra sec bits 1. Max ns ok 15 - 2378-04-22 Lower bound of 32bit >=0 timestamp. All extra sec bits on ok 16 - 2446-05-10 Upper bound of 32bit >=0 timestamp. All extra sec bits on # inode_test_xtimestamp_decoding: pass:16 fail:0 skip:0 total:16 ok 1 - inode_test_xtimestamp_decoding

Signed-off-by: David Gow davidgow@google.com Reviewed-by: Daniel Latypov dlatypov@google.com ---

Changes since v3: https://lore.kernel.org/linux-kselftest/20211028064154.2301049-4-davidgow@go... - Fix the missing log line which ended up in patch 3 by mistake.

Changes since v2: https://lore.kernel.org/linux-kselftest/20211027013702.2039566-4-davidgow@go... - No changes to this patch.

lib/kunit/test.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/lib/kunit/test.c b/lib/kunit/test.c index f96498ede2cc..c7ed4aabec04 100644 --- a/lib/kunit/test.c +++ b/lib/kunit/test.c @@ -512,6 +512,8 @@ int kunit_run_tests(struct kunit_suite *suite) /* Get initial param. */ param_desc[0] = '\0'; test.param_value = test_case->generate_params(NULL, param_desc); + kunit_log(KERN_INFO, &test, KUNIT_SUBTEST_INDENT KUNIT_SUBTEST_INDENT + "# Subtest: %s", test_case->name);

while (test.param_value) { kunit_run_case_catch_errors(suite, test_case, &test); @@ -522,9 +524,8 @@ int kunit_run_tests(struct kunit_suite *suite) }

kunit_log(KERN_INFO, &test, - KUNIT_SUBTEST_INDENT - "# %s: %s %d - %s", - test_case->name, + KUNIT_SUBTEST_INDENT KUNIT_SUBTEST_INDENT + "%s %d - %s", kunit_status_to_ok_not_ok(test.status), test.param_index + 1, param_desc);