On Sat, Jun 12, 2021 at 4:26 AM Brendan Higgins brendanhiggins@google.com wrote:
On Thu, Jun 10, 2021 at 8:57 PM David Gow davidgow@google.com wrote:
When one parameter of a parameterised test failed, its failure would be propagated to the overall test, but not to the suite result (unless it was the last parameter).
This is because test_case->success was being reset to the test->success result after each parameter was used, so a failing test's result would be overwritten by a non-failing result. The overall test result was handled in a third variable, test_result, but this was disacarded after the status line was printed.
nit: s/disacarded/discarded/g
Instead, just propagate the result after each parameter run.
Signed-off-by: David Gow davidgow@google.com Fixes: fadb08e7c750 ("kunit: Support for Parameterized Testing")
I tried to reproduce the problem described and was unable to. Anyway, from the code it definitely looks like there is a bug like you describe. And it definitely looks like your change should fix it.
I was able to reproduce this again myself. Note that the kunit_tool wrapper does its own result propagation which doesn't have a similar bug, so you won't see the issue in its parsed results. (Using the --raw_output flag does show it, though).
Here's the output from a patched UUID suite, which deliberately fails the first parameter of the first two tests and passes the other parameters and tests, which exhibits the issue:
TAP version 14 1..1 # Subtest: uuid 1..4 # uuid_correct_be: ASSERTION FAILED at lib/test_uuid.c:57 Expected uuid_parse(data->uuid, &be) == 0, but uuid_parse(data->uuid, &be) == -22
failed to parse 'c33fx4995-3701-450e-9fbf-206a2e98e576' # uuid_correct_be: not ok 1 - c33fx4995-3701-450e-9fbf-206a2e98e576 # uuid_correct_be: ok 2 - 64b4371c-77c1-48f9-8221-29f054fc023b # uuid_correct_be: ok 3 - 0cb4ddff-a545-4401-9d06-688af53e7f84 not ok 1 - uuid_correct_be # uuid_correct_le: ASSERTION FAILED at lib/test_uuid.c:46 Expected guid_parse(data->uuid, &le) == 0, but guid_parse(data->uuid, &le) == -22
failed to parse 'c33fx4995-3701-450e-9fbf-206a2e98e576' # uuid_correct_le: not ok 1 - c33fx4995-3701-450e-9fbf-206a2e98e576 # uuid_correct_le: ok 2 - 64b4371c-77c1-48f9-8221-29f054fc023b # uuid_correct_le: ok 3 - 0cb4ddff-a545-4401-9d06-688af53e7f84 not ok 2 - uuid_correct_le # uuid_wrong_be: ok 1 - c33f4995-3701-450e-9fbf206a2e98e576 # uuid_wrong_be: ok 2 - 64b4371c-77c1-48f9-8221-29f054XX023b # uuid_wrong_be: ok 3 - 0cb4ddff-a545-4401-9d06-688af53e ok 3 - uuid_wrong_be # uuid_wrong_le: ok 1 - c33f4995-3701-450e-9fbf206a2e98e576 # uuid_wrong_le: ok 2 - 64b4371c-77c1-48f9-8221-29f054XX023b # uuid_wrong_le: ok 3 - 0cb4ddff-a545-4401-9d06-688af53e ok 4 - uuid_wrong_le ok 1 - uuid
Note the "not ok 1 - uuid_correct_be" line, yet it ending in "ok 1 - uuid".
Anyway, I tried testing your fix, but given I was unable to reproduce the failure, I am not super confident in my testing. Still,
Reviewed-by: Brendan Higgins brendanhiggins@google.com