On 2020-06-16 18:58, Kees Cook wrote:
On Tue, Jun 16, 2020 at 12:44:28PM -0700, Brendan Higgins wrote:
On Tue, Jun 16, 2020 at 9:42 AM Bird, Tim Tim.Bird@sony.com wrote:
From: Paolo Bonzini pbonzini@redhat.com
On 15/06/20 21:07, Bird, Tim wrote:
Note: making the plan line required differs from TAP13 and TAP14. I think it's the right choice, but we should be clear.
As an aside, where is TAP14?
By TAP14, I was referring to the current, undocumented, KUnit conventions.
Not so. TAP14 is the proposed next version of TAP13:
https://github.com/TestAnything/testanything.github.io/pull/36 https://github.com/isaacs/testanything.github.io/blob/tap14/tap-version-14-s...
I was reading this (I haven't compared to the blob above):
https://github.com/TestAnything/Specification/blob/tap-14-specification/spec...
Based on the discussion, it seems like most of the things we wanted from TAP14 would probably make it in if TAP ever accepts another pull request.
Were our leading diagnostic lines part of their latest spec? I thought we were pretty far off in left field for that particular bit.
My personal preference is to have the dash. I think it's more human readable. I note that the TAP spec has examples of result lines both with and without the dash, so even the spec is ambiguous on this. I think not mandating it either way is probably best. For regex parsers, it's easy to ignore with '[-]?' outside the pattern groups that grab the number and description.
I don't think we care, because we don't use it.
Yeah, I'm in the same place. I don't care -- I would just like a determination. (The "implied" nature of it in TAP14 bothers me.)
XFAIL/XPASS are different from SKIP. I personally don't have a need for them, but kselftests includes XFAIL/XPASS exit codes and they aren't reflected into selftests/kselftest/runner.sh.
Likewise, kselftest.h has ksft_inc_xfail_cnt but not ksft_test_result_xfail/ksft_test_result_xpass.
I proposed fixing that recently[1]. seccomp uses XFAIL for "I have detected you lack the config to test this, so I can't say it's working or not, because it only looks like a failure without the config."
Based on that description, the case sounds like it should be a skip.
Or if the entire test depends on the missing config then Bail out might be appropriate.
[1] https://lore.kernel.org/lkml/20200611224028.3275174-7-keescook@chromium.org/