On Tue, Jun 16, 2020 at 4:52 PM Kees Cook keescook@chromium.org wrote:
On Mon, Jun 15, 2020 at 07:07:34PM +0000, Bird, Tim wrote:
From: Kees Cook keescook@chromium.org
Note: making the plan line required differs from TAP13 and TAP14. I think it's the right choice, but we should be clear.
[...] With regards to making it optional or not, I don't have a strong preference. The extra info seems helpful in some circumstances. I don't know if it's too onerous to make it a requirement or not. I'd prefer if it was always there (either at the beginning or the end), but if there is some situation where it's quite difficult to calculate, then it would be best not to mandate it. I can't think of any impossible situations at the moment.
I think we should require one of:
- starting plan line
- ending plan line
- ending with something that indicates "I'm done, but I have no idea how many tests actually ran" (Maybe "1..?")
To me, the point of the plan line is to be able to say "this test did, in fact, finish". So even if some test can't even count how many tests it _ran_, it can at least say "I am now finished".
So the counting is actually not the hard part for us, it's figuring out when we have finished. Again, the change that I am working on (I REALLY need to get that out) should fix that, but until we get that upstream, KUnit doesn't actually know when it is done running tests.
TAP13/14 makes description optional, are we making it required (I think we should). There seems to be a TAP13/14 "convention" of starting <description> with "- ", which I'm on the fence about it. It does make parsing maybe a little easier.
I would like the description to be required. I don't have a strong opinion on the dash. I'm OK with either one (dash or no dash), but we should make kselftest and KUnit consistent.
I find the dash to be distracting -- it doesn't help me scan it, and it doesn't help a parser (which only needs to find "#").
Yeah, I also prefer spaces and/or "#". I am okay if spaces are optional only to aid human readability. And honestly I don't care about this point too much. Just offering my 2 cents.
Differences between kernel test result format and TAP13:
- in KTAP the "# SKIP" directive is placed after the description on the test result line
I sent a bunch of clean-ups for kselftest.h recently[1], but it looks like we'll need more for adding "description" to skip (right now it only prints the SKIP reason).
[1] https://lore.kernel.org/lkml/20200611224028.3275174-1-keescook@chromium.org/
Yes Documentation/*.rst Not sure on name yet, but where do kselftest docs live? :)
Documentation/dev-tools/kselftest.rst
I'll put this at: Documentation/dev-tools/test-results-format.rst
Sounds good!
-- Kees Cook