On 2020-06-16 18:52, Kees Cook wrote:
On Mon, Jun 15, 2020 at 07:07:34PM +0000, Bird, Tim wrote:
From: Kees Cook keescook@chromium.org
Note: making the plan line required differs from TAP13 and TAP14. I think it's the right choice, but we should be clear.
[...] With regards to making it optional or not, I don't have a strong preference. The extra info seems helpful in some circumstances. I don't know if it's too onerous to make it a requirement or not. I'd prefer if it was always there (either at the beginning or the end), but if there is some situation where it's quite difficult to calculate, then it would be best not to mandate it. I can't think of any impossible situations at the moment.
I think we should require one of:
- starting plan line
- ending plan line
- ending with something that indicates "I'm done, but I have no idea how many tests actually ran" (Maybe "1..?")
I understand the desire to be able to say "I don't know how many tests actually ran", but the use of it should be discouraged, even if available.
-Frank
To me, the point of the plan line is to be able to say "this test did, in fact, finish". So even if some test can't even count how many tests it _ran_, it can at least say "I am now finished".
TAP13/14 makes description optional, are we making it required (I think we should). There seems to be a TAP13/14 "convention" of starting <description> with "- ", which I'm on the fence about it. It does make parsing maybe a little easier.
I would like the description to be required. I don't have a strong opinion on the dash. I'm OK with either one (dash or no dash), but we should make kselftest and KUnit consistent.
I find the dash to be distracting -- it doesn't help me scan it, and it doesn't help a parser (which only needs to find "#").
Differences between kernel test result format and TAP13:
- in KTAP the "# SKIP" directive is placed after the description on the test result line
I sent a bunch of clean-ups for kselftest.h recently[1], but it looks like we'll need more for adding "description" to skip (right now it only prints the SKIP reason).
[1] https://lore.kernel.org/lkml/20200611224028.3275174-1-keescook@chromium.org/
Yes Documentation/*.rst Not sure on name yet, but where do kselftest docs live? :)
Documentation/dev-tools/kselftest.rst
I'll put this at: Documentation/dev-tools/test-results-format.rst
Sounds good!