On Wed, Jun 10, 2020 at 06:11:06PM +0000, Bird, Tim wrote:
Some months ago I started work on a document to formalize how kselftest implements the TAP specification. However, I didn't finish that work. Maybe it's time to do so now.
kselftest has developed a few differences from the original TAP specification, and some extensions that I believe are worth documenting.
Essentially, we have created our own KTAP (kernel TAP) format. I think it is worth documenting our conventions, in order to keep everyone on the same page.
Below is a partially completed document on my understanding of KTAP, based on examination of some of the kselftest test output. I have not reconciled this with the kunit output format, which I believe has some differences (which maybe we should resolve before we get too far into this).
I submit the document now, before it is finished, because a patch was recently introduced to alter one of the result conventions (from SKIP='not ok' to SKIP='ok').
See the document include inline below
====== start of ktap-doc-rfc.txt ======
[...]
--- from here on is not-yet-organized material
Tip:
- don't change the test plan based on skipped tests.
- it is better to report that a test case was skipped, than to not report it
- that is, don't adjust the number of test cases based on skipped tests
Other things to mention: TAP13 elements not used:
- yaml for diagnostic messages
We talked about this before, but I would like some way to get failed expectation/assertion information in the test in a consistent machine parsible way. Currently we do the following:
# Subtest: example 1..1 # example_simple_test: initializing # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29 Expected 1 + 1 == 3, but 1 + 1 == 2 3 == 3 not ok 1 - example_simple_test not ok 5 - example
Technically not TAP compliant, but no one seems to mind. I am okay with keeping it the way it is, but if we don't want it in the KTAP spec, we will need some kind of recourse.
- reason: try to keep things line-based, since output from other things
may be interspersed with messages from the test itself
- TODO directive
Is this more of stating a fact or desire? We don't use TODO either, but it looks like it could be useful.
KTAP Extensions beyond TAP13:
- nesting
- via indentation
- indentation makes it easier for humans to read
- test identifier
- multiple parts, separated by ':'
Can you elabroate on this more? I am not sure what you mean.
- summary lines
- can be skipped by CI systems that do their own calculations
Other notes:
- automatic assignment of result status based on exit code
Tips:
- do NOT describe the result in the test line
- the test case description should be the same whether the test succeeds or fails
- use diagnostic lines to describe or explain results, if this is desirable
- test numbers are considered harmful
- test harnesses should use the test description as the identifier
- test numbers change when testcases are added or removed
- which means that results can't be compared between different versions of the test
- recommendations for diagnostic messages:
- reason for failure
- reason for skip
- diagnostic data should always preceding the result line
- problem: harness may emit result before test can do assessment to determine reason for result
- this is what the kernel uses
Differences between kernel test result format and TAP13:
- in KTAP the "# SKIP" directive is placed after the description on the test result line
====== start of ktap-doc-rfc.txt ====== OK - that's the end of the RFC doc.
Here are a few questions:
- is this document desired or not?
- is it too long or too short?
- if the document is desired, where should it be placed?
I like it. I don't think we can rely on the TAP people updating their stuff based on my interactions with them. So having a spec which is actually maintained would be nice.
Maybe in Documentation/dev-tools/ ?
I assume somewhere under Documentation, and put into .rst format. Suggestions for a name and location are welcome.
- is this document accurate? I think KUNIT does a few things differently than this description.
- is the intent to have kunit and kselftest have the same output format? if so, then these should be rationalized.
Yeah, I think it would be nice if all test frameworks/libraries for the kernel output tests in the same language.
Cheers