On Wed, Jan 31, 2024 at 5:22 PM Bird, Tim Tim.Bird@sony.com wrote:
-----Original Message----- From: Rae Moar rmoar@google.com Add specification for test metadata to the KTAP v2 spec.
KTAP v1 only specifies the output format of very basic test information: test result and test name. Any additional test information either gets added to general diagnostic data or is not included in the output at all.
The purpose of KTAP metadata is to create a framework to include and easily identify additional important test information in KTAP.
KTAP metadata could include any test information that is pertinent for user interaction before or after the running of the test. For example, the test file path or the test speed.
Since this includes a large variety of information, this specification will recognize notable types of KTAP metadata to ensure consistent format across test frameworks. See the full list of types in the specification.
Example of KTAP Metadata:
KTAP version 2 # ktap_test: main # ktap_arch: uml 1..1 KTAP version 2 # ktap_test: suite_1 # ktap_subsystem: example # ktap_test_file: lib/test.c 1..2 ok 1 test_1 # ktap_test: test_2 # ktap_speed: very_slow # custom_is_flaky: true ok 2 test_2 ok 1 test_suite
The changes to the KTAP specification outline the format, location, and different types of metadata.
Here is a link to a version of the KUnit parser that is able to parse test metadata lines for KTAP version 2. Note this includes test metadata lines for the main level of KTAP.
Link: https://kunit-review.googlesource.com/c/linux/+/5889
Signed-off-by: Rae Moar rmoar@google.com
Documentation/dev-tools/ktap.rst | 163 ++++++++++++++++++++++++++++++- 1 file changed, 159 insertions(+), 4 deletions(-)
diff --git a/Documentation/dev-tools/ktap.rst b/Documentation/dev-tools/ktap.rst index ff77f4aaa6ef..4480eaf5bbc3 100644 --- a/Documentation/dev-tools/ktap.rst +++ b/Documentation/dev-tools/ktap.rst @@ -17,19 +17,20 @@ KTAP test results describe a series of tests (which may be nested: i.e., test can have subtests), each of which can contain both diagnostic data -- e.g., log lines -- and a final result. The test structure and results are machine-readable, whereas the diagnostic data is unstructured and is there to -aid human debugging. +aid human debugging. One exception to this is test metadata lines - a type +of diagnostic lines. Test metadata is used to identify important supplemental +test information and can be machine-readable.
KTAP output is built from four different types of lines:
- Version lines
- Plan lines
- Test case result lines
-- Diagnostic lines +- Diagnostic lines (including test metadata)
In general, valid KTAP output should also form valid TAP output, but some information, in particular nested test results, may be lost. Also note that there is a stagnant draft specification for TAP14, KTAP diverges from this in -a couple of places (notably the "Subtest" header), which are described where -relevant later in this document. +a couple of places, which are described where relevant later in this document.
Version lines
@@ -166,6 +167,154 @@ even if they do not start with a "#": this is to capture any other useful kernel output which may help debug the test. It is nevertheless recommended that tests always prefix any diagnostic output they have with a "#" character.
+KTAP metadata lines +-------------------
+KTAP metadata lines are a subset of diagnostic lines that are used to include +and easily identify important supplemental test information in KTAP.
+.. code-block:: none
# <prefix>_<metadata type>: <metadata value>+The <prefix> indicates where to find the specification for the type of +metadata. The metadata types listed below use the prefix "ktap" (See Types of +KTAP Metadata).
+Types that are instead specified by an individual test framework use the +framework name as the prefix. For example, a metadata type documented by the +kselftest specification would use the prefix "kselftest". Any metadata type +that is not listed in a specification must use the prefix "custom". Note the +prefix must not include spaces or the characters ":" or "_".
+The format of <metadata type> and <value> varies based on the type. See the +individual specification. For "custom" types the <metadata type> can be any +string excluding ":", spaces, or newline characters and the <value> can be any +string.
+**Location:**
+The first KTAP metadata entry for a test must be "# ktap_test: <test name>", +which acts as a header to associate metadata with the correct test.
+For test cases, the location of the metadata is between the prior test result +line and the current test result line. For test suites, the location of the +metadata is between the suite's version line and test plan line. See the +example below.
+KTAP metadata for a test does not need to be contiguous. For example, a kernel +warning or other diagnostic output could interrupt metadata lines. However, it +is recommended to keep a test's metadata lines together when possible, as this +improves readability.
+**Here is an example of using KTAP metadata:**
+::
KTAP version 2# ktap_test: main# ktap_arch: uml1..1KTAP version 2# ktap_test: suite_1# ktap_subsystem: example# ktap_test_file: lib/test.c1..2ok 1 test_1# ktap_test: test_2# ktap_speed: very_slow# custom_is_flaky: trueok 2 test_2# suite_1 passedok 1 suite_1+In this example, the tests are running on UML. The test suite "suite_1" is part +of the subsystem "example" and belongs to the file "lib/example_test.c". It has
Unless I'm missing something, the file is specified in the example above as list/test.c, not "lib/example_test.c".
Other than that, the documentation looks fine to me. -- Tim
Hello!
Thanks for looking over the documentation. And sorry about that typo in the explanation. It should say "belongs to the file "lib/test.c"". I will change that for the next version.
Thanks! -Rae