Add specification for declaring test metadata to the KTAP v2 spec.
The purpose of test metadata is to allow for the declaration of essential testing information in KTAP output. This information includes test names, test configuration info, test attributes, and test files.
There have been similar ideas around the idea of test metadata such as test prefixes and test name lines. However, I propose this specification as an overall fix for these issues.
These test metadata lines are a form of diagnostic lines with the format: "# <metadata_type>: <data>". As a type of diagnostic line, test metadata lines are compliant with KTAP v1, which will help to not interfere too much with current parsers.
Specifically the "# Subtest:" line is derived from the TAP 14 spec: https://testanything.org/tap-version-14-specification.html.
The proposed location for test metadata is in the test header, between the version line and the test plan line. Note including diagnostic lines in the test header is a depature from KTAP v1.
This location provides two main benefits:
First, metadata will be printed prior to when subtests are run. Then if a test fails, test metadata can help discern which test is causing the issue and potentially why.
Second, this location ensures that the lines will not be accidentally parsed as a subtest's diagnostic lines because the lines are bordered by the version line and plan line.
Here is an example of test metadata:
KTAP version 2 # Config: CONFIG_TEST=y 1..1 KTAP version 2 # Subtest: test_suite # File: /sys/kernel/... # Attributes: slow # Other: example_test 1..2 ok 1 test_1 ok 2 test_2 ok 1 test_suite
Here is a link to a version of the KUnit parser that is able to parse test metadata lines for KTAP version 2. Note this includes test metadata lines for the main level of KTAP.
Link: https://kunit-review.googlesource.com/c/linux/+/5809
Signed-off-by: Rae Moar rmoar@google.com ---
Hi everyone,
I would like to use this proposal similar to an RFC to gather ideas on the topic of test metadata. Let me know what you think.
I am also interested in brainstorming a list of recognized metadata types. Providing recognized metadata types would be helpful in parsing and displaying test metadata in a useful way.
Current ideas: - "# Subtest: <test_name>" to indicate test name (name must match corresponding result line) - "# Attributes: <attributes list>" to indicate test attributes (list separated by commas) - "# File: <file_path>" to indicate file used in testing
Any other ideas?
Note this proposal replaces two of my previous proposals: "ktap_v2: add recognized test name line" and "ktap_v2: allow prefix to KTAP lines."
Thanks! -Rae
Note: this patch is based on Frank's ktap_spec_version_2 branch.
Documentation/dev-tools/ktap.rst | 51 ++++++++++++++++++++++++++++++-- 1 file changed, 48 insertions(+), 3 deletions(-)
diff --git a/Documentation/dev-tools/ktap.rst b/Documentation/dev-tools/ktap.rst index ff77f4aaa6ef..a2d0a196c115 100644 --- a/Documentation/dev-tools/ktap.rst +++ b/Documentation/dev-tools/ktap.rst @@ -17,7 +17,9 @@ KTAP test results describe a series of tests (which may be nested: i.e., test can have subtests), each of which can contain both diagnostic data -- e.g., log lines -- and a final result. The test structure and results are machine-readable, whereas the diagnostic data is unstructured and is there to -aid human debugging. +aid human debugging. One exception to this is test metadata lines - a type +of diagnostic lines. Test metadata is located between the version line and +plan line of a test and can be machine-readable.
KTAP output is built from four different types of lines: - Version lines @@ -28,8 +30,7 @@ KTAP output is built from four different types of lines: In general, valid KTAP output should also form valid TAP output, but some information, in particular nested test results, may be lost. Also note that there is a stagnant draft specification for TAP14, KTAP diverges from this in -a couple of places (notably the "Subtest" header), which are described where -relevant later in this document. +a couple of places, which are described where relevant later in this document.
Version lines ------------- @@ -166,6 +167,45 @@ even if they do not start with a "#": this is to capture any other useful kernel output which may help debug the test. It is nevertheless recommended that tests always prefix any diagnostic output they have with a "#" character.
+Test metadata lines +------------------- + +Test metadata lines are a type of diagnostic lines used to the declare the +name of a test and other helpful testing information in the test header. +These lines are often helpful for parsing and for providing context during +crashes. + +Test metadata lines must follow the format: "# <metadata_type>: <data>". +These lines must be located between the version line and the plan line +within a test header. + +There are a few currently recognized metadata types: +- "# Subtest: <test_name>" to indicate test name (name must match + corresponding result line) +- "# Attributes: <attributes list>" to indicate test attributes (list + separated by commas) +- "# File: <file_path>" to indicate file used in testing + +As a rule, the "# Subtest:" line is generally first to declare the test +name. Note that metadata lines do not necessarily need to use a +recognized metadata type. + +An example of using metadata lines: + +:: + + KTAP version 2 + 1..1 + # File: /sys/kernel/... + KTAP version 2 + # Subtest: example + # Attributes: slow, example_test + 1..1 + ok 1 test_1 + # example passed + ok 1 example + + Unknown lines -------------
@@ -206,6 +246,7 @@ An example of a test with two nested subtests: KTAP version 2 1..1 KTAP version 2 + # Subtest: example 1..2 ok 1 test_1 not ok 2 test_2 @@ -219,6 +260,7 @@ An example format with multiple levels of nested testing: KTAP version 2 1..2 KTAP version 2 + # Subtest: example_test_1 1..2 KTAP version 2 1..2 @@ -254,6 +296,7 @@ Example KTAP output KTAP version 2 1..1 KTAP version 2 + # Subtest: main_test 1..3 KTAP version 2 1..1 @@ -261,11 +304,13 @@ Example KTAP output ok 1 test_1 ok 1 example_test_1 KTAP version 2 + # Attributes: slow 1..2 ok 1 test_1 # SKIP test_1 skipped ok 2 test_2 ok 2 example_test_2 KTAP version 2 + # Subtest: example_test_3 1..3 ok 1 test_1 # test_2: FAIL
base-commit: 906f02e42adfbd5ae70d328ee71656ecb602aaf5
Hi Rae,
kernel test robot noticed the following build warnings:
[auto build test WARNING on 906f02e42adfbd5ae70d328ee71656ecb602aaf5]
url: https://github.com/intel-lab-lkp/linux/commits/Rae-Moar/ktap_v2-add-test-met... base: 906f02e42adfbd5ae70d328ee71656ecb602aaf5 patch link: https://lore.kernel.org/r/20230420205734.1288498-1-rmoar%40google.com patch subject: [KTAP V2 PATCH] ktap_v2: add test metadata reproduce: # https://github.com/intel-lab-lkp/linux/commit/9a01de0a8d8f9f09023184d9df00aa... git remote add linux-review https://github.com/intel-lab-lkp/linux git fetch --no-tags linux-review Rae-Moar/ktap_v2-add-test-metadata/20230421-045838 git checkout 9a01de0a8d8f9f09023184d9df00aa98a559c871 make menuconfig # enable CONFIG_COMPILE_TEST, CONFIG_WARN_MISSING_DOCUMENTS, CONFIG_WARN_ABI_ERRORS make htmldocs
If you fix the issue, kindly add following tag where applicable | Reported-by: kernel test robot lkp@intel.com | Link: https://lore.kernel.org/oe-kbuild-all/202304221132.H0D5OLHS-lkp@intel.com/
All warnings (new ones prefixed by >>):
Documentation/dev-tools/ktap.rst:184: WARNING: Unexpected indentation. Documentation/dev-tools/ktap.rst:185: WARNING: Block quote ends without a blank line; unexpected unindent.
vim +184 Documentation/dev-tools/ktap.rst
181 182 There are a few currently recognized metadata types: 183 - "# Subtest: <test_name>" to indicate test name (name must match
184 corresponding result line) 185 - "# Attributes: <attributes list>" to indicate test attributes (list
186 separated by commas) 187 - "# File: <file_path>" to indicate file used in testing 188
On 4/20/23 15:57, Rae Moar wrote:
Add specification for declaring test metadata to the KTAP v2 spec.
The purpose of test metadata is to allow for the declaration of essential testing information in KTAP output. This information includes test names, test configuration info, test attributes, and test files.
There have been similar ideas around the idea of test metadata such as test prefixes and test name lines. However, I propose this specification as an overall fix for these issues.
This seems like a cleaner approach.
These test metadata lines are a form of diagnostic lines with the format: "# <metadata_type>: <data>". As a type of diagnostic line, test metadata lines are compliant with KTAP v1, which will help to not interfere too much with current parsers.
Specifically the "# Subtest:" line is derived from the TAP 14 spec: https://testanything.org/tap-version-14-specification.html.
The proposed location for test metadata is in the test header, between the version line and the test plan line. Note including diagnostic lines in the test header is a depature from KTAP v1.
This location provides two main benefits:
First, metadata will be printed prior to when subtests are run. Then if a test fails, test metadata can help discern which test is causing the issue and potentially why.
Second, this location ensures that the lines will not be accidentally parsed as a subtest's diagnostic lines because the lines are bordered by the version line and plan line.
I like that.
Here is an example of test metadata:
KTAP version 2 # Config: CONFIG_TEST=y 1..1 KTAP version 2 # Subtest: test_suite # File: /sys/kernel/... # Attributes: slow # Other: example_test 1..2 ok 1 test_1 ok 2 test_2 ok 1 test_suite
Here is a link to a version of the KUnit parser that is able to parse test metadata lines for KTAP version 2. Note this includes test metadata lines for the main level of KTAP.
Link: https://kunit-review.googlesource.com/c/linux/+/5809
Signed-off-by: Rae Moar rmoar@google.com
Hi everyone,
I would like to use this proposal similar to an RFC to gather ideas on the topic of test metadata. Let me know what you think.
I am also interested in brainstorming a list of recognized metadata types. Providing recognized metadata types would be helpful in parsing and displaying test metadata in a useful way.
Current ideas:
- "# Subtest: <test_name>" to indicate test name (name must match corresponding result line)
I would prefer "Test" to "Subtest" because the type should be allowed for the top level test, as well as for subtest levels.
- "# Attributes: <attributes list>" to indicate test attributes (list separated by commas)
- "# File: <file_path>" to indicate file used in testing
Any other ideas?
(Already used in an example above...)
- "# Config: <config_option list> to indicate kernel configuration options (list separated by commas)
config_option format: Option XXX is enabled: CONFIG_XXX=y Option XXX is not enabled: CONFIG_XXX=n Option XXX is text: CONFIG_XXX="a text string"
Linux .config format is "#CONFIG_XXX is not set", but this would be harder to parse in a list.
A text config option also complicates parsing of a list. Maybe there should not be a list, instead have a separate "# Config:" line for each config option.
I would like to bifurcate the name space of metadata types, to names specified in the standard vs names not in the standard that can be used on an experimental or for future use in existing tests.
I can think of at least two ways to implement this:
(1) types that are in the specification all begin with a specific prefix, such as "ktap_" (bike shedding on naming welcomed), so the examples woudld be
# ktap_test: # ktap_attributes: # ktap_file: # ktap_config:
(2) types that are _not_ in the specification all begin with a specific prefix, such as "custom_" (bike shedding on naming welcomed).
Note this proposal replaces two of my previous proposals: "ktap_v2: add recognized test name line" and "ktap_v2: allow prefix to KTAP lines."
Thanks! -Rae
Note: this patch is based on Frank's ktap_spec_version_2 branch.
Documentation/dev-tools/ktap.rst | 51 ++++++++++++++++++++++++++++++-- 1 file changed, 48 insertions(+), 3 deletions(-)
diff --git a/Documentation/dev-tools/ktap.rst b/Documentation/dev-tools/ktap.rst index ff77f4aaa6ef..a2d0a196c115 100644 --- a/Documentation/dev-tools/ktap.rst +++ b/Documentation/dev-tools/ktap.rst @@ -17,7 +17,9 @@ KTAP test results describe a series of tests (which may be nested: i.e., test can have subtests), each of which can contain both diagnostic data -- e.g., log lines -- and a final result. The test structure and results are machine-readable, whereas the diagnostic data is unstructured and is there to -aid human debugging. +aid human debugging. One exception to this is test metadata lines - a type +of diagnostic lines. Test metadata is located between the version line and +plan line of a test and can be machine-readable. KTAP output is built from four different types of lines:
- Version lines
@@ -28,8 +30,7 @@ KTAP output is built from four different types of lines: In general, valid KTAP output should also form valid TAP output, but some information, in particular nested test results, may be lost. Also note that there is a stagnant draft specification for TAP14, KTAP diverges from this in -a couple of places (notably the "Subtest" header), which are described where -relevant later in this document. +a couple of places, which are described where relevant later in this document. Version lines
@@ -166,6 +167,45 @@ even if they do not start with a "#": this is to capture any other useful kernel output which may help debug the test. It is nevertheless recommended that tests always prefix any diagnostic output they have with a "#" character. +Test metadata lines +-------------------
+Test metadata lines are a type of diagnostic lines used to the declare the +name of a test and other helpful testing information in the test header. +These lines are often helpful for parsing and for providing context during +crashes.
+Test metadata lines must follow the format: "# <metadata_type>: <data>". +These lines must be located between the version line and the plan line +within a test header.
+There are a few currently recognized metadata types: +- "# Subtest: <test_name>" to indicate test name (name must match
- corresponding result line)
+- "# Attributes: <attributes list>" to indicate test attributes (list
- separated by commas)
+- "# File: <file_path>" to indicate file used in testing
+As a rule, the "# Subtest:" line is generally first to declare the test +name. Note that metadata lines do not necessarily need to use a +recognized metadata type.
+An example of using metadata lines:
+::
KTAP version 21..1# File: /sys/kernel/...KTAP version 2# Subtest: example# Attributes: slow, example_test1..1ok 1 test_1# example passedok 1 exampleUnknown lines
@@ -206,6 +246,7 @@ An example of a test with two nested subtests: KTAP version 2 1..1 KTAP version 2
# Subtest: example@@ -219,6 +260,7 @@ An example format with multiple levels of nested testing: KTAP version 2 1..2 KTAP version 2
# Subtest: example_test_1@@ -254,6 +296,7 @@ Example KTAP output KTAP version 2 1..1 KTAP version 2
# Subtest: main_test@@ -261,11 +304,13 @@ Example KTAP output ok 1 test_1 ok 1 example_test_1 KTAP version 2
# Attributes: slow 1..2 ok 1 test_1 # SKIP test_1 skipped ok 2 test_2# Subtest: example_test_3 1..3 ok 1 test_1 # test_2: FAILbase-commit: 906f02e42adfbd5ae70d328ee71656ecb602aaf5
On Tue, Apr 25, 2023 at 4:55 PM Frank Rowand frowand.list@gmail.com wrote:
On 4/20/23 15:57, Rae Moar wrote:
Add specification for declaring test metadata to the KTAP v2 spec.
The purpose of test metadata is to allow for the declaration of essential testing information in KTAP output. This information includes test names, test configuration info, test attributes, and test files.
There have been similar ideas around the idea of test metadata such as test prefixes and test name lines. However, I propose this specification as an overall fix for these issues.
This seems like a cleaner approach.
These test metadata lines are a form of diagnostic lines with the format: "# <metadata_type>: <data>". As a type of diagnostic line, test metadata lines are compliant with KTAP v1, which will help to not interfere too much with current parsers.
Specifically the "# Subtest:" line is derived from the TAP 14 spec: https://testanything.org/tap-version-14-specification.html.
The proposed location for test metadata is in the test header, between the version line and the test plan line. Note including diagnostic lines in the test header is a depature from KTAP v1.
This location provides two main benefits:
First, metadata will be printed prior to when subtests are run. Then if a test fails, test metadata can help discern which test is causing the issue and potentially why.
Second, this location ensures that the lines will not be accidentally parsed as a subtest's diagnostic lines because the lines are bordered by the version line and plan line.
I like that.
Here is an example of test metadata:
KTAP version 2 # Config: CONFIG_TEST=y 1..1 KTAP version 2 # Subtest: test_suite # File: /sys/kernel/... # Attributes: slow # Other: example_test 1..2 ok 1 test_1 ok 2 test_2 ok 1 test_suite
Here is a link to a version of the KUnit parser that is able to parse test metadata lines for KTAP version 2. Note this includes test metadata lines for the main level of KTAP.
Link: https://kunit-review.googlesource.com/c/linux/+/5809
Signed-off-by: Rae Moar rmoar@google.com
Hi everyone,
I would like to use this proposal similar to an RFC to gather ideas on the topic of test metadata. Let me know what you think.
I am also interested in brainstorming a list of recognized metadata types. Providing recognized metadata types would be helpful in parsing and displaying test metadata in a useful way.
Current ideas:
- "# Subtest: <test_name>" to indicate test name (name must match corresponding result line)
I would prefer "Test" to "Subtest" because the type should be allowed for the top level test, as well as for subtest levels.
Hi Frank!
Yes, I can see the reasoning to switch to "Test". Although this is a departure from current behavior, it would be clearer. I am happy to make this change.
- "# Attributes: <attributes list>" to indicate test attributes (list separated by commas)
- "# File: <file_path>" to indicate file used in testing
Any other ideas?
(Already used in an example above...)
"# Config: <config_option list> to indicate kernel configuration options (list separated by commas)
config_option format: Option XXX is enabled: CONFIG_XXX=y Option XXX is not enabled: CONFIG_XXX=n Option XXX is text: CONFIG_XXX="a text string"
I like this addition of the "Config" metadata. I also like all of these format options, including the text string option. Although, I would be interested in adding "Option XXX is loadable as a module: CONFIG_XXX=m" to the format list.
Linux .config format is "#CONFIG_XXX is not set", but this would be harder to parse in a list.
A text config option also complicates parsing of a list. Maybe there should not be a list, instead have a separate "# Config:" line for each config option.
I'm not sure how to deal with multiple config options. I am split between either using a list or multiple "Config" lines. I would be happy with either approach. Maybe a list would be slightly better, since it is slightly closer to the defined behavior for the attributes metadata line.
I would like to bifurcate the name space of metadata types, to names specified in the standard vs names not in the standard that can be used on an experimental or for future use in existing tests.
I can think of at least two ways to implement this:
(1) types that are in the specification all begin with a specific prefix, such as "ktap_" (bike shedding on naming welcomed), so the examples woudld be
# ktap_test: # ktap_attributes: # ktap_file: # ktap_config:
(2) types that are _not_ in the specification all begin with a specific prefix, such as "custom_" (bike shedding on naming welcomed).
This is an interesting proposal. I like this idea of using a prefix. I would be happy to add this. I like "ktap_" and "custom_".
Thanks! -Rae
Note this proposal replaces two of my previous proposals: "ktap_v2: add recognized test name line" and "ktap_v2: allow prefix to KTAP lines."
Thanks! -Rae
Note: this patch is based on Frank's ktap_spec_version_2 branch.
Documentation/dev-tools/ktap.rst | 51 ++++++++++++++++++++++++++++++-- 1 file changed, 48 insertions(+), 3 deletions(-)
diff --git a/Documentation/dev-tools/ktap.rst b/Documentation/dev-tools/ktap.rst index ff77f4aaa6ef..a2d0a196c115 100644 --- a/Documentation/dev-tools/ktap.rst +++ b/Documentation/dev-tools/ktap.rst @@ -17,7 +17,9 @@ KTAP test results describe a series of tests (which may be nested: i.e., test can have subtests), each of which can contain both diagnostic data -- e.g., log lines -- and a final result. The test structure and results are machine-readable, whereas the diagnostic data is unstructured and is there to -aid human debugging. +aid human debugging. One exception to this is test metadata lines - a type +of diagnostic lines. Test metadata is located between the version line and +plan line of a test and can be machine-readable.
KTAP output is built from four different types of lines:
- Version lines
@@ -28,8 +30,7 @@ KTAP output is built from four different types of lines: In general, valid KTAP output should also form valid TAP output, but some information, in particular nested test results, may be lost. Also note that there is a stagnant draft specification for TAP14, KTAP diverges from this in -a couple of places (notably the "Subtest" header), which are described where -relevant later in this document. +a couple of places, which are described where relevant later in this document.
Version lines
@@ -166,6 +167,45 @@ even if they do not start with a "#": this is to capture any other useful kernel output which may help debug the test. It is nevertheless recommended that tests always prefix any diagnostic output they have with a "#" character.
+Test metadata lines +-------------------
+Test metadata lines are a type of diagnostic lines used to the declare the +name of a test and other helpful testing information in the test header. +These lines are often helpful for parsing and for providing context during +crashes.
+Test metadata lines must follow the format: "# <metadata_type>: <data>". +These lines must be located between the version line and the plan line +within a test header.
+There are a few currently recognized metadata types: +- "# Subtest: <test_name>" to indicate test name (name must match
- corresponding result line)
+- "# Attributes: <attributes list>" to indicate test attributes (list
- separated by commas)
+- "# File: <file_path>" to indicate file used in testing
+As a rule, the "# Subtest:" line is generally first to declare the test +name. Note that metadata lines do not necessarily need to use a +recognized metadata type.
+An example of using metadata lines:
+::
KTAP version 21..1# File: /sys/kernel/...KTAP version 2# Subtest: example# Attributes: slow, example_test1..1ok 1 test_1# example passedok 1 exampleUnknown lines
@@ -206,6 +246,7 @@ An example of a test with two nested subtests: KTAP version 2 1..1 KTAP version 2
# Subtest: example 1..2 ok 1 test_1 not ok 2 test_2@@ -219,6 +260,7 @@ An example format with multiple levels of nested testing: KTAP version 2 1..2 KTAP version 2
# Subtest: example_test_1 1..2 KTAP version 2 1..2@@ -254,6 +296,7 @@ Example KTAP output KTAP version 2 1..1 KTAP version 2
# Subtest: main_test 1..3 KTAP version 2 1..1@@ -261,11 +304,13 @@ Example KTAP output ok 1 test_1 ok 1 example_test_1 KTAP version 2
# Attributes: slow 1..2 ok 1 test_1 # SKIP test_1 skipped ok 2 test_2 ok 2 example_test_2 KTAP version 2# Subtest: example_test_3 1..3 ok 1 test_1 # test_2: FAILbase-commit: 906f02e42adfbd5ae70d328ee71656ecb602aaf5
On 4/26/23 11:18, Rae Moar wrote:
On Tue, Apr 25, 2023 at 4:55 PM Frank Rowand frowand.list@gmail.com wrote:
On 4/20/23 15:57, Rae Moar wrote:
Add specification for declaring test metadata to the KTAP v2 spec.
The purpose of test metadata is to allow for the declaration of essential testing information in KTAP output. This information includes test names, test configuration info, test attributes, and test files.
There have been similar ideas around the idea of test metadata such as test prefixes and test name lines. However, I propose this specification as an overall fix for these issues.
This seems like a cleaner approach.
These test metadata lines are a form of diagnostic lines with the format: "# <metadata_type>: <data>". As a type of diagnostic line, test metadata lines are compliant with KTAP v1, which will help to not interfere too much with current parsers.
Specifically the "# Subtest:" line is derived from the TAP 14 spec: https://testanything.org/tap-version-14-specification.html.
The proposed location for test metadata is in the test header, between the version line and the test plan line. Note including diagnostic lines in the test header is a depature from KTAP v1.
This location provides two main benefits:
First, metadata will be printed prior to when subtests are run. Then if a test fails, test metadata can help discern which test is causing the issue and potentially why.
Second, this location ensures that the lines will not be accidentally parsed as a subtest's diagnostic lines because the lines are bordered by the version line and plan line.
I like that.
Here is an example of test metadata:
KTAP version 2 # Config: CONFIG_TEST=y 1..1 KTAP version 2 # Subtest: test_suite # File: /sys/kernel/... # Attributes: slow # Other: example_test 1..2 ok 1 test_1 ok 2 test_2 ok 1 test_suite
Here is a link to a version of the KUnit parser that is able to parse test metadata lines for KTAP version 2. Note this includes test metadata lines for the main level of KTAP.
Link: https://kunit-review.googlesource.com/c/linux/+/5809
Signed-off-by: Rae Moar rmoar@google.com
Hi everyone,
I would like to use this proposal similar to an RFC to gather ideas on the topic of test metadata. Let me know what you think.
I am also interested in brainstorming a list of recognized metadata types. Providing recognized metadata types would be helpful in parsing and displaying test metadata in a useful way.
Current ideas:
- "# Subtest: <test_name>" to indicate test name (name must match corresponding result line)
I would prefer "Test" to "Subtest" because the type should be allowed for the top level test, as well as for subtest levels.
Hi Frank!
Yes, I can see the reasoning to switch to "Test". Although this is a departure from current behavior, it would be clearer. I am happy to make this change.
- "# Attributes: <attributes list>" to indicate test attributes (list separated by commas)
Should the spec say that <attributes list> is test specific (and defined by the test) and not defined in the ktap specification?
I think there is a balance between under specifying and over specifying within the ktap specification. But I think the specification should be clear about what definition is left open to the test implementations.
My wishy washy thought for the general case is that the ktap specification should leave as much as choice as possible to the test implementers, but should provide as much detail as needed to provide general guidance to parser implementers.
- "# File: <file_path>" to indicate file used in testing
nit question: Should both relative paths and absolute paths be allowed for <file_path>? I think both have value in different contexts.
Any other ideas?
(Already used in an example above...)
"# Config: <config_option list> to indicate kernel configuration options (list separated by commas)
config_option format: Option XXX is enabled: CONFIG_XXX=y Option XXX is not enabled: CONFIG_XXX=n Option XXX is text: CONFIG_XXX="a text string"
I like this addition of the "Config" metadata. I also like all of these format options, including the text string option. Although, I would be interested in adding "Option XXX is loadable as a module: CONFIG_XXX=m" to the format list.
Yes, definitely. I thought I was blanking on an additional form of a config option when I wrote that.
Linux .config format is "#CONFIG_XXX is not set", but this would be harder to parse in a list.
A text config option also complicates parsing of a list. Maybe there should not be a list, instead have a separate "# Config:" line for each config option.
I'm not sure how to deal with multiple config options. I am split between either using a list or multiple "Config" lines. I would be happy with either approach. Maybe a list would be slightly better, since it is slightly closer to the defined behavior for the attributes metadata line.
I slightly prefer the single option per line. :-)
I would like to bifurcate the name space of metadata types, to names specified in the standard vs names not in the standard that can be used on an experimental or for future use in existing tests.
I can think of at least two ways to implement this:
(1) types that are in the specification all begin with a specific prefix, such as "ktap_" (bike shedding on naming welcomed), so the examples woudld be
# ktap_test: # ktap_attributes: # ktap_file: # ktap_config:
(2) types that are _not_ in the specification all begin with a specific prefix, such as "custom_" (bike shedding on naming welcomed).
This is an interesting proposal. I like this idea of using a prefix. I would be happy to add this. I like "ktap_" and "custom_".
Those prefixes look good to me.
-Frank
Thanks! -Rae
Note this proposal replaces two of my previous proposals: "ktap_v2: add recognized test name line" and "ktap_v2: allow prefix to KTAP lines."
Thanks! -Rae
Note: this patch is based on Frank's ktap_spec_version_2 branch.
Documentation/dev-tools/ktap.rst | 51 ++++++++++++++++++++++++++++++-- 1 file changed, 48 insertions(+), 3 deletions(-)
diff --git a/Documentation/dev-tools/ktap.rst b/Documentation/dev-tools/ktap.rst index ff77f4aaa6ef..a2d0a196c115 100644 --- a/Documentation/dev-tools/ktap.rst +++ b/Documentation/dev-tools/ktap.rst @@ -17,7 +17,9 @@ KTAP test results describe a series of tests (which may be nested: i.e., test can have subtests), each of which can contain both diagnostic data -- e.g., log lines -- and a final result. The test structure and results are machine-readable, whereas the diagnostic data is unstructured and is there to -aid human debugging. +aid human debugging. One exception to this is test metadata lines - a type +of diagnostic lines. Test metadata is located between the version line and +plan line of a test and can be machine-readable.
KTAP output is built from four different types of lines:
- Version lines
@@ -28,8 +30,7 @@ KTAP output is built from four different types of lines: In general, valid KTAP output should also form valid TAP output, but some information, in particular nested test results, may be lost. Also note that there is a stagnant draft specification for TAP14, KTAP diverges from this in -a couple of places (notably the "Subtest" header), which are described where -relevant later in this document. +a couple of places, which are described where relevant later in this document.
Version lines
@@ -166,6 +167,45 @@ even if they do not start with a "#": this is to capture any other useful kernel output which may help debug the test. It is nevertheless recommended that tests always prefix any diagnostic output they have with a "#" character.
+Test metadata lines +-------------------
+Test metadata lines are a type of diagnostic lines used to the declare the +name of a test and other helpful testing information in the test header. +These lines are often helpful for parsing and for providing context during +crashes.
+Test metadata lines must follow the format: "# <metadata_type>: <data>". +These lines must be located between the version line and the plan line +within a test header.
+There are a few currently recognized metadata types: +- "# Subtest: <test_name>" to indicate test name (name must match
- corresponding result line)
+- "# Attributes: <attributes list>" to indicate test attributes (list
- separated by commas)
+- "# File: <file_path>" to indicate file used in testing
+As a rule, the "# Subtest:" line is generally first to declare the test +name. Note that metadata lines do not necessarily need to use a +recognized metadata type.
+An example of using metadata lines:
+::
KTAP version 21..1# File: /sys/kernel/...KTAP version 2# Subtest: example# Attributes: slow, example_test1..1ok 1 test_1# example passedok 1 exampleUnknown lines
@@ -206,6 +246,7 @@ An example of a test with two nested subtests: KTAP version 2 1..1 KTAP version 2
# Subtest: example 1..2 ok 1 test_1 not ok 2 test_2@@ -219,6 +260,7 @@ An example format with multiple levels of nested testing: KTAP version 2 1..2 KTAP version 2
# Subtest: example_test_1 1..2 KTAP version 2 1..2@@ -254,6 +296,7 @@ Example KTAP output KTAP version 2 1..1 KTAP version 2
# Subtest: main_test 1..3 KTAP version 2 1..1@@ -261,11 +304,13 @@ Example KTAP output ok 1 test_1 ok 1 example_test_1 KTAP version 2
# Attributes: slow 1..2 ok 1 test_1 # SKIP test_1 skipped ok 2 test_2 ok 2 example_test_2 KTAP version 2# Subtest: example_test_3 1..3 ok 1 test_1 # test_2: FAILbase-commit: 906f02e42adfbd5ae70d328ee71656ecb602aaf5
On Thu, Apr 20, 2023 at 4:57 PM Rae Moar rmoar@google.com wrote:
Add specification for declaring test metadata to the KTAP v2 spec.
The purpose of test metadata is to allow for the declaration of essential testing information in KTAP output. This information includes test names, test configuration info, test attributes, and test files.
There have been similar ideas around the idea of test metadata such as test prefixes and test name lines. However, I propose this specification as an overall fix for these issues.
These test metadata lines are a form of diagnostic lines with the format: "# <metadata_type>: <data>". As a type of diagnostic line, test metadata lines are compliant with KTAP v1, which will help to not interfere too much with current parsers.
Specifically the "# Subtest:" line is derived from the TAP 14 spec: https://testanything.org/tap-version-14-specification.html.
The proposed location for test metadata is in the test header, between the version line and the test plan line. Note including diagnostic lines in the test header is a depature from KTAP v1.
This location provides two main benefits:
First, metadata will be printed prior to when subtests are run. Then if a test fails, test metadata can help discern which test is causing the issue and potentially why.
Second, this location ensures that the lines will not be accidentally parsed as a subtest's diagnostic lines because the lines are bordered by the version line and plan line.
Here is an example of test metadata:
KTAP version 2 # Config: CONFIG_TEST=y 1..1 KTAP version 2 # Subtest: test_suite # File: /sys/kernel/... # Attributes: slow # Other: example_test 1..2 ok 1 test_1 ok 2 test_2 ok 1 test_suite
Hi everyone!
I have been doing some more thinking on KTAP Metadata as I have been working on the KUnit Test Attributes patch set (https://lore.kernel.org/all/20230719222338.259684-1-rmoar@google.com/). Two additional ideas have come up in the discussion:
1) I wonder if it would be easier to separate "ktap_attributes" into individual attributes.
The two proposed KUnit attributes currently are speed and module name. I think it would be easier for parsing and reading if these attributes had corresponding "ktap_speed" and "ktap_module" categories. Then, in the future if there are too many attributes to print on separate lines they could be grouped into a "ktap_attributes" category later.
2) I wonder if we can shift the concept of KTAP metadata to all tests rather than just suites.
I think it would be very valuable to have a KTAP metadata format that is flexible to work for both suites and test cases. To transition this to test cases, I propose we would use the same format we have been discussing but just printed just before the test result line (David Gow originally came up with this idea). This would look something like this:
KTAP version 2 # ktap_config: CONFIG_TEST=y 1..1 KTAP version 2 # ktap_test: test_suite # ktap_module: example 1..2 ok 1 test_1 # ktap_test: test_2 # ktap_speed: slow # test initializing // diagnostic data ok 2 test_2 ok 1 test_suite
I don't love using the "ktap_test: test_2" line since the test name is repeated. However, I like that this mirrors the same format used for a suite and I currently think it is the best way to define the start of the metadata header.
The test name line could actually be useful by providing context for any test diagnostic data printed below or if the test crashes while running.
What do people think of these ideas?
Thanks! -Rae
Here is a link to a version of the KUnit parser that is able to parse test metadata lines for KTAP version 2. Note this includes test metadata lines for the main level of KTAP.
Link: https://kunit-review.googlesource.com/c/linux/+/5809
Signed-off-by: Rae Moar rmoar@google.com
Hi everyone,
I would like to use this proposal similar to an RFC to gather ideas on the topic of test metadata. Let me know what you think.
I am also interested in brainstorming a list of recognized metadata types. Providing recognized metadata types would be helpful in parsing and displaying test metadata in a useful way.
Current ideas:
- "# Subtest: <test_name>" to indicate test name (name must match corresponding result line)
- "# Attributes: <attributes list>" to indicate test attributes (list separated by commas)
- "# File: <file_path>" to indicate file used in testing
Any other ideas?
Note this proposal replaces two of my previous proposals: "ktap_v2: add recognized test name line" and "ktap_v2: allow prefix to KTAP lines."
Thanks! -Rae
Note: this patch is based on Frank's ktap_spec_version_2 branch.
On 7/20/23 16:31, Rae Moar wrote:
On Thu, Apr 20, 2023 at 4:57 PM Rae Moar rmoar@google.com wrote:
Add specification for declaring test metadata to the KTAP v2 spec.
The purpose of test metadata is to allow for the declaration of essential testing information in KTAP output. This information includes test names, test configuration info, test attributes, and test files.
There have been similar ideas around the idea of test metadata such as test prefixes and test name lines. However, I propose this specification as an overall fix for these issues.
These test metadata lines are a form of diagnostic lines with the format: "# <metadata_type>: <data>". As a type of diagnostic line, test metadata lines are compliant with KTAP v1, which will help to not interfere too much with current parsers.
Specifically the "# Subtest:" line is derived from the TAP 14 spec: https://testanything.org/tap-version-14-specification.html.
The proposed location for test metadata is in the test header, between the version line and the test plan line. Note including diagnostic lines in the test header is a depature from KTAP v1.
This location provides two main benefits:
First, metadata will be printed prior to when subtests are run. Then if a test fails, test metadata can help discern which test is causing the issue and potentially why.
Second, this location ensures that the lines will not be accidentally parsed as a subtest's diagnostic lines because the lines are bordered by the version line and plan line.
Here is an example of test metadata:
KTAP version 2 # Config: CONFIG_TEST=y 1..1 KTAP version 2 # Subtest: test_suite # File: /sys/kernel/... # Attributes: slow # Other: example_test 1..2 ok 1 test_1 ok 2 test_2 ok 1 test_suite
Hi everyone!
I have been doing some more thinking on KTAP Metadata as I have been working on the KUnit Test Attributes patch set (https://lore.kernel.org/all/20230719222338.259684-1-rmoar@google.com/). Two additional ideas have come up in the discussion:
- I wonder if it would be easier to separate "ktap_attributes" into
individual attributes.
The two proposed KUnit attributes currently are speed and module name. I think it would be easier for parsing and reading if these attributes had corresponding "ktap_speed" and "ktap_module" categories. Then, in the future if there are too many attributes to print on separate lines they could be grouped into a "ktap_attributes" category later.
I am fine with the above. This basically removes the special case of "attribute", and what we have been calling attributes are now metadata, just like any other metadata.
- I wonder if we can shift the concept of KTAP metadata to all tests
rather than just suites.
I think it would be very valuable to have a KTAP metadata format that is flexible to work for both suites and test cases. To transition this to test cases, I propose we would use the same format we have been discussing but just printed just before the test result line (David Gow originally came up with this idea). This would look something like this:
KTAP version 2 # ktap_config: CONFIG_TEST=y 1..1 KTAP version 2 # ktap_test: test_suite # ktap_module: example 1..2 ok 1 test_1 # ktap_test: test_2 # ktap_speed: slow # test initializing // diagnostic data ok 2 test_2 ok 1 test_suite
That is the way I would expect metadata to exist. I think I had implicitly expected test metadata and suite metadata to be of the same format.
It seems to me that "suite" is more a kunit concept than a KTAP concept. The kunit suite naturally maps into the top level KTAP test, but I don't think that KTAP should use the term "suite".
I don't love using the "ktap_test: test_2" line since the test name is repeated. However, I like that this mirrors the same format used for a suite and I currently think it is the best way to define the start of the metadata header.
The test name line could actually be useful by providing context for any test diagnostic data printed below or if the test crashes while running.
What do people think of these ideas?
Sounds good to me.
-Frank
Thanks! -Rae
Here is a link to a version of the KUnit parser that is able to parse test metadata lines for KTAP version 2. Note this includes test metadata lines for the main level of KTAP.
Link: https://kunit-review.googlesource.com/c/linux/+/5809
Signed-off-by: Rae Moar rmoar@google.com
Hi everyone,
I would like to use this proposal similar to an RFC to gather ideas on the topic of test metadata. Let me know what you think.
I am also interested in brainstorming a list of recognized metadata types. Providing recognized metadata types would be helpful in parsing and displaying test metadata in a useful way.
Current ideas:
- "# Subtest: <test_name>" to indicate test name (name must match corresponding result line)
- "# Attributes: <attributes list>" to indicate test attributes (list separated by commas)
- "# File: <file_path>" to indicate file used in testing
Any other ideas?
Note this proposal replaces two of my previous proposals: "ktap_v2: add recognized test name line" and "ktap_v2: allow prefix to KTAP lines."
Thanks! -Rae
Note: this patch is based on Frank's ktap_spec_version_2 branch.
linux-kselftest-mirror@lists.linaro.org
-
Frank Rowand -
kernel test robot -
Rae Moar