Hi Jon, Shuah & others,
I'd like to discuss with you with regards to test documentation.
I had some preliminary discussions with people interested on improving tests during EOSS last week in Prague, as we're working to improve media test coverage as well. During such discussions, I talked with developers from several companies that have been collaboration and/or using Kernel CI. I also talked with Nikolai from Red Hat, who gave a presentation about Kernel CI, which points that one of the areas to be improved there is documentation.
So, it seems it is worth having some discussions about how to improve Kernel test documentation.
While kernel_doc does a pretty decent job documenting functions and data structures, for tests, the most important things to be documented are:
a. what the tests do; b. what functionalities they are testing.
This is a lot more important than documenting functions - and the used data structures on tests are typically the ones that are part of the driver's kAPI or uAPI, so they should be documented somewhere else.
Usually, (b) is not so simple, as, at least for complex hardware, the tested features are grouped on an hierarchical way, like:
1. hardware 1.1 DMA engine 1.2 output ports ... 2. firmware 2.1 firmware load 2.2 firmware DMA actions ... 3. kernel features 3.1 memory allocation 3.2 mmap 3.3 bind/unbind ...
CI engines running the test sets usually want to produce a report that will be providing pass rates for the tested features and functionalites that are available at the driver's and their respective hardware and firmware.
I've doing some work at the tool we use to test DRM code [1] in order to have a decent documentation of the tests we have hosted there, focusing mostly on tests for i915 and Xe Intel drivers, also covering documentation for DRM core tests - while providing support for other vendors to also improve their test documentation for IGT - IGT GPU tools and tests.
The documentation tool I developed is generic enough to be used for other test sets and I believe it could be useful as well to document Kselftest and KUnit.
The core of the tool (at test_list.py) is a Python class, with some callers (igt_doc.py, xls_to_doc.py, doc_to_xls.py), being extensible enough to also have other callers to integrate with external tools. We are developing internally one to integrate with our internal Grafana reports to report the pass rate per documented feature, in an hierarchical way.
Something similar to:
1. hardware pass rate: 98% (98 tests passed of 100) 1.1 DMA engine pass rate: 80% (8 tests passed of 10) 1.2 output ports pass rate: 100% (10 tests passed of 10) ...
It is based on the concept that test documentation should be placed as close as possible to the actual code implementing the test sets. It was also be developed in a way that the documentation grouping is flexible. The code was written from the scratch in Python and was implemented inside a class that can also be re-used to do do other nice things, like importing/exporting test documentation to spreadsheets and integration with other tools (like Grafana).
The actual documentation tags look like this:
/** * TEST: Check if new IGT test documentation logic functionality is working * Category: Software build block * Sub-category: documentation * Functionality: test documentation * Issue: none * Description: Complete description of this test * * SUBTEST: foo * Description: do foo things * description continuing on another line * * SUBTEST: bar * Description: do bar things * description continuing on another line * Functionality: bar test doc */
And it has support for wildcards.
There, "TEST" is associated to the contents of the file, while "SUBTEST" refers to each specific subtest inside it. The valid fields are imported from JSON config files, and can be placed into an hierarchical way, in order to produce an hierarchical documentation. Fields defined at the "TEST" level are imported on "SUBTEST", but can be overriden.
The JSON config file looks like this: https://gitlab.freedesktop.org/drm/igt-gpu-tools/-/blob/158feaa20fa2b9424ee5...
The output is in ReST, which can be generated in hierarchical or per-file way. The hierarchical output looks like this:
$ ./scripts/igt_doc.py --config tests/xe/xe_test_config.json --file fubar_tests.c
=============================== Implemented Tests for Xe Driver =============================== Category: Software build block ============================== Sub-category: documentation --------------------------- Functionality: bar test doc ^^^^^^^^^^^^^^^^^^^^^^^^^^^ ``igt@fubar_tests@bar`` :Description: do bar things description continuing on another line :Issue: none Functionality: test documentation ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ``igt@fubar_tests@foo`` :Description: do foo things description continuing on another line :Issue: none
(if --file is not used, it will use all C files specified at the configuration)
The tool already skips tags like the ones used by kernel-doc[1], so one could have both function documentation and per-test documentation on the same file, if needed.
While such tool was conceived to be part of IGT, it doesn't have anything specific for it [2], and I do believe it would be a great contribution to the Kernel to have such tool upstreamed, and integrated as a Sphinx extension.
If we decide to go ahead adding it, I can work on a patchset to apply it to the Kernel, modifying the scripts to better fit at the Kernel needs and start with some documentation examples for i915, DRM core and upcoming Xe KUnit tests.
Comments?
Regards, Mauro
[1] It should be trivial to patch kernel-doc for it to skip TEST and SUBTEST tags if we decide to integrate it to the kernel.
[2] except that tests there are named after IGT, as <igt <test>@<subtest>@<dynamic_subtest>, but a change to a Kernel-specific namespace would be trivial