On 15/12/2023 13:54, Mark Brown wrote:
On Thu, Dec 14, 2023 at 04:24:34PM +0000, Ryan Roberts wrote:
When running tests on a CI system (e.g. LAVA) it is useful to output test results in TAP format so that the CI can parse the fine-grained results to show regressions. Many of the mm selftest binaries already output using the TAP format. And the kselftests runner (run_kselftest.sh) also uses the format. CI systems such as LAVA can already handle nested TAP reports. However, with the mm selftests we have 3 levels of nesting (run_kselftest.sh -> run_vmtests.sh -> individual test binaries) and the middle level did not previously support TAP, which breaks the parser.
Reviewed-by: Mark Brown broonie@kernel.org
Thanks!
Let's fix that by teaching run_vmtests.sh to output using the TAP format. Ideally this would be opt-in via a command line argument to avoid the possibility of breaking anyone's existing scripts that might scrape the output. However, it is not possible to pass arguments to tests invoked via run_kselftest.sh. So I've implemented an opt-out option (-n), which will revert to the existing output format.
What I did for ftrace which had a similar situation was make a wrapper script which invokes the test runner, make the test runner a TEST_PROGS_EXTENDED so it's not run by the kselftest infrastructure automatically and make the wrapper a normal TEST_PROGS. Neither option is especially lovely.
Yeah that's a good idea... I'll wait and see if anyone shouts that this has broken something. If nothing is broken, I think it is better to just make TAP the default rather than adding yet another wrapper.