-----Original Message----- From: Kees Cook
On Tue, Jun 16, 2020 at 09:16:01PM +0000, Bird, Tim wrote:
So far, most of the CI systems don't parse out diagnostic data, so it doesn't really matter what the format is. If it's useful for humans, it's valuable as is. However, it would be nice if that could change. But without some formalization of the format of the diagnostic data, it's an intractable problem for CI systems to parse it. So it's really a chicken and egg problem. To solve it, we would have to determine what exactly needs to be provided on a consistent basis for diagnostic data across many tests. I think that it's too big a problem to handle right now. I'm not opposed to migrating to some structure with yaml in the future, but free form text output seems OK for now.
For a CI system, if I see a test has failed, I expect to be able to click a link to get the log of that test, which includes the diagnostic lines. The other reason to have them there is to show progress during a manual run.
Agreed. You only need machine-parsable data if you expect the CI system to do something more with the data than just present it. What that would be, that would be common for all tests (or at least many test), is unclear. Maybe there are patterns in the diagnostic data that could lead to higher-level analysis, or even automated fixes, that don't become apparent if the data is unstructured. But it's hard to know until you have lots of data. I think just getting the other things consistent is a good priority right now. -- Tim
Yeah, I think it would be nice if all test frameworks/libraries for the kernel output tests in the same language.
Agreed.
$ git grep "TAP version" exec/binfmt_script:print("TAP version 1.3") kselftest.h: printf("TAP version 13\n"); kselftest/runner.sh: echo "TAP version 13" resctrl/resctrl_tests.c: printf("TAP version 13\n"); size/get_size.c: print("TAP version 13\n");
Looks like there are 2 tests to convert to kselftest.h, and then we can just change the version to 14 in the header and the runner. ;)
-- Kees Cook