kselftest running on LAVA infrastures provides test results parser from test-definitions repository. which is getting "Bad test result:".
I have noticed this on kernelci [1] and LKFT LAVA instances [2]. We need to investigate and change parse_output [3] inside test-definitions.
Report-by: Linux Kernel Functional Testing lkft@linaro.org
Test results parser showing “Bad test results: “,
Received signal: <TESTCASE> TEST_CASE_ID=arm64.sve-ptrace.SVE RESULT=FPSIMD Bad test result: FPSIMD <LAVA_SIGNAL_TESTCASE TEST_CASE_ID=arm64.sve-ptrace.SVE RESULT=FPSIMD> Received signal: <TESTCASE> TEST_CASE_ID=arm64.sve-ptrace.SVE RESULT=get_fpsimd() Bad test result: get_fpsimd() <LAVA_SIGNAL_TESTCASE TEST_CASE_ID=arm64.sve-ptrace.SVE RESULT=get_fpsimd()> Received signal: <TESTCASE> TEST_CASE_ID=arm64.sve-ptrace.SVE RESULT=SVE_PT_VL_INHERIT Bad test result: SVE_PT_VL_INHERIT <LAVA_SIGNAL_TESTCASE TEST_CASE_ID=arm64.sve-ptrace.SVE RESULT=SVE_PT_VL_INHERIT> Received signal: <TESTCASE> TEST_CASE_ID=arm64.sve-ptrace.SVE RESULT=SVE_PT_VL_INHERIT Bad test result: SVE_PT_VL_INHERIT <LAVA_SIGNAL_TESTCASE TEST_CASE_ID=arm64.sve-ptrace.SVE RESULT=SVE_PT_VL_INHERIT> Received signal: <TESTCASE> TEST_CASE_ID=arm64.sve-ptrace.Set RESULT=SVE Bad test result: SVE <LAVA_SIGNAL_TESTCASE TEST_CASE_ID=arm64.sve-ptrace.Set RESULT=SVE> <LAVA_SIGNAL_TESTCASE TEST_CASE_ID=arm64.sve-ptrace.Set RESULT=and> Received signal: <TESTCASE> TEST_CASE_ID=arm64.sve-ptrace.Set RESULT=and Bad test result: and Received signal: <TESTCASE> TEST_CASE_ID=arm64.sve-ptrace.Set RESULT=and Bad test result: and .. <LAVA_SIGNAL_TESTCASE TEST_CASE_ID=clone3.clone3_set_tid.[1710] RESULT=Result> Received signal: <TESTCASE> TEST_CASE_ID=clone3.clone3_set_tid.[1710] RESULT=Result Bad test result: Result
[1] https://storage.kernelci.org/next/master/next-20221116/arm64/defconfig+arm64... [2] https://linaro.atlassian.net/browse/LKQ-934 [3] https://github.com/Linaro/test-definitions/blob/master/automated/linux/kself... [4] https://lkft.validation.linaro.org/scheduler/job/5729151#L2511
-- Linaro LKFT https://lkft.linaro.org
On Wed, Nov 16, 2022 at 05:46:33PM +0530, Naresh Kamboju wrote:
kselftest running on LAVA infrastures provides test results parser from test-definitions repository. which is getting "Bad test result:".
I have noticed this on kernelci [1] and LKFT LAVA instances [2]. We need to investigate and change parse_output [3] inside test-definitions.
Report-by: Linux Kernel Functional Testing lkft@linaro.org
Test results parser showing “Bad test results: “,
When reporting an issue can you please try to provide some analysis which goes beyond the level of "I saw an error message" - for example here it's hard to tell if you think you're seeing an issue somewhere in your test automation system or if you're trying to report something in the tests.
On Wed, 16 Nov 2022 at 18:57, Mark Brown broonie@kernel.org wrote:
On Wed, Nov 16, 2022 at 05:46:33PM +0530, Naresh Kamboju wrote:
kselftest running on LAVA infrastures provides test results parser from test-definitions repository. which is getting "Bad test result:".
I have noticed this on kernelci [1] and LKFT LAVA instances [2]. We need to investigate and change parse_output [3] inside test-definitions.
Report-by: Linux Kernel Functional Testing lkft@linaro.org
Test results parser showing “Bad test results: “,
When reporting an issue can you please try to provide some analysis which goes beyond the level of "I saw an error message"
- for example here it's hard to tell if you think you're seeing
an issue somewhere in your test automation system or if you're trying to report something in the tests.
Let me add more information about this,
Kees Cook, has done the work of a kselftest results parser in perl which is in the test-definitions repository. which was working well for two years now. please refer to the below commit log and link to the kselftest test-definitions [3].
The new test cases output is not coping up with the old results parser and both KernelCI [1] and LKFT [2] using LAVA have noticed. Kselftest results parser problem [4].
--- Repo: test-definitions commit id: 8bd338bbcfa5a03efcf1d12e25b5d341d5a29cbc --- kselftest: Parse kTAP for test results
This stops mangling test output for humans and performs a best-effort kTAP parsing for the kselftest output (including recursive kTAP output).
Signed-off-by: Kees Cook keescook@chromium.org
[1] https://storage.kernelci.org/next/master/next-20221116/arm64/defconfig+arm64... [2] https://linaro.atlassian.net/browse/LKQ-934 [3] https://github.com/Linaro/test-definitions/blob/master/automated/linux/kself... [4] https://lkft.validation.linaro.org/scheduler/job/5729151#L2511
-- Linaro LKFT https://lkft.linaro.org
On Wed, Nov 16, 2022 at 07:36:10PM +0530, Naresh Kamboju wrote:
On Wed, 16 Nov 2022 at 18:57, Mark Brown broonie@kernel.org wrote:
On Wed, Nov 16, 2022 at 05:46:33PM +0530, Naresh Kamboju wrote:
Test results parser showing “Bad test results: “,
When reporting an issue can you please try to provide some analysis which goes beyond the level of "I saw an error message"
- for example here it's hard to tell if you think you're seeing
an issue somewhere in your test automation system or if you're trying to report something in the tests.
Let me add more information about this,
Kees Cook, has done the work of a kselftest results parser in perl which is in the test-definitions repository. which was working well for two years now. please refer to the below commit log and link to the kselftest test-definitions [3].
The new test cases output is not coping up with the old results parser and both KernelCI [1] and LKFT [2] using LAVA have noticed. Kselftest results parser problem [4].
I'm still not clear if you believe there is an issue in the tests or in your test infrastructure here. I can't identify any problem with the test output, everything appears to be within the KTAP spec:
https://www.kernel.org/doc/html/latest/dev-tools/ktap.html#test-case-result-...
My best guess is that either this script which the infrastructure is using or something else that uses the results of the script is broken when the test description includes spaces. The KTAP spec says that each test case result is a line in the form:
<result> <number> [<description>][ # [<directive>] [<diagnostic data>]]
where
The description is a description of the test, generally the name of the test, and can be any string of words (can’t include #). The description is optional, but recommended.
ie, anything after the space following the test number up to a # or the end of the line is a valid test description (it's not the intent but it does appear that the # needs to have a space before it). You could argue that non-alphabetic characters are out of spec for the description since any separators between words aren't themselves words but realistically that'd invalidate a fairly large subset of the selftests which probably isn't constructive.
-
Mark Brown -
Naresh Kamboju