On 9 September 2016 at 14:09, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
Hi,
I'm trying to get the proper relationship between requested tests and results in LAVA v2. Here is example job: https://validation.linaro.org/scheduler/job/1109234 and results for this job: https://validation.linaro.org/results/1109234
I'll add notes to the docs for the 2016.11 release based on these responses and any feedback on this list.
How can I tell:
- which result matches which test?
There is a chevron in the test case detail page, directly after the test case name, which links to the point in the log where that test was reported. The same URL can also be determined in advance by knowing the job ID, the sequence of test definitions in the test job definition and the name of the test case.
Note: Unlike V1, the test shell does not wait until the test case entry has been created before moving on, so there can be an offset between the point linked from the result (where the test case entry was created) to the point slightly earlier in the log where the test itself was executed. This wait behaviour caused various bugs as it needs to block at the shell read command which gets confused by other messages on the serial console. The offset is the consequence of removing this behaviour.
So:
https://validation.linaro.org/results/1109234/1_lamp-test/mysql-show-databas... links to https://validation.linaro.org/scheduler/job/1109234#results_1_lamp-test_mysq...
i.e. once you know the URL of the result, you can generate the URL of the point in the test job log where that result was created.
In the log file this section looks like:
Received signal: <TESTCASE> TEST_CASE_ID=mysql-show-databases RESULT=pass case: mysql-show-databases definition: 1_lamp-test result: pass
So, in this case, there was no offset.
There is a REST API using the name of the test definition and the name of the test case.
The name of the test definition comes from the test job definition:
- repository: http://git.linaro.org/lava-team/lava-functional-tests.git from: git path: lava-test-shell/single-node/singlenode03.yaml name: singlenode-advanced
The digit comes from the sequence of definitions in the list in the - test: action of the test job definition. So job 154736 on staging has three definitions to the test action, 0_env-dut-inline, 1_smoke_tests and 2_singlenode_advanced.
The test case name comes directly from the call to lava-test-case.
When an inline test definition does not report any test cases (by not calling lava-test-case anywhere, just doing setup or diagnostic calls to put data into the logs) then the metadata shows that test definition as "omitted" and it has no entry in the results table.
omitted.0.inline.name: env-dut-inline
In addition, each test job gets a set of LAVA results containing useful information like the commit hash of the test definition when it was cloned for this test job.
- if there are multiple occurrences of the same test with different
parameters, how to recognize the results?
Multiple occurrences show up in the results table: https://staging.validation.linaro.org/results/154736/2_singlenode-advanced (realpath_check occurs twice with separate results)
In this case, each occurred within the same test definition, so there is one page showing both results: https://staging.validation.linaro.org/results/154736/2_singlenode-advanced/r... (as the test case name is the same, there can only be one URL).
Each one links to a different point in the job log.
In LAVA v1 the matching was a very arduous process.
That should be much simpler now. All the information is available to the test writer once the testjob ID is known.
https://validation.linaro.org/results/1109234
There is a CSV and YAML download link for the complete set of results for the job. (Not just a list of bundles as it was with V1.) The export includes details of the test definition names.
https://validation.linaro.org/results/1109234/csv https://validation.linaro.org/results/1109234/yaml
From the test job submission, there are two entries in the test:
definitions: list. The names in the test job submission are smoke-test and lamp-test. This ordering is retained consistently, so the results for smoke-test are 0_smoke-test:
https://validation.linaro.org/results/1109234/0_smoke-test
Now the CSV and YAML links are restricted to just the smoke-test definition.
https://validation.linaro.org/results/1109234/0_smoke-test/csv https://validation.linaro.org/results/1109234/0_smoke-test/yaml
In addition, private test jobs create results which are only visible for certain users. The REST API supports this using ?name=user.name&token=tokenstring in the URL at all export levels.
e.g. ....validation.linaro.org/results/1109234/0_smoke-test/yaml?name=user.name&token=tokenstring
One had to download the job definition, look for lava-test-shell actions, pull the test definition yaml sources and match yaml ID and to ID found in result bundle. How does this work in v2?
See also the metadata section:
https://staging.validation.linaro.org/results/154736
test.2.namesinglenode-advanced
There are CSV and YAML download links for the results and YAML download links for the metadata.
BTW: one limitation with the metadata representation is that it is not possible from the git clone URL, path and filename to consistently build a full URL to the file itself without specialist knowledge of exactly which git WWW frontend is in use for which git clone URL. So LAVA can turn git://git.linaro.org/qa/test-definitions.git into http://git.linaro.org/qa/test-definitions.git but LAVA cannot reliably combine that with ubuntu/smoke-tests-basic.yaml to get a path like https://git.linaro.org/qa/test-definitions.git/blob/HEAD:/ubuntu/smoke-tests... because the "blob" "HEAD:" elements are not knowable from the URL or the git clone operation. git.linaro.org has one frontend, github.com has another and git.debian.org has yet another.