On 12 September 2016 at 11:37, Neil Williams neil.williams@linaro.org wrote:
On 12 September 2016 at 10:32, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On 12 September 2016 at 08:55, Neil Williams neil.williams@linaro.org wrote:
On 9 September 2016 at 14:09, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
Hi,
I'm trying to get the proper relationship between requested tests and results in LAVA v2. Here is example job: https://validation.linaro.org/scheduler/job/1109234 and results for this job: https://validation.linaro.org/results/1109234
I'll add notes to the docs for the 2016.11 release based on these responses and any feedback on this list.
How can I tell:
- which result matches which test?
There is a chevron in the test case detail page, directly after the test case name, which links to the point in the log where that test was reported. The same URL can also be determined in advance by knowing the job ID, the sequence of test definitions in the test job definition and the name of the test case.
The chevron seems to always point to #bottom of log file.
That's the double chevron >> on the same line as the Job ID.
Below that, there is the test case name and a single chevron.
https://validation.linaro.org/results/1109234/1_lamp-test/mysql-show-databas...
mysql-show-databases >
Suggestions on making this clearer are welcome...
OK, I was looking at the wrong chevron :) This is OK once one knows how to use it.
The URL can also be assembled from the data available in the results, allowing parsers to go directly to that position in the log file.
Note: Unlike V1, the test shell does not wait until the test case entry has been created before moving on, so there can be an offset between the point linked from the result (where the test case entry was created) to the point slightly earlier in the log where the test itself was executed. This wait behaviour caused various bugs as it needs to block at the shell read command which gets confused by other messages on the serial console. The offset is the consequence of removing this behaviour.
So:
https://validation.linaro.org/results/1109234/1_lamp-test/mysql-show-databas... links to https://validation.linaro.org/scheduler/job/1109234#results_1_lamp-test_mysq...
i.e. once you know the URL of the result, you can generate the URL of the point in the test job log where that result was created.
In the log file this section looks like:
Received signal: <TESTCASE> TEST_CASE_ID=mysql-show-databases RESULT=pass case: mysql-show-databases definition: 1_lamp-test result: pass
So, in this case, there was no offset.
There is a REST API using the name of the test definition and the name of the test case.
My question was about API. Manually it's possible to do the matching even in v1.
I'm not sure what else you want from a REST API other than having all of the data available to build the URL immediately after completion, without needing to do round-trip lookups to find hashes or other generated strings. A single call to the results for a completed testjob provides all the information you need to build URLs for all test cases including the links to the position within the log file for each test case. There is no "matching" required in V2 and no round-trips back to the server with more API calls. One call gets all the data but the job of a REST API is not to build those URLs for you, it's to provide enough information to predict those URLs in a single call. Are you looking for an API call which returns all the URLs pre-assembled?
I don't need URLs at all. All I need is to know which test results come from which 'tests' in job definition and if there is anything missing. The important part is to know that some tests screws something up and produces no results. What will I have in the 'LAVA Results' then? Will the metadata present such test as 'omitted'? It's also important to know which results come from which parametrized tests (when more than 1 parameter is present).
The name of the test definition comes from the test job definition:
- repository: http://git.linaro.org/lava-team/lava-functional-tests.git from: git path: lava-test-shell/single-node/singlenode03.yaml name: singlenode-advanced
The digit comes from the sequence of definitions in the list in the - test: action of the test job definition. So job 154736 on staging has three definitions to the test action, 0_env-dut-inline, 1_smoke_tests and 2_singlenode_advanced.
OK. So when I download the jobdefinition and test results I should get the match by order of appearance. Is 'lava' always present in the results?
Yes.
The test case name comes directly from the call to lava-test-case.
When an inline test definition does not report any test cases (by not calling lava-test-case anywhere, just doing setup or diagnostic calls to put data into the logs) then the metadata shows that test definition as "omitted" and it has no entry in the results table.
omitted.0.inline.name: env-dut-inline
lava-test-case calls are not that interesing yet as for example the test can return different number of results based on parameters passed.
However, lava-test-case can also be used to report results for things which are "hidden" within the scripts in the remote git repo. It is also the test-case which provides the link into the position in the job log file.
This approach ties tests to LAVA which I don't like as users requested to have ability to run tests 'standalone'. So anything that takes the test in the direction of being 'LAVA specific' can't be used.
In addition, each test job gets a set of LAVA results containing useful information like the commit hash of the test definition when it was cloned for this test job.
- if there are multiple occurrences of the same test with different
parameters, how to recognize the results?
Multiple occurrences show up in the results table: https://staging.validation.linaro.org/results/154736/2_singlenode-advanced (realpath_check occurs twice with separate results)
The question was about multiple occurences of the same test definition.
Will occur as discrete entries in the results - prefixed with the order.
1_smoke_tests 2_smoke_tests etc.
For example we use subsets of LTP. So I would like to test:
- LTP - syscalls
- LTP - math
As I wrote above the test cases will be different, so they're not that interesting.
That is where test-set is useful. I'll be writing up more documentation on that today.
lava-test-set start syscalls lava-test-case syscalls ... lava-test-set stop syscalls lava-test-case start math lava-test-case math ... lava-test-set stop math
This adds a set around those test cases by adding the test set to the URL.
/results/JOB_ID/2_smoke-tests/syscalls/syscall_one_test
This approach ties the test to LAVA which is a 'no go' from my point of view. Beside that there are other params which are important to know (see CTS: https://git.linaro.org/qa/test-definitions.git/blob/HEAD:/android/cts-host.y... or hackbench: https://git.linaro.org/qa/test-definitions.git/blob/HEAD:/ubuntu/hackbench.y...).
[cut]
Example of such jobs: https://validation.linaro.org/results/1107487 (not the best as the names are different) https://validation.linaro.org/scheduler/job/1113188/definition (job failed, so no results, but I'm trying to get this working)
That needs to be declared to LAVA via the test suite name or a test-set or via the test case names. LAVA cannot introspect into your remote git repo any more easily than you can.
Hmm, this approach implies there is only 1 parameter. How do I know if there are more than 1?
So if the default isn't clear, add a lava-test-case which tests that the default is what you expect - smoke-test-default-true: fail.
This looks like regression from v1 which reported all params in result bundles (default and set in job definition).
milosz