Luis,
I have the exact case that fails for restricted job. Here it is: ./lqa -c ./examples/lqa.yaml analyse 415150 Generating lqa report for job(s): 415150
Report for job(s) (Thu Jul 02 16:16:09 2015): 415150 1 test job(s) ran: 1 complete (0 fully successful, 1 wih failures), 0 incomplete
* --- Failed Jobs --- *
(F) Jobs with failed tests:
415150: https://ci.linaro.org/jenkins/job/linux-linaro-stable-lsk-v3.18/hwpack=beagl... ==================================================================================================================== 116 passed, 4 failed, 0 skipped, 0 unknown FAILED | lava:test_kernel_exception_1 FAILED | lava:test_kernel_exception_1 FAILED | lava:lava_test_shell FAILED | lava:test_kernel_exception_1 Job: https://validation.linaro.org/scheduler/job/415150 Bundle: https://validation.linaro.org/dashboard/permalink/bundle/d54379110280ae1d33c...
Hope that helps (one instance of mmtest is missing - compare results with definition). As I wrote before, I have the code that fixes the issue if you're interested. For simple cases your code works well (checked earlier).
milosz
On 1 July 2015 at 10:10, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
Luis,
On 1 July 2015 at 08:21, Luis Araujo luis.araujo@collabora.co.uk wrote:
On 06/30/2015 09:39 PM, Milosz Wasilewski wrote:
On 30 June 2015 at 10:05, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On 25 June 2015 at 16:32, Luis Araujo luis.araujo@collabora.co.uk wrote:
I checked and it doesn't work (doesn't detect missing results). Here is example job: https://validation.linaro.org/scheduler/job/382325 (I'm not sure it's publicly available)
I cannot access it (even logged in with my launchpad account).
Can you send me a link publicly available with this same problem?, I really would like to check this out.
I'll try to reproduce it with some other device.
Luis,
Not exactly the same case, but I'm able to show that the missing results are not detected. Here is the job: https://validation.linaro.org/scheduler/job/410147
Here is my output: Generating lqa report for job(s): 410147
Report for job(s) (Tue Jun 30 14:33:11 2015): 410147 1 test job(s) ran: 1 complete (0 fully successful, 1 wih failures), 0 incomplete
--- Failed Jobs --- *
(F) Jobs with failed tests:
410147:
https://ci.linaro.org/jenkins/job/linux-linaro-stable-lsk-v3.18/hwpack=beagl...
==================================================================================================================== 319 passed, 17 failed, 27 skipped, 0 unknown FAILED | rcutorture:rcutorture-start FAILED | rcutorture:lava-test-shell-run FAILED | pwrmgmt:cputopology_01 FAILED | usb-test-basic:list-all-usb-devices FAILED | usb-test-basic:examine-all-usb-devices FAILED | usb-test-basic:print-supported-protocols FAILED | usb-test-basic:print-supported-speeds FAILED | lava:test_kernel_exception_1 FAILED | lava:test_kernel_exception_1 FAILED | lava:test_kernel_exception_1 FAILED | lava:test_kernel_exception_1 FAILED | lava:test_kernel_exception_1 FAILED | lava:test_kernel_exception_1 FAILED | lava:test_kernel_exception_1 FAILED | lava:test_kernel_exception_1 FAILED | lava:test_kernel_exception_1 FAILED | lava:test_kernel_exception_1 Job: https://validation.linaro.org/scheduler/job/410147 Bundle: https://validation.linaro.org/dashboard/permalink/bundle/f022ec4a6ce822e154f...
In this case the LTP entries are there in the bundle. However one is missing the parameters. Anyway, I'll try to reproduce the original problem in some other test (maybe an artificial one, so you). In case of the original problem the entry for one LTP shell is missing from the bundle.
milosz
Right, the method won't report missing results for this job since all the testdef's specified in the yaml are available in the bundle:
https://validation.linaro.org/scheduler/job/410147/definition https://validation.linaro.org/dashboard/streams/anonymous/mwasilew/bundles/f...
The _find_missing_tests method is actually used only to check for those testdef from the lava_test_shell command that for some reason were not properly reported in the bundle stream. Now, if these tests have missing results, certainly, the method won't check for those, but this could be a later improvement for sure.
As I wrote this isn't exactly the same issue as I reported previously. If you take one of LTP result entries out from the bundle you will get what I wrote about before. So the situation is you ask for 2 LTP test shells with different parameters and result bundle only contains one result for LTP. The other one is entirely missing.
If you want to do the missing detection a bit better, take a look here: https://git.linaro.org/qa/qa-reports.git/blob/refs/heads/refactoring:/utils/... Eventually it should be fixed on LAVA side, but for the moment this is what we get.
milosz