Luis,
I got the patch but I can't test it now as we have LAVA maintenance/migration happening this week. I'll get back to it when migration is done.
milosz
On 3 July 2015 at 11:56, Luis Araujo luis.araujo@collabora.co.uk wrote:
Hello Milosz,
Thanks for going checking this issue that far and pointing me to the cause of it.
I glanced over your code and this certainly will involve quite few operations to properly get reported the exact missing test defs from LAVA. I also noticed that one of the problem for lqa not reporting the 'mmtests' is this particular case was about not properly handling duplicated tests names from the bundles.
I put a simple solution that should handle the below case better, with the patch from my branch at:
https://git.collabora.com/cgit/user/araujo/lqa.git/log/?h=WIP-fix-find-missi...
Even though we don't detect the same exact missing tests, we are mainly interested at the beginning to have an idea of what went wrong with any of the tests.
Let me know if you still don't get missing results with that patch, or maybe if you find other case giving such a failure.
Cheers,
Luis
On 07/02/2015 11:18 PM, Milosz Wasilewski wrote:
Luis,
I have the exact case that fails for restricted job. Here it is: ./lqa -c ./examples/lqa.yaml analyse 415150 Generating lqa report for job(s): 415150
Report for job(s) (Thu Jul 02 16:16:09 2015): 415150 1 test job(s) ran: 1 complete (0 fully successful, 1 wih failures), 0 incomplete
--- Failed Jobs --- *
(F) Jobs with failed tests:
415150:
https://ci.linaro.org/jenkins/job/linux-linaro-stable-lsk-v3.18/hwpack=beagl...
==================================================================================================================== 116 passed, 4 failed, 0 skipped, 0 unknown FAILED | lava:test_kernel_exception_1 FAILED | lava:test_kernel_exception_1 FAILED | lava:lava_test_shell FAILED | lava:test_kernel_exception_1 Job: https://validation.linaro.org/scheduler/job/415150 Bundle: https://validation.linaro.org/dashboard/permalink/bundle/d54379110280ae1d33c...
Hope that helps (one instance of mmtest is missing - compare results with definition). As I wrote before, I have the code that fixes the issue if you're interested. For simple cases your code works well (checked earlier).
milosz
On 1 July 2015 at 10:10, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
Luis,
On 1 July 2015 at 08:21, Luis Araujo luis.araujo@collabora.co.uk wrote:
On 06/30/2015 09:39 PM, Milosz Wasilewski wrote:
On 30 June 2015 at 10:05, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On 25 June 2015 at 16:32, Luis Araujo luis.araujo@collabora.co.uk wrote: >> >> I checked and it doesn't work (doesn't detect missing results). Here >> is example job: >> https://validation.linaro.org/scheduler/job/382325 (I'm not sure >> it's >> publicly available) > > > I cannot access it (even logged in with my launchpad account). > > Can you send me a link publicly available with this same problem?, I > really > would like to check this out. > I'll try to reproduce it with some other device.
Luis,
Not exactly the same case, but I'm able to show that the missing results are not detected. Here is the job: https://validation.linaro.org/scheduler/job/410147
Here is my output: Generating lqa report for job(s): 410147
Report for job(s) (Tue Jun 30 14:33:11 2015): 410147 1 test job(s) ran: 1 complete (0 fully successful, 1 wih failures), 0 incomplete
--- Failed Jobs --- *
(F) Jobs with failed tests:
410147:https://ci.linaro.org/jenkins/job/linux-linaro-stable-lsk-v3.18/hwpack=beagl...
==================================================================================================================== 319 passed, 17 failed, 27 skipped, 0 unknown FAILED | rcutorture:rcutorture-start FAILED | rcutorture:lava-test-shell-run FAILED | pwrmgmt:cputopology_01 FAILED | usb-test-basic:list-all-usb-devices FAILED | usb-test-basic:examine-all-usb-devices FAILED | usb-test-basic:print-supported-protocols FAILED | usb-test-basic:print-supported-speeds FAILED | lava:test_kernel_exception_1 FAILED | lava:test_kernel_exception_1 FAILED | lava:test_kernel_exception_1 FAILED | lava:test_kernel_exception_1 FAILED | lava:test_kernel_exception_1 FAILED | lava:test_kernel_exception_1 FAILED | lava:test_kernel_exception_1 FAILED | lava:test_kernel_exception_1 FAILED | lava:test_kernel_exception_1 FAILED | lava:test_kernel_exception_1 Job: https://validation.linaro.org/scheduler/job/410147 Bundle:
https://validation.linaro.org/dashboard/permalink/bundle/f022ec4a6ce822e154f...
In this case the LTP entries are there in the bundle. However one is missing the parameters. Anyway, I'll try to reproduce the original problem in some other test (maybe an artificial one, so you). In case of the original problem the entry for one LTP shell is missing from the bundle.
milosz
Right, the method won't report missing results for this job since all the testdef's specified in the yaml are available in the bundle:
https://validation.linaro.org/scheduler/job/410147/definition
https://validation.linaro.org/dashboard/streams/anonymous/mwasilew/bundles/f...
The _find_missing_tests method is actually used only to check for those testdef from the lava_test_shell command that for some reason were not properly reported in the bundle stream. Now, if these tests have missing results, certainly, the method won't check for those, but this could be a later improvement for sure.
As I wrote this isn't exactly the same issue as I reported previously. If you take one of LTP result entries out from the bundle you will get what I wrote about before. So the situation is you ask for 2 LTP test shells with different parameters and result bundle only contains one result for LTP. The other one is entirely missing.
If you want to do the missing detection a bit better, take a look here:
https://git.linaro.org/qa/qa-reports.git/blob/refs/heads/refactoring:/utils/... Eventually it should be fixed on LAVA side, but for the moment this is what we get.
milosz