Hi all!
I am working on writing LAVA test definitions and using the callback method to send test results to kernelCI.
I noticed a sort of bug (?) when the test definitions are reporting a lot of elements. The job finishes and the callback gets triggered before the end of the log parsing. I think the callback method is not waiting for the parser to finish before sending the event. The callback output is then missing some test results.
I made a simple test script that reports 300 test elements and returns. I can see in the LAVA log that they are all detected. But in the callback object there is around 80 test results. If I add a sleep (15 seconds) to hold the script before returning, the callback has the 300 test results bundled in the json.
Did anyone experienced this before ?
Thanks !
Hi Loys,
I talked to the team an nobody experienced this yet, then again, they didn't seem to have this amount of results used with callback. It seems that the callback trigger (job complete) is happening before all the results coming from the dispatcher are parsed (the timeout test you did confirms that this is the most likely case).
I filed a CTT request in our portal and will be looking into fixing this in the next release (.12) https://projects.linaro.org/servicedesk/customer/portal/1/CTT-622
Thanks, Stevan
On 11/03/2017 09:29 AM, Loys Ollivier wrote:
Hi all!
I am working on writing LAVA test definitions and using the callback method to send test results to kernelCI.
I noticed a sort of bug (?) when the test definitions are reporting a lot of elements. The job finishes and the callback gets triggered before the end of the log parsing. I think the callback method is not waiting for the parser to finish before sending the event. The callback output is then missing some test results.
I made a simple test script that reports 300 test elements and returns. I can see in the LAVA log that they are all detected. But in the callback object there is around 80 test results. If I add a sleep (15 seconds) to hold the script before returning, the callback has the 300 test results bundled in the json.
Did anyone experienced this before ?
Thanks !
-- Loys OLLIVIER Baylibre
Lava-users mailing list Lava-users@lists.linaro.org https://lists.linaro.org/mailman/listinfo/lava-users
On 3 November 2017 at 08:29, Loys Ollivier lollivier@baylibre.com wrote:
Hi all!
I am working on writing LAVA test definitions and using the callback method to send test results to kernelCI.
I noticed a sort of bug (?) when the test definitions are reporting a lot of elements. The job finishes and the callback gets triggered before the end of the log parsing. I think the callback method is not waiting for the parser to finish before sending the event. The callback output is then missing some test results.
I made a simple test script that reports 300 test elements and returns. I can see in the LAVA log that they are all detected. But in the callback object there is around 80 test results. If I add a sleep (15 seconds) to hold the script before returning, the callback has the 300 test results bundled in the json.
Did anyone experienced this before ?
This sounds familiar. I had similar issue when reporting a big number of results. However I didn't use the callback. In my case it was notification on ZMQ publisher followed by XML-RPC call. Also some results were missing. Subsequent XML-RPC call a few seconds later returned all results. However I thought the problem was fixed. https://projects.linaro.org/browse/LAVA-1030
milosz
However I thought the problem was fixed. https://projects.linaro.org/browse/LAVA-1030
You are right, this has been fixed for zmq events.
Everyone,
Thanks for your feedback. So this is something you experienced as well :) Maybe the callback was not in the picture at the time of the fix (ZMQ/XML-RPC) ? I wasn't around at that time so I can't tell.
Stevan, Here is my test definition: github https://github.com/loysollivier/agl-test-definitions/blob/master/test-suites/yocto-ptest.yaml and the test script: github https://github.com/loysollivier/agl-test-definitions/blob/master/common/scripts/lava-test.sh. It is very basic but feel free to use it if you want to replicate my procedure.
I'll keep an eye on the CTT request (if I manage to log in jira)!
Thanks
On Fri, Nov 3, 2017 at 11:07 AM, Remi Duraffort remi.duraffort@linaro.org wrote:
However I thought the problem was fixed. https://projects.linaro.org/browse/LAVA-1030
You are right, this has been fixed for zmq events.
-- Rémi Duraffort LAVA Team
Hi Team Can you share an example code or reference to understand this scenario more.
Look forward to hear from you with positive response.
On Fri, Nov 3, 2017 at 8:52 PM, Loys Ollivier lollivier@baylibre.com wrote:
Everyone,
Thanks for your feedback. So this is something you experienced as well :) Maybe the callback was not in the picture at the time of the fix (ZMQ/XML-RPC) ? I wasn't around at that time so I can't tell.
Stevan, Here is my test definition: github https://github.com/loysollivier/agl-test-definitions/blob/master/test-suites/yocto-ptest.yaml and the test script: github https://github.com/loysollivier/agl-test-definitions/blob/master/common/scripts/lava-test.sh. It is very basic but feel free to use it if you want to replicate my procedure.
I'll keep an eye on the CTT request (if I manage to log in jira)!
Thanks
On Fri, Nov 3, 2017 at 11:07 AM, Remi Duraffort <remi.duraffort@linaro.org
wrote:
However I thought the problem was fixed. https://projects.linaro.org/browse/LAVA-1030
You are right, this has been fixed for zmq events.
-- Rémi Duraffort LAVA Team
-- Loys OLLIVIER Embedded Software Engineer | Baylibre
Lava-users mailing list Lava-users@lists.linaro.org https://lists.linaro.org/mailman/listinfo/lava-users
Hello,
unfortunately I do not have the code anymore. But it is fairly easy to reproduce: - Specify a callback instance (URL + token) - Call a test script - Within the test script do something like: for i in {1..1000};do echo "lava-test-case test-$i --result pass"; done - Exit the script - When the LAVA test definition execution finishes: verify that the callback happens - Look in the callback result, test number != 1000
Hope this helps,
Best,
Loys
On Tue, Apr 3, 2018 at 10:59 AM, Chetan Sharma chetan.sharma@timesys.com wrote:
Hi Team Can you share an example code or reference to understand this scenario more.
Look forward to hear from you with positive response.
On Fri, Nov 3, 2017 at 8:52 PM, Loys Ollivier lollivier@baylibre.com wrote:
Everyone,
Thanks for your feedback. So this is something you experienced as well :) Maybe the callback was not in the picture at the time of the fix (ZMQ/XML-RPC) ? I wasn't around at that time so I can't tell.
Stevan, Here is my test definition: github https://github.com/loysollivier/agl-test-definitions/blob/master/test-suites/yocto-ptest.yaml and the test script: github https://github.com/loysollivier/agl-test-definitions/blob/master/common/scripts/lava-test.sh. It is very basic but feel free to use it if you want to replicate my procedure.
I'll keep an eye on the CTT request (if I manage to log in jira)!
Thanks
On Fri, Nov 3, 2017 at 11:07 AM, Remi Duraffort < remi.duraffort@linaro.org> wrote:
However I thought the problem was fixed. https://projects.linaro.org/browse/LAVA-1030
You are right, this has been fixed for zmq events.
-- Rémi Duraffort LAVA Team
-- Loys OLLIVIER Embedded Software Engineer | Baylibre
Lava-users mailing list Lava-users@lists.linaro.org https://lists.linaro.org/mailman/listinfo/lava-users
-- Thanks & Regards Chetan Sharma
On 27 April 2018 at 15:01, Loys Ollivier lollivier@baylibre.com wrote:
Hello,
unfortunately I do not have the code anymore. But it is fairly easy to reproduce:
- Specify a callback instance (URL + token)
- Call a test script
- Within the test script do something like: for i in {1..1000};do echo
"lava-test-case test-$i --result pass"; done
- Exit the script
- When the LAVA test definition execution finishes: verify that the callback
happens
- Look in the callback result, test number != 1000
I think this was fixed some time ago: https://projects.linaro.org/browse/LAVA-1096 https://projects.linaro.org/browse/LAVA-1030
and a few reviews: https://review.linaro.org/#/c/21322/ https://review.linaro.org/#/c/21323/ https://review.linaro.org/#/c/21246/
milosz
-
Chetan Sharma -
Loys Ollivier -
Milosz Wasilewski -
Remi Duraffort -
Stevan Radakovic