Thank you Anibal for the fast response
On 22.08.18 19:50, Anibal Limon wrote:
On Wed, 22 Aug 2018 at 11:39, Oleksandr Terentiev <oterenti@cisco.com mailto:oterenti@cisco.com> wrote:
Hi, I launched util-linux ptest using automated/linux/ptest/ptest.yaml from https://git.linaro.org/qa/test-definitions.git and received the following results: https://pastebin.com/nj9PYQzE As you can see some tests failed. However, case util-linux marked as passed. It looks like ptest.py only analyze return code of ptest-runner -d <ptest_dir> <ptest_name> command. And since ptest-runner finishes correctly exit code is 0. Therefore all tests are always marked as passed, and users never know when some of the tests fail. Maybe it worth to analyze each test?Talking about each ptest the result comes from the ptest script in the OE recipe [1], for convention if the OE ptest returns 0 means pass, so needs to be fixed in the OE ptest [2].
I’ve read https://wiki.yoctoproject.org/wiki/Ptest carefully a few times more. There are prescriptions about output format. But I didn’t find any mention about return code processing or a reference to the convention you mentioned in the answer.
I looked through some OE run-ptest scripts. I suspect they don’t verify if some of their tests failed, and exit with 0 even if all their tests failed.
http://git.openembedded.org/openembedded-core/tree/meta/recipes-core/util-li... http://git.openembedded.org/openembedded-core/tree/meta/recipes-support/attr... http://git.openembedded.org/openembedded-core/tree/meta/recipes-support/attr... http://git.openembedded.org/openembedded-core/tree/meta/recipes-core/dbus/db... http://git.openembedded.org/openembedded-core/tree/meta/recipes-devtools/e2f... http://git.openembedded.org/openembedded-core/tree/meta/recipes-extended/gaw...
Regarding the LAVA ptest.py script, I mark the run as succeed if there is no critical error in the ptest-runner and we have a QA-reports tool to analyse pass/fails in detail for every ptest executed [3].
I heard about QA-reports tool but I’ve never used it before, so maybe I missed something. From https://qa-reports.linaro.org/qcomlt/openembedded-rpb-sumo/build/37/testrun/... I see all ptests passed. Still, in log https://qa-reports.linaro.org/qcomlt/openembedded-rpb-sumo/build/37/testrun/... I found 54 failed tests and wasn’t able to find a report which indicates those failures.
Is there such a report? It would be really useful to know that some tests failed.
Thanks
[1] http://git.openembedded.org/openembedded-core/tree/meta/recipes-core/util-li... [2] https://wiki.yoctoproject.org/wiki/Ptest [3] https://qa-reports.linaro.org/qcomlt/openembedded-rpb-sumo/build/37/testrun/...
Regards, Anibal
Best regards, Alex