Yes, I will submit patch to test definitions. My previous version of plugin patch didn't actually work. I think I have a better version now. Adding tests to verify special cases now.
milosz
On Thu, 13 Jun 2019 at 14:29, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Great !
So are you still going to submit a patch to test-definitions ?
Axel
On Thu, 13 Jun 2019 at 14:41, Milosz Wasilewski < milosz.wasilewski@linaro.org> wrote:
I wrote this patch: https://github.com/Linaro/squadplugins/pull/10
In my local testing it fixes the problem and a few other issues I wansn't aware before.
milosz
On Thu, 13 Jun 2019 at 12:58, Milosz Wasilewski < milosz.wasilewski@linaro.org> wrote:
OK, found the culprit. It's the VTS test name:
arm64-v8a.VtsHalCameraProviderV2_4Target/VtsHalCameraProviderV2_4Target.CameraHidlTest.autoFocus(legacy/0)_64bit VTS decided to use "/" at the end of the test name which SQUAD isn't very happy about (the legacy/0 part). I added a fix for [] some time ago but it looks like VTS also uses () in some cases :( I'm not sure whether we should fix it in SQUAD or in test-definitions. IMHO adding to many 'special cases' to SQUAD isn't a good idea. I'll propose a patch to test-definitions.
As a side note, I found that the plugin code doesn't always recognize ABI properly. I'll fix that in the plugin.
milosz
On Thu, 13 Jun 2019 at 11:51, Milosz Wasilewski < milosz.wasilewski@linaro.org> wrote:
Thank Axel, debugging now. I don't see anything that stands out in any obvious way.
milosz
On Thu, 13 Jun 2019 at 11:21, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Milosz,
I privately shared with you the archive containing what you asked for.
On Thu, 13 Jun 2019 at 12:06, Milosz Wasilewski < milosz.wasilewski@linaro.org> wrote:
Could you share your testjob YAML and test-results.xml from VTS (may be privately). I think I know what's going on. Thanks for reporting this.
milosz
On Thu, 13 Jun 2019 at 10:58, Axel Lebourhis < axel.lebourhis@linaro.org> wrote:
> Hi Milosz, > > Today I faced a new issue, it may be a bug. I ran a full VTS run, > the plugin returned the following error : > > Jun 12 20:25:36 squad.sw.nxp.com celery[6499]: [2019-06-12 20:25:36 > +0000] [INFO] Starting CTS/VTS plugin for test job: 1841 > Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: [2019-06-12 20:25:38 > +0000] [ERROR] Plugin postprocessing error: too many values to unpack > (expected 2) > Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: Traceback (most > recent call last): > Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: File > "/usr/local/lib/python3.5/dist-packages/squad/ci/models.py", line 132, in > __postprocess_testjob__ > Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: > plugin.postprocess_testjob(test_job) > Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: File > "/usr/local/lib/python3.5/dist-packages/tradefed/__init__.py", line 207, in > postprocess_testjob > Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: > self._assign_test_log(results.test_results.contents, failed) > Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: File > "/usr/local/lib/python3.5/dist-packages/tradefed/__init__.py", line 50, in > _assign_test_log > Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: test_suite_abi, > test_suite_name = test_suite_name.split(".") > Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: ValueError: too many > values to unpack (expected 2) > > Do you have any idea ? > > Regards, > Axel > > On Tue, 11 Jun 2019 at 13:27, Axel Lebourhis < > axel.lebourhis@linaro.org> wrote: > >> Great, thanks! >> >> On Tue, 11 Jun 2019 at 12:42, Milosz Wasilewski < >> milosz.wasilewski@linaro.org> wrote: >> >>> I forgot to release it. Will do it today. >>> >>> milosz >>> >>> On Tue, 11 Jun 2019 at 11:03, Axel Lebourhis < >>> axel.lebourhis@linaro.org> wrote: >>> > >>> > Hi Milosz, >>> > >>> > I saw your fix has been merged, is it available now with pip ? >>> > >>> > Regards, >>> > Axel >>> > >>> > On Fri, 7 Jun 2019 at 14:39, Milosz Wasilewski < >>> milosz.wasilewski@linaro.org> wrote: >>> >> >>> >> https://github.com/Linaro/squadplugins/pull/9 >>> >> Antonio, could you take a look? >>> >> >>> >> milosz >>> >> >>> >> On Fri, 7 Jun 2019 at 13:34, Axel Lebourhis < >>> axel.lebourhis@linaro.org> wrote: >>> >> > >>> >> > Hi Milosz, >>> >> > >>> >> > Great thank you. >>> >> > >>> >> > Axel >>> >> > >>> >> > On Fri, 7 Jun 2019 at 14:26, Milosz Wasilewski < >>> milosz.wasilewski@linaro.org> wrote: >>> >> >> >>> >> >> I think it's interactive test section that makes the plugin >>> unhappy. >>> >> >> I'll fix that and upload new version. >>> >> >> >>> >> >> milosz >>> >> >> >>> >> >> On Thu, 6 Jun 2019 at 12:45, Axel Lebourhis < >>> axel.lebourhis@linaro.org> wrote: >>> >> >> > >>> >> >> > Hi Milosz, >>> >> >> > >>> >> >> > Yes sure, you will find it attached. This definition is >>> the one stored by SQUAD, retrieved with api/testjobs/<id>/definition/. >>> >> >> > I hid the sensible information with "****". >>> >> >> > >>> >> >> > Regards, >>> >> >> > Axel >>> >> >> > >>> >> >> > On Thu, 6 Jun 2019 at 12:45, Milosz Wasilewski < >>> milosz.wasilewski@linaro.org> wrote: >>> >> >> >> >>> >> >> >> On Thu, 6 Jun 2019 at 09:38, Axel Lebourhis < >>> axel.lebourhis@linaro.org> wrote: >>> >> >> >> > >>> >> >> >> > Hi all, >>> >> >> >> > >>> >> >> >> > I get back to this topic as I have a new issue now. >>> >> >> >> > Squad worker return a "Plugin postprocessing error" >>> when the tradefed plugin is triggered. >>> >> >> >> > The logs : >>> >> >> >> > >>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: >>> [2019-06-06 03:35:40 +0000] [INFO] Starting CTS/VTS plugin for test job: >>> 1717 >>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: >>> [2019-06-06 03:35:40 +0000] [ERROR] Plugin postprocessing error: >>> 'definitions' >>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: >>> Traceback (most recent call last): >>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: File >>> "/usr/local/lib/python3.5/dist-packages/squad/ci/models.py", line 132, in >>> __postprocess_testjob__ >>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: >>> plugin.postprocess_testjob(test_job) >>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: File >>> "/usr/local/lib/python3.5/dist-packages/tradefed/__init__.py", line 185, in >>> postprocess_testjob >>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: >>> for test_definition in test_action['test']['definitions']: >>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: >>> KeyError: 'definitions' >>> >> >> >> > >>> >> >> >> > Am I missing something ? >>> >> >> >> >>> >> >> >> Could you show your test job definition? CTS plugin will >>> first check >>> >> >> >> whether it's supposed to work on the test job by >>> inspecting the >>> >> >> >> definition. Apparently there is some mismatch between >>> what is expected >>> >> >> >> and what is actually there. You can hide/obfuscate all >>> URLs, they're >>> >> >> >> not that important. >>> >> >> >> >>> >> >> >> milosz >>> >> >> >> >>> >> >> >> > >>> >> >> >> > Best regards, >>> >> >> >> > Axel >>> >> >> >> > >>> >> >> >> > On Mon, 3 Jun 2019 at 10:19, Axel Lebourhis < >>> axel.lebourhis@linaro.org> wrote: >>> >> >> >> >> >>> >> >> >> >> Hi Milosz, >>> >> >> >> >> >>> >> >> >> >> Thank you for your email. It is now working fine. >>> >> >> >> >> I setup my own local HTTP server and updated my >>> tradefed test definition (based on yours) to match my setup. >>> >> >> >> >> The main thing that was missing in my test definition >>> was the "lava-test-reference". >>> >> >> >> >> >>> >> >> >> >> Thank you for you help. >>> >> >> >> >> >>> >> >> >> >> Regards, >>> >> >> >> >> Axel >>> >> >> >> >> >>> >> >> >> >> On Wed, 29 May 2019 at 11:26, Milosz Wasilewski < >>> milosz.wasilewski@linaro.org> wrote: >>> >> >> >> >>> >>> >> >> >> >>> On Wed, 29 May 2019 at 08:27, Axel Lebourhis < >>> axel.lebourhis@linaro.org> wrote: >>> >> >> >> >>> > >>> >> >> >> >>> > Hi all, >>> >> >> >> >>> > >>> >> >> >> >>> > I saw you have a repository with tradefed and ltp >>> plugins for SQUAD. >>> >> >> >> >>> > If I understand correctly, those plugins retrieve >>> CTS/VTS/LTP logs to >>> >> >> >> >>> > attach the error logs to failed test cases. >>> >> >> >> >>> > >>> >> >> >> >>> > My questions are : >>> >> >> >> >>> > How can I setup those plugins to work with my SQUAD >>> instance ? >>> >> >> >> >>> >>> >> >> >> >>> you need to install the package from pip >>> >> >> >> >>> (https://pypi.org/project/squad-linaro-plugins/) to >>> your squad >>> >> >> >> >>> instance. That should do the trick. Each project has >>> to have the >>> >> >> >> >>> plugin enabled manually. >>> >> >> >> >>> >>> >> >> >> >>> > Do I need to use a specific tradefed test >>> definition to interface it with >>> >> >> >> >>> > the plugin ? >>> >> >> >> >>> >>> >> >> >> >>> yes, tradefed plugin assumes that all logs are stored >>> outside of LAVA. >>> >> >> >> >>> Tradefed creates a zip file that is later uploaded to >>> 'artifactorial' >>> >> >> >> >>> (https://archive.validation.linaro.org/). You can >>> use some other >>> >> >> >> >>> service but it needs to support upload request in the >>> following form: >>> >> >> >> >>> curl "${CURL_VERBOSE_FLAG}" -F "path=@${ATTACHMENT}" >>> -F >>> >> >> >> >>> "token=${ARTIFACTORIAL_TOKEN}" "${ARTIFACTORIAL_URL}" >>> >> >> >> >>> ATTACHMENT is the zip file that tradefed created and >>> >> >> >> >>> ARTIFACTORIAL_TOKEN is used to authenticate the user >>> uploading the >>> >> >> >> >>> content. >>> >> >> >> >>> This step is done automatically when you use tradefed >>> [1] or >>> >> >> >> >>> noninteractive-tradefed [2] from test-definitions >>> >> >> >> >>> >>> >> >> >> >>> When running in squad post processing step plugin >>> downloads the zip >>> >> >> >> >>> file and extracts logs from it. Failed tests are >>> updated with logs. >>> >> >> >> >>> >>> >> >> >> >>> > Does the job submission works the same way ? using >>> the submitjob API ? >>> >> >> >> >>> >>> >> >> >> >>> Test job submission works the same way. Plugins work >>> as a post >>> >> >> >> >>> processing step after LAVA jobs are completed. >>> >> >> >> >>> >>> >> >> >> >>> [1] >>> https://git.linaro.org/qa/test-definitions.git/tree/automated/android/tradef... >>> >> >> >> >>> [2] >>> https://git.linaro.org/qa/test-definitions.git/tree/automated/android/nonint... >>> >> >> >> >>> >>> >> >> >> >>> milosz >>> >> >> >> >>> >>> >> >> >> >>> > >>> >> >> >> >>> > Thank you in advance, >>> >> >> >> >>> > Axel >>> >> >> >> >>> > _______________________________________________ >>> >> >> >> >>> > Squad-dev mailing list >>> >> >> >> >>> > Squad-dev@lists.linaro.org >>> >> >> >> >>> > https://lists.linaro.org/mailman/listinfo/squad-dev >>> >>