Hi Milosz,
Will you make a new release of the plugin package today with your recent patch ?
Thanks, Axel
On Thu, 13 Jun 2019 at 15:32, Milosz Wasilewski < milosz.wasilewski@linaro.org> wrote:
Yes, I will submit patch to test definitions. My previous version of plugin patch didn't actually work. I think I have a better version now. Adding tests to verify special cases now.
milosz
On Thu, 13 Jun 2019 at 14:29, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Great !
So are you still going to submit a patch to test-definitions ?
Axel
On Thu, 13 Jun 2019 at 14:41, Milosz Wasilewski < milosz.wasilewski@linaro.org> wrote:
I wrote this patch: https://github.com/Linaro/squadplugins/pull/10
In my local testing it fixes the problem and a few other issues I wansn't aware before.
milosz
On Thu, 13 Jun 2019 at 12:58, Milosz Wasilewski < milosz.wasilewski@linaro.org> wrote:
OK, found the culprit. It's the VTS test name:
arm64-v8a.VtsHalCameraProviderV2_4Target/VtsHalCameraProviderV2_4Target.CameraHidlTest.autoFocus(legacy/0)_64bit VTS decided to use "/" at the end of the test name which SQUAD isn't very happy about (the legacy/0 part). I added a fix for [] some time ago but it looks like VTS also uses () in some cases :( I'm not sure whether we should fix it in SQUAD or in test-definitions. IMHO adding to many 'special cases' to SQUAD isn't a good idea. I'll propose a patch to test-definitions.
As a side note, I found that the plugin code doesn't always recognize ABI properly. I'll fix that in the plugin.
milosz
On Thu, 13 Jun 2019 at 11:51, Milosz Wasilewski < milosz.wasilewski@linaro.org> wrote:
Thank Axel, debugging now. I don't see anything that stands out in any obvious way.
milosz
On Thu, 13 Jun 2019 at 11:21, Axel Lebourhis < axel.lebourhis@linaro.org> wrote:
Milosz,
I privately shared with you the archive containing what you asked for.
On Thu, 13 Jun 2019 at 12:06, Milosz Wasilewski < milosz.wasilewski@linaro.org> wrote:
> Could you share your testjob YAML and test-results.xml from VTS (may > be privately). I think I know what's going on. Thanks for reporting this. > > milosz > > On Thu, 13 Jun 2019 at 10:58, Axel Lebourhis < > axel.lebourhis@linaro.org> wrote: > >> Hi Milosz, >> >> Today I faced a new issue, it may be a bug. I ran a full VTS run, >> the plugin returned the following error : >> >> Jun 12 20:25:36 squad.sw.nxp.com celery[6499]: [2019-06-12 >> 20:25:36 +0000] [INFO] Starting CTS/VTS plugin for test job: 1841 >> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: [2019-06-12 >> 20:25:38 +0000] [ERROR] Plugin postprocessing error: too many values to >> unpack (expected 2) >> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: Traceback (most >> recent call last): >> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: File >> "/usr/local/lib/python3.5/dist-packages/squad/ci/models.py", line 132, in >> __postprocess_testjob__ >> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: >> plugin.postprocess_testjob(test_job) >> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: File >> "/usr/local/lib/python3.5/dist-packages/tradefed/__init__.py", line 207, in >> postprocess_testjob >> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: >> self._assign_test_log(results.test_results.contents, failed) >> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: File >> "/usr/local/lib/python3.5/dist-packages/tradefed/__init__.py", line 50, in >> _assign_test_log >> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: test_suite_abi, >> test_suite_name = test_suite_name.split(".") >> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: ValueError: too >> many values to unpack (expected 2) >> >> Do you have any idea ? >> >> Regards, >> Axel >> >> On Tue, 11 Jun 2019 at 13:27, Axel Lebourhis < >> axel.lebourhis@linaro.org> wrote: >> >>> Great, thanks! >>> >>> On Tue, 11 Jun 2019 at 12:42, Milosz Wasilewski < >>> milosz.wasilewski@linaro.org> wrote: >>> >>>> I forgot to release it. Will do it today. >>>> >>>> milosz >>>> >>>> On Tue, 11 Jun 2019 at 11:03, Axel Lebourhis < >>>> axel.lebourhis@linaro.org> wrote: >>>> > >>>> > Hi Milosz, >>>> > >>>> > I saw your fix has been merged, is it available now with pip ? >>>> > >>>> > Regards, >>>> > Axel >>>> > >>>> > On Fri, 7 Jun 2019 at 14:39, Milosz Wasilewski < >>>> milosz.wasilewski@linaro.org> wrote: >>>> >> >>>> >> https://github.com/Linaro/squadplugins/pull/9 >>>> >> Antonio, could you take a look? >>>> >> >>>> >> milosz >>>> >> >>>> >> On Fri, 7 Jun 2019 at 13:34, Axel Lebourhis < >>>> axel.lebourhis@linaro.org> wrote: >>>> >> > >>>> >> > Hi Milosz, >>>> >> > >>>> >> > Great thank you. >>>> >> > >>>> >> > Axel >>>> >> > >>>> >> > On Fri, 7 Jun 2019 at 14:26, Milosz Wasilewski < >>>> milosz.wasilewski@linaro.org> wrote: >>>> >> >> >>>> >> >> I think it's interactive test section that makes the plugin >>>> unhappy. >>>> >> >> I'll fix that and upload new version. >>>> >> >> >>>> >> >> milosz >>>> >> >> >>>> >> >> On Thu, 6 Jun 2019 at 12:45, Axel Lebourhis < >>>> axel.lebourhis@linaro.org> wrote: >>>> >> >> > >>>> >> >> > Hi Milosz, >>>> >> >> > >>>> >> >> > Yes sure, you will find it attached. This definition is >>>> the one stored by SQUAD, retrieved with api/testjobs/<id>/definition/. >>>> >> >> > I hid the sensible information with "****". >>>> >> >> > >>>> >> >> > Regards, >>>> >> >> > Axel >>>> >> >> > >>>> >> >> > On Thu, 6 Jun 2019 at 12:45, Milosz Wasilewski < >>>> milosz.wasilewski@linaro.org> wrote: >>>> >> >> >> >>>> >> >> >> On Thu, 6 Jun 2019 at 09:38, Axel Lebourhis < >>>> axel.lebourhis@linaro.org> wrote: >>>> >> >> >> > >>>> >> >> >> > Hi all, >>>> >> >> >> > >>>> >> >> >> > I get back to this topic as I have a new issue now. >>>> >> >> >> > Squad worker return a "Plugin postprocessing error" >>>> when the tradefed plugin is triggered. >>>> >> >> >> > The logs : >>>> >> >> >> > >>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: >>>> [2019-06-06 03:35:40 +0000] [INFO] Starting CTS/VTS plugin for test job: >>>> 1717 >>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: >>>> [2019-06-06 03:35:40 +0000] [ERROR] Plugin postprocessing error: >>>> 'definitions' >>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: >>>> Traceback (most recent call last): >>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: >>>> File "/usr/local/lib/python3.5/dist-packages/squad/ci/models.py", line >>>> 132, in __postprocess_testjob__ >>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: >>>> plugin.postprocess_testjob(test_job) >>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: >>>> File "/usr/local/lib/python3.5/dist-packages/tradefed/__init__.py", line >>>> 185, in postprocess_testjob >>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: >>>> for test_definition in test_action['test']['definitions']: >>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: >>>> KeyError: 'definitions' >>>> >> >> >> > >>>> >> >> >> > Am I missing something ? >>>> >> >> >> >>>> >> >> >> Could you show your test job definition? CTS plugin will >>>> first check >>>> >> >> >> whether it's supposed to work on the test job by >>>> inspecting the >>>> >> >> >> definition. Apparently there is some mismatch between >>>> what is expected >>>> >> >> >> and what is actually there. You can hide/obfuscate all >>>> URLs, they're >>>> >> >> >> not that important. >>>> >> >> >> >>>> >> >> >> milosz >>>> >> >> >> >>>> >> >> >> > >>>> >> >> >> > Best regards, >>>> >> >> >> > Axel >>>> >> >> >> > >>>> >> >> >> > On Mon, 3 Jun 2019 at 10:19, Axel Lebourhis < >>>> axel.lebourhis@linaro.org> wrote: >>>> >> >> >> >> >>>> >> >> >> >> Hi Milosz, >>>> >> >> >> >> >>>> >> >> >> >> Thank you for your email. It is now working fine. >>>> >> >> >> >> I setup my own local HTTP server and updated my >>>> tradefed test definition (based on yours) to match my setup. >>>> >> >> >> >> The main thing that was missing in my test definition >>>> was the "lava-test-reference". >>>> >> >> >> >> >>>> >> >> >> >> Thank you for you help. >>>> >> >> >> >> >>>> >> >> >> >> Regards, >>>> >> >> >> >> Axel >>>> >> >> >> >> >>>> >> >> >> >> On Wed, 29 May 2019 at 11:26, Milosz Wasilewski < >>>> milosz.wasilewski@linaro.org> wrote: >>>> >> >> >> >>> >>>> >> >> >> >>> On Wed, 29 May 2019 at 08:27, Axel Lebourhis < >>>> axel.lebourhis@linaro.org> wrote: >>>> >> >> >> >>> > >>>> >> >> >> >>> > Hi all, >>>> >> >> >> >>> > >>>> >> >> >> >>> > I saw you have a repository with tradefed and ltp >>>> plugins for SQUAD. >>>> >> >> >> >>> > If I understand correctly, those plugins retrieve >>>> CTS/VTS/LTP logs to >>>> >> >> >> >>> > attach the error logs to failed test cases. >>>> >> >> >> >>> > >>>> >> >> >> >>> > My questions are : >>>> >> >> >> >>> > How can I setup those plugins to work with my >>>> SQUAD instance ? >>>> >> >> >> >>> >>>> >> >> >> >>> you need to install the package from pip >>>> >> >> >> >>> (https://pypi.org/project/squad-linaro-plugins/) to >>>> your squad >>>> >> >> >> >>> instance. That should do the trick. Each project has >>>> to have the >>>> >> >> >> >>> plugin enabled manually. >>>> >> >> >> >>> >>>> >> >> >> >>> > Do I need to use a specific tradefed test >>>> definition to interface it with >>>> >> >> >> >>> > the plugin ? >>>> >> >> >> >>> >>>> >> >> >> >>> yes, tradefed plugin assumes that all logs are >>>> stored outside of LAVA. >>>> >> >> >> >>> Tradefed creates a zip file that is later uploaded >>>> to 'artifactorial' >>>> >> >> >> >>> (https://archive.validation.linaro.org/). You can >>>> use some other >>>> >> >> >> >>> service but it needs to support upload request in >>>> the following form: >>>> >> >> >> >>> curl "${CURL_VERBOSE_FLAG}" -F "path=@${ATTACHMENT}" >>>> -F >>>> >> >> >> >>> "token=${ARTIFACTORIAL_TOKEN}" "${ARTIFACTORIAL_URL}" >>>> >> >> >> >>> ATTACHMENT is the zip file that tradefed created and >>>> >> >> >> >>> ARTIFACTORIAL_TOKEN is used to authenticate the user >>>> uploading the >>>> >> >> >> >>> content. >>>> >> >> >> >>> This step is done automatically when you use >>>> tradefed [1] or >>>> >> >> >> >>> noninteractive-tradefed [2] from test-definitions >>>> >> >> >> >>> >>>> >> >> >> >>> When running in squad post processing step plugin >>>> downloads the zip >>>> >> >> >> >>> file and extracts logs from it. Failed tests are >>>> updated with logs. >>>> >> >> >> >>> >>>> >> >> >> >>> > Does the job submission works the same way ? using >>>> the submitjob API ? >>>> >> >> >> >>> >>>> >> >> >> >>> Test job submission works the same way. Plugins work >>>> as a post >>>> >> >> >> >>> processing step after LAVA jobs are completed. >>>> >> >> >> >>> >>>> >> >> >> >>> [1] >>>> https://git.linaro.org/qa/test-definitions.git/tree/automated/android/tradef... >>>> >> >> >> >>> [2] >>>> https://git.linaro.org/qa/test-definitions.git/tree/automated/android/nonint... >>>> >> >> >> >>> >>>> >> >> >> >>> milosz >>>> >> >> >> >>> >>>> >> >> >> >>> > >>>> >> >> >> >>> > Thank you in advance, >>>> >> >> >> >>> > Axel >>>> >> >> >> >>> > _______________________________________________ >>>> >> >> >> >>> > Squad-dev mailing list >>>> >> >> >> >>> > Squad-dev@lists.linaro.org >>>> >> >> >> >>> > >>>> https://lists.linaro.org/mailman/listinfo/squad-dev >>>> >>>