Great !
So are you still going to submit a patch to test-definitions ?
Axel
On Thu, 13 Jun 2019 at 14:41, Milosz Wasilewski < milosz.wasilewski@linaro.org> wrote:
I wrote this patch: https://github.com/Linaro/squadplugins/pull/10
In my local testing it fixes the problem and a few other issues I wansn't aware before.
milosz
On Thu, 13 Jun 2019 at 12:58, Milosz Wasilewski < milosz.wasilewski@linaro.org> wrote:
OK, found the culprit. It's the VTS test name:
arm64-v8a.VtsHalCameraProviderV2_4Target/VtsHalCameraProviderV2_4Target.CameraHidlTest.autoFocus(legacy/0)_64bit VTS decided to use "/" at the end of the test name which SQUAD isn't very happy about (the legacy/0 part). I added a fix for [] some time ago but it looks like VTS also uses () in some cases :( I'm not sure whether we should fix it in SQUAD or in test-definitions. IMHO adding to many 'special cases' to SQUAD isn't a good idea. I'll propose a patch to test-definitions.
As a side note, I found that the plugin code doesn't always recognize ABI properly. I'll fix that in the plugin.
milosz
On Thu, 13 Jun 2019 at 11:51, Milosz Wasilewski < milosz.wasilewski@linaro.org> wrote:
Thank Axel, debugging now. I don't see anything that stands out in any obvious way.
milosz
On Thu, 13 Jun 2019 at 11:21, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Milosz,
I privately shared with you the archive containing what you asked for.
On Thu, 13 Jun 2019 at 12:06, Milosz Wasilewski < milosz.wasilewski@linaro.org> wrote:
Could you share your testjob YAML and test-results.xml from VTS (may be privately). I think I know what's going on. Thanks for reporting this.
milosz
On Thu, 13 Jun 2019 at 10:58, Axel Lebourhis < axel.lebourhis@linaro.org> wrote:
Hi Milosz,
Today I faced a new issue, it may be a bug. I ran a full VTS run, the plugin returned the following error :
Jun 12 20:25:36 squad.sw.nxp.com celery[6499]: [2019-06-12 20:25:36 +0000] [INFO] Starting CTS/VTS plugin for test job: 1841 Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: [2019-06-12 20:25:38 +0000] [ERROR] Plugin postprocessing error: too many values to unpack (expected 2) Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: Traceback (most recent call last): Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: File "/usr/local/lib/python3.5/dist-packages/squad/ci/models.py", line 132, in __postprocess_testjob__ Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: plugin.postprocess_testjob(test_job) Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: File "/usr/local/lib/python3.5/dist-packages/tradefed/__init__.py", line 207, in postprocess_testjob Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: self._assign_test_log(results.test_results.contents, failed) Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: File "/usr/local/lib/python3.5/dist-packages/tradefed/__init__.py", line 50, in _assign_test_log Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: test_suite_abi, test_suite_name = test_suite_name.split(".") Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: ValueError: too many values to unpack (expected 2)
Do you have any idea ?
Regards, Axel
On Tue, 11 Jun 2019 at 13:27, Axel Lebourhis < axel.lebourhis@linaro.org> wrote:
> Great, thanks! > > On Tue, 11 Jun 2019 at 12:42, Milosz Wasilewski < > milosz.wasilewski@linaro.org> wrote: > >> I forgot to release it. Will do it today. >> >> milosz >> >> On Tue, 11 Jun 2019 at 11:03, Axel Lebourhis < >> axel.lebourhis@linaro.org> wrote: >> > >> > Hi Milosz, >> > >> > I saw your fix has been merged, is it available now with pip ? >> > >> > Regards, >> > Axel >> > >> > On Fri, 7 Jun 2019 at 14:39, Milosz Wasilewski < >> milosz.wasilewski@linaro.org> wrote: >> >> >> >> https://github.com/Linaro/squadplugins/pull/9 >> >> Antonio, could you take a look? >> >> >> >> milosz >> >> >> >> On Fri, 7 Jun 2019 at 13:34, Axel Lebourhis < >> axel.lebourhis@linaro.org> wrote: >> >> > >> >> > Hi Milosz, >> >> > >> >> > Great thank you. >> >> > >> >> > Axel >> >> > >> >> > On Fri, 7 Jun 2019 at 14:26, Milosz Wasilewski < >> milosz.wasilewski@linaro.org> wrote: >> >> >> >> >> >> I think it's interactive test section that makes the plugin >> unhappy. >> >> >> I'll fix that and upload new version. >> >> >> >> >> >> milosz >> >> >> >> >> >> On Thu, 6 Jun 2019 at 12:45, Axel Lebourhis < >> axel.lebourhis@linaro.org> wrote: >> >> >> > >> >> >> > Hi Milosz, >> >> >> > >> >> >> > Yes sure, you will find it attached. This definition is the >> one stored by SQUAD, retrieved with api/testjobs/<id>/definition/. >> >> >> > I hid the sensible information with "****". >> >> >> > >> >> >> > Regards, >> >> >> > Axel >> >> >> > >> >> >> > On Thu, 6 Jun 2019 at 12:45, Milosz Wasilewski < >> milosz.wasilewski@linaro.org> wrote: >> >> >> >> >> >> >> >> On Thu, 6 Jun 2019 at 09:38, Axel Lebourhis < >> axel.lebourhis@linaro.org> wrote: >> >> >> >> > >> >> >> >> > Hi all, >> >> >> >> > >> >> >> >> > I get back to this topic as I have a new issue now. >> >> >> >> > Squad worker return a "Plugin postprocessing error" when >> the tradefed plugin is triggered. >> >> >> >> > The logs : >> >> >> >> > >> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: >> [2019-06-06 03:35:40 +0000] [INFO] Starting CTS/VTS plugin for test job: >> 1717 >> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: >> [2019-06-06 03:35:40 +0000] [ERROR] Plugin postprocessing error: >> 'definitions' >> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: >> Traceback (most recent call last): >> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: File >> "/usr/local/lib/python3.5/dist-packages/squad/ci/models.py", line 132, in >> __postprocess_testjob__ >> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: >> plugin.postprocess_testjob(test_job) >> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: File >> "/usr/local/lib/python3.5/dist-packages/tradefed/__init__.py", line 185, in >> postprocess_testjob >> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: for >> test_definition in test_action['test']['definitions']: >> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: >> KeyError: 'definitions' >> >> >> >> > >> >> >> >> > Am I missing something ? >> >> >> >> >> >> >> >> Could you show your test job definition? CTS plugin will >> first check >> >> >> >> whether it's supposed to work on the test job by >> inspecting the >> >> >> >> definition. Apparently there is some mismatch between what >> is expected >> >> >> >> and what is actually there. You can hide/obfuscate all >> URLs, they're >> >> >> >> not that important. >> >> >> >> >> >> >> >> milosz >> >> >> >> >> >> >> >> > >> >> >> >> > Best regards, >> >> >> >> > Axel >> >> >> >> > >> >> >> >> > On Mon, 3 Jun 2019 at 10:19, Axel Lebourhis < >> axel.lebourhis@linaro.org> wrote: >> >> >> >> >> >> >> >> >> >> Hi Milosz, >> >> >> >> >> >> >> >> >> >> Thank you for your email. It is now working fine. >> >> >> >> >> I setup my own local HTTP server and updated my >> tradefed test definition (based on yours) to match my setup. >> >> >> >> >> The main thing that was missing in my test definition >> was the "lava-test-reference". >> >> >> >> >> >> >> >> >> >> Thank you for you help. >> >> >> >> >> >> >> >> >> >> Regards, >> >> >> >> >> Axel >> >> >> >> >> >> >> >> >> >> On Wed, 29 May 2019 at 11:26, Milosz Wasilewski < >> milosz.wasilewski@linaro.org> wrote: >> >> >> >> >>> >> >> >> >> >>> On Wed, 29 May 2019 at 08:27, Axel Lebourhis < >> axel.lebourhis@linaro.org> wrote: >> >> >> >> >>> > >> >> >> >> >>> > Hi all, >> >> >> >> >>> > >> >> >> >> >>> > I saw you have a repository with tradefed and ltp >> plugins for SQUAD. >> >> >> >> >>> > If I understand correctly, those plugins retrieve >> CTS/VTS/LTP logs to >> >> >> >> >>> > attach the error logs to failed test cases. >> >> >> >> >>> > >> >> >> >> >>> > My questions are : >> >> >> >> >>> > How can I setup those plugins to work with my SQUAD >> instance ? >> >> >> >> >>> >> >> >> >> >>> you need to install the package from pip >> >> >> >> >>> (https://pypi.org/project/squad-linaro-plugins/) to >> your squad >> >> >> >> >>> instance. That should do the trick. Each project has >> to have the >> >> >> >> >>> plugin enabled manually. >> >> >> >> >>> >> >> >> >> >>> > Do I need to use a specific tradefed test definition >> to interface it with >> >> >> >> >>> > the plugin ? >> >> >> >> >>> >> >> >> >> >>> yes, tradefed plugin assumes that all logs are stored >> outside of LAVA. >> >> >> >> >>> Tradefed creates a zip file that is later uploaded to >> 'artifactorial' >> >> >> >> >>> (https://archive.validation.linaro.org/). You can use >> some other >> >> >> >> >>> service but it needs to support upload request in the >> following form: >> >> >> >> >>> curl "${CURL_VERBOSE_FLAG}" -F "path=@${ATTACHMENT}" -F >> >> >> >> >>> "token=${ARTIFACTORIAL_TOKEN}" "${ARTIFACTORIAL_URL}" >> >> >> >> >>> ATTACHMENT is the zip file that tradefed created and >> >> >> >> >>> ARTIFACTORIAL_TOKEN is used to authenticate the user >> uploading the >> >> >> >> >>> content. >> >> >> >> >>> This step is done automatically when you use tradefed >> [1] or >> >> >> >> >>> noninteractive-tradefed [2] from test-definitions >> >> >> >> >>> >> >> >> >> >>> When running in squad post processing step plugin >> downloads the zip >> >> >> >> >>> file and extracts logs from it. Failed tests are >> updated with logs. >> >> >> >> >>> >> >> >> >> >>> > Does the job submission works the same way ? using >> the submitjob API ? >> >> >> >> >>> >> >> >> >> >>> Test job submission works the same way. Plugins work >> as a post >> >> >> >> >>> processing step after LAVA jobs are completed. >> >> >> >> >>> >> >> >> >> >>> [1] >> https://git.linaro.org/qa/test-definitions.git/tree/automated/android/tradef... >> >> >> >> >>> [2] >> https://git.linaro.org/qa/test-definitions.git/tree/automated/android/nonint... >> >> >> >> >>> >> >> >> >> >>> milosz >> >> >> >> >>> >> >> >> >> >>> > >> >> >> >> >>> > Thank you in advance, >> >> >> >> >>> > Axel >> >> >> >> >>> > _______________________________________________ >> >> >> >> >>> > Squad-dev mailing list >> >> >> >> >>> > Squad-dev@lists.linaro.org >> >> >> >> >>> > https://lists.linaro.org/mailman/listinfo/squad-dev >> >