1.8 now released and uploaded. I checked a few CTS and VTS jobs locally and all worked well. Please try with your VTS job as you run more tests than I have available. Should work (fingers crossed)!
milosz
On Fri, 14 Jun 2019 at 09:41, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Ok sure no problem !
Axel
On Fri, 14 Jun 2019 at 10:21, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
Axel,
I'll try it out first with some test jobs I have. I finished late yesterday and didn't have time to do that. I only ran unit tests that don't cover all possible cases.
milosz
On Fri, 14 Jun 2019 at 09:17, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Hi Milosz,
Will you make a new release of the plugin package today with your recent patch ?
Thanks, Axel
On Thu, 13 Jun 2019 at 15:32, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
Yes, I will submit patch to test definitions. My previous version of plugin patch didn't actually work. I think I have a better version now. Adding tests to verify special cases now.
milosz
On Thu, 13 Jun 2019 at 14:29, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Great !
So are you still going to submit a patch to test-definitions ?
Axel
On Thu, 13 Jun 2019 at 14:41, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
I wrote this patch: https://github.com/Linaro/squadplugins/pull/10
In my local testing it fixes the problem and a few other issues I wansn't aware before.
milosz
On Thu, 13 Jun 2019 at 12:58, Milosz Wasilewski milosz.wasilewski@linaro.org wrote: > > OK, found the culprit. It's the VTS test name: > arm64-v8a.VtsHalCameraProviderV2_4Target/VtsHalCameraProviderV2_4Target.CameraHidlTest.autoFocus(legacy/0)_64bit > VTS decided to use "/" at the end of the test name which SQUAD isn't very happy about (the legacy/0 part). I added a fix for [] some time ago but it looks like VTS also uses () in some cases :( I'm not sure whether we should fix it in SQUAD or in test-definitions. IMHO adding to many 'special cases' to SQUAD isn't a good idea. I'll propose a patch to test-definitions. > > As a side note, I found that the plugin code doesn't always recognize ABI properly. I'll fix that in the plugin. > > milosz > > > On Thu, 13 Jun 2019 at 11:51, Milosz Wasilewski milosz.wasilewski@linaro.org wrote: >> >> Thank Axel, debugging now. I don't see anything that stands out in any obvious way. >> >> milosz >> >> On Thu, 13 Jun 2019 at 11:21, Axel Lebourhis axel.lebourhis@linaro.org wrote: >>> >>> Milosz, >>> >>> I privately shared with you the archive containing what you asked for. >>> >>> On Thu, 13 Jun 2019 at 12:06, Milosz Wasilewski milosz.wasilewski@linaro.org wrote: >>>> >>>> Could you share your testjob YAML and test-results.xml from VTS (may be privately). I think I know what's going on. Thanks for reporting this. >>>> >>>> milosz >>>> >>>> On Thu, 13 Jun 2019 at 10:58, Axel Lebourhis axel.lebourhis@linaro.org wrote: >>>>> >>>>> Hi Milosz, >>>>> >>>>> Today I faced a new issue, it may be a bug. I ran a full VTS run, the plugin returned the following error : >>>>> >>>>> Jun 12 20:25:36 squad.sw.nxp.com celery[6499]: [2019-06-12 20:25:36 +0000] [INFO] Starting CTS/VTS plugin for test job: 1841 >>>>> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: [2019-06-12 20:25:38 +0000] [ERROR] Plugin postprocessing error: too many values to unpack (expected 2) >>>>> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: Traceback (most recent call last): >>>>> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: File "/usr/local/lib/python3.5/dist-packages/squad/ci/models.py", line 132, in __postprocess_testjob__ >>>>> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: plugin.postprocess_testjob(test_job) >>>>> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: File "/usr/local/lib/python3.5/dist-packages/tradefed/__init__.py", line 207, in postprocess_testjob >>>>> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: self._assign_test_log(results.test_results.contents, failed) >>>>> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: File "/usr/local/lib/python3.5/dist-packages/tradefed/__init__.py", line 50, in _assign_test_log >>>>> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: test_suite_abi, test_suite_name = test_suite_name.split(".") >>>>> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: ValueError: too many values to unpack (expected 2) >>>>> >>>>> Do you have any idea ? >>>>> >>>>> Regards, >>>>> Axel >>>>> >>>>> On Tue, 11 Jun 2019 at 13:27, Axel Lebourhis axel.lebourhis@linaro.org wrote: >>>>>> >>>>>> Great, thanks! >>>>>> >>>>>> On Tue, 11 Jun 2019 at 12:42, Milosz Wasilewski milosz.wasilewski@linaro.org wrote: >>>>>>> >>>>>>> I forgot to release it. Will do it today. >>>>>>> >>>>>>> milosz >>>>>>> >>>>>>> On Tue, 11 Jun 2019 at 11:03, Axel Lebourhis axel.lebourhis@linaro.org wrote: >>>>>>> > >>>>>>> > Hi Milosz, >>>>>>> > >>>>>>> > I saw your fix has been merged, is it available now with pip ? >>>>>>> > >>>>>>> > Regards, >>>>>>> > Axel >>>>>>> > >>>>>>> > On Fri, 7 Jun 2019 at 14:39, Milosz Wasilewski milosz.wasilewski@linaro.org wrote: >>>>>>> >> >>>>>>> >> https://github.com/Linaro/squadplugins/pull/9 >>>>>>> >> Antonio, could you take a look? >>>>>>> >> >>>>>>> >> milosz >>>>>>> >> >>>>>>> >> On Fri, 7 Jun 2019 at 13:34, Axel Lebourhis axel.lebourhis@linaro.org wrote: >>>>>>> >> > >>>>>>> >> > Hi Milosz, >>>>>>> >> > >>>>>>> >> > Great thank you. >>>>>>> >> > >>>>>>> >> > Axel >>>>>>> >> > >>>>>>> >> > On Fri, 7 Jun 2019 at 14:26, Milosz Wasilewski milosz.wasilewski@linaro.org wrote: >>>>>>> >> >> >>>>>>> >> >> I think it's interactive test section that makes the plugin unhappy. >>>>>>> >> >> I'll fix that and upload new version. >>>>>>> >> >> >>>>>>> >> >> milosz >>>>>>> >> >> >>>>>>> >> >> On Thu, 6 Jun 2019 at 12:45, Axel Lebourhis axel.lebourhis@linaro.org wrote: >>>>>>> >> >> > >>>>>>> >> >> > Hi Milosz, >>>>>>> >> >> > >>>>>>> >> >> > Yes sure, you will find it attached. This definition is the one stored by SQUAD, retrieved with api/testjobs/<id>/definition/. >>>>>>> >> >> > I hid the sensible information with "****". >>>>>>> >> >> > >>>>>>> >> >> > Regards, >>>>>>> >> >> > Axel >>>>>>> >> >> > >>>>>>> >> >> > On Thu, 6 Jun 2019 at 12:45, Milosz Wasilewski milosz.wasilewski@linaro.org wrote: >>>>>>> >> >> >> >>>>>>> >> >> >> On Thu, 6 Jun 2019 at 09:38, Axel Lebourhis axel.lebourhis@linaro.org wrote: >>>>>>> >> >> >> > >>>>>>> >> >> >> > Hi all, >>>>>>> >> >> >> > >>>>>>> >> >> >> > I get back to this topic as I have a new issue now. >>>>>>> >> >> >> > Squad worker return a "Plugin postprocessing error" when the tradefed plugin is triggered. >>>>>>> >> >> >> > The logs : >>>>>>> >> >> >> > >>>>>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: [2019-06-06 03:35:40 +0000] [INFO] Starting CTS/VTS plugin for test job: 1717 >>>>>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: [2019-06-06 03:35:40 +0000] [ERROR] Plugin postprocessing error: 'definitions' >>>>>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: Traceback (most recent call last): >>>>>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: File "/usr/local/lib/python3.5/dist-packages/squad/ci/models.py", line 132, in __postprocess_testjob__ >>>>>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: plugin.postprocess_testjob(test_job) >>>>>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: File "/usr/local/lib/python3.5/dist-packages/tradefed/__init__.py", line 185, in postprocess_testjob >>>>>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: for test_definition in test_action['test']['definitions']: >>>>>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: KeyError: 'definitions' >>>>>>> >> >> >> > >>>>>>> >> >> >> > Am I missing something ? >>>>>>> >> >> >> >>>>>>> >> >> >> Could you show your test job definition? CTS plugin will first check >>>>>>> >> >> >> whether it's supposed to work on the test job by inspecting the >>>>>>> >> >> >> definition. Apparently there is some mismatch between what is expected >>>>>>> >> >> >> and what is actually there. You can hide/obfuscate all URLs, they're >>>>>>> >> >> >> not that important. >>>>>>> >> >> >> >>>>>>> >> >> >> milosz >>>>>>> >> >> >> >>>>>>> >> >> >> > >>>>>>> >> >> >> > Best regards, >>>>>>> >> >> >> > Axel >>>>>>> >> >> >> > >>>>>>> >> >> >> > On Mon, 3 Jun 2019 at 10:19, Axel Lebourhis axel.lebourhis@linaro.org wrote: >>>>>>> >> >> >> >> >>>>>>> >> >> >> >> Hi Milosz, >>>>>>> >> >> >> >> >>>>>>> >> >> >> >> Thank you for your email. It is now working fine. >>>>>>> >> >> >> >> I setup my own local HTTP server and updated my tradefed test definition (based on yours) to match my setup. >>>>>>> >> >> >> >> The main thing that was missing in my test definition was the "lava-test-reference". >>>>>>> >> >> >> >> >>>>>>> >> >> >> >> Thank you for you help. >>>>>>> >> >> >> >> >>>>>>> >> >> >> >> Regards, >>>>>>> >> >> >> >> Axel >>>>>>> >> >> >> >> >>>>>>> >> >> >> >> On Wed, 29 May 2019 at 11:26, Milosz Wasilewski milosz.wasilewski@linaro.org wrote: >>>>>>> >> >> >> >>> >>>>>>> >> >> >> >>> On Wed, 29 May 2019 at 08:27, Axel Lebourhis axel.lebourhis@linaro.org wrote: >>>>>>> >> >> >> >>> > >>>>>>> >> >> >> >>> > Hi all, >>>>>>> >> >> >> >>> > >>>>>>> >> >> >> >>> > I saw you have a repository with tradefed and ltp plugins for SQUAD. >>>>>>> >> >> >> >>> > If I understand correctly, those plugins retrieve CTS/VTS/LTP logs to >>>>>>> >> >> >> >>> > attach the error logs to failed test cases. >>>>>>> >> >> >> >>> > >>>>>>> >> >> >> >>> > My questions are : >>>>>>> >> >> >> >>> > How can I setup those plugins to work with my SQUAD instance ? >>>>>>> >> >> >> >>> >>>>>>> >> >> >> >>> you need to install the package from pip >>>>>>> >> >> >> >>> (https://pypi.org/project/squad-linaro-plugins/) to your squad >>>>>>> >> >> >> >>> instance. That should do the trick. Each project has to have the >>>>>>> >> >> >> >>> plugin enabled manually. >>>>>>> >> >> >> >>> >>>>>>> >> >> >> >>> > Do I need to use a specific tradefed test definition to interface it with >>>>>>> >> >> >> >>> > the plugin ? >>>>>>> >> >> >> >>> >>>>>>> >> >> >> >>> yes, tradefed plugin assumes that all logs are stored outside of LAVA. >>>>>>> >> >> >> >>> Tradefed creates a zip file that is later uploaded to 'artifactorial' >>>>>>> >> >> >> >>> (https://archive.validation.linaro.org/). You can use some other >>>>>>> >> >> >> >>> service but it needs to support upload request in the following form: >>>>>>> >> >> >> >>> curl "${CURL_VERBOSE_FLAG}" -F "path=@${ATTACHMENT}" -F >>>>>>> >> >> >> >>> "token=${ARTIFACTORIAL_TOKEN}" "${ARTIFACTORIAL_URL}" >>>>>>> >> >> >> >>> ATTACHMENT is the zip file that tradefed created and >>>>>>> >> >> >> >>> ARTIFACTORIAL_TOKEN is used to authenticate the user uploading the >>>>>>> >> >> >> >>> content. >>>>>>> >> >> >> >>> This step is done automatically when you use tradefed [1] or >>>>>>> >> >> >> >>> noninteractive-tradefed [2] from test-definitions >>>>>>> >> >> >> >>> >>>>>>> >> >> >> >>> When running in squad post processing step plugin downloads the zip >>>>>>> >> >> >> >>> file and extracts logs from it. Failed tests are updated with logs. >>>>>>> >> >> >> >>> >>>>>>> >> >> >> >>> > Does the job submission works the same way ? using the submitjob API ? >>>>>>> >> >> >> >>> >>>>>>> >> >> >> >>> Test job submission works the same way. Plugins work as a post >>>>>>> >> >> >> >>> processing step after LAVA jobs are completed. >>>>>>> >> >> >> >>> >>>>>>> >> >> >> >>> [1] https://git.linaro.org/qa/test-definitions.git/tree/automated/android/tradef... >>>>>>> >> >> >> >>> [2] https://git.linaro.org/qa/test-definitions.git/tree/automated/android/nonint... >>>>>>> >> >> >> >>> >>>>>>> >> >> >> >>> milosz >>>>>>> >> >> >> >>> >>>>>>> >> >> >> >>> > >>>>>>> >> >> >> >>> > Thank you in advance, >>>>>>> >> >> >> >>> > Axel >>>>>>> >> >> >> >>> > _______________________________________________ >>>>>>> >> >> >> >>> > Squad-dev mailing list >>>>>>> >> >> >> >>> > Squad-dev@lists.linaro.org >>>>>>> >> >> >> >>> > https://lists.linaro.org/mailman/listinfo/squad-dev