Axel,
I'll try it out first with some test jobs I have. I finished late yesterday and didn't have time to do that. I only ran unit tests that don't cover all possible cases.
milosz
On Fri, 14 Jun 2019 at 09:17, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Hi Milosz,
Will you make a new release of the plugin package today with your recent patch ?
Thanks, Axel
On Thu, 13 Jun 2019 at 15:32, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
Yes, I will submit patch to test definitions. My previous version of plugin patch didn't actually work. I think I have a better version now. Adding tests to verify special cases now.
milosz
On Thu, 13 Jun 2019 at 14:29, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Great !
So are you still going to submit a patch to test-definitions ?
Axel
On Thu, 13 Jun 2019 at 14:41, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
I wrote this patch: https://github.com/Linaro/squadplugins/pull/10
In my local testing it fixes the problem and a few other issues I wansn't aware before.
milosz
On Thu, 13 Jun 2019 at 12:58, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
OK, found the culprit. It's the VTS test name: arm64-v8a.VtsHalCameraProviderV2_4Target/VtsHalCameraProviderV2_4Target.CameraHidlTest.autoFocus(legacy/0)_64bit VTS decided to use "/" at the end of the test name which SQUAD isn't very happy about (the legacy/0 part). I added a fix for [] some time ago but it looks like VTS also uses () in some cases :( I'm not sure whether we should fix it in SQUAD or in test-definitions. IMHO adding to many 'special cases' to SQUAD isn't a good idea. I'll propose a patch to test-definitions.
As a side note, I found that the plugin code doesn't always recognize ABI properly. I'll fix that in the plugin.
milosz
On Thu, 13 Jun 2019 at 11:51, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
Thank Axel, debugging now. I don't see anything that stands out in any obvious way.
milosz
On Thu, 13 Jun 2019 at 11:21, Axel Lebourhis axel.lebourhis@linaro.org wrote: > > Milosz, > > I privately shared with you the archive containing what you asked for. > > On Thu, 13 Jun 2019 at 12:06, Milosz Wasilewski milosz.wasilewski@linaro.org wrote: >> >> Could you share your testjob YAML and test-results.xml from VTS (may be privately). I think I know what's going on. Thanks for reporting this. >> >> milosz >> >> On Thu, 13 Jun 2019 at 10:58, Axel Lebourhis axel.lebourhis@linaro.org wrote: >>> >>> Hi Milosz, >>> >>> Today I faced a new issue, it may be a bug. I ran a full VTS run, the plugin returned the following error : >>> >>> Jun 12 20:25:36 squad.sw.nxp.com celery[6499]: [2019-06-12 20:25:36 +0000] [INFO] Starting CTS/VTS plugin for test job: 1841 >>> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: [2019-06-12 20:25:38 +0000] [ERROR] Plugin postprocessing error: too many values to unpack (expected 2) >>> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: Traceback (most recent call last): >>> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: File "/usr/local/lib/python3.5/dist-packages/squad/ci/models.py", line 132, in __postprocess_testjob__ >>> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: plugin.postprocess_testjob(test_job) >>> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: File "/usr/local/lib/python3.5/dist-packages/tradefed/__init__.py", line 207, in postprocess_testjob >>> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: self._assign_test_log(results.test_results.contents, failed) >>> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: File "/usr/local/lib/python3.5/dist-packages/tradefed/__init__.py", line 50, in _assign_test_log >>> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: test_suite_abi, test_suite_name = test_suite_name.split(".") >>> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: ValueError: too many values to unpack (expected 2) >>> >>> Do you have any idea ? >>> >>> Regards, >>> Axel >>> >>> On Tue, 11 Jun 2019 at 13:27, Axel Lebourhis axel.lebourhis@linaro.org wrote: >>>> >>>> Great, thanks! >>>> >>>> On Tue, 11 Jun 2019 at 12:42, Milosz Wasilewski milosz.wasilewski@linaro.org wrote: >>>>> >>>>> I forgot to release it. Will do it today. >>>>> >>>>> milosz >>>>> >>>>> On Tue, 11 Jun 2019 at 11:03, Axel Lebourhis axel.lebourhis@linaro.org wrote: >>>>> > >>>>> > Hi Milosz, >>>>> > >>>>> > I saw your fix has been merged, is it available now with pip ? >>>>> > >>>>> > Regards, >>>>> > Axel >>>>> > >>>>> > On Fri, 7 Jun 2019 at 14:39, Milosz Wasilewski milosz.wasilewski@linaro.org wrote: >>>>> >> >>>>> >> https://github.com/Linaro/squadplugins/pull/9 >>>>> >> Antonio, could you take a look? >>>>> >> >>>>> >> milosz >>>>> >> >>>>> >> On Fri, 7 Jun 2019 at 13:34, Axel Lebourhis axel.lebourhis@linaro.org wrote: >>>>> >> > >>>>> >> > Hi Milosz, >>>>> >> > >>>>> >> > Great thank you. >>>>> >> > >>>>> >> > Axel >>>>> >> > >>>>> >> > On Fri, 7 Jun 2019 at 14:26, Milosz Wasilewski milosz.wasilewski@linaro.org wrote: >>>>> >> >> >>>>> >> >> I think it's interactive test section that makes the plugin unhappy. >>>>> >> >> I'll fix that and upload new version. >>>>> >> >> >>>>> >> >> milosz >>>>> >> >> >>>>> >> >> On Thu, 6 Jun 2019 at 12:45, Axel Lebourhis axel.lebourhis@linaro.org wrote: >>>>> >> >> > >>>>> >> >> > Hi Milosz, >>>>> >> >> > >>>>> >> >> > Yes sure, you will find it attached. This definition is the one stored by SQUAD, retrieved with api/testjobs/<id>/definition/. >>>>> >> >> > I hid the sensible information with "****". >>>>> >> >> > >>>>> >> >> > Regards, >>>>> >> >> > Axel >>>>> >> >> > >>>>> >> >> > On Thu, 6 Jun 2019 at 12:45, Milosz Wasilewski milosz.wasilewski@linaro.org wrote: >>>>> >> >> >> >>>>> >> >> >> On Thu, 6 Jun 2019 at 09:38, Axel Lebourhis axel.lebourhis@linaro.org wrote: >>>>> >> >> >> > >>>>> >> >> >> > Hi all, >>>>> >> >> >> > >>>>> >> >> >> > I get back to this topic as I have a new issue now. >>>>> >> >> >> > Squad worker return a "Plugin postprocessing error" when the tradefed plugin is triggered. >>>>> >> >> >> > The logs : >>>>> >> >> >> > >>>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: [2019-06-06 03:35:40 +0000] [INFO] Starting CTS/VTS plugin for test job: 1717 >>>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: [2019-06-06 03:35:40 +0000] [ERROR] Plugin postprocessing error: 'definitions' >>>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: Traceback (most recent call last): >>>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: File "/usr/local/lib/python3.5/dist-packages/squad/ci/models.py", line 132, in __postprocess_testjob__ >>>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: plugin.postprocess_testjob(test_job) >>>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: File "/usr/local/lib/python3.5/dist-packages/tradefed/__init__.py", line 185, in postprocess_testjob >>>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: for test_definition in test_action['test']['definitions']: >>>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]: KeyError: 'definitions' >>>>> >> >> >> > >>>>> >> >> >> > Am I missing something ? >>>>> >> >> >> >>>>> >> >> >> Could you show your test job definition? CTS plugin will first check >>>>> >> >> >> whether it's supposed to work on the test job by inspecting the >>>>> >> >> >> definition. Apparently there is some mismatch between what is expected >>>>> >> >> >> and what is actually there. You can hide/obfuscate all URLs, they're >>>>> >> >> >> not that important. >>>>> >> >> >> >>>>> >> >> >> milosz >>>>> >> >> >> >>>>> >> >> >> > >>>>> >> >> >> > Best regards, >>>>> >> >> >> > Axel >>>>> >> >> >> > >>>>> >> >> >> > On Mon, 3 Jun 2019 at 10:19, Axel Lebourhis axel.lebourhis@linaro.org wrote: >>>>> >> >> >> >> >>>>> >> >> >> >> Hi Milosz, >>>>> >> >> >> >> >>>>> >> >> >> >> Thank you for your email. It is now working fine. >>>>> >> >> >> >> I setup my own local HTTP server and updated my tradefed test definition (based on yours) to match my setup. >>>>> >> >> >> >> The main thing that was missing in my test definition was the "lava-test-reference". >>>>> >> >> >> >> >>>>> >> >> >> >> Thank you for you help. >>>>> >> >> >> >> >>>>> >> >> >> >> Regards, >>>>> >> >> >> >> Axel >>>>> >> >> >> >> >>>>> >> >> >> >> On Wed, 29 May 2019 at 11:26, Milosz Wasilewski milosz.wasilewski@linaro.org wrote: >>>>> >> >> >> >>> >>>>> >> >> >> >>> On Wed, 29 May 2019 at 08:27, Axel Lebourhis axel.lebourhis@linaro.org wrote: >>>>> >> >> >> >>> > >>>>> >> >> >> >>> > Hi all, >>>>> >> >> >> >>> > >>>>> >> >> >> >>> > I saw you have a repository with tradefed and ltp plugins for SQUAD. >>>>> >> >> >> >>> > If I understand correctly, those plugins retrieve CTS/VTS/LTP logs to >>>>> >> >> >> >>> > attach the error logs to failed test cases. >>>>> >> >> >> >>> > >>>>> >> >> >> >>> > My questions are : >>>>> >> >> >> >>> > How can I setup those plugins to work with my SQUAD instance ? >>>>> >> >> >> >>> >>>>> >> >> >> >>> you need to install the package from pip >>>>> >> >> >> >>> (https://pypi.org/project/squad-linaro-plugins/) to your squad >>>>> >> >> >> >>> instance. That should do the trick. Each project has to have the >>>>> >> >> >> >>> plugin enabled manually. >>>>> >> >> >> >>> >>>>> >> >> >> >>> > Do I need to use a specific tradefed test definition to interface it with >>>>> >> >> >> >>> > the plugin ? >>>>> >> >> >> >>> >>>>> >> >> >> >>> yes, tradefed plugin assumes that all logs are stored outside of LAVA. >>>>> >> >> >> >>> Tradefed creates a zip file that is later uploaded to 'artifactorial' >>>>> >> >> >> >>> (https://archive.validation.linaro.org/). You can use some other >>>>> >> >> >> >>> service but it needs to support upload request in the following form: >>>>> >> >> >> >>> curl "${CURL_VERBOSE_FLAG}" -F "path=@${ATTACHMENT}" -F >>>>> >> >> >> >>> "token=${ARTIFACTORIAL_TOKEN}" "${ARTIFACTORIAL_URL}" >>>>> >> >> >> >>> ATTACHMENT is the zip file that tradefed created and >>>>> >> >> >> >>> ARTIFACTORIAL_TOKEN is used to authenticate the user uploading the >>>>> >> >> >> >>> content. >>>>> >> >> >> >>> This step is done automatically when you use tradefed [1] or >>>>> >> >> >> >>> noninteractive-tradefed [2] from test-definitions >>>>> >> >> >> >>> >>>>> >> >> >> >>> When running in squad post processing step plugin downloads the zip >>>>> >> >> >> >>> file and extracts logs from it. Failed tests are updated with logs. >>>>> >> >> >> >>> >>>>> >> >> >> >>> > Does the job submission works the same way ? using the submitjob API ? >>>>> >> >> >> >>> >>>>> >> >> >> >>> Test job submission works the same way. Plugins work as a post >>>>> >> >> >> >>> processing step after LAVA jobs are completed. >>>>> >> >> >> >>> >>>>> >> >> >> >>> [1] https://git.linaro.org/qa/test-definitions.git/tree/automated/android/tradef... >>>>> >> >> >> >>> [2] https://git.linaro.org/qa/test-definitions.git/tree/automated/android/nonint... >>>>> >> >> >> >>> >>>>> >> >> >> >>> milosz >>>>> >> >> >> >>> >>>>> >> >> >> >>> > >>>>> >> >> >> >>> > Thank you in advance, >>>>> >> >> >> >>> > Axel >>>>> >> >> >> >>> > _______________________________________________ >>>>> >> >> >> >>> > Squad-dev mailing list >>>>> >> >> >> >>> > Squad-dev@lists.linaro.org >>>>> >> >> >> >>> > https://lists.linaro.org/mailman/listinfo/squad-dev