Great !
I'm going to run VTS/CTS during the weekend, I'll get back to you on Monday :)
Thanks Milosz, Axel
On Fri, 14 Jun 2019 at 13:07, Milosz Wasilewski < milosz.wasilewski@linaro.org> wrote:
1.8 now released and uploaded. I checked a few CTS and VTS jobs locally and all worked well. Please try with your VTS job as you run more tests than I have available. Should work (fingers crossed)!
milosz
On Fri, 14 Jun 2019 at 09:41, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Ok sure no problem !
Axel
On Fri, 14 Jun 2019 at 10:21, Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
Axel,
I'll try it out first with some test jobs I have. I finished late yesterday and didn't have time to do that. I only ran unit tests that don't cover all possible cases.
milosz
On Fri, 14 Jun 2019 at 09:17, Axel Lebourhis axel.lebourhis@linaro.org
wrote:
Hi Milosz,
Will you make a new release of the plugin package today with your
recent patch ?
Thanks, Axel
On Thu, 13 Jun 2019 at 15:32, Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
Yes, I will submit patch to test definitions. My previous version of
plugin patch didn't actually work. I think I have a better version now. Adding tests to verify special cases now.
milosz
On Thu, 13 Jun 2019 at 14:29, Axel Lebourhis <
axel.lebourhis@linaro.org> wrote:
Great !
So are you still going to submit a patch to test-definitions ?
Axel
On Thu, 13 Jun 2019 at 14:41, Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
> > I wrote this patch: > https://github.com/Linaro/squadplugins/pull/10 > > In my local testing it fixes the problem and a few other issues I
wansn't aware before.
> > milosz > > > On Thu, 13 Jun 2019 at 12:58, Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
>> >> OK, found the culprit. It's the VTS test name: >>
arm64-v8a.VtsHalCameraProviderV2_4Target/VtsHalCameraProviderV2_4Target.CameraHidlTest.autoFocus(legacy/0)_64bit
>> VTS decided to use "/" at the end of the test name which SQUAD
isn't very happy about (the legacy/0 part). I added a fix for [] some time ago but it looks like VTS also uses () in some cases :( I'm not sure whether we should fix it in SQUAD or in test-definitions. IMHO adding to many 'special cases' to SQUAD isn't a good idea. I'll propose a patch to test-definitions.
>> >> As a side note, I found that the plugin code doesn't always
recognize ABI properly. I'll fix that in the plugin.
>> >> milosz >> >> >> On Thu, 13 Jun 2019 at 11:51, Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
>>> >>> Thank Axel, debugging now. I don't see anything that stands out
in any obvious way.
>>> >>> milosz >>> >>> On Thu, 13 Jun 2019 at 11:21, Axel Lebourhis <
axel.lebourhis@linaro.org> wrote:
>>>> >>>> Milosz, >>>> >>>> I privately shared with you the archive containing what you
asked for.
>>>> >>>> On Thu, 13 Jun 2019 at 12:06, Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
>>>>> >>>>> Could you share your testjob YAML and test-results.xml from
VTS (may be privately). I think I know what's going on. Thanks for reporting this.
>>>>> >>>>> milosz >>>>> >>>>> On Thu, 13 Jun 2019 at 10:58, Axel Lebourhis <
axel.lebourhis@linaro.org> wrote:
>>>>>> >>>>>> Hi Milosz, >>>>>> >>>>>> Today I faced a new issue, it may be a bug. I ran a full VTS
run, the plugin returned the following error :
>>>>>> >>>>>> Jun 12 20:25:36 squad.sw.nxp.com celery[6499]: [2019-06-12
20:25:36 +0000] [INFO] Starting CTS/VTS plugin for test job: 1841
>>>>>> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: [2019-06-12
20:25:38 +0000] [ERROR] Plugin postprocessing error: too many values to unpack (expected 2)
>>>>>> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: Traceback
(most recent call last):
>>>>>> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: File
"/usr/local/lib/python3.5/dist-packages/squad/ci/models.py", line 132, in __postprocess_testjob__
>>>>>> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]:
plugin.postprocess_testjob(test_job)
>>>>>> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: File
"/usr/local/lib/python3.5/dist-packages/tradefed/__init__.py", line 207, in postprocess_testjob
>>>>>> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]:
self._assign_test_log(results.test_results.contents, failed)
>>>>>> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: File
"/usr/local/lib/python3.5/dist-packages/tradefed/__init__.py", line 50, in _assign_test_log
>>>>>> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]:
test_suite_abi, test_suite_name = test_suite_name.split(".")
>>>>>> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: ValueError:
too many values to unpack (expected 2)
>>>>>> >>>>>> Do you have any idea ? >>>>>> >>>>>> Regards, >>>>>> Axel >>>>>> >>>>>> On Tue, 11 Jun 2019 at 13:27, Axel Lebourhis <
axel.lebourhis@linaro.org> wrote:
>>>>>>> >>>>>>> Great, thanks! >>>>>>> >>>>>>> On Tue, 11 Jun 2019 at 12:42, Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
>>>>>>>> >>>>>>>> I forgot to release it. Will do it today. >>>>>>>> >>>>>>>> milosz >>>>>>>> >>>>>>>> On Tue, 11 Jun 2019 at 11:03, Axel Lebourhis <
axel.lebourhis@linaro.org> wrote:
>>>>>>>> > >>>>>>>> > Hi Milosz, >>>>>>>> > >>>>>>>> > I saw your fix has been merged, is it available now with
pip ?
>>>>>>>> > >>>>>>>> > Regards, >>>>>>>> > Axel >>>>>>>> > >>>>>>>> > On Fri, 7 Jun 2019 at 14:39, Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
>>>>>>>> >> >>>>>>>> >> https://github.com/Linaro/squadplugins/pull/9 >>>>>>>> >> Antonio, could you take a look? >>>>>>>> >> >>>>>>>> >> milosz >>>>>>>> >> >>>>>>>> >> On Fri, 7 Jun 2019 at 13:34, Axel Lebourhis <
axel.lebourhis@linaro.org> wrote:
>>>>>>>> >> > >>>>>>>> >> > Hi Milosz, >>>>>>>> >> > >>>>>>>> >> > Great thank you. >>>>>>>> >> > >>>>>>>> >> > Axel >>>>>>>> >> > >>>>>>>> >> > On Fri, 7 Jun 2019 at 14:26, Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
>>>>>>>> >> >> >>>>>>>> >> >> I think it's interactive test section that makes the
plugin unhappy.
>>>>>>>> >> >> I'll fix that and upload new version. >>>>>>>> >> >> >>>>>>>> >> >> milosz >>>>>>>> >> >> >>>>>>>> >> >> On Thu, 6 Jun 2019 at 12:45, Axel Lebourhis <
axel.lebourhis@linaro.org> wrote:
>>>>>>>> >> >> > >>>>>>>> >> >> > Hi Milosz, >>>>>>>> >> >> > >>>>>>>> >> >> > Yes sure, you will find it attached. This
definition is the one stored by SQUAD, retrieved with api/testjobs/<id>/definition/.
>>>>>>>> >> >> > I hid the sensible information with "****". >>>>>>>> >> >> > >>>>>>>> >> >> > Regards, >>>>>>>> >> >> > Axel >>>>>>>> >> >> > >>>>>>>> >> >> > On Thu, 6 Jun 2019 at 12:45, Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
>>>>>>>> >> >> >> >>>>>>>> >> >> >> On Thu, 6 Jun 2019 at 09:38, Axel Lebourhis <
axel.lebourhis@linaro.org> wrote:
>>>>>>>> >> >> >> > >>>>>>>> >> >> >> > Hi all, >>>>>>>> >> >> >> > >>>>>>>> >> >> >> > I get back to this topic as I have a new issue
now.
>>>>>>>> >> >> >> > Squad worker return a "Plugin postprocessing
error" when the tradefed plugin is triggered.
>>>>>>>> >> >> >> > The logs : >>>>>>>> >> >> >> > >>>>>>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]:
[2019-06-06 03:35:40 +0000] [INFO] Starting CTS/VTS plugin for test job: 1717
>>>>>>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]:
[2019-06-06 03:35:40 +0000] [ERROR] Plugin postprocessing error: 'definitions'
>>>>>>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]:
Traceback (most recent call last):
>>>>>>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com
celery[26318]: File "/usr/local/lib/python3.5/dist-packages/squad/ci/models.py", line 132, in __postprocess_testjob__
>>>>>>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com
celery[26318]: plugin.postprocess_testjob(test_job)
>>>>>>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com
celery[26318]: File "/usr/local/lib/python3.5/dist-packages/tradefed/__init__.py", line 185, in postprocess_testjob
>>>>>>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com
celery[26318]: for test_definition in test_action['test']['definitions']:
>>>>>>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]:
KeyError: 'definitions'
>>>>>>>> >> >> >> > >>>>>>>> >> >> >> > Am I missing something ? >>>>>>>> >> >> >> >>>>>>>> >> >> >> Could you show your test job definition? CTS
plugin will first check
>>>>>>>> >> >> >> whether it's supposed to work on the test job by
inspecting the
>>>>>>>> >> >> >> definition. Apparently there is some mismatch
between what is expected
>>>>>>>> >> >> >> and what is actually there. You can hide/obfuscate
all URLs, they're
>>>>>>>> >> >> >> not that important. >>>>>>>> >> >> >> >>>>>>>> >> >> >> milosz >>>>>>>> >> >> >> >>>>>>>> >> >> >> > >>>>>>>> >> >> >> > Best regards, >>>>>>>> >> >> >> > Axel >>>>>>>> >> >> >> > >>>>>>>> >> >> >> > On Mon, 3 Jun 2019 at 10:19, Axel Lebourhis <
axel.lebourhis@linaro.org> wrote:
>>>>>>>> >> >> >> >> >>>>>>>> >> >> >> >> Hi Milosz, >>>>>>>> >> >> >> >> >>>>>>>> >> >> >> >> Thank you for your email. It is now working
fine.
>>>>>>>> >> >> >> >> I setup my own local HTTP server and updated my
tradefed test definition (based on yours) to match my setup.
>>>>>>>> >> >> >> >> The main thing that was missing in my test
definition was the "lava-test-reference".
>>>>>>>> >> >> >> >> >>>>>>>> >> >> >> >> Thank you for you help. >>>>>>>> >> >> >> >> >>>>>>>> >> >> >> >> Regards, >>>>>>>> >> >> >> >> Axel >>>>>>>> >> >> >> >> >>>>>>>> >> >> >> >> On Wed, 29 May 2019 at 11:26, Milosz Wasilewski
milosz.wasilewski@linaro.org wrote:
>>>>>>>> >> >> >> >>> >>>>>>>> >> >> >> >>> On Wed, 29 May 2019 at 08:27, Axel Lebourhis <
axel.lebourhis@linaro.org> wrote:
>>>>>>>> >> >> >> >>> > >>>>>>>> >> >> >> >>> > Hi all, >>>>>>>> >> >> >> >>> > >>>>>>>> >> >> >> >>> > I saw you have a repository with tradefed
and ltp plugins for SQUAD.
>>>>>>>> >> >> >> >>> > If I understand correctly, those plugins
retrieve CTS/VTS/LTP logs to
>>>>>>>> >> >> >> >>> > attach the error logs to failed test cases. >>>>>>>> >> >> >> >>> > >>>>>>>> >> >> >> >>> > My questions are : >>>>>>>> >> >> >> >>> > How can I setup those plugins to work with
my SQUAD instance ?
>>>>>>>> >> >> >> >>> >>>>>>>> >> >> >> >>> you need to install the package from pip >>>>>>>> >> >> >> >>> (
https://pypi.org/project/squad-linaro-plugins/) to your squad
>>>>>>>> >> >> >> >>> instance. That should do the trick. Each
project has to have the
>>>>>>>> >> >> >> >>> plugin enabled manually. >>>>>>>> >> >> >> >>> >>>>>>>> >> >> >> >>> > Do I need to use a specific tradefed test
definition to interface it with
>>>>>>>> >> >> >> >>> > the plugin ? >>>>>>>> >> >> >> >>> >>>>>>>> >> >> >> >>> yes, tradefed plugin assumes that all logs are
stored outside of LAVA.
>>>>>>>> >> >> >> >>> Tradefed creates a zip file that is later
uploaded to 'artifactorial'
>>>>>>>> >> >> >> >>> (https://archive.validation.linaro.org/). You
can use some other
>>>>>>>> >> >> >> >>> service but it needs to support upload request
in the following form:
>>>>>>>> >> >> >> >>> curl "${CURL_VERBOSE_FLAG}" -F
"path=@${ATTACHMENT}" -F
>>>>>>>> >> >> >> >>> "token=${ARTIFACTORIAL_TOKEN}"
"${ARTIFACTORIAL_URL}"
>>>>>>>> >> >> >> >>> ATTACHMENT is the zip file that tradefed
created and
>>>>>>>> >> >> >> >>> ARTIFACTORIAL_TOKEN is used to authenticate
the user uploading the
>>>>>>>> >> >> >> >>> content. >>>>>>>> >> >> >> >>> This step is done automatically when you use
tradefed [1] or
>>>>>>>> >> >> >> >>> noninteractive-tradefed [2] from
test-definitions
>>>>>>>> >> >> >> >>> >>>>>>>> >> >> >> >>> When running in squad post processing step
plugin downloads the zip
>>>>>>>> >> >> >> >>> file and extracts logs from it. Failed tests
are updated with logs.
>>>>>>>> >> >> >> >>> >>>>>>>> >> >> >> >>> > Does the job submission works the same way ?
using the submitjob API ?
>>>>>>>> >> >> >> >>> >>>>>>>> >> >> >> >>> Test job submission works the same way.
Plugins work as a post
>>>>>>>> >> >> >> >>> processing step after LAVA jobs are completed. >>>>>>>> >> >> >> >>> >>>>>>>> >> >> >> >>> [1]
https://git.linaro.org/qa/test-definitions.git/tree/automated/android/tradef...
>>>>>>>> >> >> >> >>> [2]
https://git.linaro.org/qa/test-definitions.git/tree/automated/android/nonint...
>>>>>>>> >> >> >> >>> >>>>>>>> >> >> >> >>> milosz >>>>>>>> >> >> >> >>> >>>>>>>> >> >> >> >>> > >>>>>>>> >> >> >> >>> > Thank you in advance, >>>>>>>> >> >> >> >>> > Axel >>>>>>>> >> >> >> >>> >
>>>>>>>> >> >> >> >>> > Squad-dev mailing list >>>>>>>> >> >> >> >>> > Squad-dev@lists.linaro.org >>>>>>>> >> >> >> >>> >