Ok sure no problem !
Axel
On Fri, 14 Jun 2019 at 10:21, Milosz Wasilewski < milosz.wasilewski@linaro.org> wrote:
Axel,
I'll try it out first with some test jobs I have. I finished late yesterday and didn't have time to do that. I only ran unit tests that don't cover all possible cases.
milosz
On Fri, 14 Jun 2019 at 09:17, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Hi Milosz,
Will you make a new release of the plugin package today with your recent
patch ?
Thanks, Axel
On Thu, 13 Jun 2019 at 15:32, Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
Yes, I will submit patch to test definitions. My previous version of
plugin patch didn't actually work. I think I have a better version now. Adding tests to verify special cases now.
milosz
On Thu, 13 Jun 2019 at 14:29, Axel Lebourhis axel.lebourhis@linaro.org
wrote:
Great !
So are you still going to submit a patch to test-definitions ?
Axel
On Thu, 13 Jun 2019 at 14:41, Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
I wrote this patch: https://github.com/Linaro/squadplugins/pull/10
In my local testing it fixes the problem and a few other issues I
wansn't aware before.
milosz
On Thu, 13 Jun 2019 at 12:58, Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
OK, found the culprit. It's the VTS test name:
arm64-v8a.VtsHalCameraProviderV2_4Target/VtsHalCameraProviderV2_4Target.CameraHidlTest.autoFocus(legacy/0)_64bit
VTS decided to use "/" at the end of the test name which SQUAD isn't
very happy about (the legacy/0 part). I added a fix for [] some time ago but it looks like VTS also uses () in some cases :( I'm not sure whether we should fix it in SQUAD or in test-definitions. IMHO adding to many 'special cases' to SQUAD isn't a good idea. I'll propose a patch to test-definitions.
As a side note, I found that the plugin code doesn't always
recognize ABI properly. I'll fix that in the plugin.
milosz
On Thu, 13 Jun 2019 at 11:51, Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
> > Thank Axel, debugging now. I don't see anything that stands out in
any obvious way.
> > milosz > > On Thu, 13 Jun 2019 at 11:21, Axel Lebourhis <
axel.lebourhis@linaro.org> wrote:
>> >> Milosz, >> >> I privately shared with you the archive containing what you asked
for.
>> >> On Thu, 13 Jun 2019 at 12:06, Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
>>> >>> Could you share your testjob YAML and test-results.xml from VTS
(may be privately). I think I know what's going on. Thanks for reporting this.
>>> >>> milosz >>> >>> On Thu, 13 Jun 2019 at 10:58, Axel Lebourhis <
axel.lebourhis@linaro.org> wrote:
>>>> >>>> Hi Milosz, >>>> >>>> Today I faced a new issue, it may be a bug. I ran a full VTS
run, the plugin returned the following error :
>>>> >>>> Jun 12 20:25:36 squad.sw.nxp.com celery[6499]: [2019-06-12
20:25:36 +0000] [INFO] Starting CTS/VTS plugin for test job: 1841
>>>> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: [2019-06-12
20:25:38 +0000] [ERROR] Plugin postprocessing error: too many values to unpack (expected 2)
>>>> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: Traceback (most
recent call last):
>>>> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: File
"/usr/local/lib/python3.5/dist-packages/squad/ci/models.py", line 132, in __postprocess_testjob__
>>>> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]:
plugin.postprocess_testjob(test_job)
>>>> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: File
"/usr/local/lib/python3.5/dist-packages/tradefed/__init__.py", line 207, in postprocess_testjob
>>>> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]:
self._assign_test_log(results.test_results.contents, failed)
>>>> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: File
"/usr/local/lib/python3.5/dist-packages/tradefed/__init__.py", line 50, in _assign_test_log
>>>> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]:
test_suite_abi, test_suite_name = test_suite_name.split(".")
>>>> Jun 12 20:25:38 squad.sw.nxp.com celery[6499]: ValueError: too
many values to unpack (expected 2)
>>>> >>>> Do you have any idea ? >>>> >>>> Regards, >>>> Axel >>>> >>>> On Tue, 11 Jun 2019 at 13:27, Axel Lebourhis <
axel.lebourhis@linaro.org> wrote:
>>>>> >>>>> Great, thanks! >>>>> >>>>> On Tue, 11 Jun 2019 at 12:42, Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
>>>>>> >>>>>> I forgot to release it. Will do it today. >>>>>> >>>>>> milosz >>>>>> >>>>>> On Tue, 11 Jun 2019 at 11:03, Axel Lebourhis <
axel.lebourhis@linaro.org> wrote:
>>>>>> > >>>>>> > Hi Milosz, >>>>>> > >>>>>> > I saw your fix has been merged, is it available now with pip
?
>>>>>> > >>>>>> > Regards, >>>>>> > Axel >>>>>> > >>>>>> > On Fri, 7 Jun 2019 at 14:39, Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
>>>>>> >> >>>>>> >> https://github.com/Linaro/squadplugins/pull/9 >>>>>> >> Antonio, could you take a look? >>>>>> >> >>>>>> >> milosz >>>>>> >> >>>>>> >> On Fri, 7 Jun 2019 at 13:34, Axel Lebourhis <
axel.lebourhis@linaro.org> wrote:
>>>>>> >> > >>>>>> >> > Hi Milosz, >>>>>> >> > >>>>>> >> > Great thank you. >>>>>> >> > >>>>>> >> > Axel >>>>>> >> > >>>>>> >> > On Fri, 7 Jun 2019 at 14:26, Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
>>>>>> >> >> >>>>>> >> >> I think it's interactive test section that makes the
plugin unhappy.
>>>>>> >> >> I'll fix that and upload new version. >>>>>> >> >> >>>>>> >> >> milosz >>>>>> >> >> >>>>>> >> >> On Thu, 6 Jun 2019 at 12:45, Axel Lebourhis <
axel.lebourhis@linaro.org> wrote:
>>>>>> >> >> > >>>>>> >> >> > Hi Milosz, >>>>>> >> >> > >>>>>> >> >> > Yes sure, you will find it attached. This definition
is the one stored by SQUAD, retrieved with api/testjobs/<id>/definition/.
>>>>>> >> >> > I hid the sensible information with "****". >>>>>> >> >> > >>>>>> >> >> > Regards, >>>>>> >> >> > Axel >>>>>> >> >> > >>>>>> >> >> > On Thu, 6 Jun 2019 at 12:45, Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
>>>>>> >> >> >> >>>>>> >> >> >> On Thu, 6 Jun 2019 at 09:38, Axel Lebourhis <
axel.lebourhis@linaro.org> wrote:
>>>>>> >> >> >> > >>>>>> >> >> >> > Hi all, >>>>>> >> >> >> > >>>>>> >> >> >> > I get back to this topic as I have a new issue now. >>>>>> >> >> >> > Squad worker return a "Plugin postprocessing error"
when the tradefed plugin is triggered.
>>>>>> >> >> >> > The logs : >>>>>> >> >> >> > >>>>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]:
[2019-06-06 03:35:40 +0000] [INFO] Starting CTS/VTS plugin for test job: 1717
>>>>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]:
[2019-06-06 03:35:40 +0000] [ERROR] Plugin postprocessing error: 'definitions'
>>>>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]:
Traceback (most recent call last):
>>>>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]:
File "/usr/local/lib/python3.5/dist-packages/squad/ci/models.py", line 132, in __postprocess_testjob__
>>>>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]:
plugin.postprocess_testjob(test_job)
>>>>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]:
File "/usr/local/lib/python3.5/dist-packages/tradefed/__init__.py", line 185, in postprocess_testjob
>>>>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]:
for test_definition in test_action['test']['definitions']:
>>>>>> >> >> >> > Jun 06 03:35:40 squad.sw.nxp.com celery[26318]:
KeyError: 'definitions'
>>>>>> >> >> >> > >>>>>> >> >> >> > Am I missing something ? >>>>>> >> >> >> >>>>>> >> >> >> Could you show your test job definition? CTS plugin
will first check
>>>>>> >> >> >> whether it's supposed to work on the test job by
inspecting the
>>>>>> >> >> >> definition. Apparently there is some mismatch between
what is expected
>>>>>> >> >> >> and what is actually there. You can hide/obfuscate
all URLs, they're
>>>>>> >> >> >> not that important. >>>>>> >> >> >> >>>>>> >> >> >> milosz >>>>>> >> >> >> >>>>>> >> >> >> > >>>>>> >> >> >> > Best regards, >>>>>> >> >> >> > Axel >>>>>> >> >> >> > >>>>>> >> >> >> > On Mon, 3 Jun 2019 at 10:19, Axel Lebourhis <
axel.lebourhis@linaro.org> wrote:
>>>>>> >> >> >> >> >>>>>> >> >> >> >> Hi Milosz, >>>>>> >> >> >> >> >>>>>> >> >> >> >> Thank you for your email. It is now working fine. >>>>>> >> >> >> >> I setup my own local HTTP server and updated my
tradefed test definition (based on yours) to match my setup.
>>>>>> >> >> >> >> The main thing that was missing in my test
definition was the "lava-test-reference".
>>>>>> >> >> >> >> >>>>>> >> >> >> >> Thank you for you help. >>>>>> >> >> >> >> >>>>>> >> >> >> >> Regards, >>>>>> >> >> >> >> Axel >>>>>> >> >> >> >> >>>>>> >> >> >> >> On Wed, 29 May 2019 at 11:26, Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
>>>>>> >> >> >> >>> >>>>>> >> >> >> >>> On Wed, 29 May 2019 at 08:27, Axel Lebourhis <
axel.lebourhis@linaro.org> wrote:
>>>>>> >> >> >> >>> > >>>>>> >> >> >> >>> > Hi all, >>>>>> >> >> >> >>> > >>>>>> >> >> >> >>> > I saw you have a repository with tradefed and
ltp plugins for SQUAD.
>>>>>> >> >> >> >>> > If I understand correctly, those plugins
retrieve CTS/VTS/LTP logs to
>>>>>> >> >> >> >>> > attach the error logs to failed test cases. >>>>>> >> >> >> >>> > >>>>>> >> >> >> >>> > My questions are : >>>>>> >> >> >> >>> > How can I setup those plugins to work with my
SQUAD instance ?
>>>>>> >> >> >> >>> >>>>>> >> >> >> >>> you need to install the package from pip >>>>>> >> >> >> >>> (https://pypi.org/project/squad-linaro-plugins/)
to your squad
>>>>>> >> >> >> >>> instance. That should do the trick. Each project
has to have the
>>>>>> >> >> >> >>> plugin enabled manually. >>>>>> >> >> >> >>> >>>>>> >> >> >> >>> > Do I need to use a specific tradefed test
definition to interface it with
>>>>>> >> >> >> >>> > the plugin ? >>>>>> >> >> >> >>> >>>>>> >> >> >> >>> yes, tradefed plugin assumes that all logs are
stored outside of LAVA.
>>>>>> >> >> >> >>> Tradefed creates a zip file that is later
uploaded to 'artifactorial'
>>>>>> >> >> >> >>> (https://archive.validation.linaro.org/). You
can use some other
>>>>>> >> >> >> >>> service but it needs to support upload request in
the following form:
>>>>>> >> >> >> >>> curl "${CURL_VERBOSE_FLAG}" -F
"path=@${ATTACHMENT}" -F
>>>>>> >> >> >> >>> "token=${ARTIFACTORIAL_TOKEN}"
"${ARTIFACTORIAL_URL}"
>>>>>> >> >> >> >>> ATTACHMENT is the zip file that tradefed created
and
>>>>>> >> >> >> >>> ARTIFACTORIAL_TOKEN is used to authenticate the
user uploading the
>>>>>> >> >> >> >>> content. >>>>>> >> >> >> >>> This step is done automatically when you use
tradefed [1] or
>>>>>> >> >> >> >>> noninteractive-tradefed [2] from test-definitions >>>>>> >> >> >> >>> >>>>>> >> >> >> >>> When running in squad post processing step plugin
downloads the zip
>>>>>> >> >> >> >>> file and extracts logs from it. Failed tests are
updated with logs.
>>>>>> >> >> >> >>> >>>>>> >> >> >> >>> > Does the job submission works the same way ?
using the submitjob API ?
>>>>>> >> >> >> >>> >>>>>> >> >> >> >>> Test job submission works the same way. Plugins
work as a post
>>>>>> >> >> >> >>> processing step after LAVA jobs are completed. >>>>>> >> >> >> >>> >>>>>> >> >> >> >>> [1]
https://git.linaro.org/qa/test-definitions.git/tree/automated/android/tradef...
>>>>>> >> >> >> >>> [2]
https://git.linaro.org/qa/test-definitions.git/tree/automated/android/nonint...
>>>>>> >> >> >> >>> >>>>>> >> >> >> >>> milosz >>>>>> >> >> >> >>> >>>>>> >> >> >> >>> > >>>>>> >> >> >> >>> > Thank you in advance, >>>>>> >> >> >> >>> > Axel >>>>>> >> >> >> >>> > _______________________________________________ >>>>>> >> >> >> >>> > Squad-dev mailing list >>>>>> >> >> >> >>> > Squad-dev@lists.linaro.org >>>>>> >> >> >> >>> >