Hi everyone,
I'm using LAVA at NXP and after talking with Neil and Milosz, I'd like to use SQUAD. I followed the instructions here https://squad.readthedocs.io/en/latest/install.html But when I try to run the command "squad", bash tells me the command is not found. I tried to run it through Python3 shell, but didn't work either. Maybe I'm missing something obvious ?
Best regards, Axel
On Mon, 17 Dec 2018 at 12:59, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Hi everyone,
I'm using LAVA at NXP and after talking with Neil and Milosz, I'd like to use SQUAD. I followed the instructions here https://squad.readthedocs.io/en/latest/install.html But when I try to run the command "squad", bash tells me the command is not found.
I just pulled a fresh ubuntu 18.04 container and repeated the steps from the docs. Everything works OK. Did you use python3? SQUAD requires p3 and doesn't work on p2. Here is the list of installed pip packages:
# pip3 freeze amqp==2.3.2 asn1crypto==0.24.0 billiard==3.5.0.5 celery==4.2.1 certifi==2018.11.29 chardet==3.0.4 coreapi==2.3.3 coreschema==0.0.4 cryptography==2.1.4 Django==1.11.17 django-cors-headers==2.4.0 django-filter==1.1.0 django-simple-history==2.6.0 djangorestframework==3.9.0 djangorestframework-filters==0.10.2.post0 gunicorn==19.9.0 idna==2.6 itypes==1.1.0 Jinja2==2.10 keyring==10.6.0 keyrings.alt==3.0 kombu==4.2.2 Markdown==3.0.1 MarkupSafe==1.1.0 msgpack==0.6.0 psycopg2==2.7.6.1 pycrypto==2.6.1 pygobject==3.26.1 pyparsing==2.3.0 python-dateutil==2.7.5 pytz==2018.7 pyxdg==0.25 PyYAML==3.13 pyzmq==17.1.2 requests==2.21.0 SecretStorage==2.3.1 six==1.11.0 sqlparse==0.2.4 squad==0.56 svgwrite==1.2.1 uritemplate==3.0.0 urllib3==1.24.1 vine==1.1.4 whitenoise==4.1.2
I tried to run it through Python3 shell, but didn't work either. Maybe I'm missing something obvious ?
Did you try to use virtualenv? After installing I'm able to import 'squad' in the python3 shell.
# python3 Python 3.6.7 (default, Oct 22 2018, 11:32:17) [GCC 8.2.0] on linux Type "help", "copyright", "credits" or "license" for more information.
import squad
So I've no idea what went wrong :( I'm available on #linaro-qa on freenode if you have a few moments to debug.
milosz
Best regards, Axel _______________________________________________ Squad-dev mailing list Squad-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/squad-dev
Milosz,
I was trying to install on a debian. I tried to deploy a LXC with Ubuntu 18.04 like you, now it's working fine. Is it impossible to run it on debian ?
Also, if I understood correctly, I need to install squad in a different host than LAVA ?
Regards, Axel
On Mon, 17 Dec 2018 at 14:28, Milosz Wasilewski < milosz.wasilewski@linaro.org> wrote:
On Mon, 17 Dec 2018 at 12:59, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Hi everyone,
I'm using LAVA at NXP and after talking with Neil and Milosz, I'd like to use SQUAD. I followed the instructions here https://squad.readthedocs.io/en/latest/install.html But when I try to run the command "squad", bash tells me the command is
not
found.
I just pulled a fresh ubuntu 18.04 container and repeated the steps from the docs. Everything works OK. Did you use python3? SQUAD requires p3 and doesn't work on p2. Here is the list of installed pip packages:
# pip3 freeze amqp==2.3.2 asn1crypto==0.24.0 billiard==3.5.0.5 celery==4.2.1 certifi==2018.11.29 chardet==3.0.4 coreapi==2.3.3 coreschema==0.0.4 cryptography==2.1.4 Django==1.11.17 django-cors-headers==2.4.0 django-filter==1.1.0 django-simple-history==2.6.0 djangorestframework==3.9.0 djangorestframework-filters==0.10.2.post0 gunicorn==19.9.0 idna==2.6 itypes==1.1.0 Jinja2==2.10 keyring==10.6.0 keyrings.alt==3.0 kombu==4.2.2 Markdown==3.0.1 MarkupSafe==1.1.0 msgpack==0.6.0 psycopg2==2.7.6.1 pycrypto==2.6.1 pygobject==3.26.1 pyparsing==2.3.0 python-dateutil==2.7.5 pytz==2018.7 pyxdg==0.25 PyYAML==3.13 pyzmq==17.1.2 requests==2.21.0 SecretStorage==2.3.1 six==1.11.0 sqlparse==0.2.4 squad==0.56 svgwrite==1.2.1 uritemplate==3.0.0 urllib3==1.24.1 vine==1.1.4 whitenoise==4.1.2
I tried to run it through Python3 shell, but didn't work either. Maybe I'm missing something obvious ?
Did you try to use virtualenv? After installing I'm able to import 'squad' in the python3 shell.
# python3 Python 3.6.7 (default, Oct 22 2018, 11:32:17) [GCC 8.2.0] on linux Type "help", "copyright", "credits" or "license" for more information.
import squad
So I've no idea what went wrong :( I'm available on #linaro-qa on freenode if you have a few moments to debug.
milosz
Best regards, Axel _______________________________________________ Squad-dev mailing list Squad-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/squad-dev
On Mon, 17 Dec 2018 at 13:52, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Milosz,
I was trying to install on a debian. I tried to deploy a LXC with Ubuntu 18.04 like you, now it's working fine. Is it impossible to run it on debian ?
I just tried on debian stable (9.6 in docker) and again it worked. No idea why it didn't work for you.
Also, if I understood correctly, I need to install squad in a different host than LAVA ?
I didn't try the same host as lava master, but it shouldn't be a problem. SQUAD connects to LAVA over API so it should not matter where it's installed as long as the API calls can go through.
milosz
Regards, Axel
On Mon, 17 Dec 2018 at 14:28, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Mon, 17 Dec 2018 at 12:59, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Hi everyone,
I'm using LAVA at NXP and after talking with Neil and Milosz, I'd like to use SQUAD. I followed the instructions here https://squad.readthedocs.io/en/latest/install.html But when I try to run the command "squad", bash tells me the command is not found.
I just pulled a fresh ubuntu 18.04 container and repeated the steps from the docs. Everything works OK. Did you use python3? SQUAD requires p3 and doesn't work on p2. Here is the list of installed pip packages:
# pip3 freeze amqp==2.3.2 asn1crypto==0.24.0 billiard==3.5.0.5 celery==4.2.1 certifi==2018.11.29 chardet==3.0.4 coreapi==2.3.3 coreschema==0.0.4 cryptography==2.1.4 Django==1.11.17 django-cors-headers==2.4.0 django-filter==1.1.0 django-simple-history==2.6.0 djangorestframework==3.9.0 djangorestframework-filters==0.10.2.post0 gunicorn==19.9.0 idna==2.6 itypes==1.1.0 Jinja2==2.10 keyring==10.6.0 keyrings.alt==3.0 kombu==4.2.2 Markdown==3.0.1 MarkupSafe==1.1.0 msgpack==0.6.0 psycopg2==2.7.6.1 pycrypto==2.6.1 pygobject==3.26.1 pyparsing==2.3.0 python-dateutil==2.7.5 pytz==2018.7 pyxdg==0.25 PyYAML==3.13 pyzmq==17.1.2 requests==2.21.0 SecretStorage==2.3.1 six==1.11.0 sqlparse==0.2.4 squad==0.56 svgwrite==1.2.1 uritemplate==3.0.0 urllib3==1.24.1 vine==1.1.4 whitenoise==4.1.2
I tried to run it through Python3 shell, but didn't work either. Maybe I'm missing something obvious ?
Did you try to use virtualenv? After installing I'm able to import 'squad' in the python3 shell.
# python3 Python 3.6.7 (default, Oct 22 2018, 11:32:17) [GCC 8.2.0] on linux Type "help", "copyright", "credits" or "license" for more information.
import squad
So I've no idea what went wrong :( I'm available on #linaro-qa on freenode if you have a few moments to debug.
milosz
Best regards, Axel _______________________________________________ Squad-dev mailing list Squad-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/squad-dev
On Mon, 17 Dec 2018 at 14:58, Milosz Wasilewski < milosz.wasilewski@linaro.org> wrote:
On Mon, 17 Dec 2018 at 13:52, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Milosz,
I was trying to install on a debian. I tried to deploy a LXC with Ubuntu
18.04 like you, now it's working fine.
Is it impossible to run it on debian ?
I just tried on debian stable (9.6 in docker) and again it worked. No idea why it didn't work for you.
I found the issue, ~/.local/bin wasn't in my PATH...
Also, if I understood correctly, I need to install squad in a different
host than LAVA ?
I didn't try the same host as lava master, but it shouldn't be a problem. SQUAD connects to LAVA over API so it should not matter where it's installed as long as the API calls can go through.
Ok I see. After reading the documentation, I don't really understand how to connect SQUAD and LAVA. What do you think is better between using the same host and a different one ?
Regards, Axel
milosz
Regards, Axel
On Mon, 17 Dec 2018 at 14:28, Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
On Mon, 17 Dec 2018 at 12:59, Axel Lebourhis axel.lebourhis@linaro.org
wrote:
Hi everyone,
I'm using LAVA at NXP and after talking with Neil and Milosz, I'd
like to
use SQUAD. I followed the instructions here https://squad.readthedocs.io/en/latest/install.html But when I try to run the command "squad", bash tells me the command
is not
found.
I just pulled a fresh ubuntu 18.04 container and repeated the steps from the docs. Everything works OK. Did you use python3? SQUAD requires p3 and doesn't work on p2. Here is the list of installed pip packages:
# pip3 freeze amqp==2.3.2 asn1crypto==0.24.0 billiard==3.5.0.5 celery==4.2.1 certifi==2018.11.29 chardet==3.0.4 coreapi==2.3.3 coreschema==0.0.4 cryptography==2.1.4 Django==1.11.17 django-cors-headers==2.4.0 django-filter==1.1.0 django-simple-history==2.6.0 djangorestframework==3.9.0 djangorestframework-filters==0.10.2.post0 gunicorn==19.9.0 idna==2.6 itypes==1.1.0 Jinja2==2.10 keyring==10.6.0 keyrings.alt==3.0 kombu==4.2.2 Markdown==3.0.1 MarkupSafe==1.1.0 msgpack==0.6.0 psycopg2==2.7.6.1 pycrypto==2.6.1 pygobject==3.26.1 pyparsing==2.3.0 python-dateutil==2.7.5 pytz==2018.7 pyxdg==0.25 PyYAML==3.13 pyzmq==17.1.2 requests==2.21.0 SecretStorage==2.3.1 six==1.11.0 sqlparse==0.2.4 squad==0.56 svgwrite==1.2.1 uritemplate==3.0.0 urllib3==1.24.1 vine==1.1.4 whitenoise==4.1.2
I tried to run it through Python3 shell, but didn't work either. Maybe I'm missing something obvious ?
Did you try to use virtualenv? After installing I'm able to import 'squad' in the python3 shell.
# python3 Python 3.6.7 (default, Oct 22 2018, 11:32:17) [GCC 8.2.0] on linux Type "help", "copyright", "credits" or "license" for more information.
import squad
So I've no idea what went wrong :( I'm available on #linaro-qa on freenode if you have a few moments to debug.
milosz
Best regards, Axel _______________________________________________ Squad-dev mailing list Squad-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/squad-dev
On Mon, 17 Dec 2018 at 15:49, Axel Lebourhis axel.lebourhis@linaro.org wrote:
On Mon, 17 Dec 2018 at 14:58, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Mon, 17 Dec 2018 at 13:52, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Milosz,
I was trying to install on a debian. I tried to deploy a LXC with Ubuntu 18.04 like you, now it's working fine. Is it impossible to run it on debian ?
I just tried on debian stable (9.6 in docker) and again it worked. No idea why it didn't work for you.
I found the issue, ~/.local/bin wasn't in my PATH...
I'll add this to docs. Thanks for reporting.
Also, if I understood correctly, I need to install squad in a different host than LAVA ?
I didn't try the same host as lava master, but it shouldn't be a problem. SQUAD connects to LAVA over API so it should not matter where it's installed as long as the API calls can go through.
Ok I see. After reading the documentation, I don't really understand how to connect SQUAD and LAVA. What do you think is better between using the same host and a different one ?
Let me try to explain. In SQUAD convention LAVA is a 'ci backend'. In order to connect to your instance you need to create a CI backend object in SQUAD. You can do it from /admin/ci/backend/add/ (or from REST API with POST request). The fields should be filled in as follows: - name: this is the unique name you will use in the API calls when submitting/retrieving jobs from LAVA instance. We use the TLD of the instance - url: https://<your instance TLD>/RPC2 - username: name of the user in LAVA instance (one that can log into LAVA) - token: LAVA API token for the username above
Other settings are described in the docs.
There are several ways of letting SQUAD know that there are LAVA jobs it should look for. 1. submitjob API https://squad.readthedocs.io/en/latest/ci.html#submitting-test-job-requests This is the one we use in Linaro most often. It will use SQUAD as a proxy to submit jobs to LAVA. 'backend' is the 'name' you used in the CI backend settings above. 'definition' is the same YAML that is submitted to LAVA for job execution. From this point SQUAD will track jobs submitted to LAVA. When the job is completed SQUAD will pull the results to SQUAD. For this to work you will need a background worker (celery) and listener (started with squad-admin command).
2. watchjob API https://squad.readthedocs.io/en/latest/ci.html#submitting-test-job-watch-req... You submit jobs to LAVA on your own and than ask SQUAD to listen to LAVA job status. It's not widely used and there might be issues especially with jobs that run Android CTS. If you decide to use this method, I'll need to fix these problems.
It's a good idea to check our ansible playbook for qa-reports.linaro.org. It's in github here: https://github.com/Linaro/qa-reports.linaro.org/tree/master/ansible
Hope this helps. If you still have issues please don't hesitate to ask
milosz
Regards, Axel
milosz
Regards, Axel
On Mon, 17 Dec 2018 at 14:28, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Mon, 17 Dec 2018 at 12:59, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Hi everyone,
I'm using LAVA at NXP and after talking with Neil and Milosz, I'd like to use SQUAD. I followed the instructions here https://squad.readthedocs.io/en/latest/install.html But when I try to run the command "squad", bash tells me the command is not found.
I just pulled a fresh ubuntu 18.04 container and repeated the steps from the docs. Everything works OK. Did you use python3? SQUAD requires p3 and doesn't work on p2. Here is the list of installed pip packages:
# pip3 freeze amqp==2.3.2 asn1crypto==0.24.0 billiard==3.5.0.5 celery==4.2.1 certifi==2018.11.29 chardet==3.0.4 coreapi==2.3.3 coreschema==0.0.4 cryptography==2.1.4 Django==1.11.17 django-cors-headers==2.4.0 django-filter==1.1.0 django-simple-history==2.6.0 djangorestframework==3.9.0 djangorestframework-filters==0.10.2.post0 gunicorn==19.9.0 idna==2.6 itypes==1.1.0 Jinja2==2.10 keyring==10.6.0 keyrings.alt==3.0 kombu==4.2.2 Markdown==3.0.1 MarkupSafe==1.1.0 msgpack==0.6.0 psycopg2==2.7.6.1 pycrypto==2.6.1 pygobject==3.26.1 pyparsing==2.3.0 python-dateutil==2.7.5 pytz==2018.7 pyxdg==0.25 PyYAML==3.13 pyzmq==17.1.2 requests==2.21.0 SecretStorage==2.3.1 six==1.11.0 sqlparse==0.2.4 squad==0.56 svgwrite==1.2.1 uritemplate==3.0.0 urllib3==1.24.1 vine==1.1.4 whitenoise==4.1.2
I tried to run it through Python3 shell, but didn't work either. Maybe I'm missing something obvious ?
Did you try to use virtualenv? After installing I'm able to import 'squad' in the python3 shell.
# python3 Python 3.6.7 (default, Oct 22 2018, 11:32:17) [GCC 8.2.0] on linux Type "help", "copyright", "credits" or "license" for more information.
> import squad >
So I've no idea what went wrong :( I'm available on #linaro-qa on freenode if you have a few moments to debug.
milosz
Best regards, Axel _______________________________________________ Squad-dev mailing list Squad-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/squad-dev
Hi Milosz,
First of all, I wish everyone an happy new year.
So, I'm currently trying to use Squad. I can access the web UI, I added a "group" and a "project". Now I want to submit a job to LAVA through Squad. Note that I created a "backend" through Squad administration and a Squad user through LAVA administration. I don't get how to submit a job, can you provide me a more explicit example ?
Best regards, Axel
On Mon, 17 Dec 2018 at 17:18, Milosz Wasilewski < milosz.wasilewski@linaro.org> wrote:
On Mon, 17 Dec 2018 at 15:49, Axel Lebourhis axel.lebourhis@linaro.org wrote:
On Mon, 17 Dec 2018 at 14:58, Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
On Mon, 17 Dec 2018 at 13:52, Axel Lebourhis axel.lebourhis@linaro.org
wrote:
Milosz,
I was trying to install on a debian. I tried to deploy a LXC with
Ubuntu 18.04 like you, now it's working fine.
Is it impossible to run it on debian ?
I just tried on debian stable (9.6 in docker) and again it worked. No idea why it didn't work for you.
I found the issue, ~/.local/bin wasn't in my PATH...
I'll add this to docs. Thanks for reporting.
Also, if I understood correctly, I need to install squad in a
different host than LAVA ?
I didn't try the same host as lava master, but it shouldn't be a problem. SQUAD connects to LAVA over API so it should not matter where it's installed as long as the API calls can go through.
Ok I see. After reading the documentation, I don't really understand how
to connect SQUAD and LAVA.
What do you think is better between using the same host and a different
one ?
Let me try to explain. In SQUAD convention LAVA is a 'ci backend'. In order to connect to your instance you need to create a CI backend object in SQUAD. You can do it from /admin/ci/backend/add/ (or from REST API with POST request). The fields should be filled in as follows:
- name: this is the unique name you will use in the API calls when
submitting/retrieving jobs from LAVA instance. We use the TLD of the instance
- url: https://<your instance TLD>/RPC2
- username: name of the user in LAVA instance (one that can log into LAVA)
- token: LAVA API token for the username above
Other settings are described in the docs.
There are several ways of letting SQUAD know that there are LAVA jobs it should look for.
- submitjob API
https://squad.readthedocs.io/en/latest/ci.html#submitting-test-job-requests This is the one we use in Linaro most often. It will use SQUAD as a proxy to submit jobs to LAVA. 'backend' is the 'name' you used in the CI backend settings above. 'definition' is the same YAML that is submitted to LAVA for job execution. From this point SQUAD will track jobs submitted to LAVA. When the job is completed SQUAD will pull the results to SQUAD. For this to work you will need a background worker (celery) and listener (started with squad-admin command).
- watchjob API
https://squad.readthedocs.io/en/latest/ci.html#submitting-test-job-watch-req... You submit jobs to LAVA on your own and than ask SQUAD to listen to LAVA job status. It's not widely used and there might be issues especially with jobs that run Android CTS. If you decide to use this method, I'll need to fix these problems.
It's a good idea to check our ansible playbook for qa-reports.linaro.org. It's in github here: https://github.com/Linaro/qa-reports.linaro.org/tree/master/ansible
Hope this helps. If you still have issues please don't hesitate to ask
milosz
Regards, Axel
milosz
Regards, Axel
On Mon, 17 Dec 2018 at 14:28, Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
On Mon, 17 Dec 2018 at 12:59, Axel Lebourhis <
axel.lebourhis@linaro.org> wrote:
Hi everyone,
I'm using LAVA at NXP and after talking with Neil and Milosz, I'd
like to
use SQUAD. I followed the instructions here https://squad.readthedocs.io/en/latest/install.html But when I try to run the command "squad", bash tells me the
command is not
found.
I just pulled a fresh ubuntu 18.04 container and repeated the steps from the docs. Everything works OK. Did you use python3? SQUAD requires p3 and doesn't work on p2. Here is the list of installed pip packages:
# pip3 freeze amqp==2.3.2 asn1crypto==0.24.0 billiard==3.5.0.5 celery==4.2.1 certifi==2018.11.29 chardet==3.0.4 coreapi==2.3.3 coreschema==0.0.4 cryptography==2.1.4 Django==1.11.17 django-cors-headers==2.4.0 django-filter==1.1.0 django-simple-history==2.6.0 djangorestframework==3.9.0 djangorestframework-filters==0.10.2.post0 gunicorn==19.9.0 idna==2.6 itypes==1.1.0 Jinja2==2.10 keyring==10.6.0 keyrings.alt==3.0 kombu==4.2.2 Markdown==3.0.1 MarkupSafe==1.1.0 msgpack==0.6.0 psycopg2==2.7.6.1 pycrypto==2.6.1 pygobject==3.26.1 pyparsing==2.3.0 python-dateutil==2.7.5 pytz==2018.7 pyxdg==0.25 PyYAML==3.13 pyzmq==17.1.2 requests==2.21.0 SecretStorage==2.3.1 six==1.11.0 sqlparse==0.2.4 squad==0.56 svgwrite==1.2.1 uritemplate==3.0.0 urllib3==1.24.1 vine==1.1.4 whitenoise==4.1.2
I tried to run it through Python3 shell, but didn't work either. Maybe I'm missing something obvious ?
Did you try to use virtualenv? After installing I'm able to import 'squad' in the python3 shell.
# python3 Python 3.6.7 (default, Oct 22 2018, 11:32:17) [GCC 8.2.0] on linux Type "help", "copyright", "credits" or "license" for more
information.
>> import squad >>
So I've no idea what went wrong :( I'm available on #linaro-qa on freenode if you have a few moments to debug.
milosz
Best regards, Axel _______________________________________________ Squad-dev mailing list Squad-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/squad-dev
Axel,
Happy New Year!
You've done most of the setup it seems. Does the LAVA user you created have an API token? You create these using either LAVA UI (https://validation.linaro.org/api/tokens/) or REST API if you have fresh enough LAVA version (https://master.lavasoftware.org/static/docs/v2/data-export.html#index-3). This token should be than added in SQUAD admin for backend you created.
For using SQUAD to proxy your job submissions you need web frontend with API and SQUAD worker. Our ansible scripts start frontend as systemd service with apache2 as proxy: https://github.com/Linaro/qa-reports.linaro.org/blob/master/ansible/roles/fr... Worker is an instance of celery process that talks to RabbitMQ. Our example systemd service: https://github.com/Linaro/qa-reports.linaro.org/blob/master/ansible/roles/wo...
You can also set this up locally for testing without systemd. Just use the ExecStart commands from systemd service files.
Submitting testjob to LAVA via SQUAD proxy is pretty easy. Here is an example: https://squad.readthedocs.io/en/latest/ci.html#submitting-test-job-requests
Make sure the URL for the request is in form: /api/submitjob/<team>/<project>/<version>/<ci-env>
ci-env doesn't have to exist in advance. It will be created on submission. Also make sure that 'backend' parameter matches the name of your backend in SQUAD (name field in admin). 'definition' has to be a valid YAML test job definition that works with your LAVA instance. You can try to submit it directly using LAVA interface to confirm it's valid and does what you expect.
The above setup will get you as far as submitting the job to LAVA. It will not however immediately fetch the results once the test job is finished. To do that you also need a SQUAD listener service and LAVA ZMQ publisher. SQUAD listener connects to LAVA's ZMQ publisher and monitors events related to the test job it submitted. Example listener systemd service file: https://github.com/Linaro/qa-reports.linaro.org/blob/master/ansible/roles/ma... In the local setup it's enough to issue 'listen' admin command. You can also force fetch the test jobs form admin panel. I use it for debugging but it's not a good idea to rely on this method in production setup.
Hope this helps. If you have any further questions, please don't hesitate to ask.
Best Regards, milosz
On Wed, 2 Jan 2019 at 11:09, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Hi Milosz,
First of all, I wish everyone an happy new year.
So, I'm currently trying to use Squad. I can access the web UI, I added a "group" and a "project". Now I want to submit a job to LAVA through Squad. Note that I created a "backend" through Squad administration and a Squad user through LAVA administration. I don't get how to submit a job, can you provide me a more explicit example ?
Best regards, Axel
On Mon, 17 Dec 2018 at 17:18, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Mon, 17 Dec 2018 at 15:49, Axel Lebourhis axel.lebourhis@linaro.org wrote:
On Mon, 17 Dec 2018 at 14:58, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Mon, 17 Dec 2018 at 13:52, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Milosz,
I was trying to install on a debian. I tried to deploy a LXC with Ubuntu 18.04 like you, now it's working fine. Is it impossible to run it on debian ?
I just tried on debian stable (9.6 in docker) and again it worked. No idea why it didn't work for you.
I found the issue, ~/.local/bin wasn't in my PATH...
I'll add this to docs. Thanks for reporting.
Also, if I understood correctly, I need to install squad in a different host than LAVA ?
I didn't try the same host as lava master, but it shouldn't be a problem. SQUAD connects to LAVA over API so it should not matter where it's installed as long as the API calls can go through.
Ok I see. After reading the documentation, I don't really understand how to connect SQUAD and LAVA. What do you think is better between using the same host and a different one ?
Let me try to explain. In SQUAD convention LAVA is a 'ci backend'. In order to connect to your instance you need to create a CI backend object in SQUAD. You can do it from /admin/ci/backend/add/ (or from REST API with POST request). The fields should be filled in as follows:
- name: this is the unique name you will use in the API calls when
submitting/retrieving jobs from LAVA instance. We use the TLD of the instance
- url: https://<your instance TLD>/RPC2
- username: name of the user in LAVA instance (one that can log into LAVA)
- token: LAVA API token for the username above
Other settings are described in the docs.
There are several ways of letting SQUAD know that there are LAVA jobs it should look for.
- submitjob API
https://squad.readthedocs.io/en/latest/ci.html#submitting-test-job-requests This is the one we use in Linaro most often. It will use SQUAD as a proxy to submit jobs to LAVA. 'backend' is the 'name' you used in the CI backend settings above. 'definition' is the same YAML that is submitted to LAVA for job execution. From this point SQUAD will track jobs submitted to LAVA. When the job is completed SQUAD will pull the results to SQUAD. For this to work you will need a background worker (celery) and listener (started with squad-admin command).
- watchjob API
https://squad.readthedocs.io/en/latest/ci.html#submitting-test-job-watch-req... You submit jobs to LAVA on your own and than ask SQUAD to listen to LAVA job status. It's not widely used and there might be issues especially with jobs that run Android CTS. If you decide to use this method, I'll need to fix these problems.
It's a good idea to check our ansible playbook for qa-reports.linaro.org. It's in github here: https://github.com/Linaro/qa-reports.linaro.org/tree/master/ansible
Hope this helps. If you still have issues please don't hesitate to ask
milosz
Regards, Axel
milosz
Regards, Axel
On Mon, 17 Dec 2018 at 14:28, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Mon, 17 Dec 2018 at 12:59, Axel Lebourhis axel.lebourhis@linaro.org wrote: > > Hi everyone, > > I'm using LAVA at NXP and after talking with Neil and Milosz, I'd like to > use SQUAD. > I followed the instructions here > https://squad.readthedocs.io/en/latest/install.html > But when I try to run the command "squad", bash tells me the command is not > found.
I just pulled a fresh ubuntu 18.04 container and repeated the steps from the docs. Everything works OK. Did you use python3? SQUAD requires p3 and doesn't work on p2. Here is the list of installed pip packages:
# pip3 freeze amqp==2.3.2 asn1crypto==0.24.0 billiard==3.5.0.5 celery==4.2.1 certifi==2018.11.29 chardet==3.0.4 coreapi==2.3.3 coreschema==0.0.4 cryptography==2.1.4 Django==1.11.17 django-cors-headers==2.4.0 django-filter==1.1.0 django-simple-history==2.6.0 djangorestframework==3.9.0 djangorestframework-filters==0.10.2.post0 gunicorn==19.9.0 idna==2.6 itypes==1.1.0 Jinja2==2.10 keyring==10.6.0 keyrings.alt==3.0 kombu==4.2.2 Markdown==3.0.1 MarkupSafe==1.1.0 msgpack==0.6.0 psycopg2==2.7.6.1 pycrypto==2.6.1 pygobject==3.26.1 pyparsing==2.3.0 python-dateutil==2.7.5 pytz==2018.7 pyxdg==0.25 PyYAML==3.13 pyzmq==17.1.2 requests==2.21.0 SecretStorage==2.3.1 six==1.11.0 sqlparse==0.2.4 squad==0.56 svgwrite==1.2.1 uritemplate==3.0.0 urllib3==1.24.1 vine==1.1.4 whitenoise==4.1.2
> I tried to run it through Python3 shell, but didn't work either. > Maybe I'm missing something obvious ?
Did you try to use virtualenv? After installing I'm able to import 'squad' in the python3 shell.
# python3 Python 3.6.7 (default, Oct 22 2018, 11:32:17) [GCC 8.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import squad >>>
So I've no idea what went wrong :( I'm available on #linaro-qa on freenode if you have a few moments to debug.
milosz
> > Best regards, > Axel > _______________________________________________ > Squad-dev mailing list > Squad-dev@lists.linaro.org > https://lists.linaro.org/mailman/listinfo/squad-dev
Milosz,
Thank you for all these information, but I still have some questions (it's all new for me).
On Wed, 2 Jan 2019 at 12:30, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
Axel,
Happy New Year!
You've done most of the setup it seems. Does the LAVA user you created have an API token? You create these using either LAVA UI (https://validation.linaro.org/api/tokens/) or REST API if you have fresh enough LAVA version (https://master.lavasoftware.org/static/docs/v2/data-export.html#index-3). This token should be than added in SQUAD admin for backend you created.
Yes, I've done that.
For using SQUAD to proxy your job submissions you need web frontend with API and SQUAD worker. Our ansible scripts start frontend as systemd service with apache2 as proxy:
https://github.com/Linaro/qa-reports.linaro.org/blob/master/ansible/roles/fr... Worker is an instance of celery process that talks to RabbitMQ. Our example systemd service:
https://github.com/Linaro/qa-reports.linaro.org/blob/master/ansible/roles/wo...
You can also set this up locally for testing without systemd. Just use the ExecStart commands from systemd service files.
To install your ansible scripts, I just need to clone the repo and follow the "deploy to production" instructions ?
Submitting testjob to LAVA via SQUAD proxy is pretty easy. Here is an example: https://squad.readthedocs.io/en/latest/ci.html#submitting-test-job-requests
Make sure the URL for the request is in form: /api/submitjob/<team>/<project>/<version>/<ci-env>
Does "team" refer to a "group" ? Do I have to put the name of the group/project or the number that refers to it ? And how do I know the "version" ?
Regards, Axel
ci-env doesn't have to exist in advance. It will be created on submission. Also make sure that 'backend' parameter matches the name of your backend in SQUAD (name field in admin). 'definition' has to be a valid YAML test job definition that works with your LAVA instance. You can try to submit it directly using LAVA interface to confirm it's valid and does what you expect.
The above setup will get you as far as submitting the job to LAVA. It will not however immediately fetch the results once the test job is finished. To do that you also need a SQUAD listener service and LAVA ZMQ publisher. SQUAD listener connects to LAVA's ZMQ publisher and monitors events related to the test job it submitted. Example listener systemd service file:
https://github.com/Linaro/qa-reports.linaro.org/blob/master/ansible/roles/ma... In the local setup it's enough to issue 'listen' admin command. You can also force fetch the test jobs form admin panel. I use it for debugging but it's not a good idea to rely on this method in production setup.
Hope this helps. If you have any further questions, please don't hesitate to ask.
Best Regards, milosz
On Wed, 2 Jan 2019 at 11:09, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Hi Milosz,
First of all, I wish everyone an happy new year.
So, I'm currently trying to use Squad. I can access the web UI, I added
a "group" and a "project".
Now I want to submit a job to LAVA through Squad. Note that I created a
"backend" through Squad administration and a Squad user through LAVA administration.
I don't get how to submit a job, can you provide me a more explicit
example ?
Best regards, Axel
On Mon, 17 Dec 2018 at 17:18, Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
On Mon, 17 Dec 2018 at 15:49, Axel Lebourhis axel.lebourhis@linaro.org
wrote:
On Mon, 17 Dec 2018 at 14:58, Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
On Mon, 17 Dec 2018 at 13:52, Axel Lebourhis <
axel.lebourhis@linaro.org> wrote:
Milosz,
I was trying to install on a debian. I tried to deploy a LXC with
Ubuntu 18.04 like you, now it's working fine.
Is it impossible to run it on debian ?
I just tried on debian stable (9.6 in docker) and again it worked. No idea why it didn't work for you.
I found the issue, ~/.local/bin wasn't in my PATH...
I'll add this to docs. Thanks for reporting.
Also, if I understood correctly, I need to install squad in a
different host than LAVA ?
I didn't try the same host as lava master, but it shouldn't be a problem. SQUAD connects to LAVA over API so it should not matter
where
it's installed as long as the API calls can go through.
Ok I see. After reading the documentation, I don't really understand
how to connect SQUAD and LAVA.
What do you think is better between using the same host and a
different one ?
Let me try to explain. In SQUAD convention LAVA is a 'ci backend'. In order to connect to your instance you need to create a CI backend object in SQUAD. You can do it from /admin/ci/backend/add/ (or from REST API with POST request). The fields should be filled in as follows:
- name: this is the unique name you will use in the API calls when
submitting/retrieving jobs from LAVA instance. We use the TLD of the instance
- url: https://<your instance TLD>/RPC2
- username: name of the user in LAVA instance (one that can log into
LAVA)
- token: LAVA API token for the username above
Other settings are described in the docs.
There are several ways of letting SQUAD know that there are LAVA jobs it should look for.
- submitjob API
https://squad.readthedocs.io/en/latest/ci.html#submitting-test-job-requests
This is the one we use in Linaro most often. It will use SQUAD as a proxy to submit jobs to LAVA. 'backend' is the 'name' you used in the CI backend settings above. 'definition' is the same YAML that is submitted to LAVA for job execution. From this point SQUAD will track jobs submitted to LAVA. When the job is completed SQUAD will pull the results to SQUAD. For this to work you will need a background worker (celery) and listener (started with squad-admin command).
- watchjob API
https://squad.readthedocs.io/en/latest/ci.html#submitting-test-job-watch-req...
You submit jobs to LAVA on your own and than ask SQUAD to listen to LAVA job status. It's not widely used and there might be issues especially with jobs that run Android CTS. If you decide to use this method, I'll need to fix these problems.
It's a good idea to check our ansible playbook for qa-reports.linaro.org. It's in github here: https://github.com/Linaro/qa-reports.linaro.org/tree/master/ansible
Hope this helps. If you still have issues please don't hesitate to ask
milosz
Regards, Axel
milosz
Regards, Axel
On Mon, 17 Dec 2018 at 14:28, Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
> > On Mon, 17 Dec 2018 at 12:59, Axel Lebourhis <
axel.lebourhis@linaro.org> wrote:
> > > > Hi everyone, > > > > I'm using LAVA at NXP and after talking with Neil and Milosz,
I'd like to
> > use SQUAD. > > I followed the instructions here > > https://squad.readthedocs.io/en/latest/install.html > > But when I try to run the command "squad", bash tells me the
command is not
> > found. > > I just pulled a fresh ubuntu 18.04 container and repeated the
steps
> from the docs. Everything works OK. Did you use python3? SQUAD > requires p3 and doesn't work on p2. Here is the list of installed
pip
> packages: > > # pip3 freeze > amqp==2.3.2 > asn1crypto==0.24.0 > billiard==3.5.0.5 > celery==4.2.1 > certifi==2018.11.29 > chardet==3.0.4 > coreapi==2.3.3 > coreschema==0.0.4 > cryptography==2.1.4 > Django==1.11.17 > django-cors-headers==2.4.0 > django-filter==1.1.0 > django-simple-history==2.6.0 > djangorestframework==3.9.0 > djangorestframework-filters==0.10.2.post0 > gunicorn==19.9.0 > idna==2.6 > itypes==1.1.0 > Jinja2==2.10 > keyring==10.6.0 > keyrings.alt==3.0 > kombu==4.2.2 > Markdown==3.0.1 > MarkupSafe==1.1.0 > msgpack==0.6.0 > psycopg2==2.7.6.1 > pycrypto==2.6.1 > pygobject==3.26.1 > pyparsing==2.3.0 > python-dateutil==2.7.5 > pytz==2018.7 > pyxdg==0.25 > PyYAML==3.13 > pyzmq==17.1.2 > requests==2.21.0 > SecretStorage==2.3.1 > six==1.11.0 > sqlparse==0.2.4 > squad==0.56 > svgwrite==1.2.1 > uritemplate==3.0.0 > urllib3==1.24.1 > vine==1.1.4 > whitenoise==4.1.2 > > > I tried to run it through Python3 shell, but didn't work either. > > Maybe I'm missing something obvious ? > > Did you try to use virtualenv? After installing I'm able to import > 'squad' in the python3 shell. > > # python3 > Python 3.6.7 (default, Oct 22 2018, 11:32:17) > [GCC 8.2.0] on linux > Type "help", "copyright", "credits" or "license" for more
information.
> >>> import squad > >>> > > So I've no idea what went wrong :( I'm available on #linaro-qa on > freenode if you have a few moments to debug. > > milosz > > > > > Best regards, > > Axel > > _______________________________________________ > > Squad-dev mailing list > > Squad-dev@lists.linaro.org > > https://lists.linaro.org/mailman/listinfo/squad-dev
On Wed, 2 Jan 2019 at 12:34, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Milosz,
Thank you for all these information, but I still have some questions (it's all new for me).
On Wed, 2 Jan 2019 at 12:30, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
Axel,
Happy New Year!
You've done most of the setup it seems. Does the LAVA user you created have an API token? You create these using either LAVA UI (https://validation.linaro.org/api/tokens/) or REST API if you have fresh enough LAVA version (https://master.lavasoftware.org/static/docs/v2/data-export.html#index-3). This token should be than added in SQUAD admin for backend you created.
Yes, I've done that.
For using SQUAD to proxy your job submissions you need web frontend with API and SQUAD worker. Our ansible scripts start frontend as systemd service with apache2 as proxy: https://github.com/Linaro/qa-reports.linaro.org/blob/master/ansible/roles/fr... Worker is an instance of celery process that talks to RabbitMQ. Our example systemd service: https://github.com/Linaro/qa-reports.linaro.org/blob/master/ansible/roles/wo...
You can also set this up locally for testing without systemd. Just use the ExecStart commands from systemd service files.
To install your ansible scripts, I just need to clone the repo and follow the "deploy to production" instructions ?
this will probably try to deploy to our servers. So not the best option. But with changes in hosts.production and looking at extra settings (we use it for LDAP authentication) this might work. Please note that our deployment repository is focused on using AWS. If your setup is different you might need to tweak the ansible scripts a bit.
Submitting testjob to LAVA via SQUAD proxy is pretty easy. Here is an example: https://squad.readthedocs.io/en/latest/ci.html#submitting-test-job-requests
Make sure the URL for the request is in form: /api/submitjob/<team>/<project>/<version>/<ci-env>
Does "team" refer to a "group" ? Do I have to put the name of the group/project or the number that refers to it ? And how do I know the "version" ?
yes, team==group. Sorry, I should have been more careful when it comes to naming convention. Version is your build number, git sha1 or any string that identifies the build. It also doesn't have to exist in advance. It will be created on submission.
milosz
Milosz,
Your help is much appreciated, I'm close to success. So now I can submit a job through Squad. The only issue remaining is that Squad doesn't fetch the results, even with Squad listener running. If I go to http://<squad url>/<group>/<project>/build/<version>/testjobs, I can see the jobs I submitted, but the status is still on "not fetched" with no fetch attempt. Maybe the ZMQ publisher is not configured well, I did nothing about this.
Regards, Axel
On Wed, 2 Jan 2019 at 13:41, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Wed, 2 Jan 2019 at 12:34, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Milosz,
Thank you for all these information, but I still have some questions
(it's all new for me).
On Wed, 2 Jan 2019 at 12:30, Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
Axel,
Happy New Year!
You've done most of the setup it seems. Does the LAVA user you created have an API token? You create these using either LAVA UI (https://validation.linaro.org/api/tokens/) or REST API if you have fresh enough LAVA version (
https://master.lavasoftware.org/static/docs/v2/data-export.html#index-3).
This token should be than added in SQUAD admin for backend you created.
Yes, I've done that.
For using SQUAD to proxy your job submissions you need web frontend with API and SQUAD worker. Our ansible scripts start frontend as systemd service with apache2 as proxy:
https://github.com/Linaro/qa-reports.linaro.org/blob/master/ansible/roles/fr...
Worker is an instance of celery process that talks to RabbitMQ. Our example systemd service:
https://github.com/Linaro/qa-reports.linaro.org/blob/master/ansible/roles/wo...
You can also set this up locally for testing without systemd. Just use the ExecStart commands from systemd service files.
To install your ansible scripts, I just need to clone the repo and
follow the "deploy to production" instructions ?
this will probably try to deploy to our servers. So not the best option. But with changes in hosts.production and looking at extra settings (we use it for LDAP authentication) this might work. Please note that our deployment repository is focused on using AWS. If your setup is different you might need to tweak the ansible scripts a bit.
Submitting testjob to LAVA via SQUAD proxy is pretty easy. Here is an
example:
https://squad.readthedocs.io/en/latest/ci.html#submitting-test-job-requests
Make sure the URL for the request is in form: /api/submitjob/<team>/<project>/<version>/<ci-env>
Does "team" refer to a "group" ? Do I have to put the name of the
group/project or the number that refers to it ?
And how do I know the "version" ?
yes, team==group. Sorry, I should have been more careful when it comes to naming convention. Version is your build number, git sha1 or any string that identifies the build. It also doesn't have to exist in advance. It will be created on submission.
milosz
On Wed, 2 Jan 2019 at 13:36, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Milosz,
Your help is much appreciated, I'm close to success. So now I can submit a job through Squad. The only issue remaining is that Squad doesn't fetch the results, even with Squad listener running. If I go to http://<squad url>/<group>/<project>/build/<version>/testjobs, I can see the jobs I submitted, but the status is still on "not fetched" with no fetch attempt.
First please try to force fetch using admin UI. Go to 'test jobs', select your test job, select 'fetch results of the selected jobs' from the dropdown and press 'go' button. This should schedule a task to fetch the results. It might take around a minute to execute the task (should be quicker than that).
Maybe the ZMQ publisher is not configured well, I did nothing about this.
There might be a problem on both ends. IIRC LAVA doesn't enable ZMQ publisher by default. Even if it's enabled, it might be using incorrect port or reporting incorrect port on xml-rpc interface. If you try 'squad-admin listen -v 3' you should get sth like this:
[2019-01-02 13:41:36 +0000] [INFO] Backend validation.linaro.org starting [2019-01-02 13:41:36 +0000] [DEBUG] validation.linaro.org: connecting to tcp://validation.linaro.org:5510 [2019-01-02 13:41:36 +0000] [WARNING] validation.linaro.org: PyZMQ has no support for heartbeat (requires ZeroMQ library 4.2+), connection may be unstable
This will tell you where SQUAD is trying to connect to. One more issue might be some firewall between SQUAD and LAVA preventing successful connection.
milosz
On Wed, 2 Jan 2019 at 14:47, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Wed, 2 Jan 2019 at 13:36, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Milosz,
Your help is much appreciated, I'm close to success. So now I can submit a job through Squad. The only issue remaining is that Squad doesn't fetch the results, even
with Squad listener running.
If I go to http://<squad
url>/<group>/<project>/build/<version>/testjobs,
I can see the jobs I submitted, but the status is still on "not fetched"
with no fetch attempt.
First please try to force fetch using admin UI. Go to 'test jobs', select your test job, select 'fetch results of the selected jobs' from the dropdown and press 'go' button. This should schedule a task to fetch the results. It might take around a minute to execute the task (should be quicker than that).
When I do that, the celery service gives an error message, you can find it in attachment.
Maybe the ZMQ publisher is not configured well, I did nothing about this.
There might be a problem on both ends. IIRC LAVA doesn't enable ZMQ publisher by default. Even if it's enabled, it might be using
How can I enable ZMQ publisher ?
incorrect port or reporting incorrect port on xml-rpc interface. If you try 'squad-admin listen -v 3' you should get sth like this:
[2019-01-02 13:41:36 +0000] [INFO] Backend validation.linaro.org starting [2019-01-02 13:41:36 +0000] [DEBUG] validation.linaro.org: connecting to tcp://validation.linaro.org:5510 [2019-01-02 13:41:36 +0000] [WARNING] validation.linaro.org: PyZMQ has no support for heartbeat (requires ZeroMQ library 4.2+), connection may be unstable
This tells me squad is trying to connect through 5500 port. How can I check this is the good one ?
This will tell you where SQUAD is trying to connect to. One more issue might be some firewall between SQUAD and LAVA preventing successful connection.
I don't think this can be an issue here as I'm testing on the same host.
Regards, Axel
milosz
On Wed, 2 Jan 2019 at 14:03, Axel Lebourhis axel.lebourhis@linaro.org wrote:
On Wed, 2 Jan 2019 at 14:47, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Wed, 2 Jan 2019 at 13:36, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Milosz,
Your help is much appreciated, I'm close to success. So now I can submit a job through Squad. The only issue remaining is that Squad doesn't fetch the results, even with Squad listener running. If I go to http://<squad url>/<group>/<project>/build/<version>/testjobs, I can see the jobs I submitted, but the status is still on "not fetched" with no fetch attempt.
First please try to force fetch using admin UI. Go to 'test jobs', select your test job, select 'fetch results of the selected jobs' from the dropdown and press 'go' button. This should schedule a task to fetch the results. It might take around a minute to execute the task (should be quicker than that).
When I do that, the celery service gives an error message, you can find it in attachment.
This looks like a bug in SQUAD. However I haven't seen it in our servers, so I'm a bit puzzled. Anyway, to avoid this, please set the following value in the project 'settings' field: CI_LAVA_SEND_ADMIN_EMAIL: false
Maybe the ZMQ publisher is not configured well, I did nothing about this.
There might be a problem on both ends. IIRC LAVA doesn't enable ZMQ publisher by default. Even if it's enabled, it might be using
How can I enable ZMQ publisher ?
I'm not an expert on this. Please check LAVA docs. They're pretty verbose: https://master.lavasoftware.org/static/docs/v2/advanced-installation.html#in... https://master.lavasoftware.org/static/docs/v2/data-export.html#index-6
incorrect port or reporting incorrect port on xml-rpc interface. If you try 'squad-admin listen -v 3' you should get sth like this:
[2019-01-02 13:41:36 +0000] [INFO] Backend validation.linaro.org starting [2019-01-02 13:41:36 +0000] [DEBUG] validation.linaro.org: connecting to tcp://validation.linaro.org:5510 [2019-01-02 13:41:36 +0000] [WARNING] validation.linaro.org: PyZMQ has no support for heartbeat (requires ZeroMQ library 4.2+), connection may be unstable
This tells me squad is trying to connect through 5500 port. How can I check this is the good one ?
5500 is the default one. If you're running both SQUAD and LAVA on the same machine, that should work as long as ZMQ publisher is running.
milosz
This will tell you where SQUAD is trying to connect to. One more issue might be some firewall between SQUAD and LAVA preventing successful connection.
I don't think this can be an issue here as I'm testing on the same host.
Regards, Axel
milosz
On Wed, 2 Jan 2019 at 15:14, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Wed, 2 Jan 2019 at 14:03, Axel Lebourhis axel.lebourhis@linaro.org wrote:
On Wed, 2 Jan 2019 at 14:47, Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
On Wed, 2 Jan 2019 at 13:36, Axel Lebourhis axel.lebourhis@linaro.org
wrote:
Milosz,
Your help is much appreciated, I'm close to success. So now I can submit a job through Squad. The only issue remaining is that Squad doesn't fetch the results,
even with Squad listener running.
If I go to http://<squad
url>/<group>/<project>/build/<version>/testjobs,
I can see the jobs I submitted, but the status is still on "not
fetched" with no fetch attempt.
First please try to force fetch using admin UI. Go to 'test jobs', select your test job, select 'fetch results of the selected jobs' from the dropdown and press 'go' button. This should schedule a task to fetch the results. It might take around a minute to execute the task (should be quicker than that).
When I do that, the celery service gives an error message, you can find
it in attachment.
This looks like a bug in SQUAD. However I haven't seen it in our servers, so I'm a bit puzzled. Anyway, to avoid this, please set the following value in the project 'settings' field: CI_LAVA_SEND_ADMIN_EMAIL: false
Ok now it's working I can force the fetch. How can I automate this ?
Maybe the ZMQ publisher is not configured well, I did nothing about
this.
There might be a problem on both ends. IIRC LAVA doesn't enable ZMQ publisher by default. Even if it's enabled, it might be using
How can I enable ZMQ publisher ?
I'm not an expert on this. Please check LAVA docs. They're pretty verbose:
https://master.lavasoftware.org/static/docs/v2/advanced-installation.html#in... https://master.lavasoftware.org/static/docs/v2/data-export.html#index-6
Thank you I configured my instance as needed.
incorrect port or reporting incorrect port on xml-rpc interface. If you try 'squad-admin listen -v 3' you should get sth like this:
[2019-01-02 13:41:36 +0000] [INFO] Backend validation.linaro.org
starting
[2019-01-02 13:41:36 +0000] [DEBUG] validation.linaro.org: connecting to tcp://validation.linaro.org:5510 [2019-01-02 13:41:36 +0000] [WARNING] validation.linaro.org: PyZMQ has no support for heartbeat (requires ZeroMQ library 4.2+), connection may be unstable
This tells me squad is trying to connect through 5500 port. How can I
check this is the good one ?
5500 is the default one. If you're running both SQUAD and LAVA on the same machine, that should work as long as ZMQ publisher is running.
milosz
This will tell you where SQUAD is trying to connect to. One more issue might be some firewall between SQUAD and LAVA preventing successful connection.
I don't think this can be an issue here as I'm testing on the same host.
Regards, Axel
milosz
On Wed, 2 Jan 2019 at 15:01, Axel Lebourhis axel.lebourhis@linaro.org wrote:
On Wed, 2 Jan 2019 at 15:14, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Wed, 2 Jan 2019 at 14:03, Axel Lebourhis axel.lebourhis@linaro.org wrote:
On Wed, 2 Jan 2019 at 14:47, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Wed, 2 Jan 2019 at 13:36, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Milosz,
Your help is much appreciated, I'm close to success. So now I can submit a job through Squad. The only issue remaining is that Squad doesn't fetch the results, even with Squad listener running. If I go to http://<squad url>/<group>/<project>/build/<version>/testjobs, I can see the jobs I submitted, but the status is still on "not fetched" with no fetch attempt.
First please try to force fetch using admin UI. Go to 'test jobs', select your test job, select 'fetch results of the selected jobs' from the dropdown and press 'go' button. This should schedule a task to fetch the results. It might take around a minute to execute the task (should be quicker than that).
When I do that, the celery service gives an error message, you can find it in attachment.
This looks like a bug in SQUAD. However I haven't seen it in our servers, so I'm a bit puzzled. Anyway, to avoid this, please set the following value in the project 'settings' field: CI_LAVA_SEND_ADMIN_EMAIL: false
Ok now it's working I can force the fetch. How can I automate this ?
This is just for debugging. Fetchin using ZMQ events should be used instead.
Maybe the ZMQ publisher is not configured well, I did nothing about this.
There might be a problem on both ends. IIRC LAVA doesn't enable ZMQ publisher by default. Even if it's enabled, it might be using
How can I enable ZMQ publisher ?
I'm not an expert on this. Please check LAVA docs. They're pretty verbose: https://master.lavasoftware.org/static/docs/v2/advanced-installation.html#in... https://master.lavasoftware.org/static/docs/v2/data-export.html#index-6
Thank you I configured my instance as needed.
Great. Could you now check whether automated fetching works when the jobs finish in LAVA?
milosz
On Wed, 2 Jan 2019 at 16:06, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Wed, 2 Jan 2019 at 15:01, Axel Lebourhis axel.lebourhis@linaro.org wrote:
On Wed, 2 Jan 2019 at 15:14, Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
On Wed, 2 Jan 2019 at 14:03, Axel Lebourhis axel.lebourhis@linaro.org
wrote:
On Wed, 2 Jan 2019 at 14:47, Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
On Wed, 2 Jan 2019 at 13:36, Axel Lebourhis <
axel.lebourhis@linaro.org> wrote:
Milosz,
Your help is much appreciated, I'm close to success. So now I can submit a job through Squad. The only issue remaining is that Squad doesn't fetch the results,
even with Squad listener running.
If I go to http://<squad
url>/<group>/<project>/build/<version>/testjobs,
I can see the jobs I submitted, but the status is still on "not
fetched" with no fetch attempt.
First please try to force fetch using admin UI. Go to 'test jobs', select your test job, select 'fetch results of the selected jobs'
from
the dropdown and press 'go' button. This should schedule a task to fetch the results. It might take around a minute to execute the task (should be quicker than that).
When I do that, the celery service gives an error message, you can
find it in attachment.
This looks like a bug in SQUAD. However I haven't seen it in our servers, so I'm a bit puzzled. Anyway, to avoid this, please set the following value in the project 'settings' field: CI_LAVA_SEND_ADMIN_EMAIL: false
Ok now it's working I can force the fetch. How can I automate this ?
This is just for debugging. Fetchin using ZMQ events should be used instead.
Maybe the ZMQ publisher is not configured well, I did nothing
about this.
There might be a problem on both ends. IIRC LAVA doesn't enable ZMQ publisher by default. Even if it's enabled, it might be using
How can I enable ZMQ publisher ?
I'm not an expert on this. Please check LAVA docs. They're pretty
verbose:
https://master.lavasoftware.org/static/docs/v2/advanced-installation.html#in...
https://master.lavasoftware.org/static/docs/v2/data-export.html#index-6
Thank you I configured my instance as needed.
Great. Could you now check whether automated fetching works when the jobs finish in LAVA?
Yes the results are fetched at the end of the job. I can see traces in logs that everything is going well. I now need to install squad on the production instance, but it should be ok :)
Thank you very much for your help, it would have been harder without it !
Regards, Axel
milosz
On Wed, 2 Jan 2019 at 15:33, Axel Lebourhis axel.lebourhis@linaro.org wrote:
On Wed, 2 Jan 2019 at 16:06, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Wed, 2 Jan 2019 at 15:01, Axel Lebourhis axel.lebourhis@linaro.org wrote:
On Wed, 2 Jan 2019 at 15:14, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Wed, 2 Jan 2019 at 14:03, Axel Lebourhis axel.lebourhis@linaro.org wrote:
On Wed, 2 Jan 2019 at 14:47, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Wed, 2 Jan 2019 at 13:36, Axel Lebourhis axel.lebourhis@linaro.org wrote: > > Milosz, > > Your help is much appreciated, I'm close to success. > So now I can submit a job through Squad. > The only issue remaining is that Squad doesn't fetch the results, even with Squad listener running. > If I go to http://<squad url>/<group>/<project>/build/<version>/testjobs, > I can see the jobs I submitted, but the status is still on "not fetched" with no fetch attempt.
First please try to force fetch using admin UI. Go to 'test jobs', select your test job, select 'fetch results of the selected jobs' from the dropdown and press 'go' button. This should schedule a task to fetch the results. It might take around a minute to execute the task (should be quicker than that).
When I do that, the celery service gives an error message, you can find it in attachment.
This looks like a bug in SQUAD. However I haven't seen it in our servers, so I'm a bit puzzled. Anyway, to avoid this, please set the following value in the project 'settings' field: CI_LAVA_SEND_ADMIN_EMAIL: false
Ok now it's working I can force the fetch. How can I automate this ?
This is just for debugging. Fetchin using ZMQ events should be used instead.
> Maybe the ZMQ publisher is not configured well, I did nothing about this.
There might be a problem on both ends. IIRC LAVA doesn't enable ZMQ publisher by default. Even if it's enabled, it might be using
How can I enable ZMQ publisher ?
I'm not an expert on this. Please check LAVA docs. They're pretty verbose: https://master.lavasoftware.org/static/docs/v2/advanced-installation.html#in... https://master.lavasoftware.org/static/docs/v2/data-export.html#index-6
Thank you I configured my instance as needed.
Great. Could you now check whether automated fetching works when the jobs finish in LAVA?
Yes the results are fetched at the end of the job. I can see traces in logs that everything is going well. I now need to install squad on the production instance, but it should be ok :)
Thank you very much for your help, it would have been harder without it !
I'm glad you got it working. However it sounds like we need some step-by-step instructions. I'll create a github ticket to make sure it happens.
milosz
Regards, Axel
milosz
Hi Milosz,
I wanted to warn you about an issue if a user uses python3.7. When submitting a job, the squad service will return an error message saying : AttributeError: module 're' has no attribute '_pattern_type' this error occurs in the file "~/.local/lib/python3.7/site-packages/celery/app/routes.py"
This can be work arounded by adding this line right after "import re": re._pattern_type = re.Pattern
Hope this helps.
Axel
On Wed, 2 Jan 2019 at 17:35, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Wed, 2 Jan 2019 at 15:33, Axel Lebourhis axel.lebourhis@linaro.org wrote:
On Wed, 2 Jan 2019 at 16:06, Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
On Wed, 2 Jan 2019 at 15:01, Axel Lebourhis axel.lebourhis@linaro.org
wrote:
On Wed, 2 Jan 2019 at 15:14, Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
On Wed, 2 Jan 2019 at 14:03, Axel Lebourhis <
axel.lebourhis@linaro.org> wrote:
On Wed, 2 Jan 2019 at 14:47, Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
> > On Wed, 2 Jan 2019 at 13:36, Axel Lebourhis <
axel.lebourhis@linaro.org> wrote:
> > > > Milosz, > > > > Your help is much appreciated, I'm close to success. > > So now I can submit a job through Squad. > > The only issue remaining is that Squad doesn't fetch the
results, even with Squad listener running.
> > If I go to http://<squad
url>/<group>/<project>/build/<version>/testjobs,
> > I can see the jobs I submitted, but the status is still on "not
fetched" with no fetch attempt.
> > First please try to force fetch using admin UI. Go to 'test jobs', > select your test job, select 'fetch results of the selected jobs'
from
> the dropdown and press 'go' button. This should schedule a task to > fetch the results. It might take around a minute to execute the
task
> (should be quicker than that).
When I do that, the celery service gives an error message, you can
find it in attachment.
This looks like a bug in SQUAD. However I haven't seen it in our servers, so I'm a bit puzzled. Anyway, to avoid this, please set the following value in the project 'settings' field: CI_LAVA_SEND_ADMIN_EMAIL: false
Ok now it's working I can force the fetch. How can I automate this ?
This is just for debugging. Fetchin using ZMQ events should be used
instead.
> > > > Maybe the ZMQ publisher is not configured well, I did nothing
about this.
> > There might be a problem on both ends. IIRC LAVA doesn't enable
ZMQ
> publisher by default. Even if it's enabled, it might be using
How can I enable ZMQ publisher ?
I'm not an expert on this. Please check LAVA docs. They're pretty
verbose:
https://master.lavasoftware.org/static/docs/v2/advanced-installation.html#in...
https://master.lavasoftware.org/static/docs/v2/data-export.html#index-6
Thank you I configured my instance as needed.
Great. Could you now check whether automated fetching works when the jobs finish in LAVA?
Yes the results are fetched at the end of the job. I can see traces in
logs that everything is going well.
I now need to install squad on the production instance, but it should be
ok :)
Thank you very much for your help, it would have been harder without it !
I'm glad you got it working. However it sounds like we need some step-by-step instructions. I'll create a github ticket to make sure it happens.
milosz
Regards, Axel
milosz
On Thu, 3 Jan 2019 at 10:50, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Hi Milosz,
I wanted to warn you about an issue if a user uses python3.7. When submitting a job, the squad service will return an error message saying : AttributeError: module 're' has no attribute '_pattern_type' this error occurs in the file "~/.local/lib/python3.7/site-packages/celery/app/routes.py"
This can be work arounded by adding this line right after "import re": re._pattern_type = re.Pattern
Hope this helps.
Thanks Alex. Reported here: https://github.com/Linaro/squad/issues/442 Should be fixed in the next release.
milosz
Axel
On Wed, 2 Jan 2019 at 17:35, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Wed, 2 Jan 2019 at 15:33, Axel Lebourhis axel.lebourhis@linaro.org wrote:
On Wed, 2 Jan 2019 at 16:06, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Wed, 2 Jan 2019 at 15:01, Axel Lebourhis axel.lebourhis@linaro.org wrote:
On Wed, 2 Jan 2019 at 15:14, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Wed, 2 Jan 2019 at 14:03, Axel Lebourhis axel.lebourhis@linaro.org wrote: > > > > On Wed, 2 Jan 2019 at 14:47, Milosz Wasilewski milosz.wasilewski@linaro.org wrote: >> >> On Wed, 2 Jan 2019 at 13:36, Axel Lebourhis axel.lebourhis@linaro.org wrote: >> > >> > Milosz, >> > >> > Your help is much appreciated, I'm close to success. >> > So now I can submit a job through Squad. >> > The only issue remaining is that Squad doesn't fetch the results, even with Squad listener running. >> > If I go to http://<squad url>/<group>/<project>/build/<version>/testjobs, >> > I can see the jobs I submitted, but the status is still on "not fetched" with no fetch attempt. >> >> First please try to force fetch using admin UI. Go to 'test jobs', >> select your test job, select 'fetch results of the selected jobs' from >> the dropdown and press 'go' button. This should schedule a task to >> fetch the results. It might take around a minute to execute the task >> (should be quicker than that). > > > When I do that, the celery service gives an error message, you can find it in attachment.
This looks like a bug in SQUAD. However I haven't seen it in our servers, so I'm a bit puzzled. Anyway, to avoid this, please set the following value in the project 'settings' field: CI_LAVA_SEND_ADMIN_EMAIL: false
Ok now it's working I can force the fetch. How can I automate this ?
This is just for debugging. Fetchin using ZMQ events should be used instead.
> >> >> >> > Maybe the ZMQ publisher is not configured well, I did nothing about this. >> >> There might be a problem on both ends. IIRC LAVA doesn't enable ZMQ >> publisher by default. Even if it's enabled, it might be using > > > How can I enable ZMQ publisher ?
I'm not an expert on this. Please check LAVA docs. They're pretty verbose: https://master.lavasoftware.org/static/docs/v2/advanced-installation.html#in... https://master.lavasoftware.org/static/docs/v2/data-export.html#index-6
Thank you I configured my instance as needed.
Great. Could you now check whether automated fetching works when the jobs finish in LAVA?
Yes the results are fetched at the end of the job. I can see traces in logs that everything is going well. I now need to install squad on the production instance, but it should be ok :)
Thank you very much for your help, it would have been harder without it !
I'm glad you got it working. However it sounds like we need some step-by-step instructions. I'll create a github ticket to make sure it happens.
milosz
Regards, Axel
milosz
-
Axel Lebourhis -
Milosz Wasilewski