Hi everyone,
I'm using Bamboo to build new binaries. My goal is to setup my builder to trigger Squad to submit tests to LAVA using the latest binaries. This part is done but I still have a problem. If there are regressions with new binaries, Bamboo will still say the last build is good because it only performs curl request and that's it.
My question is, is there a way to get a feedback from Squad concerning tests results ? For example the number of fail/pass. This would be enough to me, I'd just have to write something to compare 2 builds.
Regards, Axel
On Mon, 21 Jan 2019 at 15:52, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Hi everyone,
I'm using Bamboo to build new binaries. My goal is to setup my builder to trigger Squad to submit tests to LAVA using the latest binaries. This part is done but I still have a problem. If there are regressions with new binaries, Bamboo will still say the last build is good because it only performs curl request and that's it.
My question is, is there a way to get a feedback from Squad concerning tests results ? For example the number of fail/pass. This would be enough to me, I'd just have to write something to compare 2 builds.
you already can compare 2 builds in SQUAD using 'report' API: https://squad.readthedocs.io/en/latest/api.html#builds-api-builds
This however will only be triggered by request, not by any event in SQUAD. Could you specify whether you would like to a) query squad about the data after you push all your results b) have squad trigger a notification on receiving all your results? The latter only sends email. We don't have any option for callback yet.
You can also use a badge: https://squad.readthedocs.io/en/latest/api.html#badges Example here (under test results header): https://lkft.linaro.org/
milosz
Regards, Axel _______________________________________________ Squad-dev mailing list Squad-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/squad-dev
On Mon, 21 Jan 2019 at 18:20, Milosz Wasilewski < milosz.wasilewski@linaro.org> wrote:
On Mon, 21 Jan 2019 at 15:52, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Hi everyone,
I'm using Bamboo to build new binaries. My goal is to setup my builder to trigger Squad to submit tests to LAVA using the latest binaries. This part is done but I still have a problem. If there are regressions
with
new binaries, Bamboo will still say the last build is good because it only performs
curl
request and that's it.
My question is, is there a way to get a feedback from Squad concerning tests results ? For example the number of fail/pass. This would be enough to me, I'd just have to write something to compare 2 builds.
you already can compare 2 builds in SQUAD using 'report' API: https://squad.readthedocs.io/en/latest/api.html#builds-api-builds
Hi Milosz,
I tested this API, it returns the url of the report, I guess this is what it is supposed to do. I opened the url returned, I get some output but it doesn't give me the results. Maybe I'm missing something ?
This however will only be triggered by request, not by any event in SQUAD. Could you specify whether you would like to a) query squad about the data after you push all your results b) have squad trigger a notification on receiving all your results? The latter only sends email. We don't have any option for callback yet.
My usecase would fit the first choice. I'd like to be able to query Squad and get data like number of pass/fail. With these data I would be able to mark my build as failed is there are regressions.
Regards, Axel
You can also use a badge: https://squad.readthedocs.io/en/latest/api.html#badges Example here (under test results header): https://lkft.linaro.org/
milosz
Regards, Axel _______________________________________________ Squad-dev mailing list Squad-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/squad-dev
On Tue, 22 Jan 2019 at 09:43, Axel Lebourhis axel.lebourhis@linaro.org wrote:
On Mon, 21 Jan 2019 at 18:20, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Mon, 21 Jan 2019 at 15:52, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Hi everyone,
I'm using Bamboo to build new binaries. My goal is to setup my builder to trigger Squad to submit tests to LAVA using the latest binaries. This part is done but I still have a problem. If there are regressions with new binaries, Bamboo will still say the last build is good because it only performs curl request and that's it.
My question is, is there a way to get a feedback from Squad concerning tests results ? For example the number of fail/pass. This would be enough to me, I'd just have to write something to compare 2 builds.
you already can compare 2 builds in SQUAD using 'report' API: https://squad.readthedocs.io/en/latest/api.html#builds-api-builds
Hi Milosz,
I tested this API, it returns the url of the report, I guess this is what it is supposed to do. I opened the url returned, I get some output but it doesn't give me the results. Maybe I'm missing something ?
It's expected. Report generating might be a lengthy process so the API call is non blocking. There are several options here: 1. poll the URL you got You need to check the field status_code. It follows HTTP status codes so you're looking for 200 in this field for properly generated report. If it's empty it means report is still in cooking. If it's sth else than 200 you should check error_message for details 2. setup callback URL SQUAD can issue HTTP request on the callback URL. It can also authorize itself with the token you provide. Some details in the docs: https://squad.readthedocs.io/en/latest/api.html#builds-api-builds 3. setup email notification for the report SQUAD will send the generated report to the email address provided (you can set only one email address)
Please note that report generation tasks are sent to different queue than all other tasks. So you need to setup your worker to be aware of both queues. In our instance it's like this: /bin/celery -A squad worker -Q celery,reporting_queue --max-tasks-per-child=5000 --max-memory-per-child=1500000 --loglevel INFO
This however will only be triggered by request, not by any event in SQUAD. Could you specify whether you would like to a) query squad about the data after you push all your results b) have squad trigger a notification on receiving all your results? The latter only sends email. We don't have any option for callback yet.
My usecase would fit the first choice. I'd like to be able to query Squad and get data like number of pass/fail. With these data I would be able to mark my build as failed is there are regressions.
api/builds/<id>/status should help you with this. There is boolean 'finished' field which indicates whether SQUAD considers the build completed or not.
milosz
Regards, Axel
You can also use a badge: https://squad.readthedocs.io/en/latest/api.html#badges Example here (under test results header): https://lkft.linaro.org/
milosz
Regards, Axel _______________________________________________ Squad-dev mailing list Squad-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/squad-dev
On Tue, 22 Jan 2019 at 11:38, Milosz Wasilewski < milosz.wasilewski@linaro.org> wrote:
On Tue, 22 Jan 2019 at 09:43, Axel Lebourhis axel.lebourhis@linaro.org wrote:
On Mon, 21 Jan 2019 at 18:20, Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
On Mon, 21 Jan 2019 at 15:52, Axel Lebourhis axel.lebourhis@linaro.org
wrote:
Hi everyone,
I'm using Bamboo to build new binaries. My goal is to setup my
builder to
trigger Squad to submit tests to LAVA using the latest binaries. This part is done but I still have a problem. If there are
regressions with
new binaries, Bamboo will still say the last build is good because it only performs
curl
request and that's it.
My question is, is there a way to get a feedback from Squad concerning tests results ? For example the number of fail/pass. This would be
enough
to me, I'd just have to write something to compare 2 builds.
you already can compare 2 builds in SQUAD using 'report' API: https://squad.readthedocs.io/en/latest/api.html#builds-api-builds
Hi Milosz,
I tested this API, it returns the url of the report, I guess this is
what it is supposed to do.
I opened the url returned, I get some output but it doesn't give me the
results.
Maybe I'm missing something ?
It's expected. Report generating might be a lengthy process so the API call is non blocking. There are several options here:
- poll the URL you got
You need to check the field status_code. It follows HTTP status codes so you're looking for 200 in this field for properly generated report. If it's empty it means report is still in cooking. If it's sth else than 200 you should check error_message for details
I checked and I get "status_code": null, "error_message": null Maybe it's related to the issue I submitted here : https://github.com/Linaro/squad/issues/457
- setup callback URL
SQUAD can issue HTTP request on the callback URL. It can also authorize itself with the token you provide. Some details in the docs: https://squad.readthedocs.io/en/latest/api.html#builds-api-builds
If I understand well, Squad will request the URL set in "callback" option when the report is ready ?
- setup email notification for the report
SQUAD will send the generated report to the email address provided (you can set only one email address)
This is one of my objectives but atm I don't know how to write email templates.
Please note that report generation tasks are sent to different queue than all other tasks. So you need to setup your worker to be aware of both queues. In our instance it's like this: /bin/celery -A squad worker -Q celery,reporting_queue --max-tasks-per-child=5000 --max-memory-per-child=1500000 --loglevel INFO
I use the same command, I took it from your qa-reports.linaro.org repo on
github.
This however will only be triggered by request, not by any event in SQUAD. Could you specify whether you would like to a) query squad about the data after you push all your results b) have squad trigger a notification on receiving all your results? The latter only sends email. We don't have any option for callback yet.
My usecase would fit the first choice. I'd like to be able to query
Squad and get data like number of pass/fail.
With these data I would be able to mark my build as failed is there are
regressions.
api/builds/<id>/status should help you with this. There is boolean 'finished' field which indicates whether SQUAD considers the build completed or not.
What makes Squad consider the build is completed ?
milosz
Regards, Axel
You can also use a badge: https://squad.readthedocs.io/en/latest/api.html#badges Example here (under test results header): https://lkft.linaro.org/
milosz
Regards, Axel _______________________________________________ Squad-dev mailing list Squad-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/squad-dev
On Tue, 22 Jan 2019 at 12:50, Axel Lebourhis axel.lebourhis@linaro.org wrote:
On Tue, 22 Jan 2019 at 11:38, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Tue, 22 Jan 2019 at 09:43, Axel Lebourhis axel.lebourhis@linaro.org wrote:
On Mon, 21 Jan 2019 at 18:20, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Mon, 21 Jan 2019 at 15:52, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Hi everyone,
I'm using Bamboo to build new binaries. My goal is to setup my builder to trigger Squad to submit tests to LAVA using the latest binaries. This part is done but I still have a problem. If there are regressions with new binaries, Bamboo will still say the last build is good because it only performs curl request and that's it.
My question is, is there a way to get a feedback from Squad concerning tests results ? For example the number of fail/pass. This would be enough to me, I'd just have to write something to compare 2 builds.
you already can compare 2 builds in SQUAD using 'report' API: https://squad.readthedocs.io/en/latest/api.html#builds-api-builds
Hi Milosz,
I tested this API, it returns the url of the report, I guess this is what it is supposed to do. I opened the url returned, I get some output but it doesn't give me the results. Maybe I'm missing something ?
It's expected. Report generating might be a lengthy process so the API call is non blocking. There are several options here:
- poll the URL you got
You need to check the field status_code. It follows HTTP status codes so you're looking for 200 in this field for properly generated report. If it's empty it means report is still in cooking. If it's sth else than 200 you should check error_message for details
I checked and I get "status_code": null, "error_message": null Maybe it's related to the issue I submitted here : https://github.com/Linaro/squad/issues/457
I'm not sure. Could you check if your worker consumes messages from reporting_queue?
- setup callback URL
SQUAD can issue HTTP request on the callback URL. It can also authorize itself with the token you provide. Some details in the docs: https://squad.readthedocs.io/en/latest/api.html#builds-api-builds
If I understand well, Squad will request the URL set in "callback" option when the report is ready ?
yes, correct
- setup email notification for the report
SQUAD will send the generated report to the email address provided (you can set only one email address)
This is one of my objectives but atm I don't know how to write email templates.
You don't need a new template for this to work. SQUAD will use it's default if there is no alternative set. The only thing is to provide email address to send report to. Of course SQUAD's email sending settings have to be correct as well: https://docs.djangoproject.com/en/2.1/topics/email/
Please note that report generation tasks are sent to different queue than all other tasks. So you need to setup your worker to be aware of both queues. In our instance it's like this: /bin/celery -A squad worker -Q celery,reporting_queue --max-tasks-per-child=5000 --max-memory-per-child=1500000 --loglevel INFO
I use the same command, I took it from your qa-reports.linaro.org repo on github.
that should work then. Could you check your message queue on the message broker to see if there are any messages pending?
milosz
This however will only be triggered by request, not by any event in SQUAD. Could you specify whether you would like to a) query squad about the data after you push all your results b) have squad trigger a notification on receiving all your results? The latter only sends email. We don't have any option for callback yet.
My usecase would fit the first choice. I'd like to be able to query Squad and get data like number of pass/fail. With these data I would be able to mark my build as failed is there are regressions.
api/builds/<id>/status should help you with this. There is boolean 'finished' field which indicates whether SQUAD considers the build completed or not.
What makes Squad consider the build is completed ?
milosz
Regards, Axel
You can also use a badge: https://squad.readthedocs.io/en/latest/api.html#badges Example here (under test results header): https://lkft.linaro.org/
milosz
Regards, Axel _______________________________________________ Squad-dev mailing list Squad-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/squad-dev
On Tue, 22 Jan 2019 at 13:56, Milosz Wasilewski < milosz.wasilewski@linaro.org> wrote:
On Tue, 22 Jan 2019 at 12:50, Axel Lebourhis axel.lebourhis@linaro.org wrote:
On Tue, 22 Jan 2019 at 11:38, Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
On Tue, 22 Jan 2019 at 09:43, Axel Lebourhis axel.lebourhis@linaro.org
wrote:
On Mon, 21 Jan 2019 at 18:20, Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
On Mon, 21 Jan 2019 at 15:52, Axel Lebourhis <
axel.lebourhis@linaro.org> wrote:
Hi everyone,
I'm using Bamboo to build new binaries. My goal is to setup my
builder to
trigger Squad to submit tests to LAVA using the latest binaries. This part is done but I still have a problem. If there are
regressions with
new binaries, Bamboo will still say the last build is good because it only
performs curl
request and that's it.
My question is, is there a way to get a feedback from Squad
concerning
tests results ? For example the number of fail/pass. This would be
enough
to me, I'd just have to write something to compare 2 builds.
you already can compare 2 builds in SQUAD using 'report' API: https://squad.readthedocs.io/en/latest/api.html#builds-api-builds
Hi Milosz,
I tested this API, it returns the url of the report, I guess this is
what it is supposed to do.
I opened the url returned, I get some output but it doesn't give me
the results.
Maybe I'm missing something ?
It's expected. Report generating might be a lengthy process so the API call is non blocking. There are several options here:
- poll the URL you got
You need to check the field status_code. It follows HTTP status codes so you're looking for 200 in this field for properly generated report. If it's empty it means report is still in cooking. If it's sth else than 200 you should check error_message for details
I checked and I get "status_code": null, "error_message": null Maybe it's related to the issue I submitted here :
https://github.com/Linaro/squad/issues/457
I'm not sure. Could you check if your worker consumes messages from reporting_queue?
What kind of message should I expect in the worker's logs to see that ? Do I need to do something in Squad to make the worker consume messages ?
- setup callback URL
SQUAD can issue HTTP request on the callback URL. It can also authorize itself with the token you provide. Some details in the docs: https://squad.readthedocs.io/en/latest/api.html#builds-api-builds
If I understand well, Squad will request the URL set in "callback"
option when the report is ready ?
yes, correct
- setup email notification for the report
SQUAD will send the generated report to the email address provided (you can set only one email address)
This is one of my objectives but atm I don't know how to write email
templates.
You don't need a new template for this to work. SQUAD will use it's default if there is no alternative set. The only thing is to provide email address to send report to. Of course SQUAD's email sending settings have to be correct as well: https://docs.djangoproject.com/en/2.1/topics/email/
Ok thanks
Please note that report generation tasks are sent to different queue than all other tasks. So you need to setup your worker to be aware of both queues. In our instance it's like this: /bin/celery -A squad worker -Q celery,reporting_queue --max-tasks-per-child=5000 --max-memory-per-child=1500000 --loglevel INFO
I use the same command, I took it from your qa-reports.linaro.org repo
on github.
that should work then. Could you check your message queue on the message broker to see if there are any messages pending?
Here is the output of "$ sudo rabbitmqctl list_queues" :
Timeout: 60.0 seconds ... Listing queues for vhost / ... celery@nxf42909-vm-debian-02.celery.pidbox 0 celeryev.5b8bec38-1e65-48e7-b0b5-948e65d6d3dc 0 celery 0 reporting_queue 0
Regards, Axel
milosz
This however will only be triggered by request, not by any event in SQUAD. Could you specify whether you would like to a) query squad about the data after you push all your results b) have squad trigger a notification on receiving all your results? The latter only sends email. We don't have any option for callback
yet.
My usecase would fit the first choice. I'd like to be able to query
Squad and get data like number of pass/fail.
With these data I would be able to mark my build as failed is there
are regressions.
api/builds/<id>/status should help you with this. There is boolean 'finished' field which indicates whether SQUAD considers the build completed or not.
What makes Squad consider the build is completed ?
milosz
Regards, Axel
You can also use a badge: https://squad.readthedocs.io/en/latest/api.html#badges Example here (under test results header): https://lkft.linaro.org/
milosz
Regards, Axel _______________________________________________ Squad-dev mailing list Squad-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/squad-dev
On Tue, 22 Jan 2019 at 13:22, Axel Lebourhis axel.lebourhis@linaro.org wrote:
On Tue, 22 Jan 2019 at 13:56, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Tue, 22 Jan 2019 at 12:50, Axel Lebourhis axel.lebourhis@linaro.org wrote:
On Tue, 22 Jan 2019 at 11:38, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Tue, 22 Jan 2019 at 09:43, Axel Lebourhis axel.lebourhis@linaro.org wrote:
On Mon, 21 Jan 2019 at 18:20, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Mon, 21 Jan 2019 at 15:52, Axel Lebourhis axel.lebourhis@linaro.org wrote: > > Hi everyone, > > > I'm using Bamboo to build new binaries. My goal is to setup my builder to > trigger > Squad to submit tests to LAVA using the latest binaries. > This part is done but I still have a problem. If there are regressions with > new binaries, > Bamboo will still say the last build is good because it only performs curl > request and that's it. > > My question is, is there a way to get a feedback from Squad concerning > tests results ? For example the number of fail/pass. This would be enough > to me, I'd just have to write > something to compare 2 builds.
you already can compare 2 builds in SQUAD using 'report' API: https://squad.readthedocs.io/en/latest/api.html#builds-api-builds
Hi Milosz,
I tested this API, it returns the url of the report, I guess this is what it is supposed to do. I opened the url returned, I get some output but it doesn't give me the results. Maybe I'm missing something ?
It's expected. Report generating might be a lengthy process so the API call is non blocking. There are several options here:
- poll the URL you got
You need to check the field status_code. It follows HTTP status codes so you're looking for 200 in this field for properly generated report. If it's empty it means report is still in cooking. If it's sth else than 200 you should check error_message for details
I checked and I get "status_code": null, "error_message": null Maybe it's related to the issue I submitted here : https://github.com/Linaro/squad/issues/457
I'm not sure. Could you check if your worker consumes messages from reporting_queue?
What kind of message should I expect in the worker's logs to see that ?
Should be sth like this Jan 12 07:53:26 qa-reports-worker-1 celery[20464]: [2019-01-12 07:53:26 +0000] [INFO] Received task: squad.core.tasks.prepare_report[fcb74150-b367-4579-a710-9ae9fce25c20] Jan 12 07:53:30 qa-reports-worker-1 celery[20464]: [2019-01-12 07:53:30 +0000] [INFO] Task squad.core.tasks.prepare_report[fcb74150-b367-4579-a710-9ae9fce25c20] succeeded in 4.461465198080987s: <DelayedReport: DelayedReport object>
squad.core.tasks.prepare_report is the task you're looking for.
Do I need to do something in Squad to make the worker consume messages ?
hmm, I don't think there is anything special needed. Worker should consume the messages on it's own.
- setup callback URL
SQUAD can issue HTTP request on the callback URL. It can also authorize itself with the token you provide. Some details in the docs: https://squad.readthedocs.io/en/latest/api.html#builds-api-builds
If I understand well, Squad will request the URL set in "callback" option when the report is ready ?
yes, correct
- setup email notification for the report
SQUAD will send the generated report to the email address provided (you can set only one email address)
This is one of my objectives but atm I don't know how to write email templates.
You don't need a new template for this to work. SQUAD will use it's default if there is no alternative set. The only thing is to provide email address to send report to. Of course SQUAD's email sending settings have to be correct as well: https://docs.djangoproject.com/en/2.1/topics/email/
Ok thanks
Please note that report generation tasks are sent to different queue than all other tasks. So you need to setup your worker to be aware of both queues. In our instance it's like this: /bin/celery -A squad worker -Q celery,reporting_queue --max-tasks-per-child=5000 --max-memory-per-child=1500000 --loglevel INFO
I use the same command, I took it from your qa-reports.linaro.org repo on github.
that should work then. Could you check your message queue on the message broker to see if there are any messages pending?
Here is the output of "$ sudo rabbitmqctl list_queues" :
Timeout: 60.0 seconds ... Listing queues for vhost / ... celery@nxf42909-vm-debian-02.celery.pidbox 0 celeryev.5b8bec38-1e65-48e7-b0b5-948e65d6d3dc 0 celery 0 reporting_queue 0
this looks about right. Nothing pending.
milosz
Regards, Axel
milosz
This however will only be triggered by request, not by any event in SQUAD. Could you specify whether you would like to a) query squad about the data after you push all your results b) have squad trigger a notification on receiving all your results? The latter only sends email. We don't have any option for callback yet.
My usecase would fit the first choice. I'd like to be able to query Squad and get data like number of pass/fail. With these data I would be able to mark my build as failed is there are regressions.
api/builds/<id>/status should help you with this. There is boolean 'finished' field which indicates whether SQUAD considers the build completed or not.
What makes Squad consider the build is completed ?
milosz
Regards, Axel
You can also use a badge: https://squad.readthedocs.io/en/latest/api.html#badges Example here (under test results header): https://lkft.linaro.org/
milosz
> > Regards, > Axel > _______________________________________________ > Squad-dev mailing list > Squad-dev@lists.linaro.org > https://lists.linaro.org/mailman/listinfo/squad-dev
-
Axel Lebourhis -
Milosz Wasilewski