Hi all,
I have an issue which is certainly related to bad configuration. I run xTS test job with LAVA to test Android images, so it makes a lot of results for SQUAD to process and store (around 500k per build). My db backups start to take some space, so I want to clean old results. But when I try to delete a build, the connection is reset by SQUAD. I guess this is due to worker timeout, so how can I fix this ?
Best regards, Axel
On Tue, 3 Sep 2019 at 08:46, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Hi all,
I have an issue which is certainly related to bad configuration. I run xTS test job with LAVA to test Android images, so it makes a lot of results for SQUAD to process and store (around 500k per build). My db backups start to take some space, so I want to clean old results. But when I try to delete a build, the connection is reset by SQUAD. I guess this is due to worker timeout, so how can I fix this ?
This sounds familiar. Did you try to delete the results from web admin interface? It will likely time out as query takes a lot of time. There is one or possibly two options you can use: 1. delete old data from django shell This works and I used this approach before. Depending on the level you want to delete the data it might look sth like this squad-admin shell $> from squad.core.models import Build $> b = Build.objects.get(version="<your version goes here>", project__slug="<project slug goes here>") # this should return only 1 build $> b.delete() This will perform cascade delete on all results related to this build, but it won't remove test jobs (I'm not 100% about that). So you might need to do a similar thing for the test jobs separately.
2. delete old data using REST API I'm not 100% sure this is possible and I never tried that.
milosz
Best regards, Axel _______________________________________________ Squad-dev mailing list Squad-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/squad-dev
Hi Milosz,
Your first solution works well, thanks for that !
The second solution, using the REST API, is described as not recommended in the documentation, so I think I'll stick with the first one.
Best regards, Axel
On Tue, 3 Sep 2019 at 11:11, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Tue, 3 Sep 2019 at 08:46, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Hi all,
I have an issue which is certainly related to bad configuration. I run xTS test job with LAVA to test Android images, so it makes a lot of results for SQUAD to process and store (around 500k per build). My db backups start to take some space, so I want to clean old results. But
when
I try to delete a build, the connection is reset by SQUAD. I guess this
is
due to worker timeout, so how can I fix this ?
This sounds familiar. Did you try to delete the results from web admin interface? It will likely time out as query takes a lot of time. There is one or possibly two options you can use:
- delete old data from django shell
This works and I used this approach before. Depending on the level you want to delete the data it might look sth like this squad-admin shell $> from squad.core.models import Build $> b = Build.objects.get(version="<your version goes here>", project__slug="<project slug goes here>") # this should return only 1 build $> b.delete() This will perform cascade delete on all results related to this build, but it won't remove test jobs (I'm not 100% about that). So you might need to do a similar thing for the test jobs separately.
- delete old data using REST API
I'm not 100% sure this is possible and I never tried that.
milosz
Best regards, Axel _______________________________________________ Squad-dev mailing list Squad-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/squad-dev
On Tue, 3 Sep 2019 at 13:20, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Hi Milosz,
Your first solution works well, thanks for that !
The second solution, using the REST API, is described as not recommended in the documentation, so I think I'll stick with the first one.
I don't know if DELETE is enabled in REST API. But if it is, it would do the same thing as the delete() from shell.
milosz
Best regards, Axel
On Tue, 3 Sep 2019 at 11:11, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Tue, 3 Sep 2019 at 08:46, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Hi all,
I have an issue which is certainly related to bad configuration. I run xTS test job with LAVA to test Android images, so it makes a lot of results for SQUAD to process and store (around 500k per build). My db backups start to take some space, so I want to clean old results. But when I try to delete a build, the connection is reset by SQUAD. I guess this is due to worker timeout, so how can I fix this ?
This sounds familiar. Did you try to delete the results from web admin interface? It will likely time out as query takes a lot of time. There is one or possibly two options you can use:
- delete old data from django shell
This works and I used this approach before. Depending on the level you want to delete the data it might look sth like this squad-admin shell $> from squad.core.models import Build $> b = Build.objects.get(version="<your version goes here>", project__slug="<project slug goes here>") # this should return only 1 build $> b.delete() This will perform cascade delete on all results related to this build, but it won't remove test jobs (I'm not 100% about that). So you might need to do a similar thing for the test jobs separately.
- delete old data using REST API
I'm not 100% sure this is possible and I never tried that.
milosz
Best regards, Axel _______________________________________________ Squad-dev mailing list Squad-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/squad-dev
Hi Alex,
I don't know how often you're backing up your db, but SQUAD comes with a built-in feature that automatically deletes builds older than N number of days. It's located at `/<yourgroup>/<yourproject>/settings`
"Data retention days*". Default is 0 (never delete), so setting this
to N should delete your builds after N days of build creation. Maybe this can help in the future.
Charles
On Tue, 3 Sep 2019 at 09:42, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Tue, 3 Sep 2019 at 13:20, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Hi Milosz,
Your first solution works well, thanks for that !
The second solution, using the REST API, is described as not recommended in the documentation, so I think I'll stick with the first one.
I don't know if DELETE is enabled in REST API. But if it is, it would do the same thing as the delete() from shell.
milosz
Best regards, Axel
On Tue, 3 Sep 2019 at 11:11, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Tue, 3 Sep 2019 at 08:46, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Hi all,
I have an issue which is certainly related to bad configuration. I run xTS test job with LAVA to test Android images, so it makes a lot of results for SQUAD to process and store (around 500k per build). My db backups start to take some space, so I want to clean old results. But when I try to delete a build, the connection is reset by SQUAD. I guess this is due to worker timeout, so how can I fix this ?
This sounds familiar. Did you try to delete the results from web admin interface? It will likely time out as query takes a lot of time. There is one or possibly two options you can use:
- delete old data from django shell
This works and I used this approach before. Depending on the level you want to delete the data it might look sth like this squad-admin shell $> from squad.core.models import Build $> b = Build.objects.get(version="<your version goes here>", project__slug="<project slug goes here>") # this should return only 1 build $> b.delete() This will perform cascade delete on all results related to this build, but it won't remove test jobs (I'm not 100% about that). So you might need to do a similar thing for the test jobs separately.
- delete old data using REST API
I'm not 100% sure this is possible and I never tried that.
milosz
Best regards, Axel _______________________________________________ Squad-dev mailing list Squad-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/squad-dev
Squad-dev mailing list Squad-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/squad-dev
Hi Charles,
Yes I know this feature, but I don't use it as the need for some build results is "variable", so it's hard to find the perfect setting for data retention. So by default I keep all builds without a limit of time.
Best regards, Axel
On Tue, 3 Sep 2019 at 14:50, Charles Oliveira charles.oliveira@linaro.org wrote:
Hi Alex,
I don't know how often you're backing up your db, but SQUAD comes with a built-in feature that automatically deletes builds older than N number of days. It's located at `/<yourgroup>/<yourproject>/settings`
"Data retention days*". Default is 0 (never delete), so setting this
to N should delete your builds after N days of build creation. Maybe this can help in the future.
Charles
On Tue, 3 Sep 2019 at 09:42, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Tue, 3 Sep 2019 at 13:20, Axel Lebourhis axel.lebourhis@linaro.org
wrote:
Hi Milosz,
Your first solution works well, thanks for that !
The second solution, using the REST API, is described as not
recommended in the documentation, so I think I'll stick with the first one.
I don't know if DELETE is enabled in REST API. But if it is, it would do the same thing as the delete() from shell.
milosz
Best regards, Axel
On Tue, 3 Sep 2019 at 11:11, Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
On Tue, 3 Sep 2019 at 08:46, Axel Lebourhis <
axel.lebourhis@linaro.org> wrote:
Hi all,
I have an issue which is certainly related to bad configuration. I run xTS test job with LAVA to test Android images, so it makes a
lot of
results for SQUAD to process and store (around 500k per build). My
db
backups start to take some space, so I want to clean old results.
But when
I try to delete a build, the connection is reset by SQUAD. I guess
this is
due to worker timeout, so how can I fix this ?
This sounds familiar. Did you try to delete the results from web admin interface? It will likely time out as query takes a lot of time. There is one or possibly two options you can use:
- delete old data from django shell
This works and I used this approach before. Depending on the level you want to delete the data it might look sth like this squad-admin shell $> from squad.core.models import Build $> b = Build.objects.get(version="<your version goes here>", project__slug="<project slug goes here>") # this should return only 1 build $> b.delete() This will perform cascade delete on all results related to this build, but it won't remove test jobs (I'm not 100% about that). So you might need to do a similar thing for the test jobs separately.
- delete old data using REST API
I'm not 100% sure this is possible and I never tried that.
milosz
Best regards, Axel _______________________________________________ Squad-dev mailing list Squad-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/squad-dev
Squad-dev mailing list Squad-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/squad-dev
-
Axel Lebourhis -
Charles Oliveira -
Milosz Wasilewski