Hi,
Yesterday during SQUAD sync we discussed strategies for marking baselines instead of having them marked automatically as a previous builds. This has been suggested before here, and here.
By default, every finished build becomes the baseline of the next finished build. Two alternatives to tell SQUAD not to mark a finished build as baseline are as follows (to be implemented): a) go to build's page > build settings > check 'ignore as baseline' option; anyone with write permissions to that build's project is allowed to change its settings; b) calling qa-reports.l.o/api/createbuild passing 'is_baseline=false' as POST before calling qa-reports.l.o/api/submitjob In absence of 'is_baseline', the build will be marked as baseline by default, for backwards compatibility.
Before discussing edge cases, take example below (considering option b above is being used to submit builds to Squad):
+------------+-----------------+-------------+ | build | is_baseline | baseline | +------------+---------------- +-------------+ | build101 | false | build100 | | build102 | false | build100 | | build103 | ----- | build100 | | build104 | ----- | build103 | | build105 | false | build104 | | build106 | false | build104 | +------------+-----------------+-------------+
1. If build104 is later manually marked as not being a baseline, through UI, then build103 automatically becomes the current baseline.
2. If in the event above, Squad will *NOT* re-run test comparisons for build106 and build105 against build103.
I think we discussed other edge cases, but I can't remember them now.
Charles
Hi Charles - thanks for the write-up. I use alternative baselines quite regularly. My thoughts below.
On Tue, Feb 12, 2019 at 10:59:52AM -0200, Charles Oliveira wrote:
Hi,
Yesterday during SQUAD sync we discussed strategies for marking baselines instead of having them marked automatically as a previous builds. This has been suggested before here, and here.
By default, every finished build becomes the baseline of the next finished build. Two alternatives to tell SQUAD not to mark a finished build as baseline are as follows (to be implemented): a) go to build's page > build settings > check 'ignore as baseline' option; anyone with write permissions to that build's project is allowed to change its settings; b) calling qa-reports.l.o/api/createbuild passing 'is_baseline=false' as POST before calling qa-reports.l.o/api/submitjob In absence of 'is_baseline', the build will be marked as baseline by default, for backwards compatibility.
I think this would work for LKFT. If building a tagged build, let is_baseline be true, otherwise set it to false.
Before discussing edge cases, take example below (considering option b above is being used to submit builds to Squad):
+------------+-----------------+-------------+ | build | is_baseline | baseline | +------------+---------------- +-------------+ | build101 | false | build100 | | build102 | false | build100 | | build103 | ----- | build100 | | build104 | ----- | build103 | | build105 | false | build104 | | build106 | false | build104 | +------------+-----------------+-------------+
- If build104 is later manually marked as not being a baseline,
through UI, then build103 automatically becomes the current baseline.
- If in the event above, Squad will *NOT* re-run test comparisons for
build106 and build105 against build103.
I think we discussed other edge cases, but I can't remember them now.
I'm curious what use-cases you have for this feature? At first I thought it wouldn't be useful for LKFT, but in fact I think it would really simplify what we're currently doing to do reporting.
I'm also curious how the UI will deal with it so it's clear which builds are being compared to determine fixes/regressions.
Seems like a reasonable approach to me :)
Thanks! Dan
Charles _______________________________________________ Squad-dev mailing list Squad-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/squad-dev
Hi Dan,
My comment below.
On Tue, 12 Feb 2019 at 18:54, Dan Rue dan.rue@linaro.org wrote:
Hi Charles - thanks for the write-up. I use alternative baselines quite regularly. My thoughts below.
On Tue, Feb 12, 2019 at 10:59:52AM -0200, Charles Oliveira wrote:
Hi,
Yesterday during SQUAD sync we discussed strategies for marking baselines instead of having them marked automatically as a previous builds. This has been suggested before here, and here.
By default, every finished build becomes the baseline of the next finished build. Two alternatives to tell SQUAD not to mark a finished build as baseline are as follows (to be implemented): a) go to build's page > build settings > check 'ignore as baseline' option; anyone with write permissions to that build's project is allowed to change its settings; b) calling qa-reports.l.o/api/createbuild passing 'is_baseline=false' as POST before calling qa-reports.l.o/api/submitjob In absence of 'is_baseline', the build will be marked as baseline by default, for backwards compatibility.
I think this would work for LKFT. If building a tagged build, let is_baseline be true, otherwise set it to false.
Just out of curiosity: how do you usually tag a build as baseline?
Before discussing edge cases, take example below (considering option b above is being used to submit builds to Squad):
+------------+-----------------+-------------+ | build | is_baseline | baseline | +------------+---------------- +-------------+ | build101 | false | build100 | | build102 | false | build100 | | build103 | ----- | build100 | | build104 | ----- | build103 | | build105 | false | build104 | | build106 | false | build104 | +------------+-----------------+-------------+
- If build104 is later manually marked as not being a baseline,
through UI, then build103 automatically becomes the current baseline.
- If in the event above, Squad will *NOT* re-run test comparisons for
build106 and build105 against build103.
I think we discussed other edge cases, but I can't remember them now.
I'm curious what use-cases you have for this feature? At first I thought it wouldn't be useful for LKFT, but in fact I think it would really simplify what we're currently doing to do reporting.
We got the suggestion from you a while ago: https://github.com/Linaro/squad/issues/276 :) Please keep in mind that this new feature will affect only projects that use alternative baselines, things will continue to work as always if builds are not flagged.
I'm also curious how the UI will deal with it so it's clear which builds are being compared to determine fixes/regressions.
We would have to add the baseline reference (a link?) to each build. For builds that already exist, the reference baseline would simply be the previous build.
Seems like a reasonable approach to me :)
Thanks! Dan
Charles _______________________________________________ Squad-dev mailing list Squad-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/squad-dev
On Wed, Feb 13, 2019 at 09:21:41AM -0200, Charles Oliveira wrote:
Hi Dan,
My comment below.
On Tue, 12 Feb 2019 at 18:54, Dan Rue dan.rue@linaro.org wrote:
Hi Charles - thanks for the write-up. I use alternative baselines quite regularly. My thoughts below.
On Tue, Feb 12, 2019 at 10:59:52AM -0200, Charles Oliveira wrote:
Hi,
Yesterday during SQUAD sync we discussed strategies for marking baselines instead of having them marked automatically as a previous builds. This has been suggested before here, and here.
By default, every finished build becomes the baseline of the next finished build. Two alternatives to tell SQUAD not to mark a finished build as baseline are as follows (to be implemented): a) go to build's page > build settings > check 'ignore as baseline' option; anyone with write permissions to that build's project is allowed to change its settings; b) calling qa-reports.l.o/api/createbuild passing 'is_baseline=false' as POST before calling qa-reports.l.o/api/submitjob In absence of 'is_baseline', the build will be marked as baseline by default, for backwards compatibility.
I think this would work for LKFT. If building a tagged build, let is_baseline be true, otherwise set it to false.
Just out of curiosity: how do you usually tag a build as baseline?
When I said tag, I meant the git version that we're testing was tagged, as opposed to a git version that was not yet tagged/released. Maybe instead of saying tag I'll say release: We want to compare with the most recent release, not the most recent version. We know a release from a release-candidate based on whether or not the commit under test has been tagged in git.
Before discussing edge cases, take example below (considering option b above is being used to submit builds to Squad):
+------------+-----------------+-------------+ | build | is_baseline | baseline | +------------+---------------- +-------------+ | build101 | false | build100 | | build102 | false | build100 | | build103 | ----- | build100 | | build104 | ----- | build103 | | build105 | false | build104 | | build106 | false | build104 | +------------+-----------------+-------------+
- If build104 is later manually marked as not being a baseline,
through UI, then build103 automatically becomes the current baseline.
- If in the event above, Squad will *NOT* re-run test comparisons for
build106 and build105 against build103.
I think we discussed other edge cases, but I can't remember them now.
I'm curious what use-cases you have for this feature? At first I thought it wouldn't be useful for LKFT, but in fact I think it would really simplify what we're currently doing to do reporting.
We got the suggestion from you a while ago: https://github.com/Linaro/squad/issues/276 :) Please keep in mind that this new feature will affect only projects that use alternative baselines, things will continue to work as always if builds are not flagged.
I'm also curious how the UI will deal with it so it's clear which builds are being compared to determine fixes/regressions.
We would have to add the baseline reference (a link?) to each build. For builds that already exist, the reference baseline would simply be the previous build.
Seems like a reasonable approach to me :)
Thanks! Dan
Charles _______________________________________________ Squad-dev mailing list Squad-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/squad-dev
Squad-dev mailing list Squad-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/squad-dev
On Tue, 12 Feb 2019 at 13:00, Charles Oliveira charles.oliveira@linaro.org wrote:
Hi,
Yesterday during SQUAD sync we discussed strategies for marking baselines instead of having them marked automatically as a previous builds. This has been suggested before here, and here.
By default, every finished build becomes the baseline of the next finished build. Two alternatives to tell SQUAD not to mark a finished build as baseline are as follows (to be implemented): a) go to build's page > build settings > check 'ignore as baseline' option; anyone with write permissions to that build's project is allowed to change its settings; b) calling qa-reports.l.o/api/createbuild passing 'is_baseline=false' as POST before calling qa-reports.l.o/api/submitjob In absence of 'is_baseline', the build will be marked as baseline by default, for backwards compatibility.
Before discussing edge cases, take example below (considering option b above is being used to submit builds to Squad):
+------------+-----------------+-------------+ | build | is_baseline | baseline | +------------+---------------- +-------------+ | build101 | false | build100 | | build102 | false | build100 | | build103 | ----- | build100 | | build104 | ----- | build103 | | build105 | false | build104 | | build106 | false | build104 | +------------+-----------------+-------------+
- If build104 is later manually marked as not being a baseline,
through UI, then build103 automatically becomes the current baseline.
- If in the event above, Squad will *NOT* re-run test comparisons for
build106 and build105 against build103.
I think we discussed other edge cases, but I can't remember them now.
There is one more tricky case that we didn't discuss in the meeting. When there is a data retention policy set on the project, there is just one build marked as baseline and it gets automatically deleted. Do we go back to calculating baselines with last finished build? Do sth else? This might be very confusing.
milosz
Charles _______________________________________________ Squad-dev mailing list Squad-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/squad-dev
Hi Milosz,
This is indeed tricky. I think what has been calculated should stay the same. Therefore build reference would be not found "404 - Build Deleted due to DRP". Either that or if the current baseline is a build queued for DRP, then that build would behave like having 'keep_data=True', saving it from DRP's cruelty. After current baseline changes to a newer build, DRP would remove the old build.
How does that sound?
drp = Data Rentention Policy
On Wed, 13 Feb 2019 at 08:31, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Tue, 12 Feb 2019 at 13:00, Charles Oliveira charles.oliveira@linaro.org wrote:
Hi,
Yesterday during SQUAD sync we discussed strategies for marking baselines instead of having them marked automatically as a previous builds. This has been suggested before here, and here.
By default, every finished build becomes the baseline of the next finished build. Two alternatives to tell SQUAD not to mark a finished build as baseline are as follows (to be implemented): a) go to build's page > build settings > check 'ignore as baseline' option; anyone with write permissions to that build's project is allowed to change its settings; b) calling qa-reports.l.o/api/createbuild passing 'is_baseline=false' as POST before calling qa-reports.l.o/api/submitjob In absence of 'is_baseline', the build will be marked as baseline by default, for backwards compatibility.
Before discussing edge cases, take example below (considering option b above is being used to submit builds to Squad):
+------------+-----------------+-------------+ | build | is_baseline | baseline | +------------+---------------- +-------------+ | build101 | false | build100 | | build102 | false | build100 | | build103 | ----- | build100 | | build104 | ----- | build103 | | build105 | false | build104 | | build106 | false | build104 | +------------+-----------------+-------------+
- If build104 is later manually marked as not being a baseline,
through UI, then build103 automatically becomes the current baseline.
- If in the event above, Squad will *NOT* re-run test comparisons for
build106 and build105 against build103.
I think we discussed other edge cases, but I can't remember them now.
There is one more tricky case that we didn't discuss in the meeting. When there is a data retention policy set on the project, there is just one build marked as baseline and it gets automatically deleted. Do we go back to calculating baselines with last finished build? Do sth else? This might be very confusing.
milosz
Charles _______________________________________________ Squad-dev mailing list Squad-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/squad-dev
On Wed, 13 Feb 2019 at 11:48, Charles Oliveira charles.oliveira@linaro.org wrote:
Hi Milosz,
This is indeed tricky. I think what has been calculated should stay the same. Therefore build reference would be not found "404 - Build Deleted due to DRP". Either that or if the current baseline is a build queued for DRP, then that build would behave like having 'keep_data=True', saving it from DRP's cruelty. After current baseline changes to a newer build, DRP would remove the old build.
How does that sound?
This sounds OK but might be a bit hard to implement. Please try and we'll see if there is any contradiction we have to deal with.
milosz
drp = Data Rentention Policy
On Wed, 13 Feb 2019 at 08:31, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Tue, 12 Feb 2019 at 13:00, Charles Oliveira charles.oliveira@linaro.org wrote:
Hi,
Yesterday during SQUAD sync we discussed strategies for marking baselines instead of having them marked automatically as a previous builds. This has been suggested before here, and here.
By default, every finished build becomes the baseline of the next finished build. Two alternatives to tell SQUAD not to mark a finished build as baseline are as follows (to be implemented): a) go to build's page > build settings > check 'ignore as baseline' option; anyone with write permissions to that build's project is allowed to change its settings; b) calling qa-reports.l.o/api/createbuild passing 'is_baseline=false' as POST before calling qa-reports.l.o/api/submitjob In absence of 'is_baseline', the build will be marked as baseline by default, for backwards compatibility.
Before discussing edge cases, take example below (considering option b above is being used to submit builds to Squad):
+------------+-----------------+-------------+ | build | is_baseline | baseline | +------------+---------------- +-------------+ | build101 | false | build100 | | build102 | false | build100 | | build103 | ----- | build100 | | build104 | ----- | build103 | | build105 | false | build104 | | build106 | false | build104 | +------------+-----------------+-------------+
- If build104 is later manually marked as not being a baseline,
through UI, then build103 automatically becomes the current baseline.
- If in the event above, Squad will *NOT* re-run test comparisons for
build106 and build105 against build103.
I think we discussed other edge cases, but I can't remember them now.
There is one more tricky case that we didn't discuss in the meeting. When there is a data retention policy set on the project, there is just one build marked as baseline and it gets automatically deleted. Do we go back to calculating baselines with last finished build? Do sth else? This might be very confusing.
milosz
Charles _______________________________________________ Squad-dev mailing list Squad-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/squad-dev
Squad-dev mailing list Squad-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/squad-dev
On Wed, Feb 13, 2019 at 09:47:46AM -0200, Charles Oliveira wrote:
Hi Milosz,
This is indeed tricky. I think what has been calculated should stay the same. Therefore build reference would be not found "404 - Build Deleted due to DRP". Either that or if the current baseline is a build queued for DRP, then that build would behave like having 'keep_data=True', saving it from DRP's cruelty. After current baseline changes to a newer build, DRP would remove the old build.
How does that sound?
IMO a build marked as a baseline should just be excluded from any data cleanup due to DRP.
-
Antonio Terceiro -
Charles Oliveira -
Dan Rue -
Milosz Wasilewski