On Wed, Feb 13, 2019 at 09:21:41AM -0200, Charles Oliveira wrote:
Hi Dan,
My comment below.
On Tue, 12 Feb 2019 at 18:54, Dan Rue dan.rue@linaro.org wrote:
Hi Charles - thanks for the write-up. I use alternative baselines quite regularly. My thoughts below.
On Tue, Feb 12, 2019 at 10:59:52AM -0200, Charles Oliveira wrote:
Hi,
Yesterday during SQUAD sync we discussed strategies for marking baselines instead of having them marked automatically as a previous builds. This has been suggested before here, and here.
By default, every finished build becomes the baseline of the next finished build. Two alternatives to tell SQUAD not to mark a finished build as baseline are as follows (to be implemented): a) go to build's page > build settings > check 'ignore as baseline' option; anyone with write permissions to that build's project is allowed to change its settings; b) calling qa-reports.l.o/api/createbuild passing 'is_baseline=false' as POST before calling qa-reports.l.o/api/submitjob In absence of 'is_baseline', the build will be marked as baseline by default, for backwards compatibility.
I think this would work for LKFT. If building a tagged build, let is_baseline be true, otherwise set it to false.
Just out of curiosity: how do you usually tag a build as baseline?
When I said tag, I meant the git version that we're testing was tagged, as opposed to a git version that was not yet tagged/released. Maybe instead of saying tag I'll say release: We want to compare with the most recent release, not the most recent version. We know a release from a release-candidate based on whether or not the commit under test has been tagged in git.
Before discussing edge cases, take example below (considering option b above is being used to submit builds to Squad):
+------------+-----------------+-------------+ | build | is_baseline | baseline | +------------+---------------- +-------------+ | build101 | false | build100 | | build102 | false | build100 | | build103 | ----- | build100 | | build104 | ----- | build103 | | build105 | false | build104 | | build106 | false | build104 | +------------+-----------------+-------------+
- If build104 is later manually marked as not being a baseline,
through UI, then build103 automatically becomes the current baseline.
- If in the event above, Squad will *NOT* re-run test comparisons for
build106 and build105 against build103.
I think we discussed other edge cases, but I can't remember them now.
I'm curious what use-cases you have for this feature? At first I thought it wouldn't be useful for LKFT, but in fact I think it would really simplify what we're currently doing to do reporting.
We got the suggestion from you a while ago: https://github.com/Linaro/squad/issues/276 :) Please keep in mind that this new feature will affect only projects that use alternative baselines, things will continue to work as always if builds are not flagged.
I'm also curious how the UI will deal with it so it's clear which builds are being compared to determine fixes/regressions.
We would have to add the baseline reference (a link?) to each build. For builds that already exist, the reference baseline would simply be the previous build.
Seems like a reasonable approach to me :)
Thanks! Dan
Charles _______________________________________________ Squad-dev mailing list Squad-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/squad-dev
Squad-dev mailing list Squad-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/squad-dev