On 08/21/2012 07:48 AM, Dave Pigott wrote:
On 21 Aug 2012, at 12:16, Alexander Sack wrote:
+android list
On Tue, Aug 21, 2012 at 12:41 PM, Dave Pigott dave.pigott@linaro.org wrote:
+1 to that, but in this case, it turns out that the android image we use for testing comes from snapshots, and that particular snapshot was retired yesterday. None of the releases seem stable enough to use for health checks so, for the moment, we're re-baslining on a new working snapshot (liuyq is working on this at the moment.)
For the future, we're discussing holding the health check images cached locally, so that we're not hampered by these issues again, and we can choose when and how to re-baseline.
Hmm. I would very much prefer if we could figure a way to use released images as our health-check base. Even if it means we do a "special" promotion of a certain daily build outside the monthly cadence if needed ...
After testing, I've baselined to the 12.07 android release for health checks. Every other board type takes its images from releases, so panda was an anomaly.
When we were trying to fix health check issues, we were having troubles finding a really good panda build. I think YongQin had found this one. 12.07 seems reasonable for now.
On this front: is there an easy way to check that a certain build is suitable for health check? If so, we could make that part of our monthly validation process and daily dashboard and assign priority to bugs that would disqualify a build from being suitable as a health check...
What do you think?
Hmm. Nice idea, and I was thinking along similar lines as we were going through this crisis. We need to think about this and how we might achieve it.
I had my 1x1 with asac and have a thought on this. Lets take the monthly releases and submit like 50 jobs for that image that work like a health job. We can then look at the results and see how it compares to the previous month's releases. This should help us know what's good and also provide some valuable metrics to the other Platform teams in Linaro.