Thanks Zach, this makes it pretty clear what's currently expected to be run, and at what frequency. One thing I was thinking about, is that it might make sense to change to having a more distributed execution schedule.
What I mean by this, is that there are a lot of tests here, and I expect that to only grow over time. We should, of course, seek to automate as many as possible. And for the ones that can be automated and run in a reasonable amount of time, they can and should happen pretty much daily. I think there should be a set of tests that we consider "release critical" though. This sort of goes back to the star rating system that Alex has described in the past. The idea is that for release, the "release critical" tests would be the acceptance test. If they are all green, then we consider it release-worthy. Since we release so frequently, this would give us a small set of tests that could be quickly executed to give a green/red light on the release.
The rest of the tests are important too of course, but if we are not saying that we would block release if, for instance, angry birds fails to run properly, then why test it at release time? Instead, I'd like to see those tests dispersed through the rest of monthly cycle so that they are run on at least one weekly build. This way the tests would still be fully covered over the monthly cycle.
Thoughts?
On Mon, Mar 26, 2012 at 12:37 PM, Zach Pfeffer zach.pfeffer@linaro.orgwrote:
I've clarified what should be tested daily/weekly/monthly on Android builds. Take a look at:
https://docs.google.com/a/linaro.org/spreadsheet/ccc?key=0AkxwyUNxNaAadHpDRT...
I've marked tests with a d and w for daily and weekly. Anything without a mark will be done during the QA cycle.
As always the test descriptions can be found from a link on the test spreadsheets.
__Test Spreadsheets__
iMX53, iMX6
https://docs.google.com/a/linaro.org/spreadsheet/ccc?key=0AkxwyUNxNaAadDE1Ml...
Snowball
https://docs.google.com/a/linaro.org/spreadsheet/ccc?key=0AkxwyUNxNaAadEF1NX...
Origen
https://docs.google.com/a/linaro.org/spreadsheet/ccc?key=0AkxwyUNxNaAadDRDVl...
Panda
https://docs.google.com/a/linaro.org/spreadsheet/ccc?key=0AkxwyUNxNaAadGVWd3...
Vexpress
https://docs.google.com/a/linaro.org/spreadsheet/ccc?key=0AkxwyUNxNaAadExQdH...
-- Zach Pfeffer Android Platform Team Lead, Linaro Platform Teams Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog