Thanks Zach, this makes it pretty clear what's currently expected to be run, and at what frequency. One thing I was thinking about, is that it might make sense to change to having a more distributed execution schedule.
What I mean by this, is that there are a lot of tests here, and I expect that to only grow over time. We should, of course, seek to automate as many as possible. And for the ones that can be automated and run in a reasonable amount of time, they can and should happen pretty much daily. I think there should be a set of tests that we consider "release critical" though. This sort of goes back to the star rating system that Alex has described in the past. The idea is that for release, the "release critical" tests would be the acceptance test. If they are all green, then we consider it release-worthy. Since we release so frequently, this would give us a small set of tests that could be quickly executed to give a green/red light on the release.
The rest of the tests are important too of course, but if we are not saying that we would block release if, for instance, angry birds fails to run properly, then why test it at release time? Instead, I'd like to see those tests dispersed through the rest of monthly cycle so that they are run on at least one weekly build. This way the tests would still be fully covered over the monthly cycle.
Thoughts?