On Thu, 10 Mar 2011 11:09:21 +1300, Michael Hope michael.hope@linaro.org wrote:
* How frequently do problems get through review that are found by the test suite?
Ideally the test suite should have been run before the review and have shown no regressions. That's how upstream does it. GCC is complicated so a review will miss subtle things.
Sure, but reviews shouldn't be trying to catch every single problem that may exist.
If you don't have an idea of how much benefit test-before-merge has over just review-before-merge then it's going to be hard to estimate the risk of changing that scheme.
* How frequently do problems get past the testsuite?
Being pessamistic, say every second patch causes a failure found in the field not found in the testsuite.
That indicates that getting the code out there for (non-production) use would actually be worthwhile, rather than trying too hard to ensure everything goes through the testsuite. (Unless every patch has 10 problems that are not found in review but found by the testsuite.)
* Would you be confident in your ability to have quality releases if there wasn't a testsuite run of each change before it lands in trunk?
Not a quality release, no, but I have good confidence in the patches in general. It's unlikely that a commit would break trunk.
What sense are you using "break" here?
It sounds like prompt review->integration branch->beta testers | `->test suite->trunk
would be best if you don't think that you can have a quality release if you merge-before-test.
If you feel that you are equipped to recover from problems found by running the test suite against trunk before doing a release then it may be better to avoid the overhead and merge to trunk after prompt review, and deal with test failures when they happen.
Thanks,
James