On Thu, Mar 10, 2011 at 11:36 AM, James Westby james.westby@canonical.com wrote:
On Thu, 10 Mar 2011 11:09:21 +1300, Michael Hope michael.hope@linaro.org wrote:
* How frequently do problems get through review that are found by the test suite?
Ideally the test suite should have been run before the review and have shown no regressions. That's how upstream does it. GCC is complicated so a review will miss subtle things.
Sure, but reviews shouldn't be trying to catch every single problem that may exist.
If you don't have an idea of how much benefit test-before-merge has over just review-before-merge then it's going to be hard to estimate the risk of changing that scheme.
We currently have review+build+test before merge. If we checkin directly to trunk then it would be review+merge, then build+test automatically afterwards.
* How frequently do problems get past the testsuite?
Being pessamistic, say every second patch causes a failure found in the field not found in the testsuite.
That indicates that getting the code out there for (non-production) use would actually be worthwhile, rather than trying too hard to ensure everything goes through the testsuite. (Unless every patch has 10 problems that are not found in review but found by the testsuite.)
Yip. It's quite stable and bugs are quite rare. A huge testsuite like Ubuntu is a good way of exercising the compiler.
* Would you be confident in your ability to have quality releases if there wasn't a testsuite run of each change before it lands in trunk?
Not a quality release, no, but I have good confidence in the patches in general. It's unlikely that a commit would break trunk.
What sense are you using "break" here?
On second reading, too loosely. I meant 'fail to build', but most developers will want to run the testsuite. If the checkin before them caused a testsuite regression then they may get confused.
A timely automatic build would help with that: perhaps notify the previous author of the regression, and give the developer somewhere to easily check the state of their baseline.
It sounds like prompt review->integration branch->beta testers | `->test suite->trunk
would be best if you don't think that you can have a quality release if you merge-before-test.
Yip. The integration branch however may be Ubuntu itself. Matthias is fairly happy cherry-picking things so if these checkins/feature branches are at least reviewed and preferably build+tested then he can be the path to beta testers.
This may change when Linaro gets set up to build main.
If you feel that you are equipped to recover from problems found by running the test suite against trunk before doing a release then it may be better to avoid the overhead and merge to trunk after prompt review, and deal with test failures when they happen.
Agreed. The build+testsuite can happen automatically and probably within two days.
-- Michae