On Mon, Oct 16, 2017 at 10:41:43PM +0000, Tom Gall wrote:
Hi All,
I’m looking at the 4.14-rc5 results. I think it’s important that we establish clear green baselines so we can detect regressions.
Universally I think we want to get stuff on skip lists and then get after those skip lists working on fixes. As that happens, we basically get those fixes back ported to 4.9 and 4.4.
+1.
I support a rather strict and aggressive policy of skipping flaky or failing tests, followed by the work to make root cause fixes and get them re-introduced, and new tests introduced, but only once they are provably stable. It is not wise to try to fix tests alongside trying to finding actual regressions - the test environment should be considered production/stable and dev work should happen outside it. I don't suggest that this policy be permanent, but I think it would be effective in the short and medium term to gain trust and eyeballs. This may mask legitimate problems for the time being, but it is more important to establish a stable 'baseline'.
Dan