On Tue, Oct 10, 2017 at 04:43:09PM +0200, Greg KH wrote:
On Mon, Oct 09, 2017 at 10:30:05AM +0100, Milosz Wasilewski wrote:
Apart from that there were a few tests that didn't complete due to setup issues:
- LTP syscalls on juno (arm64) - problem with msgctl11 which is
omitted on other boards.
- LTP sched on x86 - running on NFS fails
- LTP timers on x15 (arm) - investigating the problem
When can we start to "trust" these results? Right now they are saying "no regressions" from previous tests, yet there were failures on a previous test from what I remember, so I don't know if this specific run of testing actually is any better or not.
I suspect we want to be showing the delta to some fixed baseline (ideally totally clean results!) rather than the previous test run, or including a full list of unexpected failures in the report. Otherwise any issue that doesn't get fixed immediately ends up getting hidden in the reporting which isn't ideal.
And have you all tried breaking something (a build, a test, etc.) to see if it is caught by the system? Based on the last 4.9-rc1 release, I think something is still wrong in that area...
Clearly our stable upstream software presents too little challenge for automated test systems! :)