On Tue, Oct 10, 2017 at 03:59:31PM +0100, Mark Brown wrote:
On Tue, Oct 10, 2017 at 04:43:09PM +0200, Greg KH wrote:
On Mon, Oct 09, 2017 at 10:30:05AM +0100, Milosz Wasilewski wrote:
Apart from that there were a few tests that didn't complete due to setup issues:
- LTP syscalls on juno (arm64) - problem with msgctl11 which is
omitted on other boards.
- LTP sched on x86 - running on NFS fails
- LTP timers on x15 (arm) - investigating the problem
When can we start to "trust" these results? Right now they are saying "no regressions" from previous tests, yet there were failures on a previous test from what I remember, so I don't know if this specific run of testing actually is any better or not.
I suspect we want to be showing the delta to some fixed baseline (ideally totally clean results!) rather than the previous test run, or including a full list of unexpected failures in the report. Otherwise any issue that doesn't get fixed immediately ends up getting hidden in the reporting which isn't ideal.
I agree.
And have you all tried breaking something (a build, a test, etc.) to see if it is caught by the system? Based on the last 4.9-rc1 release, I think something is still wrong in that area...
Clearly our stable upstream software presents too little challenge for automated test systems! :)
Heh.
Ok, the "why did it build" issue is now found out, you all didn't have the needed CONFIG option enabled. Which makes me ask, like I just did in public, can you all do better build testing for these arches? Like 'allmodconfig' or 'allyesconfig' or heck, a round of 'randconfig' would be good if possible...
thanks,
greg k-h