On Tue, Oct 17, 2017 at 12:14:43AM +0000, Tom Gall wrote:
This past RC cycle I think we exposed a weakness in what we have in LKFT where the ability to execute some key functional stacks in the system to drive the kernel would probably be useful for validation.
The networking bug involving dhclient for example.
So what if we used either Debian, Gentoo or akin that has a mechanism that has as part of it’s packaging system a test target for each package. Simplest build a package, runs ‘make test’ (or akin) for some key packages that exercises parts of the system that should help tickle the kernel in interesting ways to tease out regressions.
Thus wouldn’t work on modest boards but the socionext or Juno boards could probably work fine.
I think we just need to iterate on the framework we have until we're stable for a period of time. We are presently running many tests that once trusted, will find regressions that nobody else notices in a timely manner. My biggest concern is trust - our results need to be rock solid, stable, so that they are trusted and so that people jump when there is a reported regression. Currently, that is not the case.
Once we reach that point, we will be able to carefully introduce additional tests strategically, based on gaps in our existing coverage. There are still many tests in LTP that we don't run, for example.
Regarding packaging tests - I think Neil covered it well. I would be surprised if there's value there, from a kernel testing perspective. I don't think introducing another OS is a good idea, unless we are actually considering dropping one.
What I would like to see, and I don't know if it is even possible, is something that actually measures test coverage based on code paths in the linux kernel so that we have a means to actually measure our effectiveness. If we knew we were testing 43% (number pulled out of thin air) of the linux kernel code paths, then we would know what areas to focus on to bring that number up, and we would know which subsystems to have some confidence in, and which are uncovered.
Dan