On Tue, Oct 17, 2017 at 11:26:36AM -0500, Dan Rue wrote:
On Tue, Oct 17, 2017 at 03:39:15PM +0000, Neil Williams wrote:
On 17 October 2017 at 16:08, Greg KH <[1]gregkh@google.com> wrote:
On Tue, Oct 17, 2017 at 09:02:18AM -0500, Dan Rue wrote: > What I would like to see, and I don't know if it is even possible, is > something that actually measures test coverage based on code paths in > the linux kernel so that we have a means to actually measure our > effectiveness. If we knew we were testing 43% (number pulled out of thin > air) of the linux kernel code paths, then we would know what areas to > focus on to bring that number up, and we would know which subsystems to > have some confidence in, and which are uncovered. Please read: [2]http://blog.ploeh.dk/2015/11/16/code-coverage-is-a-useless-target-measure/ I worked with a team of developers over a decade ago trying to help with code-coverage analysis of the Linux kernel (many of those tests ended up in LTP). I'm pretty sure the ability is still there, but it turned out, in the end, that it means nothing at all. Heck, even when you turn on fun things like "fail kmalloc() X% of the time to exercise error paths", you still don't really test the overall system. So please, never think in terms of code coverage, but feature coverage, like what LTP is trying to accomplish, is a great metric to strive for.<snip> What we will also need is a map of which tests are stressing which features - so a sane metric for feature coverage inside and outside the kernel would be needed here.
++ ^^
From the blog post:
Some people use it to find areas where coverage is weak. There may be good reasons that some parts of a code base are sparsely covered by tests, but doing a manual inspection once in a while is a good idea. Perhaps you find that all is good, but you may also discover that a quality effort is overdue.The problem that I have is that I don't know where coverage is strong, or where it is weak. Before last week, if someone suggested adding a 'dhclient' test, I would have told them it is redundant. Now, I know that dhclient actually uses a different code path than both init and udhcpc. The only way I know to measure feature coverage is to look at the LTP tests that we're running, and which we're not, but that is a secondary measure.
Do you have a good suggestion for evaluating feature coverage? I don't disagree with your feedback, but it would be good to have some shared perspective on coverage analysis so that we can improve it strategically rather than based on gut feelings, or as a reaction to uncaught problems.
Start with LTP, we _know_ that is a good first step, combined with kselftests, to implement the basics like syscall functionality.
I also agree with Mark's response that my coverage suggestion is premature. This whole thread is premature. But it's also premature to bring in additional test suites at this time. Have to stabilize and expand on what we have, namely LTP.
Yes, it is premature, and you already have a long list of tests to add to the system after LTP is finally integrated (i.e. the list of 0-day tests). Only after you have that implemented should we start looking around for adding new tests (I have a list somewhere, but don't want to overwhelm you just yet...)
thanks,
greg k-h