On Fri, Oct 18, 2019 at 02:40:50AM +0000, Tim.Bird@sony.com wrote:
We're just talking past each other. My original e-mail was a rebuttal to your assertion that any test that was data-driven or non-deterministic was a fuzzer. I still believe that's just not the case. This is independent of the mechanics or speed of how the data is input.
Apologies, I was still focused on the original context of this thread, which was about suggested improvements to Iurii's ext4 kunit test, or perhaps adding new features to Kunit.
I also conceded (multiple times) that externally data-driven techniques are probably more aptly applied to non-unit tests. I've heard your pitch about speed, and I'm sympathetic. My point is that I believe there is a place for data-driven tests.
I guess I would put it differently. The key goal is it should be really easy for developers to run, create, and extend tests. Data-driven tests is certainly one technique to make it easier to extend tests, and indeed fs/ext4/inode-test.c is data-driven with the goal to make it easier to add additional tests.
Having the data for the test be external is certainly one option, and there will be cases where it will make sense. However, the overhead in creating the parser for the data, and additional complexity required to get the test data to be fed to the test program means that that benefits need to be pretty large in order to balance the additional costs of having an external data file, especially for Kunit.
In terms of the abstract question, is there a place for data-driven tests, I'm in complete agreement with you. I've used this many times personally, especially when writing tests which are implemented in terms of shell scripts. Examples of this include e2fsprogs's regression test suite and xfstests. I don't consider that a terribly interesting question though; I view that as on the same order as "is the sky blue?" or "are apple pies yummy?"
The more interesting, and more concrete question is whether there is a place for external data-driven tests in Kunit, and there I am *much* more skeptical. Sure, I could imagine adding some infrastructure where user-mode linux could read files from its "host" system. But there are those people who want to run KUnit tests by cross-compiling an ARM kernel and then shipping the resulting kernel to an test board, and then reading the output from the ARM board's serial console. In that case, we can't make the assumption that Kunit tests will be running under user-mode linux, so the complexity of how we get the external data file into the KUnit test just went up by another order of magnitude. It's going to be ***so*** much simpler if the test data is embedded in the kernel built by KUnit. That way, it works both in the ARCH=um case, and in the "build an ARM kernel and push it to a test board" case.
Cheers,
- Ted