-----Original Message----- From: Theodore Y. Ts'o
On Thu, Oct 17, 2019 at 11:40:01PM +0000, Tim.Bird@sony.com wrote:
No. Well, the data might be provided at some time independent of the test compilation time, but it would not be made up on the fly. So the data might be provided at run time, but that shouldn't imply that the data is random, or that there is some lengthy fabrication process that happens at test execution time.
So how would the data be provided? Via a mounted file system? There is no mounted file system when we're running a kunit test. One of the reasons why kunit is fast is because we're not running init scripts, and we're not mounting a file system.
The fabrication process isn't really lengthy, though. If I modify fs/ext4/inode-test.c to add or remove a test, it takes:
Elapsed time: 2.672s total, 0.001s configuring, 2.554s building, 0.116s running
Compare and contrast this with running "kvm-xfstests -c 4k generic/001"
The actual time to run the test generic/001 is 3 seconds. But there is a 9 second overhead in starting the VM, for a total test time of 12 seconds. So sure, with kvm-xfstests I can drop a config file in /tmp/kvm-xfstests-tytso, which is mounted as /vtmp using 9p, so you could provide "user provided data" via a text file. But the overhead of starting up a full KVM, mounting a file system, starting userspace, etc., is 9 seconds.
Contrast this with 2.5 seconds to recompile and relink fs/ext4/inode-test.c into the kunit library. I wouldn't call that a "length fabrication process".
I'm not sure I understand your point here at all. I never said that compiling the code was a lengthy fabrication process. I said that I was NOT envisioning a lengthy fabrication process at runtime for the creation of the external data. Indeed, I wasn't envisioning fabricating the data at test runtime at all. I was trying to clarify that I didn't envision a human or fuzzer in the loop at test runtime, but apparently I didn't make that clear. The clarification was based on an assumption I made about what you were asking in your question. Maybe that assumption was bad.
Is it really worth it to add in some super-complex way to feed a data text file into a Kunit test, when editing the test file and rerunning the test really doesn't take that long?
In this case, the cost of parsing the data file does add some overhead, but it's not onerous. I'm not sure how, or whether, kunit handles the issue of reading data from a file at test time. But it doesn't have to be a file read. I'm just talking separating data from code.
It's not the cost of parsing the data file is how to *feed* the data file into the test file. How exactly are we supposed to do it? 9p? Some other mounted file system? That's where all the complexity and overhead is going to be.
Not necessarily. Maybe the root privilege example is not a good one. How about a test that probes the kernel config, and executes some variation of the tests based on the config values it detects?
But that's even easier. We can put "#ifdef CONFIG_xxx" into the fs/ext4/inode-test.c file. Again, it doesn't take that long to recompile and relink the test .c file.
Apologies, but this really seems like complexity in search of a problem....
We're just talking past each other. My original e-mail was a rebuttal to your assertion that any test that was data-driven or non-deterministic was a fuzzer. I still believe that's just not the case. This is independent of the mechanics or speed of how the data is input.
I also conceded (multiple times) that externally data-driven techniques are probably more aptly applied to non-unit tests. I've heard your pitch about speed, and I'm sympathetic. My point is that I believe there is a place for data-driven tests.
I can live with having failed to convince you, which I'll put down to my inadequacy to accurately communicate my ideas. :-)
Regards, -- Tim