On 10/17/19 6:08 AM, Theodore Y. Ts'o wrote:
On Wed, Oct 16, 2019 at 05:26:29PM -0600, Shuah Khan wrote:
I don't really buy the argument that unit tests should be deterministic Possibly, but I would opt for having the ability to feed test data.
I strongly believe that unit tests should be deterministic. Non-deterministic tests are essentially fuzz tests. And fuzz tests should be different from unit tests.
Having the ability to take test data doesn't make it non-deterministic though. It just means that if user wants to test with a different set of data, there is no need to recompile the test. This could be helpful to test cases the test write didn't think about.
You could make the data in this test the default and add ability to pass in data as needed.
We want unit tests to run quickly. Fuzz tests need to be run for a large number of passes (perhaps hours) in order to be sure that we've hit any possible bad cases. We want to be able to easily bisect fuzz tests --- preferably, automatically. And any kind of flakey test is hell to bisect.
Absolutely.
It's bad enough when a test is flakey because of the underlying code. But when a test is flakey because the test inputs are non-deterministic, it's even worse.
That is fine. You can achieve both by making the test data included in the test the default for deterministic behavior and allow users to supply another set of data.
thanks, -- Shuah