-----Original Message----- From: Brendan Higgins
On Thu, Oct 17, 2019 at 3:25 PM Tim.Bird@sony.com wrote:
-----Original Message----- From: Theodore Y. Ts'o on October 17, 2019 2:09 AM
On Wed, Oct 16, 2019 at 05:26:29PM -0600, Shuah Khan wrote:
I don't really buy the argument that unit tests should be deterministic Possibly, but I would opt for having the ability to feed test data.
I strongly believe that unit tests should be deterministic. Non-deterministic tests are essentially fuzz tests. And fuzz tests should be different from unit tests.
I'm not sure I have the entire context here, but I think deterministic might not be the right word, or it might not capture the exact meaning intended.
I think there are multiple issues here:
- Does the test enclose all its data, including working data and expected
results?
Or, does the test allow someone to provide working data? This alternative implies that either the some of testcases or the results might be different
depending on
the data that is provided. IMHO the test would be deterministic if it always
produced
the same results based on the same data inputs. And if the input data was
deterministic.
I would call this a data-driven test.
Since the results would be dependent on the data provided, the results from tests using different data would not be comparable. Essentially, changing the input data changes the test so maybe it's best to consider this a different test. Like 'test-with-data-A' and 'test-with-data-B'.
That kind of sound like parameterized tests[1];
...
Both Iurii and you pointed me at parameterized tests. This sounds like what I am talking about. We have a lot of infrastructure for these concepts in Fuego, but it's called something different.
I'm not sure if I should feel smart for having developed some of these concepts independently or dumb for not having known about this (apparently) standard nomenclature. ;-)
Thanks for the reference! -- Tim