Hi!
Does anyone have ideas about crediting test authors or tests for bugs discovered? We increasingly see situations where someone adds a test then our subsystem CI uncovers a (1 in a 100 runs) bug using that test.
Using reported-by doesn't feel right. But credit should go to the person who wrote the test. Is anyone else having this dilemma?
On Fri, Jul 25, 2025 at 08:00:23AM -0700, Jakub Kicinski wrote:
Does anyone have ideas about crediting test authors or tests for bugs discovered? We increasingly see situations where someone adds a test then our subsystem CI uncovers a (1 in a 100 runs) bug using that test.
I just list the test that discovered the bug and which was cleared by the commit, and assume that the test author will get credited in the upstream test repository (e.g., such as generic/750 from fstests or inotify02 from ltp). So it's not something that I've worried about.
I suppose if the test repository isn't as well known, or if the test hasn't been checked anywhere at all, your concern that the test author should be credited is something I can understand. But it hasn't come up for me.
In other cases, if the commit hasn't been stable yet (say for kunit or kselftests coming from some other tree), I'll just throw in a Link: tag pointing at lore.kernel.org.
- Ted
Jakub Kicinski wrote:
Hi!
Does anyone have ideas about crediting test authors or tests for bugs discovered? We increasingly see situations where someone adds a test then our subsystem CI uncovers a (1 in a 100 runs) bug using that test.
Using reported-by doesn't feel right. But credit should go to the person who wrote the test. Is anyone else having this dilemma?
Is that not a "credit in the changelog" situation?
"Big thanks to DeveloperX for their recent TestY added with CommitZ for catching this case."
Reported-by: Some CI Bot
On Fri, 25 Jul 2025 09:38:28 -0700 dan.j.williams@intel.com wrote:
Jakub Kicinski wrote:
Hi!
Does anyone have ideas about crediting test authors or tests for bugs discovered? We increasingly see situations where someone adds a test then our subsystem CI uncovers a (1 in a 100 runs) bug using that test.
Using reported-by doesn't feel right. But credit should go to the person who wrote the test. Is anyone else having this dilemma?
Is that not a "credit in the changelog" situation?
"Big thanks to DeveloperX for their recent TestY added with CommitZ for catching this case."
That's what we do usually, but I'm a strong believer in (LWN) statistics to help people justify the work they do upstream. Feels even more important for testing than feature development in a way.
So a tag would be ideal. But it's a hard nut to crack. Best I can come up with would be:
Reproducer: test.case.path # 001122aabb (optimal) commit of the test case
? Could potentially be useful for backporters?
Reported-by: Some CI Bot
I guess we'd need something like:
Reported-by: subsystem CI # Person Who developed@the.test ?
Jakub Kicinski wrote:
On Fri, 25 Jul 2025 09:38:28 -0700 dan.j.williams@intel.com wrote:
Jakub Kicinski wrote:
Hi!
Does anyone have ideas about crediting test authors or tests for bugs discovered? We increasingly see situations where someone adds a test then our subsystem CI uncovers a (1 in a 100 runs) bug using that test.
Using reported-by doesn't feel right. But credit should go to the person who wrote the test. Is anyone else having this dilemma?
Is that not a "credit in the changelog" situation?
"Big thanks to DeveloperX for their recent TestY added with CommitZ for catching this case."
That's what we do usually, but I'm a strong believer in (LWN) statistics to help people justify the work they do upstream. Feels even more important for testing than feature development in a way.
So a tag would be ideal. But it's a hard nut to crack. Best I can come up with would be:
Reproducer: test.case.path # 001122aabb (optimal) commit of the test case
? Could potentially be useful for backporters?
That's true, more than a few times I have had distro folks reach out to ask "how do I verify this backport" and end up manually pointing to the new unit test that backstops a fix.
Although, from that tag I would not know where to get the commit. Maybe:
Test: <git url>
...as a new Link: type?
Reported-by: Some CI Bot
I guess we'd need something like:
Reported-by: subsystem CI # Person Who developed@the.test ?
That looks like a useful convention so that the statistics gathering script does not need to walk URLs to get author data.
Also:
Tested-by: validation person # test author
...might be another convention, because the validator likely has interest in getting Cc'd on backports, while the tool author likely wants the credit but not all the notifications on what happens with fixes.
On Fri, Jul 25, 2025 at 08:00:23AM -0700, Jakub Kicinski wrote:
Does anyone have ideas about crediting test authors or tests for bugs discovered? We increasingly see situations where someone adds a test then our subsystem CI uncovers a (1 in a 100 runs) bug using that test.
Using reported-by doesn't feel right. But credit should go to the person who wrote the test. Is anyone else having this dilemma?
Usually I'd do a reported-by for whoever actually looked at the test system, triaged the issue and reported it. Trying to credit test authorship separately to the testsuite gets cumbersome over time, tests get updated over time for a range of reasons (toolchain updates, adding more coverage, improvements in the testsuite's frameworks...) so it's often not just a single person. Hopefully the testsuite is keeping track of things well enough so mentioning the test will point people in the right direction.
On Fri, 25 Jul 2025 17:39:12 +0100 Mark Brown wrote:
On Fri, Jul 25, 2025 at 08:00:23AM -0700, Jakub Kicinski wrote:
Does anyone have ideas about crediting test authors or tests for bugs discovered? We increasingly see situations where someone adds a test then our subsystem CI uncovers a (1 in a 100 runs) bug using that test.
Using reported-by doesn't feel right. But credit should go to the person who wrote the test. Is anyone else having this dilemma?
Usually I'd do a reported-by for whoever actually looked at the test system, triaged the issue and reported it. Trying to credit test authorship separately to the testsuite gets cumbersome over time, tests get updated over time for a range of reasons (toolchain updates, adding more coverage, improvements in the testsuite's frameworks...) so it's often not just a single person. Hopefully the testsuite is keeping track of things well enough so mentioning the test will point people in the right direction.
Ack, it does get murky overtime. Also with pre-commit testing there usually wouldn't even be a bug in the tree to credit fixing.
I guess we just had a lucky(?) string of very clear cut cases where a good selftest led the maintainer noticing a crash in the CI, and fixing something.
linux-kselftest-mirror@lists.linaro.org
-
dan.j.williams＠intel.com -
Jakub Kicinski -
Mark Brown -
Theodore Ts'o