Hi!
Does anyone have ideas about crediting test authors or tests for bugs discovered? We increasingly see situations where someone adds a test then our subsystem CI uncovers a (1 in a 100 runs) bug using that test.
Using reported-by doesn't feel right. But credit should go to the person who wrote the test. Is anyone else having this dilemma?
On Fri, Jul 25, 2025 at 08:00:23AM -0700, Jakub Kicinski wrote:
Does anyone have ideas about crediting test authors or tests for bugs discovered? We increasingly see situations where someone adds a test then our subsystem CI uncovers a (1 in a 100 runs) bug using that test.
I just list the test that discovered the bug and which was cleared by the commit, and assume that the test author will get credited in the upstream test repository (e.g., such as generic/750 from fstests or inotify02 from ltp). So it's not something that I've worried about.
I suppose if the test repository isn't as well known, or if the test hasn't been checked anywhere at all, your concern that the test author should be credited is something I can understand. But it hasn't come up for me.
In other cases, if the commit hasn't been stable yet (say for kunit or kselftests coming from some other tree), I'll just throw in a Link: tag pointing at lore.kernel.org.
- Ted
Jakub Kicinski wrote:
Hi!
Does anyone have ideas about crediting test authors or tests for bugs discovered? We increasingly see situations where someone adds a test then our subsystem CI uncovers a (1 in a 100 runs) bug using that test.
Using reported-by doesn't feel right. But credit should go to the person who wrote the test. Is anyone else having this dilemma?
Is that not a "credit in the changelog" situation?
"Big thanks to DeveloperX for their recent TestY added with CommitZ for catching this case."
Reported-by: Some CI Bot
On Fri, 25 Jul 2025 09:38:28 -0700 dan.j.williams@intel.com wrote:
Jakub Kicinski wrote:
Hi!
Does anyone have ideas about crediting test authors or tests for bugs discovered? We increasingly see situations where someone adds a test then our subsystem CI uncovers a (1 in a 100 runs) bug using that test.
Using reported-by doesn't feel right. But credit should go to the person who wrote the test. Is anyone else having this dilemma?
Is that not a "credit in the changelog" situation?
"Big thanks to DeveloperX for their recent TestY added with CommitZ for catching this case."
That's what we do usually, but I'm a strong believer in (LWN) statistics to help people justify the work they do upstream. Feels even more important for testing than feature development in a way.
So a tag would be ideal. But it's a hard nut to crack. Best I can come up with would be:
Reproducer: test.case.path # 001122aabb (optimal) commit of the test case
? Could potentially be useful for backporters?
Reported-by: Some CI Bot
I guess we'd need something like:
Reported-by: subsystem CI # Person Who developed@the.test ?
Jakub Kicinski wrote:
On Fri, 25 Jul 2025 09:38:28 -0700 dan.j.williams@intel.com wrote:
Jakub Kicinski wrote:
Hi!
Does anyone have ideas about crediting test authors or tests for bugs discovered? We increasingly see situations where someone adds a test then our subsystem CI uncovers a (1 in a 100 runs) bug using that test.
Using reported-by doesn't feel right. But credit should go to the person who wrote the test. Is anyone else having this dilemma?
Is that not a "credit in the changelog" situation?
"Big thanks to DeveloperX for their recent TestY added with CommitZ for catching this case."
That's what we do usually, but I'm a strong believer in (LWN) statistics to help people justify the work they do upstream. Feels even more important for testing than feature development in a way.
So a tag would be ideal. But it's a hard nut to crack. Best I can come up with would be:
Reproducer: test.case.path # 001122aabb (optimal) commit of the test case
? Could potentially be useful for backporters?
That's true, more than a few times I have had distro folks reach out to ask "how do I verify this backport" and end up manually pointing to the new unit test that backstops a fix.
Although, from that tag I would not know where to get the commit. Maybe:
Test: <git url>
...as a new Link: type?
Reported-by: Some CI Bot
I guess we'd need something like:
Reported-by: subsystem CI # Person Who developed@the.test ?
That looks like a useful convention so that the statistics gathering script does not need to walk URLs to get author data.
Also:
Tested-by: validation person # test author
...might be another convention, because the validator likely has interest in getting Cc'd on backports, while the tool author likely wants the credit but not all the notifications on what happens with fixes.
On Fri, Jul 25, 2025 at 11:37:13AM -0700, dan.j.williams@intel.com wrote:
Jakub Kicinski wrote:
So a tag would be ideal. But it's a hard nut to crack. Best I can come up with would be:
Reproducer: test.case.path # 001122aabb (optimal) commit of the test case
That's true, more than a few times I have had distro folks reach out to ask "how do I verify this backport" and end up manually pointing to the new unit test that backstops a fix.
Although, from that tag I would not know where to get the commit. Maybe:
Test: <git url>
...as a new Link: type?
It seems like there's some overlap here with the work that people have been intermittently trying to do on test cataloging, eg:
https://lore.kernel.org/workflows/CAK18DXYitS7hL1mA3QsPLmW9-R0q6Kin0C5Uv9fj=...
That's been approached more from the "what tests should I run?" end of things since it's been driven by people interested in testing and CI, but it feels like there's a lot of overlap with the describing the suites part of things. It'd be a lot easier to write and read tags like the above if we could define some more compact names than git URLs for suites/tests.
Hello,
---- On Sat, 26 Jul 2025 18:44:29 -0300 Mark Brown broonie@kernel.org wrote ---
On Fri, Jul 25, 2025 at 11:37:13AM -0700, dan.j.williams@intel.com wrote:
Jakub Kicinski wrote:
So a tag would be ideal. But it's a hard nut to crack. Best I can come up with would be:
Reproducer: test.case.path # 001122aabb (optimal) commit of the test case
That's true, more than a few times I have had distro folks reach out to ask "how do I verify this backport" and end up manually pointing to the new unit test that backstops a fix.
Although, from that tag I would not know where to get the commit. Maybe:
Test: <git url>
...as a new Link: type?
It seems like there's some overlap here with the work that people have been intermittently trying to do on test cataloging, eg: https://lore.kernel.org/workflows/CAK18DXYitS7hL1mA3QsPLmW9-R0q6Kin0C5Uv9fj=... That's been approached more from the "what tests should I run?" end of things since it's been driven by people interested in testing and CI, but it feels like there's a lot of overlap with the describing the suites part of things. It'd be a lot easier to write and read tags like the above if we could define some more compact names than git URLs for suites/tests.
I see the overlap too. The catalog discussion envisions a mapping of which tests should be executed for this folder/file or function. This approach is being used, for example, in the Mesa project for its CI testing. When a new PR comes in, the system will trigger tests based on the files being modified.
Our discussions on the catalog side are quite basic right now and happening through the .kernelci.yml file[1]. I believe there is a possible future, built in a step by step manner, to indentify for given patchset: * configs to test * arch/hw to run tests on * tests must be executed * expectation of pass/fail for each test
KernelCI wants to work with maintainers to figure out how to make progress on that.
[1] https://www.youtube.com/watch?v=-LtK1fScFww
Best,
- Gus
On Fri, Jul 25, 2025 at 08:00:23AM -0700, Jakub Kicinski wrote:
Does anyone have ideas about crediting test authors or tests for bugs discovered? We increasingly see situations where someone adds a test then our subsystem CI uncovers a (1 in a 100 runs) bug using that test.
Using reported-by doesn't feel right. But credit should go to the person who wrote the test. Is anyone else having this dilemma?
Usually I'd do a reported-by for whoever actually looked at the test system, triaged the issue and reported it. Trying to credit test authorship separately to the testsuite gets cumbersome over time, tests get updated over time for a range of reasons (toolchain updates, adding more coverage, improvements in the testsuite's frameworks...) so it's often not just a single person. Hopefully the testsuite is keeping track of things well enough so mentioning the test will point people in the right direction.
On Fri, 25 Jul 2025 17:39:12 +0100 Mark Brown wrote:
On Fri, Jul 25, 2025 at 08:00:23AM -0700, Jakub Kicinski wrote:
Does anyone have ideas about crediting test authors or tests for bugs discovered? We increasingly see situations where someone adds a test then our subsystem CI uncovers a (1 in a 100 runs) bug using that test.
Using reported-by doesn't feel right. But credit should go to the person who wrote the test. Is anyone else having this dilemma?
Usually I'd do a reported-by for whoever actually looked at the test system, triaged the issue and reported it. Trying to credit test authorship separately to the testsuite gets cumbersome over time, tests get updated over time for a range of reasons (toolchain updates, adding more coverage, improvements in the testsuite's frameworks...) so it's often not just a single person. Hopefully the testsuite is keeping track of things well enough so mentioning the test will point people in the right direction.
Ack, it does get murky overtime. Also with pre-commit testing there usually wouldn't even be a bug in the tree to credit fixing.
I guess we just had a lucky(?) string of very clear cut cases where a good selftest led the maintainer noticing a crash in the CI, and fixing something.
linux-kselftest-mirror@lists.linaro.org
-
dan.j.williams＠intel.com -
Gustavo Padovan -
Jakub Kicinski -
Mark Brown -
Theodore Ts'o