Hi David,
On Wed, Nov 20, 2024 at 3:16 AM David Gow davidgow@google.com wrote:
On Thu, 7 Nov 2024 at 01:01, Donald Zickus dzickus@redhat.com wrote:
Hi,
Thanks for the feedback. I created a more realistic test.yaml file to start (we can split it when more tests are added) and a parser. I was going to add patch support as input to mimic get_maintainers.pl output, but that might take some time. For now, you have to manually select a subsystem. I will try to find space on kernelci.org to grow this work but you can find a git tree here[0].
From the README.md """ An attempt to map kernel subsystems to kernel tests that should be run on patches or code by humans and CI systems.
Examples:
Find test info for a subsystem
./get_tests.py -s 'KUNIT TEST' --info
Subsystem: KUNIT TEST Maintainer: David Gow davidgow@google.com Mailing List: None Version: None Dependency: ['python3-mypy'] Test: smoke: Url: None Working Directory: None Cmd: ./tools/testing/kunit/kunit.py Env: None Param: run --kunitconfig lib/kunit Hardware: arm64, x86_64
Find copy-n-pastable tests for a subsystem
./get_tests.py -s 'KUNIT TEST'
./tools/testing/kunit/kunit.pyrun --kunitconfig lib/kunit """
Is this aligning with what people were expecting?
Awesome! I've been playing around a bit with this, and I think it's an excellent start.
There are definitely some more features I'd want in an ideal world (e.g., configuration matrices, etc), but this works well enough.
Yeah, I was trying to nail down the usability angle first before expanding with bells and whistles. I would like to think the yaml file is flexible enough to handle those features though??
I've been playing around with a branch which adds the ability to actually run these tests, based on the 'run_checks.py' script we use for KUnit: https://github.com/sulix/test-catalog/tree/runtest-wip
Thanks!
In particular, this adds a '-r' option which runs the tests for the subsystem in parallel. This largely matches what I was doing manually — for instance, the KUnit section in test.yaml now has three different tests, and running it gives me this result: ../test-catalog/get_tests.py -r -s 'KUNIT TEST' Waiting on 3 checks (kunit-tool-test, uml, x86_64)... kunit-tool-test: PASSED x86_64: PASSED uml: PASSED
Interesting. Originally I was thinking this would be done serially. I didn't think tests were safe enough to run in parallel. I am definitely open to this. My python isn't the best, but I think your PR looks reasonable.
(Obviously, in the real world, I'd have more checks, including other architectures, checkpatch, etc, but this works as a proof-of-concept for me.)
I think the most interesting questions will be:
- How do we make this work with more complicated dependencies
(containers, special hardware, etc)?
I was imagining a 'hw-requires' type line to handle the hardware requests as that seemed natural for a lot of the driver work. Run a quick check before running the test to see if the required hw is present or not and bail if it isn't. The containers piece is a little trickier and ties into the test environment I think. The script would have to create an environment and inject the tests into the environment and run them. I would imagine some of this would have to be static as the setup is complicated. For example, a 'container' label would execute custom code to setup a test environment inside a container. Open to ideas here.
- How do we integrate it with CI systems — can we pull the subsystem
name for a patch from MAINTAINERS and look it up here?
There are two thoughts. First is yes. As a developer you probably want to run something like 'get_maintainers.sh <patch> | get_tests.py -s -' or something to figure out what variety of tests you should run before posting. And a CI system could probably do something similar.
There is also another thought, you already know the subsystem you want to test. For example, a patch is usually written for a particular subsystem that happens to touch code from other subsystems. You primarily want to run it against a specified subsystem. I know Red Hat's CKI will run against a known subsystem git-tree and would fall into this category. While it does leave a gap in other subsystem testing, sometimes as a human you already know running those extra tests is mostly a no-op because it doesn't really change anything.
- What about things like checkpatch, or general defconfig build tests
which aren't subsystem-specific?
My initial thought is that this is another category of testing. A lot of CI tests are workload testing and have predefined configs. Whereas a generic testing CI system (think 0-day) would focus on those types of testing. So I would lean away from those checks in this approach or we could add a category 'general' too. I do know checkpatch rules vary from maintainer to maintainer.
- How can we support more complicated configurations or groups of
configurations?
Examples?
- Do we add support for specific tools and/or parsing/combining output?
Examples? I wasn't thinking of parsing test output, just providing what to run as a good first step. My initial thought was to help nudge tests towards the KTAP output??
But I'm content to keep playing around with this a bit more for now.
Thank you! Please do!
Cheers, Don