Jakub Kicinski wrote:
On Wed, 28 Aug 2024 15:33:33 -0400 Willem de Bruijn wrote:
That could work.
Is reporting one KTAP and exitcode per directory vs per packetdrill invocation good/bad/neither?
To me "neither", hopefully and unhelpfully I won't tell you anything that's not common sense :) The balance is between:
- having test cases which don't take too long (assuming debug kernel): <15min is good,
1h I will start complaining, 1h30m is bad because we can't retry and still make the 3hdeadline that NIPA has,3h the test can't run in NIPA at all.vs
- flip side is having so many cases it's hard to keep track and render in the UI. JSON is relatively slow to process. If you have 150 cases, that's 300 per branch (debug and non-debug kernels), times 8 branches a day => 2.4k results / day I think that's still fine-ish, but on the larger side for sure. For reference net and forwarding have ~100 tests each. FWIW we do have the ability to collect and display nested KTAP so the information is not lost (but it makes queries slower so we don't fetch it by default).
Three other issues if this is calling packetdrill directly is
- passing the non-trivial IP specific flags
- running twice, for IPv4 and IPv6
- chdir into the directory of the pkt file
That can be addressed by instead calling a small wrapper shell script.
That would do the test_func_builder part of packetdrill_ksft.py. But without the need to handle netns, popen/cmd, etc, and thus the ksft dependencies.
Right!
Implemented this, but hit a snag
Kselftest install does not preserve directories.
So all .pkt files are copied into net/packetdrill root. This is messy. More fundamentally it breaks the includes in the files (e..g, `source ../common/defaults.sh`).
The output of having each test separate is also quite unreadable.
An alternative is to add each subdirectory as a separate TARGET. But that seems similarly impractical. And does nothing to group KTAP output.
In practice, most directories have a handful of .pkt scripts, and each script is fast, so serialization is not a huge issue.
I found tools/testing/selftests/net/kselftest/ktap_helpers.sh, which has helpful KTAP boilerplate for shell tests, including KSFT exit codes. Am using that instead of python, to reduce the external dependencies.