Hi
so this series carries two small fixes to the toplevel KSelfTest makefile which I found useful especially while attempting to run the suite in automation.
[1/2] While it is already possible to specify a limited list of TARGETS to run, it is not instead easily possible to state a list of targets NOT to run (say due to specific instability issues). Moreover providing such a skip list through a stripped down list of TARGETS it is cumbersome and fragile since this poses the risk to stick to an old stale stripped TARGETS list once upstream decides to add more default targets. A new SKIP_TARGETS Makefile variable is provided by this patch to easily specify a skiplist for target subsystems.
[2/2] Currently when some target fails to build, KSFT Makefile just carries on building as much subsystems as it can: unfortunately this is not properly reflected also in the generation of the runlist inside run_kselftest.sh. This patch rectifies this behaviour checking for the existence of a target directory in the INSTALL_PATH before adding the related snippet to the run_kselftest.sh script.
Thanks
Cristian
Changelog
v1 --> v2 - added Documentation - various typos fixed - added a proper override when filtering-out SKIP_TARGETS from TARGETS to make it work also when TARGETS is provided too from the cmdline
Cristian Marussi (2): kselftest: add capability to skip chosen TARGETS kselftest: exclude failed TARGETS from runlist
Documentation/dev-tools/kselftest.rst | 11 +++++++++++ tools/testing/selftests/Makefile | 11 +++++++++++ 2 files changed, 22 insertions(+)
Let the user specify an optional TARGETS skiplist through the new optional SKIP_TARGETS Makefile variable.
It is easier to skip at will using a reduced and well defined list of possibly problematic targets with SKIP_TARGETS than to provide a partially stripped down list of good targets using the usual TARGETS variable.
Signed-off-by: Cristian Marussi cristian.marussi@arm.com --- v1 --> v2 - fixed commit msg - added documentation fix - added forced override to allow TARGETS= and SKIP_TARGETS being both passed on the cmdline --- Documentation/dev-tools/kselftest.rst | 16 ++++++++++++++++ tools/testing/selftests/Makefile | 7 +++++++ 2 files changed, 23 insertions(+)
diff --git a/Documentation/dev-tools/kselftest.rst b/Documentation/dev-tools/kselftest.rst index 25604904fa6e..ecdfdc9d4b03 100644 --- a/Documentation/dev-tools/kselftest.rst +++ b/Documentation/dev-tools/kselftest.rst @@ -89,6 +89,22 @@ To build, save output files in a separate directory with KBUILD_OUTPUT ::
$ export KBUILD_OUTPUT=/tmp/kselftest; make TARGETS="size timers" kselftest
+Additionally you can use the "SKIP_TARGETS" variable on the make command +line to specify one or more targets to exclude from the TARGETS list. + +To run all tests but a single subsystem:: + + $ make -C tools/testing/selftests SKIP_TARGETS=ptrace run_tests + +You can specify multiple tests to skip:: + + $ make SKIP_TARGETS="size timers" kselftest + +You can also specify a restricted list of tests to run together with a +dedicated skiplist:: + + $ make TARGETS="bpf breakpoints size timers" SKIP_TARGETS=bpf kselftest + See the top-level tools/testing/selftests/Makefile for the list of all possible targets.
diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile index 25b43a8c2b15..9621b26ee433 100644 --- a/tools/testing/selftests/Makefile +++ b/tools/testing/selftests/Makefile @@ -63,6 +63,13 @@ TARGETS += zram TARGETS_HOTPLUG = cpu-hotplug TARGETS_HOTPLUG += memory-hotplug
+# User can optionally provide a TARGETS skiplist. +SKIP_TARGETS ?= +ifneq ($(SKIP_TARGETS),) + TMP := $(filter-out $(SKIP_TARGETS), $(TARGETS)) + override TARGETS := $(TMP) +endif + # Clear LDFLAGS and MAKEFLAGS if called from main # Makefile to avoid test build failures when test # Makefile doesn't have explicit build rules.
A TARGET which failed to be built/installed should not be included in the runlist generated inside the run_kselftest.sh script.
Signed-off-by: Cristian Marussi cristian.marussi@arm.com --- As an example, here BPF failed to compile but was included in the runlist and attempted to run anyway:
... ./KSFT_LAST_2/run_kselftest.sh: 37: cd: can't cd to bpf TAP version 13 1..49 # selftests: KSFT_LAST_2: test_verifier # Warning: file test_verifier is missing! not ok 1 selftests: KSFT_LAST_2: test_verifier # selftests: KSFT_LAST_2: test_tag # Warning: file test_tag is missing! --- tools/testing/selftests/Makefile | 4 ++++ 1 file changed, 4 insertions(+)
diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile index 9621b26ee433..d7a9df186837 100644 --- a/tools/testing/selftests/Makefile +++ b/tools/testing/selftests/Makefile @@ -205,8 +205,12 @@ ifdef INSTALL_PATH echo " cat /dev/null > $$logfile" >> $(ALL_SCRIPT) echo "fi" >> $(ALL_SCRIPT)
+ @# While building run_kselftest.sh skip also non-existent TARGET dirs: + @# they could be the result of a build failure and should NOT be + @# included in the generated runlist. for TARGET in $(TARGETS); do \ BUILD_TARGET=$$BUILD/$$TARGET; \ + [ ! -d $$INSTALL_PATH/$$TARGET ] && echo "Skipping non-existent dir: $$TARGET" && continue; \ echo "[ -w /dev/kmsg ] && echo "kselftest: Running tests in $$TARGET" >> /dev/kmsg" >> $(ALL_SCRIPT); \ echo "cd $$TARGET" >> $(ALL_SCRIPT); \ echo -n "run_many" >> $(ALL_SCRIPT); \
linux-kselftest-mirror@lists.linaro.org
-
Cristian Marussi