Hi!
I really like Hangbin Liu's intent[1] but I think we need to be a little more clean about the implementation. This extracts run_kselftest.sh from the Makefile so it can actually be changed without embeds, etc. Instead, generate the test list into a text file. Everything gets much simpler. :)
And in patch 2, I add back Hangin Liu's new options (with some extra added) with knowledge of "collections" (i.e. Makefile TARGETS) and subtests. This should work really well with LAVA too, which needs to manipulate the lists of tests being run.
Thoughts?
-Kees
[1] https://lore.kernel.org/lkml/20200914022227.437143-1-liuhangbin@gmail.com/
Kees Cook (2): selftests: Extract run_kselftest.sh and generate stand-alone test list selftests/run_kselftest.sh: Make each test individually selectable
tools/testing/selftests/Makefile | 26 ++----- tools/testing/selftests/lib.mk | 5 +- tools/testing/selftests/run_kselftest.sh | 89 ++++++++++++++++++++++++ 3 files changed, 98 insertions(+), 22 deletions(-) create mode 100755 tools/testing/selftests/run_kselftest.sh
Instead of building a script on the fly (which just repeats the same thing for each test collection), move the script out of the Makefile and into run_kselftest.sh, which reads kselftest-list.txt.
Adjust the emit_tests target to report each test on a separate line so that test running tools (e.g. LAVA) can easily remove individual tests (for example, as seen in [1]).
[1] https://github.com/Linaro/test-definitions/pull/208/commits/2e7b62155e4998e5...
Cc: Shuah Khan shuah@kernel.org Cc: linux-kselftest@vger.kernel.org Signed-off-by: Kees Cook keescook@chromium.org --- tools/testing/selftests/Makefile | 26 +++++++----------------- tools/testing/selftests/lib.mk | 5 ++--- tools/testing/selftests/run_kselftest.sh | 24 ++++++++++++++++++++++ 3 files changed, 33 insertions(+), 22 deletions(-) create mode 100755 tools/testing/selftests/run_kselftest.sh
diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile index 15c1c1359c50..d9c283503159 100644 --- a/tools/testing/selftests/Makefile +++ b/tools/testing/selftests/Makefile @@ -206,6 +206,7 @@ KSFT_INSTALL_PATH := $(abspath $(KSFT_INSTALL_PATH)) # Avoid changing the rest of the logic here and lib.mk. INSTALL_PATH := $(KSFT_INSTALL_PATH) ALL_SCRIPT := $(INSTALL_PATH)/run_kselftest.sh +TEST_LIST := $(INSTALL_PATH)/kselftest-list.txt
install: all ifdef INSTALL_PATH @@ -214,6 +215,8 @@ ifdef INSTALL_PATH install -m 744 kselftest/module.sh $(INSTALL_PATH)/kselftest/ install -m 744 kselftest/runner.sh $(INSTALL_PATH)/kselftest/ install -m 744 kselftest/prefix.pl $(INSTALL_PATH)/kselftest/ + install -m 744 run_kselftest.sh $(INSTALL_PATH)/ + rm -f $(TEST_LIST) @ret=1; \ for TARGET in $(TARGETS); do \ BUILD_TARGET=$$BUILD/$$TARGET; \ @@ -222,33 +225,18 @@ ifdef INSTALL_PATH ret=$$((ret * $$?)); \ done; exit $$ret;
- @# Ask all targets to emit their test scripts - echo "#!/bin/sh" > $(ALL_SCRIPT) - echo "BASE_DIR=$$(realpath $$(dirname $$0))" >> $(ALL_SCRIPT) - echo "cd $$BASE_DIR" >> $(ALL_SCRIPT) - echo ". ./kselftest/runner.sh" >> $(ALL_SCRIPT) - echo "ROOT=$$PWD" >> $(ALL_SCRIPT) - echo "if [ "$$1" = "--summary" ]; then" >> $(ALL_SCRIPT) - echo " logfile=$$BASE_DIR/output.log" >> $(ALL_SCRIPT) - echo " cat /dev/null > $$logfile" >> $(ALL_SCRIPT) - echo "fi" >> $(ALL_SCRIPT)
- @# While building run_kselftest.sh skip also non-existent TARGET dirs: + @# Ask all targets to emit their test scripts + @# While building kselftest-list.text skip also non-existent TARGET dirs: @# they could be the result of a build failure and should NOT be @# included in the generated runlist. for TARGET in $(TARGETS); do \ BUILD_TARGET=$$BUILD/$$TARGET; \ [ ! -d $(INSTALL_PATH)/$$TARGET ] && echo "Skipping non-existent dir: $$TARGET" && continue; \ - echo "[ -w /dev/kmsg ] && echo "kselftest: Running tests in $$TARGET" >> /dev/kmsg" >> $(ALL_SCRIPT); \ - echo "cd $$TARGET" >> $(ALL_SCRIPT); \ - echo -n "run_many" >> $(ALL_SCRIPT); \ echo -n "Emit Tests for $$TARGET\n"; \ - $(MAKE) -s --no-print-directory OUTPUT=$$BUILD_TARGET -C $$TARGET emit_tests >> $(ALL_SCRIPT); \ - echo "" >> $(ALL_SCRIPT); \ - echo "cd $$ROOT" >> $(ALL_SCRIPT); \ + $(MAKE) -s --no-print-directory OUTPUT=$$BUILD_TARGET COLLECTION=$$TARGET \ + -C $$TARGET emit_tests >> $(TEST_LIST); \ done; - - chmod u+x $(ALL_SCRIPT) else $(error Error: set INSTALL_PATH to use install) endif diff --git a/tools/testing/selftests/lib.mk b/tools/testing/selftests/lib.mk index 51124b962d56..30848ca36555 100644 --- a/tools/testing/selftests/lib.mk +++ b/tools/testing/selftests/lib.mk @@ -107,9 +107,8 @@ endif emit_tests: for TEST in $(TEST_GEN_PROGS) $(TEST_CUSTOM_PROGS) $(TEST_PROGS); do \ BASENAME_TEST=`basename $$TEST`; \ - echo " \"; \ - echo -n " "$$BASENAME_TEST""; \ - done; \ + echo "$(COLLECTION):$$BASENAME_TEST"; \ + done
# define if isn't already. It is undefined in make O= case. ifeq ($(RM),) diff --git a/tools/testing/selftests/run_kselftest.sh b/tools/testing/selftests/run_kselftest.sh new file mode 100755 index 000000000000..e07344be28ae --- /dev/null +++ b/tools/testing/selftests/run_kselftest.sh @@ -0,0 +1,24 @@ +#!/bin/sh +BASE_DIR=$(realpath $(dirname $0)) +cd $BASE_DIR +TESTS="$BASE_DIR"/kselftest-list.txt +if [ ! -r "$TESTS" ] ; then + echo "$0: Could not find list of tests to run ($TESTS)" >&2 + exit 1 +fi +available="$(cat "$TESTS")" + +. ./kselftest/runner.sh +ROOT=$PWD + +if [ "$1" = "--summary" ] ; then + logfile="$BASE_DIR"/output.log + cat /dev/null > $logfile +fi + +collections=$(echo "$available" | cut -d: -f1 | uniq) +for collection in $collections ; do + [ -w /dev/kmsg ] && echo "kselftest: Running tests in $collection" >> /dev/kmsg + tests=$(echo "$available" | grep "^$collection:" | cut -d: -f2) + (cd "$collection" && run_many $tests) +done
Currently with run_kselftest.sh there is no way to choose which test we could run. All the tests listed in kselftest-list.txt are all run every time. This patch enhanced the run_kselftest.sh to make the test collections (or tests) individually selectable. e.g.:
$ ./run_kselftest.sh -c seccomp -t timers:posix_timers -t timers:nanosleep
Additionally adds a way to list all known tests with "-l", usage with "-h", and perform a dry run without running tests with "-n".
Co-developed-by: Hangbin Liu liuhangbin@gmail.com Signed-off-by: Hangbin Liu liuhangbin@gmail.com Signed-off-by: Kees Cook keescook@chromium.org --- tools/testing/selftests/run_kselftest.sh | 77 ++++++++++++++++++++++-- 1 file changed, 71 insertions(+), 6 deletions(-)
diff --git a/tools/testing/selftests/run_kselftest.sh b/tools/testing/selftests/run_kselftest.sh index e07344be28ae..c08089f80e28 100755 --- a/tools/testing/selftests/run_kselftest.sh +++ b/tools/testing/selftests/run_kselftest.sh @@ -4,21 +4,86 @@ cd $BASE_DIR TESTS="$BASE_DIR"/kselftest-list.txt if [ ! -r "$TESTS" ] ; then echo "$0: Could not find list of tests to run ($TESTS)" >&2 - exit 1 + available="" +else + available="$(cat "$TESTS")" fi -available="$(cat "$TESTS")"
. ./kselftest/runner.sh ROOT=$PWD
-if [ "$1" = "--summary" ] ; then - logfile="$BASE_DIR"/output.log - cat /dev/null > $logfile +usage() +{ + cat <<EOF +Usage: $0 [OPTIONS] + -s | --summary Print summary with detailed log in output.log + -t | --test COLLECTION:TEST Run TEST from COLLECTION + -c | --collection COLLECTION Run all tests from COLLECTION + -l | --list List the available collection:test entries + -d | --dry-run Don't actually run any tests + -h | --help Show this usage info +EOF + exit $1 +} + +COLLECTIONS="" +TESTS="" +dryrun="" +while true; do + case "$1" in + -s | --summary) + logfile="$BASE_DIR"/output.log + cat /dev/null > $logfile + shift ;; + -t | --test) + TESTS="$TESTS $2" + shift 2 ;; + -c | --collection) + COLLECTIONS="$COLLECTIONS $2" + shift 2 ;; + -l | --list) + echo "$available" + exit 0 ;; + -n | --dry-run) + dryrun="echo" + shift ;; + -h | --help) + usage 0 ;; + "") + break ;; + *) + usage 1 ;; + esac +done + +# Add all selected collections to the explicit test list. +if [ -n "$COLLECTIONS" ]; then + for collection in $COLLECTIONS ; do + found="$(echo "$available" | grep "^$collection:")" + if [ -z "$found" ] ; then + echo "No such collection '$collection'" >&2 + exit 1 + fi + TESTS="$TESTS $found" + done +fi +# Replace available test list with explicitly selected tests. +if [ -n "$TESTS" ]; then + valid="" + for test in $TESTS ; do + found="$(echo "$available" | grep "^${test}$")" + if [ -z "$found" ] ; then + echo "No such test '$test'" >&2 + exit 1 + fi + valid="$valid $found" + done + available="$(echo "$valid" | sed -e 's/ /\n/g')" fi
collections=$(echo "$available" | cut -d: -f1 | uniq) for collection in $collections ; do [ -w /dev/kmsg ] && echo "kselftest: Running tests in $collection" >> /dev/kmsg tests=$(echo "$available" | grep "^$collection:" | cut -d: -f2) - (cd "$collection" && run_many $tests) + ($dryrun cd "$collection" && $dryrun run_many $tests) done
On Fri, Sep 25, 2020 at 04:45:27PM -0700, Kees Cook wrote:
Currently with run_kselftest.sh there is no way to choose which test we could run. All the tests listed in kselftest-list.txt are all run every time. This patch enhanced the run_kselftest.sh to make the test collections (or tests) individually selectable. e.g.:
$ ./run_kselftest.sh -c seccomp -t timers:posix_timers -t timers:nanosleep
Additionally adds a way to list all known tests with "-l", usage with "-h", and perform a dry run without running tests with "-n".
This is better than my previous patch and we can modify run_kselftest.sh easily. The Documentation/dev-tools/kselftest.rst should also be update.
Thanks Hangbin
On Sun, Sep 27, 2020 at 10:48:40AM +0800, Hangbin Liu wrote:
On Fri, Sep 25, 2020 at 04:45:27PM -0700, Kees Cook wrote:
Currently with run_kselftest.sh there is no way to choose which test we could run. All the tests listed in kselftest-list.txt are all run every time. This patch enhanced the run_kselftest.sh to make the test collections (or tests) individually selectable. e.g.:
$ ./run_kselftest.sh -c seccomp -t timers:posix_timers -t timers:nanosleep
Additionally adds a way to list all known tests with "-l", usage with "-h", and perform a dry run without running tests with "-n".
This is better than my previous patch and we can modify run_kselftest.sh easily. The Documentation/dev-tools/kselftest.rst should also be update.
Thanks! I will send a v2.
linux-kselftest-mirror@lists.linaro.org
-
Hangbin Liu -
Kees Cook