At present the kselftest header can't be used with nolibc since it makes use of vprintf() which is not available in nolibc and seems like it would be inappropriate to implement given the minimal system requirements and environment intended for nolibc. This has resulted in some open coded kselftests which use nolibc to test features that are supposed to be controlled via libc and therefore better exercised in an environment with no libc.
Rather than continue this let's factor out the I/O routines in kselftest.h into a separate header file and provide a nolibc implementation which only allows simple strings to be provided rather than full printf() support. This is limiting but a great improvement on sharing no code at all.
As an example of using this I've updated the arm64 za-fork test to use the standard kselftest.h.
Signed-off-by: Mark Brown broonie@kernel.org --- Mark Brown (2): kselftest: Support nolibc kselftest/arm64: Convert za-fork to use kselftest.h
tools/testing/selftests/arm64/fp/Makefile | 2 +- tools/testing/selftests/arm64/fp/za-fork.c | 88 +++-------------- tools/testing/selftests/kselftest-nolibc.h | 93 ++++++++++++++++++ tools/testing/selftests/kselftest-std.h | 151 +++++++++++++++++++++++++++++ tools/testing/selftests/kselftest.h | 149 +++------------------------- 5 files changed, 272 insertions(+), 211 deletions(-) --- base-commit: e8d018dd0257f744ca50a729e3d042cf2ec9da65 change-id: 20230405-kselftest-nolibc-cb2ce0446d09
Best regards,
At present the kselftest header can't be used with nolibc since it makes use of vprintf() which is not available in nolibc and seems like it would be inappropriate to implement given the minimal system requirements and environment intended for nolibc. This has resulted in some open coded kselftests which use nolibc to test features that are supposed to be controlled via libc and therefore better exercised in an environment with no libc.
Rather than continue this let's factor out the I/O routines in kselftest.h into a separate header file and provide a nolibc implementation which only allows simple strings to be provided rather than full printf() support. This is limiting but a great improvement on sharing no code at all.
Signed-off-by: Mark Brown broonie@kernel.org --- tools/testing/selftests/kselftest-nolibc.h | 93 ++++++++++++++++++ tools/testing/selftests/kselftest-std.h | 151 +++++++++++++++++++++++++++++ tools/testing/selftests/kselftest.h | 149 +++------------------------- 3 files changed, 255 insertions(+), 138 deletions(-)
diff --git a/tools/testing/selftests/kselftest-nolibc.h b/tools/testing/selftests/kselftest-nolibc.h new file mode 100644 index 000000000000..3dea10cc5490 --- /dev/null +++ b/tools/testing/selftests/kselftest-nolibc.h @@ -0,0 +1,93 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * kselftest-nolibc Cut down nolibc based kselftest output functions + * + * Copyright (c) 2014 Shuah Khan shuahkh@osg.samsung.com + * Copyright (c) 2014 Samsung Electronics Co., Ltd. + * + */ + +#ifndef __KSELFTEST_H +#error This file should never be included directly, always include kselftest.h +#endif + +static inline void ksft_print_msg(const char *msg) +{ + printf("# %s", msg); +} + +static inline void ksft_test_result_pass(const char *msg) +{ + ksft_cnt.ksft_pass++; + + printf("ok %d %s", ksft_test_num(), msg); +} + +static inline void ksft_test_result_fail(const char *msg) +{ + ksft_cnt.ksft_fail++; + + printf("not ok %d %s", ksft_test_num(), msg); +} + +/** + * ksft_test_result() - Report test success based on truth of condition + * + * @condition: if true, report test success, otherwise failure. + */ +#define ksft_test_result(condition, fmt) do { \ + if (!!(condition)) \ + ksft_test_result_pass(fmt);\ + else \ + ksft_test_result_fail(fmt);\ + } while (0) + +static inline void ksft_test_result_xfail(const char *msg) +{ + ksft_cnt.ksft_xfail++; + + printf("ok %d # XFAIL %s", ksft_test_num(), msg); +} + +static inline void ksft_test_result_skip(const char *msg) +{ + ksft_cnt.ksft_xskip++; + + printf("ok %d # SKIP %s", ksft_test_num(), msg); +} + +static inline void ksft_test_result_error(const char *msg) +{ + ksft_cnt.ksft_error++; + + printf("not ok %d # error %s", ksft_test_num(), msg); +} + +static inline int ksft_exit_fail_msg(const char *msg) +{ + printf("Bail out! %s", msg); + + ksft_print_cnts(); + exit(KSFT_FAIL); +} + +static inline int ksft_exit_skip(const char *msg) +{ + /* + * FIXME: several tests misuse ksft_exit_skip so produce + * something sensible if some tests have already been run + * or a plan has been printed. Those tests should use + * ksft_test_result_skip or ksft_exit_fail_msg instead. + */ + if (ksft_plan || ksft_test_num()) { + ksft_cnt.ksft_xskip++; + printf("ok %d # SKIP ", 1 + ksft_test_num()); + } else { + printf("1..0 # SKIP "); + } + if (msg) + printf("%s", msg); + if (ksft_test_num()) + ksft_print_cnts(); + exit(KSFT_SKIP); +} diff --git a/tools/testing/selftests/kselftest-std.h b/tools/testing/selftests/kselftest-std.h new file mode 100644 index 000000000000..b57c515f6dda --- /dev/null +++ b/tools/testing/selftests/kselftest-std.h @@ -0,0 +1,151 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * kselftest-std.h: Full stdio based kselftest output functions + * + * Copyright (c) 2014 Shuah Khan shuahkh@osg.samsung.com + * Copyright (c) 2014 Samsung Electronics Co., Ltd. + * + */ + +#ifndef __KSELFTEST_H +#error This file should never be included directly, always include kselftest.h +#endif + +static inline void ksft_print_msg(const char *msg, ...) +{ + int saved_errno = errno; + va_list args; + + va_start(args, msg); + printf("# "); + errno = saved_errno; + vprintf(msg, args); + va_end(args); +} + +static inline void ksft_test_result_pass(const char *msg, ...) +{ + int saved_errno = errno; + va_list args; + + ksft_cnt.ksft_pass++; + + va_start(args, msg); + printf("ok %d ", ksft_test_num()); + errno = saved_errno; + vprintf(msg, args); + va_end(args); +} + +static inline void ksft_test_result_fail(const char *msg, ...) +{ + int saved_errno = errno; + va_list args; + + ksft_cnt.ksft_fail++; + + va_start(args, msg); + printf("not ok %d ", ksft_test_num()); + errno = saved_errno; + vprintf(msg, args); + va_end(args); +} + +/** + * ksft_test_result() - Report test success based on truth of condition + * + * @condition: if true, report test success, otherwise failure. + */ +#define ksft_test_result(condition, fmt, ...) do { \ + if (!!(condition)) \ + ksft_test_result_pass(fmt, ##__VA_ARGS__);\ + else \ + ksft_test_result_fail(fmt, ##__VA_ARGS__);\ + } while (0) + +static inline void ksft_test_result_xfail(const char *msg, ...) +{ + int saved_errno = errno; + va_list args; + + ksft_cnt.ksft_xfail++; + + va_start(args, msg); + printf("ok %d # XFAIL ", ksft_test_num()); + errno = saved_errno; + vprintf(msg, args); + va_end(args); +} + +static inline void ksft_test_result_skip(const char *msg, ...) +{ + int saved_errno = errno; + va_list args; + + ksft_cnt.ksft_xskip++; + + va_start(args, msg); + printf("ok %d # SKIP ", ksft_test_num()); + errno = saved_errno; + vprintf(msg, args); + va_end(args); +} + +/* TODO: how does "error" differ from "fail" or "skip"? */ +static inline void ksft_test_result_error(const char *msg, ...) +{ + int saved_errno = errno; + va_list args; + + ksft_cnt.ksft_error++; + + va_start(args, msg); + printf("not ok %d # error ", ksft_test_num()); + errno = saved_errno; + vprintf(msg, args); + va_end(args); +} + +static inline int ksft_exit_fail_msg(const char *msg, ...) +{ + int saved_errno = errno; + va_list args; + + va_start(args, msg); + printf("Bail out! "); + errno = saved_errno; + vprintf(msg, args); + va_end(args); + + ksft_print_cnts(); + exit(KSFT_FAIL); +} + +static inline int ksft_exit_skip(const char *msg, ...) +{ + int saved_errno = errno; + va_list args; + + va_start(args, msg); + + /* + * FIXME: several tests misuse ksft_exit_skip so produce + * something sensible if some tests have already been run + * or a plan has been printed. Those tests should use + * ksft_test_result_skip or ksft_exit_fail_msg instead. + */ + if (ksft_plan || ksft_test_num()) { + ksft_cnt.ksft_xskip++; + printf("ok %d # SKIP ", 1 + ksft_test_num()); + } else { + printf("1..0 # SKIP "); + } + if (msg) { + errno = saved_errno; + vprintf(msg, args); + va_end(args); + } + if (ksft_test_num()) + ksft_print_cnts(); + exit(KSFT_SKIP); +} diff --git a/tools/testing/selftests/kselftest.h b/tools/testing/selftests/kselftest.h index 33a0dbd26bd3..10fb6f7c4ae2 100644 --- a/tools/testing/selftests/kselftest.h +++ b/tools/testing/selftests/kselftest.h @@ -26,6 +26,10 @@ * ksft_test_result_xfail(fmt, ...); * ksft_test_result_error(fmt, ...); * + * If building with nolibc then only a string may be provided, va_args + * arre not supported and format characters within the string will not be + * interpreted. + * * When all tests are finished, clean up and exit the program with one of: * * ksft_finished(); @@ -43,11 +47,13 @@ #ifndef __KSELFTEST_H #define __KSELFTEST_H
+#ifndef NOLIBC #include <errno.h> #include <stdlib.h> #include <unistd.h> #include <stdarg.h> #include <stdio.h> +#endif
#ifndef ARRAY_SIZE #define ARRAY_SIZE(arr) (sizeof(arr) / sizeof((arr)[0])) @@ -132,100 +138,11 @@ static inline void ksft_print_cnts(void) ksft_cnt.ksft_xskip, ksft_cnt.ksft_error); }
-static inline void ksft_print_msg(const char *msg, ...) -{ - int saved_errno = errno; - va_list args; - - va_start(args, msg); - printf("# "); - errno = saved_errno; - vprintf(msg, args); - va_end(args); -} - -static inline void ksft_test_result_pass(const char *msg, ...) -{ - int saved_errno = errno; - va_list args; - - ksft_cnt.ksft_pass++; - - va_start(args, msg); - printf("ok %d ", ksft_test_num()); - errno = saved_errno; - vprintf(msg, args); - va_end(args); -} - -static inline void ksft_test_result_fail(const char *msg, ...) -{ - int saved_errno = errno; - va_list args; - - ksft_cnt.ksft_fail++; - - va_start(args, msg); - printf("not ok %d ", ksft_test_num()); - errno = saved_errno; - vprintf(msg, args); - va_end(args); -} - -/** - * ksft_test_result() - Report test success based on truth of condition - * - * @condition: if true, report test success, otherwise failure. - */ -#define ksft_test_result(condition, fmt, ...) do { \ - if (!!(condition)) \ - ksft_test_result_pass(fmt, ##__VA_ARGS__);\ - else \ - ksft_test_result_fail(fmt, ##__VA_ARGS__);\ - } while (0) - -static inline void ksft_test_result_xfail(const char *msg, ...) -{ - int saved_errno = errno; - va_list args; - - ksft_cnt.ksft_xfail++; - - va_start(args, msg); - printf("ok %d # XFAIL ", ksft_test_num()); - errno = saved_errno; - vprintf(msg, args); - va_end(args); -} - -static inline void ksft_test_result_skip(const char *msg, ...) -{ - int saved_errno = errno; - va_list args; - - ksft_cnt.ksft_xskip++; - - va_start(args, msg); - printf("ok %d # SKIP ", ksft_test_num()); - errno = saved_errno; - vprintf(msg, args); - va_end(args); -} - -/* TODO: how does "error" differ from "fail" or "skip"? */ -static inline void ksft_test_result_error(const char *msg, ...) -{ - int saved_errno = errno; - va_list args; - - ksft_cnt.ksft_error++; - - va_start(args, msg); - printf("not ok %d # error ", ksft_test_num()); - errno = saved_errno; - vprintf(msg, args); - va_end(args); -} +#ifdef NOLIBC +#include "kselftest-nolibc.h" +#else +#include "kselftest-std.h" +#endif
static inline int ksft_exit_pass(void) { @@ -260,21 +177,6 @@ static inline int ksft_exit_fail(void) ksft_cnt.ksft_xfail + \ ksft_cnt.ksft_xskip)
-static inline int ksft_exit_fail_msg(const char *msg, ...) -{ - int saved_errno = errno; - va_list args; - - va_start(args, msg); - printf("Bail out! "); - errno = saved_errno; - vprintf(msg, args); - va_end(args); - - ksft_print_cnts(); - exit(KSFT_FAIL); -} - static inline int ksft_exit_xfail(void) { ksft_print_cnts(); @@ -287,33 +189,4 @@ static inline int ksft_exit_xpass(void) exit(KSFT_XPASS); }
-static inline int ksft_exit_skip(const char *msg, ...) -{ - int saved_errno = errno; - va_list args; - - va_start(args, msg); - - /* - * FIXME: several tests misuse ksft_exit_skip so produce - * something sensible if some tests have already been run - * or a plan has been printed. Those tests should use - * ksft_test_result_skip or ksft_exit_fail_msg instead. - */ - if (ksft_plan || ksft_test_num()) { - ksft_cnt.ksft_xskip++; - printf("ok %d # SKIP ", 1 + ksft_test_num()); - } else { - printf("1..0 # SKIP "); - } - if (msg) { - errno = saved_errno; - vprintf(msg, args); - va_end(args); - } - if (ksft_test_num()) - ksft_print_cnts(); - exit(KSFT_SKIP); -} - #endif /* __KSELFTEST_H */
Now that kselftest.h can be used with nolibc convert the za-fork test to use it. We do still have to open code ksft_print_msg() but that's not the end of the world. Some of the advantage comes from using printf() which we could have been using already.
This does change the output when tests are skipped, bringing it in line with the standard kselftest output by removing the test name - we move from
ok 0 skipped
to
ok 1 # SKIP fork_test
The old output was not following KTAP format for skips, and the numbering was not standard or consistent with the reported plan.
Signed-off-by: Mark Brown broonie@kernel.org --- tools/testing/selftests/arm64/fp/Makefile | 2 +- tools/testing/selftests/arm64/fp/za-fork.c | 88 ++++++------------------------ 2 files changed, 17 insertions(+), 73 deletions(-)
diff --git a/tools/testing/selftests/arm64/fp/Makefile b/tools/testing/selftests/arm64/fp/Makefile index 48f56c86ad45..b413b0af07f9 100644 --- a/tools/testing/selftests/arm64/fp/Makefile +++ b/tools/testing/selftests/arm64/fp/Makefile @@ -38,7 +38,7 @@ $(OUTPUT)/vec-syscfg: vec-syscfg.c $(OUTPUT)/rdvl.o $(OUTPUT)/vlset: vlset.c $(OUTPUT)/za-fork: za-fork.c $(OUTPUT)/za-fork-asm.o $(CC) -fno-asynchronous-unwind-tables -fno-ident -s -Os -nostdlib \ - -include ../../../../include/nolibc/nolibc.h \ + -include ../../../../include/nolibc/nolibc.h -I../..\ -static -ffreestanding -Wall $^ -o $@ $(OUTPUT)/za-ptrace: za-ptrace.c $(OUTPUT)/za-test: za-test.S $(OUTPUT)/asm-utils.o diff --git a/tools/testing/selftests/arm64/fp/za-fork.c b/tools/testing/selftests/arm64/fp/za-fork.c index ff475c649e96..3acd5621e468 100644 --- a/tools/testing/selftests/arm64/fp/za-fork.c +++ b/tools/testing/selftests/arm64/fp/za-fork.c @@ -9,42 +9,9 @@ #include <linux/sched.h> #include <linux/wait.h>
-#define EXPECTED_TESTS 1 - -static void putstr(const char *str) -{ - write(1, str, strlen(str)); -} - -static void putnum(unsigned int num) -{ - char c; - - if (num / 10) - putnum(num / 10); - - c = '0' + (num % 10); - write(1, &c, 1); -} +#include "kselftest.h"
-static int tests_run; -static int tests_passed; -static int tests_failed; -static int tests_skipped; - -static void print_summary(void) -{ - if (tests_passed + tests_failed + tests_skipped != EXPECTED_TESTS) - putstr("# UNEXPECTED TEST COUNT: "); - - putstr("# Totals: pass:"); - putnum(tests_passed); - putstr(" fail:"); - putnum(tests_failed); - putstr(" xfail:0 xpass:0 skip:"); - putnum(tests_skipped); - putstr(" error:0\n"); -} +#define EXPECTED_TESTS 1
int fork_test(void); int verify_fork(void); @@ -63,22 +30,21 @@ int fork_test_c(void) if (newpid == 0) { /* In child */ if (!verify_fork()) { - putstr("# ZA state invalid in child\n"); + ksft_print_msg("ZA state invalid in child\n"); exit(0); } else { exit(1); } } if (newpid < 0) { - putstr("# fork() failed: -"); - putnum(-newpid); - putstr("\n"); + printf("# fork() failed: %d\n", newpid); + return 0; }
parent_result = verify_fork(); if (!parent_result) - putstr("# ZA state invalid in parent\n"); + ksft_print_msg("ZA state invalid in parent\n");
for (;;) { waiting = waitpid(newpid, &child_status, 0); @@ -86,18 +52,16 @@ int fork_test_c(void) if (waiting < 0) { if (errno == EINTR) continue; - putstr("# waitpid() failed: "); - putnum(errno); - putstr("\n"); + printf("# waitpid() failed: %d\n", errno); return 0; } if (waiting != newpid) { - putstr("# waitpid() returned wrong PID\n"); + printf("# waitpid() returned wrong PID\n"); return 0; }
if (!WIFEXITED(child_status)) { - putstr("# child did not exit\n"); + printf("# child did not exit\n"); return 0; }
@@ -105,29 +69,14 @@ int fork_test_c(void) } }
-#define run_test(name) \ - if (name()) { \ - tests_passed++; \ - } else { \ - tests_failed++; \ - putstr("not "); \ - } \ - putstr("ok "); \ - putnum(++tests_run); \ - putstr(" " #name "\n"); - int main(int argc, char **argv) { int ret, i;
- putstr("TAP version 13\n"); - putstr("1.."); - putnum(EXPECTED_TESTS); - putstr("\n"); + ksft_print_header(); + ksft_set_plan(EXPECTED_TESTS);
- putstr("# PID: "); - putnum(getpid()); - putstr("\n"); + printf("# PID: %d\n", getpid());
/* * This test is run with nolibc which doesn't support hwcap and @@ -136,21 +85,16 @@ int main(int argc, char **argv) */ ret = open("/proc/sys/abi/sme_default_vector_length", O_RDONLY, 0); if (ret >= 0) { - run_test(fork_test); + ksft_test_result(fork_test(), "fork_test");
} else { - putstr("# SME support not present\n"); - + ksft_print_msg("SME not supported\n"); for (i = 0; i < EXPECTED_TESTS; i++) { - putstr("ok "); - putnum(i); - putstr(" skipped\n"); + ksft_test_result_skip("fork_test\n"); } - - tests_skipped += EXPECTED_TESTS; }
- print_summary(); + ksft_finished();
return 0; }
Hi Mark,
On Thu, Apr 06, 2023 at 02:56:28PM +0100, Mark Brown wrote:
At present the kselftest header can't be used with nolibc since it makes use of vprintf() which is not available in nolibc and seems like it would be inappropriate to implement given the minimal system requirements and environment intended for nolibc.
In fact we already have vfprintf(), and printf() is based on it, so wouldn't it just be a matter of adding vprintf() that calls vfprintf() for your case ? Maybe just something like this :
static int vprintf(const char *fmt, va_list args) { return vfprintf(stdout, fmt, args); }
It's possible I'm missing something, but it's also possible you didn't find vfprintf() which is why I prefer to raise my hand ;-)
This has resulted in some open coded kselftests which use nolibc to test features that are supposed to be controlled via libc and therefore better exercised in an environment with no libc.
Yeah that's ugly. In nolibc-test we now have two build targets so that we can more easily verify the compatibility between the default libc and nolibc, so my recommendation would be to stick to a common subset of both libcs, but not to rely on nolibc-specific stuff that could make tests harder to debug.
Regards, Willy
On Thu, Apr 06, 2023 at 04:20:29PM +0200, Willy Tarreau wrote:
On Thu, Apr 06, 2023 at 02:56:28PM +0100, Mark Brown wrote:
At present the kselftest header can't be used with nolibc since it makes use of vprintf() which is not available in nolibc and seems like it would be inappropriate to implement given the minimal system requirements and environment intended for nolibc.
In fact we already have vfprintf(), and printf() is based on it, so wouldn't it just be a matter of adding vprintf() that calls vfprintf() for your case ? Maybe just something like this :
static int vprintf(const char *fmt, va_list args) { return vfprintf(stdout, fmt, args); }
It's possible I'm missing something, but it's also possible you didn't find vfprintf() which is why I prefer to raise my hand ;-)
Oh, yes - I just didn't find that. Can't remember what I searched for but it didn't match.
This has resulted in some open coded kselftests which use nolibc to test features that are supposed to be controlled via libc and therefore better exercised in an environment with no libc.
Yeah that's ugly. In nolibc-test we now have two build targets so that we can more easily verify the compatibility between the default libc and nolibc, so my recommendation would be to stick to a common subset of both libcs, but not to rely on nolibc-specific stuff that could make tests harder to debug.
For these features we simply never want to run with a proper libc since if we use a libc which has support for the features then we can't meaningfully interact with them. We're trying to test interfaces that libc is supposed to use.
On Thu, Apr 06, 2023 at 03:32:20PM +0100, Mark Brown wrote:
On Thu, Apr 06, 2023 at 04:20:29PM +0200, Willy Tarreau wrote:
On Thu, Apr 06, 2023 at 02:56:28PM +0100, Mark Brown wrote:
At present the kselftest header can't be used with nolibc since it makes use of vprintf() which is not available in nolibc and seems like it would be inappropriate to implement given the minimal system requirements and environment intended for nolibc.
In fact we already have vfprintf(), and printf() is based on it, so wouldn't it just be a matter of adding vprintf() that calls vfprintf() for your case ? Maybe just something like this :
static int vprintf(const char *fmt, va_list args) { return vfprintf(stdout, fmt, args); }
It's possible I'm missing something, but it's also possible you didn't find vfprintf() which is why I prefer to raise my hand ;-)
Oh, yes - I just didn't find that. Can't remember what I searched for but it didn't match.
No problem. I just remembered it existed because we just received a new test for it a few days ago ;-)
This has resulted in some open coded kselftests which use nolibc to test features that are supposed to be controlled via libc and therefore better exercised in an environment with no libc.
Yeah that's ugly. In nolibc-test we now have two build targets so that we can more easily verify the compatibility between the default libc and nolibc, so my recommendation would be to stick to a common subset of both libcs, but not to rely on nolibc-specific stuff that could make tests harder to debug.
For these features we simply never want to run with a proper libc since if we use a libc which has support for the features then we can't meaningfully interact with them. We're trying to test interfaces that libc is supposed to use.
Indeed, this totally makes sense then! But I think you get the idea of what I was suggesting which is to try to avoid getting trapped by a single implementation in general, by using portable stuff as much as possible.
Cheers, Willy
linux-kselftest-mirror@lists.linaro.org
-
Mark Brown -
Willy Tarreau