Hi,
Here is the 2nd version of the series of "requires:" list for
simplifying and unifying requirement checks for each test case.
The previous version is here.
https://lkml.kernel.org/r/159102252279.31199.12855129586058455119.stgit@dev…
I've fixed some trival mistakes and add Tom's reviewed-by
in this version.
Currently, we have many similar requirement checker to find
unconfigured or unsupported (in older kernels) feature in
each test case. I think it is a good time to unify those similar
checks.
As same as "description:" or "flags:" line, this series introduces
new "requires:" line, and convert current checking code intor the
"requires:" line.
This requires line gives some good effects, not only simplyfies
the code, but also unifies the reason message, and because it checks
the requirements before running the testc ase, it skips unneeded
ftrace initialization.
The requires line supports following checks
- tracefs interface check: Check whether the given file or directory
in the tracefs. (No suffix) [3/7],[4/7],[5/7]
- available tracer check: Check whether the given tracer is available
(":tracer" suffix) [6/7]
- README feature check: Check whether the given string is in the
README (":README" suffix) [7/7]
This series also includes the description line fix and
unresolved -> unsupported change ([1/7] and [2/7]).
Note: Since the requires line returns UNSUPPORTED error,
the requirements must be one of ftrace feature, but not the
user-space environmental requirement. If there is some issue
in user-space (e.g. lack of the command, modules, etc) it must
report UNRESOLVED error.
Since this series depends on following 2 commits,
commit 619ee76f5c9f ("selftests/ftrace: Return unsupported if no
error_log file") on Shuah's Kselftest tree
commit bea24f766efc ("selftests/ftrace: Distinguish between hist
and synthetic event checks") on Steven's Tracing tree
This can be applied on the tree which merged both of them.
Also, you can get the series from the following.
git://git.kernel.org/pub/scm/linux/kernel/git/mhiramat/linux.git ftracetest-requires-v2
Thank you,
---
Masami Hiramatsu (7):
selftests/ftrace: Allow ":" in description
selftests/ftrace: Return unsupported for the unconfigured features
selftests/ftrace: Add "requires:" list support
selftests/ftrace: Convert required interface checks into requires list
selftests/ftrace: Convert check_filter_file() with requires list
selftests/ftrace: Support ":tracer" suffix for requires
selftests/ftrace: Support ":README" suffix for requires
tools/testing/selftests/ftrace/ftracetest | 11 ++++++-
.../selftests/ftrace/test.d/00basic/snapshot.tc | 3 +-
.../selftests/ftrace/test.d/00basic/trace_pipe.tc | 3 +-
.../ftrace/test.d/direct/kprobe-direct.tc | 6 +---
.../ftrace/test.d/dynevent/add_remove_kprobe.tc | 6 +---
.../ftrace/test.d/dynevent/add_remove_synth.tc | 5 +--
.../ftrace/test.d/dynevent/clear_select_events.tc | 11 +------
.../ftrace/test.d/dynevent/generic_clear_event.tc | 8 +----
.../selftests/ftrace/test.d/event/event-enable.tc | 6 +---
.../selftests/ftrace/test.d/event/event-no-pid.tc | 11 +------
.../selftests/ftrace/test.d/event/event-pid.tc | 11 +------
.../ftrace/test.d/event/subsystem-enable.tc | 6 +---
.../ftrace/test.d/event/toplevel-enable.tc | 6 +---
.../ftrace/test.d/ftrace/fgraph-filter-stack.tc | 14 +--------
.../ftrace/test.d/ftrace/fgraph-filter.tc | 8 +----
.../ftrace/test.d/ftrace/func-filter-glob.tc | 8 +----
.../test.d/ftrace/func-filter-notrace-pid.tc | 13 +-------
.../ftrace/test.d/ftrace/func-filter-pid.tc | 13 +-------
.../ftrace/test.d/ftrace/func-filter-stacktrace.tc | 3 +-
.../selftests/ftrace/test.d/ftrace/func_cpumask.tc | 6 +---
.../ftrace/test.d/ftrace/func_event_triggers.tc | 7 ++---
.../ftrace/test.d/ftrace/func_mod_trace.tc | 3 +-
.../ftrace/test.d/ftrace/func_profile_stat.tc | 3 +-
.../ftrace/test.d/ftrace/func_profiler.tc | 12 +-------
.../ftrace/test.d/ftrace/func_set_ftrace_file.tc | 6 ++--
.../ftrace/test.d/ftrace/func_stack_tracer.tc | 8 +----
.../test.d/ftrace/func_traceonoff_triggers.tc | 6 ++--
.../ftrace/test.d/ftrace/tracing-error-log.tc | 12 ++------
tools/testing/selftests/ftrace/test.d/functions | 28 ++++++++++++++----
.../ftrace/test.d/instances/instance-event.tc | 6 +---
.../selftests/ftrace/test.d/instances/instance.tc | 6 +---
.../ftrace/test.d/kprobe/add_and_remove.tc | 3 +-
.../selftests/ftrace/test.d/kprobe/busy_check.tc | 3 +-
.../selftests/ftrace/test.d/kprobe/kprobe_args.tc | 3 +-
.../ftrace/test.d/kprobe/kprobe_args_comm.tc | 3 +-
.../ftrace/test.d/kprobe/kprobe_args_string.tc | 3 +-
.../ftrace/test.d/kprobe/kprobe_args_symbol.tc | 3 +-
.../ftrace/test.d/kprobe/kprobe_args_syntax.tc | 5 +--
.../ftrace/test.d/kprobe/kprobe_args_type.tc | 5 +--
.../ftrace/test.d/kprobe/kprobe_args_user.tc | 4 +--
.../ftrace/test.d/kprobe/kprobe_eventname.tc | 3 +-
.../ftrace/test.d/kprobe/kprobe_ftrace.tc | 6 +---
.../ftrace/test.d/kprobe/kprobe_module.tc | 3 +-
.../ftrace/test.d/kprobe/kprobe_multiprobe.tc | 5 +--
.../ftrace/test.d/kprobe/kprobe_syntax_errors.tc | 5 +--
.../ftrace/test.d/kprobe/kretprobe_args.tc | 3 +-
.../ftrace/test.d/kprobe/kretprobe_maxactive.tc | 4 +--
.../ftrace/test.d/kprobe/multiple_kprobes.tc | 3 +-
.../selftests/ftrace/test.d/kprobe/probepoint.tc | 3 +-
.../selftests/ftrace/test.d/kprobe/profile.tc | 3 +-
.../ftrace/test.d/kprobe/uprobe_syntax_errors.tc | 5 +--
.../ftrace/test.d/preemptirq/irqsoff_tracer.tc | 4 +--
tools/testing/selftests/ftrace/test.d/template | 4 +++
.../selftests/ftrace/test.d/tracer/wakeup.tc | 6 +---
.../selftests/ftrace/test.d/tracer/wakeup_rt.tc | 6 +---
.../inter-event/trigger-action-hist-xfail.tc | 13 +-------
.../inter-event/trigger-field-variable-support.tc | 16 +---------
.../trigger-inter-event-combined-hist.tc | 16 +---------
.../inter-event/trigger-multi-actions-accept.tc | 16 +---------
.../inter-event/trigger-onchange-action-hist.tc | 8 +----
.../inter-event/trigger-onmatch-action-hist.tc | 16 +---------
.../trigger-onmatch-onmax-action-hist.tc | 16 +---------
.../inter-event/trigger-onmax-action-hist.tc | 16 +---------
.../inter-event/trigger-snapshot-action-hist.tc | 20 +------------
.../trigger-synthetic-event-createremove.tc | 11 +------
.../inter-event/trigger-synthetic-event-syntax.tc | 11 +------
.../inter-event/trigger-trace-action-hist.tc | 18 +-----------
.../ftrace/test.d/trigger/trigger-eventonoff.tc | 11 +------
.../ftrace/test.d/trigger/trigger-filter.tc | 11 +------
.../ftrace/test.d/trigger/trigger-hist-mod.tc | 16 +---------
.../test.d/trigger/trigger-hist-syntax-errors.tc | 18 +-----------
.../ftrace/test.d/trigger/trigger-hist.tc | 16 +---------
.../ftrace/test.d/trigger/trigger-multihist.tc | 16 +---------
.../ftrace/test.d/trigger/trigger-snapshot.tc | 16 +---------
.../ftrace/test.d/trigger/trigger-stacktrace.tc | 11 +------
.../test.d/trigger/trigger-trace-marker-hist.tc | 21 +-------------
.../trigger/trigger-trace-marker-snapshot.tc | 21 +-------------
.../trigger-trace-marker-synthetic-kernel.tc | 31 +-------------------
.../trigger/trigger-trace-marker-synthetic.tc | 26 +----------------
.../ftrace/test.d/trigger/trigger-traceonoff.tc | 11 +------
80 files changed, 119 insertions(+), 633 deletions(-)
--
Masami Hiramatsu (Linaro) <mhiramat(a)kernel.org>
Add an FAQ entry to the KUnit documentation with some tips for
troubleshooting KUnit and kunit_tool.
These suggestions largely came from an email thread:
https://lore.kernel.org/linux-kselftest/41db8bbd-3ba0-8bde-7352-083bf4b947f…
Signed-off-by: David Gow <davidgow(a)google.com>
Reviewed-by: Alan Maguire <alan.maguire(a)oracle.com>
Reviewed-by: Brendan Higgins <brendanhiggins(a)google.com>
---
Changes since v1[1]
- Added a section about running KUnit outside UML (based heavily on
Alan's suggestion in the thread).
[1]: https://lore.kernel.org/linux-kselftest/20200602054216.93122-1-davidgow@goo…
Documentation/dev-tools/kunit/faq.rst | 40 +++++++++++++++++++++++++++
1 file changed, 40 insertions(+)
diff --git a/Documentation/dev-tools/kunit/faq.rst b/Documentation/dev-tools/kunit/faq.rst
index ea55b2467653..1628862e7024 100644
--- a/Documentation/dev-tools/kunit/faq.rst
+++ b/Documentation/dev-tools/kunit/faq.rst
@@ -61,3 +61,43 @@ test, or an end-to-end test.
kernel by installing a production configuration of the kernel on production
hardware with a production userspace and then trying to exercise some behavior
that depends on interactions between the hardware, the kernel, and userspace.
+
+KUnit isn't working, what should I do?
+======================================
+
+Unfortunately, there are a number of things which can break, but here are some
+things to try.
+
+1. Try running ``./tools/testing/kunit/kunit.py run`` with the ``--raw_output``
+ parameter. This might show details or error messages hidden by the kunit_tool
+ parser.
+2. Instead of running ``kunit.py run``, try running ``kunit.py config``,
+ ``kunit.py build``, and ``kunit.py exec`` independently. This can help track
+ down where an issue is occurring. (If you think the parser is at fault, you
+ can run it manually against stdin or a file with ``kunit.py parse``.)
+3. Running the UML kernel directly can often reveal issues or error messages
+ kunit_tool ignores. This should be as simple as running ``./vmlinux`` after
+ building the UML kernel (e.g., by using ``kunit.py build``). Note that UML
+ has some unusual requirements (such as the host having a tmpfs filesystem
+ mounted), and has had issues in the past when built statically and the host
+ has KASLR enabled. (On older host kernels, you may need to run ``setarch
+ `uname -m` -R ./vmlinux`` to disable KASLR.)
+4. Make sure the kernel .config has ``CONFIG_KUNIT=y`` and at least one test
+ (e.g. ``CONFIG_KUNIT_EXAMPLE_TEST=y``). kunit_tool will keep its .config
+ around, so you can see what config was used after running ``kunit.py run``.
+ It also preserves any config changes you might make, so you can
+ enable/disable things with ``make ARCH=um menuconfig`` or similar, and then
+ re-run kunit_tool.
+5. Try to run ``make ARCH=um defconfig`` before running ``kunit.py run``. This
+ may help clean up any residual config items which could be causing problems.
+6. Finally, try running KUnit outside UML. KUnit and KUnit tests can run be
+ built into any kernel, or can be built as a module and loaded at runtime.
+ Doing so should allow you to determine if UML is causing the issue you're
+ seeing. When tests are built-in, they will execute when the kernel boots, and
+ modules will automatically execute associated tests when loaded. Test results
+ can be collected from ``/sys/kernel/debug/kunit/<test suite>/results``, and
+ can be parsed with ``kunit.py parse``. For more details, see "KUnit on
+ non-UML architectures" in :doc:`usage`.
+
+If none of the above tricks help, you are always welcome to email any issues to
+kunit-dev(a)googlegroups.com.
--
2.27.0.rc2.251.g90737beb825-goog
The identation before this code (`if not os.path.exists(cli_args.build_dir):``)
was with spaces instead of tabs after fixed up merge conflits,
this commit revert spaces to tabs:
[iha@bbking linux]$ tools/testing/kunit/kunit.py run
File "tools/testing/kunit/kunit.py", line 247
if not linux:
^
TabError: inconsistent use of tabs and spaces in indentation
Remove defconfig related code to fix these two errors,
the commit 9bdf64b35 was created before 45ba7a893ad,
and the commit 9bdf64b35 removes defconfig related code:
[iha@bbking linux]$ tools/testing/kunit/kunit.py run
Traceback (most recent call last):
File "tools/testing/kunit/kunit.py", line 338, in <module>
main(sys.argv[1:])
File "tools/testing/kunit/kunit.py", line 215, in main
add_config_opts(config_parser)
[iha@bbking linux]$ tools/testing/kunit/kunit.py run
Traceback (most recent call last):
File "tools/testing/kunit/kunit.py", line 337, in <module>
main(sys.argv[1:])
File "tools/testing/kunit/kunit.py", line 255, in main
result = run_tests(linux, request)
File "tools/testing/kunit/kunit.py", line 133, in run_tests
request.defconfig,
AttributeError: 'KunitRequest' object has no attribute 'defconfig'
Handles when there is no .kunitconfig, the error occurs because
commit 9bdf64b35 was created before 45ba7a893ad.
[iha@bbking linux]$ tools/testing/kunit/kunit.py run
Traceback (most recent call last):
File "tools/testing/kunit/kunit.py", line 335, in <module>
main(sys.argv[1:])
File "tools/testing/kunit/kunit.py", line 246, in main
linux = kunit_kernel.LinuxSourceTree()
File "/home/iha/sdb/opensource/lkmp/linux/tools/testing/kunit/kunit_kernel.py", line 109, in __init__
self._kconfig.read_from_file(kunitconfig_path)
File "/home/iha/sdb/opensource/lkmp/linux/tools/testing/kunit/kunit_config.py", line 88, in read_from_file
with open(path, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '.kunit/.kunitconfig'
Signed-off-by: Vitor Massaru Iha <vitor(a)massaru.org>
---
Fix the fixup on this commits: 9bdf64b, ddbd60c.
Some errors occurs because these commits were created before
this commit 45ba7a8, as explained in the commit message.
---
tools/testing/kunit/kunit.py | 27 ++++++++++++++++++---------
1 file changed, 18 insertions(+), 9 deletions(-)
diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py
index ec73b07d1265..787b6d4ad716 100755
--- a/tools/testing/kunit/kunit.py
+++ b/tools/testing/kunit/kunit.py
@@ -23,7 +23,7 @@ import kunit_parser
KunitResult = namedtuple('KunitResult', ['status','result','elapsed_time'])
KunitConfigRequest = namedtuple('KunitConfigRequest',
- ['build_dir', 'defconfig', 'make_options'])
+ ['build_dir', 'make_options'])
KunitBuildRequest = namedtuple('KunitBuildRequest',
['jobs', 'build_dir', 'alltests',
'make_options'])
@@ -130,7 +130,6 @@ def run_tests(linux: kunit_kernel.LinuxSourceTree,
run_start = time.time()
config_request = KunitConfigRequest(request.build_dir,
- request.defconfig,
request.make_options)
config_result = config_tests(linux, config_request)
if config_result.status != KunitStatus.SUCCESS:
@@ -212,7 +211,6 @@ def main(argv, linux=None):
help='Ensures that .config contains all of '
'the options in .kunitconfig')
add_common_opts(config_parser)
- add_config_opts(config_parser)
build_parser = subparser.add_parser('build', help='Builds a kernel with KUnit tests')
add_common_opts(build_parser)
@@ -238,11 +236,14 @@ def main(argv, linux=None):
cli_args = parser.parse_args(argv)
if cli_args.subcommand == 'run':
- if not os.path.exists(cli_args.build_dir):
- os.mkdir(cli_args.build_dir)
- kunit_kernel.kunitconfig_path = os.path.join(
- cli_args.build_dir,
- kunit_kernel.kunitconfig_path)
+ if not os.path.exists(cli_args.build_dir):
+ os.mkdir(cli_args.build_dir)
+ kunit_kernel.kunitconfig_path = os.path.join(
+ cli_args.build_dir,
+ kunit_kernel.kunitconfig_path)
+
+ if not os.path.exists(kunit_kernel.kunitconfig_path):
+ create_default_kunitconfig()
if not linux:
linux = kunit_kernel.LinuxSourceTree()
@@ -264,11 +265,13 @@ def main(argv, linux=None):
cli_args.build_dir,
kunit_kernel.kunitconfig_path)
+ if not os.path.exists(kunit_kernel.kunitconfig_path):
+ create_default_kunitconfig()
+
if not linux:
linux = kunit_kernel.LinuxSourceTree()
request = KunitConfigRequest(cli_args.build_dir,
- cli_args.defconfig,
cli_args.make_options)
result = config_tests(linux, request)
kunit_parser.print_with_timestamp((
@@ -284,6 +287,9 @@ def main(argv, linux=None):
cli_args.build_dir,
kunit_kernel.kunitconfig_path)
+ if not os.path.exists(kunit_kernel.kunitconfig_path):
+ create_default_kunitconfig()
+
if not linux:
linux = kunit_kernel.LinuxSourceTree()
@@ -305,6 +311,9 @@ def main(argv, linux=None):
cli_args.build_dir,
kunit_kernel.kunitconfig_path)
+ if not os.path.exists(kunit_kernel.kunitconfig_path):
+ create_default_kunitconfig()
+
if not linux:
linux = kunit_kernel.LinuxSourceTree()
--
2.26.2
Hi,
This patchset will try to enable as many KUnit test fragments as
possible for the current .config file.
This will make it easier for both developers that tests their specific
feature and also for test-systems that would like to get as much as
possible for their current .config file.
I will send a separate KCSAN KUnit patch after this patchset since that
isn't in mainline yet.
Since v2:
Fixed David's comments. KUNIT_RUN_ALL -> KUNIT_ALL_TESTS, and he
suggested a great help text.
Since v1:
Marco commented to split up the patches, and change a "." to a ",".
Cheers,
Anders
Anders Roxell (6):
kunit: Kconfig: enable a KUNIT_ALL_TESTS fragment
kunit: default KUNIT_* fragments to KUNIT_ALL_TESTS
lib: Kconfig.debug: default KUNIT_* fragments to KUNIT_ALL_TESTS
drivers: base: default KUNIT_* fragments to KUNIT_ALL_TESTS
fs: ext4: default KUNIT_* fragments to KUNIT_ALL_TESTS
security: apparmor: default KUNIT_* fragments to KUNIT_ALL_TESTS
drivers/base/Kconfig | 3 ++-
drivers/base/test/Kconfig | 3 ++-
fs/ext4/Kconfig | 3 ++-
lib/Kconfig.debug | 6 ++++--
lib/kunit/Kconfig | 23 ++++++++++++++++++++---
security/apparmor/Kconfig | 3 ++-
6 files changed, 32 insertions(+), 9 deletions(-)
--
2.20.1
Add an FAQ entry to the KUnit documentation with some tips for
troubleshooting KUnit and kunit_tool.
These suggestions largely came from an email thread:
https://lore.kernel.org/linux-kselftest/41db8bbd-3ba0-8bde-7352-083bf4b947f…
Signed-off-by: David Gow <davidgow(a)google.com>
---
Documentation/dev-tools/kunit/faq.rst | 32 +++++++++++++++++++++++++++
1 file changed, 32 insertions(+)
diff --git a/Documentation/dev-tools/kunit/faq.rst b/Documentation/dev-tools/kunit/faq.rst
index ea55b2467653..40109d425988 100644
--- a/Documentation/dev-tools/kunit/faq.rst
+++ b/Documentation/dev-tools/kunit/faq.rst
@@ -61,3 +61,35 @@ test, or an end-to-end test.
kernel by installing a production configuration of the kernel on production
hardware with a production userspace and then trying to exercise some behavior
that depends on interactions between the hardware, the kernel, and userspace.
+
+KUnit isn't working, what should I do?
+======================================
+
+Unfortunately, there are a number of things which can break, but here are some
+things to try.
+
+1. Try running ``./tools/testing/kunit/kunit.py run`` with the ``--raw_output``
+ parameter. This might show details or error messages hidden by the kunit_tool
+ parser.
+2. Instead of running ``kunit.py run``, try running ``kunit.py config``,
+ ``kunit.py build``, and ``kunit.py exec`` independently. This can help track
+ down where an issue is occurring. (If you think the parser is at fault, you
+ can run it manually against stdin or a file with ``kunit.py parse``.)
+3. Running the UML kernel directly can often reveal issues or error messages
+ kunit_tool ignores. This should be as simple as running ``./vmlinux`` after
+ building the UML kernel (e.g., by using ``kunit.py build``). Note that UML
+ has some unusual requirements (such as the host having a tmpfs filesystem
+ mounted), and has had issues in the past when built statically and the host
+ has KASLR enabled. (On older host kernels, you may need to run ``setarch
+ `uname -m` -R ./vmlinux`` to disable KASLR.)
+4. Make sure the kernel .config has ``CONFIG_KUNIT=y`` and at least one test
+ (e.g. ``CONFIG_KUNIT_EXAMPLE_TEST=y``). kunit_tool will keep its .config
+ around, so you can see what config was used after running ``kunit.py run``.
+ It also preserves any config changes you might make, so you can
+ enable/disable things with ``make ARCH=um menuconfig`` or similar, and then
+ re-run kunit_tool.
+5. Finally, running ``make ARCH=um defconfig`` before running ``kunit.py run``
+ may help clean up any residual config items which could be causing problems.
+
+If none of the above tricks help, you are always welcome to email any issues to
+kunit-dev(a)googlegroups.com.
--
2.27.0.rc2.251.g90737beb825-goog
When running with conntrack rules, the dropped overlap fragments may cause
EPERM to be returned to sendto. Instead of completely failing, just ignore
those errors and continue. If this causes packets with overlap fragments to
be dropped as expected, that is okay. And if it causes packets that are
expected to be received to be dropped, which should not happen, it will be
detected as failure.
Signed-off-by: Thadeu Lima de Souza Cascardo <cascardo(a)canonical.com>
---
tools/testing/selftests/net/ip_defrag.c | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/tools/testing/selftests/net/ip_defrag.c b/tools/testing/selftests/net/ip_defrag.c
index b53fb67f8e5e..62ee927bacae 100644
--- a/tools/testing/selftests/net/ip_defrag.c
+++ b/tools/testing/selftests/net/ip_defrag.c
@@ -192,9 +192,9 @@ static void send_fragment(int fd_raw, struct sockaddr *addr, socklen_t alen,
}
res = sendto(fd_raw, ip_frame, frag_len, 0, addr, alen);
- if (res < 0)
+ if (res < 0 && errno != EPERM)
error(1, errno, "send_fragment");
- if (res != frag_len)
+ if (res >= 0 && res != frag_len)
error(1, 0, "send_fragment: %d vs %d", res, frag_len);
frag_counter++;
@@ -313,9 +313,9 @@ static void send_udp_frags(int fd_raw, struct sockaddr *addr,
iphdr->ip_len = htons(frag_len);
}
res = sendto(fd_raw, ip_frame, frag_len, 0, addr, alen);
- if (res < 0)
+ if (res < 0 && errno != EPERM)
error(1, errno, "sendto overlap: %d", frag_len);
- if (res != frag_len)
+ if (res >= 0 && res != frag_len)
error(1, 0, "sendto overlap: %d vs %d", (int)res, frag_len);
frag_counter++;
}
--
2.25.1