In the middle of the thread about a patch to add the skip test result,
I suggested documenting the process of deprecating the KTAP v1 Specification
method of marking a skipped test:
https://lore.kernel.org/all/490271eb-1429-2217-6e38-837c6e5e328b@gmail.com/…
In a reply to that email I suggested that we ought to have a process to transition
the KTAP Specification from v1 to v2, and possibly v3 and future.
This email is meant to be the root of that discussion.
My initial thinking is that there are at least three different types of project
and/or community that may have different needs in this area.
Type 1 - project controls both the test output generation and the test output
parsing tool. Both generation and parsing code are in the same repository
and/or synchronized versions are distributed together.
Devicetree unittests are an example of Type 1. I plan to maintain changes
of test output to KTAP v2 format in coordination with updating the parser
to process KTAP v2 data.
Type 2 - project controls both the test output generation and the test output
parsing tool. The test output generation and a parser modifications may be
controlled by the project BUT there are one or more external testing projects
that (1) may have their own parsers, and (2) may have a single framework that
tests multiple versions of the tests.
I think that kselftest and kunit tests are probably examples of Type 2. I also
think that DT unittests will become a Type 2 project as a result of converting
to KTAP v2 data.
Type 3 - project may create and maintain some tests, but is primarily a consumer
of tests created by other projects. Type 3 projects typically have a single
framework that is able to execute and process multiple versions of the tests.
The Fuego test project is an example of Type 3.
Maybe adding all of this complexity of different Types in my initial thinking
was silly -- maybe everything in this topic is governed by the more complex
Type 3.
My thinking was that the three different Types of project would be impacted
in different ways by transition plans. Type 3 would be the most impacted,
so I wanted to be sure that any transition plan especially considered their
needs.
There is an important aspect of the KTAP format that might ease the transition
from one version to another: All KTAP formatted results begin with a "version
line", so as soon as a parser has processed the first line of a test, it can
apply the appropriate KTAP Specification version to all subsequent lines of
test output. A parser implementation could choose to process all versions,
could choose to invoke a version specific parser, or some other approach
all together.
In the "add skip test results" thread, I suggested deprecating the v1
method of marking a skipped test in v2, with a scheduled removal of
the v1 method in v3. But since the KTAP format version is available
in the very first line of test output, is it necessary to do a slow
deprecation and removal over two versions?
One argument to doing a two version deprecation/removal process is that
a parser that is one version older the the test output _might_ be able
to process the test output without error, but would not be able to take
advantage of features added in the newer version of the Specification.
My opinion is that a two version deprecation/removal process will slow
the Specification update process and lead to more versions of the
Specification over a given time interval.
A one version deprecation/removal process puts more of a burden on Type 3
projects and external parsers for Type 2 projects to implement parsers
that can process the newer Specification more quickly and puts a burden
on test maintainers to delay a move to the newer Specification, or possibly
pressure to support selection of more than one Specification version format
for output data.
One additional item... On the KTAP Specification version 2 process wiki page,
I suggested that it is "desirable for test result parsers that understand the
KTAP Specification version 2 data also be able to parse version 1 data."
With the implication "Converting version 1 compliant data to version 2 compliant
data should not require a "flag day" switch of test result parsers." If this
thread discussion results in a different decision, I will update the wiki.
Thoughts?
-Frank
Hi everyone,
I am new to this forum and excited to be here.
I'm interested in learning more about Linux kernel self-tests and contributing where I can.
I look forward to engaging with you all and gaining a deeper understanding of the topics discussed here.
Could someone please guide me on how to ask questions here ?
Where should I post if I have a query ?
Looking forward to your advice and connecting with you all.
Thanks,
From: Geliang Tang <tanggeliang(a)kylinos.cn>
v2:
- Although all CI tests passed on x86_64 "bpf/vmtest-bpf-next-VM_Test-22
Logs for x86_64-gcc / test (test_progs, false, 360) / test_progs on x86_64
with gcc", some unexpect "SKIP"s are showed in the log:
#29/1 bpf_tcp_ca/dctcp:SKIP
#29/2 bpf_tcp_ca/cubic:OK
#29/3 bpf_tcp_ca/invalid_license:OK
#29/4 bpf_tcp_ca/dctcp_fallback:SKIP
#29/5 bpf_tcp_ca/rel_setsockopt:OK
#29/6 bpf_tcp_ca/write_sk_pacing:OK
#29/7 bpf_tcp_ca/incompl_cong_ops:OK
#29/8 bpf_tcp_ca/unsupp_cong_op:OK
#29/9 bpf_tcp_ca/update_ca:OK
#29/10 bpf_tcp_ca/update_wrong:OK
#29/11 bpf_tcp_ca/mixed_links:OK
#29/12 bpf_tcp_ca/multi_links:OK
#29/13 bpf_tcp_ca/link_replace:OK
#29/14 bpf_tcp_ca/tcp_ca_kfunc:OK
#29/15 bpf_tcp_ca/cc_cubic:OK
#29/16 bpf_tcp_ca/dctcp_autoattach_map:SKIP
#29 bpf_tcp_ca:OK (SKIP: 3/16)
Shouldn't skip any tests on X86_64. Fix this in v2.
- add a new helper test_progs_get_error.
BPF selftests seem to have not been fully tested on Loongarch platforms.
There are so many "ENOTSUPP" (-524) errors when running BPF selftests on
them since lacking BPF trampoline on Loongarch.
For these "ENOTSUPP" tests, it's better to skip them, instead of reporting
some "ENOTSUPP" errors. This patchset skips ENOTSUPP in ASSERT_OK/
ASSERT_OK_PTR/ASSERT_GE helpers to fix them. This is useful for running BPF
selftests for other architectures too.
Geliang Tang (6):
selftests/bpf: Define ENOTSUPP in testing_helpers.h
selftests/bpf: Skip ENOTSUPP in ASSERT_OK
selftests/bpf: Use ASSERT_OK to skip ENOTSUPP
selftests/bpf: Null checks for link in bpf_tcp_ca
selftests/bpf: Skip ENOTSUPP in ASSERT_OK_PTR
selftests/bpf: Skip ENOTSUPP in ASSERT_GE
.../selftests/bpf/prog_tests/bpf_tcp_ca.c | 20 ++++++-----
.../testing/selftests/bpf/prog_tests/d_path.c | 2 +-
.../selftests/bpf/prog_tests/lsm_cgroup.c | 10 +-----
.../selftests/bpf/prog_tests/module_attach.c | 2 +-
.../selftests/bpf/prog_tests/ringbuf.c | 2 +-
.../selftests/bpf/prog_tests/sock_addr.c | 4 ---
.../selftests/bpf/prog_tests/test_bprm_opts.c | 2 +-
.../selftests/bpf/prog_tests/test_ima.c | 2 +-
.../selftests/bpf/prog_tests/trace_ext.c | 2 +-
tools/testing/selftests/bpf/test_maps.c | 4 ---
tools/testing/selftests/bpf/test_progs.h | 33 +++++++++++++++----
tools/testing/selftests/bpf/test_verifier.c | 4 ---
tools/testing/selftests/bpf/testing_helpers.h | 4 +++
13 files changed, 50 insertions(+), 41 deletions(-)
--
2.43.0
Conform individual tests to TAP output. One patch conform one test. With
this series, all vDSO tests become TAP conformant.
First patch conform the test by using kselftest_harness.h. Other patches
are conforming using default kselftest.h helpers.
All tests have been tested multiple times before and after these
patches. They are working correctly and outputting TAP messaging to find
failures quikly when they happen.
---
Changes since v1:
- Update cover letter
- Update commit message of first patch
Muhammad Usama Anjum (4):
kselftests: vdso: vdso_test_clock_getres: conform test to TAP output
kselftests: vdso: vdso_test_correctness: conform test to TAP output
kselftests: vdso: vdso_test_getcpu: conform test to TAP output
kselftests: vdso: vdso_test_gettimeofday: conform test to TAP output
.../selftests/vDSO/vdso_test_clock_getres.c | 68 ++++----
.../selftests/vDSO/vdso_test_correctness.c | 146 +++++++++---------
.../testing/selftests/vDSO/vdso_test_getcpu.c | 16 +-
.../selftests/vDSO/vdso_test_gettimeofday.c | 23 +--
4 files changed, 126 insertions(+), 127 deletions(-)
--
2.39.2
Changes since v3:
1) Rebased onto Linux 6.10-rc6+.
2) Added Muhammad's acks for the series.
Cover letter for v3:
Hi,
Dave Hansen, Muhammad Usama Anjum, here is the combined series that we
discussed yesterday [1].
As I mentioned then, this is a bit intrusive--but no more than
necessary, IMHO. Specifically, it moves some clang-un-inlineable things
out to "pure" assembly code files.
I've tested this by building with clang, then running each binary on my
x86_64 test system with today's 6.10-rc1, and comparing the console and
dmesg output to a gcc-based build without these patches applied. Aside
from timestamps and virtual addresses, it looks identical.
Earlier cover letter:
Just a bunch of build and warnings fixes that show up when building with
clang. Some of these depend on each other, so I'm sending them as a
series.
Changes since v2:
1) Dropped my test_FISTTP.c patch, and picked up Muhammad's fix instead,
seeing as how that was posted first.
2) Updated patch descriptions to reflect that Valentin Obst's build fix
for LLVM [1] has already been merged into Linux main.
3) Minor wording and typo corrections in the commit logs throughout.
Changes since the first version:
1) Rebased onto Linux 6.10-rc1
Enjoy!
[1] https://lore.kernel.org/44428518-4d21-4de7-8587-04eceefb330d@nvidia.com
thanks,
John Hubbard
John Hubbard (6):
selftests/x86: fix Makefile dependencies to work with clang
selftests/x86: build fsgsbase_restore.c with clang
selftests/x86: build sysret_rip.c with clang
selftests/x86: avoid -no-pie warnings from clang during compilation
selftests/x86: remove (or use) unused variables and functions
selftests/x86: fix printk warnings reported by clang
Muhammad Usama Anjum (1):
selftests: x86: test_FISTTP: use fisttps instead of ambiguous fisttp
tools/testing/selftests/x86/Makefile | 31 +++++++++++++++----
tools/testing/selftests/x86/amx.c | 16 ----------
.../testing/selftests/x86/clang_helpers_32.S | 11 +++++++
.../testing/selftests/x86/clang_helpers_64.S | 28 +++++++++++++++++
tools/testing/selftests/x86/fsgsbase.c | 6 ----
.../testing/selftests/x86/fsgsbase_restore.c | 11 +++----
tools/testing/selftests/x86/sigreturn.c | 2 +-
.../testing/selftests/x86/syscall_arg_fault.c | 1 -
tools/testing/selftests/x86/sysret_rip.c | 20 ++++--------
tools/testing/selftests/x86/test_FISTTP.c | 8 ++---
tools/testing/selftests/x86/test_vsyscall.c | 15 +++------
tools/testing/selftests/x86/vdso_restorer.c | 2 ++
12 files changed, 87 insertions(+), 64 deletions(-)
create mode 100644 tools/testing/selftests/x86/clang_helpers_32.S
create mode 100644 tools/testing/selftests/x86/clang_helpers_64.S
base-commit: 795c58e4c7fc6163d8fb9f2baa86cfe898fa4b19
--
2.45.2
This patch series adds unit tests for the clk fixed rate basic type and
the clk registration functions that use struct clk_parent_data. To get
there, we add support for loading device tree overlays onto the live DTB
along with probing platform drivers to bind to device nodes in the
overlays. With this series, we're able to exercise some of the code in
the common clk framework that uses devicetree lookups to find parents
and the fixed rate clk code that scans device tree directly and creates
clks. Please review.
I Cced everyone to all the patches so they get the full context. I'm
hoping I can take the whole pile through the clk tree as they all build
upon each other. Or the DT part can be merged through the DT tree to
reduce the dependencies.
Changes from v5: https://lore.kernel.org/r/20240603223811.3815762-1-sboyd@kernel.org
* Pick up reviewed-by tags
* Drop test vendor prefix bindings as dtschema allows anything now
* Use of_node_put_kunit() more to plug some reference leaks
* Select DTC config to avoid compile fails because of missing dtc
* Don't skip for OF_OVERLAY in overlay tests because they depend on it
Changes from v4: https://lore.kernel.org/r/20240422232404.213174-1-sboyd@kernel.org
* Picked up reviewed-by tags
* Check for non-NULL device pointers before calling put_device()
* Fix CFI issues with kunit actions
* Introduce platform_device_prepare_wait_for_probe() helper to wait for
a platform device to probe
* Move platform code to lib/kunit and rename functions to have kunit
prefix
* Fix issue with platform wrappers messing up reference counting
because they used kunit actions
* New patch to populate overlay devices on root node for powerpc
* Make fixed-rate binding generic single clk consumer binding
Changes from v3: https://lore.kernel.org/r/20230327222159.3509818-1-sboyd@kernel.org
* No longer depend on Frank's series[1] because it was merged upstream[2]
* Use kunit_add_action_or_reset() to shorten code
* Skip tests properly when CONFIG_OF_OVERLAY isn't set
Changes from v2: https://lore.kernel.org/r/20230315183729.2376178-1-sboyd@kernel.org
* Overlays don't depend on __symbols__ node
* Depend on Frank's always create root node if CONFIG_OF series[1]
* Added kernel-doc to KUnit API doc
* Fixed some kernel-doc on functions
* More test cases for fixed rate clk
Changes from v1: https://lore.kernel.org/r/20230302013822.1808711-1-sboyd@kernel.org
* Don't depend on UML, use unittest data approach to attach nodes
* Introduce overlay loading API for KUnit
* Move platform_device KUnit code to drivers/base/test
* Use #define macros for constants shared between unit tests and
overlays
* Settle on "test" as a vendor prefix
* Make KUnit wrappers have "_kunit" postfix
[1] https://lore.kernel.org/r/20230317053415.2254616-1-frowand.list@gmail.com
[2] https://lore.kernel.org/r/20240308195737.GA1174908-robh@kernel.org
Stephen Boyd (8):
of/platform: Allow overlays to create platform devices from the root
node
of: Add test managed wrappers for of_overlay_apply()/of_node_put()
dt-bindings: vendor-prefixes: Add "test" vendor for KUnit and friends
of: Add a KUnit test for overlays and test managed APIs
platform: Add test managed platform_device/driver APIs
clk: Add test managed clk provider/consumer APIs
clk: Add KUnit tests for clk fixed rate basic type
clk: Add KUnit tests for clks registered with struct clk_parent_data
Documentation/dev-tools/kunit/api/clk.rst | 10 +
Documentation/dev-tools/kunit/api/index.rst | 21 +
Documentation/dev-tools/kunit/api/of.rst | 13 +
.../dev-tools/kunit/api/platformdevice.rst | 10 +
.../devicetree/bindings/vendor-prefixes.yaml | 2 +
drivers/clk/.kunitconfig | 2 +
drivers/clk/Kconfig | 11 +
drivers/clk/Makefile | 9 +-
drivers/clk/clk-fixed-rate_test.c | 379 +++++++++++++++
drivers/clk/clk-fixed-rate_test.h | 8 +
drivers/clk/clk_kunit_helpers.c | 204 ++++++++
drivers/clk/clk_parent_data_test.h | 10 +
drivers/clk/clk_test.c | 453 +++++++++++++++++-
drivers/clk/kunit_clk_fixed_rate_test.dtso | 19 +
drivers/clk/kunit_clk_parent_data_test.dtso | 28 ++
drivers/of/.kunitconfig | 1 +
drivers/of/Kconfig | 10 +
drivers/of/Makefile | 2 +
drivers/of/kunit_overlay_test.dtso | 9 +
drivers/of/of_kunit_helpers.c | 74 +++
drivers/of/overlay_test.c | 114 +++++
drivers/of/platform.c | 9 +-
include/kunit/clk.h | 28 ++
include/kunit/of.h | 115 +++++
include/kunit/platform_device.h | 20 +
lib/kunit/Makefile | 4 +-
lib/kunit/platform-test.c | 223 +++++++++
lib/kunit/platform.c | 302 ++++++++++++
28 files changed, 2084 insertions(+), 6 deletions(-)
create mode 100644 Documentation/dev-tools/kunit/api/clk.rst
create mode 100644 Documentation/dev-tools/kunit/api/of.rst
create mode 100644 Documentation/dev-tools/kunit/api/platformdevice.rst
create mode 100644 drivers/clk/clk-fixed-rate_test.c
create mode 100644 drivers/clk/clk-fixed-rate_test.h
create mode 100644 drivers/clk/clk_kunit_helpers.c
create mode 100644 drivers/clk/clk_parent_data_test.h
create mode 100644 drivers/clk/kunit_clk_fixed_rate_test.dtso
create mode 100644 drivers/clk/kunit_clk_parent_data_test.dtso
create mode 100644 drivers/of/kunit_overlay_test.dtso
create mode 100644 drivers/of/of_kunit_helpers.c
create mode 100644 drivers/of/overlay_test.c
create mode 100644 include/kunit/clk.h
create mode 100644 include/kunit/of.h
create mode 100644 include/kunit/platform_device.h
create mode 100644 lib/kunit/platform-test.c
create mode 100644 lib/kunit/platform.c
base-commit: 1613e604df0cd359cf2a7fbd9be7a0bcfacfabd0
--
https://git.kernel.org/pub/scm/linux/kernel/git/clk/linux.git/https://git.kernel.org/pub/scm/linux/kernel/git/sboyd/spmi.git