On Fri, 2019-05-10 at 10:12 +0200, Daniel Vetter wrote:
On Fri, May 10, 2019 at 7:49 AM Knut Omang knut.omang@oracle.com wrote:
On Thu, 2019-05-09 at 22:18 -0700, Frank Rowand wrote:
On 5/9/19 4:40 PM, Logan Gunthorpe wrote:
On 2019-05-09 5:30 p.m., Theodore Ts'o wrote:
On Thu, May 09, 2019 at 04:20:05PM -0600, Logan Gunthorpe wrote:
The second item, arguably, does have significant overlap with kselftest. Whether you are running short tests in a light weight UML environment or higher level tests in an heavier VM the two could be using the same framework for writing or defining in-kernel tests. It *may* also be valuable for some people to be able to run all the UML tests in the heavy VM environment along side other higher level tests.
Looking at the selftests tree in the repo, we already have similar items to what Kunit is adding as I described in point (2) above. kselftest_harness.h contains macros like EXPECT_* and ASSERT_* with very similar intentions to the new KUNIT_EXECPT_* and KUNIT_ASSERT_* macros.
However, the number of users of this harness appears to be quite small. Most of the code in the selftests tree seems to be a random mismash of scripts and userspace code so it's not hard to see it as something completely different from the new Kunit:
$ git grep --files-with-matches kselftest_harness.h *
To the extent that we can unify how tests are written, I agree that this would be a good thing. However, you should note that kselftest_harness.h is currently assums that it will be included in userspace programs. This is most obviously seen if you look closely at the functions defined in the header files which makes calls to fork(), abort() and fprintf().
Ah, yes. I obviously did not dig deep enough. Using kunit for in-kernel tests and kselftest_harness for userspace tests seems like a sensible line to draw to me. Trying to unify kernel and userspace here sounds like it could be difficult so it's probably not worth forcing the issue unless someone wants to do some really fancy work to get it done.
Based on some of the other commenters, I was under the impression that kselftests had in-kernel tests but I'm not sure where or if they exist.
YES, kselftest has in-kernel tests. (Excuse the shouting...)
Here is a likely list of them in the kernel source tree:
$ grep module_init lib/test_*.c lib/test_bitfield.c:module_init(test_bitfields) lib/test_bitmap.c:module_init(test_bitmap_init); lib/test_bpf.c:module_init(test_bpf_init); lib/test_debug_virtual.c:module_init(test_debug_virtual_init); lib/test_firmware.c:module_init(test_firmware_init); lib/test_hash.c:module_init(test_hash_init); /* Does everything */ lib/test_hexdump.c:module_init(test_hexdump_init); lib/test_ida.c:module_init(ida_checks); lib/test_kasan.c:module_init(kmalloc_tests_init); lib/test_list_sort.c:module_init(list_sort_test); lib/test_memcat_p.c:module_init(test_memcat_p_init); lib/test_module.c:static int __init test_module_init(void) lib/test_module.c:module_init(test_module_init); lib/test_objagg.c:module_init(test_objagg_init); lib/test_overflow.c:static int __init test_module_init(void) lib/test_overflow.c:module_init(test_module_init); lib/test_parman.c:module_init(test_parman_init); lib/test_printf.c:module_init(test_printf_init); lib/test_rhashtable.c:module_init(test_rht_init); lib/test_siphash.c:module_init(siphash_test_init); lib/test_sort.c:module_init(test_sort_init); lib/test_stackinit.c:module_init(test_stackinit_init); lib/test_static_key_base.c:module_init(test_static_key_base_init); lib/test_static_keys.c:module_init(test_static_key_init); lib/test_string.c:module_init(string_selftest_init); lib/test_ubsan.c:module_init(test_ubsan_init); lib/test_user_copy.c:module_init(test_user_copy_init); lib/test_uuid.c:module_init(test_uuid_init); lib/test_vmalloc.c:module_init(vmalloc_test_init) lib/test_xarray.c:module_init(xarray_checks);
If they do exists, it seems like it would make sense to convert those to kunit and have Kunit tests run-able in a VM or baremetal instance.
They already run in a VM.
They already run on bare metal.
They already run in UML.
This is not to say that KUnit does not make sense. But I'm still trying to get a better description of the KUnit features (and there are some).
FYI, I have a master student who looks at converting some of these to KTF, such as for instance the XArray tests, which lended themselves quite good to a semi-automated conversion.
The result is also a somewhat more compact code as well as the flexibility provided by the Googletest executor and the KTF frameworks, such as running selected tests, output formatting, debugging features etc.
So is KTF already in upstream?
No..
Or is the plan to unify the KTF and Kunit in-kernel test harnesses?
Since KTF delegates reporting and test running to user space - reporting is based on netlink communication with the user land frontend (Googletest based, but can in principle be ported to any user land framework if need be) - little specific test harness features are needed for KTF, so there's little if no direct overlap with the infrastructure in kselftests (as far as I understand, I'd like to spend more time on the details there..)
Because there's tons of these in-kernel unit tests already, and every merge we get more (Frank's list didn't even look into drivers or anywhere else, e.g. it's missing the locking self tests I worked on in the past), and a more structured approach would really be good.
That's been my thinking too. Having a unified way to gradually move to would benefit anyone who needs to relate to these tests in that there will be less to learn. But that would be a long term evolutionary goal of course.
Also I think each of these test suites/sets would benefit from being more available for running even in kernels not made specifically for testing, and I think using the module structure and a lean, common module (ktf.ko) as a library for the tests, but no "polluting" of the "production" kernel code with anything else, is a kind of a policy that comes implicitly with KTF that can make it easier to maintain a sort of standard "language" for writing kernel tests.
Wrt KTF, we're working on making a suitable patch set for the kernel, but also have projects running internally that uses KTF as a standalone git repository, and that inspires new features and other changes, as well as a growing set of tests. I want to make sure we find a good candidate kernel integration that can coexist nicely with the separate git repo version without becoming too much of a back/forward porting challenge.
Knut
-Daniel