Signed-off-by: Knut Omang knut.omang@oracle.com --- Documentation/dev-tools/ktf/concepts.rst | 242 ++++++++++++++- Documentation/dev-tools/ktf/debugging.rst | 248 +++++++++++++++- Documentation/dev-tools/ktf/examples.rst | 26 ++- Documentation/dev-tools/ktf/features.rst | 307 ++++++++++++++++++- Documentation/dev-tools/ktf/implementation.rst | 70 ++++- Documentation/dev-tools/ktf/index.rst | 14 +- Documentation/dev-tools/ktf/installation.rst | 73 ++++- Documentation/dev-tools/ktf/introduction.rst | 134 ++++++++- Documentation/dev-tools/ktf/progref.rst | 144 ++++++++- 9 files changed, 1258 insertions(+) create mode 100644 Documentation/dev-tools/ktf/concepts.rst create mode 100644 Documentation/dev-tools/ktf/debugging.rst create mode 100644 Documentation/dev-tools/ktf/examples.rst create mode 100644 Documentation/dev-tools/ktf/features.rst create mode 100644 Documentation/dev-tools/ktf/implementation.rst create mode 100644 Documentation/dev-tools/ktf/index.rst create mode 100644 Documentation/dev-tools/ktf/installation.rst create mode 100644 Documentation/dev-tools/ktf/introduction.rst create mode 100644 Documentation/dev-tools/ktf/progref.rst
diff --git a/Documentation/dev-tools/ktf/concepts.rst b/Documentation/dev-tools/ktf/concepts.rst new file mode 100644 index 0000000..9b9ef1a --- /dev/null +++ b/Documentation/dev-tools/ktf/concepts.rst @@ -0,0 +1,242 @@ + +5. KTF Basic Concepts +--------------------- + +Tests and test suites +********************* + +The simplest form of test is to just specify:: + + TEST(suite_name, test_name) + { + <test code and assertions go here> + } + +A KTF test is declared with TEST() or TEST_F(), which both +takes both a test suite name and a test name, which are two different +name spaces. Consequently, each test belongs to one test suite, and +the test suites are created based on what tests that exists. +A test suite is just a container of tests which in user space +contributes to the extended name of a test. Test names must be +unique within a suite, and test names must also be unique within a +source file, since the test name is the only parameter needed +when adding a test. + +All tests must be added using ADD_TEST or ADD_LOOP_TEST to be visible +to KTF's runtime framework. This allows tests to be declared while +under development, but not added (or the ADD_TEST could be commented +out) if the test or the kernel module under test is not ready +yet for some reason. + +Test fixtures +************* + +As in other unit test frameworks, a test fixture is a mechanism to +allow several tests to run under the same conditions, in that setup +and teardown is done before and after each test. In KTF a test fixture +must first be declared with DECLARE_F() which takes a fixture name +followed by a list of attributes and an end brace, and initialized +with INIT_F() which takes the fixture name and a setup and teardown +function to be defined subsequently. Note that there are +no start brace, which is intentional:: + + DECLARE_F(a_fixture) + int value; + struct my_details; + ... + }; + INIT_F(a_fixture, a_setup, a_teardown); + +Then to the implementation of the fixture, in the form of actual setup and +a teardown functions that may operate on the attributes of the fixture:: + + SETUP_F(a_fixture, a_setup) + { + a_fixture->value = 42; /* or whatever.. */ + <other actions needed to set up> + /* If everything went well during setup: */ + a_fixture->ok = true; + } + + TEARDOWN_F(a_fixture, a_teardown) + { + <necessary cleanup code> + } + +Now individual tests that uses this fixture can be declared with:: + + TEST_F(a_fixture, suite_name, test_name) + { + <test code> + } + +Contexts +******** + +A context provides a way to instantiate a test in multiple ways. +A typical use case is if you have multiple similar devices +you want to run a set of tests on. Another use case could be that +you want to run a set of tests under different configurations. + +You are free to let the number and names of these contexts +vary as to where you run your test. For the devices use case, you can +have the init function loop through all available devices, to identify +the ones the tests applies to, then instantiate a context for each +device, possibly using the device name for trackability. The context +names will be prepended to the test name and the number of available +tests will be multipled by the number of contexts. + +Note that the state of a context persists through the whole "life" of +the module (until it gets unloaded) so it can be used to store more +long term bookeeping in addition to any configuration information. +The test writer must make sure that subsequent runs of the test suite +(or parallel runs!) does not interfere with +each other. Similar to fixtures, there's a generic part that KTF uses, +and it can be extended the normal way. Make sure to declare the +test specific context struct type with an element named:: + + struct ktf_context k; + +typically as the first element of the struct, then you can continue +with whatever other datastructure desired. A test module can declare +and use as many contexts as desired. Note that contexts are associated +with and unique within a ``handle`` (see below). So if you need +to use a different set of contexts for different tests, you need to +put these contexts and tests into different handles. + +A context can be added using something like:: + + KTF_CONTEXT_ADD(&my_struct.k, "mycfg") + +where the first argument is a reference to the ktf_context structure +within the test specific structure, and the second argument is a text +name to use to refer to the context. Once one or more contexts exists +for a handle, tests for that handle will show up with names postfixed +by the context name, and there will be a distinct version of the test +for each context, e.g if a handle has contexts named ``c1`` and +``c2``, and tests declared with ``TEST(x, t1)`` and ``TEST(x, t2)``, +then this will manifest as 4 tests:: + + x.t1_c1 + x.t1_c2 + x.t2_c1 + x.t2_c2 + +Tests that depends on having a context +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +It is likely that once you have a set of tests that uses contexts, +that they also depends on a context being available, e.g. that the +``ctx`` variable inside a test points to a non-nil context. One example +use case for contexts would be a certain class of device. Such a +device might or might not be available in the system. If one or more +devices are available, you might want to have tests named +``c.t1_dev1`` and ``c.t1_dev2`` where ``dev1`` and ``dev2`` are the +device names for these devices in the system, but never have any +``c.t1`` test without a device. You can +enforce this by letting tests associated with a handle requiring a +context to even show up as a test in the list or be available for +execution. Instead of using ``KTF_INIT()`` or ``KTF_HANDLE_INIT()``, +use ``KTF_CTX_INIT()`` or ``KTF_HANDLE_CTX_INIT()``. + +Configurable contexts +~~~~~~~~~~~~~~~~~~~~~ +Sometimes it might be useful to be able to configure a context for the +execution of some (or all) of the tests using the context. +This can be because the system the tests are running on might have +different hardware or software capabilities, or might rely on +differing device or network setup or naming. Typically we want a unit +test suite to have as little configuration and parameterization as +possible, so recommended use is for parameters that is not directly +related to the operation of the test, but more for situations where +parameters outside the test itself needs to be set up, such as connect +details for a network service to test against, or a peer unit test +process for network related tests that require more than one +system to run. To specify a configurable context, use:: + + int my_cfg_callback(struct ktf_context *ctx, const void* data, size_t data_sz); + + KTF_CONTEXT_ADD_CFG(&my_struct.k, "mycfg", my_cfg_callback, type_id) + +The ``data`` pointer (and it's length) should be provided from user +space, and it is up to the test specific user space and kernel space +code to decide with the configuration is all about. If 0 is returned, +KTF considers the context to be configured, otherwise it will retain +it's current state, which will initially be unconfigured. +The callback return value is stored as an errno value in ``ktf_context`` in the +variable ``config_errno``, which will initially be set to ``ENOENT``, +to indicate unconfigured. The test can use this value +to decide what to do, such as failing with a message about missing +configuration or just silently pass and ignore the case if not +configured. The ``type_id`` parameter is used as a unique +identifier for the kernel side to decide how to interpret the +parameter, which is useful if different contexts wants to implement +very different configuration options. It also allows two different +test modules to use the same context names but with different +parameters by using different context types. + +In the user space part of the test, configuration information +can be set for a context using:: + + KTF_CONTEXT_CFG(name, type_id, parameter_type, parameter_ref) + +A simple example of a configurable test can be seen in +the selftests test in ``selftest/context.c`` (kernel part) and +``user/context.cpp`` (user part) and the header file +``selftest/context_self.h`` shared between user space and kernel space. + +Context types and user space created contexts +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Contexts belong to a ``context type``, which is a mechanism to group +contexts into types that have similar properties. It is up to the +kernel test module using these contexts what the meaning of this is, +but a simple semantics can be that all contexts of a certain type has +the same parameter block and the type ID can be used to check what +type of context it is before trying to resolve or verify the +parameters. For contexts pre-created by the kernel module, type IDs +can be freely selected and does not have any further meaning beyond +this. + +Context types can however also be used to selectively allow user +space to dynamically create new contexts of a specific type. To enable +such functionality, the kernel test module will need to enable it for +one or more context types. This is done by means of the following call:: + + ktf_handle_add_ctx_type(struct ktf_handle *handle, + struct ktf_context_type *type) + +kernel side call, which tells KTF that a new context type with a given type ID +permits user applications to create new contexts. This is useful for +instance if user parameters or other information most easily +obtainable from user land at test runtime is most easily available +from user space. + +Handles +******* + +Unlike user land unit test frameworks, which can rely on everything +being cleaned up when the test program finished, KTF and test writers +must pay the normal kernel level attention to allocations, references and +deallocations. + +KTF itself uses the concept of a *handle* to track tests, +test suites and contexts associated with a kernel module. +Contexts are also associated with a handle. Since the availability of +contexts for a handle determines the availability of tests and the +naming of them, it can be useful to have separate spaces for tests +that relies on some context and tests that do not, to avoid +aggregating up multiple test cases that are identical. +Handles thus also have a namespace effect in that it is possible to +have two contexts with the same name, and possibly a different type, +by putting them in different handles. + +The simplest mode of usage is for each module to use KTF_INIT() and +KTF_CLEANUP() in it's __init and __exit functions. KTF_INIT implicitly declares and +initializes a global handle __test_handle that gets cleaned up again +in KTF_CLEANUP, making the handle something a test developer does not +need to think too much about. However, sometimes a KTF kernel module +may be such organized that it makes sense to use more than one handle. +KTF allows the creation/cleanup of explicitly named handles by means of +KTF_HANDLE_INIT(name) and KTF_HANDLE_CLEANUP(name). This can be used +as an alternative to KTF_INIT()/KTF_CLEANUP() but requires the use of +ADD_TEST_TO(handle, testname) instead of the normal ADD_TEST(testname) +for adding tests to be executed. diff --git a/Documentation/dev-tools/ktf/debugging.rst b/Documentation/dev-tools/ktf/debugging.rst new file mode 100644 index 0000000..4f1d0ff --- /dev/null +++ b/Documentation/dev-tools/ktf/debugging.rst @@ -0,0 +1,248 @@ +8. Debugging KTF +-------------------- + +Structured debugging/tracing by printf +====================================== + +The kernel side of KTF implements it's own simple printk based logging +abstraction. You can create log entries by means of calls to the macro +tlog(level, format,...) - level is a bitmask where individual bits +represents log classes. You can read and set this bitmask via a +writable ``/sys`` file:: + + # cat /sys/module/ktf/parameters/debug_mask + 0x1 + echo 0x3 > /sys/module/ktf/parameters/debug_mask + # cat /sys/module/ktf/parameters/debug_mask + 0x3 + +The default value of ``debug_mask`` is 0x1 = INFO level. +Bits are by convention supposed to be +ordered by verbosity, with the lowest bits reserved for low volume, +important messages, while the higher bist are left for more verbose +debugging. You can also use this mechanism for simple debugging of KTF's +interaction with your tests, as the core KTF code contains some log +statements to make it easier to follow and debug involved +instances of KTF objects. + +Similarly, the user library implementing the interaction with the +user land test runner can log details about this. You can enable such +logging by providing a similar bitmask via the environment variable +KTF_DEBUG_MASK. + +Debugging fatal errors in tests +=============================== + +So your KTF test crashed the kernel? Let's see how you can use crash to +examine KTF test cases, individual test logs and see which test is running. + +First step is we need to load symbols for KTF. To get text, data and +bss section locations for the ktf module (assuming it's currently +loaded):: + + # cd /sys/module/ktf/sections + # cat .text .data .bss + 0xffffffffa0bdb000 + 0xffffffffa0bdf000 + 0xffffffffa0bdf7a0 + +Now run crash on your corefile (or /proc/kcore for a live kernel):: + + # crash <path-to-Symbol.map> <path-to-vmlinux> <path-to-core> + crash> add-symbol-file /path/to/kernel/ktf.ko 0xffffffffa0bdb000 -s .data 0xffffffffa0bdf000 -s .bss 0xffffffffa0bdf7a0 + +Now we can see the global test_cases rbtree via the handy +"tree" command. It displays an rbtree, and because test_cases +is an rbtree under the hood we can display the set of test +cases as follows:: + + crash> tree -t rbtree -s ktf_case test_cases + ffff88036f710c00 + struct ktf_case { + kmap = { + node = { + __rb_parent_color = 1, + rb_right = 0x0, + rb_left = 0x0 + }, + key = "selftest\000cov\000probereturn\000probeentry\000wrongversion\000dummy\000simplemap", + map = 0xffffffffa0bdd1a0 <test_cases>, + refcount = { + refcount = { + counter = 2 + } + } + }, + tests = { + root = { + rb_node = 0xffff880250ac4a00 + }, + size = 5, + lock = { + { + rlock = { + raw_lock = { + { + head_tail = 655370, + tickets = { + head = 10, + tail = 10 + } + } + } + } + } + }, + elem_comparefn = 0x0, + elem_freefn = 0xffffffffa0bd8760 <ktf_test_free> + }, + debugfs = { + debugfs_results_testset = 0xffff88021a18a3c0, + debugfs_results_test = 0xffff88021a18aa80, + debugfs_run_testset = 0xffff88021a18a300, + debugfs_run_test = 0xffff88021a18a840 + } + } + +Here we had 1 test case - from the "key" field +we can see it is called "selftest" - in fact it is +KTF's self tests. Within that one test cases we see +the rbtree for the indivdual selftest tests has a root +rb_node:: + + tests = { + root = { + rb_node = 0xffff880250ac4a00 + }, + +By printing _that_ tree of ktf_test structures from +root node (-N) 0xffff880250ac4a00 we can see our +individual tests:: + + crash> tree -t rbtree -s ktf_test -N 0xffff880250ac4a00 + ffff880250ac4a00 + struct ktf_test { + kmap = { + node = { + __rb_parent_color = 1, + rb_right = 0xffff880250ac5b00, + rb_left = 0xffff880250ac5d00 + }, + key = "probeentry\000wrongversion\000dummy\000simplemap\000\000\000\000\000\020\276\240\377\377\377\377 \020\276\240\377\377\377\377@\020\276\240\377", + map = 0xffff88036f710c68, + refcount = { + refcount = { + counter = 2 + } + } + }, + tclass = 0xffffffffa0be41a4 "selftest", + name = 0xffffffffa0be41bd "probeentry", + fun = 0xffffffffa0be1920, + start = 0, + end = 1, + skb = 0xffff88003fc03800, + log = 0xffff88003fa58000 "", + lastrun = { + tv_sec = 1506072537, + tv_nsec = 289494591 + }, + debugfs = { + debugfs_results_testset = 0x0, + debugfs_results_test = 0xffff88021a18ac00, + debugfs_run_testset = 0x0, + debugfs_run_test = 0xffff88021a18af00 + }, + handle = 0xffffffffa0be5480 + } + ffff880250ac5d00 + struct ktf_test { + kmap = { + node = { + __rb_parent_color = 18446612142257621505, + rb_right = 0x0, + rb_left = 0xffff880250ac4b00 + }, + key = "dummy\000simplemap\000\000\000\000\000\020\276\240\377\377\377\377 \020\276\240\377\377\377\377@\020\276\240\377\377\377\377`\020\276\240\377\377\377\377\200\020\276\240\377\377\377\377\320\020\276\240\377", + map = 0xffff88036f710c68, + refcount = { + refcount = { + counter = 2 + } + } + }, + tclass = 0xffffffffa0be41a4 "selftest", + name = 0xffffffffa0be41d5 "dummy", + fun = 0xffffffffa0be10f0, + start = 0, + end = 1, + skb = 0xffff88003fc03800, + log = 0xffff88003fa59800 "", + lastrun = { + tv_sec = 1506072537, + tv_nsec = 289477354 + }, + debugfs = { + debugfs_results_testset = 0x0, + debugfs_results_test = 0xffff88021a18a900, + debugfs_run_testset = 0x0, + debugfs_run_test = 0xffff88021a18a9c0 + }, + handle = 0xffffffffa0be5480 + } + ... + crash> + + +The "log" fields are empty as each test passed, but we can +see from the "lastrun" times when the tests were run. +Logs will contain assertion failures etc in case of failure. + +Note that each test has a "handle" field also - this is +the KTF handle which was used to register the test. Each +handle also shows the currently-executing (if in the middle +of a test run) test associated with it, so if we want to +see where test execution was we can simply print the handle:: + + crash> print *(struct ktf_handle *)0xffffffffa0be5480 + $13 = { + test_list = { + next = 0xffffffffa0be5480, + prev = 0xffffffffa0be5480 + }, + handle_list = { + next = 0xffffffffa0be5490, + prev = 0xffffffffa0be5490 + }, + ctx_map = { + root = { + rb_node = 0x0 + }, + size = 0, + lock = { + { + rlock = { + raw_lock = { + { + head_tail = 0, + tickets = { + head = 0, + tail = 0 + } + } + } + } + } + }, + elem_comparefn = 0x0, + elem_freefn = 0x0 + }, + id = 0, + version = 4294967296, + current_test = 0x0 + } + crash> + +In this case current_test is NULL, but if we crashed in the +middle of executing a test it would show us which struct ktf_test * +it was. diff --git a/Documentation/dev-tools/ktf/examples.rst b/Documentation/dev-tools/ktf/examples.rst new file mode 100644 index 0000000..ec1f6e9 --- /dev/null +++ b/Documentation/dev-tools/ktf/examples.rst @@ -0,0 +1,26 @@ +6. Example test code +-------------------- + +Here is a minimal dummy example of a KTF unit test suite that defines +two tests, ``hello_ok`` and ``hello_fail``. The test is in the examples +directory and is built with KTF: + +.. literalinclude:: ../examples/hello.c + :language: c + +To run the test, cd to your KTF build tree and insmod the ktf module and +the module that provides the test:: + + insmod kernel/ktf.ko + insmod examples/hello.ko + +Now you should be able to run one or more of the tests by running the +application ``ktfrun`` built in ``user/ktfrun``. You should be able to run +that application as an ordinary user:: + + ktfrun --gtest_list_tests + ktfrun --gtest_filter='*fail' + ktfrun --gtest_filter='*ok' + +There are more examples in the examples directory. KTF also includes a +``selftest`` directory used to test/check the KTF implementation itself. diff --git a/Documentation/dev-tools/ktf/features.rst b/Documentation/dev-tools/ktf/features.rst new file mode 100644 index 0000000..2dc736c --- /dev/null +++ b/Documentation/dev-tools/ktf/features.rst @@ -0,0 +1,307 @@ +3. KTF kernel specific features +------------------------------- + +Reference to module internal symbols +************************************ + +When working with unit tests, the need to access non-public interfaces +often arises. In general non-public interfaces is of course not intended to +be used by outside programs, but a test framework is somewhat special here +in that it is often necessary or desirable to unit test internal +data structures or algorithms even if they are not exposed. The program +under test may be a complicated beast by itself and merely exercising the +public interfaces may not be flexible enough to stress the internal code. +Even if it is possible to get the necessary "pressure" from the outside +like that, it might be much more challenging or require a lot more work. + +The usual method to gain access to internal interfaces is to be part of the +internals. To some extent this is the way a lot of the kernel testing +utilities operate. The obvious advantages of this is that the test code +'automatically' follows the module and it's changes. The disadvantage is +that test code is tightly integrated with the code itself. One important +goal with KTF is to make it possible to write detailed and sophisticated +test code which does not affect the readability or complexity of the tested +code. + +KTF contains a small python program, ``resolve``, which +parses a list of symbol names on the form:: + + #module first_module + #header first_header.h + private_symbol1 + private_symbol2 + ... + #header second_header.h + #module next_module + ... + +The output is a header file and a struct containing function pointers and +some convenience macro definitions to make it possible to 'use' the +internal functions just as one would if within the module. This logic is +based on kallsyms, and would of course only work if that functionality is +enabled in the kernel KTF compiles against. Access to internal symbols +this way is controlled by the kernel config options CONFIG_KALLSYMS +and CONFIG_KALLSYMS_ALL, which must be set to "y". + +If you create a new test project using the ``ktfnew`` script, you can +put your private symbol definitions in a file ``ktf_syms.txt`` in the +kernel directory, and KTF will automatically generate ``ktf_syms.h``, +which you can then include in your test file to get to these symbols. +This functionality is also used by the KTF selftests, which might +serve as an example to get started. + +Note also that for exported symbols, if you build your module out-of-tree in +addition to KTF and the test modules, you might need to also add those +module's ``Module.symvers`` files to ``KBUILD_EXTRA_SYMBOLS`` +(See kernel documentation for this) to find them during test module build. + +Requesting callbacks when a certain function gets called/returns +**************************************************************** + +Tap into function entry using KTF entry probes. Many tests need to +move beyond kernel APIs and ensure that side effects (logging a +message etc) occur. A good way to do this is to probe entry of relevant +functions. In order to do so in KTF you need to: + + - define an entry probe function with the same return value and arguments + as the function to be probed + + - within the body of the entry probe function, ensure return is wrapped with + KTF_ENTRY_PROBE_RETURN(<return value>); + + - entry probes need to registered for use and de-registered when done via + KTF_[UN]REGISTER_ENTRY_PROBE(<kernel function name>, <handler function>). + +See example h4.c in examples/ for a simple case where we probe printk() and +ensure it is called. + +Sometimes is is also useful to check that an intermediate function is returning +an expected value. Return probes can be used to register/probe function +return. In order to probe function return: + + - define a return probe point; i.e + KTF_RETURN_PROBE(<kernel function>, <handler>) + + - within the body of the return probe the return value can be retrieved + via KTF_RETURN_VALUE(). Type will obviously depend on the function + probed so should be cast if dereferencing is required. + + - return probes need to be registered for use and unregistered when done + via KTF_[UN]REGISTER_RETURN_PROBE(<kernel function name>, <handler>). + +See example h4.c in examples/ for a simple case where we verify return value +of printk(). + +Note that this functionality is only available on kernels with CONFIG_KPPROBES +and CONFIG_KRETPROBES set to "y". + +Overriding functions +******************** +in some cases, we wish to override harmful functions when inducing failues in +tests (e.g. skb_panic()). Override is done via kprobes and we define as follows:: + + KTF_OVERRIDE(oldfunc, newfunc) + { + ... + KTF_SET_RETURN_VALUE(1); + KTF_OVERRIDE_RETURN; + } + + TEST(...) + { + KTF_REGISTER_OVERRIDE(oldfunc, newfunc); + ... + KTF_UNREGISTER_OVERRIDE(oldfunc, newfunc); + } + +Override should be used sparingly; we'd rather test the code as-is and use +entry/return probes where possible. + +Note that this functionality is only available on kernels with CONFIG_KPPROBES +and CONFIG_KRETPROBES set to "y". + +Coverage analytics +****************** + +While other coverage tools exist, they generally involve gcc-level support +which is required at compile-time. KTF offers kernel module coverage +support via kprobes instead. Tests can enable/disable coverage on a +per-module basis, and coverage data can be retrieved via:: + + # more /sys/kernel/debug/ktf/coverage + +For a given module we show how many of its functions were called versus the +total, e.g.:: + + # cat /sys/kernel/debug/ktf/coverage + MODULE #FUNCTIONS #CALLED + selftest 14 1 + +We see 1 out of 14 functions was called when coverage was enabled. + +We can also see how many times each function was called:: + + MODULE FUNCTION COUNT + selftest myelem_free 0 + selftest myelem_cmp 0 + selftest ktf_return_printk 0 + selftest cov_counted 1 + selftest dummy 0 + +In addition, we can track memory allocated via kmem_cache_alloc()/kmalloc() +originating from module functions we have enabled coverage for. This +allows us to track memory associated with the module specifically to find +leaks etc. If memory tracking is enabled, /sys/kernel/debug/ktf/coverage +will show outstanding allocations - the stack at allocation time; the +memory address and size. + +Coverage can be enabled via the "ktfcov" utility. Syntax is as follows:: + + ktfcov [-d module] [-e module [-m]] + +"-e" enables coverage for the specified module; "-d" disables coverage. +"-m" in combination with "-e" enables memory tracking for the module under +test. + +Note that this functionality is only available on kernels with CONFIG_KPPROBES +and CONFIG_KRETPROBES set to "y", and that CONFIG_KALLSYMS and +CONFIG_KALLSYMS_ALL should be set to "y" also to get all exported and +non-exported symbols. + +Thread execution +**************** + +KTF provides easy mechanisms to create and use kernel threads. +Assertions can then be carried out in the created thread context +also. Threads can be created as follows, and we can if we wish +wait for thread completion:: + + + TEST(foo, bar) + { + struct ktf_thread t; + + ... + KTF_THREAD_INIT(mythread, &t); + KTF_THREAD_RUN(&t); + KTF_THREAD_WAIT_COMPLETED(&t); + ... + } + +The thread itself is defined as follows:: + + KTF_THREAD(mythread) + { + ... + } + +We can add assertions to the thread and they will be recorded/logged +as part of the test. + +Hybrid tests +************ + +KTF also allows mixing of user and kernel side code in the same test. +This is useful if one wants for instance to verify that user land operations +has certain effects in the kernel, for instance verify that a parameter is +transferred or handled correctly in the kernel. + +Hybrid tests are specified by writing a user mode test using the special +``HTEST()`` macro instead of the normal ``TEST()`` macro. This macro takes +Inside the macro, the special variable ``self`` can be used to refer to the +test itself, and the macro ``KTF_USERDATA()`` can be used to get a pointer to +an allocated instance of a test specific parameter struct. The user land test +can then call the kernel side directly using ``ktf::run_kernel_test(self)`` An +optional context name can be specified as a second argument to the call if +needed. This can be done any number of times during the user land test and +each call will transmit the struct value out-of-band to the kernel side. To +the kernel this appears as separate test calls, but the kernel side have the +option of aggregating or otherwise maintain state for the duration of the +test. + +Declare the data structure to use for user/kernel out-of-band communication +in a header file that is included both by the user and the kernel side:: + + struct my_params + { + char expected[128]; + unsigned long mode; + }; + +The user land side of the test itself can then look like this:: + + HTEST(foo, hybrid) + { + KTF_USERDATA(self, my_params, data); + + <normal gtest code> + + strcpy(data->expected, "something"); + data->mode = 0; + ktf::run_kernel_test(self); + + strcpy(data->expected, "something_else"); + ktf::run_kernel_test(self); + + <normal gtest code> + + ... + } + +On the kernel side, a hybrid test is written as a normal kernel test using +the ``TEST()`` macro, and the test must be added using ``ADD_TEST()`` as +usual. Include the user land header file to know the data type of the +out-of-band parameter block. Invoke the macro ``KTF_USERDATA()`` to get a +size validated pointer to the user land provided data. If no data is +available, the test will silently exit. This is by purpose - if the kernel +test is executed from a test program that does not have the associated user +land code, such as for instance ``ktfrun``, it will just appear as a test +with no assertions in it, and not create any errors. If on the other hand the +parameter block does not match in size, an assertion is thrown and the test +exits:: + + TEST(foo, hybrid) + { + KTF_USERDATA(self, my_params, data); + + ... + if (strcmp(data->expected, "something") == 0) + ... + EXPECT( ... ) + + ... + } + + +Running tests and examining results via debugfs +*********************************************** + +In addition to the netlink interface used by the Googletest integrated frontend code, +we provide debugfs interfaces for examining the results of the +last test run and for running tests which do not require configuration +specification. Individual ktf testsets can be run via:: + + cat /sys/kernel/debug/ktf/run/<testset> + +Individual tests can be run via:: + + cat /sys/kernel/debug/ktf/run/<testset>-tests/<test> + +Results can be displayed for the last run via:: + + cat /sys/kernel/debug/ktf/results/<testset> + +Individual tests can be run via:: + + cat /sys/kernel/debug/ktf/results/<testset>-tests/<test> + +These interfaces bypasses use of the netlink socket API +and provide a simple way to keep track of test failures. It can +be useful to log into a machine and examine what tests were run +without having console history available. + +In particular:: + + cat /sys/kernel/debug/ktf/run/* + +...is a useful way of running all KTF tests. diff --git a/Documentation/dev-tools/ktf/implementation.rst b/Documentation/dev-tools/ktf/implementation.rst new file mode 100644 index 0000000..2bc4335 --- /dev/null +++ b/Documentation/dev-tools/ktf/implementation.rst @@ -0,0 +1,70 @@ + +2. Implementation +----------------- + +KTF consists of a kernel part and a user part. The role of the user part is to query the kernel +for available tests, and provide mechanisms for executing a selected set or all the available +tests, and report the results. The core ktf kernel module simply provides some APIs to write +assertions and run tests and to communicate about tests and results with user mode. +A simple generic Netlink protocol is used for the communication. + +User mode implementation +************************ + +Since test filtering and reporting is something existing unit test frameworks for +user space code already does well, the implementation of KTF simply leverages that. +The current version supports an integration with gtest (Googletest), which provides a lot of +these features in a flexible way, but in principle alternative implementations could +use the reporting of any other user level unit test framework. The use of gtest also allows this +documentation to be shorter, as many of the features in gtest are automatically available for KTF as well. +More information about Googletest features can be found here: https://github.com/google/googletest + +Kernel mode implementation +************************** + +The kernel side of KTF implements a simple API for tracking test modules, +writing tests, support functions and and a set of assertion macros, some +tailored for typical kernel usage, such as ``ASSERT_OK_ADDR_GOTO()`` +as a kernel specific macro to check for a valid address with a label to jump to if the +assertion fails. After all as we are still in the kernel, tests would always need to clean up for +themselves even though in the context of ktf. + +KTF supports two distinct classes of tests: + +* Pure kernel mode tests +* Hybrid tests + +Pure kernel mode tests are tests that are fully implemented in kernel space. +This is the most straightforward mode and resembles ordinary user land unit testing +in kernel mode. If you only have kernel mode tests, you will only ever need one user level program +similar to user/ktfrun.cpp, since all test development takes place on the kernel side. + +Hybrid tests are for testing and making assumptions about the user/kernel communication, for instance +if a parameter supplied from user mode is interpreted the intended way when it arrives at it's kernel +destination. For such tests you need to tell ktf (from user space) when the kernel part of the test +is going to be executed - this can happen multiple times depending on your test needs. +Apart from that it works mostly like a normal gtest user level test. + +Kernel integration of KTF or KTF as a separate git project? +*********************************************************** + +Yes. A lot of test infrastructure and utilities for the Linux kernel +is implemented as part of the linux kernel git project. +This has some obvious benefits, such as + +* Always being included +* When APIs are changed, test code can be updated atomically with the rest of the kernel +* Higher visibility and easier access +* Easier integration with internal kernel interfaces useful for testing. + +On the other hand providing KTF as a separate project allows + +* With some use of ``KERNEL_VERSION`` and ``LINUX_VERSION_CODE``, up-to-date KTF code and tests + can be allowed to work across kernel versions. +* This in turn allows a single set of newly developed tests to be + simultaneously tested against multiple older kernels, possibly + detecting more bugs, or instances of bugs not backported. + +So we will continue to support both, and have work in progress to simplify +the maintenance and synchronization of the two versions, and allow the +additional tooling to extend to KTF client test suites as well. diff --git a/Documentation/dev-tools/ktf/index.rst b/Documentation/dev-tools/ktf/index.rst new file mode 100644 index 0000000..25db49b --- /dev/null +++ b/Documentation/dev-tools/ktf/index.rst @@ -0,0 +1,14 @@ +Kernel Test Framework documentation +=================================== + +.. toctree:: + :maxdepth: 2 + + introduction + implementation + features + installation + concepts + examples + progref + debugging diff --git a/Documentation/dev-tools/ktf/installation.rst b/Documentation/dev-tools/ktf/installation.rst new file mode 100644 index 0000000..1bfccc1 --- /dev/null +++ b/Documentation/dev-tools/ktf/installation.rst @@ -0,0 +1,73 @@ +4. Building and installing KTF +------------------------------ + +KTF's user land side depends on googletest. +The googletest project has seen some structural changes in moving from a +project specific gtest-config via no package management support at all to +recently introduce pkgconfig support. This version of KTF only supports +building against a googletest (gtest) with pkgconfig support, which means +that as of February 2018 you have to build googletest from source at +github. + +Googletest has also recently been fairly in flux, and while we +try to keep up to date with the official googletest version on Github, +we have seen issues with changes that breaks KTF. We also have a small +queue of enhancements and fixes to Googletest based on our experience +and use of it a.o. with KTF. You can find the latest rebase of this +version in the ktf branch of knuto/googletest at Github, but expect it +to rebase as we move forward to keep it up-to-date. +This version will at any time have been tested with KTF by us, since +we use it internally. Let's assume for the rest of these instructions +that your source trees are below ``~/src`` and your build trees are +under ``~/build``:: + + cd ~/src + git clone https://github.com/knuto/googletest.git + +or:: + + cd ~/src + git clone https://github.com/google/googletest.git + +then:: + + mkdir -p ~/build/$(uname -r) + cd ~/build/$(uname -r) + mkdir googletest + cd googletest + cmake ~/src/googletest + make + sudo make install + +Default for googletest is to use static libraries. If you want to use shared +libraries for googletest, you can specify ``-DBUILD_SHARED_LIBS=ON`` to +cmake. If you don't want to install googletest into /usr/local, you can +specify an alternate install path using ``-DCMAKE_INSTALL_PREFIX=<your path>`` +to cmake for googletest, and similarly use ``--prefix=<your path>`` both for +KTF and your own test modules. Note that on some distros, cmake version +2 and 3 comes as different packages, make sure you get version 3, which may +require you to use ``cmake3`` as command instead of cmake above. + +Building the in-kernel version of KTF and running KTF selftests +*************************************************************** + +The environment needs to have the path to the +gtest (Googletest) build set to the directory above the lib and +include directories:: + + export GTEST_PATH=$HOME/install + +KTF can then be built using the module target, eg. from the top level +kernel build tree. + + make M=tools/testing/selftests/ktf + +You can run also build (and run) KTF tests as selftests tests +via the kselftest target:: + + make TARGETS="ktf" kselftest + +You can invoke this command to let the tests run as a normal user, but +root access is needed to load and unload ktf.ko and the test +module(s). This will happen as part of the kselftest target even as a +normal user if the user has sudo privileges. diff --git a/Documentation/dev-tools/ktf/introduction.rst b/Documentation/dev-tools/ktf/introduction.rst new file mode 100644 index 0000000..5c861f4 --- /dev/null +++ b/Documentation/dev-tools/ktf/introduction.rst @@ -0,0 +1,134 @@ +:Author: Knut Omang knut.omang@oracle.com +:Last Updated: Alan Maguire alan.maguire@oracle.com + +1. Background and motivation +---------------------------- + +Kernel Test Framework (KTF) implements a unit test framework for the Linux kernel. +There's a wide selection of unit test frameworks available for normal user land +code testing, but we have so far not seen any similar frameworks that can be used +with kernel code, to test details of both exported and non-exported kernel APIs. +The hope is that providing an easy to use and convenient way to write simple unit +tests for kernel internals, that this can promote a more test driven approach to +kernel development, where appropriate. + +An important design goal is to make KTF in a way that it lend itself well to a normal kernel +developer cycle, and that it integrates well with user land unit testing, to allow kernel and +user land tests to behave, look and feel as similar as possible. This should hopefully make it +more intuitive to use as well as more rewarding. We also believe that even a kernel test that +passes should have a nice, easy to read and pleasant output, and that a test framework must have +good observability, that is good mechanisms for debugging what went wrong, both in case of bugs +in the tests and and the test framework itself. + +KTF is designed to test the kernel in the same ways it runs. This means we want to stay away from +changing configuration options, or otherwise make changes that makes it hard to logically tell +from a high level perspective whether the kernel with KTF is really logically "the same" as the +kernel our users are exposed to. Of course we all know that it is very hard to test anything +without affecting it, with quantum mechanics as the extreme, but at least we want to make an +effort to make the footprint as small as possible. + +KTF tests kernel code by running tests in kernel context - or in the case of hybrid tests - in +both user- and kernel contexts. Doing this ensures that we test kernel codepaths in a real way, +without emulating a kernel execution environment. This gives us vastly more control over what +tests can do versus user-space driven testing, and increases confidence that what the tests test +matches what the kernel does since the test execution environment is identical. + +KTF is a product of a refactoring of code used as part of test driven development of a Linux +driver for an Infiniband HCA. It is in active use for kernel component testing within Oracle. + +Test driven development +*********************** + +Unit testing is an important component of Test driven development (TDD). +The idea of test driven development is that when you have some code to write, +whether it is a bug to fix, or a new feature or enhancement, that you start by writing +one or more tests for it, have those tests fail, and then do the actual development. + +Typically a test driven development cycle would have several rounds of development and +test using the new unit tests, and once the (new) tests pass, you would +also run all or a suitable subset (limited by execution time) of the old tests to verify +that you have not broken any old functionality by the new code. + +At this stage it is important that the tests that are run can be run quickly to allow +them to be actively used in the development cycle. When time comes for +submission of the code, a full, extensive set of both the tests the developer thinks +can touch the affected code *and* all the other tests should be run, and a longer time +to run tests can be afforded. + +KTF tries to support this by using the module system of the kernel to support +modularized test suites, where a user only need to insmod the test subsets that he/she wants +to use right then. Different test suites may touch and require different kernel APIs and have +lots of different module and device requirements. To enable as much reuse of the functionality +of code developed within KTF it is important that any piece of test code has as few dependencies +as possible. + +Good use cases for KTF +********************** + +Unit testing is at it's most valuable when the code to test is relatively error prone, but still +might be difficult to test in a systematic and reproducable way from a normal application level. +It can be difficult to trigger corner cases from a high abstraction layer, +the code paths we want to exercise might only be used occasionally, or we want to exercise +that error/exception scenarios are handled gracefully. + +KTF comes to it's strength in testing kernel APIs that are fairly integrated into the kernel, +and depend upon lots of components, making them difficult or error prone to mock. Good examples +are module APIs not easily testable from user land. Exported module APIs are usually only used +by one or a few other kernel clients, and hitting buggy corner cases with these might be hard or +impossible. This typically leads to bugs detected "down the road", when some new client appears +and starts using the API in a new way, or instabilities that go undetected because underlying +semantics that the implementation implicitly depend upon changes in subtle ways. + +KTF can use mechanisms such as KTF probes in cases where calls to other functions needs to be +intercepted and/or modified to create the right test condition, whether it means waiting for a +potential race condition to occur, or return an error value, or just collect state to make assertions. + +Typical classical use cases that lend itself well to unit testing are simple APIs with a relativ +complex implementation - such as container implementations. Typical kernel examples of these +in the kernel are scatterlist, rbtree, list, XArray etc. When testing the base implementations of such +containers, bringing them entirely out into user space and compiling them standalone require some +additional work up-front to implement mock interfaces to the services provided by the kernel, +but may nonetheless be rewarding in the longer run, as such tests have at it's disposal the whole +arsenal of user land tools, such as gdb, valgrind etc. This, however does not guarantee against +wrong use of a container, such as with interactions between a container and a driver +datastructure. + +Testing the *instantiations* of these container implementations inside drivers or +the kernels's own internals might not be that easy with a user land approach, as it very quickly +requires a prohibitive amount of mock interfaces to be written. And even when such mock +interfaces can be written, one cannot be sure that they implement exactly the same as the +environment that the code executes in within the kernel. Having the ability to make tests within +a release kernel, even run the same tests against multiple such kernels is something KTF +supports well. Our experience is that even error scenarios that are hard to reproduce by +running applications on the kernel can often be reproduced with a surprisingly small +number of lines of code in a KTF test, once the problem is understood. And writing that code can +be a very rewarding way of narrowing down a hard bug. + +When *not* to use KTF +********************* + +Writing kernel code has some challenges compared to user land code. +KTF is intended for the cases where it is not easy to get coverage by writing +simple tests from user land, using an existing rich and well proven user land unit test +framework. + +Why *you* would want to write and run KTF tests +*********************************************** + +Besides the normal write test, write code, run test cycle of development and the obvious benefits of +delivering better quality code with fewer embarrassments, there's a few other upsides from +developing unit test for a particular area of the kernel: + +* A test becomes an invariant for how the code is supposed to work. + If someone breaks it, they should detect it and either document the changes that caused the breakage + by fixing the test or realize that their fix is broken before you even get to spend time on it. + +* Kernel documentation while quite good in some places, does not always + cover the full picture, or you might not find that sentence you needed while looking for it. + If you want to better understand how a particular kernel module actually works, a good way is to + write a test that codes your assumptions. If it passes, all is well, if not, then you have gained some + understanding of the kernel. + +* Sometimes you may find yourself relying on some specific feature or property of the kernel. + If you encode a test that guards the assumptions you have made, you will capture if someone + changes it, or if your code is ported to an older kernel which does not support it. diff --git a/Documentation/dev-tools/ktf/progref.rst b/Documentation/dev-tools/ktf/progref.rst new file mode 100644 index 0000000..2f0fa48 --- /dev/null +++ b/Documentation/dev-tools/ktf/progref.rst @@ -0,0 +1,144 @@ +7. KTF programming reference +---------------------------- + +KTF itself contains no tests but provides primitives and data structures to +allow tests to be maintained and written in separate test modules that +depend on the KTF APIs. + +KTF API Overview +**************** + +For reference, the following table lists a few terms and classes of +abstractions provided by KTF. These are kernel side, if not otherwise noted: + ++----------------------------+--------------------------------------------------+ +| **Item** | **description** | ++============================+==================================================+ +| Test module | A kernel object file (.ko) with ktf tests in it | ++----------------------------+--------------------------------------------------+ +| struct ktf_handle | At least 1 per test module. | +| | Use macros KTF_INIT() and KTF_CLEANUP() to set up| +| | and tear down handles. | ++----------------------------+--------------------------------------------------+ +| struct ktf_context | 0-n per test module - test module specific | +| | context for the test, such as eg. a device or | +| | another kernel object. | ++----------------------------+--------------------------------------------------+ +| KTF_INIT() | Call this at the global level in the main file | +| | for each test module. Declares an implicit, | +| | default test handle used by macros which do not | +| | provide a handle argument. | ++----------------------------+--------------------------------------------------+ +| KTF_CTX_INIT() | Use this instead of KTF_INIT if the tests require| +| | a context to execute. Tests will only show up as | +| | options if a context has been provided. | ++----------------------------+--------------------------------------------------+ +| KTF_HANDLE_INIT(handle) | Declare a named handle to associate tests and | +| | contexts with. This is an alternative to | +| | KTF_INIT() to allow the use of separate test | +| | handles for separate sets of tests. | ++----------------------------+--------------------------------------------------+ +| KTF_HANDLE_CTX_INIT(handle)| Equivalent of KTF_CTX_INIT for a named handle | ++----------------------------+--------------------------------------------------+ +| KTF_CLEANUP() | Call this in the __exit function to clean up | ++----------------------------+--------------------------------------------------+ +| KTF_CONTEXT_ADD(ctx, name) | Add a new context to the default handle | ++----------------------------+--------------------------------------------------+ +| KTF_CONTEXT_FIND(name) | Return a struct ktf_context reference to context | +| | 'name', if it exists | ++----------------------------+--------------------------------------------------+ +| KTF_CONTEXT_GET(name,type) | Return the structure of type 'type' containing | +| | the ktf_context named 'name', if 'name' exists. | ++----------------------------+--------------------------------------------------+ +| KTF_CONTEXT_REMOVE(ctx) | Remove a previously added context from KTF | ++----------------------------+--------------------------------------------------+ +| EXPECT_* | non-fatal assertions | ++----------------------------+--------------------------------------------------+ +| ASSERT_* | "fatal" assertions that would cause return/goto | ++----------------------------+--------------------------------------------------+ +| TEST(s, n) {...} | Define a simple test named 's.n' with implicit | +| | arguments 'ctx' and '_i' for context/iteration. | ++----------------------------+--------------------------------------------------+ +| DECLARE_F(f) {...} | Declare a new test fixture named 'f' with | +| | additional data structure | ++----------------------------+--------------------------------------------------+ +| SETUP_F(f, s) {...} | Define setup function for the fixture | ++----------------------------+--------------------------------------------------+ +| TEARDOWN_F(f, t) {...} | Define teardown function for the fixture | ++----------------------------+--------------------------------------------------+ +| INIT_F(f, s, t) {...} | Declare the setup and tear down functions for the| +| | fixture | ++----------------------------+--------------------------------------------------+ +| TEST_F(s, f, n) {...} | Define a test named 's.n' operating in fixture f | ++----------------------------+--------------------------------------------------+ +| ADD_TEST(n) | Add a test previously declared with TEST or | +| | TEST_F to the default handle. | ++----------------------------+--------------------------------------------------+ +| ADD_LOOP_TEST(n, from, to) | Add a test to be executed repeatedly with a range| +| | of values [from,to] to the implicit variable _i | ++----------------------------+--------------------------------------------------+ +| DEL_TEST(n) | Remove a test previously added with ADD_TEST | ++----------------------------+--------------------------------------------------+ +| KTF_ENTRY_PROBE(f, h) | Define function entry probe for function f with | +| {...} | handler function h. Must be used at global level.| ++----------------------------+--------------------------------------------------+ +| KTF_ENTRY_PROBE_RETURN(r) | Return from probed function with return value r. | +| | Must be called within KTF_ENTRY_PROBE(). | ++----------------------------+--------------------------------------------------+ +| KTF_REGISTER_ENTRY_PROBE | Enable probe on entry to kernel function f | +| (f, h) | with handler h. | ++----------------------------+--------------------------------------------------+ +| KTF_UNREGISTER_ENTRY_PROBE | Disable probe on entry to kernel function f | +| (f, h) | which used handler h. | ++----------------------------+--------------------------------------------------+ +| KTF_RETURN_PROBE(f, h) | Define function return probe for function f with | +| {..} | handler h. Must be used at a global level. | ++----------------------------+--------------------------------------------------+ +| KTF_RETURN_VALUE() | Retrieve return value in body of return probe. | ++----------------------------+--------------------------------------------------+ +| KTF_REGISTER_RETURN_PROBE | Enable probe for return of function f with | +| (f, h) | handler h. | ++----------------------------+--------------------------------------------------+ +| KTF_UNREGISTER_RETURN_PROBE| Disable probe for return of function f and | +| (f, h) | handler h. | ++----------------------------+--------------------------------------------------+ +| ktf_cov_enable(m, flags) | Enable coverage analytics for module m. | +| | Flag must be either 0 or KTF_COV_OPT_MEM. | ++----------------------------+--------------------------------------------------+ +| ktf_cov_disable(m) | Disable coverage analytics for module m. | ++----------------------------+--------------------------------------------------+ +| KTF_THREAD_INIT(name, t) | Initialize thread name, struct ktf_thread * t. | ++----------------------------+--------------------------------------------------+ +| KTF_THREAD_RUN(t) | Run initialized struct ktf_thread * t. | ++----------------------------+--------------------------------------------------+ +| KTF_THREAD_STOP(t) | Stop thread via kthread_stop() | ++----------------------------+--------------------------------------------------+ +| KTF_THREAD_WAIT_STARTED(t) | Wait for start of struct ktf_thread * t. | ++----------------------------+--------------------------------------------------+ +| KTF_THREAD_WAIT_COMPLETED | Wait for completion of struct ktf_thread * t. | +| (t) | | ++----------------------------+--------------------------------------------------+ +| HTEST(s, n) { ... } | Declares a hybrid test. A correspondingly named | +| (NB! User mode only!) | test must be declared using TEST() from kernel | +| | space for the hybrid test to be executed. | ++----------------------------+--------------------------------------------------+ +| KTF_USERDATA(self, type, d)| Declare/get a pointer to user/kernel aux.data | +| (NB! both kernel and | for a test that declares such extra data. Used | +| user space!) | for hybrid tests. | ++----------------------------+--------------------------------------------------+ + +The ``KTF_INIT()`` macro must be called at a global level as it just +defines a variable ``__test_handle`` which is referred to, and which existence +is assumed to continue until the call to KTF_CLEANUP(), typically done in +the ``__exit`` function of the test module. + + + +Assertions +********** + +Below is example documentation for some of the available assertion macros. +For a full overview, see ``kernel/ktf.h`` + +.. kernel-doc:: kernel/ktf.h + :internal: