Re: [PATCH v2 1/2] kunit: add ability to run tests after boot using debugfs

On Thu, Sep 14, 2023 at 5:06 AM David Gow davidgow@google.com wrote:

On Sat, 9 Sept 2023 at 05:31, Rae Moar rmoar@google.com wrote:

...

Add functionality to run built-in tests after boot by writing to a debugfs file.

Add a new debugfs file labeled "run" for each test suite to use for this purpose.

As an example, write to the file using the following:

echo "any string" > /sys/kernel/debugfs/kunit/<testsuite>/run

This will trigger the test suite to run and will print results to the kernel log.

Note that what you "write" to the debugfs file will not be saved.

To guard against running tests concurrently with this feature, add a mutex lock around running kunit. This supports the current practice of not allowing tests to be run concurrently on the same kernel.

This functionality may not work for all tests.

This new functionality could be used to design a parameter injection feature in the future.

Signed-off-by: Rae Moar rmoar@google.com

This is looking pretty good, but I have a few nitpicks below and one big issue.

The big issue is that this doesn't seem to exclude test suites created with kunit_test_init_section_suite{,s}(). The init section versions of the suite declarations, by definition, won't work if run after the kernel has finished booting. At the moment, these macros just pass through to the normal versions (because we've not been able to run after boot until now), but we'll need to implement it (maybe as a separate linker section, maybe as an attribute, etc) now. I expect that the correct solution here would be to not create the 'run' debugfs file for these tests. But I could be convinced to have it exist, but to just say "this test cannot be run after boot" if you've got a good argument. In any case, grep 'test.h' for "NOTE TO KUNIT DEVS" and you'll see the details.

My one other not-totally-related thought (and this extends to module loading, too, so is possibly more useful as a separate patch) is that we're continually incrementing the test number still. This doesn't matter if we read the results from debugfs though, and it may make more sense to have this continue to increment (and thus treat all of dmesg as one long KTAP document). We could always add a reset option to debugfs in a follow-up patch if we want. But that's not something I'd hold this up with.

Hello!

Sorry for the delay in this response. I was working on other items but I have started working on the next version of this patch.

Thanks for bringing my attention to the init tests. I am currently working on a draft to remove the run files for these tests. However, if that does not work, I will resort to outputting the message you detailed above: "this test cannot be run after boot".

I am currently fine with the test number incrementing. However, I would also be fine to implement a reset to ensure all the re-run results have the test number of 1.

...

Changes since v1:

• Removed second patch as this problem has been fixed
• Added Documentation patch
• Made changes to work with new dynamically-extending log feature

Note that these patches now rely on (and are rebased on) the patch series: https://lore.kernel.org/all/20230828104111.2394344-1-rf@opensource.cirrus.co...

lib/kunit/debugfs.c | 66 +++++++++++++++++++++++++++++++++++++++++++++ lib/kunit/test.c | 13 +++++++++ 2 files changed, 79 insertions(+)

diff --git a/lib/kunit/debugfs.c b/lib/kunit/debugfs.c index 270d185737e6..8c0a970321ce 100644 --- a/lib/kunit/debugfs.c +++ b/lib/kunit/debugfs.c @@ -8,12 +8,14 @@ #include <linux/module.h>

#include <kunit/test.h> +#include <kunit/test-bug.h>

#include "string-stream.h" #include "debugfs.h"

#define KUNIT_DEBUGFS_ROOT "kunit" #define KUNIT_DEBUGFS_RESULTS "results" +#define KUNIT_DEBUGFS_RUN "run"

/*

• Create a debugfs representation of test suites:

@@ -21,6 +23,8 @@

• Path Semantics
• /sys/kernel/debug/kunit/<testsuite>/results Show results of last run for

•                                         testsuite
  

• • /sys/kernel/debug/kunit/<testsuite>/run Write to this file to trigger

• •                                         testsuite to run
    
  */

@@ -99,6 +103,51 @@ static int debugfs_results_open(struct inode *inode, struct file *file) return single_open(file, debugfs_print_results, suite); }

+/*

• • Print a usage message to the debugfs "run" file
• • (/sys/kernel/debug/kunit/<testsuite>/run) if opened.
• */

+static int debugfs_print_run(struct seq_file *seq, void *v) +{

•   struct kunit_suite *suite = (struct kunit_suite *)seq->private;
  

•   seq_puts(seq, "Write to this file to trigger the test suite to run.\n");
  

•   seq_printf(seq, "usage: echo \"any string\" > /sys/kernel/debugfs/kunit/%s/run\n",
  

•                   suite->name);
  

•   return 0;
  

+}

+/*

• • The debugfs "run" file (/sys/kernel/debug/kunit/<testsuite>/run)
• • contains no information. Write to the file to trigger the test suite
• • to run.
• */

+static int debugfs_run_open(struct inode *inode, struct file *file) +{

•   struct kunit_suite *suite;
  

•   suite = (struct kunit_suite *)inode->i_private;
  

•   return single_open(file, debugfs_print_run, suite);
  

+}

+/*

• • Trigger a test suite to run by writing to the suite's "run" debugfs
• • file found at: /sys/kernel/debug/kunit/<testsuite>/run
• • Note: what is written to this file will not be saved.
• */

+static ssize_t debugfs_run(struct file *file,

•           const char __user *buf, size_t count, loff_t *ppos)
  

+{

•   struct inode *f_inode = file->f_inode;
  

•   struct kunit_suite *suite = (struct kunit_suite *) f_inode->i_private;
  

•   __kunit_test_suites_init(&suite, 1);
  

•   return count;
  

+}

static const struct file_operations debugfs_results_fops = { .open = debugfs_results_open, .read = seq_read, @@ -106,10 +155,23 @@ static const struct file_operations debugfs_results_fops = { .release = debugfs_release, };

+static const struct file_operations debugfs_run_fops = {

•   .open = debugfs_run_open,
  

•   .read = seq_read,
  

•   .write = debugfs_run,
  

•   .llseek = seq_lseek,
  

•   .release = debugfs_release,
  

+};

void kunit_debugfs_create_suite(struct kunit_suite *suite) { struct kunit_case *test_case;

•   if (suite->log) {
  

•           /* Clear the suite log that's leftover from a previous run. */
  

•           string_stream_clear(suite->log);
  

•           return;
  

•   }
  

Can we just move this to kunit_init_suite() in test.c. It doesn't feel quite debugfs-y enough, and the return here tripped me up for a little too long.

Ideally, we'd then split up kunit_init_suite() into a one-time initialisation (which calls kunit_debugfs_create_suite()), and a reset function (which resets the state of the suite back to the beginning). We then only call init once, but reset on every execution.

I definitely think you are right here to move this into test.c. I will try to put this into a reset function.

...

    /* Allocate logs before creating debugfs representation. */
    suite->log = alloc_string_stream(GFP_KERNEL);
    string_stream_set_append_newlines(suite->log, true);

@@ -124,6 +186,10 @@ void kunit_debugfs_create_suite(struct kunit_suite *suite) debugfs_create_file(KUNIT_DEBUGFS_RESULTS, S_IFREG | 0444, suite->debugfs, suite, &debugfs_results_fops);

•   debugfs_create_file(KUNIT_DEBUGFS_RUN, S_IFREG | 0644,
  

•                       suite->debugfs,
  

•                       suite, &debugfs_run_fops);
  

}

void kunit_debugfs_destroy_suite(struct kunit_suite *suite) diff --git a/lib/kunit/test.c b/lib/kunit/test.c index 651cbda9f250..d376b886d72d 100644 --- a/lib/kunit/test.c +++ b/lib/kunit/test.c @@ -13,6 +13,7 @@ #include <linux/kernel.h> #include <linux/module.h> #include <linux/moduleparam.h> +#include <linux/mutex.h> #include <linux/panic.h> #include <linux/sched/debug.h> #include <linux/sched.h> @@ -22,6 +23,8 @@ #include "string-stream.h" #include "try-catch-impl.h"

+static struct mutex kunit_run_lock;

Should we use DEFINE_MUTEX() here rather than initialising it at runtime?

I will try to implement this using DEFINE_MUTEX.

...

/*

• Hook to fail the current test and print an error message to the log.

*/ @@ -668,6 +671,11 @@ int __kunit_test_suites_init(struct kunit_suite * const * const suites, int num_ return 0; }

•   /* Use mutex lock to guard against running tests concurrently. */
  

•   if (mutex_lock_interruptible(&kunit_run_lock)) {
  

•           pr_err("kunit: test interrupted\n");
  

•           return -EINTR;
  

•   }
    static_branch_inc(&kunit_running);
  
    for (i = 0; i < num_suites; i++) {
  

@@ -676,6 +684,7 @@ int __kunit_test_suites_init(struct kunit_suite * const * const suites, int num_ }

    static_branch_dec(&kunit_running);

•   mutex_unlock(&kunit_run_lock);
    return 0;
  

} EXPORT_SYMBOL_GPL(__kunit_test_suites_init); @@ -836,6 +845,10 @@ static int __init kunit_init(void) kunit_install_hooks();

    kunit_debugfs_init();

•   /* Initialize lock to guard against running tests concurrently. */
  

•   mutex_init(&kunit_run_lock);
  

As I understand it, we can just use DEFINE_MUTEX() above.

...

#ifdef CONFIG_MODULES return register_module_notifier(&kunit_mod_nb); #else

base-commit: b754593274e04fc840482a658b29791bc8f8b933

2.42.0.283.g2d96d420d3-goog