livepatch's consistency model requires that no live patched function must be found on any task's stack during a transition process after a live patch is applied. It is achieved by walking through stacks of all blocked tasks.
The user might also want to define more functions to search for without them being patched at all. It may either help with preparing a live patch, which would otherwise require additional touches to achieve the consistency, or it can be used to overcome deficiencies the stack checking inherently has. For example, GCC may optimize a function so that a part of it is moved to a different section and the function would jump to it. This child function would not be found on a stack in this case, but it may be important to search for it so that, again, the consistency is achieved.
The patch set adds a new API which allows the user to specify such functions.
The first patch is only preparatory. The main work is in the second one. The third patch adds a test.
Originally, I wanted to add it all to klp_patch struct, but it makes more sense to do it on klp_object level. It is also easier to reuse as much of the existing code as possible in that case.
I am not good with naming so bike-shedding is welcome. Reviews even more.
Miroslav Benes (3): livepatch: Move the initialization of old_func to a new function livepatch: Allow user to specify functions to search for on a stack selftests/livepatch: Test of the API for specifying functions to search for on a stack
include/linux/livepatch.h | 11 +++ kernel/livepatch/core.c | 50 ++++++++--- kernel/livepatch/transition.c | 21 +++-- lib/Kconfig.debug | 1 + lib/livepatch/Makefile | 4 +- lib/livepatch/test_klp_funcstack_demo.c | 61 +++++++++++++ lib/livepatch/test_klp_funcstack_mod.c | 72 +++++++++++++++ tools/testing/selftests/livepatch/Makefile | 3 +- .../selftests/livepatch/test-func-stack.sh | 88 +++++++++++++++++++ 9 files changed, 293 insertions(+), 18 deletions(-) create mode 100644 lib/livepatch/test_klp_funcstack_demo.c create mode 100644 lib/livepatch/test_klp_funcstack_mod.c create mode 100755 tools/testing/selftests/livepatch/test-func-stack.sh
struct klp_func will be used not only for functions to be patched but also for functions which must not be found on a stack. Move the initialization of needed struct members to a separate function, so the code can be reused.
Signed-off-by: Miroslav Benes mbenes@suse.cz --- kernel/livepatch/core.c | 34 +++++++++++++++++++++++----------- 1 file changed, 23 insertions(+), 11 deletions(-)
diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c index 335d988bd811..3d8e3caf9f92 100644 --- a/kernel/livepatch/core.c +++ b/kernel/livepatch/core.c @@ -767,6 +767,28 @@ static int klp_apply_object_relocs(struct klp_patch *patch, return 0; }
+static int klp_init_old_func(struct klp_object *obj, + struct klp_func *func) +{ + int ret; + + ret = klp_find_object_symbol(obj->name, func->old_name, + func->old_sympos, + (unsigned long *)&func->old_func); + if (ret) + return ret; + + ret = kallsyms_lookup_size_offset((unsigned long)func->old_func, + &func->old_size, NULL); + if (!ret) { + pr_err("kallsyms size lookup failed for '%s'\n", + func->old_name); + return -ENOENT; + } + + return 0; +} + /* parts of the initialization that is done only when the object is loaded */ static int klp_init_object_loaded(struct klp_patch *patch, struct klp_object *obj) @@ -787,20 +809,10 @@ static int klp_init_object_loaded(struct klp_patch *patch, }
klp_for_each_func(obj, func) { - ret = klp_find_object_symbol(obj->name, func->old_name, - func->old_sympos, - (unsigned long *)&func->old_func); + ret = klp_init_old_func(obj, func); if (ret) return ret;
- ret = kallsyms_lookup_size_offset((unsigned long)func->old_func, - &func->old_size, NULL); - if (!ret) { - pr_err("kallsyms size lookup failed for '%s'\n", - func->old_name); - return -ENOENT; - } - if (func->nop) func->new_func = func->old_func;
livepatch's consistency model requires that no live patched function must be found on any task's stack during a transition process after a live patch is applied. It is achieved by walking through stacks of all blocked tasks.
The user might also want to define more functions to search for without them being patched at all. It may either help with preparing a live patch, which would otherwise require additional touches to achieve the consistency, or it can be used to overcome deficiencies the stack checking inherently has. For example, GCC may optimize a function so that a part of it is moved to a different section and the function would jump to it. This child function would not be found on a stack in this case, but it may be important to search for it so that, again, the consistency is achieved.
Allow the user to specify such functions on klp_object level.
Signed-off-by: Miroslav Benes mbenes@suse.cz --- include/linux/livepatch.h | 11 +++++++++++ kernel/livepatch/core.c | 16 ++++++++++++++++ kernel/livepatch/transition.c | 21 ++++++++++++++++----- 3 files changed, 43 insertions(+), 5 deletions(-)
diff --git a/include/linux/livepatch.h b/include/linux/livepatch.h index 2614247a9781..89df578af8c3 100644 --- a/include/linux/livepatch.h +++ b/include/linux/livepatch.h @@ -106,9 +106,11 @@ struct klp_callbacks { * struct klp_object - kernel object structure for live patching * @name: module name (or NULL for vmlinux) * @funcs: function entries for functions to be patched in the object + * @funcs_stack: function entries for functions to be stack checked * @callbacks: functions to be executed pre/post (un)patching * @kobj: kobject for sysfs resources * @func_list: dynamic list of the function entries + * @func_stack_list: dynamic list of the function entries for stack checking * @node: list node for klp_patch obj_list * @mod: kernel module associated with the patched object * (NULL for vmlinux) @@ -119,11 +121,13 @@ struct klp_object { /* external */ const char *name; struct klp_func *funcs; + struct klp_func *funcs_stack; struct klp_callbacks callbacks;
/* internal */ struct kobject kobj; struct list_head func_list; + struct list_head func_stack_list; struct list_head node; struct module *mod; bool dynamic; @@ -187,12 +191,19 @@ struct klp_patch { func->old_name || func->new_func || func->old_sympos; \ func++)
+#define klp_for_each_func_stack_static(obj, func) \ + for (func = obj->funcs_stack; \ + func && (func->old_name || func->old_sympos); func++) + #define klp_for_each_func_safe(obj, func, tmp_func) \ list_for_each_entry_safe(func, tmp_func, &obj->func_list, node)
#define klp_for_each_func(obj, func) \ list_for_each_entry(func, &obj->func_list, node)
+#define klp_for_each_func_stack(obj, func) \ + list_for_each_entry(func, &obj->func_stack_list, node) + int klp_enable_patch(struct klp_patch *);
/* Called from the module loader during module coming/going states */ diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c index 3d8e3caf9f92..86fc73a06844 100644 --- a/kernel/livepatch/core.c +++ b/kernel/livepatch/core.c @@ -825,6 +825,12 @@ static int klp_init_object_loaded(struct klp_patch *patch, } }
+ klp_for_each_func_stack(obj, func) { + ret = klp_init_old_func(obj, func); + if (ret) + return ret; + } + return 0; }
@@ -853,6 +859,11 @@ static int klp_init_object(struct klp_patch *patch, struct klp_object *obj) return ret; }
+ klp_for_each_func_stack(obj, func) { + if (strlen(func->old_name) >= KSYM_NAME_LEN) + return -EINVAL; + } + if (klp_is_object_loaded(obj)) ret = klp_init_object_loaded(patch, obj);
@@ -870,6 +881,7 @@ static void klp_init_object_early(struct klp_patch *patch, struct klp_object *obj) { INIT_LIST_HEAD(&obj->func_list); + INIT_LIST_HEAD(&obj->func_stack_list); kobject_init(&obj->kobj, &klp_ktype_object); list_add_tail(&obj->node, &patch->obj_list); } @@ -899,6 +911,10 @@ static int klp_init_patch_early(struct klp_patch *patch) klp_for_each_func_static(obj, func) { klp_init_func_early(obj, func); } + + klp_for_each_func_stack_static(obj, func) { + list_add_tail(&func->node, &obj->func_stack_list); + } }
if (!try_module_get(patch->mod)) diff --git a/kernel/livepatch/transition.c b/kernel/livepatch/transition.c index 5683ac0d2566..be7afc5dc275 100644 --- a/kernel/livepatch/transition.c +++ b/kernel/livepatch/transition.c @@ -200,7 +200,10 @@ static int klp_check_stack_func(struct klp_func *func, unsigned long *entries, for (i = 0; i < nr_entries; i++) { address = entries[i];
- if (klp_target_state == KLP_UNPATCHED) { + if (!func->new_func) { + func_addr = (unsigned long)func->old_func; + func_size = func->old_size; + } else if (klp_target_state == KLP_UNPATCHED) { /* * Check for the to-be-unpatched function * (the func itself). @@ -256,14 +259,22 @@ static int klp_check_stack(struct task_struct *task, const char **oldname) continue; klp_for_each_func(obj, func) { ret = klp_check_stack_func(func, entries, nr_entries); - if (ret) { - *oldname = func->old_name; - return -EADDRINUSE; - } + if (ret) + goto err; + } + + klp_for_each_func_stack(obj, func) { + ret = klp_check_stack_func(func, entries, nr_entries); + if (ret) + goto err; } }
return 0; + +err: + *oldname = func->old_name; + return -EADDRINUSE; }
static int klp_check_and_switch_task(struct task_struct *task, void *arg)
Add a test for the API which allows the user to specify functions which are then searched for on any tasks's stack during a transition process.
Signed-off-by: Miroslav Benes mbenes@suse.cz --- lib/Kconfig.debug | 1 + lib/livepatch/Makefile | 4 +- lib/livepatch/test_klp_funcstack_demo.c | 61 +++++++++++++ lib/livepatch/test_klp_funcstack_mod.c | 72 +++++++++++++++ tools/testing/selftests/livepatch/Makefile | 3 +- .../selftests/livepatch/test-func-stack.sh | 88 +++++++++++++++++++ 6 files changed, 227 insertions(+), 2 deletions(-) create mode 100644 lib/livepatch/test_klp_funcstack_demo.c create mode 100644 lib/livepatch/test_klp_funcstack_mod.c create mode 100755 tools/testing/selftests/livepatch/test-func-stack.sh
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 9ef7ce18b4f5..aa4c97098f41 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -2529,6 +2529,7 @@ config TEST_LIVEPATCH default n depends on DYNAMIC_DEBUG depends on LIVEPATCH + depends on DEBUG_FS depends on m help Test kernel livepatching features for correctness. The tests will diff --git a/lib/livepatch/Makefile b/lib/livepatch/Makefile index dcc912b3478f..584e3b8b5415 100644 --- a/lib/livepatch/Makefile +++ b/lib/livepatch/Makefile @@ -11,4 +11,6 @@ obj-$(CONFIG_TEST_LIVEPATCH) += test_klp_atomic_replace.o \ test_klp_shadow_vars.o \ test_klp_state.o \ test_klp_state2.o \ - test_klp_state3.o + test_klp_state3.o \ + test_klp_funcstack_mod.o \ + test_klp_funcstack_demo.o diff --git a/lib/livepatch/test_klp_funcstack_demo.c b/lib/livepatch/test_klp_funcstack_demo.c new file mode 100644 index 000000000000..902798077f05 --- /dev/null +++ b/lib/livepatch/test_klp_funcstack_demo.c @@ -0,0 +1,61 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (C) 2021 Miroslav Benes mbenes@suse.cz + +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt + +#include <linux/module.h> +#include <linux/kernel.h> +#include <linux/livepatch.h> + +static int funcstack; +module_param(funcstack, int, 0644); +MODULE_PARM_DESC(funcstack, "func_stack (default=0)"); + +static noinline void livepatch_child2_function(void) +{ + pr_info("%s\n", __func__); +} + +static struct klp_func funcs[] = { + { + .old_name = "child2_function", + .new_func = livepatch_child2_function, + }, {} +}; + +static struct klp_func funcs_stack[] = { + { + .old_name = "parent_function", + }, {} +}; + +static struct klp_object objs[] = { + { + .name = "test_klp_funcstack_mod", + .funcs = funcs, + }, {} +}; + +static struct klp_patch patch = { + .mod = THIS_MODULE, + .objs = objs, +}; + +static int test_klp_funcstack_demo_init(void) +{ + if (funcstack) + objs[0].funcs_stack = funcs_stack; + + return klp_enable_patch(&patch); +} + +static void test_klp_funcstack_demo_exit(void) +{ +} + +module_init(test_klp_funcstack_demo_init); +module_exit(test_klp_funcstack_demo_exit); +MODULE_LICENSE("GPL"); +MODULE_INFO(livepatch, "Y"); +MODULE_AUTHOR("Miroslav Benes mbenes@suse.cz"); +MODULE_DESCRIPTION("Livepatch test: func_stack demo"); diff --git a/lib/livepatch/test_klp_funcstack_mod.c b/lib/livepatch/test_klp_funcstack_mod.c new file mode 100644 index 000000000000..127c6093d890 --- /dev/null +++ b/lib/livepatch/test_klp_funcstack_mod.c @@ -0,0 +1,72 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (C) 2021 Miroslav Benes mbenes@suse.cz + +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt + +#include <linux/module.h> +#include <linux/kernel.h> +#include <linux/debugfs.h> +#include <linux/delay.h> + +static int sleep_length = 10000; +module_param(sleep_length, int, 0644); +MODULE_PARM_DESC(sleep_length, "length of sleep in seconds (default=10)"); + +static noinline void child_function(void) +{ + pr_info("%s enter\n", __func__); + msleep(sleep_length); + pr_info("%s exit\n", __func__); +} + +static noinline void child2_function(void) +{ + pr_info("%s\n", __func__); +} + +static noinline void parent_function(void) +{ + pr_info("%s enter\n", __func__); + child_function(); + child2_function(); + pr_info("%s exit\n", __func__); +} + +static int parent_function_get(void *data, u64 *val) +{ + *val = 0; + parent_function(); + + return 0; +} + +DEFINE_DEBUGFS_ATTRIBUTE(fops_parent_function, parent_function_get, NULL, "%llu\n"); + +static struct dentry *debugfs_dir; + +static int test_klp_funcstack_mod_init(void) +{ + struct dentry *d; + + debugfs_dir = debugfs_create_dir("test_klp_funcstack", NULL); + if (IS_ERR(debugfs_dir)) + return PTR_ERR(debugfs_dir); + + d = debugfs_create_file("parent_function", 0400, debugfs_dir, NULL, + &fops_parent_function); + if (IS_ERR(d)) + debugfs_remove_recursive(debugfs_dir); + + return 0; +} + +static void test_klp_funcstack_mod_exit(void) +{ + debugfs_remove_recursive(debugfs_dir); +} + +module_init(test_klp_funcstack_mod_init); +module_exit(test_klp_funcstack_mod_exit); +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Miroslav Benes mbenes@suse.cz"); +MODULE_DESCRIPTION("Livepatch test: func_stack module"); diff --git a/tools/testing/selftests/livepatch/Makefile b/tools/testing/selftests/livepatch/Makefile index 1acc9e1fa3fb..40f8a3a2e9aa 100644 --- a/tools/testing/selftests/livepatch/Makefile +++ b/tools/testing/selftests/livepatch/Makefile @@ -6,7 +6,8 @@ TEST_PROGS := \ test-callbacks.sh \ test-shadow-vars.sh \ test-state.sh \ - test-ftrace.sh + test-ftrace.sh \ + test-func-stack.sh
TEST_FILES := settings
diff --git a/tools/testing/selftests/livepatch/test-func-stack.sh b/tools/testing/selftests/livepatch/test-func-stack.sh new file mode 100755 index 000000000000..b7da62c9f5a1 --- /dev/null +++ b/tools/testing/selftests/livepatch/test-func-stack.sh @@ -0,0 +1,88 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2021 Miroslav Benes mbenes@suse.cz + +. $(dirname $0)/functions.sh + +MOD_TARGET=test_klp_funcstack_mod +MOD_LIVEPATCH=test_klp_funcstack_demo + +setup_config + +# - load a target module and call its parent_function(). It will sleep in its +# child_function() callee. +# - load a live patch with new child2_function() called from parent_function() +# too. The patching does not wait for child_function() to return, because +# child2_function() is not on any stack. +# - clean up afterwards + +start_test "non-blocking patching without the function on a stack" + +load_mod $MOD_TARGET + +(cat /sys/kernel/debug/test_klp_funcstack/parent_function) >/dev/null & +PID=$! + +load_lp $MOD_LIVEPATCH + +wait $PID + +disable_lp $MOD_LIVEPATCH +unload_lp $MOD_LIVEPATCH +unload_mod $MOD_TARGET + +check_result "% modprobe $MOD_TARGET +$MOD_TARGET: parent_function enter +$MOD_TARGET: child_function enter +% modprobe $MOD_LIVEPATCH +livepatch: enabling patch '$MOD_LIVEPATCH' +livepatch: '$MOD_LIVEPATCH': initializing patching transition +livepatch: '$MOD_LIVEPATCH': starting patching transition +livepatch: '$MOD_LIVEPATCH': completing patching transition +livepatch: '$MOD_LIVEPATCH': patching complete +$MOD_TARGET: child_function exit +$MOD_LIVEPATCH: livepatch_child2_function +$MOD_TARGET: parent_function exit +% echo 0 > /sys/kernel/livepatch/$MOD_LIVEPATCH/enabled +livepatch: '$MOD_LIVEPATCH': initializing unpatching transition +livepatch: '$MOD_LIVEPATCH': starting unpatching transition +livepatch: '$MOD_LIVEPATCH': completing unpatching transition +livepatch: '$MOD_LIVEPATCH': unpatching complete +% rmmod $MOD_LIVEPATCH +% rmmod $MOD_TARGET" + +# Similar to the previous test but now the patching has to wait for +# child2_function() to return, because parent_function() is also checked for. + +start_test "patching delayed due to the function on a stack" + +load_mod $MOD_TARGET + +(cat /sys/kernel/debug/test_klp_funcstack/parent_function) >/dev/null & + +load_lp $MOD_LIVEPATCH funcstack=1 +disable_lp $MOD_LIVEPATCH +unload_lp $MOD_LIVEPATCH +unload_mod $MOD_TARGET + +check_result "% modprobe $MOD_TARGET +$MOD_TARGET: parent_function enter +$MOD_TARGET: child_function enter +% modprobe $MOD_LIVEPATCH funcstack=1 +livepatch: enabling patch '$MOD_LIVEPATCH' +livepatch: '$MOD_LIVEPATCH': initializing patching transition +livepatch: '$MOD_LIVEPATCH': starting patching transition +$MOD_TARGET: child_function exit +$MOD_TARGET: child2_function +$MOD_TARGET: parent_function exit +livepatch: '$MOD_LIVEPATCH': completing patching transition +livepatch: '$MOD_LIVEPATCH': patching complete +% echo 0 > /sys/kernel/livepatch/$MOD_LIVEPATCH/enabled +livepatch: '$MOD_LIVEPATCH': initializing unpatching transition +livepatch: '$MOD_LIVEPATCH': starting unpatching transition +livepatch: '$MOD_LIVEPATCH': completing unpatching transition +livepatch: '$MOD_LIVEPATCH': unpatching complete +% rmmod $MOD_LIVEPATCH +% rmmod $MOD_TARGET" + +exit 0
On Fri, Nov 19, 2021 at 10:03:26AM +0100, Miroslav Benes wrote:
livepatch's consistency model requires that no live patched function must be found on any task's stack during a transition process after a live patch is applied. It is achieved by walking through stacks of all blocked tasks.
The user might also want to define more functions to search for without them being patched at all. It may either help with preparing a live patch, which would otherwise require additional touches to achieve the consistency, or it can be used to overcome deficiencies the stack checking inherently has. For example, GCC may optimize a function so that a part of it is moved to a different section and the function would jump to it. This child function would not be found on a stack in this case, but it may be important to search for it so that, again, the consistency is achieved.
Allow the user to specify such functions on klp_object level.
Ok, so this relies on the patch generator to DTRT but then it should work.
Thanks!
Thanks for doing this! And at peterz-esque speed no less :-)
On Fri, Nov 19, 2021 at 10:03:26AM +0100, Miroslav Benes wrote:
livepatch's consistency model requires that no live patched function must be found on any task's stack during a transition process after a live patch is applied. It is achieved by walking through stacks of all blocked tasks.
The user might also want to define more functions to search for without them being patched at all. It may either help with preparing a live patch, which would otherwise require additional touches to achieve the consistency
Do we have any examples of this situation we can add to the commit log?
or it can be used to overcome deficiencies the stack checking inherently has. For example, GCC may optimize a function so that a part of it is moved to a different section and the function would jump to it. This child function would not be found on a stack in this case, but it may be important to search for it so that, again, the consistency is achieved.
Allow the user to specify such functions on klp_object level.
Signed-off-by: Miroslav Benes mbenes@suse.cz
include/linux/livepatch.h | 11 +++++++++++ kernel/livepatch/core.c | 16 ++++++++++++++++ kernel/livepatch/transition.c | 21 ++++++++++++++++----- 3 files changed, 43 insertions(+), 5 deletions(-)
diff --git a/include/linux/livepatch.h b/include/linux/livepatch.h index 2614247a9781..89df578af8c3 100644 --- a/include/linux/livepatch.h +++ b/include/linux/livepatch.h @@ -106,9 +106,11 @@ struct klp_callbacks {
- struct klp_object - kernel object structure for live patching
- @name: module name (or NULL for vmlinux)
- @funcs: function entries for functions to be patched in the object
- @funcs_stack: function entries for functions to be stack checked
So there are two arrays/lists of 'klp_func', and two implied meanings of what a 'klp_func' is and how it's initialized.
Might it be simpler and more explicit to just add a new external field to 'klp_func' and continue to have a single 'funcs' array? Similar to what we already do with the special-casing of 'nop', except it would be an external field, e.g. 'no_patch' or 'stack_only'.
Then instead of all the extra klp_for_each_func_stack_static() incantations, and the special cases in higher-level callers like klp_init_object() and klp_init_patch_early(), the lower-level functions like klp_init_func() and klp_init_func_early() can check the field to determine which initializations need to be made. Which is kind of nice IMO as it pushes that detail down more where it belongs. And makes the different types of 'klp_func' more explicit.
On Fri, 19 Nov 2021, Josh Poimboeuf wrote:
Thanks for doing this! And at peterz-esque speed no less :-)
On Fri, Nov 19, 2021 at 10:03:26AM +0100, Miroslav Benes wrote:
livepatch's consistency model requires that no live patched function must be found on any task's stack during a transition process after a live patch is applied. It is achieved by walking through stacks of all blocked tasks.
The user might also want to define more functions to search for without them being patched at all. It may either help with preparing a live patch, which would otherwise require additional touches to achieve the consistency
Do we have any examples of this situation we can add to the commit log?
I do not have anything at hand. Joe, do you remember the case you mentioned previously about adding a nop to a function?
or it can be used to overcome deficiencies the stack checking inherently has. For example, GCC may optimize a function so that a part of it is moved to a different section and the function would jump to it. This child function would not be found on a stack in this case, but it may be important to search for it so that, again, the consistency is achieved.
Allow the user to specify such functions on klp_object level.
Signed-off-by: Miroslav Benes mbenes@suse.cz
include/linux/livepatch.h | 11 +++++++++++ kernel/livepatch/core.c | 16 ++++++++++++++++ kernel/livepatch/transition.c | 21 ++++++++++++++++----- 3 files changed, 43 insertions(+), 5 deletions(-)
diff --git a/include/linux/livepatch.h b/include/linux/livepatch.h index 2614247a9781..89df578af8c3 100644 --- a/include/linux/livepatch.h +++ b/include/linux/livepatch.h @@ -106,9 +106,11 @@ struct klp_callbacks {
- struct klp_object - kernel object structure for live patching
- @name: module name (or NULL for vmlinux)
- @funcs: function entries for functions to be patched in the object
- @funcs_stack: function entries for functions to be stack checked
So there are two arrays/lists of 'klp_func', and two implied meanings of what a 'klp_func' is and how it's initialized.
Might it be simpler and more explicit to just add a new external field to 'klp_func' and continue to have a single 'funcs' array? Similar to what we already do with the special-casing of 'nop', except it would be an external field, e.g. 'no_patch' or 'stack_only'.
Then instead of all the extra klp_for_each_func_stack_static() incantations, and the special cases in higher-level callers like klp_init_object() and klp_init_patch_early(), the lower-level functions like klp_init_func() and klp_init_func_early() can check the field to determine which initializations need to be made. Which is kind of nice IMO as it pushes that detail down more where it belongs. And makes the different types of 'klp_func' more explicit.
I thought about doing this for a moment but then I was worried there would be many places which would require special-casing, so I tried to keep it separate. But yes, it would be cleaner, so definitely worth trying for v2.
Thanks
Miroslav
On 11/22/21 2:57 AM, Miroslav Benes wrote:
On Fri, 19 Nov 2021, Josh Poimboeuf wrote:
Thanks for doing this! And at peterz-esque speed no less :-)
On Fri, Nov 19, 2021 at 10:03:26AM +0100, Miroslav Benes wrote:
livepatch's consistency model requires that no live patched function must be found on any task's stack during a transition process after a live patch is applied. It is achieved by walking through stacks of all blocked tasks.
The user might also want to define more functions to search for without them being patched at all. It may either help with preparing a live patch, which would otherwise require additional touches to achieve the consistency
Do we have any examples of this situation we can add to the commit log?
I do not have anything at hand. Joe, do you remember the case you mentioned previously about adding a nop to a function?
I went looking in my inbox to see... Unfortunately the closest thing I found was a kpatchset in which we added nops to coax kpatch-build into reverting previous patch version changes. Not applicable here, but I was certain we entertained the same idea to increase the task stack check for some other problem.
Maybe adding a hypothetical scenario to the commit log would suffice?
or it can be used to overcome deficiencies the stack checking inherently has. For example, GCC may optimize a function so that a part of it is moved to a different section and the function would jump to it. This child function would not be found on a stack in this case, but it may be important to search for it so that, again, the consistency is achieved.
Allow the user to specify such functions on klp_object level.
Signed-off-by: Miroslav Benes mbenes@suse.cz
include/linux/livepatch.h | 11 +++++++++++ kernel/livepatch/core.c | 16 ++++++++++++++++ kernel/livepatch/transition.c | 21 ++++++++++++++++----- 3 files changed, 43 insertions(+), 5 deletions(-)
diff --git a/include/linux/livepatch.h b/include/linux/livepatch.h index 2614247a9781..89df578af8c3 100644 --- a/include/linux/livepatch.h +++ b/include/linux/livepatch.h @@ -106,9 +106,11 @@ struct klp_callbacks {
- struct klp_object - kernel object structure for live patching
- @name: module name (or NULL for vmlinux)
- @funcs: function entries for functions to be patched in the object
- @funcs_stack: function entries for functions to be stack checked
So there are two arrays/lists of 'klp_func', and two implied meanings of what a 'klp_func' is and how it's initialized.
Might it be simpler and more explicit to just add a new external field to 'klp_func' and continue to have a single 'funcs' array? Similar to what we already do with the special-casing of 'nop', except it would be an external field, e.g. 'no_patch' or 'stack_only'.
Then instead of all the extra klp_for_each_func_stack_static() incantations, and the special cases in higher-level callers like klp_init_object() and klp_init_patch_early(), the lower-level functions like klp_init_func() and klp_init_func_early() can check the field to determine which initializations need to be made. Which is kind of nice IMO as it pushes that detail down more where it belongs. And makes the different types of 'klp_func' more explicit.
I thought about doing this for a moment but then I was worried there would be many places which would require special-casing, so I tried to keep it separate. But yes, it would be cleaner, so definitely worth trying for v2.
I'll add that the first thing that came to mind when you raised this feature idea in the other thread was to support existing klp_funcs array with NULL new_func's. I didn't go look to see how invasive it would be, but it will be interesting to see if a single list approach turns out any simpler for v2.
On Mon 2021-11-22 10:53:21, Joe Lawrence wrote:
On 11/22/21 2:57 AM, Miroslav Benes wrote:
On Fri, 19 Nov 2021, Josh Poimboeuf wrote:
Thanks for doing this! And at peterz-esque speed no less :-)
On Fri, Nov 19, 2021 at 10:03:26AM +0100, Miroslav Benes wrote:
livepatch's consistency model requires that no live patched function must be found on any task's stack during a transition process after a live patch is applied. It is achieved by walking through stacks of all blocked tasks.
The user might also want to define more functions to search for without them being patched at all. It may either help with preparing a live patch, which would otherwise require additional touches to achieve the consistency
Do we have any examples of this situation we can add to the commit log?
I do not have anything at hand. Joe, do you remember the case you mentioned previously about adding a nop to a function?
Maybe adding a hypothetical scenario to the commit log would suffice?
I wonder if we could describe a scenario based on the thread about .cold code variants, see https://lore.kernel.org/all/20211112015003.pefl656m3zmir6ov@treble/
This feature would allow to safely livepatch already released kernels where the unwinder is not able to reliably detect a newly discovered problems.
or it can be used to overcome deficiencies the stack checking inherently has. For example, GCC may optimize a function so that a part of it is moved to a different section and the function would jump to it. This child function would not be found on a stack in this case, but it may be important to search for it so that, again, the consistency is achieved.
Allow the user to specify such functions on klp_object level.
Signed-off-by: Miroslav Benes mbenes@suse.cz
include/linux/livepatch.h | 11 +++++++++++ kernel/livepatch/core.c | 16 ++++++++++++++++ kernel/livepatch/transition.c | 21 ++++++++++++++++----- 3 files changed, 43 insertions(+), 5 deletions(-)
diff --git a/include/linux/livepatch.h b/include/linux/livepatch.h index 2614247a9781..89df578af8c3 100644 --- a/include/linux/livepatch.h +++ b/include/linux/livepatch.h @@ -106,9 +106,11 @@ struct klp_callbacks {
- struct klp_object - kernel object structure for live patching
- @name: module name (or NULL for vmlinux)
- @funcs: function entries for functions to be patched in the object
- @funcs_stack: function entries for functions to be stack checked
So there are two arrays/lists of 'klp_func', and two implied meanings of what a 'klp_func' is and how it's initialized.
Might it be simpler and more explicit to just add a new external field to 'klp_func' and continue to have a single 'funcs' array? Similar to what we already do with the special-casing of 'nop', except it would be an external field, e.g. 'no_patch' or 'stack_only'.
I'll add that the first thing that came to mind when you raised this feature idea in the other thread was to support existing klp_funcs array with NULL new_func's.
Please, solve this with the extra flag, e.g. .stack_only, as already suggested. It will help to distinguish mistakes and intentions. Also it will allow to find these symbols by grep.
I didn't go look to see how invasive it would be, but it will be interesting to see if a single list approach turns out any simpler for v2.
I am not sure either. But I expect that it will be easier than the extra array.
Best Regards, Petr
On Thu, 25 Nov 2021, Petr Mladek wrote:
On Mon 2021-11-22 10:53:21, Joe Lawrence wrote:
On 11/22/21 2:57 AM, Miroslav Benes wrote:
On Fri, 19 Nov 2021, Josh Poimboeuf wrote:
Thanks for doing this! And at peterz-esque speed no less :-)
On Fri, Nov 19, 2021 at 10:03:26AM +0100, Miroslav Benes wrote:
livepatch's consistency model requires that no live patched function must be found on any task's stack during a transition process after a live patch is applied. It is achieved by walking through stacks of all blocked tasks.
The user might also want to define more functions to search for without them being patched at all. It may either help with preparing a live patch, which would otherwise require additional touches to achieve the consistency
Do we have any examples of this situation we can add to the commit log?
I do not have anything at hand. Joe, do you remember the case you mentioned previously about adding a nop to a function?
Maybe adding a hypothetical scenario to the commit log would suffice?
I wonder if we could describe a scenario based on the thread about .cold code variants, see https://lore.kernel.org/all/20211112015003.pefl656m3zmir6ov@treble/
This feature would allow to safely livepatch already released kernels where the unwinder is not able to reliably detect a newly discovered problems.
and is described (well, without actually using .cold suffix) a few lines below. I'll try to improve the changelog.
or it can be used to overcome deficiencies the stack checking inherently has. For example, GCC may optimize a function so that a part of it is moved to a different section and the function would jump to it. This child function would not be found on a stack in this case, but it may be important to search for it so that, again, the consistency is achieved.
Allow the user to specify such functions on klp_object level.
Signed-off-by: Miroslav Benes mbenes@suse.cz
include/linux/livepatch.h | 11 +++++++++++ kernel/livepatch/core.c | 16 ++++++++++++++++ kernel/livepatch/transition.c | 21 ++++++++++++++++----- 3 files changed, 43 insertions(+), 5 deletions(-)
diff --git a/include/linux/livepatch.h b/include/linux/livepatch.h index 2614247a9781..89df578af8c3 100644 --- a/include/linux/livepatch.h +++ b/include/linux/livepatch.h @@ -106,9 +106,11 @@ struct klp_callbacks {
- struct klp_object - kernel object structure for live patching
- @name: module name (or NULL for vmlinux)
- @funcs: function entries for functions to be patched in the object
- @funcs_stack: function entries for functions to be stack checked
So there are two arrays/lists of 'klp_func', and two implied meanings of what a 'klp_func' is and how it's initialized.
Might it be simpler and more explicit to just add a new external field to 'klp_func' and continue to have a single 'funcs' array? Similar to what we already do with the special-casing of 'nop', except it would be an external field, e.g. 'no_patch' or 'stack_only'.
I'll add that the first thing that came to mind when you raised this feature idea in the other thread was to support existing klp_funcs array with NULL new_func's.
Please, solve this with the extra flag, e.g. .stack_only, as already suggested. It will help to distinguish mistakes and intentions. Also it will allow to find these symbols by grep.
Indeed, that is what I did for v2. I used new_func being NULL fact even in v1, but I do not like it much. Extra flag is definitely more robust.
I didn't go look to see how invasive it would be, but it will be interesting to see if a single list approach turns out any simpler for v2.
I am not sure either. But I expect that it will be easier than the extra array.
So, extra flag and one array makes certain things (initialization) definitely easier. On the other hand, there are suddenly more problems to think about (and I haven't solved them yet):
- I need to make it work with nops functions. Especially if we allow a scenario where there could be klp_object instance with just stack_only functions. Would that be useful? For example, to patch something in a module and add a stack_only for a function in vmlinux.
If yes, then the interaction with nops is not completely straightforward and also some parts of the code would have to be changed (for example how obj->patched flag is handled).
- klp_func instances are directly mirrored in sysfs. Do we want to keep stack_only functions there too? If not, it makes the whole thing slighly more difficult given how we manage kobjects.
Nothing really difficult to implement if we come up with answers.
Miroslav
On Fri 2021-11-19 10:03:27, Miroslav Benes wrote:
Add a test for the API which allows the user to specify functions which are then searched for on any tasks's stack during a transition process.
--- /dev/null +++ b/lib/livepatch/test_klp_funcstack_mod.c @@ -0,0 +1,72 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (C) 2021 Miroslav Benes mbenes@suse.cz
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+#include <linux/module.h> +#include <linux/kernel.h> +#include <linux/debugfs.h> +#include <linux/delay.h>
+static int sleep_length = 10000; +module_param(sleep_length, int, 0644); +MODULE_PARM_DESC(sleep_length, "length of sleep in seconds (default=10)");
+static noinline void child_function(void) +{
- pr_info("%s enter\n", __func__);
- msleep(sleep_length);
The hardcoded sleep is not ideal. It might be too low or non-necessary high.
If I get it correctly, we are trying to achieve here the same as busymod_work_func() in test_klp_callbacks_busy.c.
The approach with debugfs is an interesting trick. Though, I slightly prefer using the scheduled work. The workqueue API looks less tricky to me than sysfs/debugfs API. Also it does not block the module in the init() callback[*]. But I might be biased.
Anyway, it might make sense to use the same trick in both situations. It would make it easier to maintain the test modules.
[*] There is actually a race in the workqueue approach. The module init() callback should wait until the work is really scheduled and sleeping. It might be achieved by similar hand-shake like with @block_transition variable. Or completion API might be even more elegant.
- pr_info("%s exit\n", __func__);
+}
+static noinline void child2_function(void) +{
- pr_info("%s\n", __func__);
+}
+static noinline void parent_function(void) +{
- pr_info("%s enter\n", __func__);
- child_function();
- child2_function();
This would deserve some explanation what we try to simulate here and how it is achieved. It is not easy for me even with the background that I have freshly in my mind.
Also I think about more descriptive names ;-)
What about something like this (using workqueue work and completion):
/* * Simulate part of the caller code that is in another .elf section * and is reached via jump. It this was really the case then the stack * unwinder might not be able to detect that the process is sleeping * in the caller. */ static void simulate_jump_part(void) { pr_info("%s enter\n", __func__);
/* Stay in the jump part unless told to leave. */ wait_for_completion(finish_jump);
pr_info("%s exit\n", __func__); }
/* * Simulate modified part of the caller code. It should never get * livepatched when the caller is sleeping in the just_part(). */ static void simulate_modified_part(void) { pr_info("%s\n", __func__); }
static void test_not_on_stack_func_work(struct work_struct *work) { pr_info("%s enter\n", __func__);
/* Simulation ready */ complete(work_started);
simulate_jump_part(); simulate_modified_part();
pr_info("%s exit\n", __func__); }
static int test_klp_no_on_stack_init(void) { pr_info("%s\n", __func__);
schedule_work(&work); wait_for_completion(&work_started);
return 0; }
static void test_not_on_stack_exit(void) { complete(&finish_jump); flush_work(&work); pr_info("%s\n", __func__); }
module_init(test_klp_not_on_stack_init); module_exit(test_klp_not_on_stack_exit);
- pr_info("%s exit\n", __func__);
+}
Best Regards, Petr
On Thu, 25 Nov 2021, Petr Mladek wrote:
On Fri 2021-11-19 10:03:27, Miroslav Benes wrote:
Add a test for the API which allows the user to specify functions which are then searched for on any tasks's stack during a transition process.
--- /dev/null +++ b/lib/livepatch/test_klp_funcstack_mod.c @@ -0,0 +1,72 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (C) 2021 Miroslav Benes mbenes@suse.cz
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+#include <linux/module.h> +#include <linux/kernel.h> +#include <linux/debugfs.h> +#include <linux/delay.h>
+static int sleep_length = 10000; +module_param(sleep_length, int, 0644); +MODULE_PARM_DESC(sleep_length, "length of sleep in seconds (default=10)");
+static noinline void child_function(void) +{
- pr_info("%s enter\n", __func__);
- msleep(sleep_length);
The hardcoded sleep is not ideal. It might be too low or non-necessary high.
It is not.
If I get it correctly, we are trying to achieve here the same as busymod_work_func() in test_klp_callbacks_busy.c.
Yes.
The approach with debugfs is an interesting trick. Though, I slightly prefer using the scheduled work. The workqueue API looks less tricky to me than sysfs/debugfs API. Also it does not block the module in the init() callback[*]. But I might be biased.
It seemed to me that debugfs gave us more control over the process than workqueues, but I do not really care. Could you explain the blocking in the init callback? I do not follow.
Anyway, it might make sense to use the same trick in both situations. It would make it easier to maintain the test modules.
True. So I will rewrite it to workqueues as you are proposing below.
[*] There is actually a race in the workqueue approach. The module init() callback should wait until the work is really scheduled and sleeping. It might be achieved by similar hand-shake like with @block_transition variable. Or completion API might be even more elegant.
- pr_info("%s exit\n", __func__);
+}
+static noinline void child2_function(void) +{
- pr_info("%s\n", __func__);
+}
+static noinline void parent_function(void) +{
- pr_info("%s enter\n", __func__);
- child_function();
- child2_function();
This would deserve some explanation what we try to simulate here and how it is achieved. It is not easy for me even with the background that I have freshly in my mind.
Also I think about more descriptive names ;-)
Hey, I thought it was self-explaining :). So, yes, I started with the example given in the .fixup thread, but it is not really tied to .cold section, jumps or whatever. The setup is just used to test a new API. Moreover, the .fixup example is just a one scenario the new API tries to solve.
What you propose below, that is function names and comments, is a bit confusing for me. Especially if I did not know anything about the original issue (which will be the case in a couple of weeks when I forget everything).
So I think it I will stick to brevity unless you or someone else really insist.
I can improve tests description in tools/testing/selftests/livepatch/test-func-stack.sh if it helps anything.
Miroslav
What about something like this (using workqueue work and completion):
/*
- Simulate part of the caller code that is in another .elf section
- and is reached via jump. It this was really the case then the stack
- unwinder might not be able to detect that the process is sleeping
- in the caller.
*/ static void simulate_jump_part(void) { pr_info("%s enter\n", __func__);
/* Stay in the jump part unless told to leave. */ wait_for_completion(finish_jump);
pr_info("%s exit\n", __func__); }
/*
- Simulate modified part of the caller code. It should never get
- livepatched when the caller is sleeping in the just_part().
*/ static void simulate_modified_part(void) { pr_info("%s\n", __func__); }
static void test_not_on_stack_func_work(struct work_struct *work) { pr_info("%s enter\n", __func__);
/* Simulation ready */ complete(work_started);
simulate_jump_part(); simulate_modified_part();
pr_info("%s exit\n", __func__); }
static int test_klp_no_on_stack_init(void) { pr_info("%s\n", __func__);
schedule_work(&work); wait_for_completion(&work_started);
return 0; }
static void test_not_on_stack_exit(void) { complete(&finish_jump); flush_work(&work); pr_info("%s\n", __func__); }
module_init(test_klp_not_on_stack_init); module_exit(test_klp_not_on_stack_exit);
- pr_info("%s exit\n", __func__);
+}
On Fri 2021-11-26 10:20:54, Miroslav Benes wrote:
On Thu, 25 Nov 2021, Petr Mladek wrote:
On Fri 2021-11-19 10:03:27, Miroslav Benes wrote:
Add a test for the API which allows the user to specify functions which are then searched for on any tasks's stack during a transition process.
The approach with debugfs is an interesting trick. Though, I slightly prefer using the scheduled work. The workqueue API looks less tricky to me than sysfs/debugfs API. Also it does not block the module in the init() callback[*]. But I might be biased.
It seemed to me that debugfs gave us more control over the process than workqueues, but I do not really care. Could you explain the blocking in the init callback? I do not follow.
Good question about the blocking! Please, forget it. I am not sure why I thought that the module might get blocked in the module_init() script.
Anyway, it might make sense to use the same trick in both situations. It would make it easier to maintain the test modules.
True. So I will rewrite it to workqueues as you are proposing below.
[*] There is actually a race in the workqueue approach. The module init() callback should wait until the work is really scheduled and sleeping. It might be achieved by similar hand-shake like with @block_transition variable. Or completion API might be even more elegant.
- pr_info("%s exit\n", __func__);
+}
+static noinline void child2_function(void) +{
- pr_info("%s\n", __func__);
+}
+static noinline void parent_function(void) +{
- pr_info("%s enter\n", __func__);
- child_function();
- child2_function();
This would deserve some explanation what we try to simulate here and how it is achieved. It is not easy for me even with the background that I have freshly in my mind.
Also I think about more descriptive names ;-)
Hey, I thought it was self-explaining :). So, yes, I started with the example given in the .fixup thread, but it is not really tied to .cold section, jumps or whatever. The setup is just used to test a new API. Moreover, the .fixup example is just a one scenario the new API tries to solve.
OK, single child() function should be enough for testing the behavior. We might sleep/wait in the parent().
I think that I was confused by the two child() functions. It looked like sleeping in a child function was important. And the "same" name of the two children did not help much to understand and distinguish the purpose.
What you propose below, that is function names and comments, is a bit confusing for me. Especially if I did not know anything about the original issue (which will be the case in a couple of weeks when I forget everything).
So I think it I will stick to brevity unless you or someone else really insist.
No, I do not resist on the complicated exmaple. When thinking about it, the easier test case the better. It should be enough to describe the real-life purpose of the API in the patch that introduces the API.
I can improve tests description in tools/testing/selftests/livepatch/test-func-stack.sh if it helps anything.
Yes, please. I miss some top-level descriptions of the tested functionality, something like:
# Tests for "bla bla" feature. # It allows to block transition of a process when it is running # parent() function and only the child() function is livepatched.
# This test does not use the feature. The transition finishes even # before parent() exits.
# In this test case, the livepatch is instructed to check also # parent() on stack. The transition must not finish before # parent() exists.
It would be nice to have these high-level descriptions even in the test modules.
Best Regards, Petr
On Thu 2021-11-25 11:20:04, Miroslav Benes wrote:
On Thu, 25 Nov 2021, Petr Mladek wrote:
On Mon 2021-11-22 10:53:21, Joe Lawrence wrote:
On 11/22/21 2:57 AM, Miroslav Benes wrote:
On Fri, 19 Nov 2021, Josh Poimboeuf wrote:
diff --git a/include/linux/livepatch.h b/include/linux/livepatch.h index 2614247a9781..89df578af8c3 100644 --- a/include/linux/livepatch.h +++ b/include/linux/livepatch.h @@ -106,9 +106,11 @@ struct klp_callbacks {
- struct klp_object - kernel object structure for live patching
- @name: module name (or NULL for vmlinux)
- @funcs: function entries for functions to be patched in the object
- @funcs_stack: function entries for functions to be stack checked
So there are two arrays/lists of 'klp_func', and two implied meanings of what a 'klp_func' is and how it's initialized.
Might it be simpler and more explicit to just add a new external field to 'klp_func' and continue to have a single 'funcs' array? Similar to what we already do with the special-casing of 'nop', except it would be an external field, e.g. 'no_patch' or 'stack_only'.
I'll add that the first thing that came to mind when you raised this feature idea in the other thread was to support existing klp_funcs array with NULL new_func's.
Please, solve this with the extra flag, e.g. .stack_only, as already suggested. It will help to distinguish mistakes and intentions. Also it will allow to find these symbols by grep.
Indeed, that is what I did for v2. I used new_func being NULL fact even in v1, but I do not like it much. Extra flag is definitely more robust.
I didn't go look to see how invasive it would be, but it will be interesting to see if a single list approach turns out any simpler for v2.
I am not sure either. But I expect that it will be easier than the extra array.
So, extra flag and one array makes certain things (initialization) definitely easier. On the other hand, there are suddenly more problems to think about (and I haven't solved them yet):
- I need to make it work with nops functions. Especially if we allow a scenario where there could be klp_object instance with just stack_only functions. Would that be useful? For example, to patch something in a module and add a stack_only for a function in vmlinux.
My naive expectation is that the struct klp_func with @stack_only flag might be handled the same way on most locations, except when:
+ func->new_func is handled + ftrace handler is added/removed
Something like:
--- a/kernel/livepatch/core.c +++ b/kernel/livepatch/core.c @@ -724,7 +724,7 @@ static int klp_init_func(struct klp_object *obj, struct klp_func *func) * NOPs get the address later. The patched module must be loaded, * see klp_init_object_loaded(). */ - if (!func->new_func && !func->nop) + if (!func->new_func && !func->nop && !func->stack_only) return -EINVAL;
if (strlen(func->old_name) >= KSYM_NAME_LEN) @@ -801,6 +801,9 @@ static int klp_init_object_loaded(struct klp_patch *patch, return -ENOENT; }
+ if (func->stack_only) + continue; + if (func->nop) func->new_func = func->old_func;
--- a/kernel/livepatch/patch.c +++ b/kernel/livepatch/patch.c @@ -247,6 +247,9 @@ static void __klp_unpatch_object(struct klp_object *obj, bool nops_only) struct klp_func *func;
klp_for_each_func(obj, func) { + if (func->stack_only) + continue; + if (nops_only && !func->nop) continue;
@@ -273,6 +276,8 @@ int klp_patch_object(struct klp_object *obj) return -EINVAL;
klp_for_each_func(obj, func) { + if (func->stack_only) + continue; ret = klp_patch_func(func); if (ret) { klp_unpatch_object(obj);
If yes, then the interaction with nops is not completely straightforward and also some parts of the code would have to be changed (for example how obj->patched flag is handled).
I would keep func->patched disabled for @stack_only entries.
But they should not affect obj->patched state. I mean that obj->patched should be set when the patch is enabled even when there are only "stack_only" funcs.
It might look a bit unclear. A solution might be to rename the flags:
+ func->patched -> func->active (and set even for @stack_only) + obj->patched -> obj->active (same as func)
But I am not sure if it is worth it.
- klp_func instances are directly mirrored in sysfs. Do we want to keep stack_only functions there too? If not, it makes the whole thing slighly more difficult given how we manage kobjects.
I would keep them in sysfs. It will be easier and it does not harm.
Nothing really difficult to implement if we come up with answers.
I am sorry for not answering this earlier. I have missed the questions :-(
Best Regards, Petr
linux-kselftest-mirror@lists.linaro.org
-
Joe Lawrence -
Josh Poimboeuf -
Miroslav Benes -
Peter Zijlstra -
Petr Mladek