On Fri, 13 Jun 2025 17:27:53 +0900 Masami Hiramatsu (Google) mhiramat@kernel.org wrote:
Run 3:
This is the same as Run 2, and clearer.
In do_int3(), if we hit a disappeared int3, it is evacuated after all. This means kprobe_int3_handler() is hit, and call get_kprobe() to find the corresponding kprobes. But,
ffffffff8150a040 <get_kprobe>: ffffffff8150a040: f3 0f 1e fa endbr64 ffffffff8150a044: e8 07 b0 e2 ff call ffffffff81335050 <__fentry__> ffffffff8150a049: 48 b8 eb 83 b5 80 46 movabs $0x61c8864680b583eb,%rax ffffffff8150a050: 86 c8 61
It hits the ftrace and hooked by fgraph, and eventually returns via ftrace_return_to_handler()
[ 137.338572] RIP: 0010:ftrace_return_to_handler+0xd5/0x1f0 [ 137.338577] Code: 00 89 55 c8 48 85 ff 74 07 4c 89 b7 80 00 00 00 49 8b 94 24 38 0b 00 00 48 98 48 8b 04 c2 48 c1 e8 0c 0f b7 c0 48 89 45 b8 cc <90> 48 8b 05 e3 ac c2 01 48 63 80 f8 00 00 00 48 0f a3 45 b8 72 39
This address is;
$ eu-addr2line -fi -e vmlinux ftrace_return_to_handler+0xd5 arch_static_branch inlined at /builds/linux/kernel/trace/fgraph.c:839:6 in ftrace_return_to_handler /builds/linux/arch/x86/include/asm/jump_label.h:36:2 __ftrace_return_to_handler /builds/linux/kernel/trace/fgraph.c:839:6 ftrace_return_to_handler /builds/linux/kernel/trace/fgraph.c:874:9
It is for static_branch, which also uses a text_poke.
#ifdef CONFIG_HAVE_STATIC_CALL if (static_branch_likely(&fgraph_do_direct)) { <====== if (test_bit(fgraph_direct_gops->idx, &bitmap)) static_call(fgraph_retfunc)(&trace, fgraph_direct_gops, fregs);
But actually, this static_branch modifies the kernel code with smp_text_poke_single() (note, this is a wrapper of smp_text_poke_batch).
And this is MISSED by the smp_text_poke_int3_handler() again and go through the kprobes, and hit ftrace (fgraph) and caused this loop.
So the fundamental issue is that smp_text_poke_batch missed handling INT3.
I guess some text_poke user do not get text_mutex?
Hmm, I've checked the smp_text_poke_* users, but it seems no problem. Basically, those smp_text_poke* user locks text_mutex, and another suspicious ftrace_start_up is also set under ftrace_lock. ftrace_arch_code_modify_post_process() is also paired with ftrace_arch_code_modify_prepare() and under ftrace_lock.
smp_text_poke_single() ftrace_mod_jmp() ftrace_enable_ftrace_graph_caller() ftrace_modify_all_code() -> see [*1] ftrace_disable_ftrace_graph_caller() ftrace_modify_all_code() -> see [*1] ftrace_update_ftrace_func() update_ftrace_func() ftrace_modify_all_code() -> see [*1]
smp_text_poke_batch_add() arch_jump_label_transform_queue() -> lock text_mutex ftrace_replace_code() ftrace_modify_all_code() <------[*1] arch_ftrace_update_code() ftrace_run_update_code() -> lock text_mutex ftrace_modify_code_direct() (only if ftrace_poke_late != 0) ftrace_make_nop() __ftrace_replace_code() <----[*3] ftrace_replace_code(weak) --> Not used on x86 (overridden) ftrace_modify_all_code() <--- [*1] arch_ftrace_update_code() <---- [*4] ftrace_run_update_code()-> lock text_mutex __ftrace_modify_code() ftrace_run_stop_machine() arch_ftrace_update_code(weak) -> overridden on x86 see [*4] ftrace_module_enable() -> lock text_mutex (see below) ftrace_init_nop() ftrace_nop_initialize() ftrace_update_code() ftrace_module_enable() -> lock text_mutex prepare_coming_module() load_modole() ftrace_process_locs() -> lock ftrace_lock. ftrace_init() -> OK (ftrace_poke_late == 0 because its early) ftrace_module_init() -> OK (ftrace_poke_late == 0 because module is not live) load_module() ftrace_make_call() __ftrace_replace_code() -> see [*3]
smp_text_poke_batch_finish() arch_jump_label_transform_apply() -> lock text_mutex ftrace_arch_code_modify_post_process() -> must be OK because this unlock text_mutex ftrace_run_update_code()-> paired with ftrace_arch_code_modify_prepare() ftrace_module_enable()-> paired with ftrace_arch_code_modify_prepare() (depends on ftrace_lock && ftrace_start_up) ftrace_replace_code() ftrace_modify_all_code() -> see [*1]
ftrace_start_up <does variable set under ftrace_lock ?> ftrace_startup() ftrace_startup_subops() register_ftrace_graph() -> lock ftrace_lock register_ftrace_function_probe() -> lock ftrace_lock register_ftrace_function_nolock() -> lock ftrace_lock ftrace_shutdown() unregister_ftrace_function() -> lock ftrace_lock
ftrace_arch_code_modify_prepare() < this set ftrace_poke_late = 1> ftrace_module_enable() -> lock ftrace_lock. ftrace_run_update_code() ftrace_run_modify_code() ftrace_ops_update_code() __ftrace_hash_move_and_update_ops() ftrace_update_ops() ftrace_startup_subops() register_ftrace_graph() -> lock ftrace_lock ftrace_shutdown_subops() unregister_ftrace_graph() -> lock ftrace_lock ftrace_hash_move_and_update_subops() ftrace_hash_move_and_update_ops() -> [*2] ftrace_hash_move_and_update_ops() <-- [*2] process_mod_list() -> lock ftrace_lock register_ftrace_function_probe() -> lock ftrace_lock unregister_ftrace_function_probe_func() -> lock ftrace_lock ftrace_set_hash() -> lock ftrace_lock ftrace_regex_release() -> lock ftrace_lock unregister_ftrace_function_probe_func() -> lock ftrace_lock ftrace_startup_enable() ftrace_startup_all() ftrace_pid_reset() -> lock ftrace_lock pid_write() -> lock ftrace_lock ftrace_startup() ftrace_startup_subops() register_ftrace_graph() -> lock ftrace_lock register_ftrace_function_probe() -> lock ftrace_lock register_ftrace_function_nolock() -> lock ftrace_lock ftrace_startup_sysctl() ftrace_enable_sysctl() -> lock ftrace_lock ftrace_shutdown() ftrace_shutdown_subops() unregister_ftrace_graph() -> lock ftrace_lock unregister_ftrace_function_probe_func() -> lock ftrace_lock ftrace_destroy_filter_files() -> lock ftrace_lock unregister_ftrace_function() -> lock ftrace_lock ftrace_shutdown_sysctl() ftrace_enable_sysctl() -> lock ftrace_lock
Thanks,