Dear stable team,
I noticed that cfa7f3d2c526 ("perf,x86: avoid missing caller address in stack traces captured in uprobe"), got backported into 6.1.113 and 6.6.55, but it contains a race condition which Jiri Olsa has fixed in mainline b583ef82b671 ("uprobes: Fix race in uprobe_free_utask"). I think this should be backported into those stable branches.
#regzbot title: uprobes: Fix race in uprobe_free_utask #regzbot introduced: cfa7f3d2c526 #regzbot link: https://lore.kernel.org/all/20250109141440.2692173-1-jolsa@kernel.org/
Link: https://lore.kernel.org/all/20250109141440.2692173-1-jolsa@kernel.org/
Note: Sorry if I am using the wrong process/form/format, I tried to follow "Option 2" from the stable-kernel-rules.html.
Best regards, Christian
On Tue, Mar 04, 2025 at 10:33:24AM +0000, Christian Simon wrote:
Dear stable team,
I noticed that cfa7f3d2c526 ("perf,x86: avoid missing caller address in stack traces captured in uprobe"), got backported into 6.1.113 and 6.6.55, but it contains a race condition which Jiri Olsa has fixed in mainline b583ef82b671 ("uprobes: Fix race in uprobe_free_utask"). I think this should be backported into those stable branches.
#regzbot title: uprobes: Fix race in uprobe_free_utask #regzbot introduced: cfa7f3d2c526 #regzbot link: https://lore.kernel.org/all/20250109141440.2692173-1-jolsa@kernel.org/
Link: https://lore.kernel.org/all/20250109141440.2692173-1-jolsa@kernel.org/
Note: Sorry if I am using the wrong process/form/format, I tried to follow "Option 2" from the stable-kernel-rules.html.
As the patch does not apply cleanly to those kernels, how did you test this?
Can you provide some working, and tested, patches for this? We'll be glad to queue them up if so.
thanks,
greg k-h
commit b583ef82b671c9a752fbe3e95bd4c1c51eab764d upstream.
Christian Simon verified the regression exists in v6.6.80 as per method below and backported the mainline fix to the older version of uprobe_free_utask. After that change I can no longer reproduce the race with this method within 1h, while before it would show the panic under a minute.
Max Makarov reported kernel panic [1] in perf user callchain code.
The reason for that is the race between uprobe_free_utask and bpf profiler code doing the perf user stack unwind and is triggered within uprobe_free_utask function: - after current->utask is freed and - before current->utask is set to NULL
general protection fault, probably for non-canonical address 0x9e759c37ee555c76: 0000 [#1] SMP PTI RIP: 0010:is_uprobe_at_func_entry+0x28/0x80 ... ? die_addr+0x36/0x90 ? exc_general_protection+0x217/0x420 ? asm_exc_general_protection+0x26/0x30 ? is_uprobe_at_func_entry+0x28/0x80 perf_callchain_user+0x20a/0x360 get_perf_callchain+0x147/0x1d0 bpf_get_stackid+0x60/0x90 bpf_prog_9aac297fb833e2f5_do_perf_event+0x434/0x53b ? __smp_call_single_queue+0xad/0x120 bpf_overflow_handler+0x75/0x110 ... asm_sysvec_apic_timer_interrupt+0x1a/0x20 RIP: 0010:__kmem_cache_free+0x1cb/0x350 ... ? uprobe_free_utask+0x62/0x80 ? acct_collect+0x4c/0x220 uprobe_free_utask+0x62/0x80 mm_release+0x12/0xb0 do_exit+0x26b/0xaa0 __x64_sys_exit+0x1b/0x20 do_syscall_64+0x5a/0x80
It can be easily reproduced by running following commands in separate terminals:
# while :; do bpftrace -e 'uprobe:/bin/ls:_start { printf("hit\n"); }' -c ls; done # bpftrace -e 'profile:hz:100000 { @[ustack()] = count(); }'
Fixing this by making sure current->utask pointer is set to NULL before we start to release the utask object.
[1] https://github.com/grafana/pyroscope/issues/3673
Fixes: cfa7f3d2c526 ("perf,x86: avoid missing caller address in stack traces captured in uprobe") Reported-by: Max Makarov maxpain@linux.com (cherry picked from commit b583ef82b671c9a752fbe3e95bd4c1c51eab764d) Signed-off-by: Jiri Olsa jolsa@kernel.org Signed-off-by: Peter Zijlstra (Intel) peterz@infradead.org Acked-by: Oleg Nesterov oleg@redhat.com Acked-by: Andrii Nakryiko andrii@kernel.org Link: https://lore.kernel.org/r/20250109141440.2692173-1-jolsa@kernel.org [Christian Simon: Rebased for 6.6.y, due to mainline change https://lore.kernel.org/all/20240929144239.GA9475@redhat.com/] Signed-off-by: Christian Simon simon@swine.de --- kernel/events/uprobes.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 6dac0b579821..70fae75c48ca 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -1716,6 +1716,7 @@ void uprobe_free_utask(struct task_struct *t) if (!utask) return;
+ t->utask = NULL; if (utask->active_uprobe) put_uprobe(utask->active_uprobe);
@@ -1725,7 +1726,6 @@ void uprobe_free_utask(struct task_struct *t)
xol_free_insn_slot(t); kfree(utask); - t->utask = NULL; }
/*
[ Sasha's backport helper bot ]
Hi,
✅ All tests passed successfully. No issues detected. No action required from the submitter.
The upstream commit SHA1 provided is correct: b583ef82b671c9a752fbe3e95bd4c1c51eab764d
WARNING: Author mismatch between patch and upstream commit: Backport author: Christian Simonsimon@swine.de Commit author: Jiri Olsajolsa@kernel.org
Note: The patch differs from the upstream commit: --- 1: b583ef82b671c ! 1: c65c252796df6 uprobes: Fix race in uprobe_free_utask @@ ## Metadata ## -Author: Jiri Olsa jolsa@kernel.org +Author: Christian Simon simon@swine.de
## Commit message ## uprobes: Fix race in uprobe_free_utask
+ commit b583ef82b671c9a752fbe3e95bd4c1c51eab764d upstream. + + Christian Simon verified the regression exists in v6.6.80 as per method + below and backported the mainline fix to the older version of + uprobe_free_utask. After that change I can no longer reproduce + the race with this method within 1h, while before it would show the + panic under a minute. + Max Makarov reported kernel panic [1] in perf user callchain code.
The reason for that is the race between uprobe_free_utask and bpf @@ Commit message
Fixes: cfa7f3d2c526 ("perf,x86: avoid missing caller address in stack traces captured in uprobe") Reported-by: Max Makarov maxpain@linux.com + (cherry picked from commit b583ef82b671c9a752fbe3e95bd4c1c51eab764d) Signed-off-by: Jiri Olsa jolsa@kernel.org Signed-off-by: Peter Zijlstra (Intel) peterz@infradead.org Acked-by: Oleg Nesterov oleg@redhat.com Acked-by: Andrii Nakryiko andrii@kernel.org Link: https://lore.kernel.org/r/20250109141440.2692173-1-jolsa@kernel.org + [Christian Simon: Rebased for 6.6.y, due to mainline change https://lore.kernel.org/all/20240929144239.GA9475@redhat.com/] + Signed-off-by: Christian Simon simon@swine.de
## kernel/events/uprobes.c ## @@ kernel/events/uprobes.c: void uprobe_free_utask(struct task_struct *t) @@ kernel/events/uprobes.c: void uprobe_free_utask(struct task_struct *t) return;
+ t->utask = NULL; - WARN_ON_ONCE(utask->active_uprobe || utask->xol_vaddr); + if (utask->active_uprobe) + put_uprobe(utask->active_uprobe);
- timer_delete_sync(&utask->ri_timer); @@ kernel/events/uprobes.c: void uprobe_free_utask(struct task_struct *t) - ri = free_ret_instance(ri, true /* cleanup_hprobe */);
+ xol_free_insn_slot(t); kfree(utask); - t->utask = NULL; }
- #define RI_TIMER_PERIOD (HZ / 10) /* 100 ms */ + /* ---
Results of testing on various branches:
| Branch | Patch Apply | Build Test | |---------------------------|-------------|------------| | stable/linux-6.6.y | Success | Success |
commit b583ef82b671c9a752fbe3e95bd4c1c51eab764d upstream.
Christian Simon verified the regression exists in v6.1.129 as per method below and backported the mainline fix to the older version of uprobe_free_utask. After that change I can no longer reproduce the race with this method within 12 hours, while before it would show the panic in under a minute.
Max Makarov reported kernel panic [1] in perf user callchain code.
The reason for that is the race between uprobe_free_utask and bpf profiler code doing the perf user stack unwind and is triggered within uprobe_free_utask function: - after current->utask is freed and - before current->utask is set to NULL
general protection fault, probably for non-canonical address 0x9e759c37ee555c76: 0000 [#1] SMP PTI RIP: 0010:is_uprobe_at_func_entry+0x28/0x80 ... ? die_addr+0x36/0x90 ? exc_general_protection+0x217/0x420 ? asm_exc_general_protection+0x26/0x30 ? is_uprobe_at_func_entry+0x28/0x80 perf_callchain_user+0x20a/0x360 get_perf_callchain+0x147/0x1d0 bpf_get_stackid+0x60/0x90 bpf_prog_9aac297fb833e2f5_do_perf_event+0x434/0x53b ? __smp_call_single_queue+0xad/0x120 bpf_overflow_handler+0x75/0x110 ... asm_sysvec_apic_timer_interrupt+0x1a/0x20 RIP: 0010:__kmem_cache_free+0x1cb/0x350 ... ? uprobe_free_utask+0x62/0x80 ? acct_collect+0x4c/0x220 uprobe_free_utask+0x62/0x80 mm_release+0x12/0xb0 do_exit+0x26b/0xaa0 __x64_sys_exit+0x1b/0x20 do_syscall_64+0x5a/0x80
It can be easily reproduced by running following commands in separate terminals:
# while :; do bpftrace -e 'uprobe:/bin/ls:_start { printf("hit\n"); }' -c ls; done # bpftrace -e 'profile:hz:100000 { @[ustack()] = count(); }'
Fixing this by making sure current->utask pointer is set to NULL before we start to release the utask object.
[1] https://github.com/grafana/pyroscope/issues/3673
Fixes: cfa7f3d2c526 ("perf,x86: avoid missing caller address in stack traces captured in uprobe") Reported-by: Max Makarov maxpain@linux.com (cherry picked from commit b583ef82b671c9a752fbe3e95bd4c1c51eab764d) Signed-off-by: Jiri Olsa jolsa@kernel.org Signed-off-by: Peter Zijlstra (Intel) peterz@infradead.org Acked-by: Oleg Nesterov oleg@redhat.com Acked-by: Andrii Nakryiko andrii@kernel.org Link: https://lore.kernel.org/r/20250109141440.2692173-1-jolsa@kernel.org [Christian Simon: Rebased for 6.1.y, due to mainline change https://lore.kernel.org/all/20240929144239.GA9475@redhat.com/] Signed-off-by: Christian Simon simon@swine.de --- kernel/events/uprobes.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 9ee25351ceca..e9f9a669f80e 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -1719,6 +1719,7 @@ void uprobe_free_utask(struct task_struct *t) if (!utask) return;
+ t->utask = NULL; if (utask->active_uprobe) put_uprobe(utask->active_uprobe);
@@ -1728,7 +1729,6 @@ void uprobe_free_utask(struct task_struct *t)
xol_free_insn_slot(t); kfree(utask); - t->utask = NULL; }
/*
[ Sasha's backport helper bot ]
Hi,
Summary of potential issues: ℹ️ Patch is missing in 6.13.y (ignore if backport was sent) ⚠️ Commit missing in all newer stable branches
The upstream commit SHA1 provided is correct: b583ef82b671c9a752fbe3e95bd4c1c51eab764d
WARNING: Author mismatch between patch and upstream commit: Backport author: Christian Simonsimon@swine.de Commit author: Jiri Olsajolsa@kernel.org
Status in newer kernel trees: 6.13.y | Present (exact SHA1) 6.12.y | Not found 6.6.y | Not found
Note: The patch differs from the upstream commit: --- 1: b583ef82b671c ! 1: 9f50b10df7c63 uprobes: Fix race in uprobe_free_utask @@ ## Metadata ## -Author: Jiri Olsa jolsa@kernel.org +Author: Christian Simon simon@swine.de
## Commit message ## uprobes: Fix race in uprobe_free_utask
+ commit b583ef82b671c9a752fbe3e95bd4c1c51eab764d upstream. + + Christian Simon verified the regression exists in v6.1.129 as per method + below and backported the mainline fix to the older version of + uprobe_free_utask. After that change I can no longer reproduce + the race with this method within 12 hours, while before it would + show the panic in under a minute. + Max Makarov reported kernel panic [1] in perf user callchain code.
The reason for that is the race between uprobe_free_utask and bpf @@ Commit message
Fixes: cfa7f3d2c526 ("perf,x86: avoid missing caller address in stack traces captured in uprobe") Reported-by: Max Makarov maxpain@linux.com + (cherry picked from commit b583ef82b671c9a752fbe3e95bd4c1c51eab764d) Signed-off-by: Jiri Olsa jolsa@kernel.org Signed-off-by: Peter Zijlstra (Intel) peterz@infradead.org Acked-by: Oleg Nesterov oleg@redhat.com Acked-by: Andrii Nakryiko andrii@kernel.org Link: https://lore.kernel.org/r/20250109141440.2692173-1-jolsa@kernel.org + [Christian Simon: Rebased for 6.1.y, due to mainline change https://lore.kernel.org/all/20240929144239.GA9475@redhat.com/] + Signed-off-by: Christian Simon simon@swine.de
## kernel/events/uprobes.c ## @@ kernel/events/uprobes.c: void uprobe_free_utask(struct task_struct *t) @@ kernel/events/uprobes.c: void uprobe_free_utask(struct task_struct *t) return;
+ t->utask = NULL; - WARN_ON_ONCE(utask->active_uprobe || utask->xol_vaddr); + if (utask->active_uprobe) + put_uprobe(utask->active_uprobe);
- timer_delete_sync(&utask->ri_timer); @@ kernel/events/uprobes.c: void uprobe_free_utask(struct task_struct *t) - ri = free_ret_instance(ri, true /* cleanup_hprobe */);
+ xol_free_insn_slot(t); kfree(utask); - t->utask = NULL; }
- #define RI_TIMER_PERIOD (HZ / 10) /* 100 ms */ + /* ---
Results of testing on various branches:
| Branch | Patch Apply | Build Test | |---------------------------|-------------|------------| | stable/linux-6.1.y | Success | Success |
commit b583ef82b671c9a752fbe3e95bd4c1c51eab764d upstream.
Christian Simon verified the regression exists in v6.12.17 as per method below and backported the mainline fix to the older version of uprobe_free_utask. After that change I can no longer reproduce the race with this method within 1h, while before it would show the panic under a minute.
Max Makarov reported kernel panic [1] in perf user callchain code.
The reason for that is the race between uprobe_free_utask and bpf profiler code doing the perf user stack unwind and is triggered within uprobe_free_utask function: - after current->utask is freed and - before current->utask is set to NULL
general protection fault, probably for non-canonical address 0x9e759c37ee555c76: 0000 [#1] SMP PTI RIP: 0010:is_uprobe_at_func_entry+0x28/0x80 ... ? die_addr+0x36/0x90 ? exc_general_protection+0x217/0x420 ? asm_exc_general_protection+0x26/0x30 ? is_uprobe_at_func_entry+0x28/0x80 perf_callchain_user+0x20a/0x360 get_perf_callchain+0x147/0x1d0 bpf_get_stackid+0x60/0x90 bpf_prog_9aac297fb833e2f5_do_perf_event+0x434/0x53b ? __smp_call_single_queue+0xad/0x120 bpf_overflow_handler+0x75/0x110 ... asm_sysvec_apic_timer_interrupt+0x1a/0x20 RIP: 0010:__kmem_cache_free+0x1cb/0x350 ... ? uprobe_free_utask+0x62/0x80 ? acct_collect+0x4c/0x220 uprobe_free_utask+0x62/0x80 mm_release+0x12/0xb0 do_exit+0x26b/0xaa0 __x64_sys_exit+0x1b/0x20 do_syscall_64+0x5a/0x80
It can be easily reproduced by running following commands in separate terminals:
# while :; do bpftrace -e 'uprobe:/bin/ls:_start { printf("hit\n"); }' -c ls; done # bpftrace -e 'profile:hz:100000 { @[ustack()] = count(); }'
Fixing this by making sure current->utask pointer is set to NULL before we start to release the utask object.
[1] https://github.com/grafana/pyroscope/issues/3673
Fixes: cfa7f3d2c526 ("perf,x86: avoid missing caller address in stack traces captured in uprobe") Reported-by: Max Makarov maxpain@linux.com (cherry picked from commit b583ef82b671c9a752fbe3e95bd4c1c51eab764d) Signed-off-by: Jiri Olsa jolsa@kernel.org Signed-off-by: Peter Zijlstra (Intel) peterz@infradead.org Acked-by: Oleg Nesterov oleg@redhat.com Acked-by: Andrii Nakryiko andrii@kernel.org Link: https://lore.kernel.org/r/20250109141440.2692173-1-jolsa@kernel.org [Christian Simon: Rebased for 6.12.y, due to mainline change https://lore.kernel.org/all/20240929144239.GA9475@redhat.com/] Signed-off-by: Christian Simon simon@swine.de --- kernel/events/uprobes.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 4b52cb2ae6d6..fcb777ba17df 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -1770,6 +1770,7 @@ void uprobe_free_utask(struct task_struct *t) if (!utask) return;
+ t->utask = NULL; if (utask->active_uprobe) put_uprobe(utask->active_uprobe);
@@ -1779,7 +1780,6 @@ void uprobe_free_utask(struct task_struct *t)
xol_free_insn_slot(t); kfree(utask); - t->utask = NULL; }
/*
[ Sasha's backport helper bot ]
Hi,
Summary of potential issues: ℹ️ Patch is missing in 6.13.y (ignore if backport was sent) ⚠️ Commit missing in all newer stable branches
The upstream commit SHA1 provided is correct: b583ef82b671c9a752fbe3e95bd4c1c51eab764d
WARNING: Author mismatch between patch and upstream commit: Backport author: Christian Simonsimon@swine.de Commit author: Jiri Olsajolsa@kernel.org
Status in newer kernel trees: 6.13.y | Present (exact SHA1) 6.6.y | Not found
Note: The patch differs from the upstream commit: --- 1: b583ef82b671c ! 1: b3a3f2344d131 uprobes: Fix race in uprobe_free_utask @@ ## Metadata ## -Author: Jiri Olsa jolsa@kernel.org +Author: Christian Simon simon@swine.de
## Commit message ## uprobes: Fix race in uprobe_free_utask
+ commit b583ef82b671c9a752fbe3e95bd4c1c51eab764d upstream. + + Christian Simon verified the regression exists in v6.12.17 as per method + below and backported the mainline fix to the older version of + uprobe_free_utask. After that change I can no longer reproduce + the race with this method within 1h, while before it would show the + panic under a minute. + Max Makarov reported kernel panic [1] in perf user callchain code.
The reason for that is the race between uprobe_free_utask and bpf @@ Commit message
Fixes: cfa7f3d2c526 ("perf,x86: avoid missing caller address in stack traces captured in uprobe") Reported-by: Max Makarov maxpain@linux.com + (cherry picked from commit b583ef82b671c9a752fbe3e95bd4c1c51eab764d) Signed-off-by: Jiri Olsa jolsa@kernel.org Signed-off-by: Peter Zijlstra (Intel) peterz@infradead.org Acked-by: Oleg Nesterov oleg@redhat.com Acked-by: Andrii Nakryiko andrii@kernel.org Link: https://lore.kernel.org/r/20250109141440.2692173-1-jolsa@kernel.org + [Christian Simon: Rebased for 6.12.y, due to mainline change https://lore.kernel.org/all/20240929144239.GA9475@redhat.com/] + Signed-off-by: Christian Simon simon@swine.de
## kernel/events/uprobes.c ## @@ kernel/events/uprobes.c: void uprobe_free_utask(struct task_struct *t) @@ kernel/events/uprobes.c: void uprobe_free_utask(struct task_struct *t) return;
+ t->utask = NULL; - WARN_ON_ONCE(utask->active_uprobe || utask->xol_vaddr); + if (utask->active_uprobe) + put_uprobe(utask->active_uprobe);
- timer_delete_sync(&utask->ri_timer); @@ kernel/events/uprobes.c: void uprobe_free_utask(struct task_struct *t) - ri = free_ret_instance(ri, true /* cleanup_hprobe */);
+ xol_free_insn_slot(t); kfree(utask); - t->utask = NULL; }
- #define RI_TIMER_PERIOD (HZ / 10) /* 100 ms */ + /* ---
Results of testing on various branches:
| Branch | Patch Apply | Build Test | |---------------------------|-------------|------------| | stable/linux-6.12.y | Success | Success |
linux-stable-mirror@lists.linaro.org
-
Christian Simon -
Greg KH -
Sasha Levin