Commit 8d470a45d1a6 ("panic: add option to dump all CPUs backtraces in panic_print") introduced a setting for the "panic_print" kernel parameter to allow users to request a NMI backtrace on panic. Problem is that the panic_print handling happens after the secondary CPUs are already disabled, hence this option ended-up being kind of a no-op - kernel skips the NMI trace in idling CPUs, which is the case of offline CPUs.
Fix it by checking the NMI backtrace bit in the panic_print prior to the CPU disabling function.
Fixes: 8d470a45d1a6 ("panic: add option to dump all CPUs backtraces in panic_print") Cc: stable@vger.kernel.org Signed-off-by: Guilherme G. Piccoli gpiccoli@igalia.com
---
V4: - Sent as standalone patch, rebased against v6.2-rc7.
V2 / V3: - New patch, there was no V1 of this one. Link for V3: https://lore.kernel.org/lkml/20220819221731.480795-12-gpiccoli@igalia.com/
Hi folks, thanks in advance for reviews/comments.
Notice that while at it, I got rid of the "crash_kexec_post_notifiers" local copy in panic(). This was introduced by commit b26e27ddfd2a ("kexec: use core_param for crash_kexec_post_notifiers boot option"), but it is not clear from comments or commit message why this local copy is required.
My understanding is that it's a mechanism to prevent some concurrency, in case some other CPU modify this variable while panic() is running. I find it very unlikely, hence I removed it - but if people consider this copy needed, I can respin this patch and keep it, even providing a comment about that, in order to be explict about its need.
Let me know your thoughts! Cheers,
Guilherme
kernel/panic.c | 47 +++++++++++++++++++++++++++-------------------- 1 file changed, 27 insertions(+), 20 deletions(-)
diff --git a/kernel/panic.c b/kernel/panic.c index 463c9295bc28..f45ee88be8a2 100644 --- a/kernel/panic.c +++ b/kernel/panic.c @@ -211,9 +211,6 @@ static void panic_print_sys_info(bool console_flush) return; }
- if (panic_print & PANIC_PRINT_ALL_CPU_BT) - trigger_all_cpu_backtrace(); - if (panic_print & PANIC_PRINT_TASK_INFO) show_state();
@@ -243,6 +240,30 @@ void check_panic_on_warn(const char *origin) origin, limit); }
+/* + * Helper that triggers the NMI backtrace (if set in panic_print) + * and then performs the secondary CPUs shutdown - we cannot have + * the NMI backtrace after the CPUs are off! + */ +static void panic_other_cpus_shutdown(void) +{ + if (panic_print & PANIC_PRINT_ALL_CPU_BT) + trigger_all_cpu_backtrace(); + + /* + * Note that smp_send_stop() is the usual SMP shutdown function, + * which unfortunately may not be hardened to work in a panic + * situation. If we want to do crash dump after notifier calls + * and kmsg_dump, we will need architecture dependent extra + * bits in addition to stopping other CPUs, hence we rely on + * crash_smp_send_stop() for that. + */ + if (!crash_kexec_post_notifiers) + smp_send_stop(); + else + crash_smp_send_stop(); +} + /** * panic - halt the system * @fmt: The text string to print @@ -258,7 +279,6 @@ void panic(const char *fmt, ...) long i, i_next = 0, len; int state = 0; int old_cpu, this_cpu; - bool _crash_kexec_post_notifiers = crash_kexec_post_notifiers;
if (panic_on_warn) { /* @@ -333,23 +353,10 @@ void panic(const char *fmt, ...) * * Bypass the panic_cpu check and call __crash_kexec directly. */ - if (!_crash_kexec_post_notifiers) { + if (!crash_kexec_post_notifiers) __crash_kexec(NULL);
- /* - * Note smp_send_stop is the usual smp shutdown function, which - * unfortunately means it may not be hardened to work in a - * panic situation. - */ - smp_send_stop(); - } else { - /* - * If we want to do crash dump after notifier calls and - * kmsg_dump, we will need architecture dependent extra - * works in addition to stopping other CPUs. - */ - crash_smp_send_stop(); - } + panic_other_cpus_shutdown();
/* * Run any panic handlers, including those that might need to @@ -370,7 +377,7 @@ void panic(const char *fmt, ...) * * Bypass the panic_cpu check and call __crash_kexec directly. */ - if (_crash_kexec_post_notifiers) + if (crash_kexec_post_notifiers) __crash_kexec(NULL);
console_unblank();
On Fri 2023-02-10 17:35:10, Guilherme G. Piccoli wrote:
Commit 8d470a45d1a6 ("panic: add option to dump all CPUs backtraces in panic_print") introduced a setting for the "panic_print" kernel parameter to allow users to request a NMI backtrace on panic. Problem is that the panic_print handling happens after the secondary CPUs are already disabled, hence this option ended-up being kind of a no-op - kernel skips the NMI trace in idling CPUs, which is the case of offline CPUs.
Great catch!
Hi folks, thanks in advance for reviews/comments.
Notice that while at it, I got rid of the "crash_kexec_post_notifiers" local copy in panic(). This was introduced by commit b26e27ddfd2a ("kexec: use core_param for crash_kexec_post_notifiers boot option"), but it is not clear from comments or commit message why this local copy is required.
My understanding is that it's a mechanism to prevent some concurrency, in case some other CPU modify this variable while panic() is running. I find it very unlikely, hence I removed it - but if people consider this copy needed, I can respin this patch and keep it, even providing a comment about that, in order to be explict about its need.
Yes, I think that it makes the behavior consistent even when the global variable manipulated in parallel.
I would personally prefer to keep the local copy. Better safe than sorry.
Let me know your thoughts! Cheers,
Guilherme
kernel/panic.c | 47 +++++++++++++++++++++++++++-------------------- 1 file changed, 27 insertions(+), 20 deletions(-)
diff --git a/kernel/panic.c b/kernel/panic.c index 463c9295bc28..f45ee88be8a2 100644 --- a/kernel/panic.c +++ b/kernel/panic.c @@ -211,9 +211,6 @@ static void panic_print_sys_info(bool console_flush) return; }
- if (panic_print & PANIC_PRINT_ALL_CPU_BT)
trigger_all_cpu_backtrace();
Sigh, this is yet another PANIC_PRINT_ action that need special timing. We should handle both the same way.
What about the following? The parameter @mask says what actions are allowed at the given time.
--- a/kernel/panic.c +++ b/kernel/panic.c @@ -72,6 +72,9 @@ EXPORT_SYMBOL_GPL(panic_timeout); #define PANIC_PRINT_FTRACE_INFO 0x00000010 #define PANIC_PRINT_ALL_PRINTK_MSG 0x00000020 #define PANIC_PRINT_ALL_CPU_BT 0x00000040 +/* Filter out actions that need special timing. */ +#define PANIC_PRINT_COMMON_INFO_MASK ~(PANIC_PRINT_ALL_PRINTK_MSG | \ + PANIC_PRINT_ALL_CPU_BT) unsigned long panic_print;
ATOMIC_NOTIFIER_HEAD(panic_notifier_list); @@ -203,30 +206,29 @@ void nmi_panic(struct pt_regs *regs, const char *msg) } EXPORT_SYMBOL(nmi_panic);
-static void panic_print_sys_info(bool console_flush) +static void panic_print_sys_info(unsigned long mask) { - if (console_flush) { - if (panic_print & PANIC_PRINT_ALL_PRINTK_MSG) - console_flush_on_panic(CONSOLE_REPLAY_ALL); - return; - } + unsigned long panic_print_now = panic_print & mask; + + if (panic_print_now & PANIC_PRINT_ALL_PRINTK_MSG) + console_flush_on_panic(CONSOLE_REPLAY_ALL);
- if (panic_print & PANIC_PRINT_ALL_CPU_BT) + if (panic_print_now & PANIC_PRINT_ALL_CPU_BT) trigger_all_cpu_backtrace();
- if (panic_print & PANIC_PRINT_TASK_INFO) + if (panic_print_now & PANIC_PRINT_TASK_INFO) show_state();
- if (panic_print & PANIC_PRINT_MEM_INFO) + if (panic_print_now & PANIC_PRINT_MEM_INFO) show_mem(0, NULL);
- if (panic_print & PANIC_PRINT_TIMER_INFO) + if (panic_print_now & PANIC_PRINT_TIMER_INFO) sysrq_timer_list_show();
- if (panic_print & PANIC_PRINT_LOCK_INFO) + if (panic_print_now & PANIC_PRINT_LOCK_INFO) debug_show_all_locks();
- if (panic_print & PANIC_PRINT_FTRACE_INFO) + if (panic_print_now & PANIC_PRINT_FTRACE_INFO) ftrace_dump(DUMP_ALL); }
@@ -333,9 +335,12 @@ void panic(const char *fmt, ...) * * Bypass the panic_cpu check and call __crash_kexec directly. */ - if (!_crash_kexec_post_notifiers) { + if (!_crash_kexec_post_notifiers) __crash_kexec(NULL);
+ panic_print_sys_info(PANIC_PRINT_ALL_CPU_BT); + + if (!_crash_kexec_post_notifiers) { /* * Note smp_send_stop is the usual smp shutdown function, which * unfortunately means it may not be hardened to work in a @@ -357,7 +362,7 @@ void panic(const char *fmt, ...) */ atomic_notifier_call_chain(&panic_notifier_list, 0, buf);
- panic_print_sys_info(false); + panic_print_sys_info(PANIC_PRINT_COMMON_INFO_MASK);
kmsg_dump(KMSG_DUMP_PANIC);
@@ -386,7 +391,7 @@ void panic(const char *fmt, ...) debug_locks_off(); console_flush_on_panic(CONSOLE_FLUSH_PENDING);
- panic_print_sys_info(true); + panic_print_sys_info(PANIC_PRINT_ALL_PRINTK_MSG);
if (!panic_blink) panic_blink = no_blink;
Best Regards, Petr
On 14/02/2023 11:46, Petr Mladek wrote:
[...]
My understanding is that it's a mechanism to prevent some concurrency, in case some other CPU modify this variable while panic() is running. I find it very unlikely, hence I removed it - but if people consider this copy needed, I can respin this patch and keep it, even providing a comment about that, in order to be explict about its need.
Yes, I think that it makes the behavior consistent even when the global variable manipulated in parallel.
I would personally prefer to keep the local copy. Better safe than sorry.
Hi Petr, thanks for your review! OK, we could keep this local copy, makes sense...even adding a comment, to make its purpose really clear.
[...] @@ -211,9 +211,6 @@ static void panic_print_sys_info(bool console_flush) return; }
- if (panic_print & PANIC_PRINT_ALL_CPU_BT)
trigger_all_cpu_backtrace();Sigh, this is yet another PANIC_PRINT_ action that need special timing. We should handle both the same way.
What about the following? The parameter @mask says what actions are allowed at the given time. < ..code..>
I think your approach is interesting, it's very "organized".
But I think it's a bit conflicting with that purpose we had on notifiers refactor, to deprecate "bogus" usages of panic_print, as in https://lore.kernel.org/lkml/20220427224924.592546-26-gpiccoli@igalia.com/ .
So, the idea of my approach is to allow:
(a) Easy removal of panic_print_sys_info() of panic(), once we move it to a panic notifier;
(b) Better separate and identify the "bogus" cases. The CPU backtrace one is less a bogus case in my opinion, more a "complicated" one, since it's related with the CPUs stop routines. But the console flush, as we discussed, it's clearly something that calls for a new parameter (and such param was added in the refactor patch).
In the end, I think your approach is interesting but it's kinda like we're adding the fix to later, on refactor, entirely remove/rework it. With my approach we wouldn't be calling panic_print_sys_info() again (3rd time!) on panic(), and also would be more natural to move it later to a new panic notifier.
What you / others think? If your approach is in the end preferred, it's fine by me - I'd just ask you to submit as a full patch so we can get it merged as a fix in 6.3, if possible (and backport it to the 6.1/6.2 stable). Now, if my approach is fine, I can resubmit as a V5 keeping the local variable - lemme know.
Cheers,
Guilherme
On Fri, 10 Feb 2023 17:35:10 -0300 "Guilherme G. Piccoli" gpiccoli@igalia.com wrote:
Commit 8d470a45d1a6 ("panic: add option to dump all CPUs backtraces in panic_print") introduced a setting for the "panic_print" kernel parameter to allow users to request a NMI backtrace on panic. Problem is that the panic_print handling happens after the secondary CPUs are already disabled, hence this option ended-up being kind of a no-op - kernel skips the NMI trace in idling CPUs, which is the case of offline CPUs.
Fix it by checking the NMI backtrace bit in the panic_print prior to the CPU disabling function.
...
Notice that while at it, I got rid of the "crash_kexec_post_notifiers" local copy in panic(). This was introduced by commit b26e27ddfd2a ("kexec: use core_param for crash_kexec_post_notifiers boot option"), but it is not clear from comments or commit message why this local copy is required.
My understanding is that it's a mechanism to prevent some concurrency, in case some other CPU modify this variable while panic() is running. I find it very unlikely, hence I removed it - but if people consider this copy needed, I can respin this patch and keep it, even providing a comment about that, in order to be explict about its need.
Only two sites change crash_kexec_post_notifiers, in arch/powerpc/kernel/fadump.c and drivers/hv/hv_common.c. Yes, it's very unlikely that this will be altered while panic() is running and the consequences will be slight anyway.
But formally, we shouldn't do this, especially in a -stable backportable patch. So please, let's have the minimal bugfix for now and we can look at removing that local at a later time?
On 26/02/2023 02:44, Andrew Morton wrote:
On Fri, 10 Feb 2023 17:35:10 -0300 "Guilherme G. Piccoli" gpiccoli@igalia.com wrote: [...]
Notice that while at it, I got rid of the "crash_kexec_post_notifiers" local copy in panic(). This was introduced by commit b26e27ddfd2a ("kexec: use core_param for crash_kexec_post_notifiers boot option"), but it is not clear from comments or commit message why this local copy is required.
My understanding is that it's a mechanism to prevent some concurrency, in case some other CPU modify this variable while panic() is running. I find it very unlikely, hence I removed it - but if people consider this copy needed, I can respin this patch and keep it, even providing a comment about that, in order to be explict about its need.
Only two sites change crash_kexec_post_notifiers, in arch/powerpc/kernel/fadump.c and drivers/hv/hv_common.c. Yes, it's very unlikely that this will be altered while panic() is running and the consequences will be slight anyway.
But formally, we shouldn't do this, especially in a -stable backportable patch. So please, let's have the minimal bugfix for now and we can look at removing that local at a later time?
Thanks Andrew, I agree with you! I just sent a V5 with the bugfix alone, not changing this local/global variable behavior.
Cheers,
Guilherme
linux-stable-mirror@lists.linaro.org
-
Andrew Morton -
Guilherme G. Piccoli -
Petr Mladek