From: Li RongQing lirongqing@baidu.com
Currently, when 'hung_task_panic' is enabled, the kernel panics immediately upon detecting the first hung task. However, some hung tasks are transient and allow system recovery, while persistent hangs should trigger a panic when accumulating beyond a threshold.
Extend the 'hung_task_panic' sysctl to accept a threshold value specifying the number of hung tasks that must be detected before triggering a kernel panic. This provides finer control for environments where transient hangs may occur but persistent hangs should be fatal.
The sysctl now accepts: - 0: don't panic (maintains original behavior) - 1: panic on first hung task (maintains original behavior) - N > 1: panic after N hung tasks are detected in a single scan
This maintains backward compatibility while providing flexibility for different hang scenarios.
Signed-off-by: Li RongQing lirongqing@baidu.com Cc: Andrew Jeffery andrew@codeconstruct.com.au Cc: Anshuman Khandual anshuman.khandual@arm.com Cc: Arnd Bergmann arnd@arndb.de Cc: David Hildenbrand david@redhat.com Cc: Florian Wesphal fw@strlen.de Cc: Jakub Kacinski kuba@kernel.org Cc: Jason A. Donenfeld jason@zx2c4.com Cc: Joel Granados joel.granados@kernel.org Cc: Joel Stanley joel@jms.id.au Cc: Jonathan Corbet corbet@lwn.net Cc: Kees Cook kees@kernel.org Cc: Lance Yang lance.yang@linux.dev Cc: Liam Howlett liam.howlett@oracle.com Cc: Lorenzo Stoakes lorenzo.stoakes@oracle.com Cc: "Masami Hiramatsu (Google)" mhiramat@kernel.org Cc: "Paul E . McKenney" paulmck@kernel.org Cc: Pawan Gupta pawan.kumar.gupta@linux.intel.com Cc: Petr Mladek pmladek@suse.com Cc: Phil Auld pauld@redhat.com Cc: Randy Dunlap rdunlap@infradead.org Cc: Russell King linux@armlinux.org.uk Cc: Shuah Khan shuah@kernel.org Cc: Simon Horman horms@kernel.org Cc: Stanislav Fomichev sdf@fomichev.me Cc: Steven Rostedt rostedt@goodmis.org --- diff with v3: comments modification, suggested by Lance, Masami, Randy and Petr diff with v2: do not add a new sysctl, extend hung_task_panic, suggested by Kees Cook
Documentation/admin-guide/kernel-parameters.txt | 20 +++++++++++++------- Documentation/admin-guide/sysctl/kernel.rst | 9 +++++---- arch/arm/configs/aspeed_g5_defconfig | 2 +- kernel/configs/debug.config | 2 +- kernel/hung_task.c | 15 ++++++++++----- lib/Kconfig.debug | 9 +++++---- tools/testing/selftests/wireguard/qemu/kernel.config | 2 +- 7 files changed, 36 insertions(+), 23 deletions(-)
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index a51ab46..492f0bc 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -1992,14 +1992,20 @@ the added memory block itself do not be affected.
hung_task_panic= - [KNL] Should the hung task detector generate panics. - Format: 0 | 1 + [KNL] Number of hung tasks to trigger kernel panic. + Format: <int> + + When set to a non-zero value, a kernel panic will be triggered if + the number of detected hung tasks reaches this value. + + 0: don't panic + 1: panic immediately on first hung task + N: panic after N hung tasks are detected in a single scan
- A value of 1 instructs the kernel to panic when a - hung task is detected. The default value is controlled - by the CONFIG_BOOTPARAM_HUNG_TASK_PANIC build-time - option. The value selected by this boot parameter can - be changed later by the kernel.hung_task_panic sysctl. + The default value is controlled by the + CONFIG_BOOTPARAM_HUNG_TASK_PANIC build-time option. The value + selected by this boot parameter can be changed later by the + kernel.hung_task_panic sysctl.
hvc_iucv= [S390] Number of z/VM IUCV hypervisor console (HVC) terminal devices. Valid values: 0..8 diff --git a/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/admin-guide/sysctl/kernel.rst index f3ee807..0065a55 100644 --- a/Documentation/admin-guide/sysctl/kernel.rst +++ b/Documentation/admin-guide/sysctl/kernel.rst @@ -397,13 +397,14 @@ a hung task is detected. hung_task_panic ===============
-Controls the kernel's behavior when a hung task is detected. +When set to a non-zero value, a kernel panic will be triggered if the +number of hung tasks found during a single scan reaches this value. This file shows up if ``CONFIG_DETECT_HUNG_TASK`` is enabled.
-= ================================================= += ======================================================= 0 Continue operation. This is the default behavior. -1 Panic immediately. -= ================================================= +N Panic when N hung tasks are found during a single scan. += =======================================================
hung_task_check_count diff --git a/arch/arm/configs/aspeed_g5_defconfig b/arch/arm/configs/aspeed_g5_defconfig index 61cee1e..c3b0d5f 100644 --- a/arch/arm/configs/aspeed_g5_defconfig +++ b/arch/arm/configs/aspeed_g5_defconfig @@ -308,7 +308,7 @@ CONFIG_PANIC_ON_OOPS=y CONFIG_PANIC_TIMEOUT=-1 CONFIG_SOFTLOCKUP_DETECTOR=y CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC=y -CONFIG_BOOTPARAM_HUNG_TASK_PANIC=y +CONFIG_BOOTPARAM_HUNG_TASK_PANIC=1 CONFIG_WQ_WATCHDOG=y # CONFIG_SCHED_DEBUG is not set CONFIG_FUNCTION_TRACER=y diff --git a/kernel/configs/debug.config b/kernel/configs/debug.config index e81327d..9f6ab7d 100644 --- a/kernel/configs/debug.config +++ b/kernel/configs/debug.config @@ -83,7 +83,7 @@ CONFIG_SLUB_DEBUG_ON=y # # Debug Oops, Lockups and Hangs # -# CONFIG_BOOTPARAM_HUNG_TASK_PANIC is not set +CONFIG_BOOTPARAM_HUNG_TASK_PANIC=0 # CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC is not set CONFIG_DEBUG_ATOMIC_SLEEP=y CONFIG_DETECT_HUNG_TASK=y diff --git a/kernel/hung_task.c b/kernel/hung_task.c index b2c1f14..84b4b04 100644 --- a/kernel/hung_task.c +++ b/kernel/hung_task.c @@ -81,7 +81,7 @@ static unsigned int __read_mostly sysctl_hung_task_all_cpu_backtrace; * hung task is detected: */ static unsigned int __read_mostly sysctl_hung_task_panic = - IS_ENABLED(CONFIG_BOOTPARAM_HUNG_TASK_PANIC); + CONFIG_BOOTPARAM_HUNG_TASK_PANIC;
static int hung_task_panic(struct notifier_block *this, unsigned long event, void *ptr) @@ -218,8 +218,11 @@ static inline void debug_show_blocker(struct task_struct *task, unsigned long ti } #endif
-static void check_hung_task(struct task_struct *t, unsigned long timeout) +static void check_hung_task(struct task_struct *t, unsigned long timeout, + unsigned long prev_detect_count) { + unsigned long total_hung_task; + if (!task_is_hung(t, timeout)) return;
@@ -229,9 +232,10 @@ static void check_hung_task(struct task_struct *t, unsigned long timeout) */ sysctl_hung_task_detect_count++;
+ total_hung_task = sysctl_hung_task_detect_count - prev_detect_count; trace_sched_process_hang(t);
- if (sysctl_hung_task_panic) { + if (sysctl_hung_task_panic && total_hung_task >= sysctl_hung_task_panic) { console_verbose(); hung_task_show_lock = true; hung_task_call_panic = true; @@ -300,6 +304,7 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout) int max_count = sysctl_hung_task_check_count; unsigned long last_break = jiffies; struct task_struct *g, *t; + unsigned long prev_detect_count = sysctl_hung_task_detect_count;
/* * If the system crashed already then all bets are off, @@ -320,7 +325,7 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout) last_break = jiffies; }
- check_hung_task(t, timeout); + check_hung_task(t, timeout, prev_detect_count); } unlock: rcu_read_unlock(); @@ -389,7 +394,7 @@ static const struct ctl_table hung_task_sysctls[] = { .mode = 0644, .proc_handler = proc_dointvec_minmax, .extra1 = SYSCTL_ZERO, - .extra2 = SYSCTL_ONE, + .extra2 = SYSCTL_INT_MAX, }, { .procname = "hung_task_check_count", diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 3034e294..3976c90 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -1258,12 +1258,13 @@ config DEFAULT_HUNG_TASK_TIMEOUT Keeping the default should be fine in most cases.
config BOOTPARAM_HUNG_TASK_PANIC - bool "Panic (Reboot) On Hung Tasks" + int "Number of hung tasks to trigger kernel panic" depends on DETECT_HUNG_TASK + default 0 help - Say Y here to enable the kernel to panic on "hung tasks", - which are bugs that cause the kernel to leave a task stuck - in uninterruptible "D" state. + When set to a non-zero value, a kernel panic will be triggered + if the number of hung tasks found during a single scan reaches + this value.
The panic can be used in combination with panic_timeout, to cause the system to reboot automatically after a diff --git a/tools/testing/selftests/wireguard/qemu/kernel.config b/tools/testing/selftests/wireguard/qemu/kernel.config index 936b18b..0504c11 100644 --- a/tools/testing/selftests/wireguard/qemu/kernel.config +++ b/tools/testing/selftests/wireguard/qemu/kernel.config @@ -81,7 +81,7 @@ CONFIG_WQ_WATCHDOG=y CONFIG_DETECT_HUNG_TASK=y CONFIG_BOOTPARAM_HARDLOCKUP_PANIC=y CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC=y -CONFIG_BOOTPARAM_HUNG_TASK_PANIC=y +CONFIG_BOOTPARAM_HUNG_TASK_PANIC=1 CONFIG_PANIC_TIMEOUT=-1 CONFIG_STACKTRACE=y CONFIG_EARLY_PRINTK=y
LGTM. It works as expected, thanks!
On 2025/10/15 14:36, lirongqing wrote:
From: Li RongQing lirongqing@baidu.com
For the commit message, I'd suggest the following for better clarity:
``` The hung_task_panic sysctl is currently a blunt instrument: it's all or nothing.
Panicking on a single hung task can be an overreaction to a transient glitch. A more reliable indicator of a systemic problem is when multiple tasks hang simultaneously.
Extend hung_task_panic to accept an integer threshold, allowing the kernel to panic only when N hung tasks are detected in a single scan. This provides finer control to distinguish between isolated incidents and system-wide failures.
The accepted values are: - 0: Don't panic (unchanged) - 1: Panic on the first hung task (unchanged) - N > 1: Panic after N hung tasks are detected in a single scan
The original behavior is preserved for values 0 and 1, maintaining full backward compatibility. ```
If you agree, likely no need to resend - Andrew could pick it up directly when applying :)
Currently, when 'hung_task_panic' is enabled, the kernel panics immediately upon detecting the first hung task. However, some hung tasks are transient and allow system recovery, while persistent hangs should trigger a panic when accumulating beyond a threshold.
Extend the 'hung_task_panic' sysctl to accept a threshold value specifying the number of hung tasks that must be detected before triggering a kernel panic. This provides finer control for environments where transient hangs may occur but persistent hangs should be fatal.
The sysctl now accepts:
- 0: don't panic (maintains original behavior)
- 1: panic on first hung task (maintains original behavior)
- N > 1: panic after N hung tasks are detected in a single scan
This maintains backward compatibility while providing flexibility for different hang scenarios.
Signed-off-by: Li RongQing lirongqing@baidu.com Cc: Andrew Jeffery andrew@codeconstruct.com.au Cc: Anshuman Khandual anshuman.khandual@arm.com Cc: Arnd Bergmann arnd@arndb.de Cc: David Hildenbrand david@redhat.com Cc: Florian Wesphal fw@strlen.de Cc: Jakub Kacinski kuba@kernel.org Cc: Jason A. Donenfeld jason@zx2c4.com Cc: Joel Granados joel.granados@kernel.org Cc: Joel Stanley joel@jms.id.au Cc: Jonathan Corbet corbet@lwn.net Cc: Kees Cook kees@kernel.org Cc: Lance Yang lance.yang@linux.dev Cc: Liam Howlett liam.howlett@oracle.com Cc: Lorenzo Stoakes lorenzo.stoakes@oracle.com Cc: "Masami Hiramatsu (Google)" mhiramat@kernel.org Cc: "Paul E . McKenney" paulmck@kernel.org Cc: Pawan Gupta pawan.kumar.gupta@linux.intel.com Cc: Petr Mladek pmladek@suse.com Cc: Phil Auld pauld@redhat.com Cc: Randy Dunlap rdunlap@infradead.org Cc: Russell King linux@armlinux.org.uk Cc: Shuah Khan shuah@kernel.org Cc: Simon Horman horms@kernel.org Cc: Stanislav Fomichev sdf@fomichev.me Cc: Steven Rostedt rostedt@goodmis.org
So:
Reviewed-by: Lance Yang lance.yang@linux.dev Tested-by: Lance Yang lance.yang@linux.dev
Cheers, Lance
LGTM. It works as expected, thanks!
On 2025/10/15 14:36, lirongqing wrote:
From: Li RongQing lirongqing@baidu.com
For the commit message, I'd suggest the following for better clarity:
The hung_task_panic sysctl is currently a blunt instrument: it's all or nothing. Panicking on a single hung task can be an overreaction to a transient glitch. A more reliable indicator of a systemic problem is when multiple tasks hang simultaneously. Extend hung_task_panic to accept an integer threshold, allowing the kernel to panic only when N hung tasks are detected in a single scan. This provides finer control to distinguish between isolated incidents and system-wide failures. The accepted values are: - 0: Don't panic (unchanged) - 1: Panic on the first hung task (unchanged) - N > 1: Panic after N hung tasks are detected in a single scan The original behavior is preserved for values 0 and 1, maintaining full backward compatibility.If you agree, likely no need to resend - Andrew could pick it up directly when applying :)
This is better;
Andrew, could you pick it up directly
Thanks
-Li
Currently, when 'hung_task_panic' is enabled, the kernel panics immediately upon detecting the first hung task. However, some hung tasks are transient and allow system recovery, while persistent hangs should trigger a panic when accumulating beyond a threshold.
Extend the 'hung_task_panic' sysctl to accept a threshold value specifying the number of hung tasks that must be detected before triggering a kernel panic. This provides finer control for environments where transient hangs may occur but persistent hangs
should be fatal.
The sysctl now accepts:
- 0: don't panic (maintains original behavior)
- 1: panic on first hung task (maintains original behavior)
- N > 1: panic after N hung tasks are detected in a single scan
This maintains backward compatibility while providing flexibility for different hang scenarios.
Signed-off-by: Li RongQing lirongqing@baidu.com Cc: Andrew Jeffery andrew@codeconstruct.com.au Cc: Anshuman Khandual anshuman.khandual@arm.com Cc: Arnd Bergmann arnd@arndb.de Cc: David Hildenbrand david@redhat.com Cc: Florian Wesphal fw@strlen.de Cc: Jakub Kacinski kuba@kernel.org Cc: Jason A. Donenfeld jason@zx2c4.com Cc: Joel Granados joel.granados@kernel.org Cc: Joel Stanley joel@jms.id.au Cc: Jonathan Corbet corbet@lwn.net Cc: Kees Cook kees@kernel.org Cc: Lance Yang lance.yang@linux.dev Cc: Liam Howlett liam.howlett@oracle.com Cc: Lorenzo Stoakes lorenzo.stoakes@oracle.com Cc: "Masami Hiramatsu (Google)" mhiramat@kernel.org Cc: "Paul E . McKenney" paulmck@kernel.org Cc: Pawan Gupta pawan.kumar.gupta@linux.intel.com Cc: Petr Mladek pmladek@suse.com Cc: Phil Auld pauld@redhat.com Cc: Randy Dunlap rdunlap@infradead.org Cc: Russell King linux@armlinux.org.uk Cc: Shuah Khan shuah@kernel.org Cc: Simon Horman horms@kernel.org Cc: Stanislav Fomichev sdf@fomichev.me Cc: Steven Rostedt rostedt@goodmis.org
So:
Reviewed-by: Lance Yang lance.yang@linux.dev Tested-by: Lance Yang lance.yang@linux.dev
Cheers, Lance
On Thu, 16 Oct 2025 05:57:34 +0000 "Li,Rongqing" lirongqing@baidu.com wrote:
If you agree, likely no need to resend - Andrew could pick it up directly when applying :)
This is better;
Andrew, could you pick it up directly
No problems, thanks.
On Wed, 15 Oct 2025 14:36:15 +0800 lirongqing lirongqing@baidu.com wrote:
From: Li RongQing lirongqing@baidu.com
Currently, when 'hung_task_panic' is enabled, the kernel panics immediately upon detecting the first hung task. However, some hung tasks are transient and allow system recovery, while persistent hangs should trigger a panic when accumulating beyond a threshold.
Extend the 'hung_task_panic' sysctl to accept a threshold value specifying the number of hung tasks that must be detected before triggering a kernel panic. This provides finer control for environments where transient hangs may occur but persistent hangs should be fatal.
The sysctl now accepts:
- 0: don't panic (maintains original behavior)
- 1: panic on first hung task (maintains original behavior)
- N > 1: panic after N hung tasks are detected in a single scan
This maintains backward compatibility while providing flexibility for different hang scenarios.
Looks good to me.
Reviewed-by: Masami Hiramatsu (Google) mhiramat@kernel.org
Thank you,
Signed-off-by: Li RongQing lirongqing@baidu.com Cc: Andrew Jeffery andrew@codeconstruct.com.au Cc: Anshuman Khandual anshuman.khandual@arm.com Cc: Arnd Bergmann arnd@arndb.de Cc: David Hildenbrand david@redhat.com Cc: Florian Wesphal fw@strlen.de Cc: Jakub Kacinski kuba@kernel.org Cc: Jason A. Donenfeld jason@zx2c4.com Cc: Joel Granados joel.granados@kernel.org Cc: Joel Stanley joel@jms.id.au Cc: Jonathan Corbet corbet@lwn.net Cc: Kees Cook kees@kernel.org Cc: Lance Yang lance.yang@linux.dev Cc: Liam Howlett liam.howlett@oracle.com Cc: Lorenzo Stoakes lorenzo.stoakes@oracle.com Cc: "Masami Hiramatsu (Google)" mhiramat@kernel.org Cc: "Paul E . McKenney" paulmck@kernel.org Cc: Pawan Gupta pawan.kumar.gupta@linux.intel.com Cc: Petr Mladek pmladek@suse.com Cc: Phil Auld pauld@redhat.com Cc: Randy Dunlap rdunlap@infradead.org Cc: Russell King linux@armlinux.org.uk Cc: Shuah Khan shuah@kernel.org Cc: Simon Horman horms@kernel.org Cc: Stanislav Fomichev sdf@fomichev.me Cc: Steven Rostedt rostedt@goodmis.org
diff with v3: comments modification, suggested by Lance, Masami, Randy and Petr diff with v2: do not add a new sysctl, extend hung_task_panic, suggested by Kees Cook
Documentation/admin-guide/kernel-parameters.txt | 20 +++++++++++++------- Documentation/admin-guide/sysctl/kernel.rst | 9 +++++---- arch/arm/configs/aspeed_g5_defconfig | 2 +- kernel/configs/debug.config | 2 +- kernel/hung_task.c | 15 ++++++++++----- lib/Kconfig.debug | 9 +++++---- tools/testing/selftests/wireguard/qemu/kernel.config | 2 +- 7 files changed, 36 insertions(+), 23 deletions(-)
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index a51ab46..492f0bc 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -1992,14 +1992,20 @@ the added memory block itself do not be affected. hung_task_panic=
[KNL] Should the hung task detector generate panics.Format: 0 | 1
[KNL] Number of hung tasks to trigger kernel panic.Format: <int>When set to a non-zero value, a kernel panic will be triggered ifthe number of detected hung tasks reaches this value.0: don't panic1: panic immediately on first hung taskN: panic after N hung tasks are detected in a single scan
A value of 1 instructs the kernel to panic when ahung task is detected. The default value is controlledby the CONFIG_BOOTPARAM_HUNG_TASK_PANIC build-timeoption. The value selected by this boot parameter canbe changed later by the kernel.hung_task_panic sysctl.
The default value is controlled by theCONFIG_BOOTPARAM_HUNG_TASK_PANIC build-time option. The valueselected by this boot parameter can be changed later by thekernel.hung_task_panic sysctl.hvc_iucv= [S390] Number of z/VM IUCV hypervisor console (HVC) terminal devices. Valid values: 0..8 diff --git a/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/admin-guide/sysctl/kernel.rst index f3ee807..0065a55 100644 --- a/Documentation/admin-guide/sysctl/kernel.rst +++ b/Documentation/admin-guide/sysctl/kernel.rst @@ -397,13 +397,14 @@ a hung task is detected. hung_task_panic =============== -Controls the kernel's behavior when a hung task is detected. +When set to a non-zero value, a kernel panic will be triggered if the +number of hung tasks found during a single scan reaches this value. This file shows up if ``CONFIG_DETECT_HUNG_TASK`` is enabled. -= ================================================= += ======================================================= 0 Continue operation. This is the default behavior. -1 Panic immediately. -= ================================================= +N Panic when N hung tasks are found during a single scan. += ======================================================= hung_task_check_count diff --git a/arch/arm/configs/aspeed_g5_defconfig b/arch/arm/configs/aspeed_g5_defconfig index 61cee1e..c3b0d5f 100644 --- a/arch/arm/configs/aspeed_g5_defconfig +++ b/arch/arm/configs/aspeed_g5_defconfig @@ -308,7 +308,7 @@ CONFIG_PANIC_ON_OOPS=y CONFIG_PANIC_TIMEOUT=-1 CONFIG_SOFTLOCKUP_DETECTOR=y CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC=y -CONFIG_BOOTPARAM_HUNG_TASK_PANIC=y +CONFIG_BOOTPARAM_HUNG_TASK_PANIC=1 CONFIG_WQ_WATCHDOG=y # CONFIG_SCHED_DEBUG is not set CONFIG_FUNCTION_TRACER=y diff --git a/kernel/configs/debug.config b/kernel/configs/debug.config index e81327d..9f6ab7d 100644 --- a/kernel/configs/debug.config +++ b/kernel/configs/debug.config @@ -83,7 +83,7 @@ CONFIG_SLUB_DEBUG_ON=y # # Debug Oops, Lockups and Hangs # -# CONFIG_BOOTPARAM_HUNG_TASK_PANIC is not set +CONFIG_BOOTPARAM_HUNG_TASK_PANIC=0 # CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC is not set CONFIG_DEBUG_ATOMIC_SLEEP=y CONFIG_DETECT_HUNG_TASK=y diff --git a/kernel/hung_task.c b/kernel/hung_task.c index b2c1f14..84b4b04 100644 --- a/kernel/hung_task.c +++ b/kernel/hung_task.c @@ -81,7 +81,7 @@ static unsigned int __read_mostly sysctl_hung_task_all_cpu_backtrace;
- hung task is detected:
*/ static unsigned int __read_mostly sysctl_hung_task_panic =
- IS_ENABLED(CONFIG_BOOTPARAM_HUNG_TASK_PANIC);
- CONFIG_BOOTPARAM_HUNG_TASK_PANIC;
static int hung_task_panic(struct notifier_block *this, unsigned long event, void *ptr) @@ -218,8 +218,11 @@ static inline void debug_show_blocker(struct task_struct *task, unsigned long ti } #endif -static void check_hung_task(struct task_struct *t, unsigned long timeout) +static void check_hung_task(struct task_struct *t, unsigned long timeout,
unsigned long prev_detect_count){
- unsigned long total_hung_task;
- if (!task_is_hung(t, timeout)) return;
@@ -229,9 +232,10 @@ static void check_hung_task(struct task_struct *t, unsigned long timeout) */ sysctl_hung_task_detect_count++;
- total_hung_task = sysctl_hung_task_detect_count - prev_detect_count; trace_sched_process_hang(t);
- if (sysctl_hung_task_panic) {
- if (sysctl_hung_task_panic && total_hung_task >= sysctl_hung_task_panic) { console_verbose(); hung_task_show_lock = true; hung_task_call_panic = true;
@@ -300,6 +304,7 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout) int max_count = sysctl_hung_task_check_count; unsigned long last_break = jiffies; struct task_struct *g, *t;
- unsigned long prev_detect_count = sysctl_hung_task_detect_count;
/* * If the system crashed already then all bets are off, @@ -320,7 +325,7 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout) last_break = jiffies; }
check_hung_task(t, timeout);
check_hung_task(t, timeout, prev_detect_count);@@ -389,7 +394,7 @@ static const struct ctl_table hung_task_sysctls[] = { .mode = 0644, .proc_handler = proc_dointvec_minmax, .extra1 = SYSCTL_ZERO,
.extra2 = SYSCTL_ONE,
.extra2 = SYSCTL_INT_MAX,diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 3034e294..3976c90 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -1258,12 +1258,13 @@ config DEFAULT_HUNG_TASK_TIMEOUT Keeping the default should be fine in most cases. config BOOTPARAM_HUNG_TASK_PANIC
- bool "Panic (Reboot) On Hung Tasks"
- int "Number of hung tasks to trigger kernel panic" depends on DETECT_HUNG_TASK
- default 0 help
Say Y here to enable the kernel to panic on "hung tasks",which are bugs that cause the kernel to leave a task stuckin uninterruptible "D" state.
When set to a non-zero value, a kernel panic will be triggeredif the number of hung tasks found during a single scan reachesthis value.The panic can be used in combination with panic_timeout, to cause the system to reboot automatically after a diff --git a/tools/testing/selftests/wireguard/qemu/kernel.config b/tools/testing/selftests/wireguard/qemu/kernel.config index 936b18b..0504c11 100644 --- a/tools/testing/selftests/wireguard/qemu/kernel.config +++ b/tools/testing/selftests/wireguard/qemu/kernel.config @@ -81,7 +81,7 @@ CONFIG_WQ_WATCHDOG=y CONFIG_DETECT_HUNG_TASK=y CONFIG_BOOTPARAM_HARDLOCKUP_PANIC=y CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC=y -CONFIG_BOOTPARAM_HUNG_TASK_PANIC=y +CONFIG_BOOTPARAM_HUNG_TASK_PANIC=1 CONFIG_PANIC_TIMEOUT=-1 CONFIG_STACKTRACE=y CONFIG_EARLY_PRINTK=y -- 2.9.4
Dear RongQing,
Thank you for the patch. One minor comment regarding the Kconfig description.
Am 15.10.25 um 08:36 schrieb lirongqing:
From: Li RongQing lirongqing@baidu.com
Currently, when 'hung_task_panic' is enabled, the kernel panics immediately upon detecting the first hung task. However, some hung tasks are transient and allow system recovery, while persistent hangs should trigger a panic when accumulating beyond a threshold.
Extend the 'hung_task_panic' sysctl to accept a threshold value specifying the number of hung tasks that must be detected before triggering a kernel panic. This provides finer control for environments where transient hangs may occur but persistent hangs should be fatal.
The sysctl now accepts:
- 0: don't panic (maintains original behavior)
- 1: panic on first hung task (maintains original behavior)
- N > 1: panic after N hung tasks are detected in a single scan
This maintains backward compatibility while providing flexibility for different hang scenarios.
Signed-off-by: Li RongQing lirongqing@baidu.com Cc: Andrew Jeffery andrew@codeconstruct.com.au Cc: Anshuman Khandual anshuman.khandual@arm.com Cc: Arnd Bergmann arnd@arndb.de Cc: David Hildenbrand david@redhat.com Cc: Florian Wesphal fw@strlen.de Cc: Jakub Kacinski kuba@kernel.org Cc: Jason A. Donenfeld jason@zx2c4.com Cc: Joel Granados joel.granados@kernel.org Cc: Joel Stanley joel@jms.id.au Cc: Jonathan Corbet corbet@lwn.net Cc: Kees Cook kees@kernel.org Cc: Lance Yang lance.yang@linux.dev Cc: Liam Howlett liam.howlett@oracle.com Cc: Lorenzo Stoakes lorenzo.stoakes@oracle.com Cc: "Masami Hiramatsu (Google)" mhiramat@kernel.org Cc: "Paul E . McKenney" paulmck@kernel.org Cc: Pawan Gupta pawan.kumar.gupta@linux.intel.com Cc: Petr Mladek pmladek@suse.com Cc: Phil Auld pauld@redhat.com Cc: Randy Dunlap rdunlap@infradead.org Cc: Russell King linux@armlinux.org.uk Cc: Shuah Khan shuah@kernel.org Cc: Simon Horman horms@kernel.org Cc: Stanislav Fomichev sdf@fomichev.me Cc: Steven Rostedt rostedt@goodmis.org
diff with v3: comments modification, suggested by Lance, Masami, Randy and Petr diff with v2: do not add a new sysctl, extend hung_task_panic, suggested by Kees Cook
Documentation/admin-guide/kernel-parameters.txt | 20 +++++++++++++------- Documentation/admin-guide/sysctl/kernel.rst | 9 +++++---- arch/arm/configs/aspeed_g5_defconfig | 2 +- kernel/configs/debug.config | 2 +- kernel/hung_task.c | 15 ++++++++++----- lib/Kconfig.debug | 9 +++++---- tools/testing/selftests/wireguard/qemu/kernel.config | 2 +- 7 files changed, 36 insertions(+), 23 deletions(-)
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index a51ab46..492f0bc 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -1992,14 +1992,20 @@ the added memory block itself do not be affected. hung_task_panic=
[KNL] Should the hung task detector generate panics.Format: 0 | 1
[KNL] Number of hung tasks to trigger kernel panic.Format: <int>When set to a non-zero value, a kernel panic will be triggered ifthe number of detected hung tasks reaches this value.0: don't panic1: panic immediately on first hung taskN: panic after N hung tasks are detected in a single scan
A value of 1 instructs the kernel to panic when ahung task is detected. The default value is controlledby the CONFIG_BOOTPARAM_HUNG_TASK_PANIC build-timeoption. The value selected by this boot parameter canbe changed later by the kernel.hung_task_panic sysctl.
The default value is controlled by theCONFIG_BOOTPARAM_HUNG_TASK_PANIC build-time option. The valueselected by this boot parameter can be changed later by thekernel.hung_task_panic sysctl.hvc_iucv= [S390] Number of z/VM IUCV hypervisor console (HVC) terminal devices. Valid values: 0..8 diff --git a/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/admin-guide/sysctl/kernel.rst index f3ee807..0065a55 100644 --- a/Documentation/admin-guide/sysctl/kernel.rst +++ b/Documentation/admin-guide/sysctl/kernel.rst @@ -397,13 +397,14 @@ a hung task is detected. hung_task_panic =============== -Controls the kernel's behavior when a hung task is detected. +When set to a non-zero value, a kernel panic will be triggered if the +number of hung tasks found during a single scan reaches this value. This file shows up if ``CONFIG_DETECT_HUNG_TASK`` is enabled. -= ================================================= += ======================================================= 0 Continue operation. This is the default behavior. -1 Panic immediately. -= ================================================= +N Panic when N hung tasks are found during a single scan. += ======================================================= hung_task_check_count
[…]
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 3034e294..3976c90 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -1258,12 +1258,13 @@ config DEFAULT_HUNG_TASK_TIMEOUT Keeping the default should be fine in most cases. config BOOTPARAM_HUNG_TASK_PANIC
- bool "Panic (Reboot) On Hung Tasks"
- int "Number of hung tasks to trigger kernel panic" depends on DETECT_HUNG_TASK
- default 0 help
Say Y here to enable the kernel to panic on "hung tasks",which are bugs that cause the kernel to leave a task stuckin uninterruptible "D" state.
When set to a non-zero value, a kernel panic will be triggeredif the number of hung tasks found during a single scan reachesthis value.The panic can be used in combination with panic_timeout, to cause the system to reboot automatically after a
Why not leave the sentence about the uninterruptible "D" state in there?
Also, it sounds like, some are actually using this in production. Maybe it should be moved out of `Kconfig.debug` too?
Kind regards,
Paul
Am 15.10.25 um 08:36 schrieb lirongqing:
From: Li RongQing lirongqing@baidu.com
Currently, when 'hung_task_panic' is enabled, the kernel panics immediately upon detecting the first hung task. However, some hung tasks are transient and allow system recovery, while persistent hangs should trigger a panic when accumulating beyond a threshold.
Extend the 'hung_task_panic' sysctl to accept a threshold value specifying the number of hung tasks that must be detected before triggering a kernel panic. This provides finer control for environments where transient hangs may occur but persistent hangs
should be fatal.
The sysctl now accepts:
- 0: don't panic (maintains original behavior)
- 1: panic on first hung task (maintains original behavior)
- N > 1: panic after N hung tasks are detected in a single scan
This maintains backward compatibility while providing flexibility for different hang scenarios.
Signed-off-by: Li RongQing lirongqing@baidu.com Cc: Andrew Jeffery andrew@codeconstruct.com.au Cc: Anshuman Khandual anshuman.khandual@arm.com Cc: Arnd Bergmann arnd@arndb.de Cc: David Hildenbrand david@redhat.com Cc: Florian Wesphal fw@strlen.de Cc: Jakub Kacinski kuba@kernel.org Cc: Jason A. Donenfeld jason@zx2c4.com Cc: Joel Granados joel.granados@kernel.org Cc: Joel Stanley joel@jms.id.au Cc: Jonathan Corbet corbet@lwn.net Cc: Kees Cook kees@kernel.org Cc: Lance Yang lance.yang@linux.dev Cc: Liam Howlett liam.howlett@oracle.com Cc: Lorenzo Stoakes lorenzo.stoakes@oracle.com Cc: "Masami Hiramatsu (Google)" mhiramat@kernel.org Cc: "Paul E . McKenney" paulmck@kernel.org Cc: Pawan Gupta pawan.kumar.gupta@linux.intel.com Cc: Petr Mladek pmladek@suse.com Cc: Phil Auld pauld@redhat.com Cc: Randy Dunlap rdunlap@infradead.org Cc: Russell King linux@armlinux.org.uk Cc: Shuah Khan shuah@kernel.org Cc: Simon Horman horms@kernel.org Cc: Stanislav Fomichev sdf@fomichev.me Cc: Steven Rostedt rostedt@goodmis.org
diff with v3: comments modification, suggested by Lance, Masami, Randy and Petr diff with v2: do not add a new sysctl, extend hung_task_panic, suggested by Kees Cook
Documentation/admin-guide/kernel-parameters.txt | 20
+++++++++++++-------
Documentation/admin-guide/sysctl/kernel.rst | 9 +++++---- arch/arm/configs/aspeed_g5_defconfig | 2 +- kernel/configs/debug.config | 2 +- kernel/hung_task.c | 15
++++++++++-----
lib/Kconfig.debug | 9
+++++----
tools/testing/selftests/wireguard/qemu/kernel.config | 2 +- 7 files changed, 36 insertions(+), 23 deletions(-)
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index a51ab46..492f0bc 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -1992,14 +1992,20 @@ the added memory block itself do not be affected.
hung_task_panic=
[KNL] Should the hung task detector generate panics.Format: 0 | 1
[KNL] Number of hung tasks to trigger kernel panic.Format: <int>When set to a non-zero value, a kernel panic will be triggeredif
the number of detected hung tasks reaches this value.0: don't panic1: panic immediately on first hung taskN: panic after N hung tasks are detected in a single scan
A value of 1 instructs the kernel to panic when ahung task is detected. The default value is controlledby the CONFIG_BOOTPARAM_HUNG_TASK_PANIC build-timeoption. The value selected by this boot parameter canbe changed later by the kernel.hung_task_panic sysctl.
The default value is controlled by theCONFIG_BOOTPARAM_HUNG_TASK_PANIC build-time option.The value
selected by this boot parameter can be changed later by thekernel.hung_task_panic sysctl.hvc_iucv= [S390] Number of z/VM IUCV hypervisor console
(HVC)
terminal devices. Valid values: 0..8 diff --gita/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/admin-guide/sysctl/kernel.rst index f3ee807..0065a55 100644 --- a/Documentation/admin-guide/sysctl/kernel.rst +++ b/Documentation/admin-guide/sysctl/kernel.rst @@ -397,13 +397,14 @@ a hung task is detected. hung_task_panic ===============
-Controls the kernel's behavior when a hung task is detected. +When set to a non-zero value, a kernel panic will be triggered if the +number of hung tasks found during a single scan reaches this value. This file shows up if ``CONFIG_DETECT_HUNG_TASK`` is enabled.
-= ================================================= += ======================================================= 0 Continue operation. This is the default behavior. -1 Panic immediately. -= ================================================= +N Panic when N hung tasks are found during a single scan. += =======================================================
hung_task_check_count
[…]
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 3034e294..3976c90 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -1258,12 +1258,13 @@ config DEFAULT_HUNG_TASK_TIMEOUT Keeping the default should be fine in most cases.
config BOOTPARAM_HUNG_TASK_PANIC
- bool "Panic (Reboot) On Hung Tasks"
- int "Number of hung tasks to trigger kernel panic" depends on DETECT_HUNG_TASK
- default 0 help
Say Y here to enable the kernel to panic on "hung tasks",which are bugs that cause the kernel to leave a task stuckin uninterruptible "D" state.
When set to a non-zero value, a kernel panic will be triggeredif the number of hung tasks found during a single scan reachesthis value.The panic can be used in combination with panic_timeout, to cause the system to reboot automatically after a
Why not leave the sentence about the uninterruptible "D" state in there?
This seem to say a kernel bug to cause hung task, but it maybe hardware failure(or virtio backend bug); so I do not keep it
Also, it sounds like, some are actually using this in production. Maybe it should be moved out of `Kconfig.debug` too?
I think hung task panic is a useful feature, it should move out of Kconfig.debug
Thanks
-Li
Kind regards,
Paul
On Wed, 2025-10-15 at 14:36 +0800, lirongqing wrote:
From: Li RongQing lirongqing@baidu.com
Currently, when 'hung_task_panic' is enabled, the kernel panics immediately upon detecting the first hung task. However, some hung tasks are transient and allow system recovery, while persistent hangs should trigger a panic when accumulating beyond a threshold.
Extend the 'hung_task_panic' sysctl to accept a threshold value specifying the number of hung tasks that must be detected before triggering a kernel panic. This provides finer control for environments where transient hangs may occur but persistent hangs should be fatal.
The sysctl now accepts:
- 0: don't panic (maintains original behavior)
- 1: panic on first hung task (maintains original behavior)
- N > 1: panic after N hung tasks are detected in a single scan
This maintains backward compatibility while providing flexibility for different hang scenarios.
Signed-off-by: Li RongQing lirongqing@baidu.com Cc: Andrew Jeffery andrew@codeconstruct.com.au Cc: Anshuman Khandual anshuman.khandual@arm.com Cc: Arnd Bergmann arnd@arndb.de Cc: David Hildenbrand david@redhat.com Cc: Florian Wesphal fw@strlen.de Cc: Jakub Kacinski kuba@kernel.org Cc: Jason A. Donenfeld jason@zx2c4.com Cc: Joel Granados joel.granados@kernel.org Cc: Joel Stanley joel@jms.id.au Cc: Jonathan Corbet corbet@lwn.net Cc: Kees Cook kees@kernel.org Cc: Lance Yang lance.yang@linux.dev Cc: Liam Howlett liam.howlett@oracle.com Cc: Lorenzo Stoakes lorenzo.stoakes@oracle.com Cc: "Masami Hiramatsu (Google)" mhiramat@kernel.org Cc: "Paul E . McKenney" paulmck@kernel.org Cc: Pawan Gupta pawan.kumar.gupta@linux.intel.com Cc: Petr Mladek pmladek@suse.com Cc: Phil Auld pauld@redhat.com Cc: Randy Dunlap rdunlap@infradead.org Cc: Russell King linux@armlinux.org.uk Cc: Shuah Khan shuah@kernel.org Cc: Simon Horman horms@kernel.org Cc: Stanislav Fomichev sdf@fomichev.me Cc: Steven Rostedt rostedt@goodmis.org
diff with v3: comments modification, suggested by Lance, Masami, Randy and Petr diff with v2: do not add a new sysctl, extend hung_task_panic, suggested by Kees Cook
Documentation/admin-guide/kernel-parameters.txt | 20 +++++++++++++------- Documentation/admin-guide/sysctl/kernel.rst | 9 +++++---- arch/arm/configs/aspeed_g5_defconfig | 2 +-
For the aspeed_g5_defconfig change:
Acked-by: Andrew Jeffery andrew@codeconstruct.com.au
linux-kselftest-mirror@lists.linaro.org
-
Andrew Jeffery -
Andrew Morton -
Lance Yang -
Li,Rongqing -
lirongqing
-
Masami Hiramatsu -
Paul Menzel