The following boot warning is noticed on qemu-arm64 devices booting Linux next-20251215 on wards.
Regression Analysis: - New regression? yes - Reproducibility? yes
First seen on next-20251215 Bad: next-20251215 and next-20251216 Good: next-20251212
Boot regression: arm64: WARNING: kernel/sched/core.c:10851 at sched_change_end
Reported-by: Linux Kernel Functional Testing lkft@linaro.org
$ git blame -L 10842 kernel/sched/core.c 5d1f0b2f278eb (Peter Zijlstra 2025-12-10 09:06:50 +0100 10842) ----> if (ctx->running && sched_class_above(p->sched_class, ctx->class)) { 5d1f0b2f278eb (Peter Zijlstra 2025-12-10 09:06:50 +0100 10843) rq->next_class->wakeup_preempt(rq, p, 0); 5d1f0b2f278eb (Peter Zijlstra 2025-12-10 09:06:50 +0100 10844) rq->next_class = p->sched_class; 5d1f0b2f278eb (Peter Zijlstra 2025-12-10 09:06:50 +0100 10845) } 5d1f0b2f278eb (Peter Zijlstra 2025-12-10 09:06:50 +0100 10846) 47efe2ddccb1f (Peter Zijlstra 2025-10-30 12:56:34 +0100 10847) /* 47efe2ddccb1f (Peter Zijlstra 2025-10-30 12:56:34 +0100 10848) * If this was a degradation in class someone should have set 47efe2ddccb1f (Peter Zijlstra 2025-10-30 12:56:34 +0100 10849) * need_resched by now. 47efe2ddccb1f (Peter Zijlstra 2025-10-30 12:56:34 +0100 10850) */ 47efe2ddccb1f (Peter Zijlstra 2025-10-30 12:56:34 +0100 10851) ----> WARN_ON_ONCE(sched_class_above(ctx->class, p->sched_class) && 47efe2ddccb1f (Peter Zijlstra 2025-10-30 12:56:34 +0100 10852) !test_tsk_need_resched(p)); 6455ad5346c9c (Peter Zijlstra 2024-11-01 14:16:10 +0100 10853) } else { 6455ad5346c9c (Peter Zijlstra 2024-11-01 14:16:10 +0100 10854) p->sched_class->prio_changed(rq, p, ctx->prio); 6455ad5346c9c (Peter Zijlstra 2024-11-01 14:16:10 +0100 10855) } f0e1a0643a59b (Tejun Heo 2024-06-18 10:09:17 -1000 10856) }
## Boot log [ 0.000000] Linux version 6.19.0-rc1-next-20251215 (tuxmake@tuxmake) (aarch64-linux-gnu-gcc (Debian 13.3.0-16) 13.3.0, GNU ld (GNU Binutils for Debian) 2.44) #1 SMP PREEMPT @1765798467 ... [ 14.659447] mlx5_core 0000:01:00.1 enp1s0f1np1: renamed from eth1 [ 14.691267] CPPC Cpufreq:FIE not enabled on systems with registers in PCC ** replaying previous printk message ** [ 14.691267] CPPC Cpufreq:FIE not enabled on systems with registers in PCC [ 14.694920] mlx5_core 0000:01:00.0 enp1s0f0np0: renamed from eth0 (while UP) [ 14.696414] ------------[ cut here ]------------ [ 14.696418] WARNING: kernel/sched/core.c:10851 at sched_change_end+0x168/0x188, CPU#12: ktimers/12/117 [ 14.729321] Modules linked in: cppc_cpufreq(+) arm_dsu_pmu(+) fuse drm backlight [ 14.736718] CPU: 12 UID: 0 PID: 117 Comm: ktimers/12 Not tainted 6.19.0-rc1-next-20251216 #1 PREEMPT_RT [ 14.746190] Hardware name: WIWYNN Mt.Jade Server System B81.030Z1.0010/Mt.Jade Motherboard, BIOS 2.10.20250506-1P (SCP: 2.10.20250506) 2025/05/06 [ 14.759217] pstate: 804000c9 (Nzcv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 14.766169] pc : sched_change_end (kernel/sched/core.c:10851 (discriminator 7)) [ 14.770519] lr : sched_change_end (kernel/sched/core.c:10842 (discriminator 1)) [ 14.774781] sp : ffff800080d33b90 [ 14.778084] x29: ffff800080d33b90 x28: ffff07ff822f5500 x27: ffff80008e0c3660 [ 14.785213] x26: 00000000000000f0 x25: ffff07ff822f5500 x24: 0000000000000000 [ 14.792342] x23: 00000000ffffffff x22: ffffb4099c0d64d0 x21: ffff07ff822f5500 [ 14.799469] x20: ffff083e5eeaf140 x19: ffff083e5ee9b290 x18: 0000000000000000 [ 14.806597] x17: 0000000000000000 x16: ffffb4099a40ee68 x15: 0000000000000000 [ 14.813725] x14: 0000000000000000 x13: 0000000000000000 x12: 0000020000000000 [ 14.820853] x11: 00000000000000c0 x10: 0000000000000c60 x9 : ffffb40999b8d640 [ 14.827981] x8 : 000000000000000c x7 : ffff07ff92ad8400 x6 : 00000000ffffffff [ 14.835109] x5 : 00000000000000ee x4 : 000000036b338800 x3 : 000000036b33b800 [ 14.842238] x2 : 0000000000000001 x1 : ffffb4099c0d63d8 x0 : 0000000000000010 [ 14.849366] Call trace: [ 14.851802] sched_change_end (kernel/sched/core.c:10851 (discriminator 7)) (P) [ 14.856153] rt_mutex_setprio (kernel/sched/sched.h:1813 kernel/sched/core.c:7368) [ 14.860157] rt_mutex_slowunlock (kernel/locking/rtmutex.c:1341 kernel/locking/rtmutex.c:1466) [ 14.864422] rt_spin_unlock (kernel/locking/spinlock_rt.c:86) [ 14.868165] __hrtimer_run_queues (include/linux/spinlock_rt.h:44 kernel/time/hrtimer.c:1398 kernel/time/hrtimer.c:1843) [ 14.872513] hrtimer_run_softirq (kernel/time/hrtimer.c:579 kernel/time/hrtimer.c:1193 kernel/time/hrtimer.c:1861) [ 14.876600] handle_softirqs.isra.0 (arch/arm64/include/asm/jump_label.h:36 include/trace/events/irq.h:142 kernel/softirq.c:627) [ 14.881125] run_ktimerd (kernel/softirq.c:325 kernel/softirq.c:1144) [ 14.884520] smpboot_thread_fn (kernel/smpboot.c:160) [ 14.888608] kthread (kernel/kthread.c:463) [ 14.891828] ret_from_fork (arch/arm64/kernel/entry.S:861) [ 14.895396] ---[ end trace 0000000000000000 ]---
## Source * Kernel version: 6.19.0-rc1 * Git tree: https://kernel.googlesource.com/pub/scm/linux/kernel/git/next/linux-next.git * Git commit: 4a5663c04bb679631985a15efab774da58c37815 * Git Describe: next-20251215 * Test Details: https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20251215
## Boot details * Boot log: https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20251215/tes... * Test details: https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20251215/tes... * Test plan: https://tuxapi.tuxsuite.com/v1/groups/linaro/projects/lkft/tests/36sbHUVfnUK... * Build link: https://storage.tuxsuite.com/public/linaro/lkft/builds/36sbH5DQckAagO0V1UEWB... * Kernel config: https://storage.tuxsuite.com/public/linaro/lkft/builds/36sbH5DQckAagO0V1UEWB...
-- Linaro LKFT https://lkft.linaro.org
On Wed, Dec 17, 2025 at 12:41:39AM +0530, Naresh Kamboju wrote:
Hi Mark,
Hi Naresh,
Sorry for the prior reply being a little blunt, and thanks for the good natured reply; I've tried to provide some more constructive notes below. :)
On Tue, 16 Dec 2025 at 22:53, Mark Rutland mark.rutland@arm.com wrote:
On Tue, Dec 16, 2025 at 05:09:52PM +0530, Naresh Kamboju wrote:
The following boot warning is noticed on qemu-arm64 devices booting Linux next-20251215 on wards.
- New regression? yes
- Reproducibility? yes
First seen on next-20251215 Bad: next-20251215 and next-20251216 Good: next-20251212
Boot regression: arm64: WARNING: kernel/sched/core.c:10851 at sched_change_end
The warning you quote should be the *FIRST* line after you mention "The following boot warning".
That should be quoted *exactly* as the kernel logged it, without being prefixed by "Boot regression: arm64:", which the kernel didn't log, and which is redundant given the title and surrounding context.
The additional prefixes were introduced for internal statistics tracking and report classification purposes. However, I acknowledge that they reduced clarity in this case.
That's fair enough. My key complaint here is just that the initial message had:
| The following boot warning is noticed on qemu-arm64 devices booting | Linux next-20251215 onwards. | | [ several lines that are not the warning ] | | Boot regression: arm64: WARNING: kernel/sched/core.c:10851 at sched_change_end
... where a reader has to skim through all the irrelevant lines, and then it's not clear that the line beginning with "Boot regression: arm64:" was the actual warning.
Whereas if that were formatted something like:
| The following boot warning is noticed on qemu-arm64 devices booting | Linux next-20251215 onwards: | | WARNING: kernel/sched/core.c:10851 at sched_change_end | | Note: full splat at the end of this mail. | | The testing system has characterized this as follows: | | - Architecture: arm64 | - Failure: Boot regression | - New regression? yes | - Reproducible? yes | - First seen on next-20251215 | - Bad: next-20251215 and next-20251216 | - Good: next-20251212 | - Some other attribute: foo | - Yet another attribute: bar
... it'd be much easier for a reviewer to spot the exact warning, since it's up-front, where they'd expect it, and exactly matching what the kernel output, without any additional notes.
All the metadata being grouped together after that makes it easier for someone to skip over it when they don't need it.
[...]
[ 14.696414] ------------[ cut here ]------------ [ 14.696418] WARNING: kernel/sched/core.c:10851 at sched_change_end+0x168/0x188, CPU#12: ktimers/12/117 [ 14.729321] Modules linked in: cppc_cpufreq(+) arm_dsu_pmu(+) fuse drm backlight [ 14.736718] CPU: 12 UID: 0 PID: 117 Comm: ktimers/12 Not tainted 6.19.0-rc1-next-20251216 #1 PREEMPT_RT [ 14.746190] Hardware name: WIWYNN Mt.Jade Server System B81.030Z1.0010/Mt.Jade Motherboard, BIOS 2.10.20250506-1P (SCP: 2.10.20250506) 2025/05/06
This doesn't look like a "qemu-arm64 device" to me. Are you *sure* this wasn't bare-metal on a "WIWYNN Mt.Jade Server System"?
If not, why is QEMU passing that gunk to the guest!?
Apologies for the confusion regarding the platform identification. This warning was reproduced on both qemu-arm64 and Mt. Jade. The log is from the Mt. Jade system.
Cool; thanks for confirming!
Mark.
On Tue, Dec 16, 2025 at 05:09:52PM +0530, Naresh Kamboju wrote:
The following boot warning is noticed on qemu-arm64 devices booting Linux next-20251215 on wards.
Regression Analysis:
- New regression? yes
- Reproducibility? yes
First seen on next-20251215 Bad: next-20251215 and next-20251216 Good: next-20251212
Boot regression: arm64: WARNING: kernel/sched/core.c:10851 at sched_change_end
Reported-by: Linux Kernel Functional Testing lkft@linaro.org
This is absolutely unreadable due to all the wrapping.
On Tue, Dec 16, 2025 at 05:09:52PM +0530, Naresh Kamboju wrote:
The following boot warning is noticed on qemu-arm64 devices booting Linux next-20251215 on wards.
I didn't realise LKFT was operating from a hospital!
Regression Analysis:
- New regression? yes
- Reproducibility? yes
First seen on next-20251215 Bad: next-20251215 and next-20251216 Good: next-20251212
Boot regression: arm64: WARNING: kernel/sched/core.c:10851 at sched_change_end
This email is really painful to read, because the information is out-of-order and has random words added to obscure relevant information.
The warning you quote should be the *FIRST* line after you mention "The following boot warning".
That should be quoted *exactly* as the kernel logged it, without being prefixed by "Boot regression: arm64:", which the kernel didn't log, and which is redundant given the title and surrounding context.
It'd be *significantly* clearer to have:
| I'm seeing the following warning consistently on qemu-arm64 when | booting next-20251215 and later: | | WARNING: kernel/sched/core.c:10851 at sched_change_end | | Note: full splat at the end of this mail. | | First seen on next-20251215 | Bad: next-20251215 and next-20251216 | Good: next-20251212 | | This looks to be a regression.
... where someone can see all the relevant details at a glance.
Reported-by: Linux Kernel Functional Testing lkft@linaro.org
$ git blame -L 10842 kernel/sched/core.c 5d1f0b2f278eb (Peter Zijlstra 2025-12-10 09:06:50 +0100 10842) ----> if (ctx->running && sched_class_above(p->sched_class, ctx->class)) { 5d1f0b2f278eb (Peter Zijlstra 2025-12-10 09:06:50 +0100 10843) rq->next_class->wakeup_preempt(rq, p, 0); 5d1f0b2f278eb (Peter Zijlstra 2025-12-10 09:06:50 +0100 10844) rq->next_class = p->sched_class; 5d1f0b2f278eb (Peter Zijlstra 2025-12-10 09:06:50 +0100 10845) } 5d1f0b2f278eb (Peter Zijlstra 2025-12-10 09:06:50 +0100 10846) 47efe2ddccb1f (Peter Zijlstra 2025-10-30 12:56:34 +0100 10847) /* 47efe2ddccb1f (Peter Zijlstra 2025-10-30 12:56:34 +0100 10848)
- If this was a degradation in class someone should have set
47efe2ddccb1f (Peter Zijlstra 2025-10-30 12:56:34 +0100 10849)
- need_resched by now.
47efe2ddccb1f (Peter Zijlstra 2025-10-30 12:56:34 +0100 10850) */ 47efe2ddccb1f (Peter Zijlstra 2025-10-30 12:56:34 +0100 10851) ----> WARN_ON_ONCE(sched_class_above(ctx->class, p->sched_class) && 47efe2ddccb1f (Peter Zijlstra 2025-10-30 12:56:34 +0100 10852) !test_tsk_need_resched(p)); 6455ad5346c9c (Peter Zijlstra 2024-11-01 14:16:10 +0100 10853) } else { 6455ad5346c9c (Peter Zijlstra 2024-11-01 14:16:10 +0100 10854) p->sched_class->prio_changed(rq, p, ctx->prio); 6455ad5346c9c (Peter Zijlstra 2024-11-01 14:16:10 +0100 10855) } f0e1a0643a59b (Tejun Heo 2024-06-18 10:09:17 -1000 10856) }
This is all formatted illegibly since it was line-wrapped, and you didn't mention why you're dumping this. AFAICT the only relevant bit is that the warning is from:
WARN_ON_ONCE(sched_class_above(ctx->class, p->sched_class) && !test_tsk_need_resched(p));
... which Peter added in commit:
47efe2ddccb1f ("sched/core: Add assertions to QUEUE_CLASS")
Have you tried looking around that commit, or bisecting?
[...]
Note: I've fixed the line-wrapping for the below. Please fix that in future.
[ 14.696414] ------------[ cut here ]------------ [ 14.696418] WARNING: kernel/sched/core.c:10851 at sched_change_end+0x168/0x188, CPU#12: ktimers/12/117 [ 14.729321] Modules linked in: cppc_cpufreq(+) arm_dsu_pmu(+) fuse drm backlight [ 14.736718] CPU: 12 UID: 0 PID: 117 Comm: ktimers/12 Not tainted 6.19.0-rc1-next-20251216 #1 PREEMPT_RT [ 14.746190] Hardware name: WIWYNN Mt.Jade Server System B81.030Z1.0010/Mt.Jade Motherboard, BIOS 2.10.20250506-1P (SCP: 2.10.20250506) 2025/05/06
This doesn't look like a "qemu-arm64 device" to me. Are you *sure* this wasn't bare-metal on a "WIWYNN Mt.Jade Server System"?
If not, why is QEMU passing that gunk to the guest!?
Mark.
[ 14.759217] pstate: 804000c9 (Nzcv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 14.766169] pc : sched_change_end (kernel/sched/core.c:10851 (discriminator 7)) [ 14.770519] lr : sched_change_end (kernel/sched/core.c:10842 (discriminator 1)) [ 14.774781] sp : ffff800080d33b90 [ 14.778084] x29: ffff800080d33b90 x28: ffff07ff822f5500 x27: ffff80008e0c3660 [ 14.785213] x26: 00000000000000f0 x25: ffff07ff822f5500 x24: 0000000000000000 [ 14.792342] x23: 00000000ffffffff x22: ffffb4099c0d64d0 x21: ffff07ff822f5500 [ 14.799469] x20: ffff083e5eeaf140 x19: ffff083e5ee9b290 x18: 0000000000000000 [ 14.806597] x17: 0000000000000000 x16: ffffb4099a40ee68 x15: 0000000000000000 [ 14.813725] x14: 0000000000000000 x13: 0000000000000000 x12: 0000020000000000 [ 14.820853] x11: 00000000000000c0 x10: 0000000000000c60 x9 : ffffb40999b8d640 [ 14.827981] x8 : 000000000000000c x7 : ffff07ff92ad8400 x6 : 00000000ffffffff [ 14.835109] x5 : 00000000000000ee x4 : 000000036b338800 x3 : 000000036b33b800 [ 14.842238] x2 : 0000000000000001 x1 : ffffb4099c0d63d8 x0 : 0000000000000010 [ 14.849366] Call trace: [ 14.851802] sched_change_end (kernel/sched/core.c:10851 (discriminator 7)) (P) [ 14.856153] rt_mutex_setprio (kernel/sched/sched.h:1813 kernel/sched/core.c:7368) [ 14.860157] rt_mutex_slowunlock (kernel/locking/rtmutex.c:1341 kernel/locking/rtmutex.c:1466) [ 14.864422] rt_spin_unlock (kernel/locking/spinlock_rt.c:86) [ 14.868165] __hrtimer_run_queues (include/linux/spinlock_rt.h:44 kernel/time/hrtimer.c:1398 kernel/time/hrtimer.c:1843) [ 14.872513] hrtimer_run_softirq (kernel/time/hrtimer.c:579 kernel/time/hrtimer.c:1193 kernel/time/hrtimer.c:1861) [ 14.876600] handle_softirqs.isra.0 (arch/arm64/include/asm/jump_label.h:36 include/trace/events/irq.h:142 kernel/softirq.c:627) [ 14.881125] run_ktimerd (kernel/softirq.c:325 kernel/softirq.c:1144) [ 14.884520] smpboot_thread_fn (kernel/smpboot.c:160) [ 14.888608] kthread (kernel/kthread.c:463) [ 14.891828] ret_from_fork (arch/arm64/kernel/entry.S:861) [ 14.895396] ---[ end trace 0000000000000000 ]---
## Source
- Kernel version: 6.19.0-rc1
- Git tree: https://kernel.googlesource.com/pub/scm/linux/kernel/git/next/linux-next.git
- Git commit: 4a5663c04bb679631985a15efab774da58c37815
- Git Describe: next-20251215
- Test Details: https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20251215
## Boot details
- Boot log: https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20251215/tes...
- Test details: https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20251215/tes...
- Test plan: https://tuxapi.tuxsuite.com/v1/groups/linaro/projects/lkft/tests/36sbHUVfnUK...
- Build link: https://storage.tuxsuite.com/public/linaro/lkft/builds/36sbH5DQckAagO0V1UEWB...
- Kernel config: https://storage.tuxsuite.com/public/linaro/lkft/builds/36sbH5DQckAagO0V1UEWB...
-- Linaro LKFT https://lkft.linaro.org
Hi Mark,
On Tue, 16 Dec 2025 at 22:53, Mark Rutland mark.rutland@arm.com wrote:
On Tue, Dec 16, 2025 at 05:09:52PM +0530, Naresh Kamboju wrote:
The following boot warning is noticed on qemu-arm64 devices booting Linux next-20251215 on wards.
I didn't realise LKFT was operating from a hospital!
:) No hospital ward involved :) “onwards” it is. Thanks for spotting that.
Thank you for taking the time to review the report and for the detailed feedback. I appreciate you pointing out the issues with ordering, formatting, and accuracy. This is very helpful.
Regression Analysis:
- New regression? yes
- Reproducibility? yes
First seen on next-20251215 Bad: next-20251215 and next-20251216 Good: next-20251212
Boot regression: arm64: WARNING: kernel/sched/core.c:10851 at sched_change_end
This email is really painful to read, because the information is out-of-order and has random words added to obscure relevant information.
This one’s on me. I used Gmail instead of git send-email/mutt for the kernel logs. I’ll use git send-email or mutt for regression reporting going forward.
The warning you quote should be the *FIRST* line after you mention "The following boot warning".
That should be quoted *exactly* as the kernel logged it, without being prefixed by "Boot regression: arm64:", which the kernel didn't log, and which is redundant given the title and surrounding context.
The additional prefixes were introduced for internal statistics tracking and report classification purposes. However, I acknowledge that they reduced clarity in this case.
For example, Build regression: Boot regression: Test regression:
Any suggestions for a regression classification identifier for reporting ?
It'd be *significantly* clearer to have:
| I'm seeing the following warning consistently on qemu-arm64 when | booting next-20251215 and later: | | WARNING: kernel/sched/core.c:10851 at sched_change_end | | Note: full splat at the end of this mail. | | First seen on next-20251215 | Bad: next-20251215 and next-20251216 | Good: next-20251212 | | This looks to be a regression.
Noted.
... where someone can see all the relevant details at a glance.
Reported-by: Linux Kernel Functional Testing lkft@linaro.org
<trim>
This is all formatted illegibly since it was line-wrapped, and you didn't mention why you're dumping this. AFAICT the only relevant bit is that the warning is from:
WARN_ON_ONCE(sched_class_above(ctx->class, p->sched_class) && !test_tsk_need_resched(p));... which Peter added in commit:
47efe2ddccb1f ("sched/core: Add assertions to QUEUE_CLASS")
Have you tried looking around that commit, or bisecting?
My point was simply to share the git blame details for the recent change.
[...]
Note: I've fixed the line-wrapping for the below. Please fix that in future.
I’ll make sure to follow your suggested regression reporting format going forward.
[ 14.696414] ------------[ cut here ]------------ [ 14.696418] WARNING: kernel/sched/core.c:10851 at sched_change_end+0x168/0x188, CPU#12: ktimers/12/117 [ 14.729321] Modules linked in: cppc_cpufreq(+) arm_dsu_pmu(+) fuse drm backlight [ 14.736718] CPU: 12 UID: 0 PID: 117 Comm: ktimers/12 Not tainted 6.19.0-rc1-next-20251216 #1 PREEMPT_RT [ 14.746190] Hardware name: WIWYNN Mt.Jade Server System B81.030Z1.0010/Mt.Jade Motherboard, BIOS 2.10.20250506-1P (SCP: 2.10.20250506) 2025/05/06
This doesn't look like a "qemu-arm64 device" to me. Are you *sure* this wasn't bare-metal on a "WIWYNN Mt.Jade Server System"?
If not, why is QEMU passing that gunk to the guest!?
Apologies for the confusion regarding the platform identification. This warning was reproduced on both qemu-arm64 and Mt. Jade. The log is from the Mt. Jade system.
Mark.
- Naresh
-
Mark Rutland -
Naresh Kamboju -
Peter Zijlstra