On Tue, 10 Jun 2025 at 20:22, Steven Rostedt rostedt@goodmis.org wrote:
On Tue, 10 Jun 2025 18:50:05 +0530 Naresh Kamboju naresh.kamboju@linaro.org wrote:
Is this bug reproducible easier recently?
Yes. It is easy to reproduce.
Can you test before and after this commit:
4334336e769b ("x86/alternatives: Improve code-patching scalability by removing false sharing in poke_int3_handler()")
I think that may be the culprit.
Even if Masami's patches work, I want to know what exactly caused it.
Steven,
Since the reported regressions are intermittent, It is not easy to bisect. However, The commit merged into Linux next-20250414 tag and then started noticing from next-20250415 onwards this regression on both x86_64 devices and qemu-x86_64 intermittently with and without compat mode.
- https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20250606/tes... - https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20250606/tes...
And above commit landed into Linus master branch on 2025-05-13 and then started noticing this regression intermittently on x86 with and without compat mode.
- https://qa-reports.linaro.org/lkft/linux-mainline-master/build/v6.16-rc1/tes...
Masami San,
case 1) compat mode x86_64 (64-bit kernel + 32-bit rootfs) I have tested your patch on top of linux next-20250606 tag and tested on real x86_64 (64-bit kernel + 32-bit rootfs) hardware for 7 test runs.
ftrace_regression01 - pass ftrace_regression02 - pass ftrace-stress-test - pass dynamic_debug01 - Hangs (No crash log on serial console)
Case 1.1) Above case noticed on qemu-x86_64 with compat mode ^ with 12 test runs.
- https://lkft.validation.linaro.org/scheduler/job/8312811#L1687
case 2) x86_64 (64-bit kernel + 64-bit rootfs) I have tested your patch on top of linux next-20250606 tag and tested on real x86_64 (64-bit kernel + 64-bit rootfs) hardware for 4 runs and out of these 3 runs failed and found these kernel warnings, kernel BUG and invalid opcode while running LTP tracing test cases.
Here I am sharing the crash log snippet and boot and test log links and build link.
Test logs: [ 112.596591] Ring buffer clock went backwards: 113864910133 -> 112596588266 [ 115.829620] cat (5762) used greatest stack depth: 10936 bytes left [ 120.922517] ------------[ cut here ]------------ [ 120.927198] WARNING: CPU: 2 PID: 6639 at kernel/trace/trace_functions_graph.c:985 print_graph_entry+0x579/0x590 [ 120.937364] Modules linked in: x86_pkg_temp_thermal [ 120.942405] CPU: 2 UID: 0 PID: 6639 Comm: cat Tainted: G S 6.15.0-next-20250606 #1 PREEMPT(voluntary) [ 120.953380] Tainted: [S]=CPU_OUT_OF_SPEC [ 120.957477] Hardware name: Supermicro SYS-5019S-ML/X11SSH-F, BIOS 2.7 12/07/2021 [ 120.965036] RIP: 0010:print_graph_entry+0x579/0x590
Run 1: - https://lkft.validation.linaro.org/scheduler/job/8311136#L1700
ftrace-stress-test: [ 58.963898] /usr/local/bin/kirk[340]: starting test ftrace-stress-test (ftrace_stress_test.sh 90) [ 60.316588] ------------[ cut here ]------------ [ 60.316588] ------------[ cut here ]------------ [ 60.316590] ------------[ cut here ]------------ [ 60.316593] ------------[ cut here ]------------ [ 60.316593] ------------[ cut here ]------------ [ 60.316594] ------------[ cut here ]------------ [ 60.316594] kernel BUG at kernel/entry/common.c:328! [ 60.316594] kernel BUG at kernel/entry/common.c:328! [ 60.316595] kernel BUG at kernel/entry/common.c:328! [ 60.316600] Oops: invalid opcode: 0000 [#1] SMP PTI [ 60.316604] CPU: 2 UID: 0 PID: 1556 Comm: sh Tainted: G S 6.15.0-next-20250606 #1 PREEMPT(voluntary) [ 60.316608] Tainted: [S]=CPU_OUT_OF_SPEC [ 60.316609] Hardware name: Supermicro SYS-5019S-ML/X11SSH-F, BIOS 2.7 12/07/2021 [ 60.316614] ------------[ cut here ]------------ [ 60.316615] kernel BUG at kernel/entry/common.c:328! [ 60.316617] Oops: invalid opcode: 0000 [#2] SMP PTI [ 60.316620] CPU: 2 UID: 0 PID: 1556 Comm: sh Tainted: G S 6.15.0-next-20250606 #1 PREEMPT(voluntary) [ 60.316622] Tainted: [S]=CPU_OUT_OF_SPEC [ 60.316623] Hardware name: Supermicro SYS-5019S-ML/X11SSH-F, BIOS 2.7 12/07/2021 [ 60.316625] RIP: 0010:irqentry_nmi_enter+0x6c/0x70
Run 2: - https://lkft.validation.linaro.org/scheduler/job/8311138#L1703
ftrace-stress-test: [ 78.877495] /usr/local/bin/kirk[343]: starting test ftrace-stress-test (ftrace_stress_test.sh 90) [ 78.977303] Scheduler tracepoints stat_sleep, stat_iowait, stat_blocked and stat_runtime require the kernel parameter schedstats=enable or kernel.sched_schedstats=1 [ 82.299799] cat (2322) used greatest stack depth: 11520 bytes left [ 82.327708] cat (2327) used greatest stack depth: 11256 bytes left [ 82.632183] cat (2375) used greatest stack depth: 10992 bytes left [ 137.335901] ------------[ cut here ]------------ [ 137.335901] ------------[ cut here ]------------ [ 137.335902] ------------[ cut here ]------------ [ 137.335907] kernel BUG at kernel/entry/common.c:328! [ 137.335908] ------------[ cut here ]------------ [ 137.335909] ------------[ cut here ]------------ [ 137.335912] kernel BUG at kernel/entry/common.c:328! [ 137.335912] kernel BUG at kernel/entry/common.c:328! [ 137.335915] Oops: invalid opcode: 0000 [#1] SMP PTI [ 137.335921] CPU: 0 UID: 0 PID: 544 Comm: sh Tainted: G S 6.15.0-next-20250606 #1 PREEMPT(voluntary) [ 137.335926] Tainted: [S]=CPU_OUT_OF_SPEC [ 137.335929] Hardware name: Supermicro SYS-5019S-ML/X11SSH-F, BIOS 2.7 12/07/2021 [ 137.335937] ------------[ cut here ]------------ [ 137.335939] kernel BUG at kernel/entry/common.c:328! [ 137.335945] Oops: invalid opcode: 0000 [#2] SMP PTI [ 137.335949] CPU: 0 UID: 0 PID: 544 Comm: sh Tainted: G S 6.15.0-next-20250606 #1 PREEMPT(voluntary) [ 137.335953] Tainted: [S]=CPU_OUT_OF_SPEC [ 137.335956] Hardware name: Supermicro SYS-5019S-ML/X11SSH-F, BIOS 2.7 12/07/2021 [ 137.335959] RIP: 0010:irqentry_nmi_enter+0x6c/0x70
Run 3: - https://lkft.validation.linaro.org/scheduler/job/8311139#L1703
Build log: - https://storage.tuxsuite.com/public/linaro/naresh/builds/2yM9krm5KgE5a57QFvO...
- Naresh
-- Steve