Hi all,

On 6/28/24 1:15 오전, Sven Schnelle wrote:

Yunseong Kim yskelg@gmail.com writes:

...

In my s390x archtecture, kprobe_eventname selftest have always failed because of rcu_sched stalls.

My environment is QEMU Ubuntu 24.04 KVM Machine Linux version 6.8.0-36-generic (buildd@bos01-s390x-012) (s390x-linux-gnu-gcc-13 (Ubuntu 13.2.0-23ubuntu4) 13.2.0, GNU ld (GNU Binutils for Ubuntu) 2.42) #36

1 configured CPUs, Linux is running under KVM in 64-bit mode

qemu-system-s390x -no-reboot -name auto-inst-test -nographic -m 4096 \ -drive file=disk-image.qcow2,format=qcow2,cache=none,if=virtio \ -netdev user,id=enc0,hostfwd=tcp::10000-:22 \ -device virtio-net-ccw,netdev=enc0 \ -qmp tcp:localhost:4444,server,nowait

Currently, This failure can be always reproduced by this kselftests script:

# tools/testing/selftests/ftrace/ftracetest tools/testing/selftests/ftrace/test.d/kprobe/kprobe_eventname.tc

Could you see if you have the same issue on the most recent kernel?

-- Steve

Thanks you Steve,

As soon as I saw your mail, I pulled the latest kernel.

Linux version 6.10.0-rc5+ (paran@fedora) (s390x-linux-gnu-gcc (GCC) 14.1.1 20240507 (Red Hat Cross 14.1.1-1), GNU ld version 2.41-1.fc40) #30 SMP Fri Jun 28 04:32:20 KST 2024

My Linux is running under QEMU KVM in 64-bit mode, cpu: 1 configured CPUs, 0 standby CPUs.

[ 726.069032] Kernel panic - not syncing: Corrupt kernel stack, can't continue. [ 726.069297] CPU: 0 PID: 1023 Comm: ftracetest Not tainted 6.10.0-rc5+ #30 [ 726.069335] Hardware name: QEMU 8561 QEMU (KVM/Linux) [ 726.069361] Call Trace: [ 726.069383] [<000003ffe0dc545e>] dump_stack_lvl+0x76/0xa8 [ 726.069446] [<000003ffe0db7170>] panic+0x170/0x358 [ 726.069485] [<000003ffe0000f16>] kernel_stack_overflow+0x56/0x60 [ 726.069522] [<0000000000000200>] 0x200 [ 726.069987] Dumping ftrace buffer: [ 726.070144] --------------------------------- [ 726.071356] <...>-1023 0..... 1829038us : p_kernel_clone_0: (kernel_clone+0x0/0x390) [ 726.071452] <...>-1023 0dN... 1829717us : p_kernel_clone_6: (kernel_clone+0x6/0x390) [ 726.071506] <...>-1023 0dN... 1830009us : p_kernel_clone_12: (kernel_clone+0xc/0x390) [ 726.071558] <...>-1023 0dN... 1830064us : p_kernel_clone_16: ...

I have changed the QEMU setting a bit to run latest kernel.

qemu-system-s390x -no-reboot -name auto-inst-test -nographic -m 4096 \ -kernel bzImage-6.10.0-rc5-00035-gafcd48134c58 \ -initrd 6.8.0-36-generic/initrd.img-6.8.0-36-generic \ -append "root=/dev/disk/by-id/dm-uuid-LVM-7JeH1Hjp3H1U60GH8m82Od6YXP47ItB01EcvyJ33Ya80VPeRinI1vfMjI9Ndzcc3 crashkernel=512M-:1024M oops=panic panic_on_warn=1 ftrace_dump_on_oops=orig_cpu slub_debug=UZ nokaslr gcov_persist=0 " \ -drive file=disk-image.qcow2,format=qcow2,cache=none,if=virtio \ -netdev user,id=enc0,hostfwd=tcp::10000-:22 \ -device virtio-net-ccw,netdev=enc0 \ -qmp tcp:localhost:4444,server,nowait

Hi,

Thanks for reporting!

Hi, Masami, It's my pleasure.

...

# tools/testing/selftests/ftrace/ftracetest tools/testing/selftests/ftrace/test.d/kprobe/kprobe_eventname.tc

I've investigating cause of line, then I find this line.

for i in `seq 0 255`; do echo p $FUNCTION_FORK+${i} >> kprobe_events || continue done

cat kprobe_events >> $testlog

echo 1 > events/kprobes/enable # <<<

Isn't it kprobe_insn_boundary.tc?

That's exactly right! I'm sorry, I slipped up again. :(

This is my exactly tested command.

$ cd tools/testing/selftests/ftrace/ $ sudo ./ftracetest --console test.d/kprobe/kprobe_insn_boundary.tc

Also, can you modify it as

for i in `seq 0 255`; do echo p $FUNCTION_FORK+${i} >> kprobe_events || continue echo 1 > events/kprobes/enable done

And run it with --console option? It may tell us when the stall happens with what offset.

Also, please try it with the latest kernel instead of 6.8.x

Thank you,

I took what you said and applied it in test script.

--- a/tools/testing/selftests/ftrace/test.d/kprobe/kprobe_insn_boundary.tc +++ b/tools/testing/selftests/ftrace/test.d/kprobe/kprobe_insn_boundary.tc @@ -6,11 +6,11 @@

for i in `seq 0 255`; do echo p $FUNCTION_FORK+${i} >> kprobe_events || continue + echo 1 > events/kprobes/enable done

cat kprobe_events >> $testlog

-echo 1 > events/kprobes/enable ( echo "forked" ) echo 0 > events/kprobes/enable echo > kprobe_events

This is the results from console.

=== Ftrace unit tests === [1] Register multiple kprobe events in a function+ checkreq /home/paran/linux-source/tools/testing/selftests/ftrace/test.d/kprobe/kprobe_insn_boundary.tc + + grep ^#[ \t]*requires: /home/paran/linux-source-6.8.0/tools/testing/selftests/ftrace/test.d/kprobe/kprobe_insn_boundary.tc cut -f2- -d: + requires= kprobe_events + eval check_requires kprobe_events + check_requires kprobe_events + p=kprobe_events + r=kprobe_events + t=kprobe_events + [ kprobe_events != kprobe_events ] + [ kprobe_events != kprobe_events ] + [ kprobe_events != kprobe_events ] + [ ! -e kprobe_events ] + initialize_ftrace + disable_tracing + echo 0 + reset_tracer + echo nop + reset_trigger + [ -d events/synthetic ] + reset_trigger_file events/alarmtimer/alarmtimer_cancel/trigger events/alarmtimer/alarmtimer_fired/trigger events/alarmtimer+

....

clear_trace + echo + enable_tracing + echo 1 + . /home/paran/linux-source/tools/testing/selftests/ftrace/test.d/kprobe/kprobe_insn_boundary.tc + seq 0 255 + echo p kernel_clone+0 + echo 1 + echo p kernel_clone+1 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+2 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+3 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+4 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+5 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+6 + echo 1 + echo p kernel_clone+7 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+8 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+9 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+10 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+11 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+12 + echo 1 + echo p kernel_clone+13 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+14 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+15 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+16 + echo 1 + echo p kernel_clone+17 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+18 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+19 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+20 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+21 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+22 + echo 1 + echo p kernel_clone+23 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+24 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+25 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+26 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+27 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+28 + echo 1 + echo p kernel_clone+29 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+30 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+31 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+32 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+33 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+34 + echo 1 + echo p kernel_clone+35 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+36 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+37 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+38 + echo 1 + echo p kernel_clone+39 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+40 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+41 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+42 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+43 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+44 + echo 1 + echo p kernel_clone+45 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+46 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+47 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+48 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+49 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+50 + echo 1 + echo p kernel_clone+51 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+52 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+53 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+54 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+55 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+56 + echo 1 + echo p kernel_clone+57 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+58 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+59 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+60 + echo 1 + echo p kernel_clone+61 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+62 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+63 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+64 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+65 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+66 + echo 1 + echo p kernel_clone+67 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+68 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+69 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+70 + echo 1 + echo p kernel_clone+71 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+72 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+73 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+74 + echo 1 + echo p kernel_clone+75 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+76 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+77 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+78 + echo 1 + echo p kernel_clone+79 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+80 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+81 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+82 + echo 1 + echo p kernel_clone+83 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+84 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+85 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+86 + echo 1 + echo p kernel_clone+87 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+88 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+89 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+90 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+91 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+92 + echo 1 + echo p kernel_clone+93 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+94 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+95 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+96 + echo 1 + echo p kernel_clone+97 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+98 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+99 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+100 + echo 1 + echo p kernel_clone+101 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+102 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+103 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+104 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+105 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+106 + echo 1 + echo p kernel_clone+107 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+108 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+109 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+110 + echo 1 + echo p kernel_clone+111 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+112 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+113 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+114 + echo 1 + echo p kernel_clone+115 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+116 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+117 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+118 + echo 1 + echo p kernel_clone+119 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+120 + echo 1 + echo p kernel_clone+121 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+122 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+123 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+124 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+125 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+126 + echo 1 + echo p kernel_clone+127 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+128 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+129 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+130 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+131 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+132 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+133 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+134 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+135 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+136 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+137 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+138 + echo 1 + echo p kernel_clone+139 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+140 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+141 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+142 + echo 1 + echo p kernel_clone+143 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+144 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+145 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+146 + echo 1 + echo p kernel_clone+147 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+148 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+149 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+150 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+151 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+152 + echo 1 + echo p kernel_clone+153 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+154 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+155 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+156 + echo 1 + echo p kernel_clone+157 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+158 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+159 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+160 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+161 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+162 + echo 1 + echo p kernel_clone+163 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+164 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+165 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+166 + echo 1 + echo p kernel_clone+167 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+168 + echo 1 + echo p kernel_clone+169 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+170 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+171 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+172 + echo 1 + echo p kernel_clone+173 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+174 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+175 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+176 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+177 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+178 + echo 1 + echo p kernel_clone+179 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+180 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+181 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+182 + echo 1 + echo p kernel_clone+183 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+184 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+185 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+186 + echo 1 + echo p kernel_clone+187 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+188 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+189 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+190 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+191 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+192 + echo 1 + echo p kernel_clone+193 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+194 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+195 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+196 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+197 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+198 + echo 1 + echo p kernel_clone+199 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+200 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+201 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+202 + echo 1 + echo p kernel_clone+203 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+204 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+205 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+206 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+207 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+208 + echo 1 + echo p kernel_clone+209 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+210 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+211 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+212 + echo 1 + echo p kernel_clone+213 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+214 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+215 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+216 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+217 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+218 + echo 1 + echo p kernel_clone+219 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+220 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+221 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+222 + echo 1 + echo p kernel_clone+223 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+224 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+225 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+226 + echo 1 + echo p kernel_clone+227 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+228 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+229 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+230 + echo 1 + echo p kernel_clone+231 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+232 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+233 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+234 + echo 1 + echo p kernel_clone+235 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+236 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+237 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+238 + echo 1 + echo p kernel_clone+239 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+240 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+241 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+242 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+243 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+244 + echo 1 + echo p kernel_clone+245 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+246 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+247 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+248 + echo 1 + echo p kernel_clone+249 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+250 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+251 ./ftracetest: 8: echo: echo: I/O error + continue + echo p kernel_clone+252

...

I've investigating cause of line, then I find this line.

for i in `seq 0 255`; do echo p $FUNCTION_FORK+${i} >> kprobe_events || continue done

I think this is fixed with:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h...

Regards, Sven

Hi, Sven, Oh thank you so much for the link.

I have also checked that the patch(c239c83ed5c55) you mentioned has been applied to the code.

It looks like have another problem.

This is the whole call trace.

[ 726.069032] Kernel panic - not syncing: Corrupt kernel stack, can't continue. [ 726.069297] CPU: 0 PID: 1023 Comm: ftracetest Not tainted 6.10.0-rc5+ #30 [ 726.069335] Hardware name: QEMU 8561 QEMU (KVM/Linux) [ 726.069361] Call Trace: [ 726.069383] [<000003ffe0dc545e>] dump_stack_lvl+0x76/0xa8 [ 726.069446] [<000003ffe0db7170>] panic+0x170/0x358 [ 726.069485] [<000003ffe0000f16>] kernel_stack_overflow+0x56/0x60 [ 726.069522] [<0000000000000200>] 0x200 [ 726.069987] Dumping ftrace buffer: [ 726.070144] --------------------------------- [ 726.071356] <...>-1023 0..... 1829038us : p_kernel_clone_0: (kernel_clone+0x0/0x390) [ 726.071452] <...>-1023 0dN... 1829717us : p_kernel_clone_6: (kernel_clone+0x6/0x390) [ 726.071506] <...>-1023 0dN... 1830009us : p_kernel_clone_12: (kernel_clone+0xc/0x390) [ 726.071558] <...>-1023 0dN... 1830064us : p_kernel_clone_16: (kernel_clone+0x10/0x390) [ 726.071610] <...>-1023 0dN... 1830116us : p_kernel_clone_22: (kernel_clone+0x16/0x390) [ 726.071661] <...>-1023 0dN... 1830161us : p_kernel_clone_28: (kernel_clone+0x1c/0x390) [ 726.071711] <...>-1023 0dN... 1830204us : p_kernel_clone_34: (kernel_clone+0x22/0x390) [ 726.071762] <...>-1023 0dN... 1830252us : p_kernel_clone_38: (kernel_clone+0x26/0x390) [ 726.071815] <...>-1023 0dN... 1830301us : p_kernel_clone_44: (kernel_clone+0x2c/0x390) [ 726.071866] <...>-1023 0dN... 1830355us : p_kernel_clone_50: (kernel_clone+0x32/0x390) [ 726.071917] <...>-1023 0dN... 1830399us : p_kernel_clone_56: (kernel_clone+0x38/0x390) [ 726.071967] <...>-1023 0dN... 1830441us : p_kernel_clone_60: (kernel_clone+0x3c/0x390) [ 726.072018] <...>-1023 0dN... 1830495us : p_kernel_clone_66: (kernel_clone+0x42/0x390) [ 726.072068] <...>-1023 0dN... 1830553us : p_kernel_clone_70: (kernel_clone+0x46/0x390) [ 726.072118] <...>-1023 0dN... 1830597us : p_kernel_clone_74: (kernel_clone+0x4a/0x390) [ 726.072168] <...>-1023 0dN... 1830639us : p_kernel_clone_78: (kernel_clone+0x4e/0x390) [ 726.072217] <...>-1023 0dN... 1830681us : p_kernel_clone_82: (kernel_clone+0x52/0x390) [ 726.072267] <...>-1023 0dN... 1830722us : p_kernel_clone_86: (kernel_clone+0x56/0x390) [ 726.072318] <...>-1023 0dN... 1830765us : p_kernel_clone_92: (kernel_clone+0x5c/0x390) [ 726.072368] <...>-1023 0dN... 1830806us : p_kernel_clone_96: (kernel_clone+0x60/0x390) [ 726.072418] <...>-1023 0dN... 1830847us : p_kernel_clone_100: (kernel_clone+0x64/0x390) [ 726.072468] <...>-1023 0dN... 1830911us : p_kernel_clone_106: (kernel_clone+0x6a/0x390) [ 726.072518] <...>-1023 0dN... 1830953us : p_kernel_clone_110: (kernel_clone+0x6e/0x390) [ 726.072568] <...>-1023 0dN... 1830994us : p_kernel_clone_114: (kernel_clone+0x72/0x390) [ 726.072619] <...>-1023 0dN... 1831035us : p_kernel_clone_118: (kernel_clone+0x76/0x390) [ 726.072669] <...>-1023 0dN... 1831092us : p_kernel_clone_120: (kernel_clone+0x78/0x390) [ 726.072718] <...>-1023 0dN... 1831133us : p_kernel_clone_126: (kernel_clone+0x7e/0x390) [ 726.072769] <...>-1023 0dN... 1831175us : p_kernel_clone_138: (kernel_clone+0x8a/0x390) [ 726.072819] <...>-1023 0dN... 1831221us : p_kernel_clone_142: (kernel_clone+0x8e/0x390) [ 726.072869] <...>-1023 0dN... 1831262us : p_kernel_clone_146: (kernel_clone+0x92/0x390) [ 726.072919] <...>-1023 0dN... 1831303us : p_kernel_clone_152: (kernel_clone+0x98/0x390) [ 726.072970] <...>-1023 0dN... 1831344us : p_kernel_clone_156: (kernel_clone+0x9c/0x390) [ 726.073020] <...>-1023 0dN... 1831385us : p_kernel_clone_162: (kernel_clone+0xa2/0x390) [ 726.073070] <...>-1023 0dN... 1831427us : p_kernel_clone_166: (kernel_clone+0xa6/0x390) [ 726.073120] <...>-1023 0dN... 1831469us : p_kernel_clone_168: (kernel_clone+0xa8/0x390) [ 726.073910] <...>-1023 0dN... 1831511us : p_kernel_clone_172: (kernel_clone+0xac/0x390) [ 726.073983] <...>-1023 0dN... 1831554us : p_kernel_clone_178: (kernel_clone+0xb2/0x390) [ 726.074047] <...>-1023 0dN... 1831601us : p_kernel_clone_182: (kernel_clone+0xb6/0x390) [ 726.074109] <...>-1023 0dN... 1831642us : p_kernel_clone_186: (kernel_clone+0xba/0x390) [ 726.074172] <...>-1023 0dN... 1831683us : p_kernel_clone_192: (kernel_clone+0xc0/0x390) [ 726.074234] <...>-1023 0dN... 1831725us : p_kernel_clone_198: (kernel_clone+0xc6/0x390) [ 726.074296] <...>-1023 0dN... 1831766us : p_kernel_clone_202: (kernel_clone+0xca/0x390) [ 726.074358] <...>-1023 0dN... 1831807us : p_kernel_clone_208: (kernel_clone+0xd0/0x390) [ 726.074419] <...>-1023 0dN... 1831848us : p_kernel_clone_212: (kernel_clone+0xd4/0x390) [ 726.074481] <...>-1023 0dN... 1831895us : p_kernel_clone_218: (kernel_clone+0xda/0x390) [ 726.074542] <...>-1023 0dN... 1831941us : p_kernel_clone_222: (kernel_clone+0xde/0x390) [ 726.074603] <...>-1023 0dN... 1831982us : p_kernel_clone_226: (kernel_clone+0xe2/0x390) [ 726.074665] <...>-1023 0dN... 1832023us : p_kernel_clone_230: (kernel_clone+0xe6/0x390) [ 726.074727] <...>-1023 0dN... 1832064us : p_kernel_clone_234: (kernel_clone+0xea/0x390) [ 726.074787] <...>-1023 0dN... 1832105us : p_kernel_clone_238: (kernel_clone+0xee/0x390) [ 726.074846] <...>-1023 0dN... 1832146us : p_kernel_clone_244: (kernel_clone+0xf4/0x390) [ 726.074906] <...>-1023 0dN... 1832188us : p_kernel_clone_248: (kernel_clone+0xf8/0x390) [ 726.074965] <...>-1023 0dN... 1832229us : p_kernel_clone_252: (kernel_clone+0xfc/0x390) [ 726.075024] <...>-1023 0dN... 1832286us : p_kernel_clone_74: (kernel_clone+0x4a/0x390) [ 726.075083] <...>-1023 0dN... 1832324us : p_kernel_clone_78: (kernel_clone+0x4e/0x390) [ 726.075142] <...>-1023 0dN... 1832361us : p_kernel_clone_82: (kernel_clone+0x52/0x390) [ 726.075200] <...>-1023 0dN... 1832398us : p_kernel_clone_86: (kernel_clone+0x56/0x390) [ 726.075259] <...>-1023 0dN... 1832436us : p_kernel_clone_92: (kernel_clone+0x5c/0x390) [ 726.075317] <...>-1023 0dN... 1832472us : p_kernel_clone_96: (kernel_clone+0x60/0x390) [ 726.075376] <...>-1023 0dN... 1832510us : p_kernel_clone_100: (kernel_clone+0x64/0x390) [ 726.075434] <...>-1023 0dN... 1832547us : p_kernel_clone_106: (kernel_clone+0x6a/0x390) [ 726.075493] <...>-1023 0dN... 1832584us : p_kernel_clone_110: (kernel_clone+0x6e/0x390) [ 726.075552] <...>-1023 0dN... 1832620us : p_kernel_clone_114: (kernel_clone+0x72/0x390) [ 726.075611] <...>-1023 0dN... 1832657us : p_kernel_clone_118: (kernel_clone+0x76/0x390) [ 726.075669] <...>-1023 0dN... 1832694us : p_kernel_clone_120: (kernel_clone+0x78/0x390) [ 726.075734] <...>-1023 0dN... 1832731us : p_kernel_clone_126: (kernel_clone+0x7e/0x390) [ 726.075793] <...>-1023 0dN... 1832768us : p_kernel_clone_138: (kernel_clone+0x8a/0x390) [ 726.075860] <...>-1023 0dN... 1832805us : p_kernel_clone_142: (kernel_clone+0x8e/0x390) [ 726.075922] <...>-1023 0dN... 1832848us : p_kernel_clone_146: (kernel_clone+0x92/0x390) [ 726.075980] <...>-1023 0dN... 1832885us : p_kernel_clone_152: (kernel_clone+0x98/0x390) [ 726.076046] <...>-1023 0dN... 1832922us : p_kernel_clone_156: (kernel_clone+0x9c/0x390) [ 726.076107] <...>-1023 0dN... 1832959us : p_kernel_clone_162: (kernel_clone+0xa2/0x390) [ 726.076168] <...>-1023 0dN... 1832996us : p_kernel_clone_166: (kernel_clone+0xa6/0x390) [ 726.076229] <...>-1023 0dN... 1833038us : p_kernel_clone_168: (kernel_clone+0xa8/0x390) [ 726.076290] <...>-1023 0dN... 1833078us : p_kernel_clone_172: (kernel_clone+0xac/0x390) [ 726.076376] <...>-1023 0dN... 1833117us : p_kernel_clone_178: (kernel_clone+0xb2/0x390) [ 726.076438] <...>-1023 0dN... 1833154us : p_kernel_clone_182: (kernel_clone+0xb6/0x390) [ 726.076505] <...>-1023 0dN... 1833190us : p_kernel_clone_186: (kernel_clone+0xba/0x390) [ 726.076565] <...>-1023 0dN... 1833229us : p_kernel_clone_192: (kernel_clone+0xc0/0x390) [ 726.076634] <...>-1023 0dN... 1833266us : p_kernel_clone_198: (kernel_clone+0xc6/0x390) [ 726.076695] <...>-1023 0dN... 1833302us : p_kernel_clone_202: (kernel_clone+0xca/0x390) [ 726.076779] <...>-1023 0dN... 1833339us : p_kernel_clone_208: (kernel_clone+0xd0/0x390) [ 726.076842] <...>-1023 0dN... 1833376us : p_kernel_clone_212: (kernel_clone+0xd4/0x390) [ 726.076886] ---------------------------------

Thank you Steve, Masami and Sven for the great advice on what I reported.

If there are any additional aspects I should check or any advice, please let me know and I will look into it.

Warm regards, Yunseong Kim