Hi,
There is new WARNING and Oops on ppc64le in v6.6.37 when running LTP tests: bpf_prog01, bpf_prog02, bpf_prog04, bpf_prog05, prctl04. Logs excerpt below. I see there is 1 commit in v6.6.36..v6.6.37 with call to bpf_jit_binary_pack_finalize, backported from 5 patch mainline patchset:
f99feda5684a powerpc/bpf: use bpf_jit_binary_pack_[alloc|finalize|free]
Log:
[ 8.822079] LTP: starting bpf_prog01 [ 8.841853] ------------[ cut here ]------------ [ 8.841946] Trying to vfree() bad address (00000000453be747) [ 8.842024] WARNING: CPU: 6 PID: 689 at mm/vmalloc.c:2700 remove_vm_area+0xb4/0xf0 [ 8.842103] Modules linked in: virtio_rng rng_core virtio_net net_failover failover sd_mod ata_generic ata_piix libata scsi_mod scsi_common virtio_blk virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev 9pnet_virtio virtio_ring virtio 9p 9pnet netfs [ 8.842323] CPU: 6 PID: 689 Comm: bpf_prog01 Not tainted 6.6.37-un-def-alt1 #1 [ 8.842396] Hardware name: IBM pSeries (emulated by qemu) POWER8 (raw) 0x4d0200 0xf000004 of:SLOF,git-3a259d hv:linux,kvm pSeries [ 8.842519] NIP: c0000000004faf04 LR: c0000000004faf00 CTR: 0000000000000000 [ 8.842598] REGS: c000000009b6f250 TRAP: 0700 Not tainted (6.6.37-un-def-alt1) [ 8.842669] MSR: 8000000000029033 <SF,EE,ME,IR,DR,RI,LE> CR: 28002822 XER: 00000000 [ 8.842748] CFAR: c00000000015df94 IRQMASK: 0 [ 8.842748] GPR00: 0000000000000000 c000000009b6f4f0 c000000001ac7f00 0000000000000000 [ 8.842748] GPR04: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 8.842748] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 8.842748] GPR12: 0000000000000000 c00000003fff7a00 0000000000000000 0000000000000000 [ 8.842748] GPR16: 0000000000000012 0000000000000000 000000000000008c 0000000000000000 [ 8.842748] GPR20: c008000000040a40 0000000000000002 c0000000022a7560 c008000000040a4c [ 8.842748] GPR24: c000000005716480 0000000000000000 c000000002155698 c0000000022a7680 [ 8.842748] GPR28: c000000002155688 c008000000040a40 c008000000040a40 c008000000040a40 [ 8.843347] NIP [c0000000004faf04] remove_vm_area+0xb4/0xf0 [ 8.843398] LR [c0000000004faf00] remove_vm_area+0xb0/0xf0 [ 8.843448] Call Trace: [ 8.843484] [c000000009b6f4f0] [c0000000004faf00] remove_vm_area+0xb0/0xf0 (unreliable) [ 8.843559] [c000000009b6f560] [c0000000004fb360] vfree+0x60/0x2a0 [ 8.843621] [c000000009b6f5e0] [c000000000269c6c] module_memfree+0x3c/0x60 [ 8.843685] [c000000009b6f600] [c00000000038cf60] bpf_jit_free_exec+0x20/0x40 [ 8.843759] [c000000009b6f620] [c00000000038f518] bpf_prog_pack_free+0x2f8/0x390 [ 8.843832] [c000000009b6f6b0] [c00000000038f878] bpf_jit_binary_pack_finalize+0x98/0xd0 [ 8.843906] [c000000009b6f6e0] [c000000000118240] bpf_int_jit_compile+0x2c0/0x710 [ 8.843979] [c000000009b6f830] [c00000000038ef64] bpf_prog_select_runtime+0x154/0x1b0 [ 8.844053] [c000000009b6f880] [c000000000398edc] bpf_prog_load+0x94c/0xe90 [ 8.844114] [c000000009b6f990] [c00000000039c878] __sys_bpf+0x418/0x2970 [ 8.844176] [c000000009b6fac0] [c00000000039f1a0] sys_bpf+0x30/0x50 [ 8.844237] [c000000009b6fae0] [c000000000030230] system_call_exception+0x190/0x390 [ 8.844312] [c000000009b6fe50] [c00000000000c7d4] system_call_common+0xf4/0x258 [ 8.844386] --- interrupt: c00 at 0x7fffb0839ad4 [ 8.844437] NIP: 00007fffb0839ad4 LR: 000000012a027fb4 CTR: 0000000000000000 [ 8.844524] REGS: c000000009b6fe80 TRAP: 0c00 Not tainted (6.6.37-un-def-alt1) [ 8.844596] MSR: 800000000280f033 <SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE> CR: 22002240 XER: 00000000 [ 8.844690] IRQMASK: 0 [ 8.844690] GPR00: 0000000000000169 00007fffd8534200 00007fffb0936d00 0000000000000005 [ 8.844690] GPR04: 00007fffb06aff90 0000000000000070 000000012a0538a0 0000000000000001 [ 8.844690] GPR08: 000000012a0801f4 0000000000000000 0000000000000000 0000000000000000 [ 8.844690] GPR12: 0000000000000000 00007fffb09ea540 0000000000000000 0000000000000000 [ 8.844690] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 8.844690] GPR20: 00007fffd85344b0 0000000000000000 0000000000000001 0000000000000000 [ 8.844690] GPR24: 000000012a0801f4 00007fffb06ce000 0000000000000000 00000000000f4240 [ 8.844690] GPR28: 00007fffb06aff90 00007fffb09e3550 0000000000000001 0000000000001118 [ 8.845267] NIP [00007fffb0839ad4] 0x7fffb0839ad4 [ 8.845315] LR [000000012a027fb4] 0x12a027fb4 [ 8.845363] --- interrupt: c00 [ 8.845399] Code: 38000000 38800000 39200000 4e800020 60000000 60000000 60420000 3c62ffa2 7fe4fb78 3863e698 4bc62f8d 60000000 <0fe00000> 38210070 3bc00000 e8010010 [ 8.845550] ---[ end trace 0000000000000000 ]--- [ 8.845603] ------------[ cut here ]------------ [ 8.845651] Trying to vfree() nonexistent vm area (00000000453be747) [ 8.845714] WARNING: CPU: 6 PID: 689 at mm/vmalloc.c:2835 vfree+0x1d8/0x2a0 [ 8.845776] Modules linked in: virtio_rng rng_core virtio_net net_failover failover sd_mod ata_generic ata_piix libata scsi_mod scsi_common virtio_blk virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev 9pnet_virtio virtio_ring virtio 9p 9pnet netfs [ 8.845989] CPU: 6 PID: 689 Comm: bpf_prog01 Tainted: G W 6.6.37-un-def-alt1 #1 [ 8.846072] Hardware name: IBM pSeries (emulated by qemu) POWER8 (raw) 0x4d0200 0xf000004 of:SLOF,git-3a259d hv:linux,kvm pSeries [ 8.846177] NIP: c0000000004fb4d8 LR: c0000000004fb4d4 CTR: 0000000000000000 [ 8.846248] REGS: c000000009b6f2c0 TRAP: 0700 Tainted: G W (6.6.37-un-def-alt1) [ 8.846330] MSR: 8000000000029033 <SF,EE,ME,IR,DR,RI,LE> CR: 28002222 XER: 00000000 [ 8.846408] CFAR: c00000000015df94 IRQMASK: 0 [ 8.846408] GPR00: 0000000000000000 c000000009b6f560 c000000001ac7f00 0000000000000000 [ 8.846408] GPR04: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 8.846408] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 8.846408] GPR12: 0000000000000000 c00000003fff7a00 0000000000000000 0000000000000000 [ 8.846408] GPR16: 0000000000000012 0000000000000000 000000000000008c 0000000000000000 [ 8.846408] GPR20: c008000000040a40 0000000000000002 c0000000022a7560 c008000000040a4c [ 8.846408] GPR24: c000000005716480 0000000000000000 c000000002155698 c0000000022a7680 [ 8.846408] GPR28: c000000002155688 0000000000000000 c008000000040a40 0000000000000000 [ 8.851030] NIP [c0000000004fb4d8] vfree+0x1d8/0x2a0 [ 8.851085] LR [c0000000004fb4d4] vfree+0x1d4/0x2a0 [ 8.851135] Call Trace: [ 8.851160] [c000000009b6f560] [c0000000004fb4d4] vfree+0x1d4/0x2a0 (unreliable) [ 8.851234] [c000000009b6f5e0] [c000000000269c6c] module_memfree+0x3c/0x60 [ 8.851297] [c000000009b6f600] [c00000000038cf60] bpf_jit_free_exec+0x20/0x40 [ 8.851371] [c000000009b6f620] [c00000000038f518] bpf_prog_pack_free+0x2f8/0x390 [ 8.851445] [c000000009b6f6b0] [c00000000038f878] bpf_jit_binary_pack_finalize+0x98/0xd0 [ 8.851529] [c000000009b6f6e0] [c000000000118240] bpf_int_jit_compile+0x2c0/0x710 [ 8.851602] [c000000009b6f830] [c00000000038ef64] bpf_prog_select_runtime+0x154/0x1b0 [ 8.851675] [c000000009b6f880] [c000000000398edc] bpf_prog_load+0x94c/0xe90 [ 8.851737] [c000000009b6f990] [c00000000039c878] __sys_bpf+0x418/0x2970 [ 8.851798] [c000000009b6fac0] [c00000000039f1a0] sys_bpf+0x30/0x50 [ 8.851860] [c000000009b6fae0] [c000000000030230] system_call_exception+0x190/0x390 [ 8.851934] [c000000009b6fe50] [c00000000000c7d4] system_call_common+0xf4/0x258 [ 8.852007] --- interrupt: c00 at 0x7fffb0839ad4 [ 8.852057] NIP: 00007fffb0839ad4 LR: 000000012a027fb4 CTR: 0000000000000000 [ 8.852128] REGS: c000000009b6fe80 TRAP: 0c00 Tainted: G W (6.6.37-un-def-alt1) [ 8.852212] MSR: 800000000280f033 <SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE> CR: 22002240 XER: 00000000 [ 8.852307] IRQMASK: 0 [ 8.852307] GPR00: 0000000000000169 00007fffd8534200 00007fffb0936d00 0000000000000005 [ 8.852307] GPR04: 00007fffb06aff90 0000000000000070 000000012a0538a0 0000000000000001 [ 8.852307] GPR08: 000000012a0801f4 0000000000000000 0000000000000000 0000000000000000 [ 8.852307] GPR12: 0000000000000000 00007fffb09ea540 0000000000000000 0000000000000000 [ 8.852307] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 8.852307] GPR20: 00007fffd85344b0 0000000000000000 0000000000000001 0000000000000000 [ 8.852307] GPR24: 000000012a0801f4 00007fffb06ce000 0000000000000000 00000000000f4240 [ 8.852307] GPR28: 00007fffb06aff90 00007fffb09e3550 0000000000000001 0000000000001118 [ 8.852889] NIP [00007fffb0839ad4] 0x7fffb0839ad4 [ 8.852938] LR [000000012a027fb4] 0x12a027fb4 [ 8.852986] --- interrupt: c00 [ 8.853022] Code: 4e800020 60420000 3949ffff 4bffff0c 38210080 ebe1fff8 4bfffd68 3c62ffa2 7fc4f378 3863e6f0 4bc629b9 60000000 <0fe00000> eba10068 4bffff8c 2c080000 [ 8.853164] ---[ end trace 0000000000000000 ]--- [ 8.856619] kernel tried to execute exec-protected page (c008000000040a4c) - exploit attempt? (uid: 0) [ 8.856717] BUG: Unable to handle kernel instruction fetch [ 8.856763] Faulting instruction address: 0xc008000000040a4c [ 8.856825] Oops: Kernel access of bad area, sig: 11 [#1] [ 8.856875] LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries [ 8.856937] Modules linked in: virtio_rng rng_core virtio_net net_failover failover sd_mod ata_generic ata_piix libata scsi_mod scsi_common virtio_blk virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev 9pnet_virtio virtio_ring virtio 9p 9pnet netfs [ 8.857154] CPU: 6 PID: 689 Comm: bpf_prog01 Tainted: G W 6.6.37-un-def-alt1 #1 [ 8.857236] Hardware name: IBM pSeries (emulated by qemu) POWER8 (raw) 0x4d0200 0xf000004 of:SLOF,git-3a259d hv:linux,kvm pSeries [ 8.857342] NIP: c008000000040a4c LR: c000000000ed25d0 CTR: c008000000040a4c [ 8.857413] REGS: c000000009b6f6f0 TRAP: 0400 Tainted: G W (6.6.37-un-def-alt1) [ 8.857510] MSR: 8000000010009033 <SF,EE,ME,IR,DR,RI,LE> CR: 28008286 XER: 00000000 [ 8.857588] CFAR: c000000000ed25cc IRQMASK: 0 [ 8.857588] GPR00: c000000000ed25a8 c000000009b6f990 c000000001ac7f00 c000000006130400 [ 8.857588] GPR04: c008000000920048 0000000000000001 0000000000000000 0000000000000000 [ 8.857588] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 8.857588] GPR12: c008000000040a4c c00000003fff7a00 0000000000000000 0000000000000000 [ 8.857588] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 8.857588] GPR20: 7fffffffffffffff 0000000000000000 0000000000000001 0000000000000000 [ 8.857588] GPR24: c000000006130400 c000000006510a00 c000000006510f00 c0000000041a9000 [ 8.857588] GPR28: 0000000000000001 c000000006130400 0000000000000000 c008000000920000 [ 8.858184] NIP [c008000000040a4c] bpf_prog_2fb4fda3a3499517+0x0/0x8c [ 8.858245] LR [c000000000ed25d0] sk_filter_trim_cap+0xc0/0x370 [ 8.858308] Call Trace: [ 8.858333] [c000000009b6f990] [c000000000ed2574] sk_filter_trim_cap+0x64/0x370 (unreliable) [ 8.858421] [c000000009b6fa10] [c000000001068b64] unix_dgram_sendmsg+0x214/0xb10 [ 8.858511] [c000000009b6fad0] [c000000000e4c59c] sock_write_iter+0x19c/0x1e0 [ 8.858586] [c000000009b6fb80] [c0000000005b1b58] vfs_write+0x258/0x4e0 [ 8.858648] [c000000009b6fc40] [c0000000005b21d4] ksys_write+0x114/0x170 [ 8.858711] [c000000009b6fc90] [c000000000030230] system_call_exception+0x190/0x390 [ 8.858785] [c000000009b6fe50] [c00000000000c7d4] system_call_common+0xf4/0x258 [ 8.858859] --- interrupt: c00 at 0x7fffb082b884 [ 8.858908] NIP: 00007fffb082b884 LR: 000000012a02ab70 CTR: 0000000000000000 [ 8.858979] REGS: c000000009b6fe80 TRAP: 0c00 Tainted: G W (6.6.37-un-def-alt1) [ 8.859060] MSR: 800000000280f033 <SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE> CR: 28002281 XER: 00000000 [ 8.859153] IRQMASK: 0 [ 8.859153] GPR00: 0000000000000004 00007fffd85341f0 00007fffb0936d00 0000000000000005 [ 8.859153] GPR04: 00007fffb068fffa 0000000000000006 0000000000000001 0000000000000005 [ 8.859153] GPR08: 00007fffb068fffa 0000000000000000 0000000000000000 0000000000000000 [ 8.859153] GPR12: 0000000000000000 00007fffb09ea540 0000000000000000 0000000000000000 [ 8.859153] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 8.859153] GPR20: 00007fffd85344b0 0000000000000000 0000000000000001 0000000000000000 [ 8.859153] GPR24: 000000012a053698 000000000000008b 0000000000000000 0000000000000001 [ 8.859153] GPR28: 00007fffb068fffa 0000000000000005 0000000000000006 000000012a053698 [ 8.859738] NIP [00007fffb082b884] 0x7fffb082b884 [ 8.859786] LR [000000012a02ab70] 0x12a02ab70 [ 8.859836] --- interrupt: c00 [ 8.859872] Code: 7fe00008 7fe00008 7fe00008 7fe00008 7fe00008 7fe00008 7fe00008 7fe00008 7fe00008 7fe00008 7fe00008 7fe00008 <7fe00008> 7fe00008 7fe00008 7fe00008 [ 8.860013] ---[ end trace 0000000000000000 ]--- [ 8.863088] pstore: backend (nvram) writing error (-1) [ 8.863141] [ 8.863166] note: bpf_prog01[689] exited with irqs disabled
And so on. Temporary build/test log is at https://git.altlinux.org/tasks/352218/build/100/ppc64le/log
Other stable/longterm branches or other architectures does not exhibit this.
Thanks,
On 2024-07-05 16:34, Vitaly Chikunov wrote:
Hi,
There is new WARNING and Oops on ppc64le in v6.6.37 when running LTP tests: bpf_prog01, bpf_prog02, bpf_prog04, bpf_prog05, prctl04. Logs excerpt below. I see there is 1 commit in v6.6.36..v6.6.37 with call to bpf_jit_binary_pack_finalize, backported from 5 patch mainline patchset:
f99feda5684a powerpc/bpf: use bpf_jit_binary_pack_[alloc|finalize|free]
Log:
[ 8.822079] LTP: starting bpf_prog01 [ 8.841853] ------------[ cut here ]------------ [ 8.841946] Trying to vfree() bad address (00000000453be747) [ 8.842024] WARNING: CPU: 6 PID: 689 at mm/vmalloc.c:2700 remove_vm_area+0xb4/0xf0 [ 8.842103] Modules linked in: virtio_rng rng_core virtio_net net_failover failover sd_mod ata_generic ata_piix libata scsi_mod scsi_common virtio_blk virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev 9pnet_virtio virtio_ring virtio 9p 9pnet netfs [ 8.842323] CPU: 6 PID: 689 Comm: bpf_prog01 Not tainted 6.6.37-un-def-alt1 #1 [ 8.842396] Hardware name: IBM pSeries (emulated by qemu) POWER8 (raw) 0x4d0200 0xf000004 of:SLOF,git-3a259d hv:linux,kvm pSeries [ 8.842519] NIP: c0000000004faf04 LR: c0000000004faf00 CTR: 0000000000000000 [ 8.842598] REGS: c000000009b6f250 TRAP: 0700 Not tainted (6.6.37-un-def-alt1) [ 8.842669] MSR: 8000000000029033 <SF,EE,ME,IR,DR,RI,LE> CR: 28002822 XER: 00000000 [ 8.842748] CFAR: c00000000015df94 IRQMASK: 0 [ 8.842748] GPR00: 0000000000000000 c000000009b6f4f0 c000000001ac7f00 0000000000000000 [ 8.842748] GPR04: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 8.842748] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 8.842748] GPR12: 0000000000000000 c00000003fff7a00 0000000000000000 0000000000000000 [ 8.842748] GPR16: 0000000000000012 0000000000000000 000000000000008c 0000000000000000 [ 8.842748] GPR20: c008000000040a40 0000000000000002 c0000000022a7560 c008000000040a4c [ 8.842748] GPR24: c000000005716480 0000000000000000 c000000002155698 c0000000022a7680 [ 8.842748] GPR28: c000000002155688 c008000000040a40 c008000000040a40 c008000000040a40 [ 8.843347] NIP [c0000000004faf04] remove_vm_area+0xb4/0xf0 [ 8.843398] LR [c0000000004faf00] remove_vm_area+0xb0/0xf0 [ 8.843448] Call Trace: [ 8.843484] [c000000009b6f4f0] [c0000000004faf00] remove_vm_area+0xb0/0xf0 (unreliable) [ 8.843559] [c000000009b6f560] [c0000000004fb360] vfree+0x60/0x2a0 [ 8.843621] [c000000009b6f5e0] [c000000000269c6c] module_memfree+0x3c/0x60 [ 8.843685] [c000000009b6f600] [c00000000038cf60] bpf_jit_free_exec+0x20/0x40 [ 8.843759] [c000000009b6f620] [c00000000038f518] bpf_prog_pack_free+0x2f8/0x390 [ 8.843832] [c000000009b6f6b0] [c00000000038f878] bpf_jit_binary_pack_finalize+0x98/0xd0 [ 8.843906] [c000000009b6f6e0] [c000000000118240] bpf_int_jit_compile+0x2c0/0x710 [ 8.843979] [c000000009b6f830] [c00000000038ef64] bpf_prog_select_runtime+0x154/0x1b0 [ 8.844053] [c000000009b6f880] [c000000000398edc] bpf_prog_load+0x94c/0xe90 [ 8.844114] [c000000009b6f990] [c00000000039c878] __sys_bpf+0x418/0x2970 [ 8.844176] [c000000009b6fac0] [c00000000039f1a0] sys_bpf+0x30/0x50 [ 8.844237] [c000000009b6fae0] [c000000000030230] system_call_exception+0x190/0x390 [ 8.844312] [c000000009b6fe50] [c00000000000c7d4] system_call_common+0xf4/0x258 [ 8.844386] --- interrupt: c00 at 0x7fffb0839ad4 [ 8.844437] NIP: 00007fffb0839ad4 LR: 000000012a027fb4 CTR: 0000000000000000 [ 8.844524] REGS: c000000009b6fe80 TRAP: 0c00 Not tainted (6.6.37-un-def-alt1) [ 8.844596] MSR: 800000000280f033 <SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE> CR: 22002240 XER: 00000000 [ 8.844690] IRQMASK: 0 [ 8.844690] GPR00: 0000000000000169 00007fffd8534200 00007fffb0936d00 0000000000000005 [ 8.844690] GPR04: 00007fffb06aff90 0000000000000070 000000012a0538a0 0000000000000001 [ 8.844690] GPR08: 000000012a0801f4 0000000000000000 0000000000000000 0000000000000000 [ 8.844690] GPR12: 0000000000000000 00007fffb09ea540 0000000000000000 0000000000000000 [ 8.844690] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 8.844690] GPR20: 00007fffd85344b0 0000000000000000 0000000000000001 0000000000000000 [ 8.844690] GPR24: 000000012a0801f4 00007fffb06ce000 0000000000000000 00000000000f4240 [ 8.844690] GPR28: 00007fffb06aff90 00007fffb09e3550 0000000000000001 0000000000001118 [ 8.845267] NIP [00007fffb0839ad4] 0x7fffb0839ad4 [ 8.845315] LR [000000012a027fb4] 0x12a027fb4 [ 8.845363] --- interrupt: c00 [ 8.845399] Code: 38000000 38800000 39200000 4e800020 60000000 60000000 60420000 3c62ffa2 7fe4fb78 3863e698 4bc62f8d 60000000 <0fe00000> 38210070 3bc00000 e8010010 [ 8.845550] ---[ end trace 0000000000000000 ]--- [ 8.845603] ------------[ cut here ]------------ [ 8.845651] Trying to vfree() nonexistent vm area (00000000453be747) [ 8.845714] WARNING: CPU: 6 PID: 689 at mm/vmalloc.c:2835 vfree+0x1d8/0x2a0 [ 8.845776] Modules linked in: virtio_rng rng_core virtio_net net_failover failover sd_mod ata_generic ata_piix libata scsi_mod scsi_common virtio_blk virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev 9pnet_virtio virtio_ring virtio 9p 9pnet netfs [ 8.845989] CPU: 6 PID: 689 Comm: bpf_prog01 Tainted: G W 6.6.37-un-def-alt1 #1 [ 8.846072] Hardware name: IBM pSeries (emulated by qemu) POWER8 (raw) 0x4d0200 0xf000004 of:SLOF,git-3a259d hv:linux,kvm pSeries [ 8.846177] NIP: c0000000004fb4d8 LR: c0000000004fb4d4 CTR: 0000000000000000 [ 8.846248] REGS: c000000009b6f2c0 TRAP: 0700 Tainted: G W (6.6.37-un-def-alt1) [ 8.846330] MSR: 8000000000029033 <SF,EE,ME,IR,DR,RI,LE> CR: 28002222 XER: 00000000 [ 8.846408] CFAR: c00000000015df94 IRQMASK: 0 [ 8.846408] GPR00: 0000000000000000 c000000009b6f560 c000000001ac7f00 0000000000000000 [ 8.846408] GPR04: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 8.846408] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 8.846408] GPR12: 0000000000000000 c00000003fff7a00 0000000000000000 0000000000000000 [ 8.846408] GPR16: 0000000000000012 0000000000000000 000000000000008c 0000000000000000 [ 8.846408] GPR20: c008000000040a40 0000000000000002 c0000000022a7560 c008000000040a4c [ 8.846408] GPR24: c000000005716480 0000000000000000 c000000002155698 c0000000022a7680 [ 8.846408] GPR28: c000000002155688 0000000000000000 c008000000040a40 0000000000000000 [ 8.851030] NIP [c0000000004fb4d8] vfree+0x1d8/0x2a0 [ 8.851085] LR [c0000000004fb4d4] vfree+0x1d4/0x2a0 [ 8.851135] Call Trace: [ 8.851160] [c000000009b6f560] [c0000000004fb4d4] vfree+0x1d4/0x2a0 (unreliable) [ 8.851234] [c000000009b6f5e0] [c000000000269c6c] module_memfree+0x3c/0x60 [ 8.851297] [c000000009b6f600] [c00000000038cf60] bpf_jit_free_exec+0x20/0x40 [ 8.851371] [c000000009b6f620] [c00000000038f518] bpf_prog_pack_free+0x2f8/0x390 [ 8.851445] [c000000009b6f6b0] [c00000000038f878] bpf_jit_binary_pack_finalize+0x98/0xd0 [ 8.851529] [c000000009b6f6e0] [c000000000118240] bpf_int_jit_compile+0x2c0/0x710 [ 8.851602] [c000000009b6f830] [c00000000038ef64] bpf_prog_select_runtime+0x154/0x1b0 [ 8.851675] [c000000009b6f880] [c000000000398edc] bpf_prog_load+0x94c/0xe90 [ 8.851737] [c000000009b6f990] [c00000000039c878] __sys_bpf+0x418/0x2970 [ 8.851798] [c000000009b6fac0] [c00000000039f1a0] sys_bpf+0x30/0x50 [ 8.851860] [c000000009b6fae0] [c000000000030230] system_call_exception+0x190/0x390 [ 8.851934] [c000000009b6fe50] [c00000000000c7d4] system_call_common+0xf4/0x258 [ 8.852007] --- interrupt: c00 at 0x7fffb0839ad4 [ 8.852057] NIP: 00007fffb0839ad4 LR: 000000012a027fb4 CTR: 0000000000000000 [ 8.852128] REGS: c000000009b6fe80 TRAP: 0c00 Tainted: G W (6.6.37-un-def-alt1) [ 8.852212] MSR: 800000000280f033 <SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE> CR: 22002240 XER: 00000000 [ 8.852307] IRQMASK: 0 [ 8.852307] GPR00: 0000000000000169 00007fffd8534200 00007fffb0936d00 0000000000000005 [ 8.852307] GPR04: 00007fffb06aff90 0000000000000070 000000012a0538a0 0000000000000001 [ 8.852307] GPR08: 000000012a0801f4 0000000000000000 0000000000000000 0000000000000000 [ 8.852307] GPR12: 0000000000000000 00007fffb09ea540 0000000000000000 0000000000000000 [ 8.852307] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 8.852307] GPR20: 00007fffd85344b0 0000000000000000 0000000000000001 0000000000000000 [ 8.852307] GPR24: 000000012a0801f4 00007fffb06ce000 0000000000000000 00000000000f4240 [ 8.852307] GPR28: 00007fffb06aff90 00007fffb09e3550 0000000000000001 0000000000001118 [ 8.852889] NIP [00007fffb0839ad4] 0x7fffb0839ad4 [ 8.852938] LR [000000012a027fb4] 0x12a027fb4 [ 8.852986] --- interrupt: c00 [ 8.853022] Code: 4e800020 60420000 3949ffff 4bffff0c 38210080 ebe1fff8 4bfffd68 3c62ffa2 7fc4f378 3863e6f0 4bc629b9 60000000 <0fe00000> eba10068 4bffff8c 2c080000 [ 8.853164] ---[ end trace 0000000000000000 ]--- [ 8.856619] kernel tried to execute exec-protected page (c008000000040a4c) - exploit attempt? (uid: 0) [ 8.856717] BUG: Unable to handle kernel instruction fetch [ 8.856763] Faulting instruction address: 0xc008000000040a4c [ 8.856825] Oops: Kernel access of bad area, sig: 11 [#1] [ 8.856875] LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries [ 8.856937] Modules linked in: virtio_rng rng_core virtio_net net_failover failover sd_mod ata_generic ata_piix libata scsi_mod scsi_common virtio_blk virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev 9pnet_virtio virtio_ring virtio 9p 9pnet netfs [ 8.857154] CPU: 6 PID: 689 Comm: bpf_prog01 Tainted: G W 6.6.37-un-def-alt1 #1 [ 8.857236] Hardware name: IBM pSeries (emulated by qemu) POWER8 (raw) 0x4d0200 0xf000004 of:SLOF,git-3a259d hv:linux,kvm pSeries [ 8.857342] NIP: c008000000040a4c LR: c000000000ed25d0 CTR: c008000000040a4c [ 8.857413] REGS: c000000009b6f6f0 TRAP: 0400 Tainted: G W (6.6.37-un-def-alt1) [ 8.857510] MSR: 8000000010009033 <SF,EE,ME,IR,DR,RI,LE> CR: 28008286 XER: 00000000 [ 8.857588] CFAR: c000000000ed25cc IRQMASK: 0 [ 8.857588] GPR00: c000000000ed25a8 c000000009b6f990 c000000001ac7f00 c000000006130400 [ 8.857588] GPR04: c008000000920048 0000000000000001 0000000000000000 0000000000000000 [ 8.857588] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 8.857588] GPR12: c008000000040a4c c00000003fff7a00 0000000000000000 0000000000000000 [ 8.857588] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 8.857588] GPR20: 7fffffffffffffff 0000000000000000 0000000000000001 0000000000000000 [ 8.857588] GPR24: c000000006130400 c000000006510a00 c000000006510f00 c0000000041a9000 [ 8.857588] GPR28: 0000000000000001 c000000006130400 0000000000000000 c008000000920000 [ 8.858184] NIP [c008000000040a4c] bpf_prog_2fb4fda3a3499517+0x0/0x8c [ 8.858245] LR [c000000000ed25d0] sk_filter_trim_cap+0xc0/0x370 [ 8.858308] Call Trace: [ 8.858333] [c000000009b6f990] [c000000000ed2574] sk_filter_trim_cap+0x64/0x370 (unreliable) [ 8.858421] [c000000009b6fa10] [c000000001068b64] unix_dgram_sendmsg+0x214/0xb10 [ 8.858511] [c000000009b6fad0] [c000000000e4c59c] sock_write_iter+0x19c/0x1e0 [ 8.858586] [c000000009b6fb80] [c0000000005b1b58] vfs_write+0x258/0x4e0 [ 8.858648] [c000000009b6fc40] [c0000000005b21d4] ksys_write+0x114/0x170 [ 8.858711] [c000000009b6fc90] [c000000000030230] system_call_exception+0x190/0x390 [ 8.858785] [c000000009b6fe50] [c00000000000c7d4] system_call_common+0xf4/0x258 [ 8.858859] --- interrupt: c00 at 0x7fffb082b884 [ 8.858908] NIP: 00007fffb082b884 LR: 000000012a02ab70 CTR: 0000000000000000 [ 8.858979] REGS: c000000009b6fe80 TRAP: 0c00 Tainted: G W (6.6.37-un-def-alt1) [ 8.859060] MSR: 800000000280f033 <SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE> CR: 28002281 XER: 00000000 [ 8.859153] IRQMASK: 0 [ 8.859153] GPR00: 0000000000000004 00007fffd85341f0 00007fffb0936d00 0000000000000005 [ 8.859153] GPR04: 00007fffb068fffa 0000000000000006 0000000000000001 0000000000000005 [ 8.859153] GPR08: 00007fffb068fffa 0000000000000000 0000000000000000 0000000000000000 [ 8.859153] GPR12: 0000000000000000 00007fffb09ea540 0000000000000000 0000000000000000 [ 8.859153] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 8.859153] GPR20: 00007fffd85344b0 0000000000000000 0000000000000001 0000000000000000 [ 8.859153] GPR24: 000000012a053698 000000000000008b 0000000000000000 0000000000000001 [ 8.859153] GPR28: 00007fffb068fffa 0000000000000005 0000000000000006 000000012a053698 [ 8.859738] NIP [00007fffb082b884] 0x7fffb082b884 [ 8.859786] LR [000000012a02ab70] 0x12a02ab70 [ 8.859836] --- interrupt: c00 [ 8.859872] Code: 7fe00008 7fe00008 7fe00008 7fe00008 7fe00008 7fe00008 7fe00008 7fe00008 7fe00008 7fe00008 7fe00008 7fe00008 <7fe00008> 7fe00008 7fe00008 7fe00008 [ 8.860013] ---[ end trace 0000000000000000 ]--- [ 8.863088] pstore: backend (nvram) writing error (-1) [ 8.863141] [ 8.863166] note: bpf_prog01[689] exited with irqs disabled
And so on. Temporary build/test log is at https://git.altlinux.org/tasks/352218/build/100/ppc64le/log
Other stable/longterm branches or other architectures does not exhibit this.
Thanks,
Hi all - this just took down a production server for me, on POWER9 bare metal. Not running tests, just booting normally, before services even came up. Had to perform manual restoration, reverting to 6.6.36 worked. Also running 64k kernel, unsure if it's better on 4k kernel.
In case it's helpful, here's the log from my boot: https://dpaste.org/Gyxxg/raw
On Mon, Jul 08, 2024 at 11:16:48PM -0400, matoro wrote:
On 2024-07-05 16:34, Vitaly Chikunov wrote:
Hi,
There is new WARNING and Oops on ppc64le in v6.6.37 when running LTP tests: bpf_prog01, bpf_prog02, bpf_prog04, bpf_prog05, prctl04. Logs excerpt below. I see there is 1 commit in v6.6.36..v6.6.37 with call to bpf_jit_binary_pack_finalize, backported from 5 patch mainline patchset:
f99feda5684a powerpc/bpf: use bpf_jit_binary_pack_[alloc|finalize|free]
Log:
[ 8.822079] LTP: starting bpf_prog01 [ 8.841853] ------------[ cut here ]------------ [ 8.841946] Trying to vfree() bad address (00000000453be747) [ 8.842024] WARNING: CPU: 6 PID: 689 at mm/vmalloc.c:2700 remove_vm_area+0xb4/0xf0 [ 8.842103] Modules linked in: virtio_rng rng_core virtio_net net_failover failover sd_mod ata_generic ata_piix libata scsi_mod scsi_common virtio_blk virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev 9pnet_virtio virtio_ring virtio 9p 9pnet netfs [ 8.842323] CPU: 6 PID: 689 Comm: bpf_prog01 Not tainted 6.6.37-un-def-alt1 #1 [ 8.842396] Hardware name: IBM pSeries (emulated by qemu) POWER8 (raw) 0x4d0200 0xf000004 of:SLOF,git-3a259d hv:linux,kvm pSeries [ 8.842519] NIP: c0000000004faf04 LR: c0000000004faf00 CTR: 0000000000000000 [ 8.842598] REGS: c000000009b6f250 TRAP: 0700 Not tainted (6.6.37-un-def-alt1) [ 8.842669] MSR: 8000000000029033 <SF,EE,ME,IR,DR,RI,LE> CR: 28002822 XER: 00000000 [ 8.842748] CFAR: c00000000015df94 IRQMASK: 0 [ 8.842748] GPR00: 0000000000000000 c000000009b6f4f0 c000000001ac7f00 0000000000000000 [ 8.842748] GPR04: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 8.842748] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 8.842748] GPR12: 0000000000000000 c00000003fff7a00 0000000000000000 0000000000000000 [ 8.842748] GPR16: 0000000000000012 0000000000000000 000000000000008c 0000000000000000 [ 8.842748] GPR20: c008000000040a40 0000000000000002 c0000000022a7560 c008000000040a4c [ 8.842748] GPR24: c000000005716480 0000000000000000 c000000002155698 c0000000022a7680 [ 8.842748] GPR28: c000000002155688 c008000000040a40 c008000000040a40 c008000000040a40 [ 8.843347] NIP [c0000000004faf04] remove_vm_area+0xb4/0xf0 [ 8.843398] LR [c0000000004faf00] remove_vm_area+0xb0/0xf0 [ 8.843448] Call Trace: [ 8.843484] [c000000009b6f4f0] [c0000000004faf00] remove_vm_area+0xb0/0xf0 (unreliable) [ 8.843559] [c000000009b6f560] [c0000000004fb360] vfree+0x60/0x2a0 [ 8.843621] [c000000009b6f5e0] [c000000000269c6c] module_memfree+0x3c/0x60 [ 8.843685] [c000000009b6f600] [c00000000038cf60] bpf_jit_free_exec+0x20/0x40 [ 8.843759] [c000000009b6f620] [c00000000038f518] bpf_prog_pack_free+0x2f8/0x390 [ 8.843832] [c000000009b6f6b0] [c00000000038f878] bpf_jit_binary_pack_finalize+0x98/0xd0 [ 8.843906] [c000000009b6f6e0] [c000000000118240] bpf_int_jit_compile+0x2c0/0x710 [ 8.843979] [c000000009b6f830] [c00000000038ef64] bpf_prog_select_runtime+0x154/0x1b0 [ 8.844053] [c000000009b6f880] [c000000000398edc] bpf_prog_load+0x94c/0xe90 [ 8.844114] [c000000009b6f990] [c00000000039c878] __sys_bpf+0x418/0x2970 [ 8.844176] [c000000009b6fac0] [c00000000039f1a0] sys_bpf+0x30/0x50 [ 8.844237] [c000000009b6fae0] [c000000000030230] system_call_exception+0x190/0x390 [ 8.844312] [c000000009b6fe50] [c00000000000c7d4] system_call_common+0xf4/0x258 [ 8.844386] --- interrupt: c00 at 0x7fffb0839ad4 [ 8.844437] NIP: 00007fffb0839ad4 LR: 000000012a027fb4 CTR: 0000000000000000 [ 8.844524] REGS: c000000009b6fe80 TRAP: 0c00 Not tainted (6.6.37-un-def-alt1) [ 8.844596] MSR: 800000000280f033 <SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE> CR: 22002240 XER: 00000000 [ 8.844690] IRQMASK: 0 [ 8.844690] GPR00: 0000000000000169 00007fffd8534200 00007fffb0936d00 0000000000000005 [ 8.844690] GPR04: 00007fffb06aff90 0000000000000070 000000012a0538a0 0000000000000001 [ 8.844690] GPR08: 000000012a0801f4 0000000000000000 0000000000000000 0000000000000000 [ 8.844690] GPR12: 0000000000000000 00007fffb09ea540 0000000000000000 0000000000000000 [ 8.844690] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 8.844690] GPR20: 00007fffd85344b0 0000000000000000 0000000000000001 0000000000000000 [ 8.844690] GPR24: 000000012a0801f4 00007fffb06ce000 0000000000000000 00000000000f4240 [ 8.844690] GPR28: 00007fffb06aff90 00007fffb09e3550 0000000000000001 0000000000001118 [ 8.845267] NIP [00007fffb0839ad4] 0x7fffb0839ad4 [ 8.845315] LR [000000012a027fb4] 0x12a027fb4 [ 8.845363] --- interrupt: c00 [ 8.845399] Code: 38000000 38800000 39200000 4e800020 60000000 60000000 60420000 3c62ffa2 7fe4fb78 3863e698 4bc62f8d 60000000 <0fe00000> 38210070 3bc00000 e8010010 [ 8.845550] ---[ end trace 0000000000000000 ]--- [ 8.845603] ------------[ cut here ]------------ [ 8.845651] Trying to vfree() nonexistent vm area (00000000453be747) [ 8.845714] WARNING: CPU: 6 PID: 689 at mm/vmalloc.c:2835 vfree+0x1d8/0x2a0 [ 8.845776] Modules linked in: virtio_rng rng_core virtio_net net_failover failover sd_mod ata_generic ata_piix libata scsi_mod scsi_common virtio_blk virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev 9pnet_virtio virtio_ring virtio 9p 9pnet netfs [ 8.845989] CPU: 6 PID: 689 Comm: bpf_prog01 Tainted: G W 6.6.37-un-def-alt1 #1 [ 8.846072] Hardware name: IBM pSeries (emulated by qemu) POWER8 (raw) 0x4d0200 0xf000004 of:SLOF,git-3a259d hv:linux,kvm pSeries [ 8.846177] NIP: c0000000004fb4d8 LR: c0000000004fb4d4 CTR: 0000000000000000 [ 8.846248] REGS: c000000009b6f2c0 TRAP: 0700 Tainted: G W (6.6.37-un-def-alt1) [ 8.846330] MSR: 8000000000029033 <SF,EE,ME,IR,DR,RI,LE> CR: 28002222 XER: 00000000 [ 8.846408] CFAR: c00000000015df94 IRQMASK: 0 [ 8.846408] GPR00: 0000000000000000 c000000009b6f560 c000000001ac7f00 0000000000000000 [ 8.846408] GPR04: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 8.846408] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 8.846408] GPR12: 0000000000000000 c00000003fff7a00 0000000000000000 0000000000000000 [ 8.846408] GPR16: 0000000000000012 0000000000000000 000000000000008c 0000000000000000 [ 8.846408] GPR20: c008000000040a40 0000000000000002 c0000000022a7560 c008000000040a4c [ 8.846408] GPR24: c000000005716480 0000000000000000 c000000002155698 c0000000022a7680 [ 8.846408] GPR28: c000000002155688 0000000000000000 c008000000040a40 0000000000000000 [ 8.851030] NIP [c0000000004fb4d8] vfree+0x1d8/0x2a0 [ 8.851085] LR [c0000000004fb4d4] vfree+0x1d4/0x2a0 [ 8.851135] Call Trace: [ 8.851160] [c000000009b6f560] [c0000000004fb4d4] vfree+0x1d4/0x2a0 (unreliable) [ 8.851234] [c000000009b6f5e0] [c000000000269c6c] module_memfree+0x3c/0x60 [ 8.851297] [c000000009b6f600] [c00000000038cf60] bpf_jit_free_exec+0x20/0x40 [ 8.851371] [c000000009b6f620] [c00000000038f518] bpf_prog_pack_free+0x2f8/0x390 [ 8.851445] [c000000009b6f6b0] [c00000000038f878] bpf_jit_binary_pack_finalize+0x98/0xd0 [ 8.851529] [c000000009b6f6e0] [c000000000118240] bpf_int_jit_compile+0x2c0/0x710 [ 8.851602] [c000000009b6f830] [c00000000038ef64] bpf_prog_select_runtime+0x154/0x1b0 [ 8.851675] [c000000009b6f880] [c000000000398edc] bpf_prog_load+0x94c/0xe90 [ 8.851737] [c000000009b6f990] [c00000000039c878] __sys_bpf+0x418/0x2970 [ 8.851798] [c000000009b6fac0] [c00000000039f1a0] sys_bpf+0x30/0x50 [ 8.851860] [c000000009b6fae0] [c000000000030230] system_call_exception+0x190/0x390 [ 8.851934] [c000000009b6fe50] [c00000000000c7d4] system_call_common+0xf4/0x258 [ 8.852007] --- interrupt: c00 at 0x7fffb0839ad4 [ 8.852057] NIP: 00007fffb0839ad4 LR: 000000012a027fb4 CTR: 0000000000000000 [ 8.852128] REGS: c000000009b6fe80 TRAP: 0c00 Tainted: G W (6.6.37-un-def-alt1) [ 8.852212] MSR: 800000000280f033 <SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE> CR: 22002240 XER: 00000000 [ 8.852307] IRQMASK: 0 [ 8.852307] GPR00: 0000000000000169 00007fffd8534200 00007fffb0936d00 0000000000000005 [ 8.852307] GPR04: 00007fffb06aff90 0000000000000070 000000012a0538a0 0000000000000001 [ 8.852307] GPR08: 000000012a0801f4 0000000000000000 0000000000000000 0000000000000000 [ 8.852307] GPR12: 0000000000000000 00007fffb09ea540 0000000000000000 0000000000000000 [ 8.852307] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 8.852307] GPR20: 00007fffd85344b0 0000000000000000 0000000000000001 0000000000000000 [ 8.852307] GPR24: 000000012a0801f4 00007fffb06ce000 0000000000000000 00000000000f4240 [ 8.852307] GPR28: 00007fffb06aff90 00007fffb09e3550 0000000000000001 0000000000001118 [ 8.852889] NIP [00007fffb0839ad4] 0x7fffb0839ad4 [ 8.852938] LR [000000012a027fb4] 0x12a027fb4 [ 8.852986] --- interrupt: c00 [ 8.853022] Code: 4e800020 60420000 3949ffff 4bffff0c 38210080 ebe1fff8 4bfffd68 3c62ffa2 7fc4f378 3863e6f0 4bc629b9 60000000 <0fe00000> eba10068 4bffff8c 2c080000 [ 8.853164] ---[ end trace 0000000000000000 ]--- [ 8.856619] kernel tried to execute exec-protected page (c008000000040a4c) - exploit attempt? (uid: 0) [ 8.856717] BUG: Unable to handle kernel instruction fetch [ 8.856763] Faulting instruction address: 0xc008000000040a4c [ 8.856825] Oops: Kernel access of bad area, sig: 11 [#1] [ 8.856875] LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries [ 8.856937] Modules linked in: virtio_rng rng_core virtio_net net_failover failover sd_mod ata_generic ata_piix libata scsi_mod scsi_common virtio_blk virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev 9pnet_virtio virtio_ring virtio 9p 9pnet netfs [ 8.857154] CPU: 6 PID: 689 Comm: bpf_prog01 Tainted: G W 6.6.37-un-def-alt1 #1 [ 8.857236] Hardware name: IBM pSeries (emulated by qemu) POWER8 (raw) 0x4d0200 0xf000004 of:SLOF,git-3a259d hv:linux,kvm pSeries [ 8.857342] NIP: c008000000040a4c LR: c000000000ed25d0 CTR: c008000000040a4c [ 8.857413] REGS: c000000009b6f6f0 TRAP: 0400 Tainted: G W (6.6.37-un-def-alt1) [ 8.857510] MSR: 8000000010009033 <SF,EE,ME,IR,DR,RI,LE> CR: 28008286 XER: 00000000 [ 8.857588] CFAR: c000000000ed25cc IRQMASK: 0 [ 8.857588] GPR00: c000000000ed25a8 c000000009b6f990 c000000001ac7f00 c000000006130400 [ 8.857588] GPR04: c008000000920048 0000000000000001 0000000000000000 0000000000000000 [ 8.857588] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 8.857588] GPR12: c008000000040a4c c00000003fff7a00 0000000000000000 0000000000000000 [ 8.857588] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 8.857588] GPR20: 7fffffffffffffff 0000000000000000 0000000000000001 0000000000000000 [ 8.857588] GPR24: c000000006130400 c000000006510a00 c000000006510f00 c0000000041a9000 [ 8.857588] GPR28: 0000000000000001 c000000006130400 0000000000000000 c008000000920000 [ 8.858184] NIP [c008000000040a4c] bpf_prog_2fb4fda3a3499517+0x0/0x8c [ 8.858245] LR [c000000000ed25d0] sk_filter_trim_cap+0xc0/0x370 [ 8.858308] Call Trace: [ 8.858333] [c000000009b6f990] [c000000000ed2574] sk_filter_trim_cap+0x64/0x370 (unreliable) [ 8.858421] [c000000009b6fa10] [c000000001068b64] unix_dgram_sendmsg+0x214/0xb10 [ 8.858511] [c000000009b6fad0] [c000000000e4c59c] sock_write_iter+0x19c/0x1e0 [ 8.858586] [c000000009b6fb80] [c0000000005b1b58] vfs_write+0x258/0x4e0 [ 8.858648] [c000000009b6fc40] [c0000000005b21d4] ksys_write+0x114/0x170 [ 8.858711] [c000000009b6fc90] [c000000000030230] system_call_exception+0x190/0x390 [ 8.858785] [c000000009b6fe50] [c00000000000c7d4] system_call_common+0xf4/0x258 [ 8.858859] --- interrupt: c00 at 0x7fffb082b884 [ 8.858908] NIP: 00007fffb082b884 LR: 000000012a02ab70 CTR: 0000000000000000 [ 8.858979] REGS: c000000009b6fe80 TRAP: 0c00 Tainted: G W (6.6.37-un-def-alt1) [ 8.859060] MSR: 800000000280f033 <SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE> CR: 28002281 XER: 00000000 [ 8.859153] IRQMASK: 0 [ 8.859153] GPR00: 0000000000000004 00007fffd85341f0 00007fffb0936d00 0000000000000005 [ 8.859153] GPR04: 00007fffb068fffa 0000000000000006 0000000000000001 0000000000000005 [ 8.859153] GPR08: 00007fffb068fffa 0000000000000000 0000000000000000 0000000000000000 [ 8.859153] GPR12: 0000000000000000 00007fffb09ea540 0000000000000000 0000000000000000 [ 8.859153] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 8.859153] GPR20: 00007fffd85344b0 0000000000000000 0000000000000001 0000000000000000 [ 8.859153] GPR24: 000000012a053698 000000000000008b 0000000000000000 0000000000000001 [ 8.859153] GPR28: 00007fffb068fffa 0000000000000005 0000000000000006 000000012a053698 [ 8.859738] NIP [00007fffb082b884] 0x7fffb082b884 [ 8.859786] LR [000000012a02ab70] 0x12a02ab70 [ 8.859836] --- interrupt: c00 [ 8.859872] Code: 7fe00008 7fe00008 7fe00008 7fe00008 7fe00008 7fe00008 7fe00008 7fe00008 7fe00008 7fe00008 7fe00008 7fe00008 <7fe00008> 7fe00008 7fe00008 7fe00008 [ 8.860013] ---[ end trace 0000000000000000 ]--- [ 8.863088] pstore: backend (nvram) writing error (-1) [ 8.863141] [ 8.863166] note: bpf_prog01[689] exited with irqs disabled
And so on. Temporary build/test log is at https://git.altlinux.org/tasks/352218/build/100/ppc64le/log
Other stable/longterm branches or other architectures does not exhibit this.
Thanks,
Hi all - this just took down a production server for me, on POWER9 bare metal. Not running tests, just booting normally, before services even came up. Had to perform manual restoration, reverting to 6.6.36 worked. Also running 64k kernel, unsure if it's better on 4k kernel.
In case it's helpful, here's the log from my boot: https://dpaste.org/Gyxxg/raw
Ok, this isn't good, something went wrong with my backports here. Let me go revert them all and push out a new 6.6.y release right away.
thanks for the report!
greg k-h
Greg Kroah-Hartman wrote:
On Mon, Jul 08, 2024 at 11:16:48PM -0400, matoro wrote:
On 2024-07-05 16:34, Vitaly Chikunov wrote:
Hi,
There is new WARNING and Oops on ppc64le in v6.6.37 when running LTP tests: bpf_prog01, bpf_prog02, bpf_prog04, bpf_prog05, prctl04. Logs excerpt below. I see there is 1 commit in v6.6.36..v6.6.37 with call to bpf_jit_binary_pack_finalize, backported from 5 patch mainline patchset:
f99feda5684a powerpc/bpf: use bpf_jit_binary_pack_[alloc|finalize|free]
<snip>
And so on. Temporary build/test log is at https://git.altlinux.org/tasks/352218/build/100/ppc64le/log
Other stable/longterm branches or other architectures does not exhibit this.
Thanks,
Hi all - this just took down a production server for me, on POWER9 bare metal. Not running tests, just booting normally, before services even came up. Had to perform manual restoration, reverting to 6.6.36 worked. Also running 64k kernel, unsure if it's better on 4k kernel.
In case it's helpful, here's the log from my boot: https://dpaste.org/Gyxxg/raw
Ok, this isn't good, something went wrong with my backports here. Let me go revert them all and push out a new 6.6.y release right away.
I think the problem is that the series adding support for bpf prog_pack was partially backported. In particular, the below patches are missing from stable v6.6: 465cabc97b42 powerpc/code-patching: introduce patch_instructions() 033ffaf0af1f powerpc/bpf: implement bpf_arch_text_invalidate for bpf_prog_pack 6efc1675acb8 powerpc/bpf: implement bpf_arch_text_copy
It should be sufficient to revert commit f99feda5684a (powerpc/bpf: use bpf_jit_binary_pack_[alloc|finalize|free]) to allow the above to apply cleanly, followed by cherry picking commit 90d862f370b6 (powerpc/bpf: use bpf_jit_binary_pack_[alloc|finalize|free]) from upstream.
Alternately, commit f99feda5684a (powerpc/bpf: use bpf_jit_binary_pack_[alloc|finalize|free]) can be reverted.
- Naveen
On Tue, Jul 09, 2024 at 03:02:13PM +0530, Naveen N Rao wrote:
Greg Kroah-Hartman wrote:
On Mon, Jul 08, 2024 at 11:16:48PM -0400, matoro wrote:
On 2024-07-05 16:34, Vitaly Chikunov wrote:
Hi,
There is new WARNING and Oops on ppc64le in v6.6.37 when running
LTP tests:
bpf_prog01, bpf_prog02, bpf_prog04, bpf_prog05, prctl04. Logs excerpt below. I see there is 1 commit in v6.6.36..v6.6.37 with call to bpf_jit_binary_pack_finalize, backported from 5 patch mainline patchset:
f99feda5684a powerpc/bpf: use
bpf_jit_binary_pack_[alloc|finalize|free]
<snip>
And so on. Temporary build/test log is at
https://git.altlinux.org/tasks/352218/build/100/ppc64le/log
Other stable/longterm branches or other architectures does not
exhibit this.
Thanks,
Hi all - this just took down a production server for me, on POWER9 bare metal. Not running tests, just booting normally, before services even came up. Had to perform manual restoration, reverting to 6.6.36 worked. Also running 64k kernel, unsure if it's better on 4k kernel.
In case it's helpful, here's the log from my boot: https://dpaste.org/Gyxxg/raw
Ok, this isn't good, something went wrong with my backports here. Let me go revert them all and push out a new 6.6.y release right away.
I think the problem is that the series adding support for bpf prog_pack was partially backported. In particular, the below patches are missing from stable v6.6: 465cabc97b42 powerpc/code-patching: introduce patch_instructions() 033ffaf0af1f powerpc/bpf: implement bpf_arch_text_invalidate for bpf_prog_pack 6efc1675acb8 powerpc/bpf: implement bpf_arch_text_copy
It should be sufficient to revert commit f99feda5684a (powerpc/bpf: use bpf_jit_binary_pack_[alloc|finalize|free]) to allow the above to apply cleanly, followed by cherry picking commit 90d862f370b6 (powerpc/bpf: use bpf_jit_binary_pack_[alloc|finalize|free]) from upstream.
Alternately, commit f99feda5684a (powerpc/bpf: use bpf_jit_binary_pack_[alloc|finalize|free]) can be reverted.
I'm dropping them all now, if you want to submit a working series for this, I'll be glad to queue them all up.
thanks,
greg k-h
Greg Kroah-Hartman gregkh@linuxfoundation.org writes:
On Tue, Jul 09, 2024 at 03:02:13PM +0530, Naveen N Rao wrote:
Greg Kroah-Hartman wrote:
On Mon, Jul 08, 2024 at 11:16:48PM -0400, matoro wrote:
On 2024-07-05 16:34, Vitaly Chikunov wrote:
Hi,
There is new WARNING and Oops on ppc64le in v6.6.37 when running
LTP tests:
bpf_prog01, bpf_prog02, bpf_prog04, bpf_prog05, prctl04. Logs excerpt below. I see there is 1 commit in v6.6.36..v6.6.37 with call to bpf_jit_binary_pack_finalize, backported from 5 patch mainline patchset:
f99feda5684a powerpc/bpf: use
bpf_jit_binary_pack_[alloc|finalize|free]
<snip>
And so on. Temporary build/test log is at
https://git.altlinux.org/tasks/352218/build/100/ppc64le/log
Other stable/longterm branches or other architectures does not
exhibit this.
Thanks,
Hi all - this just took down a production server for me, on POWER9 bare metal. Not running tests, just booting normally, before services even came up. Had to perform manual restoration, reverting to 6.6.36 worked. Also running 64k kernel, unsure if it's better on 4k kernel.
In case it's helpful, here's the log from my boot: https://dpaste.org/Gyxxg/raw
Ok, this isn't good, something went wrong with my backports here. Let me go revert them all and push out a new 6.6.y release right away.
I think the problem is that the series adding support for bpf prog_pack was partially backported. In particular, the below patches are missing from stable v6.6: 465cabc97b42 powerpc/code-patching: introduce patch_instructions() 033ffaf0af1f powerpc/bpf: implement bpf_arch_text_invalidate for bpf_prog_pack 6efc1675acb8 powerpc/bpf: implement bpf_arch_text_copy
It should be sufficient to revert commit f99feda5684a (powerpc/bpf: use bpf_jit_binary_pack_[alloc|finalize|free]) to allow the above to apply cleanly, followed by cherry picking commit 90d862f370b6 (powerpc/bpf: use bpf_jit_binary_pack_[alloc|finalize|free]) from upstream.
Alternately, commit f99feda5684a (powerpc/bpf: use bpf_jit_binary_pack_[alloc|finalize|free]) can be reverted.
I'm dropping them all now, if you want to submit a working series for this, I'll be glad to queue them all up.
Thanks, revert is good for now.
With the revert there will be a build warning/error, only in stable, which I think can be fixed with the diff below. I'll get it tested and submit it properly.
cheers
diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c index 5f57a8ba3cc8..cdd9db8f8684 100644 --- a/arch/powerpc/net/bpf_jit_comp.c +++ b/arch/powerpc/net/bpf_jit_comp.c @@ -205,7 +205,10 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
bpf_flush_icache(bpf_hdr, (u8 *)bpf_hdr + bpf_hdr->size); if (!fp->is_func || extra_pass) { - bpf_jit_binary_lock_ro(bpf_hdr); + if (bpf_jit_binary_lock_ro(bpf_hdr)) { + fp = org_fp; + goto out_addrs; + } bpf_prog_fill_jited_linfo(fp, addrs); out_addrs: kfree(addrs);
Michael Ellerman mpe@ellerman.id.au writes:
Greg Kroah-Hartman gregkh@linuxfoundation.org writes:
On Tue, Jul 09, 2024 at 03:02:13PM +0530, Naveen N Rao wrote:
Greg Kroah-Hartman wrote:
On Mon, Jul 08, 2024 at 11:16:48PM -0400, matoro wrote:
On 2024-07-05 16:34, Vitaly Chikunov wrote:
Hi, > There is new WARNING and Oops on ppc64le in v6.6.37 when running
LTP tests:
bpf_prog01, bpf_prog02, bpf_prog04, bpf_prog05, prctl04. Logs excerpt below. I see there is 1 commit in v6.6.36..v6.6.37 with call to bpf_jit_binary_pack_finalize, backported from 5 patch mainline patchset: > f99feda5684a powerpc/bpf: use
bpf_jit_binary_pack_[alloc|finalize|free]
<snip>
> And so on. Temporary build/test log is at https://git.altlinux.org/tasks/352218/build/100/ppc64le/log > Other stable/longterm branches or other architectures does not
exhibit this.
> Thanks,
Hi all - this just took down a production server for me, on POWER9 bare metal. Not running tests, just booting normally, before services even came up. Had to perform manual restoration, reverting to 6.6.36 worked. Also running 64k kernel, unsure if it's better on 4k kernel.
In case it's helpful, here's the log from my boot: https://dpaste.org/Gyxxg/raw
Ok, this isn't good, something went wrong with my backports here. Let me go revert them all and push out a new 6.6.y release right away.
I think the problem is that the series adding support for bpf prog_pack was partially backported. In particular, the below patches are missing from stable v6.6: 465cabc97b42 powerpc/code-patching: introduce patch_instructions() 033ffaf0af1f powerpc/bpf: implement bpf_arch_text_invalidate for bpf_prog_pack 6efc1675acb8 powerpc/bpf: implement bpf_arch_text_copy
It should be sufficient to revert commit f99feda5684a (powerpc/bpf: use bpf_jit_binary_pack_[alloc|finalize|free]) to allow the above to apply cleanly, followed by cherry picking commit 90d862f370b6 (powerpc/bpf: use bpf_jit_binary_pack_[alloc|finalize|free]) from upstream.
Alternately, commit f99feda5684a (powerpc/bpf: use bpf_jit_binary_pack_[alloc|finalize|free]) can be reverted.
I'm dropping them all now, if you want to submit a working series for this, I'll be glad to queue them all up.
Thanks, revert is good for now.
With the revert there will be a build warning/error, only in stable, which I think can be fixed with the diff below.
Oh I see you also reverted the commit that introduces that warning, so the build should be OK now.
cheers
linux-stable-mirror@lists.linaro.org
-
Greg Kroah-Hartman -
matoro -
Michael Ellerman -
Naveen N Rao -
Vitaly Chikunov