#regzbot introduced: 6.14.2..6.14.3
Since 6.14.3 I have been seeing random panics, all in htb_dequeue. 6.14.2 was fine.
One only happened 8 hours after the reboot, so bisecting would be prolonged, since I have no idea what is triggering the crash.
I've captured three panics with netconsole. All are very similar.
I've also included the script I use to initialise tc. As well as when the ppp over ethernet interface comes up, I also run this every 5 minutes as a cron job since my ADSL line can fluctuate between 22 and 30 Mb/s. However, from the timings of the last few lines in /var/log/messages before the crash, it doesn't seem that this is directly related.
Finally, I've decoded the first panic.
I'm more than happy to help with debugging, if necessary.
The system is running up-to-date Gentoo.
Linux version 6.14.3 (alan@bilbo) (gcc (Gentoo Hardened 14.2.1_p20241221 p7) 14.2.1 20241221, GNU ld (Gentoo 2.44 p1) 2.44.0) #20 SMP PREEMPT_DYNAMIC Sun Apr 20 21:18:54 BST 2025
Linux bilbo 6.14.3 #20 SMP PREEMPT_DYNAMIC Sun Apr 20 21:18:54 BST 2025 x86_64 AMD FX(tm)-4300 Quad-Core Processor AuthenticAMD GNU/Linux
# equery -q list iproute2 sys-apps/iproute2-6.13.0
BUG: kernel NULL pointer dereference, address: 0000000000000000 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 0 P4D 0 Oops: Oops: 0000 [#1] PREEMPT SMP NOPTI CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G O 6.14.3 #20 Tainted: [O]=OOT_MODULE Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./970A-DS3P, BIOS FD 02/26/2016 RIP: 0010:rb_next+0x0/0x50 Code: e8 d5 fa ff ff 5b 4c 89 e0 5d 41 5c 41 5d 41 5e e9 85 73 01 00 5b 5d 41 5c 41 5d 41 5e e9 38 76 01 00 0f 1f 84 00 00 00 00 00 <48> 3b 3f 48 89 f8 74 38 48 8b 57 08 48 85 d2 74 11 48 89 d0 48 8b RSP: 0018:ffffc90000003e50 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff88811a764000 RCX: ffff88811a764180 RDX: ffff88835e4c6c00 RSI: ffff888162c998e8 RDI: 0000000000000000 RBP: 0000000000000000 R08: ffff88811a7642b0 R09: 00000000a535eebc R10: 0000000000000d09 R11: ffffc90000003ff8 R12: ffff88835e4c6c00 R13: ffff88811a7642b8 R14: 00001a951355b383 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88842ec00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 00000001084b4000 CR4: 00000000000406f0 Call Trace: <IRQ> htb_dequeue+0x42f/0x610 [sch_htb] __qdisc_run+0x253/0x480 ? timerqueue_del+0x2c/0x40 qdisc_run+0x15/0x30 net_tx_action+0x182/0x1b0 handle_softirqs+0x102/0x240 __irq_exit_rcu+0x3e/0xb0 sysvec_apic_timer_interrupt+0x5b/0x70 </IRQ> <TASK> asm_sysvec_apic_timer_interrupt+0x16/0x20 RIP: 0010:cpuidle_enter_state+0x126/0x220 Code: 18 4c 6f 00 85 c0 7e 0b 8b 73 04 83 cf ff e8 a1 22 e5 ff 31 ff e8 9a 2e 98 ff 45 84 ff 74 07 31 ff e8 0e 58 9d ff fb 45 85 ed <0f> 88 cc 00 00 00 49 63 c5 48 8b 3c 24 48 6b c8 68 48 6b d0 30 49 RSP: 0018:ffffffff81e03e40 EFLAGS: 00000202 RAX: ffff88842ec00000 RBX: ffff8881008d9400 RCX: 0000000000000000 RDX: 00001a94d9502071 RSI: fffffffbb3498394 RDI: 0000000000000000 RBP: 0000000000000002 R08: 0000000000000002 R09: 00001a94d7bd7640 R10: 0000000000000006 R11: 0000000000000020 R12: ffffffff81f98280 R13: 0000000000000002 R14: 00001a94d9502071 R15: 0000000000000000 cpuidle_enter+0x2a/0x40 do_idle+0x12d/0x1a0 cpu_startup_entry+0x29/0x30 rest_init+0xbc/0xc0 start_kernel+0x630/0x630 x86_64_start_reservations+0x25/0x30 x86_64_start_kernel+0x73/0x80 common_startup_64+0x12c/0x138 </TASK> Modules linked in: udp_diag netconsole sch_htb cls_u32 sch_ingress sch_cake ifb act_mirred xt_hl xt_nat ts_bm xt_string xt_TARPIT(O) xt_CT xt_tcpudp xt_helper nf_nat_ftp nf_conntrack_ftp ip6t_rt ip6table_nat xt_MASQUERADE iptable_nat nf_nat xt_TCPMSS xt_LOG nf_log_syslog ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 ip6table_raw iptable_raw ip6table_mangle iptable_mangle xt_multiport xt_state xt_limit xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6table_filter ip6_tables iptable_filter ip_tables x_tables tun pppoe binfmt_misc pppox ppp_generic slhc af_packet bridge stp llc ctr ccm dm_crypt radeon drm_client_lib video wmi drm_exec drm_suballoc_helper ath9k drm_ttm_helper syscopyarea ttm sysfillrect ath9k_common sysimgblt ath9k_hw fb_sys_fops drm_display_helper drm_kms_helper ath mac80211 agpgart pl2303 snd_hda_codec_realtek cfbfillrect snd_hda_codec_generic usbserial snd_hda_codec_hdmi snd_hda_scodec_component cfbimgblt snd_hda_intel fb_io_fops snd_intel_dspcfg cfbcopyarea snd_hda_codec i2c_algo_bit fb snd_hda_core aesni_intel cfg80211 cdc_acm snd_pcm crypto_simd font snd_timer cryptd snd at24 e1000 regmap_i2c acpi_cpufreq libarc4 soundcore k10temp fam15h_power evdev nfsd sch_fq_codel auth_rpcgss lockd grace drm sunrpc drm_panel_orientation_quirks fuse backlight configfs loop nfnetlink usbhid ohci_pci xhci_pci xhci_hcd ohci_hcd ehci_pci ehci_hcd sha512_ssse3 usbcore sha256_ssse3 sha1_ssse3 sha1_generic gf128mul usb_common dm_mirror dm_region_hash dm_log cpuid i2c_piix4 i2c_smbus i2c_dev i2c_core it87 hwmon_vid msr dmi_sysfs autofs4 CR2: 0000000000000000 ---[ end trace 0000000000000000 ]--- RIP: 0010:rb_next+0x0/0x50 Code: e8 d5 fa ff ff 5b 4c 89 e0 5d 41 5c 41 5d 41 5e e9 85 73 01 00 5b 5d 41 5c 41 5d 41 5e e9 38 76 01 00 0f 1f 84 00 00 00 00 00 <48> 3b 3f 48 89 f8 74 38 48 8b 57 08 48 85 d2 74 11 48 89 d0 48 8b RSP: 0018:ffffc90000003e50 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff88811a764000 RCX: ffff88811a764180 RDX: ffff88835e4c6c00 RSI: ffff888162c998e8 RDI: 0000000000000000 RBP: 0000000000000000 R08: ffff88811a7642b0 R09: 00000000a535eebc R10: 0000000000000d09 R11: ffffc90000003ff8 R12: ffff88835e4c6c00 R13: ffff88811a7642b8 R14: 00001a951355b383 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88842ec00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 00000001084b4000 CR4: 00000000000406f0 Kernel panic - not syncing: Fatal exception in interrupt Kernel Offset: disabled ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
BUG: kernel NULL pointer dereference, address: 0000000000000000 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 1062b1067 P4D 1062b1067 PUD 1062ae067 PMD 0 Oops: Oops: 0000 [#1] PREEMPT SMP NOPTI CPU: 1 UID: 0 PID: 0 Comm: swapper/1 Tainted: G O 6.14.3 #20 Tainted: [O]=OOT_MODULE Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./970A-DS3P, BIOS FD 02/26/2016 RIP: 0010:rb_next+0x0/0x50 Code: e8 d5 fa ff ff 5b 4c 89 e0 5d 41 5c 41 5d 41 5e e9 85 73 01 00 5b 5d 41 5c 41 5d 41 5e e9 38 76 01 00 0f 1f 84 00 00 00 00 00 <48> 3b 3f 48 89 f8 74 38 48 8b 57 08 48 85 d2 74 11 48 89 d0 48 8b RSP: 0018:ffffc9000010ce50 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff88812106e000 RCX: ffff88812106e180 RDX: ffff888129726c00 RSI: ffff888106a052e8 RDI: 0000000000000000 RBP: 0000000000000000 R08: ffff88812106e2b0 R09: 0000000036705a4e R10: 0000000000000d03 R11: ffffc9000010cff8 R12: ffff888129726c00 R13: ffff88812106e2b8 R14: 000000d9fd03fce6 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88842ec80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 0000000112716000 CR4: 00000000000406f0 Call Trace: <IRQ> htb_dequeue+0x42f/0x610 [sch_htb] __qdisc_run+0x253/0x480 ? timerqueue_del+0x2c/0x40 qdisc_run+0x15/0x30 net_tx_action+0x182/0x1b0 handle_softirqs+0x102/0x240 __irq_exit_rcu+0x3e/0xb0 sysvec_apic_timer_interrupt+0x5b/0x70 </IRQ> <TASK> asm_sysvec_apic_timer_interrupt+0x16/0x20 RIP: 0010:acpi_safe_halt+0x22/0x30 Code: 0f 1f 84 00 00 00 00 00 65 48 8b 05 b8 38 71 7e 48 8b 00 a8 08 75 14 8b 05 a3 92 bb 00 85 c0 7e 07 0f 00 2d 20 4f 15 00 fb f4 <fa> e9 18 77 00 00 0f 1f 84 00 00 00 00 00 8a 47 08 3c 01 75 05 e9 RSP: 0018:ffffc900000c7e80 EFLAGS: 00000246 RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffff88842ec80000 RDX: ffff888100ddd464 RSI: ffff888100ddd400 RDI: ffff888100ddd464 RBP: 0000000000000001 R08: 0000000000000001 R09: 071c71c71c71c71c R10: 0000000000000006 R11: 0000000000000020 R12: ffffffff81f98280 R13: ffffffff81f982e8 R14: ffffffff81f98300 R15: 0000000000000000 acpi_idle_enter+0x8f/0xa0 cpuidle_enter_state+0xb3/0x220 cpuidle_enter+0x2a/0x40 do_idle+0x12d/0x1a0 cpu_startup_entry+0x29/0x30 start_secondary+0xed/0xf0 common_startup_64+0x12c/0x138 </TASK> Modules linked in: netconsole sch_htb cls_u32 sch_ingress sch_cake ifb act_mirred xt_hl xt_nat ts_bm xt_string xt_TARPIT(O) xt_CT xt_tcpudp xt_helper nf_nat_ftp nf_conntrack_ftp ip6t_rt ip6table_nat xt_MASQUERADE iptable_nat nf_nat xt_TCPMSS xt_LOG nf_log_syslog ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 ip6table_raw iptable_raw ip6table_mangle iptable_mangle xt_multiport xt_state xt_limit xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6table_filter ip6_tables iptable_filter ip_tables x_tables tun pppoe pppox binfmt_misc ppp_generic slhc af_packet bridge stp llc ctr ccm dm_crypt radeon drm_client_lib ath9k video wmi drm_exec ath9k_common drm_suballoc_helper ath9k_hw drm_ttm_helper syscopyarea ttm sysfillrect sysimgblt ath pl2303 snd_hda_codec_realtek fb_sys_fops snd_hda_codec_generic usbserial mac80211 drm_display_helper snd_hda_codec_hdmi snd_hda_scodec_component drm_kms_helper snd_hda_intel snd_intel_dspcfg snd_hda_codec agpgart cfbfillrect snd_hda_core cfbimgblt fb_io_fops aesni_intel snd_pcm cfg80211 e1000 cfbcopyarea i2c_algo_bit cdc_acm fb crypto_simd snd_timer snd cryptd acpi_cpufreq at24 font fam15h_power libarc4 soundcore k10temp regmap_i2c evdev nfsd sch_fq_codel auth_rpcgss lockd drm grace sunrpc drm_panel_orientation_quirks fuse backlight configfs loop nfnetlink usbhid xhci_pci ohci_pci xhci_hcd ohci_hcd ehci_pci ehci_hcd usbcore sha512_ssse3 sha256_ssse3 sha1_ssse3 sha1_generic gf128mul usb_common dm_mirror dm_region_hash dm_log cpuid i2c_piix4 i2c_smbus i2c_dev i2c_core it87 hwmon_vid msr dmi_sysfs autofs4 CR2: 0000000000000000 ---[ end trace 0000000000000000 ]--- RIP: 0010:rb_next+0x0/0x50 Code: e8 d5 fa ff ff 5b 4c 89 e0 5d 41 5c 41 5d 41 5e e9 85 73 01 00 5b 5d 41 5c 41 5d 41 5e e9 38 76 01 00 0f 1f 84 00 00 00 00 00 <48> 3b 3f 48 89 f8 74 38 48 8b 57 08 48 85 d2 74 11 48 89 d0 48 8b RSP: 0018:ffffc9000010ce50 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff88812106e000 RCX: ffff88812106e180 RDX: ffff888129726c00 RSI: ffff888106a052e8 RDI: 0000000000000000 RBP: 0000000000000000 R08: ffff88812106e2b0 R09: 0000000036705a4e R10: 0000000000000d03 R11: ffffc9000010cff8 R12: ffff888129726c00 R13: ffff88812106e2b8 R14: 000000d9fd03fce6 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88842ec80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 0000000112716000 CR4: 00000000000406f0 Kernel panic - not syncing: Fatal exception in interrupt Kernel Offset: disabled ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
BUG: kernel NULL pointer dereference, address: 0000000000000000 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 0 P4D 0 Oops: Oops: 0000 [#1] PREEMPT SMP NOPTI CPU: 1 UID: 0 PID: 0 Comm: swapper/1 Tainted: G O 6.14.3 #20 Tainted: [O]=OOT_MODULE Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./970A-DS3P, BIOS FD 02/26/2016 RIP: 0010:rb_next+0x0/0x50 Code: e8 d5 fa ff ff 5b 4c 89 e0 5d 41 5c 41 5d 41 5e e9 85 73 01 00 5b 5d 41 5c 41 5d 41 5e e9 38 76 01 00 0f 1f 84 00 00 00 00 00 <48> 3b 3f 48 89 f8 74 38 48 8b 57 08 48 85 d2 74 11 48 89 d0 48 8b RSP: 0018:ffffc9000010ce50 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff88811899d000 RCX: ffff88811899d180 RDX: ffff8881845f2800 RSI: ffff8881069b7ae8 RDI: 0000000000000000 RBP: 0000000000000000 R08: ffff88811899d2b0 R09: 000000002997d2aa R10: 0000000000003fbf R11: ffffc9000010cff8 R12: ffff8881845f2800 R13: ffff88811899d2b8 R14: 000000a69ae56401 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88842ec80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 0000000103c80000 CR4: 00000000000406f0 Call Trace: <IRQ> htb_dequeue+0x42f/0x610 [sch_htb] __qdisc_run+0x253/0x480 ? timerqueue_del+0x2c/0x40 qdisc_run+0x15/0x30 net_tx_action+0x182/0x1b0 handle_softirqs+0x102/0x240 __irq_exit_rcu+0x3e/0xb0 sysvec_apic_timer_interrupt+0x5b/0x70 </IRQ> <TASK> asm_sysvec_apic_timer_interrupt+0x16/0x20 RIP: 0010:cpuidle_enter_state+0x126/0x220 Code: 18 4c 6f 00 85 c0 7e 0b 8b 73 04 83 cf ff e8 a1 22 e5 ff 31 ff e8 9a 2e 98 ff 45 84 ff 74 07 31 ff e8 0e 58 9d ff fb 45 85 ed <0f> 88 cc 00 00 00 49 63 c5 48 8b 3c 24 48 6b c8 68 48 6b d0 30 49 RSP: 0018:ffffc900000c7e98 EFLAGS: 00000202 RAX: ffff88842ec80000 RBX: ffff888101d7f800 RCX: 0000000000000000 RDX: 000000a6607c6802 RSI: fffffffc350c4254 RDI: 0000000000000000 RBP: 0000000000000002 R08: 0000000000000002 R09: 000000e8ec11c440 R10: 0000000000000006 R11: 0000000000000020 R12: ffffffff81f98280 R13: 0000000000000002 R14: 000000a6607c6802 R15: 0000000000000000 ? cpuidle_enter_state+0x116/0x220 cpuidle_enter+0x2a/0x40 do_idle+0x12d/0x1a0 cpu_startup_entry+0x29/0x30 start_secondary+0xed/0xf0 common_startup_64+0x12c/0x138 </TASK> Modules linked in: sch_htb cls_u32 sch_ingress sch_cake ifb act_mirred xt_hl xt_nat ts_bm xt_string xt_TARPIT(O) xt_CT xt_tcpudp xt_helper nf_nat_ftp nf_conntrack_ftp ip6t_rt ip6table_nat xt_MASQUERADE iptable_nat nf_nat xt_TCPMSS xt_LOG nf_log_syslog ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 ip6table_raw iptable_raw ip6table_mangle iptable_mangle xt_multiport xt_state xt_limit xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6table_filter ip6_tables iptable_filter ip_tables x_tables pppoe tun pppox binfmt_misc ppp_generic slhc netconsole af_packet bridge stp llc ctr ccm dm_crypt radeon drm_client_lib video wmi drm_exec ath9k drm_suballoc_helper drm_ttm_helper syscopyarea ttm ath9k_common ath9k_hw sysfillrect sysimgblt fb_sys_fops drm_display_helper ath pl2303 drm_kms_helper usbserial mac80211 snd_hda_codec_realtek snd_hda_codec_generic snd_hda_codec_hdmi agpgart snd_hda_scodec_component cfbfillrect snd_hda_intel cfbimgblt snd_intel_dspcfg snd_hda_codec fb_io_fops snd_hda_core cfbcopyarea aesni_intel i2c_algo_bit cfg80211 snd_pcm fb snd_timer e1000 snd crypto_simd cdc_acm cryptd at24 font acpi_cpufreq libarc4 soundcore fam15h_power regmap_i2c k10temp evdev nfsd sch_fq_codel auth_rpcgss lockd grace sunrpc drm fuse configfs drm_panel_orientation_quirks backlight loop nfnetlink usbhid xhci_pci ohci_pci xhci_hcd ohci_hcd ehci_pci ehci_hcd sha512_ssse3 sha256_ssse3 usbcore sha1_ssse3 sha1_generic gf128mul usb_common dm_mirror dm_region_hash dm_log cpuid i2c_piix4 i2c_smbus i2c_dev i2c_core it87 hwmon_vid msr dmi_sysfs autofs4 CR2: 0000000000000000 ---[ end trace 0000000000000000 ]--- RIP: 0010:rb_next+0x0/0x50 Code: e8 d5 fa ff ff 5b 4c 89 e0 5d 41 5c 41 5d 41 5e e9 85 73 01 00 5b 5d 41 5c 41 5d 41 5e e9 38 76 01 00 0f 1f 84 00 00 00 00 00 <48> 3b 3f 48 89 f8 74 38 48 8b 57 08 48 85 d2 74 11 48 89 d0 48 8b RSP: 0018:ffffc9000010ce50 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff88811899d000 RCX: ffff88811899d180 RDX: ffff8881845f2800 RSI: ffff8881069b7ae8 RDI: 0000000000000000 RBP: 0000000000000000 R08: ffff88811899d2b0 R09: 000000002997d2aa R10: 0000000000003fbf R11: ffffc9000010cff8 R12: ffff8881845f2800 R13: ffff88811899d2b8 R14: 000000a69ae56401 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88842ec80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 0000000103c80000 CR4: 00000000000406f0 Kernel panic - not syncing: Fatal exception in interrupt Kernel Offset: disabled ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
#!/bin/bash
# https://www.bufferbloat.net/projects/codel/wiki/Cake/#installing-cake-out-of... # https://trofi.github.io/posts/217-mitigating%20bufferbloat.html
#set -x
set -o nounset set -o errexit
export PATH=/sbin:/bin:/usr/sbin:/usr/bin
ext=ppp0 ext_ingress=ppp0ifb0
# [query ADSL modem for up and down rates]
echo -e "pppd pppoe UP $UP DN $DN" | systemd-cat -t traffic-control
ext_up=$((UP * 95 / 100))kbit ext_down=$((DN * 95 / 100))kbit
# below taken from https://wiki.gentoo.org/wiki/Traffic_shaping
q=1486 # HTB Quantum = 1500bytes IP + 14 bytes ethernet. # Higher bandwidths may require a higher htb quantum. MEASURE. # Some ADSL devices might require a stab setting.
quantum=300 # fq_codel quantum 300 gives a boost to interactive flows # At higher bandwidths (50Mbit+) don't bother
modprobe act_mirred modprobe ifb modprobe sch_cake modprobe sch_fq_codel
ethtool -K "$ext" tso off gso off gro off # Also turn of gro on ALL interfaces # e.g ethtool -K eth1 gro off if you have eth1 # some devices you may need to run these # commands independently
# Clear old queuing disciplines (qdisc) on the interfaces tc qdisc del dev "$ext" root >& /dev/null || true tc qdisc del dev "$ext" ingress >& /dev/null || true tc qdisc del dev "$ext_ingress" root >& /dev/null || true tc qdisc del dev "$ext_ingress" ingress >& /dev/null || true ip link del "$ext_ingress" >& /dev/null || true
######### # INGRESS #########
# Create ingress on external interface tc qdisc add dev "$ext" handle ffff: ingress
ip link add name "$ext_ingress" type ifb ip link set dev "$ext_ingress" up || true # if the interace is not up bad things happen
# Forward all ingress traffic to the IFB device tc filter add dev "$ext" parent ffff: protocol all u32 match u32 0 0 action mirred egress redirect dev "$ext_ingress"
# Create an EGRESS filter on the IFB device
# Warning: sch_htb: quantum of class 10001 is big. Consider r2q change # https://web.archive.org/web/20030514055053/http://www.docum.org/stef.coene/q... # default r2q is 10 # since up ADSL rate went from 24.4 Mb/s to 26.8 Mb/s, "r2q 15" started giving a "too big" error # up it to 20, now OK again tc qdisc add dev "$ext_ingress" root handle 1: htb default 11 r2q 20
# Add root class HTB with rate limiting
tc class add dev "$ext_ingress" parent 1: classid 1:1 htb rate $ext_down #|& grep -v "Consider r2q change" || true tc class add dev "$ext_ingress" parent 1:1 classid 1:11 htb rate $ext_down prio 0 quantum $q
# Add FQ_CODEL qdisc with ECN support (if you want ecn) tc qdisc add dev "$ext_ingress" parent 1:11 fq_codel quantum $quantum ecn
######### # EGRESS ######### # Add FQ_CODEL to EGRESS on external interface tc qdisc add dev "$ext" root handle 1: htb default 11
# Add root class HTB with rate limiting tc class add dev "$ext" parent 1: classid 1:1 htb rate $ext_up tc class add dev "$ext" parent 1:1 classid 1:11 htb rate $ext_up prio 0 quantum $q
# Note: You can apply a packet limit here and on ingress if you are memory constrained - e.g # for low bandwidths and machines with < 64MB of ram, limit 1000 is good, otherwise no point
# Add FQ_CODEL qdisc without ECN support - on egress it's generally better to just drop the packet # but feel free to enable it if you want.
tc qdisc add dev "$ext" parent 1:11 fq_codel quantum $quantum noecn
$ cat ~/1.panic | scripts/decode_stacktrace.sh vmlinux BUG: kernel NULL pointer dereference, address: 0000000000000000 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 0 P4D 0 Oops: Oops: 0000 [#1] PREEMPT SMP NOPTI CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G O 6.14.3 #20 Tainted: [O]=OOT_MODULE Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./970A-DS3P, BIOS FD 02/26/2016 RIP: 0010:rb_next (lib/rbtree.c:496) Code: e8 d5 fa ff ff 5b 4c 89 e0 5d 41 5c 41 5d 41 5e e9 85 73 01 00 5b 5d 41 5c 41 5d 41 5e e9 38 76 01 00 0f 1f 84 00 00 00 00 00 <48> 3b 3f 48 89 f8 74 38 48 8b 57 08 48 85 d2 74 11 48 89 d0 48 8b All code ======== 0: e8 d5 fa ff ff call 0xfffffffffffffada 5: 5b pop %rbx 6: 4c 89 e0 mov %r12,%rax 9: 5d pop %rbp a: 41 5c pop %r12 c: 41 5d pop %r13 e: 41 5e pop %r14 10: e9 85 73 01 00 jmp 0x1739a 15: 5b pop %rbx 16: 5d pop %rbp 17: 41 5c pop %r12 19: 41 5d pop %r13 1b: 41 5e pop %r14 1d: e9 38 76 01 00 jmp 0x1765a 22: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1) 29: 00 2a:* 48 3b 3f cmp (%rdi),%rdi <-- trapping instruction 2d: 48 89 f8 mov %rdi,%rax 30: 74 38 je 0x6a 32: 48 8b 57 08 mov 0x8(%rdi),%rdx 36: 48 85 d2 test %rdx,%rdx 39: 74 11 je 0x4c 3b: 48 89 d0 mov %rdx,%rax 3e: 48 rex.W 3f: 8b .byte 0x8b
Code starting with the faulting instruction =========================================== 0: 48 3b 3f cmp (%rdi),%rdi 3: 48 89 f8 mov %rdi,%rax 6: 74 38 je 0x40 8: 48 8b 57 08 mov 0x8(%rdi),%rdx c: 48 85 d2 test %rdx,%rdx f: 74 11 je 0x22 11: 48 89 d0 mov %rdx,%rax 14: 48 rex.W 15: 8b .byte 0x8b RSP: 0018:ffffc90000003e50 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff88811a764000 RCX: ffff88811a764180 RDX: ffff88835e4c6c00 RSI: ffff888162c998e8 RDI: 0000000000000000 RBP: 0000000000000000 R08: ffff88811a7642b0 R09: 00000000a535eebc R10: 0000000000000d09 R11: ffffc90000003ff8 R12: ffff88835e4c6c00 R13: ffff88811a7642b8 R14: 00001a951355b383 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88842ec00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 00000001084b4000 CR4: 00000000000406f0 Call Trace: <IRQ> htb_dequeue (net/sched/sch_htb.c:351 (discriminator 1) net/sched/sch_htb.c:924 (discriminator 1) net/sched/sch_htb.c:982 (discriminator 1)) sch_htb __qdisc_run (net/sched/sch_generic.c:294 net/sched/sch_generic.c:398 net/sched/sch_generic.c:416) ? timerqueue_del (lib/timerqueue.c:58) qdisc_run (./include/net/pkt_sched.h:128 ./include/net/pkt_sched.h:124) net_tx_action (net/core/dev.c:5553) handle_softirqs (./arch/x86/include/asm/atomic.h:23 ./include/linux/atomic/atomic-arch-fallback.h:457 ./include/linux/jump_label.h:262 ./include/trace/events/irq.h:142 kernel/softirq.c:562) __irq_exit_rcu (kernel/softirq.c:435 kernel/softirq.c:662) sysvec_apic_timer_interrupt (arch/x86/kernel/apic/apic.c:1049 (discriminator 35) arch/x86/kernel/apic/apic.c:1049 (discriminator 35)) </IRQ> <TASK> asm_sysvec_apic_timer_interrupt (./arch/x86/include/asm/idtentry.h:574) RIP: 0010:cpuidle_enter_state (drivers/cpuidle/cpuidle.c:292) Code: 18 4c 6f 00 85 c0 7e 0b 8b 73 04 83 cf ff e8 a1 22 e5 ff 31 ff e8 9a 2e 98 ff 45 84 ff 74 07 31 ff e8 0e 58 9d ff fb 45 85 ed <0f> 88 cc 00 00 00 49 63 c5 48 8b 3c 24 48 6b c8 68 48 6b d0 30 49 All code ======== 0: 18 4c 6f 00 sbb %cl,0x0(%rdi,%rbp,2) 4: 85 c0 test %eax,%eax 6: 7e 0b jle 0x13 8: 8b 73 04 mov 0x4(%rbx),%esi b: 83 cf ff or $0xffffffff,%edi e: e8 a1 22 e5 ff call 0xffffffffffe522b4 13: 31 ff xor %edi,%edi 15: e8 9a 2e 98 ff call 0xffffffffff982eb4 1a: 45 84 ff test %r15b,%r15b 1d: 74 07 je 0x26 1f: 31 ff xor %edi,%edi 21: e8 0e 58 9d ff call 0xffffffffff9d5834 26: fb sti 27: 45 85 ed test %r13d,%r13d 2a:* 0f 88 cc 00 00 00 js 0xfc <-- trapping instruction 30: 49 63 c5 movslq %r13d,%rax 33: 48 8b 3c 24 mov (%rsp),%rdi 37: 48 6b c8 68 imul $0x68,%rax,%rcx 3b: 48 6b d0 30 imul $0x30,%rax,%rdx 3f: 49 rex.WB
Code starting with the faulting instruction =========================================== 0: 0f 88 cc 00 00 00 js 0xd2 6: 49 63 c5 movslq %r13d,%rax 9: 48 8b 3c 24 mov (%rsp),%rdi d: 48 6b c8 68 imul $0x68,%rax,%rcx 11: 48 6b d0 30 imul $0x30,%rax,%rdx 15: 49 rex.WB RSP: 0018:ffffffff81e03e40 EFLAGS: 00000202 RAX: ffff88842ec00000 RBX: ffff8881008d9400 RCX: 0000000000000000 RDX: 00001a94d9502071 RSI: fffffffbb3498394 RDI: 0000000000000000 RBP: 0000000000000002 R08: 0000000000000002 R09: 00001a94d7bd7640 R10: 0000000000000006 R11: 0000000000000020 R12: ffffffff81f98280 R13: 0000000000000002 R14: 00001a94d9502071 R15: 0000000000000000 cpuidle_enter (drivers/cpuidle/cpuidle.c:391 (discriminator 2)) do_idle (kernel/sched/idle.c:234 kernel/sched/idle.c:325) cpu_startup_entry (kernel/sched/idle.c:422) rest_init (init/main.c:743) start_kernel (init/main.c:1525) x86_64_start_reservations (arch/x86/kernel/head64.c:513) x86_64_start_kernel (??:?) common_startup_64 (arch/x86/kernel/head_64.S:421) </TASK> Modules linked in: udp_diag netconsole sch_htb cls_u32 sch_ingress sch_cake ifb act_mirred xt_hl xt_nat ts_bm xt_string xt_TARPIT(O) xt_CT xt_tcpudp xt_helper nf_nat_ftp nf_conntrack_ftp ip6t_rt ip6table_nat xt_MASQUERADE iptable_nat nf_nat xt_TCPMSS xt_LOG nf_log_syslog ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 ip6table_raw iptable_raw ip6table_mangle iptable_mangle xt_multiport xt_state xt_limit xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6table_filter ip6_tables iptable_filter ip_tables x_tables tun pppoe binfmt_misc pppox ppp_generic slhc af_packet bridge stp llc ctr ccm dm_crypt radeon drm_client_lib video wmi drm_exec drm_suballoc_helper ath9k drm_ttm_helper syscopyarea ttm sysfillrect ath9k_common sysimgblt ath9k_hw fb_sys_fops drm_display_helper drm_kms_helper ath mac80211 agpgart pl2303 snd_hda_codec_realtek cfbfillrect snd_hda_codec_generic usbserial snd_hda_codec_hdmi snd_hda_scodec_component cfbimgblt snd_hda_intel fb_io_fops snd_intel_dspcfg cfbcopyarea snd_hda_codec i2c_algo_bit fb snd_hda_core aesni_intel cfg80211 cdc_acm snd_pcm crypto_simd font snd_timer cryptd snd at24 e1000 regmap_i2c acpi_cpufreq libarc4 soundcore k10temp fam15h_power evdev nfsd sch_fq_codel auth_rpcgss lockd grace drm sunrpc drm_panel_orientation_quirks fuse backlight configfs loop nfnetlink usbhid ohci_pci xhci_pci xhci_hcd ohci_hcd ehci_pci ehci_hcd sha512_ssse3 usbcore sha256_ssse3 sha1_ssse3 sha1_generic gf128mul usb_common dm_mirror dm_region_hash dm_log cpuid i2c_piix4 i2c_smbus i2c_dev i2c_core it87 hwmon_vid msr dmi_sysfs autofs4 CR2: 0000000000000000 ---[ end trace 0000000000000000 ]--- RIP: 0010:rb_next (lib/rbtree.c:496) Code: e8 d5 fa ff ff 5b 4c 89 e0 5d 41 5c 41 5d 41 5e e9 85 73 01 00 5b 5d 41 5c 41 5d 41 5e e9 38 76 01 00 0f 1f 84 00 00 00 00 00 <48> 3b 3f 48 89 f8 74 38 48 8b 57 08 48 85 d2 74 11 48 89 d0 48 8b All code ======== 0: e8 d5 fa ff ff call 0xfffffffffffffada 5: 5b pop %rbx 6: 4c 89 e0 mov %r12,%rax 9: 5d pop %rbp a: 41 5c pop %r12 c: 41 5d pop %r13 e: 41 5e pop %r14 10: e9 85 73 01 00 jmp 0x1739a 15: 5b pop %rbx 16: 5d pop %rbp 17: 41 5c pop %r12 19: 41 5d pop %r13 1b: 41 5e pop %r14 1d: e9 38 76 01 00 jmp 0x1765a 22: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1) 29: 00 2a:* 48 3b 3f cmp (%rdi),%rdi <-- trapping instruction 2d: 48 89 f8 mov %rdi,%rax 30: 74 38 je 0x6a 32: 48 8b 57 08 mov 0x8(%rdi),%rdx 36: 48 85 d2 test %rdx,%rdx 39: 74 11 je 0x4c 3b: 48 89 d0 mov %rdx,%rax 3e: 48 rex.W 3f: 8b .byte 0x8b
Code starting with the faulting instruction =========================================== 0: 48 3b 3f cmp (%rdi),%rdi 3: 48 89 f8 mov %rdi,%rax 6: 74 38 je 0x40 8: 48 8b 57 08 mov 0x8(%rdi),%rdx c: 48 85 d2 test %rdx,%rdx f: 74 11 je 0x22 11: 48 89 d0 mov %rdx,%rax 14: 48 rex.W 15: 8b .byte 0x8b RSP: 0018:ffffc90000003e50 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff88811a764000 RCX: ffff88811a764180 RDX: ffff88835e4c6c00 RSI: ffff888162c998e8 RDI: 0000000000000000 RBP: 0000000000000000 R08: ffff88811a7642b0 R09: 00000000a535eebc R10: 0000000000000d09 R11: ffffc90000003ff8 R12: ffff88835e4c6c00 R13: ffff88811a7642b8 R14: 00001a951355b383 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88842ec00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 00000001084b4000 CR4: 00000000000406f0 Kernel panic - not syncing: Fatal exception in interrupt Kernel Offset: disabled ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]--- $
On 2025-04-21 11:40, Alan J. Wylie wrote:
#regzbot introduced: 6.14.2..6.14.3
Since 6.14.3 I have been seeing random panics, all in htb_dequeue. 6.14.2 was fine.
6.14.3 contains: "codel: remove sch->q.qlen check before qdisc_tree_reduce_backlog()" aka https://web.git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/...
Is your HTB backed by fq_codel by any chance? If so, try either reverting the above or adding: "sch_htb: make htb_qlen_notify() idempotent" aka https://web.git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commi...
which was successfully not added to 6.14.3, along with the rest of the series: https://lore.kernel.org/all/20250403211033.166059-2-xiyou.wangcong@gmail.com...
Hope this helps. I am running fq_codel without issue but not behind htb.
cheers Holger
On Mon, 21 Apr 2025 13:50:52 +0200 Holger Hoffstätte holger@applied-asynchrony.com wrote:
On 2025-04-21 11:40, Alan J. Wylie wrote:
#regzbot introduced: 6.14.2..6.14.3
Since 6.14.3 I have been seeing random panics, all in htb_dequeue. 6.14.2 was fine.
6.14.3 contains: "codel: remove sch->q.qlen check before qdisc_tree_reduce_backlog()" aka https://web.git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/...
Is your HTB backed by fq_codel by any chance?
Yes
# grep fq 91-tc.sh quantum=300 # fq_codel quantum 300 gives a boost to interactive flows modprobe sch_fq_codel tc qdisc add dev "$ext_ingress" parent 1:11 fq_codel quantum $quantum ecn tc qdisc add dev "$ext" parent 1:11 fq_codel quantum $quantum noecn
If so, try either reverting the above or adding: "sch_htb: make htb_qlen_notify() idempotent" aka https://web.git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commi...
which was successfully not added to 6.14.3, along with the rest of the series: https://lore.kernel.org/all/20250403211033.166059-2-xiyou.wangcong@gmail.com...
"successfully not added"?
$ git cherry-pick 5ba8b837b522d7051ef81bacf3d95383ff8edce5 [linux-6.14.y 2285c724bf7d] sch_htb: make htb_qlen_notify() idempotent Author: Cong Wang xiyou.wangcong@gmail.com Date: Thu Apr 3 14:10:23 2025 -0700 1 file changed, 2 insertions(+)
It will take a while (perhaps days?) before I can confirm success.
Thanks Alan
On Mon, 21 Apr 2025 13:10:00 +0100 "Alan J. Wylie" alan@wylie.me.uk wrote:
On Mon, 21 Apr 2025 13:50:52 +0200 Holger Hoffstätte holger@applied-asynchrony.com wrote:
If so, try either reverting the above or adding: "sch_htb: make htb_qlen_notify() idempotent" aka https://web.git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commi...
which was successfully not added to 6.14.3, along with the rest of the series: https://lore.kernel.org/all/20250403211033.166059-2-xiyou.wangcong@gmail.com...
"successfully not added"?
$ git cherry-pick 5ba8b837b522d7051ef81bacf3d95383ff8edce5 [linux-6.14.y 2285c724bf7d] sch_htb: make htb_qlen_notify() idempotent Author: Cong Wang xiyou.wangcong@gmail.com Date: Thu Apr 3 14:10:23 2025 -0700 1 file changed, 2 insertions(+)
It will take a while (perhaps days?) before I can confirm success.
I'm afraid that didn't help. Same panic.
BUG: kernel NULL pointer dereference, address: 0000000000000000 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 139bfa067 P4D 139bfa067 PUD 133bf1067 PMD 0 Oops: Oops: 0000 [#1] PREEMPT SMP NOPTI CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G O 6.14.3-00001-g2285c724bf7d #21 Tainted: [O]=OOT_MODULE Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./970A-DS3P, BIOS FD 02/26/2016 RIP: 0010:rb_next+0x0/0x50 Code: e8 d5 fa ff ff 5b 4c 89 e0 5d 41 5c 41 5d 41 5e e9 85 73 01 00 5b 5d 41 5c 41 5d 41 5e e9 38 76 01 00 0f 1f 84 00 00 00 00 00 <48> 3b 3f 48 89 f8 74 38 48 8b 57 08 48 85 d2 74 11 48 89 d0 48 8b RSP: 0018:ffffc90000003e50 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff8881052e9000 RCX: ffff8881052e9180 RDX: ffff8881e5e4f400 RSI: ffff888190075ee8 RDI: 0000000000000000 RBP: 0000000000000000 R08: ffff8881052e92b0 R09: 00000000e49ea9dc R10: 000000000000278a R11: 001dcd6500000000 R12: ffff8881e5e4f400 R13: ffff8881052e92b8 R14: 00000b92b6422fbf R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88842ec00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 0000000139952000 CR4: 00000000000406f0 Call Trace: <IRQ> htb_dequeue+0x42f/0x610 [sch_htb] __qdisc_run+0x253/0x480 ? timerqueue_del+0x2c/0x40 qdisc_run+0x15/0x30 net_tx_action+0x182/0x1b0 handle_softirqs+0x102/0x240 __irq_exit_rcu+0x3e/0xb0 sysvec_apic_timer_interrupt+0x5b/0x70 </IRQ> <TASK> asm_sysvec_apic_timer_interrupt+0x16/0x20 RIP: 0010:cpuidle_enter_state+0x126/0x220 Code: 18 4c 6f 00 85 c0 7e 0b 8b 73 04 83 cf ff e8 a1 22 e5 ff 31 ff e8 9a 2e 98 ff 45 84 ff 74 07 31 ff e8 0e 58 9d ff fb 45 85 ed <0f> 88 cc 00 00 00 49 63 c5 48 8b 3c 24 48 6b c8 68 48 6b d0 30 49 RSP: 0018:ffffffff81e03e40 EFLAGS: 00000202 RAX: ffff88842ec00000 RBX: ffff8881008e7400 RCX: 0000000000000000 RDX: 00000b927d3d4a20 RSI: fffffffc3199eb61 RDI: 0000000000000000 RBP: 0000000000000002 R08: 0000000000000002 R09: 00000b927aa897c0 R10: 0000000000000006 R11: 0000000000000020 R12: ffffffff81f98280 R13: 0000000000000002 R14: 00000b927d3d4a20 R15: 0000000000000000 cpuidle_enter+0x2a/0x40 do_idle+0x12d/0x1a0 cpu_startup_entry+0x29/0x30 rest_init+0xbc/0xc0 start_kernel+0x630/0x630 x86_64_start_reservations+0x25/0x30 x86_64_start_kernel+0x73/0x80 common_startup_64+0x12c/0x138 </TASK> Modules linked in: sch_htb cls_u32 sch_ingress sch_cake ifb act_mirred netconsole xt_hl xt_nat ts_bm xt_string xt_TARPIT(O) xt_CT xt_tcpudp xt_helper nf_nat_ftp nf_conntrack_ftp ip6t_rt ip6table_nat xt_MASQUERADE iptable_nat nf_nat xt_TCPMSS xt_LOG nf_log_syslog ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 ip6table_raw iptable_raw ip6table_mangle iptable_mangle xt_multiport xt_state xt_limit xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6table_filter ip6_tables iptable_filter ip_tables x_tables pppoe tun pppox binfmt_misc ppp_generic slhc af_packet bridge stp llc ctr ccm dm_crypt radeon drm_client_lib ath9k video wmi drm_exec ath9k_common drm_suballoc_helper ath9k_hw drm_ttm_helper syscopyarea ttm ath sysfillrect sysimgblt fb_sys_fops mac80211 pl2303 drm_display_helper snd_hda_codec_realtek usbserial drm_kms_helper snd_hda_codec_generic snd_hda_codec_hdmi snd_hda_scodec_component snd_hda_intel agpgart snd_intel_dspcfg snd_hda_codec cfbfillrect cfbimgblt cfg80211 snd_hda_core fb_io_fops cfbcopyarea i2c_algo_bit fb e1000 snd_pcm font cdc_acm snd_timer aesni_intel libarc4 snd at24 acpi_cpufreq k10temp crypto_simd soundcore fam15h_power cryptd regmap_i2c evdev nfsd sch_fq_codel auth_rpcgss lockd grace sunrpc drm configfs drm_panel_orientation_quirks fuse backlight loop nfnetlink usbhid xhci_pci ohci_pci xhci_hcd ohci_hcd ehci_pci ehci_hcd sha512_ssse3 sha256_ssse3 sha1_ssse3 usbcore sha1_generic gf128mul usb_common dm_mirror dm_region_hash dm_log cpuid i2c_piix4 i2c_smbus i2c_dev i2c_core it87 hwmon_vid msr dmi_sysfs autofs4 CR2: 0000000000000000 ---[ end trace 0000000000000000 ]--- RIP: 0010:rb_next+0x0/0x50 Code: e8 d5 fa ff ff 5b 4c 89 e0 5d 41 5c 41 5d 41 5e e9 85 73 01 00 5b 5d 41 5c 41 5d 41 5e e9 38 76 01 00 0f 1f 84 00 00 00 00 00 <48> 3b 3f 48 89 f8 74 38 48 8b 57 08 48 85 d2 74 11 48 89 d0 48 8b RSP: 0018:ffffc90000003e50 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff8881052e9000 RCX: ffff8881052e9180 RDX: ffff8881e5e4f400 RSI: ffff888190075ee8 RDI: 0000000000000000 RBP: 0000000000000000 R08: ffff8881052e92b0 R09: 00000000e49ea9dc R10: 000000000000278a R11: 001dcd6500000000 R12: ffff8881e5e4f400 R13: ffff8881052e92b8 R14: 00000b92b6422fbf R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88842ec00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 0000000139952000 CR4: 00000000000406f0 Kernel panic - not syncing: Fatal exception in interrupt Kernel Offset: disabled Rebooting in 3 seconds..
On 2025-04-21 21:06, Alan J. Wylie wrote:
On Mon, 21 Apr 2025 13:10:00 +0100 "Alan J. Wylie" alan@wylie.me.uk wrote:
On Mon, 21 Apr 2025 13:50:52 +0200 Holger Hoffstätte holger@applied-asynchrony.com wrote:
If so, try either reverting the above or adding: "sch_htb: make htb_qlen_notify() idempotent" aka https://web.git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commi...
which was successfully not added to 6.14.3, along with the rest of the series: https://lore.kernel.org/all/20250403211033.166059-2-xiyou.wangcong@gmail.com...
"successfully not added"?
$ git cherry-pick 5ba8b837b522d7051ef81bacf3d95383ff8edce5 [linux-6.14.y 2285c724bf7d] sch_htb: make htb_qlen_notify() idempotent Author: Cong Wang xiyou.wangcong@gmail.com Date: Thu Apr 3 14:10:23 2025 -0700 1 file changed, 2 insertions(+)
It will take a while (perhaps days?) before I can confirm success.
I'm afraid that didn't help. Same panic.
Bummer :-(
Might be something else missing then - so for now the only other thing I'd suggest is to revert the removal of the qlen check in fq_codel.
Holger
On Mon, 21 Apr 2025 21:47:44 +0200 Holger Hoffstätte holger@applied-asynchrony.com wrote:
I'm afraid that didn't help. Same panic.
Bummer :-(
Might be something else missing then - so for now the only other thing I'd suggest is to revert the removal of the qlen check in fq_codel.
Like this?
$ git diff sch_fq_codel.c diff --git a/net/sched/sch_fq_codel.c b/net/sched/sch_fq_codel.c index 6c9029f71e88..4fdf317b82ec 100644 --- a/net/sched/sch_fq_codel.c +++ b/net/sched/sch_fq_codel.c @@ -316,7 +316,7 @@ static struct sk_buff *fq_codel_dequeue(struct Qdisc *sch) qdisc_bstats_update(sch, skb); flow->deficit -= qdisc_pkt_len(skb);
- if (q->cstats.drop_count) { + if (q->cstats.drop_count && sch->q.qlen) { qdisc_tree_reduce_backlog(sch, q->cstats.drop_count, q->cstats.drop_len); q->cstats.drop_count = 0; $
I'll be off to bed soon, but I'll leave it running overnight.
I might be able to do a quick report in the morning, but I'll have to set off early to go digging down a cave all day tomorrow.
On Mon, 21 Apr 2025 21:09:27 +0100 "Alan J. Wylie" alan@wylie.me.uk wrote:
On Mon, 21 Apr 2025 21:47:44 +0200 Holger Hoffstätte holger@applied-asynchrony.com wrote:
I'm afraid that didn't help. Same panic.
Bummer :-(
Might be something else missing then - so for now the only other thing I'd suggest is to revert the removal of the qlen check in fq_codel.
Like this?
$ git diff sch_fq_codel.c diff --git a/net/sched/sch_fq_codel.c b/net/sched/sch_fq_codel.c index 6c9029f71e88..4fdf317b82ec 100644 --- a/net/sched/sch_fq_codel.c +++ b/net/sched/sch_fq_codel.c @@ -316,7 +316,7 @@ static struct sk_buff *fq_codel_dequeue(struct Qdisc *sch) qdisc_bstats_update(sch, skb); flow->deficit -= qdisc_pkt_len(skb);
if (q->cstats.drop_count) {
if (q->cstats.drop_count && sch->q.qlen) { qdisc_tree_reduce_backlog(sch, q->cstats.drop_count, q->cstats.drop_len); q->cstats.drop_count = 0;$
It's been about 21 hours and no crash yet. I had an excellent day down a cave, so there's not been as much Internet traffic as usual, but there's a good chance the above patch as at least worked around, if not fixed the issue.
Regards Alan
(cc: Greg KH)
On 2025-04-22 18:51, Alan J. Wylie wrote:
On Mon, 21 Apr 2025 21:09:27 +0100 "Alan J. Wylie" alan@wylie.me.uk wrote:
On Mon, 21 Apr 2025 21:47:44 +0200 Holger Hoffstätte holger@applied-asynchrony.com wrote:
I'm afraid that didn't help. Same panic.
Bummer :-(
Might be something else missing then - so for now the only other thing I'd suggest is to revert the removal of the qlen check in fq_codel.
Like this?
$ git diff sch_fq_codel.c diff --git a/net/sched/sch_fq_codel.c b/net/sched/sch_fq_codel.c index 6c9029f71e88..4fdf317b82ec 100644 --- a/net/sched/sch_fq_codel.c +++ b/net/sched/sch_fq_codel.c @@ -316,7 +316,7 @@ static struct sk_buff *fq_codel_dequeue(struct Qdisc *sch) qdisc_bstats_update(sch, skb); flow->deficit -= qdisc_pkt_len(skb);
if (q->cstats.drop_count) {
if (q->cstats.drop_count && sch->q.qlen) { qdisc_tree_reduce_backlog(sch, q->cstats.drop_count, q->cstats.drop_len); q->cstats.drop_count = 0;$
It's been about 21 hours and no crash yet. I had an excellent day down a cave, so there's not been as much Internet traffic as usual, but there's a good chance the above patch as at least worked around, if not fixed the issue.
Thought so .. \o/
I guess now the question is what to do about it. IIUC the fix series [1] addressed some kind of UAF problem, but obviously was not applied correctly or is missing follow-ups. It's also a bit mysterious why adding the HTB patch didn't work.
Maybe Cong Wang can advise what to do here?
So unless someone else has any ideas: Greg, please revert:
6.14.y/a57fe60ef4cf96bfbb6b58397ec28bdb5a5c6b31 ("codel: remove sch->q.qlen check before qdisc_tree_reduce_backlog()")
and probably from 6.12 as well.
cheers Holger
[1] https://lore.kernel.org/all/20250403211033.166059-1-xiyou.wangcong@gmail.com...
On Tue, Apr 22, 2025 at 07:20:24PM +0200, Holger Hoffstätte wrote:
(cc: Greg KH)
On 2025-04-22 18:51, Alan J. Wylie wrote:
On Mon, 21 Apr 2025 21:09:27 +0100 "Alan J. Wylie" alan@wylie.me.uk wrote:
On Mon, 21 Apr 2025 21:47:44 +0200 Holger Hoffstätte holger@applied-asynchrony.com wrote:
I'm afraid that didn't help. Same panic.
Bummer :-(
Might be something else missing then - so for now the only other thing I'd suggest is to revert the removal of the qlen check in fq_codel.
Like this?
$ git diff sch_fq_codel.c diff --git a/net/sched/sch_fq_codel.c b/net/sched/sch_fq_codel.c index 6c9029f71e88..4fdf317b82ec 100644 --- a/net/sched/sch_fq_codel.c +++ b/net/sched/sch_fq_codel.c @@ -316,7 +316,7 @@ static struct sk_buff *fq_codel_dequeue(struct Qdisc *sch) qdisc_bstats_update(sch, skb); flow->deficit -= qdisc_pkt_len(skb);
if (q->cstats.drop_count) {
if (q->cstats.drop_count && sch->q.qlen) { qdisc_tree_reduce_backlog(sch, q->cstats.drop_count, q->cstats.drop_len); q->cstats.drop_count = 0;$
It's been about 21 hours and no crash yet. I had an excellent day down a cave, so there's not been as much Internet traffic as usual, but there's a good chance the above patch as at least worked around, if not fixed the issue.
Thought so .. \o/
I guess now the question is what to do about it. IIUC the fix series [1] addressed some kind of UAF problem, but obviously was not applied correctly or is missing follow-ups. It's also a bit mysterious why adding the HTB patch didn't work.
Maybe Cong Wang can advise what to do here?
I guess my patch caused some regression, I am still decoding the crashes reported here.
Meanwhile, if you could provide a reliable (and ideally minimum) reproducer, it would help me a lot to debug.
Thanks!
On Tue, 22 Apr 2025 13:42:19 -0700 Cong Wang xiyou.wangcong@gmail.com wrote:
On Tue, Apr 22, 2025 at 07:20:24PM +0200, Holger Hoffstätte wrote:
(cc: Greg KH)
On 2025-04-22 18:51, Alan J. Wylie wrote:
On Mon, 21 Apr 2025 21:09:27 +0100 "Alan J. Wylie" alan@wylie.me.uk wrote:
On Mon, 21 Apr 2025 21:47:44 +0200 Holger Hoffstätte holger@applied-asynchrony.com wrote:
I'm afraid that didn't help. Same panic.
Bummer :-(
Might be something else missing then - so for now the only other thing I'd suggest is to revert the removal of the qlen check in fq_codel.
Like this?
$ git diff sch_fq_codel.c diff --git a/net/sched/sch_fq_codel.c b/net/sched/sch_fq_codel.c index 6c9029f71e88..4fdf317b82ec 100644 --- a/net/sched/sch_fq_codel.c +++ b/net/sched/sch_fq_codel.c @@ -316,7 +316,7 @@ static struct sk_buff *fq_codel_dequeue(struct Qdisc *sch) qdisc_bstats_update(sch, skb); flow->deficit -= qdisc_pkt_len(skb);
if (q->cstats.drop_count) {
if (q->cstats.drop_count && sch->q.qlen) { qdisc_tree_reduce_backlog(sch,q->cstats.drop_count, q->cstats.drop_len); q->cstats.drop_count = 0; $
It's been about 21 hours and no crash yet. I had an excellent day down a cave, so there's not been as much Internet traffic as usual, but there's a good chance the above patch as at least worked around, if not fixed the issue.
Thought so .. \o/
I guess now the question is what to do about it. IIUC the fix series [1] addressed some kind of UAF problem, but obviously was not applied correctly or is missing follow-ups. It's also a bit mysterious why adding the HTB patch didn't work.
Maybe Cong Wang can advise what to do here?
I guess my patch caused some regression, I am still decoding the crashes reported here.
Meanwhile, if you could provide a reliable (and ideally minimum) reproducer, it would help me a lot to debug.
Thanks!
Sorry. No reproducer. The crashes seemed to be totally random.
I posted the script I use to set up tc in my initial report.
FYI, here's the resulting config.
# tc qdisc show qdisc noqueue 0: dev lo root refcnt 2 qdisc fq_codel 0: dev enp3s0 root refcnt 2 limit 10240p flows 1024 quantum 6014 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64 qdisc fq_codel 0: dev enp4s0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64 qdisc fq_codel 0: dev enp5s6f0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64 qdisc noqueue 0: dev wlp5s7 root refcnt 2 qdisc noqueue 0: dev brdmz root refcnt 2 qdisc noqueue 0: dev heipv6 root refcnt 2 qdisc fq_codel 0: dev tun0 root refcnt 2 limit 10240p flows 1024 quantum 1464 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64 qdisc htb 1: dev ppp0 root refcnt 2 r2q 10 default 0x11 direct_packets_stat 0 direct_qlen 3 qdisc fq_codel 824c: dev ppp0 parent 1:11 limit 10240p flows 1024 quantum 300 target 5ms interval 100ms memory_limit 32Mb drop_batch 64 qdisc ingress ffff: dev ppp0 parent ffff:fff1 ---------------- qdisc fq_codel 0: dev tun1 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64 qdisc htb 1: dev ppp0ifb0 root refcnt 2 r2q 20 default 0x11 direct_packets_stat 0 direct_qlen 32 qdisc fq_codel 824b: dev ppp0ifb0 parent 1:11 limit 10240p flows 1024 quantum 300 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64 #
Hi, Alan
Although I am still trying to understand the NULL pointer, which seems likely from:
478 if (p->inner.clprio[prio].ptr == cl->node + prio) { 479 /* we are removing child which is pointed to from 480 * parent feed - forget the pointer but remember 481 * classid 482 */ 483 p->inner.clprio[prio].last_ptr_id = cl->common.classid; 484 p->inner.clprio[prio].ptr = NULL; 485 }
Does the following patch work? I mean not just fixing the crash, but also not causing any other problem.
Please give it a try.
Thanks!
---
diff --git a/net/sched/sch_htb.c b/net/sched/sch_htb.c index 4b9a639b642e..0cdc778fddef 100644 --- a/net/sched/sch_htb.c +++ b/net/sched/sch_htb.c @@ -348,7 +348,8 @@ static void htb_add_to_wait_tree(struct htb_sched *q, */ static inline void htb_next_rb_node(struct rb_node **n) { - *n = rb_next(*n); + if (*n) + *n = rb_next(*n); }
/**
On Tue, 22 Apr 2025 14:49:27 -0700 Cong Wang xiyou.wangcong@gmail.com wrote:
Although I am still trying to understand the NULL pointer, which seems likely from:
478 if (p->inner.clprio[prio].ptr == cl->node + prio) { 479 /* we are removing child which is pointed to from 480 * parent feed - forget the pointer but remember 481 * classid 482 */ 483 p->inner.clprio[prio].last_ptr_id = cl->common.classid; 484 p->inner.clprio[prio].ptr = NULL; 485 }
Does the following patch work? I mean not just fixing the crash, but also not causing any other problem.
diff --git a/net/sched/sch_htb.c b/net/sched/sch_htb.c index 4b9a639b642e..0cdc778fddef 100644 --- a/net/sched/sch_htb.c +++ b/net/sched/sch_htb.c @@ -348,7 +348,8 @@ static void htb_add_to_wait_tree(struct htb_sched *q, */ static inline void htb_next_rb_node(struct rb_node **n) {
- *n = rb_next(*n);
- if (*n)
*n = rb_next(*n);} /**
There's been three of these:
Apr 23 08:08:32 bilbo kernel: WARNING: CPU: 0 PID: 0 at htb_deactivate+0xd/0x30 [sch_htb] Apr 23 08:08:32 bilbo kernel: WARNING: CPU: 0 PID: 0 at htb_deactivate+0xd/0x30 [sch_htb] Apr 23 10:41:36 bilbo kernel: WARNING: CPU: 1 PID: 0 at htb_deactivate+0xd/0x30 [sch_htb]
But no panic.
I've run scripts/decode.sh on the last one.
Apr 23 08:08:32 bilbo kernel: ------------[ cut here ]------------ Apr 23 08:08:32 bilbo kernel: WARNING: CPU: 0 PID: 0 at htb_deactivate+0xd/0x30 [sch_htb] Apr 23 08:08:32 bilbo kernel: Modules linked in: sch_htb cls_u32 sch_ingress sch_cake ifb act_mirred xt_hl xt_nat ts_bm xt_string xt_TARPIT(O) xt_CT xt_tcpudp xt_helper nf_nat_ftp nf_conntrack_ftp ip6t_rt ip6table_nat xt_MASQUERADE iptable_nat nf_nat xt_TCPMSS xt_LOG nf_log_syslog ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 ip6table_raw iptable_raw ip6table_mangle iptable_mangle xt_multiport xt_state xt_limit xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6table_filter ip6_tables iptable_filter ip_tables x_tables pppoe pppox ppp_generic binfmt_misc tun slhc netconsole af_packet bridge stp llc ctr ccm dm_crypt radeon ath9k drm_client_lib ath9k_common video ath9k_hw wmi drm_exec drm_suballoc_helper snd_hda_codec_realtek drm_ttm_helper snd_hda_codec_generic snd_hda_codec_hdmi ath syscopyarea snd_hda_scodec_component ttm pl2303 snd_hda_intel usbserial mac80211 sysfillrect snd_intel_dspcfg sysimgblt snd_hda_codec fb_sys_fops drm_display_helper drm_kms_helper snd_hda_co re agpgart snd_pcm cfbfillrect cfbimgblt snd_timer Apr 23 08:08:32 bilbo kernel: cfg80211 fb_io_fops cdc_acm cfbcopyarea aesni_intel i2c_algo_bit e1000 crypto_simd snd fb cryptd at24 libarc4 regmap_i2c font fam15h_power soundcore acpi_cpufreq k10temp evdev nfsd sch_fq_codel auth_rpcgss lockd drm grace sunrpc drm_panel_orientation_quirks backlight fuse loop configfs nfnetlink usbhid xhci_pci ohci_pci xhci_hcd ohci_hcd ehci_pci ehci_hcd usbcore sha512_ssse3 sha256_ssse3 sha1_ssse3 sha1_generic gf128mul usb_common dm_mirror dm_region_hash dm_log cpuid i2c_piix4 i2c_smbus i2c_dev i2c_core it87 hwmon_vid msr dmi_sysfs autofs4 Apr 23 08:08:32 bilbo kernel: CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G O 6.14.3-dirty #23 Apr 23 08:08:32 bilbo kernel: Tainted: [O]=OOT_MODULE Apr 23 08:08:32 bilbo kernel: Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./970A-DS3P, BIOS FD 02/26/2016 Apr 23 08:08:32 bilbo kernel: RIP: 0010:htb_deactivate+0xd/0x30 [sch_htb] Apr 23 08:08:32 bilbo kernel: Code: d4 45 21 a4 87 08 01 00 00 48 83 c4 18 5b 5d 41 5c 41 5d 41 5e 41 5f e9 c1 c5 a7 e0 90 53 83 be a8 01 00 00 00 48 89 f3 75 02 <0f> 0b 48 89 de e8 29 fe ff ff 31 c0 89 83 a8 01 00 00 5b e9 9b c5 Apr 23 08:08:32 bilbo kernel: RSP: 0018:ffffc90000003e50 EFLAGS: 00010246 Apr 23 08:08:32 bilbo kernel: RAX: ffff8881b7311c00 RBX: ffff8881b7312000 RCX: ffff8881b73121c8 Apr 23 08:08:32 bilbo kernel: RDX: ffff8881b7312000 RSI: ffff8881b7312000 RDI: ffff88811c353180 Apr 23 08:08:32 bilbo kernel: RBP: 0000000000000000 R08: ffff88811c3532b0 R09: 000000009ceae056 Apr 23 08:08:32 bilbo kernel: R10: 0000000000005de4 R11: ffffc90000003ff8 R12: 0000000000000000 Apr 23 08:08:32 bilbo kernel: R13: ffff8881b7312000 R14: 00000273e71c1348 R15: 0000000000000000 Apr 23 08:08:32 bilbo kernel: FS: 0000000000000000(0000) GS:ffff88842ec00000(0000) knlGS:0000000000000000 Apr 23 08:08:32 bilbo kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Apr 23 08:08:32 bilbo kernel: CR2: 000000c00050b000 CR3: 000000018920e000 CR4: 00000000000406f0 Apr 23 08:08:32 bilbo kernel: Call Trace: Apr 23 08:08:32 bilbo kernel: <IRQ> Apr 23 08:08:32 bilbo kernel: htb_dequeue+0x3f1/0x5a0 [sch_htb] Apr 23 08:08:32 bilbo kernel: __qdisc_run+0x253/0x480 Apr 23 08:08:32 bilbo kernel: ? timerqueue_del+0x2c/0x40 Apr 23 08:08:32 bilbo kernel: qdisc_run+0x15/0x30 Apr 23 08:08:32 bilbo kernel: net_tx_action+0x182/0x1b0 Apr 23 08:08:32 bilbo kernel: handle_softirqs+0x102/0x240 Apr 23 08:08:32 bilbo kernel: __irq_exit_rcu+0x3e/0xb0 Apr 23 08:08:32 bilbo kernel: sysvec_apic_timer_interrupt+0x5b/0x70 Apr 23 08:08:32 bilbo kernel: </IRQ> Apr 23 08:08:32 bilbo kernel: <TASK> Apr 23 08:08:32 bilbo kernel: asm_sysvec_apic_timer_interrupt+0x16/0x20 Apr 23 08:08:32 bilbo kernel: RIP: 0010:cpuidle_enter_state+0x126/0x220 Apr 23 08:08:32 bilbo kernel: Code: 18 4c 6f 00 85 c0 7e 0b 8b 73 04 83 cf ff e8 a1 22 e5 ff 31 ff e8 9a 2e 98 ff 45 84 ff 74 07 31 ff e8 0e 58 9d ff fb 45 85 ed <0f> 88 cc 00 00 00 49 63 c5 48 8b 3c 24 48 6b c8 68 48 6b d0 30 49 Apr 23 08:08:32 bilbo kernel: RSP: 0018:ffffffff81e03e40 EFLAGS: 00000202 Apr 23 08:08:32 bilbo kernel: RAX: ffff88842ec00000 RBX: ffff8881008d8000 RCX: 0000000000000000 Apr 23 08:08:32 bilbo kernel: RDX: 00000273acf9a9e7 RSI: fffffff6533d45e7 RDI: 0000000000000000 Apr 23 08:08:32 bilbo kernel: RBP: 0000000000000002 R08: 0000000000000002 R09: 071c71c71c71c71c Apr 23 08:08:32 bilbo kernel: R10: 0000000000000006 R11: 0000000000000020 R12: ffffffff81f98280 Apr 23 08:08:32 bilbo kernel: R13: 0000000000000002 R14: 00000273acf9a9e7 R15: 0000000000000000 Apr 23 08:08:32 bilbo kernel: cpuidle_enter+0x2a/0x40 Apr 23 08:08:32 bilbo kernel: do_idle+0x12d/0x1a0 Apr 23 08:08:32 bilbo kernel: cpu_startup_entry+0x29/0x30 Apr 23 08:08:32 bilbo kernel: rest_init+0xbc/0xc0 Apr 23 08:08:32 bilbo kernel: start_kernel+0x630/0x630 Apr 23 08:08:32 bilbo kernel: x86_64_start_reservations+0x25/0x30 Apr 23 08:08:32 bilbo kernel: x86_64_start_kernel+0x73/0x80 Apr 23 08:08:32 bilbo kernel: common_startup_64+0x12c/0x138 Apr 23 08:08:32 bilbo kernel: </TASK> Apr 23 08:08:32 bilbo kernel: ---[ end trace 0000000000000000 ]---
Apr 23 08:08:32 bilbo kernel: ------------[ cut here ]------------ Apr 23 08:08:32 bilbo kernel: WARNING: CPU: 0 PID: 0 at htb_deactivate+0xd/0x30 [sch_htb] Apr 23 08:08:32 bilbo kernel: Modules linked in: sch_htb cls_u32 sch_ingress sch_cake ifb act_mirred xt_hl xt_nat ts_bm xt_string xt_TARPIT(O) xt_CT xt_tcpudp xt_helper nf_nat_ftp nf_conntrack_ftp ip6t_rt ip6table_nat xt_MASQUERADE iptable_nat nf_nat xt_TCPMSS xt_LOG nf_log_syslog ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 ip6table_raw iptable_raw ip6table_mangle iptable_mangle xt_multiport xt_state xt_limit xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6table_filter ip6_tables iptable_filter ip_tables x_tables pppoe pppox ppp_generic binfmt_misc tun slhc netconsole af_packet bridge stp llc ctr ccm dm_crypt radeon ath9k drm_client_lib ath9k_common video ath9k_hw wmi drm_exec drm_suballoc_helper snd_hda_codec_realtek drm_ttm_helper snd_hda_codec_generic snd_hda_codec_hdmi ath syscopyarea snd_hda_scodec_component ttm pl2303 snd_hda_intel usbserial mac80211 sysfillrect snd_intel_dspcfg sysimgblt snd_hda_codec fb_sys_fops drm_display_helper drm_kms_helper snd_hda_co re agpgart snd_pcm cfbfillrect cfbimgblt snd_timer Apr 23 08:08:32 bilbo kernel: cfg80211 fb_io_fops cdc_acm cfbcopyarea aesni_intel i2c_algo_bit e1000 crypto_simd snd fb cryptd at24 libarc4 regmap_i2c font fam15h_power soundcore acpi_cpufreq k10temp evdev nfsd sch_fq_codel auth_rpcgss lockd drm grace sunrpc drm_panel_orientation_quirks backlight fuse loop configfs nfnetlink usbhid xhci_pci ohci_pci xhci_hcd ohci_hcd ehci_pci ehci_hcd usbcore sha512_ssse3 sha256_ssse3 sha1_ssse3 sha1_generic gf128mul usb_common dm_mirror dm_region_hash dm_log cpuid i2c_piix4 i2c_smbus i2c_dev i2c_core it87 hwmon_vid msr dmi_sysfs autofs4 Apr 23 08:08:32 bilbo kernel: CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G W O 6.14.3-dirty #23 Apr 23 08:08:32 bilbo kernel: Tainted: [W]=WARN, [O]=OOT_MODULE Apr 23 08:08:32 bilbo kernel: Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./970A-DS3P, BIOS FD 02/26/2016 Apr 23 08:08:32 bilbo kernel: RIP: 0010:htb_deactivate+0xd/0x30 [sch_htb] Apr 23 08:08:32 bilbo kernel: Code: d4 45 21 a4 87 08 01 00 00 48 83 c4 18 5b 5d 41 5c 41 5d 41 5e 41 5f e9 c1 c5 a7 e0 90 53 83 be a8 01 00 00 00 48 89 f3 75 02 <0f> 0b 48 89 de e8 29 fe ff ff 31 c0 89 83 a8 01 00 00 5b e9 9b c5 Apr 23 08:08:32 bilbo kernel: RSP: 0018:ffffc90000003e50 EFLAGS: 00010246 Apr 23 08:08:32 bilbo kernel: RAX: ffff8881b7311c00 RBX: ffff8881b7312000 RCX: ffff8881b73121c8 Apr 23 08:08:32 bilbo kernel: RDX: ffff8881b7312000 RSI: ffff8881b7312000 RDI: ffff88811c353180 Apr 23 08:08:32 bilbo kernel: RBP: 0000000000000000 R08: ffff88811c3532b0 R09: 000000009cee5629 Apr 23 08:08:32 bilbo kernel: R10: 00000000000033ab R11: 001dcd6500000000 R12: 0000000000000000 Apr 23 08:08:32 bilbo kernel: R13: ffff8881b7312000 R14: 00000273f4f3639c R15: 0000000000000000 Apr 23 08:08:32 bilbo kernel: FS: 0000000000000000(0000) GS:ffff88842ec00000(0000) knlGS:0000000000000000 Apr 23 08:08:32 bilbo kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Apr 23 08:08:32 bilbo kernel: CR2: 00007fc3b0cb0000 CR3: 0000000126e70000 CR4: 00000000000406f0 Apr 23 08:08:32 bilbo kernel: Call Trace: Apr 23 08:08:32 bilbo kernel: <IRQ> Apr 23 08:08:32 bilbo kernel: htb_dequeue+0x3f1/0x5a0 [sch_htb] Apr 23 08:08:32 bilbo kernel: __qdisc_run+0x253/0x480 Apr 23 08:08:32 bilbo kernel: ? timerqueue_del+0x2c/0x40 Apr 23 08:08:32 bilbo kernel: qdisc_run+0x15/0x30 Apr 23 08:08:32 bilbo kernel: net_tx_action+0x182/0x1b0 Apr 23 08:08:32 bilbo kernel: handle_softirqs+0x102/0x240 Apr 23 08:08:32 bilbo kernel: __irq_exit_rcu+0x3e/0xb0 Apr 23 08:08:32 bilbo kernel: sysvec_apic_timer_interrupt+0x5b/0x70 Apr 23 08:08:32 bilbo kernel: </IRQ> Apr 23 08:08:32 bilbo kernel: <TASK> Apr 23 08:08:32 bilbo kernel: asm_sysvec_apic_timer_interrupt+0x16/0x20 Apr 23 08:08:32 bilbo kernel: RIP: 0010:cpuidle_enter_state+0x126/0x220 Apr 23 08:08:32 bilbo kernel: Code: 18 4c 6f 00 85 c0 7e 0b 8b 73 04 83 cf ff e8 a1 22 e5 ff 31 ff e8 9a 2e 98 ff 45 84 ff 74 07 31 ff e8 0e 58 9d ff fb 45 85 ed <0f> 88 cc 00 00 00 49 63 c5 48 8b 3c 24 48 6b c8 68 48 6b d0 30 49 Apr 23 08:08:32 bilbo kernel: RSP: 0018:ffffffff81e03e40 EFLAGS: 00000202 Apr 23 08:08:32 bilbo kernel: RAX: ffff88842ec00000 RBX: ffff8881008d8000 RCX: 0000000000000000 Apr 23 08:08:32 bilbo kernel: RDX: 00000273bad0f26e RSI: fffffff6533d45e7 RDI: 0000000000000000 Apr 23 08:08:32 bilbo kernel: RBP: 0000000000000002 R08: 0000000000000002 R09: 000002b2b12dc100 Apr 23 08:08:32 bilbo kernel: R10: 0000000000000006 R11: 0000000000000020 R12: ffffffff81f98280 Apr 23 08:08:32 bilbo kernel: R13: 0000000000000002 R14: 00000273bad0f26e R15: 0000000000000000 Apr 23 08:08:32 bilbo kernel: cpuidle_enter+0x2a/0x40 Apr 23 08:08:32 bilbo kernel: do_idle+0x12d/0x1a0 Apr 23 08:08:32 bilbo kernel: cpu_startup_entry+0x29/0x30 Apr 23 08:08:32 bilbo kernel: rest_init+0xbc/0xc0 Apr 23 08:08:32 bilbo kernel: start_kernel+0x630/0x630 Apr 23 08:08:32 bilbo kernel: x86_64_start_reservations+0x25/0x30 Apr 23 08:08:32 bilbo kernel: x86_64_start_kernel+0x73/0x80 Apr 23 08:08:32 bilbo kernel: common_startup_64+0x12c/0x138 Apr 23 08:08:32 bilbo kernel: </TASK> Apr 23 08:08:32 bilbo kernel: ---[ end trace 0000000000000000 ]--- Apr 23 08:08:35 bilbo kernel: AIF:UNPRIV TCP packet: IN=ppp0 OUT= MAC= SRC=23.94.171.218 DST=82.68.155.94 LEN=44 TOS=0x00 PREC=0x00 TTL=245 ID=16537 PROTO=TCP SPT=49012 DPT=25634 WINDOW=1024 RES=0x00 SYN URGP=0
Apr 23 10:41:36 bilbo kernel: ------------[ cut here ]------------ Apr 23 10:41:36 bilbo kernel: WARNING: CPU: 1 PID: 0 at htb_deactivate+0xd/0x30 [sch_htb] Apr 23 10:41:36 bilbo kernel: Modules linked in: sch_htb cls_u32 sch_ingress sch_cake ifb act_mirred xt_hl xt_nat ts_bm xt_string xt_TARPIT(O) xt_CT xt_tcpudp xt_helper nf_nat_ftp nf_conntrack_ftp ip6t_rt ip6table_nat xt_MASQUERADE iptable_nat nf_nat xt_TCPMSS xt_LOG nf_log_syslog ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 ip6table_raw iptable_raw ip6table_mangle iptable_mangle xt_multiport xt_state xt_limit xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6table_filter ip6_tables iptable_filter ip_tables x_tables pppoe pppox ppp_generic binfmt_misc tun slhc netconsole af_packet bridge stp llc ctr ccm dm_crypt radeon ath9k drm_client_lib ath9k_common video ath9k_hw wmi drm_exec drm_suballoc_helper snd_hda_codec_realtek drm_ttm_helper snd_hda_codec_generic snd_hda_codec_hdmi ath syscopyarea snd_hda_scodec_component ttm pl2303 snd_hda_intel usbserial mac80211 sysfillrect snd_intel_dspcfg sysimgblt snd_hda_codec fb_sys_fops drm_display_helper drm_kms_helper snd_hda_co re agpgart snd_pcm cfbfillrect cfbimgblt snd_timer Apr 23 10:41:36 bilbo kernel: cfg80211 fb_io_fops cdc_acm cfbcopyarea aesni_intel i2c_algo_bit e1000 crypto_simd snd fb cryptd at24 libarc4 regmap_i2c font fam15h_power soundcore acpi_cpufreq k10temp evdev nfsd sch_fq_codel auth_rpcgss lockd drm grace sunrpc drm_panel_orientation_quirks backlight fuse loop configfs nfnetlink usbhid xhci_pci ohci_pci xhci_hcd ohci_hcd ehci_pci ehci_hcd usbcore sha512_ssse3 sha256_ssse3 sha1_ssse3 sha1_generic gf128mul usb_common dm_mirror dm_region_hash dm_log cpuid i2c_piix4 i2c_smbus i2c_dev i2c_core it87 hwmon_vid msr dmi_sysfs autofs4 Apr 23 10:41:36 bilbo kernel: CPU: 1 UID: 0 PID: 0 Comm: swapper/1 Tainted: G W O 6.14.3-dirty #23 Apr 23 10:41:36 bilbo kernel: Tainted: [W]=WARN, [O]=OOT_MODULE Apr 23 10:41:36 bilbo kernel: Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./970A-DS3P, BIOS FD 02/26/2016 Apr 23 10:41:36 bilbo kernel: RIP: 0010:htb_deactivate+0xd/0x30 [sch_htb] Apr 23 10:41:36 bilbo kernel: Code: d4 45 21 a4 87 08 01 00 00 48 83 c4 18 5b 5d 41 5c 41 5d 41 5e 41 5f e9 c1 c5 a7 e0 90 53 83 be a8 01 00 00 00 48 89 f3 75 02 <0f> 0b 48 89 de e8 29 fe ff ff 31 c0 89 83 a8 01 00 00 5b e9 9b c5 Apr 23 10:41:36 bilbo kernel: RSP: 0018:ffffc9000010ce50 EFLAGS: 00010246 Apr 23 10:41:36 bilbo kernel: RAX: ffff8881aab77800 RBX: ffff8881b7368400 RCX: ffff8881b73685c8 Apr 23 10:41:36 bilbo kernel: RDX: ffff8881b7368400 RSI: ffff8881b7368400 RDI: ffff88811c27a180 Apr 23 10:41:36 bilbo kernel: RBP: 0000000000000000 R08: ffff88811c27a2b0 R09: 00000000b37f4031 Apr 23 10:41:36 bilbo kernel: R10: 0000000000003819 R11: ffffc9000010cff8 R12: 0000000000000000 Apr 23 10:41:36 bilbo kernel: R13: ffff8881b7368400 R14: 00000ace389b7f34 R15: 0000000000000000 Apr 23 10:41:36 bilbo kernel: FS: 0000000000000000(0000) GS:ffff88842ec80000(0000) knlGS:0000000000000000 Apr 23 10:41:36 bilbo kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Apr 23 10:41:36 bilbo kernel: CR2: 00007f0030446000 CR3: 00000002c2cd6000 CR4: 00000000000406f0 Apr 23 10:41:36 bilbo kernel: Call Trace: Apr 23 10:41:36 bilbo kernel: <IRQ> Apr 23 10:41:36 bilbo kernel: htb_dequeue+0x3f1/0x5a0 [sch_htb] Apr 23 10:41:36 bilbo kernel: __qdisc_run+0x253/0x480 Apr 23 10:41:36 bilbo kernel: ? timerqueue_del+0x2c/0x40 Apr 23 10:41:36 bilbo kernel: qdisc_run+0x15/0x30 Apr 23 10:41:36 bilbo kernel: net_tx_action+0x182/0x1b0 Apr 23 10:41:36 bilbo kernel: handle_softirqs+0x102/0x240 Apr 23 10:41:36 bilbo kernel: __irq_exit_rcu+0x3e/0xb0 Apr 23 10:41:36 bilbo kernel: sysvec_apic_timer_interrupt+0x5b/0x70 Apr 23 10:41:36 bilbo kernel: </IRQ> Apr 23 10:41:36 bilbo kernel: <TASK> Apr 23 10:41:36 bilbo kernel: asm_sysvec_apic_timer_interrupt+0x16/0x20 Apr 23 10:41:36 bilbo kernel: RIP: 0010:acpi_safe_halt+0x22/0x30 Apr 23 10:41:36 bilbo kernel: Code: 0f 1f 84 00 00 00 00 00 65 48 8b 05 b8 38 71 7e 48 8b 00 a8 08 75 14 8b 05 a3 92 bb 00 85 c0 7e 07 0f 00 2d 20 4f 15 00 fb f4 <fa> e9 18 77 00 00 0f 1f 84 00 00 00 00 00 8a 47 08 3c 01 75 05 e9 Apr 23 10:41:36 bilbo kernel: RSP: 0018:ffffc900000c7e80 EFLAGS: 00000246 Apr 23 10:41:36 bilbo kernel: RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffff88842ec80000 Apr 23 10:41:36 bilbo kernel: RDX: ffff888100ddc864 RSI: ffff888100ddc800 RDI: ffff888100ddc864 Apr 23 10:41:36 bilbo kernel: RBP: 0000000000000001 R08: 0000000000000001 R09: 00000acdfd2a9600 Apr 23 10:41:36 bilbo kernel: R10: 0000000000000006 R11: 0000000000000020 R12: ffffffff81f98280 Apr 23 10:41:36 bilbo kernel: R13: ffffffff81f982e8 R14: ffffffff81f98300 R15: 0000000000000000 Apr 23 10:41:36 bilbo kernel: acpi_idle_enter+0x8f/0xa0 Apr 23 10:41:36 bilbo kernel: cpuidle_enter_state+0xb3/0x220 Apr 23 10:41:36 bilbo kernel: cpuidle_enter+0x2a/0x40 Apr 23 10:41:36 bilbo kernel: do_idle+0x12d/0x1a0 Apr 23 10:41:36 bilbo kernel: cpu_startup_entry+0x29/0x30 Apr 23 10:41:36 bilbo kernel: start_secondary+0xed/0xf0 Apr 23 10:41:36 bilbo kernel: common_startup_64+0x12c/0x138 Apr 23 10:41:36 bilbo kernel: </TASK> Apr 23 10:41:36 bilbo kernel: ---[ end trace 0000000000000000 ]---
$ scripts/decode_stacktrace.sh vmlinux
Apr 23 10:41:36 bilbo kernel: Modules linked in: sch_htb cls_u32 sch_ingress sch_cake ifb act_mirred xt_hl xt_nat ts_bm xt_string xt_TARPIT(O) xt_CT xt_tcpudp xt_helper nf_nat_ftp nf_conntrack_ftp ip6t_rt ip6table_nat xt_MASQUERADE iptable_nat nf_nat xt_TCPMSS xt_LOG nf_log_syslog ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 ip6table_raw iptable_raw ip6table_mangle iptable_mangle xt_multiport xt_state xt_limit xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6table_filter ip6_tables iptable_filter ip_tables x_tables pppoe pppox ppp_generic binfmt_misc tun slhc netconsole af_packet bridge stp llc ctr ccm dm_crypt radeon ath9k drm_client_lib ath9k_common video ath9k_hw wmi drm_exec drm_suballoc_helper snd_hda_codec_realtek drm_ttm_helper snd_hda_codec_generic snd_hda_codec_hdmi ath syscopyarea snd_hda_scodec_component ttm pl2303 snd_hda_intel usbserial mac80211 sysfillrect snd_intel_dspcfg sysimgblt snd_hda_codec fb_sys_fops drm_display_helper drm_kms_helper snd_hda_co re agpgart snd_pcm cfbfillrect cfbimgblt snd_timer Apr 23 10:41:36 bilbo kernel: cfg80211 fb_io_fops cdc_acm cfbcopyarea aesni_intel i2c_algo_bit e1000 crypto_simd snd fb cryptd at24 libarc4 regmap_i2c font fam15h_power soundcore acpi_cpufreq k10temp evdev nfsd sch_fq_codel auth_rpcgss lockd drm grace sunrpc drm_panel_orientation_quirks backlight fuse loop configfs nfnetlink usbhid xhci_pci ohci_pci xhci_hcd ohci_hcd ehci_pci ehci_hcd usbcore sha512_ssse3 sha256_ssse3 sha1_ssse3 sha1_generic gf128mul usb_common dm_mirror dm_region_hash dm_log cpuid i2c_piix4 i2c_smbus i2c_dev i2c_core it87 hwmon_vid msr dmi_sysfs autofs4 Apr 23 10:41:36 bilbo kernel: CPU: 1 UID: 0 PID: 0 Comm: swapper/1 Tainted: G W O 6.14.3-dirty #23 Apr 23 10:41:36 bilbo kernel: Tainted: [W]=WARN, [O]=OOT_MODULE Apr 23 10:41:36 bilbo kernel: Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./970A-DS3P, BIOS FD 02/26/2016 Apr 23 10:41:36 bilbo kernel: RIP: 0010:htb_deactivate (net/sched/sch_htb.c:613 (discriminator 1)) sch_htb Apr 23 10:41:36 bilbo kernel: Code: d4 45 21 a4 87 08 01 00 00 48 83 c4 18 5b 5d 41 5c 41 5d 41 5e 41 5f e9 c1 c5 a7 e0 90 53 83 be a8 01 00 00 00 48 89 f3 75 02 <0f> 0b 48 89 de e8 29 fe ff ff 31 c0 89 83 a8 01 00 00 5b e9 9b c5 All code ======== 0: d4 (bad) 1: 45 21 a4 87 08 01 00 and %r12d,0x108(%r15,%rax,4) 8: 00 9: 48 83 c4 18 add $0x18,%rsp d: 5b pop %rbx e: 5d pop %rbp f: 41 5c pop %r12 11: 41 5d pop %r13 13: 41 5e pop %r14 15: 41 5f pop %r15 17: e9 c1 c5 a7 e0 jmp 0xffffffffe0a7c5dd 1c: 90 nop 1d: 53 push %rbx 1e: 83 be a8 01 00 00 00 cmpl $0x0,0x1a8(%rsi) 25: 48 89 f3 mov %rsi,%rbx 28: 75 02 jne 0x2c 2a:* 0f 0b ud2 <-- trapping instruction 2c: 48 89 de mov %rbx,%rsi 2f: e8 29 fe ff ff call 0xfffffffffffffe5d 34: 31 c0 xor %eax,%eax 36: 89 83 a8 01 00 00 mov %eax,0x1a8(%rbx) 3c: 5b pop %rbx 3d: e9 .byte 0xe9 3e: 9b fwait 3f: c5 .byte 0xc5
Code starting with the faulting instruction =========================================== 0: 0f 0b ud2 2: 48 89 de mov %rbx,%rsi 5: e8 29 fe ff ff call 0xfffffffffffffe33 a: 31 c0 xor %eax,%eax c: 89 83 a8 01 00 00 mov %eax,0x1a8(%rbx) 12: 5b pop %rbx 13: e9 .byte 0xe9 14: 9b fwait 15: c5 .byte 0xc5 Apr 23 10:41:36 bilbo kernel: RSP: 0018:ffffc9000010ce50 EFLAGS: 00010246 Apr 23 10:41:36 bilbo kernel: RAX: ffff8881aab77800 RBX: ffff8881b7368400 RCX: ffff8881b73685c8 Apr 23 10:41:36 bilbo kernel: RDX: ffff8881b7368400 RSI: ffff8881b7368400 RDI: ffff88811c27a180 Apr 23 10:41:36 bilbo kernel: RBP: 0000000000000000 R08: ffff88811c27a2b0 R09: 00000000b37f4031 Apr 23 10:41:36 bilbo kernel: R10: 0000000000003819 R11: ffffc9000010cff8 R12: 0000000000000000 Apr 23 10:41:36 bilbo kernel: R13: ffff8881b7368400 R14: 00000ace389b7f34 R15: 0000000000000000 Apr 23 10:41:36 bilbo kernel: FS: 0000000000000000(0000) GS:ffff88842ec80000(0000) knlGS:0000000000000000 Apr 23 10:41:36 bilbo kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Apr 23 10:41:36 bilbo kernel: CR2: 00007f0030446000 CR3: 00000002c2cd6000 CR4: 00000000000406f0 Apr 23 10:41:36 bilbo kernel: Call Trace: Apr 23 10:41:36 bilbo kernel: <IRQ> Apr 23 10:41:36 bilbo kernel: htb_dequeue (./include/net/sch_generic.h:821 (discriminator 1) net/sched/sch_htb.c:702 (discriminator 1) net/sched/sch_htb.c:933 (discriminator 1) net/sched/sch_htb.c:983 (discriminator 1)) sch_htb Apr 23 10:41:36 bilbo kernel: __qdisc_run (net/sched/sch_generic.c:294 net/sched/sch_generic.c:398 net/sched/sch_generic.c:416) Apr 23 10:41:36 bilbo kernel: ? timerqueue_del (lib/timerqueue.c:58) Apr 23 10:41:36 bilbo kernel: qdisc_run (./include/net/pkt_sched.h:128 ./include/net/pkt_sched.h:124) Apr 23 10:41:36 bilbo kernel: net_tx_action (net/core/dev.c:5553) Apr 23 10:41:36 bilbo kernel: handle_softirqs (./arch/x86/include/asm/atomic.h:23 ./include/linux/atomic/atomic-arch-fallback.h:457 ./include/linux/jump_label.h:262 ./include/trace/events/irq.h:142 kernel/softirq.c:562) Apr 23 10:41:36 bilbo kernel: __irq_exit_rcu (kernel/softirq.c:435 kernel/softirq.c:662) Apr 23 10:41:36 bilbo kernel: sysvec_apic_timer_interrupt (arch/x86/kernel/apic/apic.c:1049 (discriminator 35) arch/x86/kernel/apic/apic.c:1049 (discriminator 35)) Apr 23 10:41:36 bilbo kernel: </IRQ> Apr 23 10:41:36 bilbo kernel: <TASK> Apr 23 10:41:36 bilbo kernel: asm_sysvec_apic_timer_interrupt (./arch/x86/include/asm/idtentry.h:574) Apr 23 10:41:36 bilbo kernel: RIP: 0010:acpi_safe_halt (./arch/x86/include/asm/irqflags.h:37 ./arch/x86/include/asm/irqflags.h:114 drivers/acpi/processor_idle.c:112) Apr 23 10:41:36 bilbo kernel: Code: 0f 1f 84 00 00 00 00 00 65 48 8b 05 b8 38 71 7e 48 8b 00 a8 08 75 14 8b 05 a3 92 bb 00 85 c0 7e 07 0f 00 2d 20 4f 15 00 fb f4 <fa> e9 18 77 00 00 0f 1f 84 00 00 00 00 00 8a 47 08 3c 01 75 05 e9 All code ======== 0: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1) 7: 00 8: 65 48 8b 05 b8 38 71 mov %gs:0x7e7138b8(%rip),%rax # 0x7e7138c8 f: 7e 10: 48 8b 00 mov (%rax),%rax 13: a8 08 test $0x8,%al 15: 75 14 jne 0x2b 17: 8b 05 a3 92 bb 00 mov 0xbb92a3(%rip),%eax # 0xbb92c0 1d: 85 c0 test %eax,%eax 1f: 7e 07 jle 0x28 21: 0f 00 2d 20 4f 15 00 verw 0x154f20(%rip) # 0x154f48 28: fb sti 29: f4 hlt 2a:* fa cli <-- trapping instruction 2b: e9 18 77 00 00 jmp 0x7748 30: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1) 37: 00 38: 8a 47 08 mov 0x8(%rdi),%al 3b: 3c 01 cmp $0x1,%al 3d: 75 05 jne 0x44 3f: e9 .byte 0xe9
Code starting with the faulting instruction =========================================== 0: fa cli 1: e9 18 77 00 00 jmp 0x771e 6: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1) d: 00 e: 8a 47 08 mov 0x8(%rdi),%al 11: 3c 01 cmp $0x1,%al 13: 75 05 jne 0x1a 15: e9 .byte 0xe9 Apr 23 10:41:36 bilbo kernel: RSP: 0018:ffffc900000c7e80 EFLAGS: 00000246 Apr 23 10:41:36 bilbo kernel: RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffff88842ec80000 Apr 23 10:41:36 bilbo kernel: RDX: ffff888100ddc864 RSI: ffff888100ddc800 RDI: ffff888100ddc864 Apr 23 10:41:36 bilbo kernel: RBP: 0000000000000001 R08: 0000000000000001 R09: 00000acdfd2a9600 Apr 23 10:41:36 bilbo kernel: R10: 0000000000000006 R11: 0000000000000020 R12: ffffffff81f98280 Apr 23 10:41:36 bilbo kernel: R13: ffffffff81f982e8 R14: ffffffff81f98300 R15: 0000000000000000 Apr 23 10:41:36 bilbo kernel: acpi_idle_enter (drivers/acpi/processor_idle.c:705) Apr 23 10:41:36 bilbo kernel: cpuidle_enter_state (drivers/cpuidle/cpuidle.c:268) Apr 23 10:41:36 bilbo kernel: cpuidle_enter (drivers/cpuidle/cpuidle.c:391 (discriminator 2)) Apr 23 10:41:36 bilbo kernel: do_idle (kernel/sched/idle.c:234 kernel/sched/idle.c:325) Apr 23 10:41:36 bilbo kernel: cpu_startup_entry (kernel/sched/idle.c:422) Apr 23 10:41:36 bilbo kernel: start_secondary (arch/x86/kernel/smpboot.c:315) Apr 23 10:41:36 bilbo kernel: common_startup_64 (arch/x86/kernel/head_64.S:421) Apr 23 10:41:36 bilbo kernel: </TASK> Apr 23 10:41:36 bilbo kernel: ---[ end trace 0000000000000000 ]---
On Wed, Apr 23, 2025 at 10:51:49AM +0100, Alan J. Wylie wrote:
On Tue, 22 Apr 2025 14:49:27 -0700 Cong Wang xiyou.wangcong@gmail.com wrote:
Although I am still trying to understand the NULL pointer, which seems likely from:
478 if (p->inner.clprio[prio].ptr == cl->node + prio) { 479 /* we are removing child which is pointed to from 480 * parent feed - forget the pointer but remember 481 * classid 482 */ 483 p->inner.clprio[prio].last_ptr_id = cl->common.classid; 484 p->inner.clprio[prio].ptr = NULL; 485 }
Does the following patch work? I mean not just fixing the crash, but also not causing any other problem.
diff --git a/net/sched/sch_htb.c b/net/sched/sch_htb.c index 4b9a639b642e..0cdc778fddef 100644 --- a/net/sched/sch_htb.c +++ b/net/sched/sch_htb.c @@ -348,7 +348,8 @@ static void htb_add_to_wait_tree(struct htb_sched *q, */ static inline void htb_next_rb_node(struct rb_node **n) {
- *n = rb_next(*n);
- if (*n)
*n = rb_next(*n);} /**
There's been three of these:
Apr 23 08:08:32 bilbo kernel: WARNING: CPU: 0 PID: 0 at htb_deactivate+0xd/0x30 [sch_htb] Apr 23 08:08:32 bilbo kernel: WARNING: CPU: 0 PID: 0 at htb_deactivate+0xd/0x30 [sch_htb] Apr 23 10:41:36 bilbo kernel: WARNING: CPU: 1 PID: 0 at htb_deactivate+0xd/0x30 [sch_htb]
But no panic.
I've run scripts/decode.sh on the last one.
Thanks a lot for testing! This helped a lot to verify how far we can go beyond the panic and what I still missed. To me it looks a bit complicated for -stable if we make everything idempotent along the path.
Do you mind testing the following one instead? Please revert the above one for htb_next_rb_node(). I think maybe this is the safest fix we could have for -stable.
Thanks!
--------->
diff --git a/net/sched/sch_htb.c b/net/sched/sch_htb.c index 4b9a639b642e..3786abbdc4c3 100644 --- a/net/sched/sch_htb.c +++ b/net/sched/sch_htb.c @@ -1487,7 +1487,8 @@ static void htb_qlen_notify(struct Qdisc *sch, unsigned long arg)
if (!cl->prio_activity) return; - htb_deactivate(qdisc_priv(sch), cl); + if (!cl->leaf.q->q.qlen) + htb_deactivate(qdisc_priv(sch), cl); }
static inline int htb_parent_last_child(struct htb_class *cl)
On Wed, 23 Apr 2025 12:32:58 -0700 Cong Wang xiyou.wangcong@gmail.com wrote:
On Wed, Apr 23, 2025 at 10:51:49AM +0100, Alan J. Wylie wrote:
On Tue, 22 Apr 2025 14:49:27 -0700 Cong Wang xiyou.wangcong@gmail.com wrote:
Although I am still trying to understand the NULL pointer, which seems likely from:
478 if (p->inner.clprio[prio].ptr == cl->node + prio) { 479 /* we are removing child which is pointed to from 480 * parent feed - forget the pointer but remember 481 * classid 482 */ 483 p->inner.clprio[prio].last_ptr_id = cl->common.classid; 484 p->inner.clprio[prio].ptr = NULL; 485 }
Does the following patch work? I mean not just fixing the crash, but also not causing any other problem.
diff --git a/net/sched/sch_htb.c b/net/sched/sch_htb.c index 4b9a639b642e..0cdc778fddef 100644 --- a/net/sched/sch_htb.c +++ b/net/sched/sch_htb.c @@ -348,7 +348,8 @@ static void htb_add_to_wait_tree(struct htb_sched *q, */ static inline void htb_next_rb_node(struct rb_node **n) {
- *n = rb_next(*n);
- if (*n)
*n = rb_next(*n);} /**
There's been three of these:
Apr 23 08:08:32 bilbo kernel: WARNING: CPU: 0 PID: 0 at htb_deactivate+0xd/0x30 [sch_htb] Apr 23 08:08:32 bilbo kernel: WARNING: CPU: 0 PID: 0 at htb_deactivate+0xd/0x30 [sch_htb] Apr 23 10:41:36 bilbo kernel: WARNING: CPU: 1 PID: 0 at htb_deactivate+0xd/0x30 [sch_htb]
But no panic.
I've run scripts/decode.sh on the last one.
Thanks a lot for testing! This helped a lot to verify how far we can go beyond the panic and what I still missed. To me it looks a bit complicated for -stable if we make everything idempotent along the path.
Do you mind testing the following one instead? Please revert the above one for htb_next_rb_node(). I think maybe this is the safest fix we could have for -stable.
Thanks!
--------->
diff --git a/net/sched/sch_htb.c b/net/sched/sch_htb.c index 4b9a639b642e..3786abbdc4c3 100644 --- a/net/sched/sch_htb.c +++ b/net/sched/sch_htb.c @@ -1487,7 +1487,8 @@ static void htb_qlen_notify(struct Qdisc *sch, unsigned long arg) if (!cl->prio_activity) return;
- htb_deactivate(qdisc_priv(sch), cl);
- if (!cl->leaf.q->q.qlen)
htb_deactivate(qdisc_priv(sch), cl);} static inline int htb_parent_last_child(struct htb_class *cl)
With that patch, on top of origin/linux-6.14.y and git cherry-pick 5ba8b837b522d7051ef81bacf3d95383ff8edce5
we're back to panics again. No WARNINGs in the log.
All at htb_dequeue+0x42f/0x610 [sch_htb]
I've decoded the last one.
reboot system boot 6.14.3-00001-ge4 Thu Apr 24 05:41 still running reboot system boot 6.14.3-00001-ge4 Thu Apr 24 05:32 still running reboot system boot 6.14.3-00001-ge4 Thu Apr 24 02:06 still running reboot system boot 6.14.3-00001-ge4 Thu Apr 24 00:35 still running reboot system boot 6.14.3-00001-ge4 Wed Apr 23 23:12 still running
BUG: kernel NULL pointer dereference, address: 0000000000000000 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 0 P4D 0 Oops: Oops: 0000 [#1] PREEMPT SMP NOPTI CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G O 6.14.3-00001-ge44dd93d7907-dirty #24 Tainted: [O]=OOT_MODULE Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./970A-DS3P, BIOS FD 02/26/2016 RIP: 0010:rb_next+0x0/0x50 Code: e8 d5 fa ff ff 5b 4c 89 e0 5d 41 5c 41 5d 41 5e e9 85 73 01 00 5b 5d 41 5c 41 5d 41 5e e9 38 76 01 00 0f 1f 84 00 00 00 00 00 <48> 3b 3f 48 89 f8 74 38 48 8b 57 08 48 85 d2 74 11 48 89 d0 48 8b RSP: 0018:ffffc90000003e50 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff888219993000 RCX: ffff888219993180 RDX: ffff888177830400 RSI: ffff888313064ce8 RDI: 0000000000000000 RBP: 0000000000000000 R08: ffff8882199932b0 R09: 00000000e928705c R10: 0000000000002aef R11: ffffc90000003ff8 R12: ffff888177830400 R13: ffff8882199932b8 R14: 000003a4dd5c3455 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88842ec00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 00000001572e8000 CR4: 00000000000406f0 Call Trace: <IRQ> htb_dequeue+0x42f/0x610 [sch_htb] __qdisc_run+0x253/0x480 ? timerqueue_del+0x2c/0x40 qdisc_run+0x15/0x30 net_tx_action+0x182/0x1b0 handle_softirqs+0x102/0x240 __irq_exit_rcu+0x3e/0xb0 sysvec_apic_timer_interrupt+0x5b/0x70 </IRQ> <TASK> asm_sysvec_apic_timer_interrupt+0x16/0x20 RIP: 0010:acpi_safe_halt+0x22/0x30 Code: 0f 1f 84 00 00 00 00 00 65 48 8b 05 b8 38 71 7e 48 8b 00 a8 08 75 14 8b 05 a3 92 bb 00 85 c0 7e 07 0f 00 2d 60 4f 15 00 fb f4 <fa> e9 18 77 00 00 0f 1f 84 00 00 00 00 00 8a 47 08 3c 01 75 05 e9 RSP: 0018:ffffffff81e03e28 EFLAGS: 00000246 RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffff88842ec00000 RDX: ffff888100ddc864 RSI: ffff888100ddc800 RDI: ffff888100ddc864 RBP: 0000000000000001 R08: 0000000000000001 R09: 000003e3dca10680 R10: 0000000000000006 R11: 0000000000000020 R12: ffffffff81f98280 R13: ffffffff81f982e8 R14: ffffffff81f98300 R15: 0000000000000000 acpi_idle_enter+0x8f/0xa0 cpuidle_enter_state+0xb3/0x220 cpuidle_enter+0x2a/0x40 do_idle+0x12d/0x1a0 cpu_startup_entry+0x29/0x30 rest_init+0xbc/0xc0 start_kernel+0x630/0x630 x86_64_start_reservations+0x25/0x30 x86_64_start_kernel+0x73/0x80 common_startup_64+0x12c/0x138 </TASK> Modules linked in: sch_htb cls_u32 sch_ingress sch_cake ifb act_mirred xt_hl xt_nat ts_bm xt_string xt_TARPIT(O) xt_CT xt_tcpudp xt_helper nf_nat_ftp nf_conntrack_ftp ip6t_rt ip6table_nat xt_MASQUERADE iptable_nat nf_nat xt_TCPMSS xt_LOG nf_log_syslog ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 ip6table_raw iptable_raw ip6table_mangle iptable_mangle xt_multiport xt_state xt_limit xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6table_filter ip6_tables iptable_filter ip_tables x_tables binfmt_misc pppoe tun pppox ppp_generic slhc netconsole af_packet bridge stp llc ctr ccm dm_crypt radeon drm_client_lib video wmi ath9k drm_exec drm_suballoc_helper drm_ttm_helper syscopyarea ath9k_common ttm ath9k_hw sysfillrect sysimgblt fb_sys_fops drm_display_helper ath drm_kms_helper mac80211 agpgart snd_hda_codec_realtek cfbfillrect snd_hda_codec_generic cfbimgblt snd_hda_codec_hdmi snd_hda_scodec_component snd_hda_intel fb_io_fops snd_intel_dspcfg pl2303 cfbcopyarea snd_hda_cod ec usbserial i2c_algo_bit snd_hda_core fb cfg80211 snd_pcm cdc_acm snd_timer font snd aesni_intel at24 e1000 crypto_simd soundcore cryptd k10temp regmap_i2c libarc4 acpi_cpufreq fam15h_power evdev nfsd sch_fq_codel auth_rpcgss lockd drm grace sunrpc drm_panel_orientation_quirks fuse backlight loop configfs nfnetlink usbhid xhci_pci ohci_pci xhci_hcd ohci_hcd ehci_pci ehci_hcd usbcore sha512_ssse3 sha256_ssse3 sha1_ssse3 sha1_generic gf128mul usb_common dm_mirror dm_region_hash dm_log cpuid i2c_piix4 i2c_smbus i2c_dev i2c_core it87 hwmon_vid msr dmi_sysfs autofs4 CR2: 0000000000000000 ---[ end trace 0000000000000000 ]--- RIP: 0010:rb_next+0x0/0x50 Code: e8 d5 fa ff ff 5b 4c 89 e0 5d 41 5c 41 5d 41 5e e9 85 73 01 00 5b 5d 41 5c 41 5d 41 5e e9 38 76 01 00 0f 1f 84 00 00 00 00 00 <48> 3b 3f 48 89 f8 74 38 48 8b 57 08 48 85 d2 74 11 48 89 d0 48 8b RSP: 0018:ffffc90000003e50 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff888219993000 RCX: ffff888219993180 RDX: ffff888177830400 RSI: ffff888313064ce8 RDI: 0000000000000000 RBP: 0000000000000000 R08: ffff8882199932b0 R09: 00000000e928705c R10: 0000000000002aef R11: ffffc90000003ff8 R12: ffff888177830400 R13: ffff8882199932b8 R14: 000003a4dd5c3455 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88842ec00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 00000001572e8000 CR4: 00000000000406f0 Kernel panic - not syncing: Fatal exception in interrupt Kernel Offset: disabled Rebooting in 3 seconds.. netconsole: network logging started Wed 23 Apr 23:13:45 BST 2025
BUG: kernel NULL pointer dereference, address: 0000000000000000 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 13978b067 P4D 13978b067 PUD 13979b067 PMD 0 Oops: Oops: 0000 [#1] PREEMPT SMP NOPTI CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G O 6.14.3-00001-ge44dd93d7907-dirty #24 Tainted: [O]=OOT_MODULE Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./970A-DS3P, BIOS FD 02/26/2016 RIP: 0010:rb_next+0x0/0x50 Code: e8 d5 fa ff ff 5b 4c 89 e0 5d 41 5c 41 5d 41 5e e9 85 73 01 00 5b 5d 41 5c 41 5d 41 5e e9 38 76 01 00 0f 1f 84 00 00 00 00 00 <48> 3b 3f 48 89 f8 74 38 48 8b 57 08 48 85 d2 74 11 48 89 d0 48 8b RSP: 0018:ffffc90000003e50 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff88811a783000 RCX: ffff88811a783180 RDX: ffff8882bd40e400 RSI: ffff8882dbc58ee8 RDI: 0000000000000000 RBP: 0000000000000000 R08: ffff88811a7832b0 R09: 000000002068e72f R10: 0000000000000cb0 R11: ffffc90000003ff8 R12: ffff8882bd40e400 R13: ffff88811a7832b8 R14: 00000481df377e05 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88842ec00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 0000000104ef2000 CR4: 00000000000406f0 Call Trace: <IRQ> htb_dequeue+0x42f/0x610 [sch_htb] __qdisc_run+0x253/0x480 ? timerqueue_del+0x2c/0x40 qdisc_run+0x15/0x30 net_tx_action+0x182/0x1b0 handle_softirqs+0x102/0x240 __irq_exit_rcu+0x3e/0xb0 sysvec_apic_timer_interrupt+0x5b/0x70 </IRQ> <TASK> asm_sysvec_apic_timer_interrupt+0x16/0x20 RIP: 0010:cpuidle_enter_state+0x126/0x220 Code: 18 4c 6f 00 85 c0 7e 0b 8b 73 04 83 cf ff e8 a1 22 e5 ff 31 ff e8 9a 2e 98 ff 45 84 ff 74 07 31 ff e8 0e 58 9d ff fb 45 85 ed <0f> 88 cc 00 00 00 49 63 c5 48 8b 3c 24 48 6b c8 68 48 6b d0 30 49 RSP: 0018:ffffffff81e03e40 EFLAGS: 00000202 RAX: ffff88842ec00000 RBX: ffff8881008e6000 RCX: 0000000000000000 RDX: 00000481a5333d42 RSI: fffffff65224a3d4 RDI: 0000000000000000 RBP: 0000000000000002 R08: 0000000000000002 R09: 00000481a3a82cc0 R10: 0000000000000006 R11: 0000000000000020 R12: ffffffff81f98280 R13: 0000000000000002 R14: 00000481a5333d42 R15: 0000000000000000 cpuidle_enter+0x2a/0x40 do_idle+0x12d/0x1a0 cpu_startup_entry+0x29/0x30 rest_init+0xbc/0xc0 start_kernel+0x630/0x630 x86_64_start_reservations+0x25/0x30 x86_64_start_kernel+0x73/0x80 common_startup_64+0x12c/0x138 </TASK> Modules linked in: sch_htb cls_u32 sch_ingress sch_cake ifb act_mirred xt_hl xt_nat ts_bm xt_string xt_TARPIT(O) xt_CT xt_tcpudp xt_helper nf_nat_ftp nf_conntrack_ftp ip6t_rt ip6table_nat xt_MASQUERADE iptable_nat nf_nat xt_TCPMSS xt_LOG nf_log_syslog ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 ip6table_raw iptable_raw ip6table_mangle iptable_mangle xt_multiport xt_state xt_limit xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6table_filter ip6_tables iptable_filter ip_tables x_tables pppoe tun pppox binfmt_misc ppp_generic slhc netconsole af_packet bridge stp llc ctr ccm dm_crypt radeon drm_client_lib video wmi drm_exec ath9k drm_suballoc_helper drm_ttm_helper syscopyarea ath9k_common ttm ath9k_hw sysfillrect sysimgblt fb_sys_fops ath snd_hda_codec_realtek drm_display_helper drm_kms_helper snd_hda_codec_generic snd_hda_codec_hdmi pl2303 snd_hda_scodec_component mac80211 usbserial agpgart snd_hda_intel snd_intel_dspcfg cfbfillrect cfbimgblt snd_hda_codec fb_io_fops cfbcopyarea snd_hda_core i2c_algo_bit snd_pcm fb cfg80211 snd_timer cdc_acm aesni_intel font snd acpi_cpufreq at24 crypto_simd e1000 cryptd libarc4 soundcore k10temp regmap_i2c fam15h_power evdev nfsd sch_fq_codel auth_rpcgss lockd drm grace sunrpc drm_panel_orientation_quirks backlight fuse loop configfs nfnetlink usbhid xhci_pci ohci_pci xhci_hcd ohci_hcd ehci_pci ehci_hcd sha512_ssse3 sha256_ssse3 usbcore sha1_ssse3 sha1_generic gf128mul usb_common dm_mirror dm_region_hash dm_log cpuid i2c_piix4 i2c_smbus i2c_dev i2c_core it87 hwmon_vid msr dmi_sysfs autofs4 CR2: 0000000000000000 ---[ end trace 0000000000000000 ]--- RIP: 0010:rb_next+0x0/0x50 Code: e8 d5 fa ff ff 5b 4c 89 e0 5d 41 5c 41 5d 41 5e e9 85 73 01 00 5b 5d 41 5c 41 5d 41 5e e9 38 76 01 00 0f 1f 84 00 00 00 00 00 <48> 3b 3f 48 89 f8 74 38 48 8b 57 08 48 85 d2 74 11 48 89 d0 48 8b RSP: 0018:ffffc90000003e50 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff88811a783000 RCX: ffff88811a783180 RDX: ffff8882bd40e400 RSI: ffff8882dbc58ee8 RDI: 0000000000000000 RBP: 0000000000000000 R08: ffff88811a7832b0 R09: 000000002068e72f R10: 0000000000000cb0 R11: ffffc90000003ff8 R12: ffff8882bd40e400 R13: ffff88811a7832b8 R14: 00000481df377e05 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88842ec00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 0000000104ef2000 CR4: 00000000000406f0 Kernel panic - not syncing: Fatal exception in interrupt Kernel Offset: disabled Rebooting in 3 seconds.. netconsole: network logging started Thu 24 Apr 00:36:40 BST 2025
BUG: kernel NULL pointer dereference, address: 0000000000000000 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 0 P4D 0 Oops: Oops: 0000 [#1] PREEMPT SMP NOPTI CPU: 1 UID: 0 PID: 0 Comm: swapper/1 Tainted: G O 6.14.3-00001-ge44dd93d7907-dirty #24 Tainted: [O]=OOT_MODULE Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./970A-DS3P, BIOS FD 02/26/2016 RIP: 0010:rb_next+0x0/0x50 Code: e8 d5 fa ff ff 5b 4c 89 e0 5d 41 5c 41 5d 41 5e e9 85 73 01 00 5b 5d 41 5c 41 5d 41 5e e9 38 76 01 00 0f 1f 84 00 00 00 00 00 <48> 3b 3f 48 89 f8 74 38 48 8b 57 08 48 85 d2 74 11 48 89 d0 48 8b RSP: 0018:ffffc9000010ce50 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff88811d22c000 RCX: ffff88811d22c180 RDX: ffff88829c80a400 RSI: ffff88810445c500 RDI: 0000000000000000 RBP: 0000000000000000 R08: ffff88811d22c2b0 R09: 000000003b4da56b R10: 0000000000000db3 R11: 001dcd6500000000 R12: ffff88829c80a400 R13: ffff88811d22c2b8 R14: 000004ed72306e80 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88842ec80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 000000028f056000 CR4: 00000000000406f0 Call Trace: <IRQ> htb_dequeue+0x42f/0x610 [sch_htb] __qdisc_run+0x253/0x480 ? timerqueue_del+0x2c/0x40 qdisc_run+0x15/0x30 net_tx_action+0x182/0x1b0 handle_softirqs+0x102/0x240 __irq_exit_rcu+0x3e/0xb0 sysvec_apic_timer_interrupt+0x5b/0x70 </IRQ> <TASK> asm_sysvec_apic_timer_interrupt+0x16/0x20 RIP: 0010:cpuidle_enter_state+0x126/0x220 Code: 18 4c 6f 00 85 c0 7e 0b 8b 73 04 83 cf ff e8 a1 22 e5 ff 31 ff e8 9a 2e 98 ff 45 84 ff 74 07 31 ff e8 0e 58 9d ff fb 45 85 ed <0f> 88 cc 00 00 00 49 63 c5 48 8b 3c 24 48 6b c8 68 48 6b d0 30 49 RSP: 0018:ffffc900000c7e98 EFLAGS: 00000202 RAX: ffff88842ec80000 RBX: ffff888101c7d400 RCX: 0000000000000000 RDX: 000004ed38350e30 RSI: fffffffc356687b0 RDI: 0000000000000000 RBP: 0000000000000002 R08: 0000000000000002 R09: 000004ed369cb580 R10: 0000000000000006 R11: 0000000000000020 R12: ffffffff81f98280 R13: 0000000000000002 R14: 000004ed38350e30 R15: 0000000000000000 ? cpuidle_enter_state+0x116/0x220 cpuidle_enter+0x2a/0x40 do_idle+0x12d/0x1a0 cpu_startup_entry+0x29/0x30 start_secondary+0xed/0xf0 common_startup_64+0x12c/0x138 </TASK> Modules linked in: sch_htb cls_u32 sch_ingress sch_cake ifb act_mirred xt_hl xt_nat ts_bm xt_string xt_TARPIT(O) xt_CT xt_tcpudp xt_helper nf_nat_ftp nf_conntrack_ftp ip6t_rt ip6table_nat xt_MASQUERADE iptable_nat nf_nat xt_TCPMSS xt_LOG nf_log_syslog ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 ip6table_raw iptable_raw ip6table_mangle iptable_mangle xt_multiport xt_state xt_limit xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6table_filter ip6_tables iptable_filter ip_tables x_tables binfmt_misc pppoe tun pppox ppp_generic slhc netconsole af_packet bridge stp llc ctr ccm dm_crypt radeon ath9k drm_client_lib video wmi ath9k_common drm_exec ath9k_hw drm_suballoc_helper drm_ttm_helper syscopyarea ttm ath sysfillrect snd_hda_codec_realtek sysimgblt snd_hda_codec_generic fb_sys_fops mac80211 snd_hda_codec_hdmi drm_display_helper pl2303 snd_hda_scodec_component snd_hda_intel usbserial drm_kms_helper snd_intel_dspcfg snd_hda_codec agpgart cfbfillrect snd_hda_core cfbimgb lt fb_io_fops cfg80211 cfbcopyarea snd_pcm i2c_algo_bit snd_timer fb aesni_intel cdc_acm snd e1000 font crypto_simd at24 cryptd libarc4 k10temp regmap_i2c soundcore acpi_cpufreq fam15h_power evdev nfsd auth_rpcgss sch_fq_codel lockd drm grace sunrpc drm_panel_orientation_quirks backlight loop fuse configfs nfnetlink usbhid xhci_pci ohci_pci xhci_hcd ohci_hcd ehci_pci ehci_hcd sha512_ssse3 usbcore sha256_ssse3 sha1_ssse3 sha1_generic gf128mul usb_common dm_mirror dm_region_hash dm_log cpuid i2c_piix4 i2c_smbus i2c_dev i2c_core it87 hwmon_vid msr dmi_sysfs autofs4 CR2: 0000000000000000 ---[ end trace 0000000000000000 ]--- RIP: 0010:rb_next+0x0/0x50 Code: e8 d5 fa ff ff 5b 4c 89 e0 5d 41 5c 41 5d 41 5e e9 85 73 01 00 5b 5d 41 5c 41 5d 41 5e e9 38 76 01 00 0f 1f 84 00 00 00 00 00 <48> 3b 3f 48 89 f8 74 38 48 8b 57 08 48 85 d2 74 11 48 89 d0 48 8b RSP: 0018:ffffc9000010ce50 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff88811d22c000 RCX: ffff88811d22c180 RDX: ffff88829c80a400 RSI: ffff88810445c500 RDI: 0000000000000000 RBP: 0000000000000000 R08: ffff88811d22c2b0 R09: 000000003b4da56b R10: 0000000000000db3 R11: 001dcd6500000000 R12: ffff88829c80a400 R13: ffff88811d22c2b8 R14: 000004ed72306e80 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88842ec80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 000000028f056000 CR4: 00000000000406f0 Kernel panic - not syncing: Fatal exception in interrupt Kernel Offset: disabled Rebooting in 3 seconds.. netconsole: network logging started Thu 24 Apr 02:07:13 BST 2025
BUG: kernel NULL pointer dereference, address: 0000000000000000 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 0 P4D 0 Oops: Oops: 0000 [#1] PREEMPT SMP NOPTI CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G O 6.14.3-00001-ge44dd93d7907-dirty #24 Tainted: [O]=OOT_MODULE Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./970A-DS3P, BIOS FD 02/26/2016 RIP: 0010:rb_next+0x0/0x50 Code: e8 d5 fa ff ff 5b 4c 89 e0 5d 41 5c 41 5d 41 5e e9 85 73 01 00 5b 5d 41 5c 41 5d 41 5e e9 38 76 01 00 0f 1f 84 00 00 00 00 00 <48> 3b 3f 48 89 f8 74 38 48 8b 57 08 48 85 d2 74 11 48 89 d0 48 8b RSP: 0018:ffffc90000003e50 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff88813d31a000 RCX: ffff88813d31a180 RDX: ffff8883ff834800 RSI: ffff88810aede6e8 RDI: 0000000000000000 RBP: 0000000000000000 R08: ffff88813d31a2b0 R09: 00000000cdfec307 R10: 0000000000000d0a R11: 001dcd6500000000 R12: ffff8883ff834800 R13: ffff88813d31a2b8 R14: 00000b3836a6dc49 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88842ec00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 00000001507ca000 CR4: 00000000000406f0 Call Trace: <IRQ> htb_dequeue+0x42f/0x610 [sch_htb] __qdisc_run+0x253/0x480 ? timerqueue_del+0x2c/0x40 qdisc_run+0x15/0x30 net_tx_action+0x182/0x1b0 handle_softirqs+0x102/0x240 __irq_exit_rcu+0x3e/0xb0 sysvec_apic_timer_interrupt+0x5b/0x70 </IRQ> <TASK> asm_sysvec_apic_timer_interrupt+0x16/0x20 RIP: 0010:cpuidle_enter_state+0x126/0x220 Code: 18 4c 6f 00 85 c0 7e 0b 8b 73 04 83 cf ff e8 a1 22 e5 ff 31 ff e8 9a 2e 98 ff 45 84 ff 74 07 31 ff e8 0e 58 9d ff fb 45 85 ed <0f> 88 cc 00 00 00 49 63 c5 48 8b 3c 24 48 6b c8 68 48 6b d0 30 49 RSP: 0018:ffffffff81e03e40 EFLAGS: 00000202 RAX: ffff88842ec00000 RBX: ffff8881008dac00 RCX: 0000000000000000 RDX: 00000b37fdd1d950 RSI: fffffffc371b8b62 RDI: 0000000000000000 RBP: 0000000000000002 R08: 0000000000000002 R09: 00000b37fb496cc0 R10: 0000000000000006 R11: 0000000000000020 R12: ffffffff81f98280 R13: 0000000000000002 R14: 00000b37fdd1d950 R15: 0000000000000000 cpuidle_enter+0x2a/0x40 do_idle+0x12d/0x1a0 cpu_startup_entry+0x29/0x30 rest_init+0xbc/0xc0 start_kernel+0x630/0x630 x86_64_start_reservations+0x25/0x30 x86_64_start_kernel+0x73/0x80 common_startup_64+0x12c/0x138 </TASK> Modules linked in: udp_diag sch_htb cls_u32 sch_ingress sch_cake ifb act_mirred xt_hl xt_nat ts_bm xt_string xt_TARPIT(O) xt_CT xt_tcpudp xt_helper nf_nat_ftp nf_conntrack_ftp ip6t_rt ip6table_nat xt_MASQUERADE iptable_nat nf_nat xt_TCPMSS xt_LOG nf_log_syslog ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 ip6table_raw iptable_raw ip6table_mangle iptable_mangle xt_multiport xt_state xt_limit xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6table_filter ip6_tables iptable_filter ip_tables x_tables binfmt_misc pppoe pppox ppp_generic tun slhc netconsole af_packet bridge stp llc ctr ccm dm_crypt radeon ath9k ath9k_common drm_client_lib video ath9k_hw wmi drm_exec drm_suballoc_helper drm_ttm_helper ath syscopyarea ttm sysfillrect mac80211 sysimgblt fb_sys_fops snd_hda_codec_realtek drm_display_helper snd_hda_codec_generic snd_hda_codec_hdmi drm_kms_helper pl2303 snd_hda_scodec_component usbserial snd_hda_intel agpgart snd_intel_dspcfg cfbfillrect snd_hda_codec cfbimgblt f b_io_fops cfbcopyarea cfg80211 snd_hda_core cdc_acm i2c_algo_bit snd_pcm aesni_intel fb snd_timer e1000 at24 font snd crypto_simd acpi_cpufreq k10temp cryptd libarc4 soundcore regmap_i2c fam15h_power evdev nfsd sch_fq_codel auth_rpcgss lockd grace sunrpc drm drm_panel_orientation_quirks loop fuse backlight configfs nfnetlink usbhid xhci_pci ohci_pci xhci_hcd ohci_hcd ehci_pci ehci_hcd sha512_ssse3 sha256_ssse3 sha1_ssse3 sha1_generic usbcore gf128mul usb_common dm_mirror dm_region_hash dm_log cpuid i2c_piix4 i2c_smbus i2c_dev i2c_core it87 hwmon_vid msr dmi_sysfs autofs4 CR2: 0000000000000000 ---[ end trace 0000000000000000 ]--- RIP: 0010:rb_next+0x0/0x50 Code: e8 d5 fa ff ff 5b 4c 89 e0 5d 41 5c 41 5d 41 5e e9 85 73 01 00 5b 5d 41 5c 41 5d 41 5e e9 38 76 01 00 0f 1f 84 00 00 00 00 00 <48> 3b 3f 48 89 f8 74 38 48 8b 57 08 48 85 d2 74 11 48 89 d0 48 8b RSP: 0018:ffffc90000003e50 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff88813d31a000 RCX: ffff88813d31a180 RDX: ffff8883ff834800 RSI: ffff88810aede6e8 RDI: 0000000000000000 RBP: 0000000000000000 R08: ffff88813d31a2b0 R09: 00000000cdfec307 R10: 0000000000000d0a R11: 001dcd6500000000 R12: ffff8883ff834800 R13: ffff88813d31a2b8 R14: 00000b3836a6dc49 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88842ec00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 00000001507ca000 CR4: 00000000000406f0 Kernel panic - not syncing: Fatal exception in interrupt Kernel Offset: disabled Rebooting in 3 seconds.. netconsole: network logging started Thu 24 Apr 05:33:20 BST 2025
BUG: kernel NULL pointer dereference, address: 0000000000000000 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 11d659067 P4D 11d659067 PUD 11d66a067 PMD 0 Oops: Oops: 0000 [#1] PREEMPT SMP NOPTI CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G O 6.14.3-00001-ge44dd93d7907-dirty #24 Tainted: [O]=OOT_MODULE Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./970A-DS3P, BIOS FD 02/26/2016 RIP: 0010:rb_next+0x0/0x50 Code: e8 d5 fa ff ff 5b 4c 89 e0 5d 41 5c 41 5d 41 5e e9 85 73 01 00 5b 5d 41 5c 41 5d 41 5e e9 38 76 01 00 0f 1f 84 00 00 00 00 00 <48> 3b 3f 48 89 f8 74 38 48 8b 57 08 48 85 d2 74 11 48 89 d0 48 8b RSP: 0018:ffffc90000003e50 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff88811c311000 RCX: ffff88811c311180 RDX: ffff888124639c00 RSI: ffff8881cf56d2e8 RDI: 0000000000000000 RBP: 0000000000000000 R08: ffff88811c3112b0 R09: 000000001cdf6fee R10: 0000000000000bad R11: ffffc90000003ff8 R12: ffff888124639c00 R13: ffff88811c3112b8 R14: 00000073b95a79bb R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88842ec00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 000000012e18c000 CR4: 00000000000406f0 Call Trace: <IRQ> htb_dequeue+0x42f/0x610 [sch_htb] __qdisc_run+0x253/0x480 ? timerqueue_del+0x2c/0x40 qdisc_run+0x15/0x30 net_tx_action+0x182/0x1b0 handle_softirqs+0x102/0x240 __irq_exit_rcu+0x3e/0xb0 sysvec_apic_timer_interrupt+0x5b/0x70 </IRQ> <TASK> asm_sysvec_apic_timer_interrupt+0x16/0x20 RIP: 0010:cpuidle_enter_state+0x126/0x220 Code: 18 4c 6f 00 85 c0 7e 0b 8b 73 04 83 cf ff e8 a1 22 e5 ff 31 ff e8 9a 2e 98 ff 45 84 ff 74 07 31 ff e8 0e 58 9d ff fb 45 85 ed <0f> 88 cc 00 00 00 49 63 c5 48 8b 3c 24 48 6b c8 68 48 6b d0 30 49 RSP: 0018:ffffffff81e03e40 EFLAGS: 00000202 RAX: ffff88842ec00000 RBX: ffff888101b26800 RCX: 0000000000000000 RDX: 000000737ef85069 RSI: fffffffc38107585 RDI: 0000000000000000 RBP: 0000000000000002 R08: 0000000000000002 R09: 000000737dcfa800 R10: 0000000000000006 R11: 0000000000000020 R12: ffffffff81f98280 R13: 0000000000000002 R14: 000000737ef85069 R15: 0000000000000000 cpuidle_enter+0x2a/0x40 do_idle+0x12d/0x1a0 cpu_startup_entry+0x29/0x30 rest_init+0xbc/0xc0 start_kernel+0x630/0x630 x86_64_start_reservations+0x25/0x30 x86_64_start_kernel+0x73/0x80 common_startup_64+0x12c/0x138 </TASK> Modules linked in: sch_htb cls_u32 sch_ingress sch_cake ifb act_mirred xt_hl xt_nat ts_bm xt_string xt_TARPIT(O) xt_CT xt_tcpudp xt_helper nf_nat_ftp nf_conntrack_ftp ip6t_rt ip6table_nat xt_MASQUERADE iptable_nat nf_nat xt_TCPMSS xt_LOG nf_log_syslog ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 ip6table_raw iptable_raw ip6table_mangle iptable_mangle xt_multiport xt_state xt_limit xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6table_filter ip6_tables iptable_filter ip_tables x_tables tun binfmt_misc pppoe pppox ppp_generic slhc netconsole af_packet bridge stp llc ctr ccm dm_crypt radeon drm_client_lib video wmi ath9k drm_exec drm_suballoc_helper ath9k_common drm_ttm_helper syscopyarea ath9k_hw ttm sysfillrect sysimgblt ath fb_sys_fops drm_display_helper snd_hda_codec_realtek pl2303 drm_kms_helper snd_hda_codec_generic mac80211 usbserial snd_hda_codec_hdmi snd_hda_scodec_component agpgart snd_hda_intel cfbfillrect snd_intel_dspcfg cfbimgblt snd_hda_codec fb_io_fops cfbcopyarea aesni_intel snd_hda_core i2c_algo_bit cdc_acm cfg80211 fb snd_pcm snd_timer font crypto_simd e1000 at24 snd cryptd libarc4 soundcore acpi_cpufreq regmap_i2c fam15h_power k10temp evdev nfsd sch_fq_codel auth_rpcgss lockd grace sunrpc drm drm_panel_orientation_quirks fuse backlight configfs loop nfnetlink usbhid xhci_pci ohci_pci xhci_hcd ohci_hcd ehci_pci sha512_ssse3 ehci_hcd sha256_ssse3 sha1_ssse3 sha1_generic usbcore gf128mul usb_common dm_mirror dm_region_hash dm_log cpuid i2c_piix4 i2c_smbus i2c_dev i2c_core it87 hwmon_vid msr dmi_sysfs autofs4 CR2: 0000000000000000 ---[ end trace 0000000000000000 ]--- RIP: 0010:rb_next+0x0/0x50 Code: e8 d5 fa ff ff 5b 4c 89 e0 5d 41 5c 41 5d 41 5e e9 85 73 01 00 5b 5d 41 5c 41 5d 41 5e e9 38 76 01 00 0f 1f 84 00 00 00 00 00 <48> 3b 3f 48 89 f8 74 38 48 8b 57 08 48 85 d2 74 11 48 89 d0 48 8b RSP: 0018:ffffc90000003e50 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff88811c311000 RCX: ffff88811c311180 RDX: ffff888124639c00 RSI: ffff8881cf56d2e8 RDI: 0000000000000000 RBP: 0000000000000000 R08: ffff88811c3112b0 R09: 000000001cdf6fee R10: 0000000000000bad R11: ffffc90000003ff8 R12: ffff888124639c00 R13: ffff88811c3112b8 R14: 00000073b95a79bb R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88842ec00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 000000012e18c000 CR4: 00000000000406f0 Kernel panic - not syncing: Fatal exception in interrupt Kernel Offset: disabled Rebooting in 3 seconds.. netconsole: network logging started Thu 24 Apr 05:42:15 BST 2025
$ ./scripts/decode_stacktrace.sh vmlinux BUG: kernel NULL pointer dereference, address: 0000000000000000 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 11d659067 P4D 11d659067 PUD 11d66a067 PMD 0 Oops: Oops: 0000 [#1] PREEMPT SMP NOPTI CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G O 6.14.3-00001-ge44dd93d7907-dirty #24 Tainted: [O]=OOT_MODULE Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./970A-DS3P, BIOS FD 02/26/2016 RIP: 0010:rb_next+0x0/0x50 Code: e8 d5 fa ff ff 5b 4c 89 e0 5d 41 5c 41 5d 41 5e e9 85 73 01 00 5b 5d 41 5c 41 5d 41 5e e9 38 76 01 00 0f 1f 84 00 00 00 00 00 <48> 3b 3f 48 89 f8 74 38 48 8b 57 08 48 85 d2 74 11 48 89 d0 48 8b RSP: 0018:ffffc90000003e50 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff88811c311000 RCX: ffff88811c311180 RDX: ffff888124639c00 RSI: ffff8881cf56d2e8 RDI: 0000000000000000 RBP: 0000000000000000 R08: ffff88811c3112b0 R09: 000000001cdf6fee R10: 0000000000000bad R11: ffffc90000003ff8 R12: ffff888124639BUG: kernel NULL pointer dereference, address: 0000000000000000 c00 R13: ffff88811c3112b8 R14: 00000073b95a79bb R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88842ec00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 000000012e18c000 CR4: 00000000000406f0 Call Trace: <IRQ> htb_dequeue+0x42f/0x610 [sch_htb] __qdisc_run+0x253/0x480 ? timerqueue_del+0x2c/0x40 qdisc_run+0x15/0x30 net_tx_action+0x182/0x1b0 handle_softirqs+0x102/0x240 __irq_exit_rcu+0x3e/0xb0 sysvec_apic#PF: supervisor read access in kernel mode _timer_interrupt+0x5b/0x70 </IRQ> <TASK> asm_sysvec_apic_timer_interrupt+0x16/0x20 RIP: 0010:cpuidle_enter_state+0x126/0x220 Code: 18 4c 6f 00 85 c0 7e 0b 8b 73 04 83 cf ff e8 a1 22 e5 ff 31 ff e8 9a 2e 98 ff 45 84 ff 74 07 31 ff e8 0e 58 9d ff fb 45 85 ed <0f> 88 cc 00 00 00 49 63 c5 48 8b 3c 24 48 6b c8 68 48 6b d0 30 49 RSP: 0018:ffffffff81e03e40 EFLAGS: 00000202 RAX: ffff88842e#PF: error_code(0x0000) - not-present page c00000 RBX: ffff888101b26800 RCX: 0000000000000000 RDX: 000000737ef85069 RSI: fffffffc38107585 RDI: 0000000000000000 RBP: 0000000000000002 R08: 0000000000000002 R09: 000000737dcfa800 R10: 0000000000000006 R11: 0000000000000020 R12: ffffffff81f98280 R13: 0000000000000002 R14: 000000737ef85069 R15: 0000000000000000 cpuidle_enter+0x2a/0x40 do_idle+0x12d/0x1a0 cpu_startup_entry+0x29/0x30 rest_init+0xbc/0xc0 start_kernePGD 11d659067 P4D 11d659067 PUD 11d66a067 PMD 0 l+0x630/0x630 x86_64_start_reservations+0x25/0x30 x86_64_start_kernel+0x73/0x80 common_startup_64+0x12c/0x138 </TASK> Modules linked in: sch_htb cls_u32 sch_ingress sch_cake ifb act_mirred xt_hl xt_nat ts_bm xt_string xt_TARPIT(O) xt_CT xt_tcpudp xt_helper nf_nat_ftp nf_conntrack_ftp ip6t_rt ip6table_nat xt_MASQUERADE iptable_nat nf_nat xt_TCPMSS xt_LOG nf_log_syslog ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 ip6table_raw iptable_raw ip6table_mangle iptable_mangle xt_multiport xt_state xOops: Oops: 0000 [#1] PREEMPT SMP NOPTI t_limit xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6table_filter ip6_tables iptable_filter ip_tables x_tables tun binfmt_misc pppoe pppox ppp_generic slhc netconsole af_packet bridge stp llc ctr ccm dm_crypt radeon drm_client_lib video wmi ath9k drm_exec drm_suballoc_helper ath9k_common drm_ttm_helper syscopyarea ath9k_hw ttm sysfillrect sysimgblt ath fb_sys_fops drm_display_helper snd_hda_codec_realtek pl2303 drm_kms_helper snd_hda_codec_generic mac80211 usbserial snd_hda_codec_hdmi snd_hda_scodCPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G O 6.14.3-00001-ge44dd93d7907-dirty #24 ec_component agpgart snd_hda_intel cfbfillrect snd_intel_dspcfg cfbimgblt snd_hda_codec fb_io_fops cfbcopyarea aesni_intel snd_hda_core i2c_algo_bit cdc_acm cfg80211 fb snd_pcm snd_timer font crypto_simd e1000 at24 snd cryptd libarc4 soundcore acpi_cpufreq regmap_i2c fam15h_power k10temp evdev nfsd sch_fq_codel auth_rpcgss lockd grace sunrpc drm drm_panel_orientation_quirks fuse backlight configfs loop nfnetlink usbhid xhci_pci ohci_pci xhci_hcd ohci_hcd ehci_pci sha512_ssse3 ehci_hcd sha256_ssse3 sha1_sTainted: [O]=OOT_MODULE sse3 sha1_generic usbcore gf128mul usb_common dm_mirror dm_region_hash dm_log cpuid i2c_piix4 i2c_smbus i2c_dev i2c_core it87 hwmon_vid msr dmi_sysfs autofs4 CR2: 0000000000000000 ---[ end trace 0000000000000000 ]--- RIP: 0010:rb_next+0x0/0x50 Code: e8 d5 fa ff ff 5b 4c 89 e0 5d 41 5c 41 5d 41 5e e9 85 73 01 00 5b 5d 41 5c 41 5d 41 5e e9 38 76 01 00 0f 1f 84 00 00 00 00 00 <48> 3b 3f 48 89 f8 74 38 48 8b 57 08 48 85 d2 74 11 48 89 d0 48 8b RSP: 0018:ffffc90000003e50 EHardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./970A-DS3P, BIOS FD 02/26/2016 FLAGS: 00010246 RAX: 0000000000000000 RBX: ffff88811c311000 RCX: ffff88811c311180 RDX: ffff888124639c00 RSI: ffff8881cf56d2e8 RDI: 0000000000000000 RBP: 0000000000000000 R08: ffff88811c3112b0 R09: 000000001cdf6fee R10: 0000000000000bad R11: ffffc90000003ff8 R12: ffff888124639c00 R13: ffff888RIP: 0010:rb_next (lib/rbtree.c:496) 11c3112b8 R14: 00000073b95a79bb R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88842ec00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000000000Code: e8 d5 fa ff ff 5b 4c 89 e0 5d 41 5c 41 5d 41 5e e9 85 73 01 00 5b 5d 41 5c 41 5d 41 5e e9 38 76 01 00 0f 1f 84 00 00 00 00 00 <48> 3b 3f 48 89 f8 74 38 48 8b 57 08 48 85 d2 74 11 48 89 d0 48 8b All code ======== 0: e8 d5 fa ff ff call 0xfffffffffffffada 5: 5b pop %rbx 6: 4c 89 e0 mov %r12,%rax 9: 5d pop %rbp a: 41 5c pop %r12 c: 41 5d pop %r13 e: 41 5e pop %r14 10: e9 85 73 01 00 jmp 0x1739a 15: 5b pop %rbx 16: 5d pop %rbp 17: 41 5c pop %r12 19: 41 5d pop %r13 1b: 41 5e pop %r14 1d: e9 38 76 01 00 jmp 0x1765a 22: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1) 29: 00 2a:* 48 3b 3f cmp (%rdi),%rdi <-- trapping instruction 2d: 48 89 f8 mov %rdi,%rax 30: 74 38 je 0x6a 32: 48 8b 57 08 mov 0x8(%rdi),%rdx 36: 48 85 d2 test %rdx,%rdx 39: 74 11 je 0x4c 3b: 48 89 d0 mov %rdx,%rax 3e: 48 rex.W 3f: 8b .byte 0x8b
Code starting with the faulting instruction =========================================== 0: 48 3b 3f cmp (%rdi),%rdi 3: 48 89 f8 mov %rdi,%rax 6: 74 38 je 0x40 8: 48 8b 57 08 mov 0x8(%rdi),%rdx c: 48 85 d2 test %rdx,%rdx f: 74 11 je 0x22 11: 48 89 d0 mov %rdx,%rax 14: 48 rex.W 15: 8b .byte 0x8b 00 CR3: 000000012e18c000 CR4: 00000000000406RSP: 0018:ffffc90000003e50 EFLAGS: 00010246 f0 Kernel panic - not syncing: Fatal exception in interrupt KernelRAX: 0000000000000000 RBX: ffff88811c311000 RCX: ffff88811c311180 Offset: disabled Rebooting in 3 seconds.. RDX: ffff888124639c00 RSI: ffff8881cf56d2e8 RDI: 0000000000000000 RBP: 0000000000000000 R08: ffff88811c3112b0 R09: 000000001cdf6fee R10: 0000000000000bad R11: ffffc90000003ff8 R12: ffff888124639c00 R13: ffff88811c3112b8 R14: 00000073b95a79bb R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88842ec00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 000000012e18c000 CR4: 00000000000406f0 Call Trace: <IRQ> htb_dequeue (net/sched/sch_htb.c:351 (discriminator 1) net/sched/sch_htb.c:924 (discriminator 1) net/sched/sch_htb.c:982 (discriminator 1)) sch_htb __qdisc_run (net/sched/sch_generic.c:294 net/sched/sch_generic.c:398 net/sched/sch_generic.c:416) ? timerqueue_del (lib/timerqueue.c:58) qdisc_run (./include/net/pkt_sched.h:128 ./include/net/pkt_sched.h:124) net_tx_action (net/core/dev.c:5553) handle_softirqs (./arch/x86/include/asm/atomic.h:23 ./include/linux/atomic/atomic-arch-fallback.h:457 ./include/linux/jump_label.h:262 ./include/trace/events/irq.h:142 kernel/softirq.c:562) __irq_exit_rcu (kernel/softirq.c:435 kernel/softirq.c:662) sysvec_apic_timer_interrupt (arch/x86/kernel/apic/apic.c:1049 (discriminator 35) arch/x86/kernel/apic/apic.c:1049 (discriminator 35)) </IRQ> <TASK> asm_sysvec_apic_timer_interrupt (./arch/x86/include/asm/idtentry.h:574) RIP: 0010:cpuidle_enter_state (drivers/cpuidle/cpuidle.c:292) Code: 18 4c 6f 00 85 c0 7e 0b 8b 73 04 83 cf ff e8 a1 22 e5 ff 31 ff e8 9a 2e 98 ff 45 84 ff 74 07 31 ff e8 0e 58 9d ff fb 45 85 ed <0f> 88 cc 00 00 00 49 63 c5 48 8b 3c 24 48 6b c8 68 48 6b d0 30 49 All code ======== 0: 18 4c 6f 00 sbb %cl,0x0(%rdi,%rbp,2) 4: 85 c0 test %eax,%eax 6: 7e 0b jle 0x13 8: 8b 73 04 mov 0x4(%rbx),%esi b: 83 cf ff or $0xffffffff,%edi e: e8 a1 22 e5 ff call 0xffffffffffe522b4 13: 31 ff xor %edi,%edi 15: e8 9a 2e 98 ff call 0xffffffffff982eb4 1a: 45 84 ff test %r15b,%r15b 1d: 74 07 je 0x26 1f: 31 ff xor %edi,%edi 21: e8 0e 58 9d ff call 0xffffffffff9d5834 26: fb sti 27: 45 85 ed test %r13d,%r13d 2a:* 0f 88 cc 00 00 00 js 0xfc <-- trapping instruction 30: 49 63 c5 movslq %r13d,%rax 33: 48 8b 3c 24 mov (%rsp),%rdi 37: 48 6b c8 68 imul $0x68,%rax,%rcx 3b: 48 6b d0 30 imul $0x30,%rax,%rdx 3f: 49 rex.WB
Code starting with the faulting instruction =========================================== 0: 0f 88 cc 00 00 00 js 0xd2 6: 49 63 c5 movslq %r13d,%rax 9: 48 8b 3c 24 mov (%rsp),%rdi d: 48 6b c8 68 imul $0x68,%rax,%rcx 11: 48 6b d0 30 imul $0x30,%rax,%rdx 15: 49 rex.WB RSP: 0018:ffffffff81e03e40 EFLAGS: 00000202 RAX: ffff88842ec00000 RBX: ffff888101b26800 RCX: 0000000000000000 RDX: 000000737ef85069 RSI: fffffffc38107585 RDI: 0000000000000000 RBP: 0000000000000002 R08: 0000000000000002 R09: 000000737dcfa800 R10: 0000000000000006 R11: 0000000000000020 R12: ffffffff81f98280 R13: 0000000000000002 R14: 000000737ef85069 R15: 0000000000000000 cpuidle_enter (drivers/cpuidle/cpuidle.c:391 (discriminator 2)) do_idle (kernel/sched/idle.c:234 kernel/sched/idle.c:325) cpu_startup_entry (kernel/sched/idle.c:422) rest_init (init/main.c:743) start_kernel (init/main.c:1525) x86_64_start_reservations (arch/x86/kernel/head64.c:513) x86_64_start_kernel (??:?) common_startup_64 (arch/x86/kernel/head_64.S:421) </TASK> Modules linked in: sch_htb cls_u32 sch_ingress sch_cake ifb act_mirred xt_hl xt_nat ts_bm xt_string xt_TARPIT(O) xt_CT xt_tcpudp xt_helper nf_nat_ftp nf_conntrack_ftp ip6t_rt ip6table_nat xt_MASQUERADE iptable_nat nf_nat xt_TCPMSS xt_LOG nf_log_syslog ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 ip6table_raw iptable_raw ip6table_mangle iptable_mangle xt_multiport xt_state xt_limit xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6table_filter ip6_tables iptable_filter ip_tables x_tables tun binfmt_misc pppoe pppox ppp_generic slhc netconsole af_packet bridge stp llc ctr ccm dm_crypt radeon drm_client_lib video wmi ath9k drm_exec drm_suballoc_helper ath9k_common drm_ttm_helper syscopyarea ath9k_hw ttm sysfillrect sysimgblt ath fb_sys_fops drm_display_helper snd_hda_codec_realtek pl2303 drm_kms_helper snd_hda_codec_generic mac80211 usbserial snd_hda_codec_hdmi snd_hda_scodec_component agpgart snd_hda_intel cfbfillrect snd_intel_dspcfg cfbimgblt snd_hda_codec fb_io_fops cfbcopyarea aesni_intel snd_hda_core i2c_algo_bit cdc_acm cfg80211 fb snd_pcm snd_timer font crypto_simd e1000 at24 snd cryptd libarc4 soundcore acpi_cpufreq regmap_i2c fam15h_power k10temp evdev nfsd sch_fq_codel auth_rpcgss lockd grace sunrpc drm drm_panel_orientation_quirks fuse backlight configfs loop nfnetlink usbhid xhci_pci ohci_pci xhci_hcd ohci_hcd ehci_pci sha512_ssse3 ehci_hcd sha256_ssse3 sha1_ssse3 sha1_generic usbcore gf128mul usb_common dm_mirror dm_region_hash dm_log cpuid i2c_piix4 i2c_smbus i2c_dev i2c_core it87 hwmon_vid msr dmi_sysfs autofs4 CR2: 0000000000000000 ---[ end trace 0000000000000000 ]--- RIP: 0010:rb_next (lib/rbtree.c:496) Code: e8 d5 fa ff ff 5b 4c 89 e0 5d 41 5c 41 5d 41 5e e9 85 73 01 00 5b 5d 41 5c 41 5d 41 5e e9 38 76 01 00 0f 1f 84 00 00 00 00 00 <48> 3b 3f 48 89 f8 74 38 48 8b 57 08 48 85 d2 74 11 48 89 d0 48 8b All code ======== 0: e8 d5 fa ff ff call 0xfffffffffffffada 5: 5b pop %rbx 6: 4c 89 e0 mov %r12,%rax 9: 5d pop %rbp a: 41 5c pop %r12 c: 41 5d pop %r13 e: 41 5e pop %r14 10: e9 85 73 01 00 jmp 0x1739a 15: 5b pop %rbx 16: 5d pop %rbp 17: 41 5c pop %r12 19: 41 5d pop %r13 1b: 41 5e pop %r14 1d: e9 38 76 01 00 jmp 0x1765a 22: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1) 29: 00 2a:* 48 3b 3f cmp (%rdi),%rdi <-- trapping instruction 2d: 48 89 f8 mov %rdi,%rax 30: 74 38 je 0x6a 32: 48 8b 57 08 mov 0x8(%rdi),%rdx 36: 48 85 d2 test %rdx,%rdx 39: 74 11 je 0x4c 3b: 48 89 d0 mov %rdx,%rax 3e: 48 rex.W 3f: 8b .byte 0x8b
Code starting with the faulting instruction =========================================== 0: 48 3b 3f cmp (%rdi),%rdi 3: 48 89 f8 mov %rdi,%rax 6: 74 38 je 0x40 8: 48 8b 57 08 mov 0x8(%rdi),%rdx c: 48 85 d2 test %rdx,%rdx f: 74 11 je 0x22 11: 48 89 d0 mov %rdx,%rax 14: 48 rex.W 15: 8b .byte 0x8b RSP: 0018:ffffc90000003e50 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff88811c311000 RCX: ffff88811c311180 RDX: ffff888124639c00 RSI: ffff8881cf56d2e8 RDI: 0000000000000000 RBP: 0000000000000000 R08: ffff88811c3112b0 R09: 000000001cdf6fee R10: 0000000000000bad R11: ffffc90000003ff8 R12: ffff888124639c00 R13: ffff88811c3112b8 R14: 00000073b95a79bb R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88842ec00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 000000012e18c000 CR4: 00000000000406f0 Kernel panic - not syncing: Fatal exception in interrupt Kernel Offset: disabled Rebooting in 3 seconds..
On Tue, Apr 22, 2025 at 07:20:24PM +0200, Holger Hoffstätte wrote:
Meanwhile, if you could provide a reliable (and ideally minimum) reproducer, it would help me a lot to debug.
I've found a reproducer. Below is a stripped down version of the shell script that I posted in my initial report.
Running this in a 1 second loop is enough to cause the panic very quickly.
It seems a bit of network traffic is needed, too.
# while true; do ./tc.sh; sleep 1; done 13:33:43 7196kbit 29296kbit 13:33:44 7196kbit 29296kbit ... 13:35:38 7196kbit 29296kbit 13:35:39 7196kbit 29296kbit [panic]
# while true; do ./tc.sh; sleep 1; done 13:44:31 7196kbit 29296kbit 13:44:32 7196kbit 29296kbit ... 13:44:52 7196kbit 29296kbit 13:44:53 7196kbit 29296kbit [panic]
The same place as usual htb_dequeue+0x42f/0x610 [sch_htb]
--------8<--------8<--------8<--------8<--------8<--------8<--------8<--------8<--------8< #!/usr/bin/bash
set -o nounset set -o errexit
export PATH=/usr/bin
ext=ppp0 ext_ingress=ppp0ifb0
ext_up=7196kbit ext_down=29296kbit
printf "%(%T)T $ext_up $ext_down\n"
q=1486 quantum=300
modprobe act_mirred modprobe ifb modprobe sch_cake modprobe sch_fq_codel
ethtool -K "$ext" tso off gso off gro off
tc qdisc del dev "$ext" root >& /dev/null || true tc qdisc del dev "$ext" ingress >& /dev/null || true tc qdisc del dev "$ext_ingress" root >& /dev/null || true tc qdisc del dev "$ext_ingress" ingress >& /dev/null || true ip link del "$ext_ingress" >& /dev/null || true
tc qdisc add dev "$ext" handle ffff: ingress
ip link add name "$ext_ingress" type ifb ip link set dev "$ext_ingress" up || true
tc filter add dev "$ext" parent ffff: protocol all u32 match u32 0 0 action mirred egress redirect dev "$ext_ingress" tc qdisc add dev "$ext_ingress" root handle 1: htb default 11 r2q 20 tc class add dev "$ext_ingress" parent 1: classid 1:1 htb rate $ext_down tc class add dev "$ext_ingress" parent 1:1 classid 1:11 htb rate $ext_down prio 0 quantum $q tc qdisc add dev "$ext_ingress" parent 1:11 fq_codel quantum $quantum ecn tc qdisc add dev "$ext" root handle 1: htb default 11 tc class add dev "$ext" parent 1: classid 1:1 htb rate $ext_up tc class add dev "$ext" parent 1:1 classid 1:11 htb rate $ext_up prio 0 quantum $q tc qdisc add dev "$ext" parent 1:11 fq_codel quantum $quantum noecn --------8<--------8<--------8<--------8<--------8<--------8<--------8<--------8<--------8<
On Thu, Apr 24, 2025 at 01:53:31PM +0100, Alan J. Wylie wrote:
On Tue, Apr 22, 2025 at 07:20:24PM +0200, Holger Hoffstätte wrote:
Meanwhile, if you could provide a reliable (and ideally minimum) reproducer, it would help me a lot to debug.
I've found a reproducer. Below is a stripped down version of the shell script that I posted in my initial report.
Running this in a 1 second loop is enough to cause the panic very quickly.
It seems a bit of network traffic is needed, too.
Excellent! Now I can try it on my side instead of bothering you. :)
I will check if applying all patches here could make crash and warning go away. And of course, I need to really understand why the crash still happened even after the qlen check adding to htb_qlen_notify(), which is unexpected to me.
Thanks a lot!
linux-stable-mirror@lists.linaro.org
-
Alan J. Wylie -
Cong Wang -
Holger Hoffstätte