Regressions on x86_64 boot failed with Linux next-20250324 tag kernel 6.14.0-rc7-next-20250324
First seen on the next-20250324 Good: next-20250321 Bad: next-20250324 ..next-20250325
Regressions found on x86_84: - boot
Regression Analysis: - New regression? Yes - Reproducible? Yes
Boot regression: x86 boot fail kernel panic __pci_enable_msi_range Reported-by: Linux Kernel Functional Testing lkft@linaro.org
## Boot log <1>[ 1.525485] BUG: kernel NULL pointer dereference, address: 0000000000000002 <1>[ 1.525573] #PF: supervisor read access in kernel mode <1>[ 1.525573] #PF: error_code(0x0000) - not-present page <6>[ 1.525573] PGD 0 P4D 0 <4>[ 1.525573] Oops: Oops: 0000 [#1] SMP PTI <4>[ 1.525573] CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.14.0-rc7-next-20250324 #1 PREEMPT(voluntary) <4>[ 1.525573] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014 <4>[ 1.525573] RIP: 0010:__pci_enable_msi_range+0x306/0x6b0 <4>[ 1.525573] Code: ff ff ff e8 1c 05 fe ff f6 83 21 08 00 00 10 0f b7 85 6e ff ff ff 74 0c 0d 00 01 00 00 66 89 85 6e ff ff ff 8b 8d 68 ff ff ff <41> f6 44 24 02 40 74 0c 25 ff fe 00 00 66 89 85 6e ff ff ff 89 8d <4>[ 1.525573] RSP: 0000:ffffa83b00013740 EFLAGS: 00010246 <4>[ 1.525573] RAX: 0000000000000080 RBX: ffffa11c8023e000 RCX: 0000000000000001 <4>[ 1.525573] RDX: 0000000000000000 RSI: ffffffff9e60c683 RDI: ffffffff9e6519a8 <4>[ 1.525573] RBP: ffffa83b00013810 R08: 0000000000000002 R09: ffffa83b0001370c <4>[ 1.525573] R10: 0000000000000001 R11: ffffffff9e60c5b0 R12: 0000000000000000 <4>[ 1.525573] R13: 0000000000000000 R14: 0000000000000001 R15: ffffa11c8023e000 <4>[ 1.525573] FS: 0000000000000000(0000) GS:ffffa11d5c339000(0000) knlGS:0000000000000000 <4>[ 1.525573] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 <4>[ 1.525573] CR2: 0000000000000002 CR3: 000000007d844000 CR4: 00000000000006f0 <4>[ 1.525573] Call Trace: <4>[ 1.525573] <TASK> <4>[ 1.525573] ? __die_body+0xb4/0xc0 <4>[ 1.525573] ? __die+0x2e/0x40 <4>[ 1.525573] ? page_fault_oops+0x3ae/0x410 <4>[ 1.525573] ? kernelmode_fixup_or_oops+0x54/0x70 <4>[ 1.525573] ? __bad_area_nosemaphore+0x52/0x240 <4>[ 1.525573] ? bad_area_nosemaphore+0x16/0x20 <4>[ 1.525573] ? do_user_addr_fault+0x738/0x7a0 <4>[ 1.525573] ? irqentry_enter+0x2d/0x50 <4>[ 1.525573] ? exc_page_fault+0x4d/0x120 <4>[ 1.525573] ? exc_page_fault+0x70/0x120 <4>[ 1.525573] ? asm_exc_page_fault+0x2b/0x30 <4>[ 1.525573] ? __pfx_pci_conf1_read+0x10/0x10 <4>[ 1.525573] ? pci_conf1_read+0xd3/0xf0 <4>[ 1.525573] ? _raw_spin_unlock_irqrestore+0x28/0x50 <4>[ 1.525573] ? __pci_enable_msi_range+0x306/0x6b0 <4>[ 1.525573] ? _raw_spin_unlock_irqrestore+0x28/0x50 <4>[ 1.525573] pci_alloc_irq_vectors_affinity+0xbf/0x140 <4>[ 1.525573] pci_alloc_irq_vectors+0x15/0x20 <4>[ 1.525573] ahci_init_irq+0x90/0xc0 <4>[ 1.525573] ahci_init_one+0x82c/0xd10 <4>[ 1.525573] pci_device_probe+0x198/0x230 <4>[ 1.525573] really_probe+0x146/0x450 <4>[ 1.525573] __driver_probe_device+0x7a/0xf0 <4>[ 1.525573] driver_probe_device+0x24/0x190 <4>[ 1.525573] __driver_attach+0x104/0x250 <4>[ 1.525573] ? __pfx___driver_attach+0x10/0x10 <4>[ 1.525573] bus_for_each_dev+0x10e/0x160 <4>[ 1.525573] driver_attach+0x22/0x30 <4>[ 1.525573] bus_add_driver+0x175/0x2c0 <4>[ 1.525573] driver_register+0x65/0xf0 <4>[ 1.525573] ? __pfx_ahci_pci_driver_init+0x10/0x10 <4>[ 1.525573] __pci_register_driver+0x68/0x70 <4>[ 1.525573] ahci_pci_driver_init+0x22/0x30 <4>[ 1.525573] do_one_initcall+0x121/0x330 <4>[ 1.525573] ? alloc_pages_mpol+0x170/0x1c0 <4>[ 1.525573] ? alloc_pages_mpol+0x170/0x1c0 <4>[ 1.525573] ? trace_preempt_on+0x12/0x80 <4>[ 1.525573] ? alloc_pages_mpol+0x170/0x1c0 <4>[ 1.525573] ? preempt_count_sub+0x63/0x80 <4>[ 1.525573] ? alloc_pages_mpol+0x170/0x1c0 <4>[ 1.525573] ? trace_hardirqs_on+0x29/0xa0 <4>[ 1.525573] ? irqentry_exit+0x57/0x60 <4>[ 1.525573] ? sysvec_apic_timer_interrupt+0x52/0x90 <4>[ 1.525573] ? next_arg+0xcd/0x150 <4>[ 1.525573] ? next_arg+0x138/0x150 <4>[ 1.525573] ? parse_args+0x16e/0x440 <4>[ 1.525573] do_initcall_level+0x80/0xf0 <4>[ 1.525573] do_initcalls+0x48/0x80 <4>[ 1.525573] do_basic_setup+0x1d/0x30 <4>[ 1.525573] kernel_init_freeable+0x10c/0x180 <4>[ 1.525573] ? __pfx_kernel_init+0x10/0x10 <4>[ 1.525573] kernel_init+0x1e/0x130 <4>[ 1.525573] ret_from_fork+0x45/0x50 <4>[ 1.525573] ? __pfx_kernel_init+0x10/0x10 <4>[ 1.525573] ret_from_fork_asm+0x1a/0x30 <4>[ 1.525573] </TASK> <4>[ 1.525573] Modules linked in: <4>[ 1.525573] CR2: 0000000000000002 <4>[ 1.525573] ---[ end trace 0000000000000000 ]--- <4>[ 1.525573] RIP: 0010:__pci_enable_msi_range+0x306/0x6b0 <4>[ 1.525573] Code: ff ff ff e8 1c 05 fe ff f6 83 21 08 00 00 10 0f b7 85 6e ff ff ff 74 0c 0d 00 01 00 00 66 89 85 6e ff ff ff 8b 8d 68 ff ff ff <41> f6 44 24 02 40 74 0c 25 ff fe 00 00 66 89 85 6e ff ff ff 89 8d <4>[ 1.525573] RSP: 0000:ffffa83b00013740 EFLAGS: 00010246 <4>[ 1.525573] RAX: 0000000000000080 RBX: ffffa11c8023e000 RCX: 0000000000000001 <4>[ 1.525573] RDX: 0000000000000000 RSI: ffffffff9e60c683 RDI: ffffffff9e6519a8 <4>[ 1.525573] RBP: ffffa83b00013810 R08: 0000000000000002 R09: ffffa83b0001370c <4>[ 1.525573] R10: 0000000000000001 R11: ffffffff9e60c5b0 R12: 0000000000000000 <4>[ 1.525573] R13: 0000000000000000 R14: 0000000000000001 R15: ffffa11c8023e000 <4>[ 1.525573] FS: 0000000000000000(0000) GS:ffffa11d5c339000(0000) knlGS:0000000000000000 <4>[ 1.525573] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 <4>[ 1.525573] CR2: 0000000000000002 CR3: 000000007d844000 CR4: 00000000000006f0 <6>[ 1.525573] note: swapper/0[1] exited with irqs disabled <0>[ 1.553459] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000009 <0>[ 1.553844] Kernel Offset: 0x1c000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) <0>[ 1.553844] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000009 ]---
## Source * Kernel version: 6.14.0-rc7-next-20250324 * Git tree: https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git * Git sha: 882a18c2c14fc79adb30fe57a9758283aa20efaa * Git describe: next-20250324 * Project details: https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20250324/
## Test * Test log: https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20250324/tes... * Test history: https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20250324/tes... * Test details: https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20250324/tes... * Build link: https://storage.tuxsuite.com/public/linaro/lkft/builds/2ulLSNJUAxmyv6UZdUMeo... * Kernel config: https://storage.tuxsuite.com/public/linaro/lkft/builds/2ulLSNJUAxmyv6UZdUMeo...
-- Linaro LKFT https://lkft.linaro.org
If I had to guess, I'd say that it was related to Fixes: d9f2164238d8 ("PCI/MSI: Convert pci_msi_ignore_mask to per MSI domain flag"). I suspect d->host_data can be NULL. I could be wrong, but let's add Roger to the CC list just in case.
regards, dan carpenter
On Tue, Mar 25, 2025 at 05:11:20PM +0530, Naresh Kamboju wrote:
Regressions on x86_64 boot failed with Linux next-20250324 tag kernel 6.14.0-rc7-next-20250324
First seen on the next-20250324 Good: next-20250321 Bad: next-20250324 ..next-20250325
Regressions found on x86_84:
- boot
Regression Analysis:
- New regression? Yes
- Reproducible? Yes
Boot regression: x86 boot fail kernel panic __pci_enable_msi_range Reported-by: Linux Kernel Functional Testing lkft@linaro.org
## Boot log <1>[ 1.525485] BUG: kernel NULL pointer dereference, address: 0000000000000002 <1>[ 1.525573] #PF: supervisor read access in kernel mode <1>[ 1.525573] #PF: error_code(0x0000) - not-present page <6>[ 1.525573] PGD 0 P4D 0 <4>[ 1.525573] Oops: Oops: 0000 [#1] SMP PTI <4>[ 1.525573] CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.14.0-rc7-next-20250324 #1 PREEMPT(voluntary) <4>[ 1.525573] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014 <4>[ 1.525573] RIP: 0010:__pci_enable_msi_range+0x306/0x6b0 <4>[ 1.525573] Code: ff ff ff e8 1c 05 fe ff f6 83 21 08 00 00 10 0f b7 85 6e ff ff ff 74 0c 0d 00 01 00 00 66 89 85 6e ff ff ff 8b 8d 68 ff ff ff <41> f6 44 24 02 40 74 0c 25 ff fe 00 00 66 89 85 6e ff ff ff 89 8d <4>[ 1.525573] RSP: 0000:ffffa83b00013740 EFLAGS: 00010246 <4>[ 1.525573] RAX: 0000000000000080 RBX: ffffa11c8023e000 RCX: 0000000000000001 <4>[ 1.525573] RDX: 0000000000000000 RSI: ffffffff9e60c683 RDI: ffffffff9e6519a8 <4>[ 1.525573] RBP: ffffa83b00013810 R08: 0000000000000002 R09: ffffa83b0001370c <4>[ 1.525573] R10: 0000000000000001 R11: ffffffff9e60c5b0 R12: 0000000000000000 <4>[ 1.525573] R13: 0000000000000000 R14: 0000000000000001 R15: ffffa11c8023e000 <4>[ 1.525573] FS: 0000000000000000(0000) GS:ffffa11d5c339000(0000) knlGS:0000000000000000 <4>[ 1.525573] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 <4>[ 1.525573] CR2: 0000000000000002 CR3: 000000007d844000 CR4: 00000000000006f0 <4>[ 1.525573] Call Trace: <4>[ 1.525573] <TASK> <4>[ 1.525573] ? __die_body+0xb4/0xc0 <4>[ 1.525573] ? __die+0x2e/0x40 <4>[ 1.525573] ? page_fault_oops+0x3ae/0x410 <4>[ 1.525573] ? kernelmode_fixup_or_oops+0x54/0x70 <4>[ 1.525573] ? __bad_area_nosemaphore+0x52/0x240 <4>[ 1.525573] ? bad_area_nosemaphore+0x16/0x20 <4>[ 1.525573] ? do_user_addr_fault+0x738/0x7a0 <4>[ 1.525573] ? irqentry_enter+0x2d/0x50 <4>[ 1.525573] ? exc_page_fault+0x4d/0x120 <4>[ 1.525573] ? exc_page_fault+0x70/0x120 <4>[ 1.525573] ? asm_exc_page_fault+0x2b/0x30 <4>[ 1.525573] ? __pfx_pci_conf1_read+0x10/0x10 <4>[ 1.525573] ? pci_conf1_read+0xd3/0xf0 <4>[ 1.525573] ? _raw_spin_unlock_irqrestore+0x28/0x50 <4>[ 1.525573] ? __pci_enable_msi_range+0x306/0x6b0 <4>[ 1.525573] ? _raw_spin_unlock_irqrestore+0x28/0x50 <4>[ 1.525573] pci_alloc_irq_vectors_affinity+0xbf/0x140 <4>[ 1.525573] pci_alloc_irq_vectors+0x15/0x20 <4>[ 1.525573] ahci_init_irq+0x90/0xc0 <4>[ 1.525573] ahci_init_one+0x82c/0xd10 <4>[ 1.525573] pci_device_probe+0x198/0x230 <4>[ 1.525573] really_probe+0x146/0x450 <4>[ 1.525573] __driver_probe_device+0x7a/0xf0 <4>[ 1.525573] driver_probe_device+0x24/0x190 <4>[ 1.525573] __driver_attach+0x104/0x250 <4>[ 1.525573] ? __pfx___driver_attach+0x10/0x10 <4>[ 1.525573] bus_for_each_dev+0x10e/0x160 <4>[ 1.525573] driver_attach+0x22/0x30 <4>[ 1.525573] bus_add_driver+0x175/0x2c0 <4>[ 1.525573] driver_register+0x65/0xf0 <4>[ 1.525573] ? __pfx_ahci_pci_driver_init+0x10/0x10 <4>[ 1.525573] __pci_register_driver+0x68/0x70 <4>[ 1.525573] ahci_pci_driver_init+0x22/0x30 <4>[ 1.525573] do_one_initcall+0x121/0x330 <4>[ 1.525573] ? alloc_pages_mpol+0x170/0x1c0 <4>[ 1.525573] ? alloc_pages_mpol+0x170/0x1c0 <4>[ 1.525573] ? trace_preempt_on+0x12/0x80 <4>[ 1.525573] ? alloc_pages_mpol+0x170/0x1c0 <4>[ 1.525573] ? preempt_count_sub+0x63/0x80 <4>[ 1.525573] ? alloc_pages_mpol+0x170/0x1c0 <4>[ 1.525573] ? trace_hardirqs_on+0x29/0xa0 <4>[ 1.525573] ? irqentry_exit+0x57/0x60 <4>[ 1.525573] ? sysvec_apic_timer_interrupt+0x52/0x90 <4>[ 1.525573] ? next_arg+0xcd/0x150 <4>[ 1.525573] ? next_arg+0x138/0x150 <4>[ 1.525573] ? parse_args+0x16e/0x440 <4>[ 1.525573] do_initcall_level+0x80/0xf0 <4>[ 1.525573] do_initcalls+0x48/0x80 <4>[ 1.525573] do_basic_setup+0x1d/0x30 <4>[ 1.525573] kernel_init_freeable+0x10c/0x180 <4>[ 1.525573] ? __pfx_kernel_init+0x10/0x10 <4>[ 1.525573] kernel_init+0x1e/0x130 <4>[ 1.525573] ret_from_fork+0x45/0x50 <4>[ 1.525573] ? __pfx_kernel_init+0x10/0x10 <4>[ 1.525573] ret_from_fork_asm+0x1a/0x30 <4>[ 1.525573] </TASK> <4>[ 1.525573] Modules linked in: <4>[ 1.525573] CR2: 0000000000000002 <4>[ 1.525573] ---[ end trace 0000000000000000 ]--- <4>[ 1.525573] RIP: 0010:__pci_enable_msi_range+0x306/0x6b0 <4>[ 1.525573] Code: ff ff ff e8 1c 05 fe ff f6 83 21 08 00 00 10 0f b7 85 6e ff ff ff 74 0c 0d 00 01 00 00 66 89 85 6e ff ff ff 8b 8d 68 ff ff ff <41> f6 44 24 02 40 74 0c 25 ff fe 00 00 66 89 85 6e ff ff ff 89 8d <4>[ 1.525573] RSP: 0000:ffffa83b00013740 EFLAGS: 00010246 <4>[ 1.525573] RAX: 0000000000000080 RBX: ffffa11c8023e000 RCX: 0000000000000001 <4>[ 1.525573] RDX: 0000000000000000 RSI: ffffffff9e60c683 RDI: ffffffff9e6519a8 <4>[ 1.525573] RBP: ffffa83b00013810 R08: 0000000000000002 R09: ffffa83b0001370c <4>[ 1.525573] R10: 0000000000000001 R11: ffffffff9e60c5b0 R12: 0000000000000000 <4>[ 1.525573] R13: 0000000000000000 R14: 0000000000000001 R15: ffffa11c8023e000 <4>[ 1.525573] FS: 0000000000000000(0000) GS:ffffa11d5c339000(0000) knlGS:0000000000000000 <4>[ 1.525573] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 <4>[ 1.525573] CR2: 0000000000000002 CR3: 000000007d844000 CR4: 00000000000006f0 <6>[ 1.525573] note: swapper/0[1] exited with irqs disabled <0>[ 1.553459] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000009 <0>[ 1.553844] Kernel Offset: 0x1c000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) <0>[ 1.553844] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000009 ]---
## Source
- Kernel version: 6.14.0-rc7-next-20250324
- Git tree: https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
- Git sha: 882a18c2c14fc79adb30fe57a9758283aa20efaa
- Git describe: next-20250324
- Project details:
https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20250324/
## Test
- Test log: https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20250324/tes...
- Test history:
https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20250324/tes...
- Test details:
https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20250324/tes...
- Build link: https://storage.tuxsuite.com/public/linaro/lkft/builds/2ulLSNJUAxmyv6UZdUMeo...
- Kernel config:
https://storage.tuxsuite.com/public/linaro/lkft/builds/2ulLSNJUAxmyv6UZdUMeo...
-- Linaro LKFT https://lkft.linaro.org
On Tue, Mar 25, 2025 at 04:56:33PM +0300, Dan Carpenter wrote:
If I had to guess, I'd say that it was related to Fixes: d9f2164238d8 ("PCI/MSI: Convert pci_msi_ignore_mask to per MSI domain flag"). I suspect d->host_data can be NULL. I could be wrong, but let's add Roger to the CC list just in case.
Indeed, sorry. There's a patch from Thomas to switch to using pci_msi_domain_supports() for fetching the flag, as there's no guarantee all call contexts will have an associated msi_domain_info:
https://lore.kernel.org/xen-devel/87v7rxzct0.ffs@tglx/
Regards, Roger.
On Tue, 25 Mar 2025 at 20:23, Roger Pau Monné roger.pau@citrix.com wrote:
On Tue, Mar 25, 2025 at 04:56:33PM +0300, Dan Carpenter wrote:
If I had to guess, I'd say that it was related to Fixes: d9f2164238d8 ("PCI/MSI: Convert pci_msi_ignore_mask to per MSI domain flag"). I suspect d->host_data can be NULL. I could be wrong, but let's add Roger to the CC list just in case.
Indeed, sorry. There's a patch from Thomas to switch to using pci_msi_domain_supports() for fetching the flag, as there's no guarantee all call contexts will have an associated msi_domain_info:
Thanks Roger for the clarification. LKFT started noticing this issue on the Linus Torvalds master branch from March 26, 2025 at git describe: v6.14-1979-g61af143fbea4
Anders bisected and confirmed that, # first bad commit: [c3164d2e0d181027da8fc94f8179d8607c3d440f] PCI/MSI: Convert pci_msi_ignore_mask to per MSI domain flag
Lore report link, - https://lore.kernel.org/all/CA+G9fYs4-4y=edxddERXQ_fMsW_nUJU+V0bSMHFDL3St7Ni...
Regards, Roger.
On Thu, Mar 27, 2025 at 02:10:21PM +0530, Naresh Kamboju wrote:
On Tue, 25 Mar 2025 at 20:23, Roger Pau Monné roger.pau@citrix.com wrote:
On Tue, Mar 25, 2025 at 04:56:33PM +0300, Dan Carpenter wrote:
If I had to guess, I'd say that it was related to Fixes: d9f2164238d8 ("PCI/MSI: Convert pci_msi_ignore_mask to per MSI domain flag"). I suspect d->host_data can be NULL. I could be wrong, but let's add Roger to the CC list just in case.
Indeed, sorry. There's a patch from Thomas to switch to using pci_msi_domain_supports() for fetching the flag, as there's no guarantee all call contexts will have an associated msi_domain_info:
Thanks Roger for the clarification. LKFT started noticing this issue on the Linus Torvalds master branch from March 26, 2025 at git describe: v6.14-1979-g61af143fbea4
Anders bisected and confirmed that, # first bad commit: [c3164d2e0d181027da8fc94f8179d8607c3d440f] PCI/MSI: Convert pci_msi_ignore_mask to per MSI domain flag
Hello,
The fix has already been committed to Linux master, commit hash 3ece3e8e5976c49c3f887e5923f998eabd54ff40.
Regards, Roger.
-
Dan Carpenter -
Naresh Kamboju -
Roger Pau Monné