Hi Robin,
On Fri, 25 Apr 2025 at 20:48, Robin Murphy robin.murphy@arm.com wrote:
On 11/04/2025 8:11 pm, Robin Murphy wrote: [...]
OK, I found it, but I'm still not sure what exactly to make of it - it's the pci_request_acs() in of_iommu_configure(), now being called early enough to actually have an effect. Booting with EDK2 already using PCI prior to Linux, here's what I get for `sudo lspci -vv | grep ACSctl`
Linux version 6.14.9-rc1 # lscpi 00:00.0 PCI bridge: PLDA PCI Express Core Reference Design 01:00.0 PCI bridge: Microsemi / PMC / IDT 89HPES16NT16G2 16-Lane 16-Port PCIe Gen2 System Interconnect Switch (rev 02) 02:01.0 PCI bridge: Microsemi / PMC / IDT 89HPES16NT16G2 16-Lane 16-Port PCIe Gen2 System Interconnect Switch (rev 02) 02:02.0 PCI bridge: Microsemi / PMC / IDT 89HPES16NT16G2 16-Lane 16-Port PCIe Gen2 System Interconnect Switch (rev 02) 02:03.0 PCI bridge: Microsemi / PMC / IDT 89HPES16NT16G2 16-Lane 16-Port PCIe Gen2 System Interconnect Switch (rev 02) 02:0c.0 PCI bridge: Microsemi / PMC / IDT 89HPES16NT16G2 16-Lane 16-Port PCIe Gen2 System Interconnect Switch (rev 02) 02:10.0 PCI bridge: Microsemi / PMC / IDT 89HPES16NT16G2 16-Lane 16-Port PCIe Gen2 System Interconnect Switch (rev 02) 02:1f.0 PCI bridge: Microsemi / PMC / IDT 89HPES16NT16G2 16-Lane 16-Port PCIe Gen2 System Interconnect Switch (rev 02) 03:00.0 Mass storage controller: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller (rev 01) 08:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8057 PCI-E Gigabit Ethernet Controller
Linux version 6.15.0 # lscpi 00:00.0 PCI bridge: PLDA PCI Express Core Reference Design 01:00.0 PCI bridge: Microsemi / PMC / IDT 89HPES16NT16G2 16-Lane 16-Port PCIe Gen2 System Interconnect Switch (rev 02) 02:01.0 PCI bridge: Microsemi / PMC / IDT 89HPES16NT16G2 16-Lane 16-Port PCIe Gen2 System Interconnect Switch (rev 02) 02:02.0 PCI bridge: Microsemi / PMC / IDT 89HPES16NT16G2 16-Lane 16-Port PCIe Gen2 System Interconnect Switch (rev 02) 02:03.0 PCI bridge: Microsemi / PMC / IDT 89HPES16NT16G2 16-Lane 16-Port PCIe Gen2 System Interconnect Switch (rev 02) 02:0c.0 PCI bridge: Microsemi / PMC / IDT 89HPES16NT16G2 16-Lane 16-Port PCIe Gen2 System Interconnect Switch (rev 02) 02:10.0 PCI bridge: Microsemi / PMC / IDT 89HPES16NT16G2 16-Lane 16-Port PCIe Gen2 System Interconnect Switch (rev 02) 02:1f.0 PCI bridge: Microsemi / PMC / IDT 89HPES16NT16G2 16-Lane 16-Port PCIe Gen2 System Interconnect Switch (rev 02)
with 6.15-rc1:
ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+UpstreamFwd+ EgressCtrl- DirectTrans- ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
whereas with the 6.14 behaviour they are all '-'. I don't have a working root filesystem with the U-Boot setup, but if I boot it with "pci=config_acs=000000@pci:0:0" then the kernel does assign the bridge windows and discover the ethernet/SATA endpoints again. I can spend some time getting NFS working next week, but if you're able to get lspci output off a machine in the "broken" state easily that would be handy to compare.
On the 6.15, After adding this into Kernel boot args pci=config_acs=000000@pci:0:0
The SSD was detected and mounted successfully.
Linux version 6.15.0 + pci=config_acs=000000@pci:0:0 # lspci 00:00.0 PCI bridge: PLDA PCI Express Core Reference Design 01:00.0 PCI bridge: Microsemi / PMC / IDT 89HPES16NT16G2 16-Lane 16-Port PCIe Gen2 System Interconnect Switch (rev 02) 02:01.0 PCI bridge: Microsemi / PMC / IDT 89HPES16NT16G2 16-Lane 16-Port PCIe Gen2 System Interconnect Switch (rev 02) 02:02.0 PCI bridge: Microsemi / PMC / IDT 89HPES16NT16G2 16-Lane 16-Port PCIe Gen2 System Interconnect Switch (rev 02) 02:03.0 PCI bridge: Microsemi / PMC / IDT 89HPES16NT16G2 16-Lane 16-Port PCIe Gen2 System Interconnect Switch (rev 02) 02:0c.0 PCI bridge: Microsemi / PMC / IDT 89HPES16NT16G2 16-Lane 16-Port PCIe Gen2 System Interconnect Switch (rev 02) 02:10.0 PCI bridge: Microsemi / PMC / IDT 89HPES16NT16G2 16-Lane 16-Port PCIe Gen2 System Interconnect Switch (rev 02) 02:1f.0 PCI bridge: Microsemi / PMC / IDT 89HPES16NT16G2 16-Lane 16-Port PCIe Gen2 System Interconnect Switch (rev 02) 03:00.0 Mass storage controller: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller (rev 01) 08:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8057 PCI-E Gigabit Ethernet Controller
So at this point it would seem to be something about how Linux configures ACS when doing it from scratch. What I don't really know is where to go from there. I do know Juno's possibly a bit odd in that the switch supports ACS, but both the root port and endpoints either side of it don't. Could this be tickling some subtle bug in the PCI layer, and what is EDK2 doing that makes it not happen?
Just following up on where I ran out of ideas. I poked at things a little more, and from a process of elimination, the culprit appears to be is that we enable ACS source validation on the downstream port while its secondary bus is still 0, *then* we get to the "bridge configuration invalid" bit and reconfigure the bus numbers, but after that, config space accesses to the secondary bus still apparently fail to work as expected.
What's now beyond me is whether this is just an ACS quirk of this particular switch, and/or whether there's something we could or should be doing in the PCI layer.
All I can suggest a this point is that you could at least sidestep the problem on the LKFT boards by updating them to a less-ancient version of U-Boot which supports PCIe for Juno (looks like that landed in 2020.10), which should then configure the switch at boot such that the bus numbering doesn't need to change when Linux probes it - that appears to be the only "magic" thing that EDK2 is doing.
I will work with my team to address these issues. Thanks for your suggestions.
Thanks, Robin.
- Naresh