Am 28.03.2018 um 20:57 schrieb Logan Gunthorpe:
On 28/03/18 12:28 PM, Christian König wrote:
I'm just using amdgpu as blueprint because I'm the co-maintainer of it and know it mostly inside out.
Ah, I see.
The resource addresses are translated using dma_map_resource(). As far as I know that should be sufficient to offload all the architecture specific stuff to the DMA subsystem.
It's not. The dma_map infrastructure currently has no concept of peer-to-peer mappings and is designed for system memory only. No architecture I'm aware of will translate PCI CPU addresses into PCI Bus addresses which is necessary for any transfer that doesn't go through the root complex (though on arches like x86 the CPU and Bus address happen to be the same). There's a lot of people that would like to see this change but it's likely going to be a long road before it does.
Well, isn't that exactly what dma_map_resource() is good for? As far as I can see it makes sure IOMMU is aware of the access route and translates a CPU address into a PCI Bus address.
Furthermore, one of the reasons our patch-set avoids going through the root complex at all is that IOMMU drivers will need to be made aware that it is operating on P2P memory and do arch-specific things accordingly. There will also need to be flags that indicate whether a given IOMMU driver supports this. None of this work is done or easy.
I'm using that with the AMD IOMMU driver and at least there it works perfectly fine.
Yeah, but not for ours. See if you want to do real peer 2 peer you need to keep both the operation as well as the direction into account.
Not sure what you are saying here... I'm pretty sure we are doing "real" peer 2 peer...
For example when you can do writes between A and B that doesn't mean that writes between B and A work. And reads are generally less likely to work than writes. etc...
If both devices are behind a switch then the PCI spec guarantees that A can both read and write B and vice versa.
Sorry to say that, but I know a whole bunch of PCI devices which horrible ignores that.
For example all AMD APUs fall under that category...
Only once you involve root complexes do you have this problem. Ie. you have unknown support which may be no support, or partial support (stores but not loads); or sometimes bad performance; or a combination of both... and you need some way to figure out all this mess and that is hard. Whoever tries to implement a white list will have to sort all this out.
Yes, exactly and unfortunately it looks like I'm the poor guy who needs to do this :)
Regards, Christian.
Logan _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx