On Tue, Jun 22, 2021 at 05:29:01PM +0200, Christian König wrote:
Am 22.06.21 um 17:23 schrieb Jason Gunthorpe:
On Tue, Jun 22, 2021 at 02:23:03PM +0200, Christian König wrote:
Am 22.06.21 um 14:01 schrieb Jason Gunthorpe:
On Tue, Jun 22, 2021 at 11:42:27AM +0300, Oded Gabbay wrote:
On Tue, Jun 22, 2021 at 9:37 AM Christian König ckoenig.leichtzumerken@gmail.com wrote:
Am 22.06.21 um 01:29 schrieb Jason Gunthorpe: > On Mon, Jun 21, 2021 at 10:24:16PM +0300, Oded Gabbay wrote: > > > Another thing I want to emphasize is that we are doing p2p only > > through the export/import of the FD. We do *not* allow the user to > > mmap the dma-buf as we do not support direct IO. So there is no access > > to these pages through the userspace. > Arguably mmaping the memory is a better choice, and is the direction > that Logan's series goes in. Here the use of DMABUF was specifically > designed to allow hitless revokation of the memory, which this isn't > even using. The major problem with this approach is that DMA-buf is also used for memory which isn't CPU accessible.
That isn't an issue here because the memory is only intended to be used with P2P transfers so it must be CPU accessible.
No, especially P2P is often done on memory resources which are not even remotely CPU accessible.
That is a special AMD thing, P2P here is PCI P2P and all PCI memory is CPU accessible.
No absolutely not. NVidia GPUs work exactly the same way.
And you have tons of similar cases in embedded and SoC systems where intermediate memory between devices isn't directly addressable with the CPU.
None of that is PCI P2P.
It is all some specialty direct transfer.
You can't reasonably call dma_map_resource() on non CPU mapped memory for instance, what address would you pass?
Do not confuse "I am doing transfers between two HW blocks" with PCI Peer to Peer DMA transfers - the latter is a very narrow subcase.
No, just using the dma_map_resource() interface.
Ik, but yes that does "work". Logan's series is better.
I'll go and read Logan's patch-set to see if that will work for us in the future. Please remember, as Daniel said, we don't have struct page backing our device memory, so if that is a requirement to connect to Logan's work, then I don't think we will want to do it at this point.
It is trivial to get the struct page for a PCI BAR.
Yeah, but it doesn't make much sense. Why should we create a struct page for something that isn't even memory in a lot of cases?
Because the iommu and other places need this handle to setup their stuff. Nobody has yet been brave enough to try to change those flows to be able to use a physical CPU address.
Well that is certainly not true. I'm just not sure if that works with all IOMMU drivers thought.
Huh? All the iommu interfaces except for the dma_map_resource() are struct page based. dma_map_resource() is slow ad limited in what it can do.
Jason