Am 29.03.2018 um 18:25 schrieb Logan Gunthorpe:

On 29/03/18 10:10 AM, Christian König wrote:

...

Why not? I mean the dma_map_resource() function is for P2P while other dma_map_* functions are only for system memory.

Oh, hmm, I wasn't aware dma_map_resource was exclusively for mapping P2P. Though it's a bit odd seeing we've been working under the assumption that PCI P2P is different as it has to translate the PCI bus address. Where as P2P for devices on other buses is a big unknown.

Yeah, completely agree. On my TODO list (but rather far down) is actually supporting P2P with USB devices.

And no, I don't have the slightest idea how to do this at the moment.

...

...

And this is necessary to check if the DMA ops in use support it or not. We can't have the dma_map_X() functions do the wrong thing because they don't support it yet.

Well that sounds like we should just return an error from dma_map_resources() when an architecture doesn't support P2P yet as Alex suggested.

Yes, well except in our patch-set we can't easily use dma_map_resources() as we either have SGLs to deal with or we need to create whole new interfaces to a number of subsystems.

Agree as well. I was also in clear favor of extending the SGLs to have a flag for this instead of the dma_map_resource() interface, but for some reason that didn't made it into the kernel.

...

You don't seem to understand the implications: The devices do have a common upstream bridge! In other words your code would currently claim that P2P is supported, but in practice it doesn't work.

Do they? They don't on any of the Intel machines I'm looking at. The previous version of the patchset not only required a common upstream bridge but two layers of upstream bridges on both devices which would effectively limit transfers to PCIe switches only. But Bjorn did not like this.

At least to me that sounds like a good idea, it would at least disable (the incorrect) auto detection of P2P for such devices.

...

You need to include both drivers which participate in the P2P transaction to make sure that both supports this and give them opportunity to chicken out and in the case of AMD APUs even redirect the request to another location (e.g. participate in the DMA translation).

I don't think it's the drivers responsibility to reject P2P . The topology is what governs support or not. The discussions we had with Bjorn settled on if the devices are all behind the same bridge they can communicate with each other. This is essentially guaranteed by the PCI spec.

Well it is not only rejecting P2P, see the devices I need to worry about are essentially part of the CPU. Their resources looks like a PCI BAR to the BIOS and OS, but are actually backed by stolen system memory.

So as crazy as it sounds what you get is an operation which starts as P2P, but then the GPU drivers sees it and says: Hey please don't write that to my PCIe BAR, but rather system memory location X.

...

DMA-buf fortunately seems to handle all this already, that's why we choose it as base for our implementation.

Well, unfortunately DMA-buf doesn't help for the drivers we are working with as neither the block layer nor the RDMA subsystem have any interfaces for it.

A fact that gives me quite some sleepless nights as well. I think we sooner or later need to extend those interfaces to work with DMA-bufs as well.

I will try to give your patch set a review when I'm back from vacation and rebase my DMA-buf work on top of that.

Regards, Christian.

Logan