Am 22.06.21 um 17:40 schrieb Oded Gabbay:

On Tue, Jun 22, 2021 at 6:31 PM Christian König christian.koenig@amd.com wrote:

...

Am 22.06.21 um 17:28 schrieb Jason Gunthorpe:

...

On Tue, Jun 22, 2021 at 05:24:08PM +0200, Christian König wrote:

...

...

...

I will take two GAUDI devices and use one as an exporter and one as an importer. I want to see that the solution works end-to-end, with real device DMA from importer to exporter.

I can tell you it doesn't. Stuffing physical addresses directly into the sg list doesn't involve any of the IOMMU code so any configuration that requires IOMMU page table setup will not work.

Sure it does. See amdgpu_vram_mgr_alloc_sgt:

      amdgpu_res_first(res, offset, length, &cursor);

       ^^^^^^^^^^

I'm not talking about the AMD driver, I'm talking about this patch.

•         bar_address = hdev->dram_pci_bar_start +
  

•                         (pages[cur_page] - prop->dram_base_address);
  

•         sg_dma_address(sg) = bar_address;
  

Yeah, that is indeed not working.

Oded you need to use dma_map_resource() for this.

Christian.

Yes, of course. But will it be enough ? Jason said that supporting IOMMU isn't nice when we don't have struct pages. I fail to understand the connection, I need to dig into this.

Question is what you want to do with this?

A struct page is always needed if you want to do stuff like HMM with it, if you only want P2P between device I actually recommend to avoid it.

Christian.

Oded

...

...

Jason