Am 13.03.20 um 12:21 schrieb Christoph Hellwig:

On Thu, Mar 12, 2020 at 11:19:28AM -0300, Jason Gunthorpe wrote:

...

The non-page scatterlist is also a big concern for RDMA as we have drivers that want the page list, so even if we did as this series contemplates I'd have still have to split the drivers and create the notion of a dma-only SGL.

The drivers I looked at want a list of IOVA address, aligned to the device "page size". What other data do drivers want?

Well for GPUs I have the requirement that those IOVA addresses allow random access.

That's the reason why we currently convert the sg_table into a linear arrays of addresses and pages. To solve that keeping the length in separate optional array would be ideal for us.

But this is so a special use case that I'm not sure if we want to support this in the common framework or not.

Execept for the software protocol stack drivers, which of couse need pages for the stack futher down.

Yes completely agree.

For the GPUs I will propose a patch to stop copying the page from the sg_table over into our linear arrays and see if anybody starts to scream.

I don't think so, but probably better to double check.

Thanks, Christian.

...

I haven't used bio_vecs before, do they support chaining like SGL so they can be very big? RDMA dma maps gigabytes of memory

bio_vecs itself don't have the chaining, but the bios build around them do. But each entry can map a huge pile. If needed we could use the same chaining scheme we use for scatterlists for bio_vecs as well, but lets see if we really end up needing that.

...

So I'm guessing the path forward is something like

• Add some generic dma_sg data structure and helper
• Add dma mapping code to go from pages to dma_sg

That has been on my todo list for a while. All the DMA consolidatation is to prepare for that and we're finally getting close.