Linaro-mm-sig November 2025

linaro-mm-sig@lists.linaro.org

17 participants
84 discussions

[PATCH v6 00/11] vfio/pci: Allow MMIO regions to be exported through dma-buf

by Leon Romanovsky

Changelog: v6: * Fixed wrong error check from pcim_p2pdma_init(). * Documented pcim_p2pdma_provider() function. * Improved commit messages. * Added VFIO DMA-BUF selftest. * Added __counted_by(nr_ranges) annotation to struct vfio_device_feature_dma_buf. * Fixed error unwind when dma_buf_fd() fails. * Document latest changes to p2pmem. * Removed EXPORT_SYMBOL_GPL from pci_p2pdma_map_type. * Moved DMA mapping logic to DMA-BUF. * Removed types patch to avoid dependencies between subsystems. * Moved vfio_pci_dma_buf_move() in err_undo block. * Added nvgrace patch. v5: https://lore.kernel.org/all/cover.1760368250.git.leon@kernel.org * Rebased on top of v6.18-rc1. * Added more validation logic to make sure that DMA-BUF length doesn't overflow in various scenarios. * Hide kernel config from the users. * Fixed type conversion issue. DMA ranges are exposed with u64 length, but DMA-BUF uses "unsigned int" as a length for SG entries. * Added check to prevent from VFIO drivers which reports BAR size different from PCI, do not use DMA-BUF functionality. v4: https://lore.kernel.org/all/cover.1759070796.git.leon@kernel.org * Split pcim_p2pdma_provider() to two functions, one that initializes array of providers and another to return right provider pointer. v3: https://lore.kernel.org/all/cover.1758804980.git.leon@kernel.org * Changed pcim_p2pdma_enable() to be pcim_p2pdma_provider(). * Cache provider in vfio_pci_dma_buf struct instead of BAR index. * Removed misleading comment from pcim_p2pdma_provider(). * Moved MMIO check to be in pcim_p2pdma_provider(). v2: https://lore.kernel.org/all/cover.1757589589.git.leon@kernel.org/ * Added extra patch which adds new CONFIG, so next patches can reuse * it. * Squashed "PCI/P2PDMA: Remove redundant bus_offset from map state" into the other patch. * Fixed revoke calls to be aligned with true->false semantics. * Extended p2pdma_providers to be per-BAR and not global to whole * device. * Fixed possible race between dmabuf states and revoke. * Moved revoke to PCI BAR zap block. v1: https://lore.kernel.org/all/cover.1754311439.git.leon@kernel.org * Changed commit messages. * Reused DMA_ATTR_MMIO attribute. * Returned support for multiple DMA ranges per-dMABUF. v0: https://lore.kernel.org/all/cover.1753274085.git.leonro@nvidia.com --------------------------------------------------------------------------- Based on "[PATCH v6 00/16] dma-mapping: migrate to physical address-based API" https://lore.kernel.org/all/cover.1757423202.git.leonro@nvidia.com/ series. --------------------------------------------------------------------------- This series extends the VFIO PCI subsystem to support exporting MMIO regions from PCI device BARs as dma-buf objects, enabling safe sharing of non-struct page memory with controlled lifetime management. This allows RDMA and other subsystems to import dma-buf FDs and build them into memory regions for PCI P2P operations. The series supports a use case for SPDK where a NVMe device will be owned by SPDK through VFIO but interacting with a RDMA device. The RDMA device may directly access the NVMe CMB or directly manipulate the NVMe device's doorbell using PCI P2P. However, as a general mechanism, it can support many other scenarios with VFIO. This dmabuf approach can be usable by iommufd as well for generic and safe P2P mappings. In addition to the SPDK use-case mentioned above, the capability added in this patch series can also be useful when a buffer (located in device memory such as VRAM) needs to be shared between any two dGPU devices or instances (assuming one of them is bound to VFIO PCI) as long as they are P2P DMA compatible. The implementation provides a revocable attachment mechanism using dma-buf move operations. MMIO regions are normally pinned as BARs don't change physical addresses, but access is revoked when the VFIO device is closed or a PCI reset is issued. This ensures kernel self-defense against potentially hostile userspace. The series includes significant refactoring of the PCI P2PDMA subsystem to separate core P2P functionality from memory allocation features, making it more modular and suitable for VFIO use cases that don't need struct page support. ----------------------------------------------------------------------- The series is based originally on https://lore.kernel.org/all/20250307052248.405803-1-vivek.kasireddy@intel.c… but heavily rewritten to be based on DMA physical API. ----------------------------------------------------------------------- The WIP branch can be found here: https://git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git/log/?h=… Thanks --- Jason Gunthorpe (2): PCI/P2PDMA: Document DMABUF model vfio/nvgrace: Support get_dmabuf_phys Leon Romanovsky (7): PCI/P2PDMA: Separate the mmap() support from the core logic PCI/P2PDMA: Simplify bus address mapping API PCI/P2PDMA: Refactor to separate core P2P functionality from memory allocation PCI/P2PDMA: Provide an access to pci_p2pdma_map_type() function dma-buf: provide phys_vec to scatter-gather mapping routine vfio/pci: Enable peer-to-peer DMA transactions by default vfio/pci: Add dma-buf export support for MMIO regions Vivek Kasireddy (2): vfio: Export vfio device get and put registration helpers vfio/pci: Share the core device pointer while invoking feature functions Documentation/driver-api/pci/p2pdma.rst | 95 +++++++--- block/blk-mq-dma.c | 2 +- drivers/dma-buf/dma-buf.c | 235 ++++++++++++++++++++++++ drivers/iommu/dma-iommu.c | 4 +- drivers/pci/p2pdma.c | 182 +++++++++++++----- drivers/vfio/pci/Kconfig | 3 + drivers/vfio/pci/Makefile | 1 + drivers/vfio/pci/nvgrace-gpu/main.c | 56 ++++++ drivers/vfio/pci/vfio_pci.c | 5 + drivers/vfio/pci/vfio_pci_config.c | 22 ++- drivers/vfio/pci/vfio_pci_core.c | 56 ++++-- drivers/vfio/pci/vfio_pci_dmabuf.c | 315 ++++++++++++++++++++++++++++++++ drivers/vfio/pci/vfio_pci_priv.h | 23 +++ drivers/vfio/vfio_main.c | 2 + include/linux/dma-buf.h | 18 ++ include/linux/pci-p2pdma.h | 120 +++++++----- include/linux/vfio.h | 2 + include/linux/vfio_pci_core.h | 42 +++++ include/uapi/linux/vfio.h | 27 +++ kernel/dma/direct.c | 4 +- mm/hmm.c | 2 +- 21 files changed, 1077 insertions(+), 139 deletions(-) --- base-commit: 3a8660878839faadb4f1a6dd72c3179c1df56787 change-id: 20251016-dmabuf-vfio-6cef732adf5a Best regards, -- Leon Romanovsky <leonro(a)nvidia.com>

3 weeks, 1 day

Re: [PATCH v6 00/11] vfio/pci: Allow MMIO regions to be exported through dma-buf

by Jason Gunthorpe

On Mon, Nov 03, 2025 at 12:07:12PM -0800, Alex Mastro wrote: > On Sun, Nov 02, 2025 at 10:00:48AM +0200, Leon Romanovsky wrote: > > Changelog: > > v6: > > * Fixed wrong error check from pcim_p2pdma_init(). > > * Documented pcim_p2pdma_provider() function. > > * Improved commit messages. > > * Added VFIO DMA-BUF selftest. > > * Added __counted_by(nr_ranges) annotation to struct vfio_device_feature_dma_buf. > > * Fixed error unwind when dma_buf_fd() fails. > > * Document latest changes to p2pmem. > > * Removed EXPORT_SYMBOL_GPL from pci_p2pdma_map_type. > > * Moved DMA mapping logic to DMA-BUF. > > * Removed types patch to avoid dependencies between subsystems. > > * Moved vfio_pci_dma_buf_move() in err_undo block. > > * Added nvgrace patch. > > Thanks Leon. Attaching a toy program which sanity tests the dma-buf export UAPI > by feeding the allocated dma-buf into an dma-buf importer (libibverbs + CX-7). Oh! Here is my toy program to do the same with iommufd as the importer: #define _GNU_SOURCE #define __user #include <fcntl.h> #include <stdio.h> #include <stdlib.h> #include <sys/ioctl.h> #include <unistd.h> #include "include/uapi/linux/vfio.h" #include "include/uapi/linux/iommufd.h" #include <string.h> #include <sys/mman.h> #include <errno.h> int main(int argc, const char *argv[]) { int vfio_dev_fd, iommufd_fd, ret; // Open the per-device VFIO file (e.g., /dev/vfio/devices/vfio3) vfio_dev_fd = open("/dev/vfio/devices/vfio0", O_RDWR); if (vfio_dev_fd < 0) { perror("Failed to open VFIO per-device file"); return 1; } // Open /dev/iommu for iommufd iommufd_fd = open("/dev/iommu", O_RDWR); if (iommufd_fd < 0) { perror("Failed to open /dev/iommu"); close(vfio_dev_fd); return 1; } // Bind device FD to iommufd struct vfio_device_bind_iommufd bind = { .argsz = sizeof(bind), .flags = 0, .iommufd = iommufd_fd, }; ret = ioctl(vfio_dev_fd, VFIO_DEVICE_BIND_IOMMUFD, &bind); if (ret < 0) { perror("VFIO_DEVICE_BIND_IOMMUFD failed"); close(vfio_dev_fd); close(iommufd_fd); return 1; } // Allocate an IOAS (I/O address space) struct iommu_ioas_alloc alloc_data = { .size = sizeof(alloc_data), .flags = 0, }; ret = ioctl(iommufd_fd, IOMMU_IOAS_ALLOC, &alloc_data); if (ret < 0) { perror("IOMMU_IOAS_ALLOC failed"); close(vfio_dev_fd); close(iommufd_fd); return 1; } // Attach the device to the IOAS struct vfio_device_attach_iommufd_pt attach_data = { .argsz = sizeof(attach_data), .flags = 0, .pt_id = alloc_data.out_ioas_id, }; ret = ioctl(vfio_dev_fd, VFIO_DEVICE_ATTACH_IOMMUFD_PT, &attach_data); if (ret < 0) { perror("VFIO_DEVICE_ATTACH_IOMMUFD_PT failed"); close(vfio_dev_fd); close(iommufd_fd); return 1; } #if 0 int mapfd = memfd_create("test", MFD_CLOEXEC); if (mapfd == -1) { perror("memfd_create failed"); return 1; } ftruncate(mapfd, 4096); #else struct dmabuf_arg { struct vfio_device_feature hdr; struct vfio_device_feature_dma_buf dma_buf; struct vfio_region_dma_range range; } dma_buf_feature = { .hdr = { .argsz = sizeof(dma_buf_feature), .flags = VFIO_DEVICE_FEATURE_GET | VFIO_DEVICE_FEATURE_DMA_BUF }, .dma_buf = { .region_index = VFIO_PCI_BAR0_REGION_INDEX, .open_flags = O_CLOEXEC, .nr_ranges = 1 }, .range = { .length = 4096 }, }; ret = ioctl(vfio_dev_fd, VFIO_DEVICE_FEATURE, &dma_buf_feature); if (ret < 0) { perror("VFIO_DEVICE_FEATURE_GET failed"); return 1; } int mapfd = ret; #endif struct iommu_ioas_map_file map_file = { .size = sizeof(map_file), .flags = IOMMU_IOAS_MAP_WRITEABLE | IOMMU_IOAS_MAP_READABLE, .ioas_id = alloc_data.out_ioas_id, .fd = mapfd, .start = 0, .length = 4096, }; ret = ioctl(iommufd_fd, IOMMU_IOAS_MAP_FILE, &map_file); if (ret < 0) { perror("IOMMU_IOAS_MAP_FILE failed"); return 1; } printf("Successfully attached device to IOAS ID: %u\n", alloc_data.out_ioas_id); close(vfio_dev_fd); close(iommufd_fd); return 0; }

3 weeks, 1 day

Re: [PATCH v6 10/11] vfio/pci: Add dma-buf export support for MMIO regions

by Leon Romanovsky

On Sun, Nov 2, 2025, at 19:11, Alex Williamson wrote: > On Sun, 2 Nov 2025 17:12:53 +0200 > Leon Romanovsky <leon(a)kernel.org> wrote: >> On Sun, Nov 02, 2025 at 08:01:37AM -0700, Alex Williamson wrote: >> > We don't need the separate loop or flag, and adding it breaks the >> > existing reverse list walk. Thanks, >> >> Do you want me to send v7? I have a feeling that v6 is good to be merged. > > Let's hold off, if this ends up being the only fixup I can roll it in. > Thanks, Thanks > > Alex > >> diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c >> index 24204893e221..51a3bcc26f8b 100644 >> --- a/drivers/vfio/pci/vfio_pci_core.c >> +++ b/drivers/vfio/pci/vfio_pci_core.c >> @@ -2403,7 +2403,6 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, >> struct iommufd_ctx *iommufd_ctx) >> { >> struct vfio_pci_core_device *vdev; >> - bool restore_revoke = false; >> struct pci_dev *pdev; >> int ret; >> >> @@ -2473,7 +2472,6 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, >> } >> >> vfio_pci_dma_buf_move(vdev, true); >> - restore_revoke = true; >> vfio_pci_zap_bars(vdev); >> } >> >> @@ -2501,15 +2499,12 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, >> struct vfio_pci_core_device, vdev.dev_set_list); >> >> err_undo: >> - if (restore_revoke) { >> - list_for_each_entry(vdev, &dev_set->device_list, vdev.dev_set_list) >> - if (__vfio_pci_memory_enabled(vdev)) >> - vfio_pci_dma_buf_move(vdev, false); >> - } >> - >> list_for_each_entry_from_reverse(vdev, &dev_set->device_list, >> - vdev.dev_set_list) >> + vdev.dev_set_list) { >> + if (__vfio_pci_memory_enabled(vdev)) >> + vfio_pci_dma_buf_move(vdev, false); >> up_write(&vdev->memory_lock); >> + } >> >> list_for_each_entry(vdev, &dev_set->device_list, vdev.dev_set_list) >> pm_runtime_put(&vdev->pdev->dev); >> >> >> > >> > Alex >> > >>

3 weeks, 2 days

Re: [PATCH v6 10/11] vfio/pci: Add dma-buf export support for MMIO regions

by Leon Romanovsky

On Sun, Nov 02, 2025 at 08:01:37AM -0700, Alex Williamson wrote: > On Sun, 2 Nov 2025 10:00:58 +0200 > Leon Romanovsky <leon(a)kernel.org> wrote: > > @@ -2391,6 +2403,7 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, > > struct iommufd_ctx *iommufd_ctx) > > { > > struct vfio_pci_core_device *vdev; > > + bool restore_revoke = false; > > struct pci_dev *pdev; > > int ret; > > > > @@ -2459,6 +2472,8 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, > > break; > > } > > > > + vfio_pci_dma_buf_move(vdev, true); > > + restore_revoke = true; > > vfio_pci_zap_bars(vdev); > > } > > > > @@ -2486,6 +2501,12 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, > > struct vfio_pci_core_device, vdev.dev_set_list); > > > > err_undo: > > + if (restore_revoke) { > > + list_for_each_entry(vdev, &dev_set->device_list, vdev.dev_set_list) > > + if (__vfio_pci_memory_enabled(vdev)) > > + vfio_pci_dma_buf_move(vdev, false); > > + } > > + > > list_for_each_entry_from_reverse(vdev, &dev_set->device_list, > > vdev.dev_set_list) > > We don't need the separate loop or flag, and adding it breaks the > existing reverse list walk. Thanks, Do you want me to send v7? I have a feeling that v6 is good to be merged. Thanks diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index 24204893e221..51a3bcc26f8b 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -2403,7 +2403,6 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, struct iommufd_ctx *iommufd_ctx) { struct vfio_pci_core_device *vdev; - bool restore_revoke = false; struct pci_dev *pdev; int ret; @@ -2473,7 +2472,6 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, } vfio_pci_dma_buf_move(vdev, true); - restore_revoke = true; vfio_pci_zap_bars(vdev); } @@ -2501,15 +2499,12 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, struct vfio_pci_core_device, vdev.dev_set_list); err_undo: - if (restore_revoke) { - list_for_each_entry(vdev, &dev_set->device_list, vdev.dev_set_list) - if (__vfio_pci_memory_enabled(vdev)) - vfio_pci_dma_buf_move(vdev, false); - } - list_for_each_entry_from_reverse(vdev, &dev_set->device_list, - vdev.dev_set_list) + vdev.dev_set_list) { + if (__vfio_pci_memory_enabled(vdev)) + vfio_pci_dma_buf_move(vdev, false); up_write(&vdev->memory_lock); + } list_for_each_entry(vdev, &dev_set->device_list, vdev.dev_set_list) pm_runtime_put(&vdev->pdev->dev); > > Alex >

3 weeks, 2 days

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

Linaro-mm-sig November 2025