Linaro-mm-sig

linaro-mm-sig@lists.linaro.org

13 participants
6543 discussions

Re: [RFC PATCH] misc: fgds: enable GPU-NVMe direct I/O via POSIX and io_uring

by Christian König

On 9/10/26 05:57, Li Wang wrote: ... >> AMD came up with something similar, but all those approaches are so fundamentally broken that we didn't even considered upstreaming it. >>> And you are using this as a "bypass" for the normal accel subsystem, >>> shouldn't this be part of that subsystem instead of a custom user/kernel >>> api like you are creating here? >> >> As far as I know there is a patch set under review and even already partially merged which enables exactly that functionality as general feature for DMA-buf which is vendor independent and should at least in theory work with all drivers. >> >> I'm really surprised that somebody is still working on the vendor specific stuff. > As you pointed out, every vendor has been inventing their own way and interfaces to support GDS, > introducing custom kernel modules and proprietary UAPI interfaces, with varying performance that > leaves developers heavily frustrated. Apologies for not making this clear enough in our commit > messages, which understandably caused some confusion. We merely borrowed the name "GDS" to describe > the functional purpose of fgds. > > In fact, we believe fgds offers four key advantages: > (1) GPU platform independence; > (2) POSIX/io_uring interface compatibility; > (3) Higher performance than GDS; > (4) Minimal kernel footprint and UAPI footprint > > Regarding (1), (2), and (3), please allow me to briefly explain the design mechanism of fgds: > fgds turns a GPU memory buffer into a POSIX/io_uring-compatible user-space virtual address via > three main steps: > > Step 1: Utilizing ZONE_DEVICE support, we remap the GPU memory exposed via PCIe BAR into struct pages > using devm_memremap_pages(); > > Step 2: Utilizing dma-buf support, the GPU memory buffer is exported as a dma-buf file descriptor (fd). > Using this fd as a bridge, we look up the corresponding DMA addresses for the GPU memory buffer inside > the kernel; > > Step 3: Through mmap, we insert the struct pages corresponding to the GPU memory buffer into the userspace > VMA, mapping their physical/DMA addresses directly. The virtual address returned by mmap can then be directly > passed into standard POSIX or io_uring interfaces. Well long story short what you do here is completely broken. Approaches like those have been suggested before and we added both documentation as well as code to prevent such hacks from working. Please see Pavel Begunkov patch set on the LKML which adds DMA-buf support to io_uring for how to do it correctly. Just google for "Add dmabuf read/write via io_uring". Regards, Christian.

2 hours, 22 minutes

Re: [PATCH v6 04/12] arm64/mm: Export set_direct_map_*_noflush() APIs

by Christoph Hellwig

On Fri, Sep 04, 2026 at 12:44:55PM +0200, Thierry Reding wrote: > From: Thierry Reding <treding(a)nvidia.com> > > Drivers that use this may want to be built as a module, so export them. That's one of the worst commit log ever. No, we don't just export core symbols dealing with the kernel direct map because "Drivers that use this may want to be built as a module". For one exporting this at all needs a very good justification and not just hand waiving. But more importantly if we can't avoid exporting it, it needs to be exported at the tightest sensible scope. E.g. for a given module if it is so special, or a namespace if it's not that special. But in doubt we should have a proper core abstraction instead of opening up direct map manipulation to random modules.

5 hours, 6 minutes

Re: [RFC PATCH] misc: fgds: enable GPU-NVMe direct I/O via POSIX and io_uring

by Greg Kroah-Hartman

On Wed, Sep 09, 2026 at 06:42:03PM +0800, Li Wang wrote: > Hi Greg, > Thanks for the review! > > > > > That's not really needed in a changelog text, it could be in the 0/X > > patch :) > > > Sorry for the clutter. I will move most of them into the 0/X patch in v2. > > > Anyway, you didn't cc: the io_uring list, why? > > > `scripts/get_maintainer.pl` didn't output the io_uring mailing list, likely > because this patch doesn't directly touch the io_uring codebase itself. It only > enables remapping GPU memory buffers to CPU virtual addresses, which can then be > consumed via standard io_uring APIs. > > I've added io-uring(a)vger.kernel.org to CC for this reply and will keep it in v2. Great, as you are using that as the api, there might be some parts that will need to be reviewed by them. > > Nor why "fgds" is the name, that's going to be hard to remember, does it > > stand for something? > > > "FGDS" stands for Fast GPUDirect Storage. GPUDirect Storage (GDS) is NVIDIA's > technology enabling direct I/O between GPU memory and files on NVMe, > widely used in LLM workloads to bypass CPU overhead. That's nvidia's specific solution, but this works on other devices, right? Or just for that one platform? And you are using this as a "bypass" for the normal accel subsystem, shouldn't this be part of that subsystem instead of a custom user/kernel api like you are creating here? thanks, greg k-h

21 hours, 12 minutes

Re: [PATCH v6 08/12] dma-buf: heaps: Add debugfs support

by Christian König

Since AMDs mail servers blocked me from sending this out, here once more from my gmail account. Regards, Christian. On 9/4/26 15:00, Christian König wrote: > On 9/4/26 12:44, Thierry Reding wrote: >> From: Thierry Reding <treding(a)nvidia.com> >> >> Add a callback to struct dma_heap_ops that heap providers can implement >> to show information about the state of the heap in debugfs. A top-level >> directory named "dma_heap" is created in debugfs and individual files >> will be named after the heaps. >> >> Reviewed-by: Maxime Ripard <mripard(a)kernel.org> >> Signed-off-by: Thierry Reding <treding(a)nvidia.com> > > One minimal nit pick below, with that taken care of Reviewed-by: Christian König <christian.koenig(a)amd.com>. > >> --- >> Changes in v5: >> - fix failure handling during debugfs root directory creation >> - add more cleanup to the newly introduced dma_heap_exit() >> --- >> drivers/dma-buf/dma-heap.c | 52 ++++++++++++++++++++++++++++++++++++++++++++++ >> include/linux/dma-heap.h | 2 ++ >> 2 files changed, 54 insertions(+) >> >> diff --git a/drivers/dma-buf/dma-heap.c b/drivers/dma-buf/dma-heap.c >> index 8a6c2b6e0cc6..65c80dac6500 100644 >> --- a/drivers/dma-buf/dma-heap.c >> +++ b/drivers/dma-buf/dma-heap.c >> @@ -7,6 +7,7 @@ >> */ >> >> #include <linux/cdev.h> >> +#include <linux/debugfs.h> >> #include <linux/device.h> >> #include <linux/dma-buf.h> >> #include <linux/dma-heap.h> >> @@ -225,6 +226,40 @@ const char *dma_heap_get_name(struct dma_heap *heap) >> } >> EXPORT_SYMBOL_NS_GPL(dma_heap_get_name, "DMA_BUF_HEAP"); >> >> +#ifdef CONFIG_DEBUG_FS >> +static int dma_heap_debug_show(struct seq_file *s, void *unused) >> +{ >> + struct dma_heap *heap = s->private; >> + int err = 0; >> + >> + if (heap->ops && heap->ops->show) >> + err = heap->ops->show(s, heap); >> + >> + return err; >> +} >> +DEFINE_SHOW_ATTRIBUTE(dma_heap_debug); >> + >> +static struct dentry *dma_heap_debugfs_dir; >> + >> +static void dma_heap_init_debugfs(void) >> +{ >> + dma_heap_debugfs_dir = debugfs_create_dir("dma_heap", NULL); >> +} >> + >> +static void dma_heap_exit_debugfs(void) >> +{ >> + debugfs_remove_recursive(dma_heap_debugfs_dir); >> +} >> +#else >> +static void dma_heap_init_debugfs(void) >> +{ >> +} >> + >> +static void dma_heap_exit_debugfs(void) >> +{ >> +} >> +#endif >> + >> /** >> * dma_heap_add - adds a heap to dmabuf heaps >> * @exp_info: information needed to register this heap >> @@ -299,6 +334,13 @@ struct dma_heap *dma_heap_add(const struct dma_heap_export_info *exp_info) >> >> /* Add heap to the list */ >> list_add(&heap->list, &heap_list); >> + >> +#ifdef CONFIG_DEBUG_FS >> + if (heap->ops && heap->ops->show) >> + debugfs_create_file(heap->name, 0444, dma_heap_debugfs_dir, >> + heap, &dma_heap_debug_fops); >> +#endif >> + >> mutex_unlock(&heap_list_lock); >> >> return heap; >> @@ -335,6 +377,16 @@ static int dma_heap_init(void) >> } >> dma_heap_class->devnode = dma_heap_devnode; >> >> + dma_heap_init_debugfs(); >> + >> return 0; >> } >> subsys_initcall(dma_heap_init); >> + >> +static void __exit dma_heap_exit(void) >> +{ >> + dma_heap_exit_debugfs(); >> + class_destroy(dma_heap_class); >> + unregister_chrdev_region(dma_heap_devt, NUM_HEAP_MINORS); >> +} >> +__exitcall(dma_heap_exit); >> diff --git a/include/linux/dma-heap.h b/include/linux/dma-heap.h >> index 648328a64b27..1c9bed1f4dde 100644 >> --- a/include/linux/dma-heap.h >> +++ b/include/linux/dma-heap.h >> @@ -12,6 +12,7 @@ >> #include <linux/types.h> >> >> struct dma_heap; >> +struct seq_file; >> >> /** >> * struct dma_heap_ops - ops to operate on a given heap >> @@ -24,6 +25,7 @@ struct dma_heap_ops { >> unsigned long len, >> u32 fd_flags, >> u64 heap_flags); >> + int (*show)(struct seq_file *s, struct dma_heap *heap); > > I found it good coding practice for objects ops pointer tables to have the object the function works with as first parameter. > > Absolutely not a must have, but just looks a little bit more consistent. > > Regards, > Christian. > >> }; >> >> /** >> >

23 hours, 53 minutes

Re: [RFC PATCH] misc: fgds: enable GPU-NVMe direct I/O via POSIX and io_uring

by Greg Kroah-Hartman

On Tue, Sep 08, 2026 at 09:15:45PM +0800, Li Wang wrote: > From: Mengmeng Zhao <zhaomengmeng(a)kylinos.cn> > > Inspired by the paper published in SC'25 [1], we implemented a character > device named fgds that provides two ioctl interfaces: > `REG_BUFFER/UNREG_BUFFER`. It enables applications to perform direct I/O > between GPU memory and NVMe via POSIX and io_uring APIs. This is > particularly useful for LLM workloads, such as model loading, KV cache > offloading, and checkpointing. The fgds device corresponds one-to-one with > the PCIe GPU on the machine. The usage is straightforward: an application > simply opens the corresponding fgds device, calls ioctl on the returned fd > with REG_BUFFER, taking the target GPU memory buffer address (represented > as a dma-buf fd), and the buffer length as inputs, and then invokes mmap on > the fgds device fd, using the return value of ioctl as the input. The mmap > call returns a CPU virtual address (call it cpu_vaddr). Afterward, > cpu_vaddr can be passed directly to pread/pwrite, or > io_uring_prep_read/io_uring_prep_write to perform direct I/O between files > on NVMe and GPU memory. A minimal working example can be found in [2]. > The underlying mechanism is that, with the support of fgds device, > cpu_vaddr is made to point directly to the GPU memory buffer corresponding > to the dma-buf fd. This solution is loosely coupled with the GPU vendor's > driver; the GPU vendor only needs to support exporting the allocated GPU > memory buffer through the standard Linux kernel dma-buf framework, which > the vast majority of mainstream GPUs already support. This allows both > applications and the fgds device to work seamlessly with GPUs from > different vendors without any modifications. Furthermore, applications no > longer need to call vendor-specific proprietary APIs (such as NVIDIA's > cuFile API) or install vendor-specific kernel modules (such as NVIDIA's > nvidia-fs.ko) for different GPU vendors. We have tested fgds on GPU cards > from NVIDIA, AMD, and several other vendors, and it works well. > > Besides the benefits in ease of use and compatibility, another key > advantage of this solution is higher performance. [2] presents the > performance comparison results between fgds and NVIDIA GDS. Because fgds > eliminates the overhead of phony buffers incurred by NVIDIA GDS, it > achieves significantly higher performance. For example, for reads, fgds > outperforms GDS by 11% to 109%; for writes, fgds outperforms GDS by 10% > to 71%. > > To further accelerate the read and write operations of large files or > massive data volumes—which are very common in LLM scenarios—we have > implemented library functions `fgds_read` and `fgds_write`. Under the hood, > these interfaces split large data into chunks and submit them > asynchronously and in parallel via io_uring, thereby further boosting I/O > performance, with read performance improved by up to 115% and write > performance by up to 40%. In addition, we also provide the `fgds_register` > library interface to encapsulate the `open`, `ioctl' and `mmap` operations. > Readers who are interested can refer to [2]. > > In addition, we have added the LMCache backend, enabling vLLM to offload KV > cache via LMCache using fgds, which accelerates inference performance. We > also added PyTorch APIs, compatible with the PyTorch GDS API, to improve > the performance of reading and writing checkpoints during LLM training. > > We look forward to community feedback and are fully committed to iterating > on this series to work towards upstreaming. That's not really needed in a changelog text, it could be in the 0/X patch :) Anyway, you didn't cc: the io_uring list, why? Also, as a first cut, please see the sashiko comments on this patch: https://sashiko.dev/#/patchset/20260908131545.105987-1-liwang@kylinos.cn > > [1] https://dl.acm.org/doi/10.1145/3712285.3759862 > [2] https://github.com/Storage-and-OS-for-AI/fgds > > Signed-off-by: Mengmeng Zhao <zhaomengmeng(a)kylinos.cn> > Signed-off-by: Li Wang <liwang(a)kylinos.cn> > --- > drivers/misc/Kconfig | 9 + > drivers/misc/Makefile | 1 + > drivers/misc/fgds.c | 989 ++++++++++++++++++++++++++++++++++++++ > include/uapi/linux/fgds.h | 54 +++ > 4 files changed, 1053 insertions(+) > create mode 100644 drivers/misc/fgds.c > create mode 100644 include/uapi/linux/fgds.h > > diff --git a/drivers/misc/Kconfig b/drivers/misc/Kconfig > index 7364931dad3a..2f3a5a5fd0bf 100644 > --- a/drivers/misc/Kconfig > +++ b/drivers/misc/Kconfig > @@ -568,6 +568,15 @@ config MCHP_LAN966X_PCI > - lan966x-miim (MDIO_MSCC_MIIM) > - lan966x-switch (LAN966X_SWITCH) > > +config FGDS > + tristate "GPU-NVMe direct I/O control driver" > + depends on PCI && DMA_SHARED_BUFFER && ZONE_DEVICE > + help > + Say Y here if you want to support GPU-NVME direct I/O > + via POSIX/io_uring interfaces. > + > + If unsure, say N. Module name is not listed here. Nor why "fgds" is the name, that's going to be hard to remember, does it stand for something? > + > source "drivers/misc/c2port/Kconfig" > source "drivers/misc/eeprom/Kconfig" > source "drivers/misc/cb710/Kconfig" > diff --git a/drivers/misc/Makefile b/drivers/misc/Makefile > index e8d8d5d88c0d..04985abe1678 100644 > --- a/drivers/misc/Makefile > +++ b/drivers/misc/Makefile > @@ -71,3 +71,4 @@ obj-y += keba/ > obj-y += amd-sbi/ > obj-$(CONFIG_MISC_RP1) += rp1/ > obj-$(CONFIG_INTEL_SSEI) += issei/ > +obj-$(CONFIG_FGDS) += fgds.o > diff --git a/drivers/misc/fgds.c b/drivers/misc/fgds.c > new file mode 100644 > index 000000000000..3aa4945f701b > --- /dev/null > +++ b/drivers/misc/fgds.c > @@ -0,0 +1,989 @@ > +// SPDX-License-Identifier: GPL-2.0 > +/* > + * Fast GPU Direct Storage via dma-buf. > + * > + * Copyright (C) 2026 KylinSoft. Co., Ltd. All rights reserved. > + * > + * Maps GPU memory into user space to enable direct NVME-to-GPU DMA > + * pread/pwrite syscalls. BAR pages are remapped into ZONE_DEVICE via > + * devm_memremap_pages() and populated using dma-buf backing pages. > + */ > +#define pr_fmt(fmt) "fgds: " fmt You are a driver, always use dev_*() print functions, not pr_() functions, as you will loose the device information. For example: > +/* > + * BAR-based mapping requires device physical addresses. When using > + * IOMMU, DMA addresses are IOVAs, which cannot be mapped directly. > + */ > +static int fgds_check_gpu_iommu(struct pci_dev *pdev) > +{ > + struct iommu_domain *domain; > + > + domain = iommu_get_domain_for_dev(&pdev->dev); > + if (domain && domain->type != IOMMU_DOMAIN_IDENTITY) { > + pr_warn("%s: reject attaching a translating IOMMU domain (requires iommu=pt or off\n", > + dev_name(&pdev->dev)); Should be dev_warn(), right? But what can userspace do with that warning, did something just break? > + pr_info("loaded successfully: %u GPU(s) active\n", fgds_dev_count); When drivers work, they are quiet, please remove this, and the other pr_info() lines, as they seem to be left over from your debugging. thanks, greg k-h

1 day, 4 hours

Re: [PATCH v2 00/15] accel/qda: Qualcomm DSP Accelerator driver

by Bjorn Andersson

On Wed, Aug 26, 2026 at 06:37:00PM +0530, Ekansh Gupta wrote: > On 20-08-2026 20:17, Rob Clark wrote: > > On Wed, Aug 19, 2026 at 11:15 PM Krzysztof Kozlowski <krzk(a)kernel.org> wrote: > >> > >> On 19/08/2026 17:48, Rob Clark wrote: > >>> On Wed, Aug 19, 2026 at 8:27 AM Krzysztof Kozlowski <krzk(a)kernel.org> wrote: > >>>> The rule of usptream development is that we do not accept duplicated > >>>> code, just because a vendor wants to write something new. This is > >>>> basically the concept applied all over the drivers tree, where we pushed > >>>> back against all sorts of duplications all over the vendors. > >>>> > >>>> What I miss in this thread is why would there be any exception here. We > >>>> do not grant exceptions from standard practices on "I want" reasons. > >>> > >>> I agree that we should not have duplicated drivers just for vendor > >>> lolz. But when it comes to adopting common frameworks and integrating > >>> better into the ecosystem, this doesn't seem like something we should > >>> actively discourage. I don't think this is a case of vendor lolz, but > >> > >> No one discourages it. Following standard Linux kernel practices and > >> requirements is not discouraging, do not twist the narrative here. > >> Again, it is standard upstream review telling that we do not duplicate > >> drivers. Ever, unless there is serious exception needed. > > > > I wasn't trying to twist the narrative, just trying to come up with a > > path forward that isn't "no" or "improve existing driver", since > > neither of those gets us towards a future using common frameworks. > > > >> I asked why there should be an exception granted? Is the reason for > >> exception following: > >> "We want to adopt common framework" > >> ? > > > > Possibly? But I don't think we want two drivers to be any sort of > > long term solution. (Ie. as long as venus/iris have co-exist.) > > > >> > >>> rather reacting to drm/accel emerging as the standard framework for > >>> this sort of driver. > >>> > >>> So how do we get from here to there? > >> > >> What is wrong with my proposal? > > > > Maybe I missed something, my understanding was your proposal was > > "Grow/replace/improve existing driver instead of coming with a > > duplicate".. grow or improve doesn't move us toward common > > frameworks. Maybe "replace" is a valid option. If there is something > > I missed, then I apologize. > > > > Options I can think of are: > > > > 1. Hardware cutoff.. new hw gets new driver, existing hw gets existing > > driver > > 2. Backwards compat chardev registered by new driver, providing existing > > UABI. I'm not 100% sure about the feasibility/drawbacks of this.. > > AFAIU the fastrpc folks where planning a backwards compat layer in > > userspace, so maybe it is possible. > > 3. exception? > > > > I'd like to know what the feasibility of #2 is, since at a high level > > that sounds like the best option. Possibly limit exposure of legacy > > UABI to existing hw so we don't get into a place of needing to extend > > the legacy UABI for new hw? > > > > But #1 sounds like a non-controversial place to start regardless. > > Possibly with #2 coming as followup and necessary step before eventual > > migration to new driver for existing hw? > > > > Even if we start with #2, how do we handle first-merge-window > > bugs/regressions without reverting addition of new driver and removal > > of old? It seems like we'd need a window of a couple release cycles > > where both drivers exist? > > > > Maybe others have other/better options in mind? > To all, I'm seeking on the approach I should follow to go ahead here. I > can work on implementing #1(as per Rob's list) with hw specific > compatible for v4 if it's acceptable. > I don't see any reason for you to define a "hw specific compatible", because as you have shown in this series (and as Rob point out), there's no difference in the "hardware". The only reason for your "hw specific compatible" is to make a software selection in Linux - and that's not what DeviceTree is for. As such, I don't see that you have a DeviceTree problem at all, because this is a Linux-internal problem. > #2(compat driver) is something that we are still exploring as we > couldn't find any standard way to achieve it. We might start a separate > discussion for that once we have few possible designs with us. > This is the actual problem! We have existing user space that depends on the ioctl interface exposed by the current misc driver. You must not break these. Hardware cutoff is not a viable solution, because that's just a declaration that we'll let the old platforms rotten - or alternatively you commit to maintain two drivers to the very same feature and quality level. So the only reasonable solution is #2; from there it's a valid question if you reach that point my stepwise migrating the current misc driver that solution, or if you present a new driver with the fully backwards compatible interface, alongside the new ABI. But this does bring to a question which the cover letter should explain - but doesn't: what problem does this patch series actually solve? Regards, Bjorn > Happy to take any other suggestion also.> > > BR, > > -R > >

1 day, 12 hours

Re: [PATCH v6 06/12] of: Export of_node_to_nid()

by Rob Herring

On Fri, Sep 04, 2026 at 12:44:57PM +0200, Thierry Reding wrote: > From: Thierry Reding <treding(a)nvidia.com> > > Drivers buildable as a module may want to use this, so export it. We need a specific reason/driver. Otherwise, that's every API function. It doesn't look to me like the VPR driver ever actually uses the NID. > > Signed-off-by: Thierry Reding <treding(a)nvidia.com> > --- > drivers/of/of_numa.c | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/drivers/of/of_numa.c b/drivers/of/of_numa.c > index 230d5f628c1b..da2e9bd1717c 100644 > --- a/drivers/of/of_numa.c > +++ b/drivers/of/of_numa.c > @@ -171,6 +171,7 @@ int of_node_to_nid(struct device_node *device) > > return NUMA_NO_NODE; > } > +EXPORT_SYMBOL(of_node_to_nid); Assuming this is still needed, EXPORT_SYMBOL_GPL. Rob

1 day, 19 hours

Re: [PATCH v6 09/12] dma-buf: heaps: Add support for Tegra VPR

by Rob Herring

On Fri, Sep 04, 2026 at 12:45:00PM +0200, Thierry Reding wrote: > From: Thierry Reding <treding(a)nvidia.com> > > NVIDIA Tegra SoCs commonly define a Video-Protection-Region, which is a > region of memory dedicated to content-protected video decode and > playback. This memory cannot be accessed by the CPU and only certain > hardware devices have access to it. > > Expose the VPR as a DMA heap so that applications and drivers can > allocate buffers from this region for use-cases that require this kind > of protected memory. > > VPR has a few very critical peculiarities. First, it must be a single > contiguous region of memory (there is a single pair of registers that > set the base address and size of the region), which is configured by > calling back into the secure monitor. The memory region also needs to > quite large for some use-cases because it needs to fit multiple video > frames (8K video should be supported), so VPR sizes of ~2 GiB are > expected. However, some devices cannot afford to reserve this amount > of memory for a particular use-case, and therefore the VPR must be > resizable. > > Unfortunately, resizing the VPR is slightly tricky because the GPU found > on Tegra SoCs must be in reset during the VPR resize operation. This is > currently implemented by freezing all userspace processes and calling > invoking the GPU's freeze() implementation, resizing and the thawing the > GPU and userspace processes. This is quite heavy-handed, so eventually > it might be better to implement thawing/freezing in the GPU driver in > such a way that they block accesses to the GPU so that the VPR resize > operation can happen without suspending all userspace. > > In order to balance the memory usage versus the amount of resizing that > needs to happen, the VPR is divided into multiple chunks. Each chunk is > implemented as a section of the CMA area that is completely allocated on > first use to guarantee the contiguity of the VPR. Once all buffers from > a chunk have been freed, the subsection is freed and the memory returned > to the system. > > The Tegra VPR driver is split into two pieces: one tiny part of the > driver is always built-in and sets up the CMA area during early boot, > whereas the second, larger, part provides the DMA heap implementation > and can be built as a module. Note that tearing down VPR is tricky and > usually not necessary, so it can currently not be unloaded. > > Signed-off-by: Thierry Reding <treding(a)nvidia.com> > --- > Changes in v6: > - split code into a small core and the main chunk so the latter can be > built as a module > > Changes in v5: > - use newly introduced cma_alloc_at() and work with a single CMA area > - setup CMA early and initialize VPR later during boot > - remove some unused variables > - use kalloc_objs() > > Changes in v4: > - address Sashiko and checkpatch comments > - fully remove from linear map while chunks are allocated > - improve error handling > - remove freezer support > > Changes in v3: > - use set_memory_device() and set_memory_normal() helpers > - use kzalloc_obj() instead of kzalloc() with sizeof() > > Changes in v2: > - cluster allocations to reduce the number of resize operations > - support cross-chunk allocation > --- > drivers/dma-buf/heaps/Kconfig | 12 + > drivers/dma-buf/heaps/Makefile | 10 + > drivers/dma-buf/heaps/tegra-vpr-init.c | 133 ++++ > drivers/dma-buf/heaps/tegra-vpr.c | 1210 ++++++++++++++++++++++++++++++++ > drivers/dma-buf/heaps/tegra-vpr.h | 73 ++ > include/trace/events/tegra_vpr.h | 57 ++ > 6 files changed, 1495 insertions(+) > > diff --git a/drivers/dma-buf/heaps/Kconfig b/drivers/dma-buf/heaps/Kconfig > index bb729e91545c..28d2c0800fb5 100644 > --- a/drivers/dma-buf/heaps/Kconfig > +++ b/drivers/dma-buf/heaps/Kconfig > @@ -20,3 +20,15 @@ config DMABUF_HEAPS_CMA > Choose this option to enable dma-buf CMA heap. This heap is backed > by the Contiguous Memory Allocator (CMA). If your system has these > regions, you should say Y here. > + > +config DMABUF_HEAPS_TEGRA_VPR > + tristate "NVIDIA Tegra Video-Protected-Region DMA-BUF Heap" > + depends on DMABUF_HEAPS && DMA_CMA > + help > + Choose this option to enable Video-Protected-Region (VPR) support on > + a range of NVIDIA Tegra devices. Access to VPR memory is limited to > + a subset of hardware engines and specifically disallowed from the > + CPU. The region can be fixed, in which case no linear mapping exists > + for the memory, or it can be resizable on systems that want to reuse > + the memory for other uses when content-protected video is not played > + back. > diff --git a/drivers/dma-buf/heaps/Makefile b/drivers/dma-buf/heaps/Makefile > index 974467791032..481fcb78f757 100644 > --- a/drivers/dma-buf/heaps/Makefile > +++ b/drivers/dma-buf/heaps/Makefile > @@ -1,3 +1,13 @@ > # SPDX-License-Identifier: GPL-2.0 > obj-$(CONFIG_DMABUF_HEAPS_SYSTEM) += system_heap.o > obj-$(CONFIG_DMABUF_HEAPS_CMA) += cma_heap.o > + > +# > +# The reserved-memory bits always need to be built-in so that the CMA > +# initialization runs during early boot. The VPR driver itself can be > +# built as a module. > +# > +ifneq ($(CONFIG_DMABUF_HEAPS_TEGRA_VPR),) > +obj-y += tegra-vpr-init.o > +obj-$(CONFIG_DMABUF_HEAPS_TEGRA_VPR) += tegra-vpr.o > +endif > diff --git a/drivers/dma-buf/heaps/tegra-vpr-init.c b/drivers/dma-buf/heaps/tegra-vpr-init.c > new file mode 100644 > index 000000000000..65e917713c6d > --- /dev/null > +++ b/drivers/dma-buf/heaps/tegra-vpr-init.c > @@ -0,0 +1,133 @@ > +// SPDX-License-Identifier: GPL-2.0 > +/* > + * DMA-BUF restricted heap exporter for NVIDIA Video-Protection-Region (VPR) > + * > + * Copyright (C) 2024-2026 NVIDIA Corporation > + */ > + > +#define pr_fmt(fmt) "tegra-vpr: " fmt > + > +#include "tegra-vpr.h" > + > +#include <linux/cma.h> > +#include <linux/of_reserved_mem.h> > + > +static DEFINE_MUTEX(vpr_lock); > +static LIST_HEAD_GUARDED(vpr_list, vpr_lock); > + > +static int __init tegra_vpr_node_init(unsigned long offset, > + struct reserved_mem *rmem) > +{ > + struct cma *cma; > + int err; > + > + if (!IS_ALIGNED(rmem->base, SZ_1M)) { > + pr_err("%s: base is not aligned to 1 MiB\n", rmem->name); > + return -EINVAL; > + } > + > + if (!IS_ALIGNED(rmem->size, SZ_1M)) { > + pr_err("%s: size is not aligned to 1 MiB\n", rmem->name); > + return -EINVAL; > + } > + > + err = cma_init_reserved_mem(rmem->base, rmem->size, 0, rmem->name, > + &cma); > + if (err < 0) { > + pr_err("%s: failed to initialize CMA: %d\n", __func__, err); > + return err; > + } > + > + rmem->priv = cma; > + > + return 0; > +} > + > +static struct tegra_vpr *tegra_vpr_lookup(struct cma *cma) > +{ > + struct tegra_vpr *vpr; > + > + mutex_lock(&vpr_lock); > + > + list_for_each_entry(vpr, &vpr_list, list) { > + if (vpr->cma == cma) { > + mutex_unlock(&vpr_lock); > + return vpr; > + } > + } > + > + mutex_unlock(&vpr_lock); > + > + return ERR_PTR(-EPROBE_DEFER); > +} > + > +static int tegra_vpr_device_init(struct reserved_mem *rmem, struct device *dev) > +{ > + const struct dev_pm_ops *pm = dev->driver->pm; > + struct tegra_vpr_device *node; > + struct cma *cma = rmem->priv; > + struct tegra_vpr *vpr; > + > + vpr = tegra_vpr_lookup(cma); > + if (IS_ERR(vpr)) > + return PTR_ERR(vpr); > + > + if (!pm || !pm->freeze || !pm->thaw) > + return -EINVAL; > + > + node = kzalloc_obj(*node, GFP_KERNEL); > + if (!node) > + return -ENOMEM; > + > + INIT_LIST_HEAD(&node->node); > + node->dev = dev; > + > + mutex_lock(&vpr->lock); > + list_add_tail(&node->node, &vpr->devices); > + mutex_unlock(&vpr->lock); > + > + return 0; > +} > + > +static void tegra_vpr_device_release(struct reserved_mem *rmem, > + struct device *dev) > +{ > + struct tegra_vpr_device *node, *tmp; > + struct cma *cma = rmem->priv; > + struct tegra_vpr *vpr; > + > + vpr = tegra_vpr_lookup(cma); > + if (IS_ERR(vpr)) { > + dev_WARN(dev, "failed to find VPR for CMA '%s'\n", > + cma_get_name(cma)); > + return; > + } > + > + mutex_lock(&vpr->lock); > + > + list_for_each_entry_safe(node, tmp, &vpr->devices, node) { > + if (node->dev == dev) { > + list_del(&node->node); > + kfree(node); > + } > + } > + > + mutex_unlock(&vpr->lock); > +} > + > +static const struct reserved_mem_ops tegra_vpr_rmem_ops = { > + .node_init = tegra_vpr_node_init, > + .device_init = tegra_vpr_device_init, > + .device_release = tegra_vpr_device_release, > +}; > + > +RESERVEDMEM_OF_DECLARE(tegra_vpr, "nvidia,tegra-video-protection-region", > + &tegra_vpr_rmem_ops); > + > +void tegra_vpr_add(struct tegra_vpr *vpr) > +{ > + mutex_lock(&vpr_lock); > + list_add_tail(&vpr->list, &vpr_list); > + mutex_unlock(&vpr_lock); > +} > +EXPORT_SYMBOL(tegra_vpr_add); > diff --git a/drivers/dma-buf/heaps/tegra-vpr.c b/drivers/dma-buf/heaps/tegra-vpr.c > new file mode 100644 > index 000000000000..d8cff7e07ea0 > --- /dev/null > +++ b/drivers/dma-buf/heaps/tegra-vpr.c > @@ -0,0 +1,1210 @@ > +// SPDX-License-Identifier: GPL-2.0 > +/* > + * DMA-BUF restricted heap exporter for NVIDIA Video-Protection-Region (VPR) > + * > + * Copyright (C) 2024-2026 NVIDIA Corporation > + */ > + > +#define pr_fmt(fmt) "tegra-vpr: " fmt > + > +#include <linux/arm-smccc.h> > +#include <linux/cma.h> > +#include <linux/debugfs.h> > +#include <linux/dma-buf.h> > +#include <linux/dma-heap.h> > +#include <linux/find.h> > +#include <linux/memory.h> > +#include <linux/of_reserved_mem.h> > +#include <linux/platform_device.h> > +#include <linux/pm_runtime.h> > +#include <linux/reset.h> > +#include <linux/set_memory.h> > + > +#include "tegra-vpr.h" > + > +MODULE_IMPORT_NS("DMA_BUF_HEAP"); > +MODULE_IMPORT_NS("DMA_BUF"); > + > +#define CREATE_TRACE_POINTS > +#include <trace/events/tegra_vpr.h> > + > +#define TEGRA_VPR_MAX_CHUNKS 64 > + > +struct tegra_vpr_buffer { > + struct list_head attachments; > + struct tegra_vpr *vpr; > + struct list_head list; > + > + /** > + * @lock: Protects concurrent access to the list of attachments. > + */ > + struct mutex lock; > + > + struct page **pages; > + pgoff_t num_pages; > + phys_addr_t start; > + phys_addr_t limit; > + size_t size; > + int pageno; > + int order; > + > + DECLARE_BITMAP(chunks, TEGRA_VPR_MAX_CHUNKS); > +}; > + > +struct tegra_vpr_attachment { > + struct device *dev; > + struct sg_table sgt; > + struct list_head list; > +}; > + > +#define ARM_SMCCC_TE_FUNC_PROGRAM_VPR 0x3 > + > +#define ARM_SMCCC_VENDOR_SIP_TE_PROGRAM_VPR_FUNC_ID \ > + ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL, \ > + ARM_SMCCC_SMC_32, \ > + ARM_SMCCC_OWNER_SIP, \ > + ARM_SMCCC_TE_FUNC_PROGRAM_VPR) > + > +static int tegra_vpr_set(phys_addr_t base, phys_addr_t size) > +{ > + struct arm_smccc_res res; > + > + arm_smccc_smc(ARM_SMCCC_VENDOR_SIP_TE_PROGRAM_VPR_FUNC_ID, base, size, > + 0, 0, 0, 0, 0, &res); > + > + return res.a0; > +} > + > +static int tegra_vpr_get_extents(struct tegra_vpr *vpr, phys_addr_t *base, > + phys_addr_t *size) > +{ > + phys_addr_t start = ~0, limit = 0; > + unsigned int i; > + > + for (i = 0; i < vpr->num_chunks; i++) { > + struct tegra_vpr_chunk *chunk = &vpr->chunks[i]; > + > + if (chunk->active) { > + if (chunk->start < start) > + start = chunk->start; > + > + if (chunk->limit > limit) > + limit = chunk->limit; > + } > + } > + > + if (limit > start) { > + *size = limit - start; > + *base = start; > + } else { > + *base = *size = 0; > + } > + > + return 0; > +} > + > +static int tegra_vpr_resize(struct tegra_vpr *vpr) > +{ > + struct tegra_vpr_device *node; > + phys_addr_t base, size; > + int err, status = 0; > + > + err = tegra_vpr_get_extents(vpr, &base, &size); > + if (err < 0) { > + pr_err("%s(): failed to get VPR extents: %d\n", __func__, err); > + return err; > + } > + > + list_for_each_entry(node, &vpr->devices, node) { > + err = pm_generic_freeze(node->dev); > + if (err < 0) { > + pr_err("failed to freeze %s: %d\n", > + dev_name(node->dev), err); > + status = err; > + goto thaw; > + } > + } > + > + trace_tegra_vpr_set(base, size); > + > + err = tegra_vpr_set(base, size); > + if (err < 0) { > + pr_err("failed to secure VPR: %d\n", err); > + status = err; > + } > + > +thaw: > + list_for_each_entry_continue_reverse(node, &vpr->devices, node) { > + err = pm_generic_thaw(node->dev); > + if (err < 0) { > + pr_err("failed to thaw %s: %d\n", > + dev_name(node->dev), err); > + continue; > + } > + } > + > + return status; > +} > + > +static int tegra_vpr_chunk_init(struct tegra_vpr *vpr, > + struct tegra_vpr_chunk *chunk, > + phys_addr_t start, size_t size, > + unsigned int order, const char *name) > +{ > + chunk->start = start; > + chunk->limit = start + size; > + chunk->size = size; > + chunk->vpr = vpr; > + > + chunk->offset = (start - vpr->base) >> PAGE_SHIFT; > + chunk->num_pages = size >> PAGE_SHIFT; > + chunk->num_buffers = 0; > + > + /* CMA area is not reserved yet */ > + chunk->start_page = NULL; > + chunk->virt = 0; > + > + return 0; > +} > + > +static void tegra_vpr_chunk_free(struct tegra_vpr_chunk *chunk) > +{ > +} > + > +static inline bool tegra_vpr_chunk_is_last(const struct tegra_vpr_chunk *chunk) > +{ > + phys_addr_t limit = chunk->vpr->base + chunk->vpr->size; > + > + return chunk->limit == limit; > +} > + > +static inline bool tegra_vpr_chunk_is_leaf(const struct tegra_vpr_chunk *chunk) > +{ > + const struct tegra_vpr_chunk *next = chunk + 1; > + > + if (tegra_vpr_chunk_is_last(chunk)) > + return true; > + > + return !next->active; > +} > + > +static int tegra_vpr_chunk_alloc(struct tegra_vpr_chunk *chunk) > +{ > + chunk->start_page = cma_alloc_at(chunk->vpr->cma, chunk->offset, > + chunk->num_pages, false); > + if (!chunk->start_page) > + return -ENOMEM; > + > + chunk->virt = (unsigned long)page_to_virt(chunk->start_page); > + > + return 0; > +} > + > +static int tegra_vpr_chunk_activate(struct tegra_vpr_chunk *chunk) > +{ > + int err; > + > + trace_tegra_vpr_chunk_activate(chunk->start, chunk->limit); > + > + err = set_direct_map_invalid_noflush(chunk->start_page, > + chunk->num_pages); > + if (err) > + return err; > + > + flush_tlb_kernel_range(chunk->virt, chunk->virt + chunk->size); > + chunk->invalid = false; > + chunk->active = true; > + > + return 0; > +} > + > +static int tegra_vpr_chunk_deactivate(struct tegra_vpr_chunk *chunk) > +{ > + int err; > + > + if (!chunk->active) > + return 0; > + > + /* do not deactivate if there are buffers left in this chunk */ > + if (WARN_ON(chunk->num_buffers > 0)) > + return -EBUSY; > + > + trace_tegra_vpr_chunk_deactivate(chunk->start, chunk->limit); > + > + err = set_direct_map_default_noflush(chunk->start_page, > + chunk->num_pages); > + if (err) > + return err; > + > + flush_tlb_kernel_range(chunk->virt, chunk->virt + chunk->size); > + chunk->invalid = false; > + chunk->active = false; > + > + return 0; > +} > + > +static void tegra_vpr_chunk_release(struct tegra_vpr_chunk *chunk) > +{ > + if (!WARN_ON(chunk->active || chunk->invalid)) { > + cma_release(chunk->vpr->cma, chunk->start_page, > + chunk->num_pages); > + chunk->start_page = NULL; > + chunk->virt = 0; > + } > +} > + > +static bool tegra_vpr_chunk_overlaps(struct tegra_vpr_chunk *chunk, > + unsigned int start, unsigned int limit) > +{ > + unsigned int first = chunk->offset; > + unsigned int last = chunk->offset + chunk->num_pages - 1; > + > + if (last < start || first >= limit) > + return false; > + > + return true; > +} > + > +static int tegra_vpr_activate_chunks(struct tegra_vpr *vpr, > + struct tegra_vpr_buffer *buffer) > +{ > + DECLARE_BITMAP(dirty, vpr->num_chunks); > + unsigned int i, bottom, top; > + int err = 0, ret; > + > + bitmap_zero(dirty, vpr->num_chunks); > + > + /* activate any inactive chunks that overlap this buffer */ > + for_each_set_bit(i, buffer->chunks, vpr->num_chunks) { > + struct tegra_vpr_chunk *chunk = &vpr->chunks[i]; > + > + if (chunk->active) > + continue; > + > + err = tegra_vpr_chunk_alloc(chunk); > + if (err < 0) > + goto deactivate; > + > + err = tegra_vpr_chunk_activate(chunk); > + if (err < 0) { > + tegra_vpr_chunk_release(chunk); > + goto deactivate; > + } > + > + set_bit(i, vpr->active); > + set_bit(i, dirty); > + } > + > + /* > + * Activating chunks above may have created holes, but since the VPR > + * can only ever be a single contiguous region, make sure to activate > + * any missing chunks. > + */ > + for_each_clear_bitrange(bottom, top, vpr->active, vpr->num_chunks) { > + /* inactive chunks at the bottom or the top are harmless */ > + if (bottom == 0 || top == vpr->num_chunks) > + continue; > + > + for (i = bottom; i < top; i++) { > + struct tegra_vpr_chunk *chunk = &vpr->chunks[i]; > + > + err = tegra_vpr_chunk_alloc(chunk); > + if (err < 0) > + goto deactivate; > + > + err = tegra_vpr_chunk_activate(chunk); > + if (err < 0) { > + tegra_vpr_chunk_release(chunk); > + goto deactivate; > + } > + > + set_bit(i, vpr->active); > + set_bit(i, dirty); > + } > + } > + > + /* if any chunks have been activated, VPR needs to be resized */ > + if (!bitmap_empty(dirty, vpr->num_chunks)) { > + err = tegra_vpr_resize(vpr); > + if (err < 0) { > + pr_err("failed to grow VPR: %d\n", err); > + goto deactivate; > + } > + } > + > + /* increment buffer count for each chunk */ > + for_each_set_bit(i, buffer->chunks, vpr->num_chunks) > + vpr->chunks[i].num_buffers++; > + > + return 0; > + > +deactivate: > + /* deactivate any of the previously inactive chunks on failure */ > + for_each_set_bit(i, dirty, vpr->num_chunks) { > + struct tegra_vpr_chunk *chunk = &vpr->chunks[i]; > + > + ret = tegra_vpr_chunk_deactivate(chunk); > + if (WARN_ON(ret < 0)) { > + pr_err("failed to deactivate chunk #%u: %d\n", i, ret); > + } else { > + tegra_vpr_chunk_release(chunk); > + clear_bit(i, vpr->active); > + } > + } > + > + return err; > +} > + > +/* > + * Retrieve the range of pages within the activate region of the VPR. > + */ > +static bool tegra_vpr_get_active_range(struct tegra_vpr *vpr, > + unsigned int *first, > + unsigned int *last) > +{ > + unsigned long i, j; > + > + i = find_first_bit(vpr->active, vpr->num_chunks); > + if (i >= vpr->num_chunks) > + return false; > + > + j = find_last_bit(vpr->active, vpr->num_chunks); > + if (j >= vpr->num_chunks) > + return false; > + > + *first = vpr->chunks[i].offset; > + *last = vpr->chunks[j].offset + vpr->chunks[j].num_pages; > + > + return true; > +} > + > +/* > + * Try to find and allocate a free region within a specific page range. > + * Returns the page number if successful, -ENOSPC otherwise. > + * > + * This function mimics bitmap_find_free_region() but restricts the search > + * to a specific range to enable allocation within individual chunks. > + */ > +static int tegra_vpr_find_free_region_in_range(struct tegra_vpr *vpr, > + unsigned int start_page, > + unsigned int end_page, > + unsigned int num_pages, > + unsigned int align) > +{ > + unsigned int pos, next = ALIGN(start_page, align); > + > + /* Scan through aligned positions, trying to allocate at each one */ > + for (pos = next; pos + num_pages <= end_page; pos = next) { > + next = find_next_bit(vpr->bitmap, pos + num_pages, pos); > + > + if (next >= pos + num_pages) { > + bitmap_set(vpr->bitmap, pos, num_pages); > + return pos; > + } > + > + next = find_next_zero_bit(vpr->bitmap, vpr->num_pages, next); > + next = ALIGN(next, align); > + } > + > + return -ENOSPC; > +} > + > +static int tegra_vpr_find_free_region(struct tegra_vpr *vpr, > + unsigned int num_pages, > + unsigned long align) > +{ > + return tegra_vpr_find_free_region_in_range(vpr, 0, vpr->num_pages - 1, > + num_pages, align); > +} > + > +static int tegra_vpr_find_free_region_clustered(struct tegra_vpr *vpr, > + unsigned int num_pages, > + unsigned int align) > +{ > + unsigned int target, first, last; > + int pageno; > + > + /* > + * If there are no allocations, abort the clustered allocation scheme > + * and use the generic allocation scheme instead. > + */ > + if (vpr->first > vpr->last) > + return -ENOSPC; > + > + /* > + * First, try to allocate within the currently allocated region. This > + * keeps allocations tightly packed and minimizes the VPR size needed. > + */ > + pageno = tegra_vpr_find_free_region_in_range(vpr, vpr->first, > + vpr->last + 1, num_pages, > + align); > + if (pageno >= 0) > + return pageno; > + > + /* > + * If not enough free space exists within the currently allocated > + * region, check to see if the allocation fits anywhere within the > + * active region, avoiding the need to resize the VPR. > + */ > + if (tegra_vpr_get_active_range(vpr, &first, &last)) { > + pageno = tegra_vpr_find_free_region_in_range(vpr, first, last, > + num_pages, align); > + if (pageno >= 0) > + return pageno; > + } > + > + /* > + * If not enough free space exists within the currently active region, > + * try to allocate adjacent to it to grow it contiguously and ensure > + * optimal packing. > + */ > + > + /* > + * Calculate where the allocation should start to end right at the > + * first allocated page, with proper alignment. > + */ > + if (vpr->first >= num_pages) { > + target = ALIGN_DOWN(vpr->first - num_pages, align); > + > + if (!bitmap_allocate(vpr->bitmap, target, num_pages)) > + return target; > + } > + > + /* Try after the last allocation */ > + target = ALIGN(vpr->last + 1, align); > + > + if (target + num_pages <= vpr->num_pages && > + !bitmap_allocate(vpr->bitmap, target, num_pages)) > + return target; > + > + /* > + * Couldn't allocate at the ideal adjacent position, search for any > + * available space before the first allocated page. > + */ > + pageno = tegra_vpr_find_free_region_in_range(vpr, 0, vpr->first, > + num_pages, align); > + if (pageno >= 0) > + return pageno; > + > + /* > + * Couldn't allocate at the ideal adjacent position, search > + * for any available space after the last allocated page. > + */ > + pageno = tegra_vpr_find_free_region_in_range(vpr, vpr->last + 1, > + vpr->num_pages, num_pages, > + align); > + if (pageno >= 0) > + return pageno; > + > + return -ENOSPC; > +} > + > +/* > + * Find a free region, preferring locations near existing allocations to > + * minimize VPR fragmentation. The allocation strategy is to first allocate > + * within or adjacent to the existing region to keep allocations clustered. > + * Otherwise fall back to a generic allocation using the first available > + * space. > + * > + * This approach focuses on page-level allocation first, then the chunk > + * system determines which chunks need to be activated based on where the > + * pages ended up. > + */ > +static int tegra_vpr_allocate_region(struct tegra_vpr *vpr, > + unsigned int num_pages, > + unsigned int align) > +{ > + int pageno; > + > + /* > + * For non-resizable VPR (no chunks), use simple first-fit allocation. > + * Clustering optimization is only beneficial for resizable VPR where > + * keeping allocations together minimizes the active VPR size. > + */ > + if (!vpr->resizable) > + return tegra_vpr_find_free_region(vpr, num_pages, align); > + > + /* > + * Check if there are any existing allocations in the bitmap. If so, > + * try to allocate near them to minimize fragmentation. > + */ > + pageno = tegra_vpr_find_free_region_clustered(vpr, num_pages, align); > + if (pageno >= 0) > + return pageno; > + > + /* > + * If there are no existing allocations, or no space adjacent to them, > + * fall back to the first available space anywhere in the VPR. > + */ > + pageno = tegra_vpr_find_free_region(vpr, num_pages, align); > + if (pageno >= 0) > + return pageno; > + > + return -ENOSPC; > +} > + > +static struct tegra_vpr_buffer * > +tegra_vpr_buffer_allocate(struct tegra_vpr *vpr, size_t size) > +{ > + unsigned int num_pages = size >> PAGE_SHIFT; > + unsigned int order = get_order(size); > + struct tegra_vpr_buffer *buffer; > + unsigned long first, last; > + int pageno, err; > + > + /* > + * Quick sanity check that we're not trying to allocate a buffer that > + * has no chance of fitting into the VPR. > + */ > + if (size > vpr->size) > + return ERR_PTR(-EINVAL); > + > + /* > + * "order" defines the alignment and size, so this may result in > + * fragmented memory depending on the allocation patterns. However, > + * since this is used primarily for video frames, it is expected that > + * a number of buffers of the same size will be allocated, so > + * fragmentation should be negligible. > + */ > + pageno = tegra_vpr_allocate_region(vpr, num_pages, 1); > + if (pageno < 0) > + return ERR_PTR(pageno); > + > + first = find_first_bit(vpr->bitmap, vpr->num_pages); > + last = find_last_bit(vpr->bitmap, vpr->num_pages); > + > + buffer = kzalloc_obj(*buffer, GFP_KERNEL); > + if (!buffer) { > + err = -ENOMEM; > + goto release; > + } > + > + INIT_LIST_HEAD(&buffer->attachments); > + INIT_LIST_HEAD(&buffer->list); > + mutex_init(&buffer->lock); > + buffer->start = vpr->base + (pageno << PAGE_SHIFT); > + buffer->limit = buffer->start + size; > + buffer->size = size; > + buffer->num_pages = num_pages; > + buffer->pageno = pageno; > + buffer->order = order; > + > + /* track which chunks this buffer overlaps */ > + if (vpr->resizable) { > + unsigned int limit = buffer->pageno + buffer->num_pages; > + pgoff_t i; > + > + /* > + * Memory is backed by struct page, so track which ones we > + * use. > + */ > + buffer->pages = kvmalloc_array(buffer->num_pages, > + sizeof(*buffer->pages), > + GFP_KERNEL); > + if (!buffer->pages) { > + err = -ENOMEM; > + goto free; > + } > + > + for (i = 0; i < buffer->num_pages; i++) > + buffer->pages[i] = &vpr->start_page[pageno + i]; > + > + for (i = 0; i < vpr->num_chunks; i++) { > + struct tegra_vpr_chunk *chunk = &vpr->chunks[i]; > + > + if (tegra_vpr_chunk_overlaps(chunk, pageno, limit)) > + set_bit(i, buffer->chunks); > + } > + > + /* activate chunks if necessary */ > + err = tegra_vpr_activate_chunks(vpr, buffer); > + if (err < 0) { > + kfree(buffer->pages); > + goto free; > + } > + > + /* track first and last allocated pages */ > + if (buffer->pageno < vpr->first) > + vpr->first = buffer->pageno; > + > + if (limit - 1 > vpr->last) > + vpr->last = limit - 1; > + } > + > + return buffer; > + > +free: > + kfree(buffer); > +release: > + bitmap_clear(vpr->bitmap, pageno, num_pages); > + return ERR_PTR(err); > +} > + > +static void tegra_vpr_buffer_release(struct tegra_vpr_buffer *buffer) > +{ > + struct tegra_vpr *vpr = buffer->vpr; > + struct tegra_vpr_buffer *entry; > + unsigned int i; > + > + /* > + * Decrement buffer count for each overlapping chunk. Note that chunks > + * are not deactivated here yet, that's done in tegra_vpr_recycle() > + * instead. > + */ > + for_each_set_bit(i, buffer->chunks, vpr->num_chunks) { > + if (!WARN_ON(vpr->chunks[i].num_buffers == 0)) > + vpr->chunks[i].num_buffers--; > + } > + > + /* track first and last allocated pages */ > + if (list_is_first(&buffer->list, &vpr->buffers) && > + list_is_last(&buffer->list, &vpr->buffers)) { > + /* if there are no remaining buffers after this, reset */ > + vpr->first = ~0U; > + vpr->last = 0U; > + } else if (list_is_first(&buffer->list, &vpr->buffers)) { > + entry = list_next_entry(buffer, list); > + vpr->first = entry->pageno; > + } else if (list_is_last(&buffer->list, &vpr->buffers)) { > + entry = list_prev_entry(buffer, list); > + vpr->last = entry->pageno + entry->num_pages - 1; > + } > + > + bitmap_clear(vpr->bitmap, buffer->pageno, buffer->num_pages); > + list_del(&buffer->list); > + kfree(buffer->pages); > + kfree(buffer); > +} > + > +static int tegra_vpr_attach(struct dma_buf *buf, > + struct dma_buf_attachment *attachment) > +{ > + struct tegra_vpr_buffer *buffer = buf->priv; > + struct tegra_vpr_attachment *attach; > + int err; > + > + attach = kzalloc_obj(*attach, GFP_KERNEL); > + if (!attach) > + return -ENOMEM; > + > + /* > + * For resizable VPR, the memory is backed by struct page, so we can > + * use the convenient helper to create the SG table. > + */ > + if (buffer->pages) { > + err = sg_alloc_table_from_pages(&attach->sgt, buffer->pages, > + buffer->num_pages, 0, > + buffer->size, GFP_KERNEL); > + if (err < 0) > + goto free; > + } else { > + if (sg_alloc_table(&attach->sgt, 1, GFP_KERNEL)) { > + err = -ENOMEM; > + goto free; > + } > + > + sg_set_page(attach->sgt.sgl, NULL, buffer->size, 0); > + sg_dma_address(attach->sgt.sgl) = buffer->start; > + sg_dma_len(attach->sgt.sgl) = buffer->size; > + } > + > + attach->dev = attachment->dev; > + INIT_LIST_HEAD(&attach->list); > + attachment->priv = attach; > + > + mutex_lock(&buffer->lock); > + list_add(&attach->list, &buffer->attachments); > + mutex_unlock(&buffer->lock); > + > + return 0; > + > +free: > + kfree(attach); > + return err; > +} > + > +static void tegra_vpr_detach(struct dma_buf *buf, > + struct dma_buf_attachment *attachment) > +{ > + struct tegra_vpr_buffer *buffer = buf->priv; > + struct tegra_vpr_attachment *attach = attachment->priv; > + > + mutex_lock(&buffer->lock); > + list_del(&attach->list); > + mutex_unlock(&buffer->lock); > + > + sg_free_table(&attach->sgt); > + kfree(attach); > +} > + > +static struct sg_table * > +tegra_vpr_map_dma_buf(struct dma_buf_attachment *attachment, > + enum dma_data_direction direction) > +{ > + struct tegra_vpr_attachment *attach = attachment->priv; > + struct sg_table *sgt = &attach->sgt; > + int err; > + > + err = dma_map_sgtable(attachment->dev, sgt, direction, > + DMA_ATTR_SKIP_CPU_SYNC); > + if (err < 0) > + return ERR_PTR(err); > + > + return sgt; > +} > + > +static void tegra_vpr_unmap_dma_buf(struct dma_buf_attachment *attachment, > + struct sg_table *sgt, > + enum dma_data_direction direction) > +{ > + dma_unmap_sgtable(attachment->dev, sgt, direction, > + DMA_ATTR_SKIP_CPU_SYNC); > +} > + > +static void tegra_vpr_recycle(struct tegra_vpr *vpr) > +{ > + DECLARE_BITMAP(dirty, vpr->num_chunks); > + unsigned int i; > + int err; > + > + if (!vpr->resizable) > + return; > + > + bitmap_zero(dirty, vpr->num_chunks); > + > + /* > + * Deactivate any unused chunks from the bottom... > + */ > + for (i = 0; i < vpr->num_chunks; i++) { > + struct tegra_vpr_chunk *chunk = &vpr->chunks[i]; > + > + if (!chunk->active) > + continue; > + > + if (chunk->num_buffers > 0) > + break; > + > + err = tegra_vpr_chunk_deactivate(chunk); > + if (err < 0) { > + pr_err("failed to deactivate chunk #%u: %d\n", i, err); > + goto activate; > + } else { > + clear_bit(i, vpr->active); > + set_bit(i, dirty); > + } > + } > + > + /* > + * ... and the top. > + */ > + for (i = 0; i < vpr->num_chunks; i++) { > + unsigned int index = vpr->num_chunks - i - 1; > + struct tegra_vpr_chunk *chunk = &vpr->chunks[index]; > + > + if (!chunk->active) > + continue; > + > + if (chunk->num_buffers > 0) > + break; > + > + err = tegra_vpr_chunk_deactivate(chunk); > + if (err < 0) { > + pr_err("failed to deactivate chunk #%u: %d\n", index, > + err); > + goto activate; > + } else { > + clear_bit(index, vpr->active); > + set_bit(index, dirty); > + } > + } > + > + if (!bitmap_empty(dirty, vpr->num_chunks)) { > + err = tegra_vpr_resize(vpr); > + if (err < 0) { > + pr_err("failed to shrink VPR: %d\n", err); > + goto activate; > + } > + } > + > + /* release the CMA memory associated with deactivated chunks */ > + for_each_set_bit(i, dirty, vpr->num_chunks) > + tegra_vpr_chunk_release(&vpr->chunks[i]); > + > + return; > + > +activate: > + for_each_set_bit(i, dirty, vpr->num_chunks) { > + err = tegra_vpr_chunk_activate(&vpr->chunks[i]); > + if (WARN_ON(err < 0)) > + pr_err("failed to activate chunk #%u: %d\n", i, err); > + > + /* > + * This may not be fully activated at this point, but we need > + * to keep track of it anyway to make sure the CMA region can > + * eventually be released. The WARN_ON above tells us when it > + * happens: here be dragons. > + */ > + set_bit(i, vpr->active); > + } > +} > + > +static void tegra_vpr_release(struct dma_buf *buf) > +{ > + struct tegra_vpr_buffer *buffer = buf->priv; > + struct tegra_vpr *vpr = buffer->vpr; > + > + mutex_lock(&vpr->lock); > + > + tegra_vpr_buffer_release(buffer); > + tegra_vpr_recycle(vpr); > + > + mutex_unlock(&vpr->lock); > +} > + > +/* > + * Prohibit userspace mapping because the CPU cannot access this memory > + * anyway. > + */ > +static int tegra_vpr_begin_cpu_access(struct dma_buf *buf, > + enum dma_data_direction direction) > +{ > + return -EPERM; > +} > + > +static int tegra_vpr_end_cpu_access(struct dma_buf *buf, > + enum dma_data_direction direction) > +{ > + return -EPERM; > +} > + > +static int tegra_vpr_mmap(struct dma_buf *buf, struct vm_area_struct *vma) > +{ > + return -EPERM; > +} > + > +static const struct dma_buf_ops tegra_vpr_buf_ops = { > + .attach = tegra_vpr_attach, > + .detach = tegra_vpr_detach, > + .map_dma_buf = tegra_vpr_map_dma_buf, > + .unmap_dma_buf = tegra_vpr_unmap_dma_buf, > + .release = tegra_vpr_release, > + .begin_cpu_access = tegra_vpr_begin_cpu_access, > + .end_cpu_access = tegra_vpr_end_cpu_access, > + .mmap = tegra_vpr_mmap, > +}; > + > +static struct dma_buf *tegra_vpr_allocate(struct dma_heap *heap, > + unsigned long len, u32 fd_flags, > + u64 heap_flags) > +{ > + struct tegra_vpr *vpr = dma_heap_get_drvdata(heap); > + struct tegra_vpr_buffer *buffer, *entry; > + size_t size = ALIGN(len, vpr->align); > + DEFINE_DMA_BUF_EXPORT_INFO(export); > + struct dma_buf *buf; > + > + mutex_lock(&vpr->lock); > + > + buffer = tegra_vpr_buffer_allocate(vpr, size); > + if (IS_ERR(buffer)) { > + mutex_unlock(&vpr->lock); > + return ERR_CAST(buffer); > + } > + > + /* insert in the correct order */ > + if (!list_empty(&vpr->buffers)) { > + list_for_each_entry(entry, &vpr->buffers, list) { > + if (buffer->pageno < entry->pageno) { > + list_add_tail(&buffer->list, &entry->list); > + break; > + } > + } > + } > + > + if (list_empty(&buffer->list)) > + list_add_tail(&buffer->list, &vpr->buffers); > + > + buffer->vpr = vpr; > + > + /* > + * If a valid buffer was allocated, wrap it in a dma_buf > + * and return it. > + */ > + export.exp_name = dma_heap_get_name(heap); > + export.ops = &tegra_vpr_buf_ops; > + export.size = buffer->size; > + export.flags = fd_flags; > + export.priv = buffer; > + > + buf = dma_buf_export(&export); > + if (IS_ERR(buf)) { > + tegra_vpr_buffer_release(buffer); > + tegra_vpr_recycle(vpr); > + } > + > + mutex_unlock(&vpr->lock); > + return buf; > +} > + > +static void tegra_vpr_debugfs_show_buffers(struct tegra_vpr *vpr, > + struct seq_file *s) > +{ > + struct tegra_vpr_buffer *buffer; > + char buf[16]; > + > + mutex_lock(&vpr->lock); > + > + list_for_each_entry(buffer, &vpr->buffers, list) { > + string_get_size(buffer->size, 1, STRING_UNITS_2, buf, > + sizeof(buf)); > + seq_printf(s, " %pap-%pap (%s)\n", &buffer->start, > + &buffer->limit, buf); > + } > + > + mutex_unlock(&vpr->lock); > +} > + > +static void tegra_vpr_debugfs_show_chunks(struct tegra_vpr *vpr, > + struct seq_file *s) > +{ > + struct tegra_vpr_buffer *buffer; > + unsigned int i; > + char buf[16]; > + > + for (i = 0; i < vpr->num_chunks; i++) { > + const struct tegra_vpr_chunk *chunk = &vpr->chunks[i]; > + > + string_get_size(chunk->size, 1, STRING_UNITS_2, buf, > + sizeof(buf)); > + seq_printf(s, " %pap-%pap (%s) (%s, %u buffers)\n", > + &chunk->start, &chunk->limit, buf, > + chunk->active ? "active" : "inactive", > + chunk->num_buffers); > + } > + > + list_for_each_entry(buffer, &vpr->buffers, list) { > + string_get_size(buffer->size, 1, STRING_UNITS_2, buf, > + sizeof(buf)); > + seq_printf(s, "%pap-%pap (%s, chunks: %*pbl)\n", > + &buffer->start, &buffer->limit, buf, > + vpr->num_chunks, buffer->chunks); > + } > +} > + > +static int tegra_vpr_debugfs_show(struct seq_file *s, struct dma_heap *heap) > +{ > + struct tegra_vpr *vpr = dma_heap_get_drvdata(heap); > + phys_addr_t limit = vpr->base + vpr->size; > + char buf[16]; > + > + string_get_size(vpr->size, 1, STRING_UNITS_2, buf, sizeof(buf)); > + seq_printf(s, "%pap-%pap (%s)\n", &vpr->base, &limit, buf); > + > + if (!vpr->resizable) > + tegra_vpr_debugfs_show_buffers(vpr, s); > + else > + tegra_vpr_debugfs_show_chunks(vpr, s); > + > + return 0; > +} > + > +static const struct dma_heap_ops tegra_vpr_heap_ops = { > + .allocate = tegra_vpr_allocate, > + .show = tegra_vpr_debugfs_show, > +}; > + > +static int tegra_vpr_setup_chunks(struct tegra_vpr *vpr, const char *name) > +{ > + phys_addr_t start, limit; > + unsigned int order, i = 0; > + size_t max_size; > + int err; > + > + /* Memory is backed by struct page, so track the first one. */ > + vpr->start_page = phys_to_page(vpr->base); > + > + /* This seems a reasonable value, so hard-code it for now. */ > + vpr->num_chunks = 4; > + > + vpr->chunks = kzalloc_objs(*vpr->chunks, vpr->num_chunks); > + if (!vpr->chunks) > + return -ENOMEM; > + > + vpr->active = bitmap_zalloc(vpr->num_chunks, GFP_KERNEL); > + if (!vpr->active) { > + err = -ENOMEM; > + goto free; > + } > + > + max_size = PAGE_SIZE << (get_order(vpr->size) - ilog2(vpr->num_chunks)); > + order = get_order(vpr->align); > + > + /* > + * Allocate CMA areas for VPR. All areas will be roughtly the same > + * size, with the last area taking up the rest. > + */ > + start = vpr->base; > + limit = vpr->base + vpr->size; > + > + pr_debug("VPR: %pap-%pap (%lu pages, %u chunks, %lu MiB)\n", &start, > + &limit, vpr->num_pages, vpr->num_chunks, > + (unsigned long)vpr->size / 1024 / 1024); > + > + for (i = 0; i < vpr->num_chunks; i++) { > + size_t size = limit - start; > + phys_addr_t end; > + > + size = min_t(size_t, size, max_size); > + end = start + size - 1; > + > + err = tegra_vpr_chunk_init(vpr, &vpr->chunks[i], start, size, > + order, name); > + if (err < 0) { > + pr_err("failed to create VPR chunk: %d\n", err); > + goto free; > + } > + > + pr_debug(" %2u: %pap-%pap (%lu MiB)\n", i, &start, &end, > + size / 1024 / 1024); > + start += size; > + } > + > + vpr->first = ~0U; > + vpr->last = 0U; > + > + return 0; > + > +free: > + while (i--) > + tegra_vpr_chunk_free(&vpr->chunks[i]); > + > + kfree(vpr->active); > + kfree(vpr->chunks); > + return err; > +} > + > +static void tegra_vpr_free_chunks(struct tegra_vpr *vpr) > +{ > + unsigned int i; > + > + for (i = 0; i < vpr->num_chunks; i++) > + tegra_vpr_chunk_free(&vpr->chunks[i]); > + > + kfree(vpr->chunks); > +} > + > +static int tegra_vpr_setup_static(struct tegra_vpr *vpr) > +{ > + phys_addr_t start, limit; > + > + start = vpr->base; > + limit = vpr->base + vpr->size; > + > + pr_debug("VPR: %pap-%pap (%lu pages, %lu MiB)\n", &start, &limit, > + vpr->num_pages, (unsigned long)vpr->size / 1024 / 1024); > + > + return 0; > +} > + > +static int tegra_vpr_add_heap(struct reserved_mem *rmem, > + struct device_node *np) > +{ > + struct dma_heap_export_info info = {}; > + unsigned long first, last; > + struct dma_heap *heap; > + struct tegra_vpr *vpr; > + int err; > + > + vpr = kzalloc_obj(*vpr, GFP_KERNEL); > + if (!vpr) > + return -ENOMEM; > + > + INIT_LIST_HEAD(&vpr->list); > + INIT_LIST_HEAD(&vpr->buffers); > + INIT_LIST_HEAD(&vpr->devices); > + mutex_init(&vpr->lock); > + > + vpr->resizable = !of_property_read_bool(np, "no-map"); > + vpr->dev_node = of_node_get(np); > + vpr->align = PAGE_SIZE; > + vpr->base = rmem->base; > + vpr->size = rmem->size; > + vpr->num_pages = vpr->size >> PAGE_SHIFT; > + vpr->nid = of_node_to_nid(np); If this gets used, I can't find it. Rob

1 day, 20 hours

Re: [PATCH v6 07/12] mm/cma: Introduce cma_alloc_at() API

by Marek Szyprowski

On 04.09.2026 12:44, Thierry Reding wrote: > From: Thierry Reding <treding(a)nvidia.com> > > This API can be used to allocate a number of CMA pages starting at a > fixed offset. This is useful, for example, if the CMA area is used as > backing storage for a nested allocator that has stricter requirements > than CMA itself. > > In order to reuse most of the existing code, the cma_range_alloc() > function is enhanced to take as inputs a range of pages to look for a > free region. If this range encompasses the entire CMA range, the > behavior is as before, but the range can also be specified to match > the offset and size passed in from cma_alloc_at(), in which case that > is the range that will be allocated (or allocation will fail if the > range is not available). > > Suggested-by: Marek Szyprowski <m.szyprowski(a)samsung.com> > Signed-off-by: Thierry Reding <treding(a)nvidia.com> Acked-by: Marek Szyprowski <m.szyprowski(a)samsung.com> > --- > Changes in v6: > - refactor cma_range_alloc() so it can be reused for _at() code paths > --- > include/linux/cma.h | 4 +++ > include/trace/events/cma.h | 77 ++++++++++++++++++++++++++++++++++++++-- > mm/cma.c | 88 +++++++++++++++++++++++++++++++++++----------- > 3 files changed, 146 insertions(+), 23 deletions(-) > > diff --git a/include/linux/cma.h b/include/linux/cma.h > index 8555d38a97b1..844404459a42 100644 > --- a/include/linux/cma.h > +++ b/include/linux/cma.h > @@ -49,11 +49,15 @@ extern int cma_init_reserved_mem(phys_addr_t base, phys_addr_t size, > struct cma **res_cma); > extern struct page *cma_alloc(struct cma *cma, unsigned long count, unsigned int align, > bool no_warn); > +extern struct page *cma_alloc_at(struct cma *cma, unsigned long offset, > + unsigned long count, bool no_warn); > extern bool cma_release(struct cma *cma, const struct page *pages, unsigned long count); > > struct page *cma_alloc_frozen(struct cma *cma, unsigned long count, > unsigned int align, bool no_warn); > struct page *cma_alloc_frozen_compound(struct cma *cma, unsigned int order); > +struct page *cma_alloc_at_frozen(struct cma *cma, unsigned long offset, > + unsigned long count, bool no_warn); > bool cma_release_frozen(struct cma *cma, const struct page *pages, > unsigned long count); > > diff --git a/include/trace/events/cma.h b/include/trace/events/cma.h > index 37195edf2498..4e0ab63e303f 100644 > --- a/include/trace/events/cma.h > +++ b/include/trace/events/cma.h > @@ -38,13 +38,17 @@ TRACE_EVENT(cma_release, > > TRACE_EVENT(cma_alloc_start, > > - TP_PROTO(const char *name, unsigned long request_count, unsigned long available_count, > + TP_PROTO(const char *name, unsigned long start, unsigned long end, > + unsigned long request_count, unsigned long available_count, > unsigned long total_count, unsigned int align), > > - TP_ARGS(name, request_count, available_count, total_count, align), > + TP_ARGS(name, start, end, request_count, available_count, total_count, > + align), > > TP_STRUCT__entry( > __string(name, name) > + __field(unsigned long, start) > + __field(unsigned long, end) > __field(unsigned long, request_count) > __field(unsigned long, available_count) > __field(unsigned long, total_count) > @@ -53,14 +57,18 @@ TRACE_EVENT(cma_alloc_start, > > TP_fast_assign( > __assign_str(name); > + __entry->start = start; > + __entry->end = end; > __entry->request_count = request_count; > __entry->available_count = available_count; > __entry->total_count = total_count; > __entry->align = align; > ), > > - TP_printk("name=%s request_count=%lu available_count=%lu total_count=%lu align=%u", > + TP_printk("name=%s start=%lu end=%lu request_count=%lu available_count=%lu total_count=%lu align=%u", > __get_str(name), > + __entry->start, > + __entry->end, > __entry->request_count, > __entry->available_count, > __entry->total_count, > @@ -132,6 +140,69 @@ TRACE_EVENT(cma_alloc_busy_retry, > __entry->align) > ); > > +TRACE_EVENT(cma_alloc_at_start, > + > + TP_PROTO(const char *name, unsigned long pfn, > + unsigned long request_count, unsigned long available_count, > + unsigned long total_count), > + > + TP_ARGS(name, pfn, request_count, available_count, total_count), > + > + TP_STRUCT__entry( > + __string(name, name) > + __field(unsigned long, pfn) > + __field(unsigned long, request_count) > + __field(unsigned long, available_count) > + __field(unsigned long, total_count) > + ), > + > + TP_fast_assign( > + __assign_str(name); > + __entry->pfn = pfn; > + __entry->request_count = request_count; > + __entry->available_count = available_count; > + __entry->total_count = total_count; > + ), > + > + TP_printk("name=%s pfn=%lx, request_count=%lu available_count=%lu total_count=%lu", > + __get_str(name), > + __entry->pfn, > + __entry->request_count, > + __entry->available_count, > + __entry->total_count) > +); > + > +TRACE_EVENT(cma_alloc_at_finish, > + > + TP_PROTO(const char *name, unsigned long pfn, const struct page *page, > + unsigned long count, int errorno), > + > + TP_ARGS(name, pfn, page, count, errorno), > + > + TP_STRUCT__entry( > + __string(name, name) > + __field(unsigned long, pfn) > + __field(const struct page *, page) > + __field(unsigned long, count) > + __field(int, errorno) > + ), > + > + TP_fast_assign( > + __assign_str(name); > + __entry->pfn = pfn; > + __entry->page = page; > + __entry->count = count; > + __entry->errorno = errorno; > + ), > + > + TP_printk("name=%s pfn=0x%lx page=%p count=%lu errorno=%d", > + __get_str(name), > + __entry->pfn, > + __entry->page, > + __entry->count, > + __entry->errorno) > +); > + > #endif /* _TRACE_CMA_H */ > > /* This part must be outside protection */ > diff --git a/mm/cma.c b/mm/cma.c > index a10ea37a261d..f92d3a188209 100644 > --- a/mm/cma.c > +++ b/mm/cma.c > @@ -783,25 +783,32 @@ static void cma_debug_show_areas(struct cma *cma) > spin_unlock_irq(&cma->lock); > } > > +/* > + * Searches the CMA memrange, from @start to @end, for a free region of > + * @count bits. If @end is less or equal to @start, will search the entire > + * memrange. > + */ > static int cma_range_alloc(struct cma *cma, struct cma_memrange *cmr, > - unsigned long count, unsigned int align, > - struct page **pagep, gfp_t gfp) > + unsigned long start, unsigned long end, > + unsigned long count, unsigned int align, > + struct page **pagep, gfp_t gfp) > { > unsigned long bitmap_maxno, bitmap_no, bitmap_count; > - unsigned long start, pfn, mask, offset; > - int ret = -EBUSY; > + unsigned long pfn, mask, offset; > struct page *page = NULL; > + int ret = -EBUSY; > > mask = cma_bitmap_aligned_mask(cma, align); > offset = cma_bitmap_aligned_offset(cma, cmr, align); > - bitmap_maxno = cma_bitmap_maxno(cma, cmr); > + bitmap_maxno = (end > start) ? end : cma_bitmap_maxno(cma, cmr); > bitmap_count = cma_bitmap_pages_to_bits(cma, count); > > if (bitmap_count > bitmap_maxno) > goto out; > > - for (start = 0; ; start = bitmap_no + mask + 1) { > + while (true) { > spin_lock_irq(&cma->lock); > + > /* > * If the request is larger than the available number > * of pages, stop right away. > @@ -810,6 +817,7 @@ static int cma_range_alloc(struct cma *cma, struct cma_memrange *cmr, > spin_unlock_irq(&cma->lock); > break; > } > + > bitmap_no = bitmap_find_next_zero_area_off(cmr->bitmap, > bitmap_maxno, start, bitmap_count, mask, > offset); > @@ -835,10 +843,11 @@ static int cma_range_alloc(struct cma *cma, struct cma_memrange *cmr, > > bitmap_set(cmr->bitmap, bitmap_no, bitmap_count); > cma->available_count -= count; > + > /* > - * It's safe to drop the lock here. We've marked this region for > - * our exclusive use. If the migration fails we will take the > - * lock again and unmark it. > + * It's safe to drop the lock here. We've marked this region > + * for our exclusive use. If the migration fails we will take > + * the lock again and unmark it. > */ > spin_unlock_irq(&cma->lock); > > @@ -856,37 +865,45 @@ static int cma_range_alloc(struct cma *cma, struct cma_memrange *cmr, > __func__, pfn, page); > > trace_cma_alloc_busy_retry(cma->name, pfn, page, count, align); > + start = bitmap_no + mask + 1; > } > + > out: > if (!ret) > *pagep = page; > + > return ret; > } > > -static struct page *__cma_alloc_frozen(struct cma *cma, > - unsigned long count, unsigned int align, gfp_t gfp) > +static struct page *__cma_alloc_frozen(struct cma *cma, unsigned long start, > + unsigned long end, unsigned long count, > + unsigned int align, gfp_t gfp) > { > struct page *page = NULL; > int ret = -ENOMEM, r; > unsigned long i; > - const char *name = cma ? cma->name : NULL; > + > + /* cma_alloc_at() and friends will only work with single-range CMA */ > + if (WARN_ON_ONCE(start > 0 && cma->nranges != 1)) > + return page; > > if (!cma || !cma->count) > return page; > > - pr_debug("%s(cma %p, name: %s, count %lu, align %d)\n", __func__, > - (void *)cma, cma->name, count, align); > + pr_debug("%s(cma %p, name: %s, start %lu, end %lu, count %lu, align %d)\n", > + __func__, (void *)cma, cma->name, start, end, count, align); > > if (!count) > return page; > > - trace_cma_alloc_start(name, count, cma->available_count, cma->count, align); > + trace_cma_alloc_start(cma->name, start, end, count, > + cma->available_count, cma->count, align); > > for (r = 0; r < cma->nranges; r++) { > page = NULL; > > - ret = cma_range_alloc(cma, &cma->ranges[r], count, align, > - &page, gfp); > + ret = cma_range_alloc(cma, &cma->ranges[r], start, end, > + count, align, &page, gfp); > if (ret != -EBUSY || page) > break; > } > @@ -908,7 +925,7 @@ static struct page *__cma_alloc_frozen(struct cma *cma, > } > > pr_debug("%s(): returned %p\n", __func__, page); > - trace_cma_alloc_finish(name, page ? page_to_pfn(page) : 0, > + trace_cma_alloc_finish(cma->name, page ? page_to_pfn(page) : 0, > page, count, align, ret); > if (page) { > count_vm_event(CMA_ALLOC_SUCCESS); > @@ -926,14 +943,22 @@ struct page *cma_alloc_frozen(struct cma *cma, unsigned long count, > { > gfp_t gfp = GFP_KERNEL | (no_warn ? __GFP_NOWARN : 0); > > - return __cma_alloc_frozen(cma, count, align, gfp); > + return __cma_alloc_frozen(cma, 0, 0, count, align, gfp); > } > > struct page *cma_alloc_frozen_compound(struct cma *cma, unsigned int order) > { > gfp_t gfp = GFP_KERNEL | __GFP_COMP | __GFP_NOWARN; > > - return __cma_alloc_frozen(cma, 1 << order, order, gfp); > + return __cma_alloc_frozen(cma, 0, 0, 1 << order, order, gfp); > +} > + > +struct page *cma_alloc_at_frozen(struct cma *cma, unsigned long offset, > + unsigned long count, bool no_warn) > +{ > + gfp_t gfp = GFP_KERNEL | (no_warn ? __GFP_NOWARN : 0); > + > + return __cma_alloc_frozen(cma, offset, offset + count, count, 0, gfp); > } > > /** > @@ -959,6 +984,29 @@ struct page *cma_alloc(struct cma *cma, unsigned long count, > } > EXPORT_SYMBOL_GPL(cma_alloc); > > +/** > + * cma_alloc_at() - allocate pages from contiguous area at fixed offset > + * @cma: Contiguous memory region for which the allocation is performed. > + * @offset: Index of the first page to allocate. > + * @count: Requested number of pages (in PAGE_SIZE order). > + * @no_warn: Avoid printing message about failed allocation. > + * > + * This function allocates a part of the contiguous memory on a specific > + * contiguous memory area. > + */ > +struct page *cma_alloc_at(struct cma *cma, unsigned long offset, > + unsigned long count, bool no_warn) > +{ > + struct page *page; > + > + page = cma_alloc_at_frozen(cma, offset, count, no_warn); > + if (page) > + set_pages_refcounted(page, count); > + > + return page; > +} > +EXPORT_SYMBOL_GPL(cma_alloc_at); > + > static struct cma_memrange *find_cma_memrange(struct cma *cma, > const struct page *pages, unsigned long count) > { > Best regards -- Marek Szyprowski, PhD Samsung R&D Institute Poland

2 days, 2 hours

[PATCH v2] dma-buf: Document how exporters and importers agree on mapping lifetime

by Leon Romanovsky

From: Leon Romanovsky <leonro(a)nvidia.com> Nothing in DMA-buf declares whether a mapping may be moved or taken away. Nearly every callback on both sides is optional, so the answer follows from which ones are implemented and from whether dma_buf_pin() succeeds. Documentation/ says none of this, and the rules sit in the kdoc of dma_buf_ops.pin, dma_buf_attach_ops.invalidate_mappings and dma_buf_invalidate_mappings(), which a driver author has to know by name before finding them. Describe the three flows an importer has to handle, the callbacks each one asks of both sides, and dma_buf_pin() as the runtime negotiation. Signed-off-by: Leon Romanovsky <leonro(a)nvidia.com> --- Let's try this variant from importer POV. Thanks Changes in v2: - Split pci p2p documentation patch to separate series - Improved dma-buf documentation patch. - Link to v1: https://patch.msgid.link/20260825-document-dma-buf-v1-0-5ecfb3e1371c@nvidia… --- Documentation/driver-api/dma-buf.rst | 6 ++++ drivers/dma-buf/dma-buf.c | 69 +++++++++++++++++++++++++++++++++++- 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/Documentation/driver-api/dma-buf.rst b/Documentation/driver-api/dma-buf.rst index 2f36c21d9948..39c201f38aa6 100644 --- a/Documentation/driver-api/dma-buf.rst +++ b/Documentation/driver-api/dma-buf.rst @@ -113,6 +113,12 @@ Basic Operation and Device DMA Access .. kernel-doc:: drivers/dma-buf/dma-buf.c :doc: dma buf device access +Mapping Lifetime Negotiation +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. kernel-doc:: drivers/dma-buf/dma-buf.c + :doc: mapping lifetime negotiation + CPU Access to DMA Buffer Objects ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/drivers/dma-buf/dma-buf.c b/drivers/dma-buf/dma-buf.c index d504c636dc29..aa47142cd9ed 100644 --- a/drivers/dma-buf/dma-buf.c +++ b/drivers/dma-buf/dma-buf.c @@ -684,7 +684,74 @@ static struct file *dma_buf_getfile(size_t size, int flags) * reference acquired with dma_buf_get() by calling dma_buf_put(). * * For the detailed semantics exporters are expected to implement see - * &dma_buf_ops. + * &dma_buf_ops. Whether the exporter may still move or take away the backing + * storage after step 3 depends on what both sides implement. See the mapping + * lifetime negotiation section below. + */ + +/** + * DOC: mapping lifetime negotiation + * + * Nearly everything in DMA-buf is optional. No flag or enum says whether the + * exporter may move or take away the backing storage while an importer holds + * a mapping. Each side simply implements the callbacks it can offer, and + * dma_buf_pin() settles the result at runtime. + * + * Out of that optionality the importer sees three main flows. They are named + * from the importer's point of view, because each one demands a different + * capability of its hardware: + * + * - Pinned: the memory is never taken away. The importer offers no way + * to stop DMA. + * - Revoked: the storage never moves, but the exporter may take it away. The + * importer must be able to stop DMA, and may hit user visible errors while + * doing so. + * - Movable: the exporter may relocate the storage at any time. The importer + * must be able to pause DMA, and must raise no error while the storage is + * moving. + * + * An importer reaches its flow like this: + * + * 1. Attach with dma_buf_dynamic_attach(). Leaving out the optional + * &dma_buf_attach_ops.invalidate_mappings callback pins the buffer for as + * long as the attachment exists. + * 2. Call dma_buf_pin() under the reservation lock. + * 3. On failure, run the movable flow or give up. + * 4. On success, the flow is the revoked one if the optional + * &dma_buf_attach_ops.invalidate_mappings is implemented, and the pinned + * one if it is not. + * + * Pinned flow: + * + * - Exporter: implement &dma_buf_ops.pin and &dma_buf_ops.unpin to hold the + * storage still on request. An exporter whose storage never moves implements + * neither, and dma_buf_pin() then succeeds on its own. An exporter which + * refuses to be pinned implements &dma_buf_ops.pin and fails it. + * - Importer: nothing more. The mapping stays valid until the importer unmaps. + * + * Revoked flow: + * + * - Exporter: answer dma_buf_pin() as above. Call + * dma_buf_invalidate_mappings() when the storage goes away and fail + * &dma_buf_ops.map_dma_buf while it is gone. The two waits which complete a + * revocation are described in dma_buf_invalidate_mappings(). + * - Importer: &dma_buf_attach_ops.invalidate_mappings has to unmap within + * bounded time and drop the pin. + * + * Movable flow: + * + * - Exporter: call dma_buf_invalidate_mappings() before each move, then wait + * for the &dma_buf.resv fences. &dma_buf_ops.pin and &dma_buf_ops.unpin play + * no part here. + * - Importer: hold no pin. &dma_buf_attach_ops.invalidate_mappings drops the + * cached mapping and has to lead to dma_buf_unmap_attachment() within + * bounded time. It need not stop the hardware, because access runs until the + * importer's &dma_buf.resv fences retire. The importer maps again before the + * next DMA. + * + * &dma_buf_ops.attach is the best place for an exporter to turn an importer + * away, because the importer can still fall back to another flow and attach + * again. */ /** --- base-commit: 63367df6e7255067ad6a83abe0d2799dfa491876 change-id: 20260820-document-dma-buf-3f8b41e32f57 Best regards, -- Leon Romanovsky <leonro(a)nvidia.com>

3 days, 22 hours

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

Linaro-mm-sig