Hi,
Here's another attempt at supporting user-space allocations from a specific carved-out reserved memory region.
The initial problem we were discussing was that I'm currently working on a platform which has a memory layout with ECC enabled. However, enabling the ECC has a number of drawbacks on that platform: lower performance, increased memory usage, etc. So for things like framebuffers, the trade-off isn't great and thus there's a memory region with ECC disabled to allocate from for such use cases.
After a suggestion from John, I chose to first start using heap allocations flags to allow for userspace to ask for a particular ECC setup. This is then backed by a new heap type that runs from reserved memory chunks flagged as such, and the existing DT properties to specify the ECC properties.
After further discussion, it was considered that flags were not the right solution, and relying on the names of the heaps would be enough to let userspace know the kind of buffer it deals with.
Thus, even though the uAPI part of it had been dropped in this second version, we still needed a driver to create heaps out of carved-out memory regions. In addition to the original usecase, a similar driver can be found in BSPs from most vendors, so I believe it would be a useful addition to the kernel.
Some extra discussion with Rob Herring [1] came to the conclusion that some specific compatible for this is not great either, and as such an new driver probably isn't called for either.
Some other discussions we had with John [2] also dropped some hints that multiple CMA heaps might be a good idea, and some vendors seem to do that too.
So here's another attempt that doesn't affect the device tree at all and will just create a heap for every CMA reserved memory region.
It also falls nicely into the current plan we have to support cgroups in DRM/KMS and v4l2, which is an additional benefit.
Let me know what you think, Maxime
1: https://lore.kernel.org/all/20250707-cobalt-dingo-of-serenity-dbf92c@houat/ 2: https://lore.kernel.org/all/CANDhNCroe6ZBtN_o=c71kzFFaWK-fF5rCdnr9P5h1sgPOWS...
Let me know what you think, Maxime
Signed-off-by: Maxime Ripard mripard@kernel.org --- Changes in v6: - Drop the new driver and allocate a CMA heap for each region now - Dropped the binding - Rebased on 6.16-rc5 - Link to v5: https://lore.kernel.org/r/20250617-dma-buf-ecc-heap-v5-0-0abdc5863a4f@kernel...
Changes in v5: - Rebased on 6.16-rc2 - Switch from property to dedicated binding - Link to v4: https://lore.kernel.org/r/20250520-dma-buf-ecc-heap-v4-1-bd2e1f1bb42c@kernel...
Changes in v4: - Rebased on 6.15-rc7 - Map buffers only when map is actually called, not at allocation time - Deal with restricted-dma-pool and shared-dma-pool - Reword Kconfig options - Properly report dma_map_sgtable failures - Link to v3: https://lore.kernel.org/r/20250407-dma-buf-ecc-heap-v3-0-97cdd36a5f29@kernel...
Changes in v3: - Reworked global variable patch - Link to v2: https://lore.kernel.org/r/20250401-dma-buf-ecc-heap-v2-0-043fd006a1af@kernel...
Changes in v2: - Add vmap/vunmap operations - Drop ECC flags uapi - Rebase on top of 6.14 - Link to v1: https://lore.kernel.org/r/20240515-dma-buf-ecc-heap-v1-0-54cbbd049511@kernel...
--- Maxime Ripard (2): dma/contiguous: Add helper to test reserved memory type dma-buf: heaps: cma: Create CMA heap for each CMA reserved region
drivers/dma-buf/heaps/cma_heap.c | 52 +++++++++++++++++++++++++++++++++++++++- include/linux/dma-map-ops.h | 13 ++++++++++ kernel/dma/contiguous.c | 7 ++++++ 3 files changed, 71 insertions(+), 1 deletion(-) --- base-commit: 47633099a672fc7bfe604ef454e4f116e2c954b1 change-id: 20240515-dma-buf-ecc-heap-28a311d2c94e prerequisite-message-id: 20250610131231.1724627-1-jkangas@redhat.com prerequisite-patch-id: bc44be5968feb187f2bc1b8074af7209462b18e7 prerequisite-patch-id: f02a91b723e5ec01fbfedf3c3905218b43d432da prerequisite-patch-id: e944d0a3e22f2cdf4d3b3906e5603af934696deb
Best regards,
A given reserved-memory region can be of multiple types.
We have currently four types defined in the tree: contiguous, backed by CMA, coherent and swiotlb, backed by their respective allocators, and a platform-specific one for tegra.
However, some users, like dma-buf heaps, might be interested in the exact type of a reserved memory region they are getting. It would thus be useful to have helpers to test if a given region is of a given type.
Since we only care about CMA for now though, let's create one for CMA only.
Signed-off-by: Maxime Ripard mripard@kernel.org --- include/linux/dma-map-ops.h | 13 +++++++++++++ kernel/dma/contiguous.c | 7 +++++++ 2 files changed, 20 insertions(+)
diff --git a/include/linux/dma-map-ops.h b/include/linux/dma-map-ops.h index f48e5fb88bd5dd346094bbf2ce1b79e5f5bfe1a6..ea646acb6367bd062619b337013db221749f85ab 100644 --- a/include/linux/dma-map-ops.h +++ b/include/linux/dma-map-ops.h @@ -153,10 +153,23 @@ static inline void dma_free_contiguous(struct device *dev, struct page *page, { __free_pages(page, get_order(size)); } #endif /* CONFIG_DMA_CMA*/
+#if defined(CONFIG_DMA_CMA) && defined(CONFIG_OF_RESERVED_MEM) +struct reserved_mem; + +bool of_reserved_mem_is_contiguous(const struct reserved_mem *rmem); +#else +struct reserved_mem; + +static inline bool of_reserved_mem_is_contiguous(const struct reserved_mem *rmem) +{ + return false; +} +#endif + #ifdef CONFIG_DMA_DECLARE_COHERENT int dma_declare_coherent_memory(struct device *dev, phys_addr_t phys_addr, dma_addr_t device_addr, size_t size); void dma_release_coherent_memory(struct device *dev); int dma_alloc_from_dev_coherent(struct device *dev, ssize_t size, diff --git a/kernel/dma/contiguous.c b/kernel/dma/contiguous.c index 8df0dfaaca18eeb0a20145512ba64425d2e7601e..ace4982e928e404315cf38551e1596f7ed445156 100644 --- a/kernel/dma/contiguous.c +++ b/kernel/dma/contiguous.c @@ -493,6 +493,13 @@ static int __init rmem_cma_setup(struct reserved_mem *rmem) &rmem->base, (unsigned long)rmem->size / SZ_1M);
return 0; } RESERVEDMEM_OF_DECLARE(cma, "shared-dma-pool", rmem_cma_setup); + +bool of_reserved_mem_is_contiguous(const struct reserved_mem *rmem) +{ + return rmem->ops == &rmem_cma_ops; +} +EXPORT_SYMBOL_GPL(of_reserved_mem_is_contiguous); + #endif
On 7/9/25 7:44 AM, Maxime Ripard wrote:
A given reserved-memory region can be of multiple types.
We have currently four types defined in the tree: contiguous, backed by CMA, coherent and swiotlb, backed by their respective allocators, and a platform-specific one for tegra.
However, some users, like dma-buf heaps, might be interested in the exact type of a reserved memory region they are getting. It would thus be useful to have helpers to test if a given region is of a given type.
Since we only care about CMA for now though, let's create one for CMA only.
Signed-off-by: Maxime Ripard mripard@kernel.org
include/linux/dma-map-ops.h | 13 +++++++++++++ kernel/dma/contiguous.c | 7 +++++++ 2 files changed, 20 insertions(+)
diff --git a/include/linux/dma-map-ops.h b/include/linux/dma-map-ops.h index f48e5fb88bd5dd346094bbf2ce1b79e5f5bfe1a6..ea646acb6367bd062619b337013db221749f85ab 100644 --- a/include/linux/dma-map-ops.h +++ b/include/linux/dma-map-ops.h @@ -153,10 +153,23 @@ static inline void dma_free_contiguous(struct device *dev, struct page *page, { __free_pages(page, get_order(size)); } #endif /* CONFIG_DMA_CMA*/ +#if defined(CONFIG_DMA_CMA) && defined(CONFIG_OF_RESERVED_MEM) +struct reserved_mem;
+bool of_reserved_mem_is_contiguous(const struct reserved_mem *rmem); +#else +struct reserved_mem;
+static inline bool of_reserved_mem_is_contiguous(const struct reserved_mem *rmem) +{
- return false;
+} +#endif
Should this all go in linux/of_reserved_mem.h?
#ifdef CONFIG_DMA_DECLARE_COHERENT int dma_declare_coherent_memory(struct device *dev, phys_addr_t phys_addr, dma_addr_t device_addr, size_t size); void dma_release_coherent_memory(struct device *dev); int dma_alloc_from_dev_coherent(struct device *dev, ssize_t size, diff --git a/kernel/dma/contiguous.c b/kernel/dma/contiguous.c index 8df0dfaaca18eeb0a20145512ba64425d2e7601e..ace4982e928e404315cf38551e1596f7ed445156 100644 --- a/kernel/dma/contiguous.c +++ b/kernel/dma/contiguous.c @@ -493,6 +493,13 @@ static int __init rmem_cma_setup(struct reserved_mem *rmem) &rmem->base, (unsigned long)rmem->size / SZ_1M); return 0; } RESERVEDMEM_OF_DECLARE(cma, "shared-dma-pool", rmem_cma_setup);
+bool of_reserved_mem_is_contiguous(const struct reserved_mem *rmem)
Needing to check where the reserved mem comes from seems wrong, it hints that the reserved mem region drivers, like this one, are not in full control of their regions. Instead of looping over all the regions in DT in the next patch and searching for the owner, how about the owner (this driver) call into __add_cma_heap() if it chooses to expose the region in that way.
(I know RESERVEDMEM_OF_DECLARE callbacks are done very early and the CMA-Heap driver might not be able to deal with adding heaps at this point, so maybe keeping a table the heaps driver can later iterate over would also work).
Andrew
+{
- return rmem->ops == &rmem_cma_ops;
+} +EXPORT_SYMBOL_GPL(of_reserved_mem_is_contiguous);
- #endif
Hi Andrew,
On Wed, Jul 09, 2025 at 10:55:40AM -0500, Andrew Davis wrote:
On 7/9/25 7:44 AM, Maxime Ripard wrote:
A given reserved-memory region can be of multiple types.
We have currently four types defined in the tree: contiguous, backed by CMA, coherent and swiotlb, backed by their respective allocators, and a platform-specific one for tegra.
However, some users, like dma-buf heaps, might be interested in the exact type of a reserved memory region they are getting. It would thus be useful to have helpers to test if a given region is of a given type.
Since we only care about CMA for now though, let's create one for CMA only.
Signed-off-by: Maxime Ripard mripard@kernel.org
include/linux/dma-map-ops.h | 13 +++++++++++++ kernel/dma/contiguous.c | 7 +++++++ 2 files changed, 20 insertions(+)
diff --git a/include/linux/dma-map-ops.h b/include/linux/dma-map-ops.h index f48e5fb88bd5dd346094bbf2ce1b79e5f5bfe1a6..ea646acb6367bd062619b337013db221749f85ab 100644 --- a/include/linux/dma-map-ops.h +++ b/include/linux/dma-map-ops.h @@ -153,10 +153,23 @@ static inline void dma_free_contiguous(struct device *dev, struct page *page, { __free_pages(page, get_order(size)); } #endif /* CONFIG_DMA_CMA*/ +#if defined(CONFIG_DMA_CMA) && defined(CONFIG_OF_RESERVED_MEM) +struct reserved_mem;
+bool of_reserved_mem_is_contiguous(const struct reserved_mem *rmem); +#else +struct reserved_mem;
+static inline bool of_reserved_mem_is_contiguous(const struct reserved_mem *rmem) +{
- return false;
+} +#endif
Should this all go in linux/of_reserved_mem.h?
#ifdef CONFIG_DMA_DECLARE_COHERENT int dma_declare_coherent_memory(struct device *dev, phys_addr_t phys_addr, dma_addr_t device_addr, size_t size); void dma_release_coherent_memory(struct device *dev); int dma_alloc_from_dev_coherent(struct device *dev, ssize_t size, diff --git a/kernel/dma/contiguous.c b/kernel/dma/contiguous.c index 8df0dfaaca18eeb0a20145512ba64425d2e7601e..ace4982e928e404315cf38551e1596f7ed445156 100644 --- a/kernel/dma/contiguous.c +++ b/kernel/dma/contiguous.c @@ -493,6 +493,13 @@ static int __init rmem_cma_setup(struct reserved_mem *rmem) &rmem->base, (unsigned long)rmem->size / SZ_1M); return 0; } RESERVEDMEM_OF_DECLARE(cma, "shared-dma-pool", rmem_cma_setup);
+bool of_reserved_mem_is_contiguous(const struct reserved_mem *rmem)
Needing to check where the reserved mem comes from seems wrong, it hints that the reserved mem region drivers, like this one, are not in full control of their regions. Instead of looping over all the regions in DT in the next patch and searching for the owner, how about the owner (this driver) call into __add_cma_heap() if it chooses to expose the region in that way.
(I know RESERVEDMEM_OF_DECLARE callbacks are done very early and the CMA-Heap driver might not be able to deal with adding heaps at this point, so maybe keeping a table the heaps driver can later iterate over would also work).
It's something I considered but wasn't too sure about, so I went the less intrusive way.
I'll work on that for the next version, thanks! Maxime
Aside from the main CMA region, it can be useful to allow userspace to allocate from the other CMA reserved regions.
Indeed, those regions can have specific properties that can be useful to a specific us-case.
For example, one of them platform I've been with has ECC enabled on the entire memory but for a specific region. Using that region to allocate framebuffers can be particular beneficial because enabling the ECC has a performance and memory footprint cost.
Thus, exposing these regions as heaps user-space can allocate from and import wherever needed allows to cover that use-case.
For now, only shared-dma-pools regions with the reusable property (ie, backed by CMA) are supported, but eventually we'll want to support other DMA pools types.
Signed-off-by: Maxime Ripard mripard@kernel.org --- drivers/dma-buf/heaps/cma_heap.c | 52 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 51 insertions(+), 1 deletion(-)
diff --git a/drivers/dma-buf/heaps/cma_heap.c b/drivers/dma-buf/heaps/cma_heap.c index 0df007111975447d555714d61ead9699287fd65a..31a18683ee25788a800f3f878fd958718a930ff7 100644 --- a/drivers/dma-buf/heaps/cma_heap.c +++ b/drivers/dma-buf/heaps/cma_heap.c @@ -19,10 +19,12 @@ #include <linux/err.h> #include <linux/highmem.h> #include <linux/io.h> #include <linux/mm.h> #include <linux/module.h> +#include <linux/of.h> +#include <linux/of_reserved_mem.h> #include <linux/scatterlist.h> #include <linux/slab.h> #include <linux/vmalloc.h>
#define DEFAULT_CMA_NAME "default_cma_region" @@ -421,7 +423,55 @@ static int __init add_default_cma_heap(void) ERR_PTR(ret)); }
return 0; } -module_init(add_default_cma_heap); + +static int __init add_cma_heaps(void) +{ + struct device_node *rmem_node; + struct device_node *node; + int ret; + + ret = add_default_cma_heap(); + if (ret) + return ret; + + rmem_node = of_find_node_by_path("/reserved-memory"); + if (!rmem_node) + goto out; + + for_each_child_of_node(rmem_node, node) { + struct reserved_mem *rmem; + struct cma *cma; + + rmem = of_reserved_mem_lookup(node); + if (!rmem) { + ret = -EINVAL; + goto err_put_node; + } + + if (!of_reserved_mem_is_contiguous(rmem)) + continue; + + cma = rmem->priv; + if (!cma) { + ret = -EINVAL; + goto err_put_node; + } + + ret = __add_cma_heap(cma, of_node_full_name(node)); + if (ret) + goto err_put_node; + } + +out: + of_node_put(rmem_node); + return 0; + +err_put_node: + of_node_put(rmem_node); + return ret; +} + +module_init(add_cma_heaps); MODULE_DESCRIPTION("DMA-BUF CMA Heap");
On 7/9/25 7:44 AM, Maxime Ripard wrote:
Aside from the main CMA region, it can be useful to allow userspace to allocate from the other CMA reserved regions.
Indeed, those regions can have specific properties that can be useful to a specific us-case.
For example, one of them platform I've been with has ECC enabled on the entire memory but for a specific region. Using that region to allocate framebuffers can be particular beneficial because enabling the ECC has a performance and memory footprint cost.
Thus, exposing these regions as heaps user-space can allocate from and import wherever needed allows to cover that use-case.
For now, only shared-dma-pools regions with the reusable property (ie, backed by CMA) are supported, but eventually we'll want to support other DMA pools types.
Signed-off-by: Maxime Ripard mripard@kernel.org
drivers/dma-buf/heaps/cma_heap.c | 52 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 51 insertions(+), 1 deletion(-)
diff --git a/drivers/dma-buf/heaps/cma_heap.c b/drivers/dma-buf/heaps/cma_heap.c index 0df007111975447d555714d61ead9699287fd65a..31a18683ee25788a800f3f878fd958718a930ff7 100644 --- a/drivers/dma-buf/heaps/cma_heap.c +++ b/drivers/dma-buf/heaps/cma_heap.c @@ -19,10 +19,12 @@ #include <linux/err.h> #include <linux/highmem.h> #include <linux/io.h> #include <linux/mm.h> #include <linux/module.h> +#include <linux/of.h> +#include <linux/of_reserved_mem.h> #include <linux/scatterlist.h> #include <linux/slab.h> #include <linux/vmalloc.h> #define DEFAULT_CMA_NAME "default_cma_region" @@ -421,7 +423,55 @@ static int __init add_default_cma_heap(void) ERR_PTR(ret)); } return 0; } -module_init(add_default_cma_heap);
+static int __init add_cma_heaps(void) +{
- struct device_node *rmem_node;
- struct device_node *node;
- int ret;
- ret = add_default_cma_heap();
Will this double add the default CMA region if it was declared using DT (reserved-memory) when all those nodes are again scanned through below? Might need a check in that loop for linux,cma-default.
- if (ret)
return ret;- rmem_node = of_find_node_by_path("/reserved-memory");
- if (!rmem_node)
goto out;
Can just return here, "out" path doesn't need to put a NULL node.
Andrew
- for_each_child_of_node(rmem_node, node) {
struct reserved_mem *rmem;struct cma *cma;rmem = of_reserved_mem_lookup(node);if (!rmem) {ret = -EINVAL;goto err_put_node;}if (!of_reserved_mem_is_contiguous(rmem))continue;cma = rmem->priv;if (!cma) {ret = -EINVAL;goto err_put_node;}ret = __add_cma_heap(cma, of_node_full_name(node));if (ret)goto err_put_node;- }
+out:
- of_node_put(rmem_node);
- return 0;
+err_put_node:
- of_node_put(rmem_node);
- return ret;
+}
+module_init(add_cma_heaps); MODULE_DESCRIPTION("DMA-BUF CMA Heap");
On Wed, Jul 09, 2025 at 11:14:37AM -0500, Andrew Davis wrote:
On 7/9/25 7:44 AM, Maxime Ripard wrote:
Aside from the main CMA region, it can be useful to allow userspace to allocate from the other CMA reserved regions.
Indeed, those regions can have specific properties that can be useful to a specific us-case.
For example, one of them platform I've been with has ECC enabled on the entire memory but for a specific region. Using that region to allocate framebuffers can be particular beneficial because enabling the ECC has a performance and memory footprint cost.
Thus, exposing these regions as heaps user-space can allocate from and import wherever needed allows to cover that use-case.
For now, only shared-dma-pools regions with the reusable property (ie, backed by CMA) are supported, but eventually we'll want to support other DMA pools types.
Signed-off-by: Maxime Ripard mripard@kernel.org
drivers/dma-buf/heaps/cma_heap.c | 52 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 51 insertions(+), 1 deletion(-)
diff --git a/drivers/dma-buf/heaps/cma_heap.c b/drivers/dma-buf/heaps/cma_heap.c index 0df007111975447d555714d61ead9699287fd65a..31a18683ee25788a800f3f878fd958718a930ff7 100644 --- a/drivers/dma-buf/heaps/cma_heap.c +++ b/drivers/dma-buf/heaps/cma_heap.c @@ -19,10 +19,12 @@ #include <linux/err.h> #include <linux/highmem.h> #include <linux/io.h> #include <linux/mm.h> #include <linux/module.h> +#include <linux/of.h> +#include <linux/of_reserved_mem.h> #include <linux/scatterlist.h> #include <linux/slab.h> #include <linux/vmalloc.h> #define DEFAULT_CMA_NAME "default_cma_region" @@ -421,7 +423,55 @@ static int __init add_default_cma_heap(void) ERR_PTR(ret)); } return 0; } -module_init(add_default_cma_heap);
+static int __init add_cma_heaps(void) +{
- struct device_node *rmem_node;
- struct device_node *node;
- int ret;
- ret = add_default_cma_heap();
Will this double add the default CMA region if it was declared using DT (reserved-memory) when all those nodes are again scanned through below? Might need a check in that loop for linux,cma-default.
Yeah, but we probably should anyway. Otherwise, if linux,cma-default ever change on a platform, we would get heaps appearing/disappearing as we reboot, which doesn't sound great from a regression perspective.
Both would allocate from the same pool though, so we don't risk stepping into each others toes. Or am I missing something?
- if (ret)
return ret;- rmem_node = of_find_node_by_path("/reserved-memory");
- if (!rmem_node)
goto out;Can just return here, "out" path doesn't need to put a NULL node.
Oh, right. Thanks! Maxime
On 7/10/25 2:44 AM, Maxime Ripard wrote:
On Wed, Jul 09, 2025 at 11:14:37AM -0500, Andrew Davis wrote:
On 7/9/25 7:44 AM, Maxime Ripard wrote:
Aside from the main CMA region, it can be useful to allow userspace to allocate from the other CMA reserved regions.
Indeed, those regions can have specific properties that can be useful to a specific us-case.
For example, one of them platform I've been with has ECC enabled on the entire memory but for a specific region. Using that region to allocate framebuffers can be particular beneficial because enabling the ECC has a performance and memory footprint cost.
Thus, exposing these regions as heaps user-space can allocate from and import wherever needed allows to cover that use-case.
For now, only shared-dma-pools regions with the reusable property (ie, backed by CMA) are supported, but eventually we'll want to support other DMA pools types.
Signed-off-by: Maxime Ripard mripard@kernel.org
drivers/dma-buf/heaps/cma_heap.c | 52 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 51 insertions(+), 1 deletion(-)
diff --git a/drivers/dma-buf/heaps/cma_heap.c b/drivers/dma-buf/heaps/cma_heap.c index 0df007111975447d555714d61ead9699287fd65a..31a18683ee25788a800f3f878fd958718a930ff7 100644 --- a/drivers/dma-buf/heaps/cma_heap.c +++ b/drivers/dma-buf/heaps/cma_heap.c @@ -19,10 +19,12 @@ #include <linux/err.h> #include <linux/highmem.h> #include <linux/io.h> #include <linux/mm.h> #include <linux/module.h> +#include <linux/of.h> +#include <linux/of_reserved_mem.h> #include <linux/scatterlist.h> #include <linux/slab.h> #include <linux/vmalloc.h> #define DEFAULT_CMA_NAME "default_cma_region" @@ -421,7 +423,55 @@ static int __init add_default_cma_heap(void) ERR_PTR(ret)); } return 0; } -module_init(add_default_cma_heap);
+static int __init add_cma_heaps(void) +{
- struct device_node *rmem_node;
- struct device_node *node;
- int ret;
- ret = add_default_cma_heap();
Will this double add the default CMA region if it was declared using DT (reserved-memory) when all those nodes are again scanned through below? Might need a check in that loop for linux,cma-default.
Yeah, but we probably should anyway. Otherwise, if linux,cma-default ever change on a platform, we would get heaps appearing/disappearing as we reboot, which doesn't sound great from a regression perspective.
Both would allocate from the same pool though, so we don't risk stepping into each others toes. Or am I missing something?
You are not missing anything, having both wouldn't cause anything to break, but would cause heaps to appear/disappear based on how the CMA region was defined (DT vs kernel cmd line) which we should avoid.
Andrew
- if (ret)
return ret;- rmem_node = of_find_node_by_path("/reserved-memory");
- if (!rmem_node)
goto out;Can just return here, "out" path doesn't need to put a NULL node.
Oh, right. Thanks! Maxime
On Thu, Jul 10, 2025 at 09:46:56AM -0500, Andrew Davis wrote:
On 7/10/25 2:44 AM, Maxime Ripard wrote:
On Wed, Jul 09, 2025 at 11:14:37AM -0500, Andrew Davis wrote:
On 7/9/25 7:44 AM, Maxime Ripard wrote:
Aside from the main CMA region, it can be useful to allow userspace to allocate from the other CMA reserved regions.
Indeed, those regions can have specific properties that can be useful to a specific us-case.
For example, one of them platform I've been with has ECC enabled on the entire memory but for a specific region. Using that region to allocate framebuffers can be particular beneficial because enabling the ECC has a performance and memory footprint cost.
Thus, exposing these regions as heaps user-space can allocate from and import wherever needed allows to cover that use-case.
For now, only shared-dma-pools regions with the reusable property (ie, backed by CMA) are supported, but eventually we'll want to support other DMA pools types.
Signed-off-by: Maxime Ripard mripard@kernel.org
drivers/dma-buf/heaps/cma_heap.c | 52 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 51 insertions(+), 1 deletion(-)
diff --git a/drivers/dma-buf/heaps/cma_heap.c b/drivers/dma-buf/heaps/cma_heap.c index 0df007111975447d555714d61ead9699287fd65a..31a18683ee25788a800f3f878fd958718a930ff7 100644 --- a/drivers/dma-buf/heaps/cma_heap.c +++ b/drivers/dma-buf/heaps/cma_heap.c @@ -19,10 +19,12 @@ #include <linux/err.h> #include <linux/highmem.h> #include <linux/io.h> #include <linux/mm.h> #include <linux/module.h> +#include <linux/of.h> +#include <linux/of_reserved_mem.h> #include <linux/scatterlist.h> #include <linux/slab.h> #include <linux/vmalloc.h> #define DEFAULT_CMA_NAME "default_cma_region" @@ -421,7 +423,55 @@ static int __init add_default_cma_heap(void) ERR_PTR(ret)); } return 0; } -module_init(add_default_cma_heap);
+static int __init add_cma_heaps(void) +{
- struct device_node *rmem_node;
- struct device_node *node;
- int ret;
- ret = add_default_cma_heap();
Will this double add the default CMA region if it was declared using DT (reserved-memory) when all those nodes are again scanned through below? Might need a check in that loop for linux,cma-default.
Yeah, but we probably should anyway. Otherwise, if linux,cma-default ever change on a platform, we would get heaps appearing/disappearing as we reboot, which doesn't sound great from a regression perspective.
Both would allocate from the same pool though, so we don't risk stepping into each others toes. Or am I missing something?
You are not missing anything, having both wouldn't cause anything to break, but would cause heaps to appear/disappear based on how the CMA region was defined (DT vs kernel cmd line) which we should avoid.
I'm sorry I guess I don't follow you then. It looked like you were arguing for avoiding double registration (under the CMA default name, and the region specific name).
I'm arguing that we should always double register to avoid linux,cma-default having any effect on the userspace.
In the latter, even if linux,cma-default isn't set at all, then the default CMA heap would still be registered from the command line CMA region.
Maxime
Hi Maxime,
Le mercredi 09 juillet 2025 à 14:44 +0200, Maxime Ripard a écrit :
Hi,
Here's another attempt at supporting user-space allocations from a specific carved-out reserved memory region.
The initial problem we were discussing was that I'm currently working on a platform which has a memory layout with ECC enabled. However, enabling the ECC has a number of drawbacks on that platform: lower performance, increased memory usage, etc. So for things like framebuffers, the trade-off isn't great and thus there's a memory region with ECC disabled to allocate from for such use cases.
After a suggestion from John, I chose to first start using heap allocations flags to allow for userspace to ask for a particular ECC setup. This is then backed by a new heap type that runs from reserved memory chunks flagged as such, and the existing DT properties to specify the ECC properties.
After further discussion, it was considered that flags were not the right solution, and relying on the names of the heaps would be enough to let userspace know the kind of buffer it deals with.
Thus, even though the uAPI part of it had been dropped in this second version, we still needed a driver to create heaps out of carved-out memory regions. In addition to the original usecase, a similar driver can be found in BSPs from most vendors, so I believe it would be a useful addition to the kernel.
Some extra discussion with Rob Herring [1] came to the conclusion that some specific compatible for this is not great either, and as such an new driver probably isn't called for either.
Some other discussions we had with John [2] also dropped some hints that multiple CMA heaps might be a good idea, and some vendors seem to do that too.
So here's another attempt that doesn't affect the device tree at all and will just create a heap for every CMA reserved memory region.
Does that means that if we carve-out memory for a co-processor operating system, that memory region is now available to userspace to allocate from ? Or is there a nuance to that ?
For other carveout, such as RK3588 HDMI receiver, that is clearly a win, giving user the ability to allocate using externally supplied constraints rather then having to convince the v4l2 driver to match these. While keeping the safety that this carveout will yield valid addresses for the IP.
Will there be a generic way to find out which driver/device this carveout belongs to ? In V4L2, only complex cameras have userspace drivers, everything else is generic code.
Nicolas
It also falls nicely into the current plan we have to support cgroups in DRM/KMS and v4l2, which is an additional benefit.
Let me know what you think, Maxime
1: https://lore.kernel.org/all/20250707-cobalt-dingo-of-serenity-dbf92c@houat/ 2: https://lore.kernel.org/all/CANDhNCroe6ZBtN_o=c71kzFFaWK-fF5rCdnr9P5h1sgPOWS...
Let me know what you think, Maxime
Signed-off-by: Maxime Ripard mripard@kernel.org
Changes in v6:
- Drop the new driver and allocate a CMA heap for each region now
- Dropped the binding
- Rebased on 6.16-rc5
- Link to v5:
https://lore.kernel.org/r/20250617-dma-buf-ecc-heap-v5-0-0abdc5863a4f@kernel...
Changes in v5:
- Rebased on 6.16-rc2
- Switch from property to dedicated binding
- Link to v4:
https://lore.kernel.org/r/20250520-dma-buf-ecc-heap-v4-1-bd2e1f1bb42c@kernel...
Changes in v4:
- Rebased on 6.15-rc7
- Map buffers only when map is actually called, not at allocation time
- Deal with restricted-dma-pool and shared-dma-pool
- Reword Kconfig options
- Properly report dma_map_sgtable failures
- Link to v3:
https://lore.kernel.org/r/20250407-dma-buf-ecc-heap-v3-0-97cdd36a5f29@kernel...
Changes in v3:
- Reworked global variable patch
- Link to v2:
https://lore.kernel.org/r/20250401-dma-buf-ecc-heap-v2-0-043fd006a1af@kernel...
Changes in v2:
- Add vmap/vunmap operations
- Drop ECC flags uapi
- Rebase on top of 6.14
- Link to v1:
https://lore.kernel.org/r/20240515-dma-buf-ecc-heap-v1-0-54cbbd049511@kernel...
Maxime Ripard (2): dma/contiguous: Add helper to test reserved memory type dma-buf: heaps: cma: Create CMA heap for each CMA reserved region
drivers/dma-buf/heaps/cma_heap.c | 52 +++++++++++++++++++++++++++++++++++++++- include/linux/dma-map-ops.h | 13 ++++++++++ kernel/dma/contiguous.c | 7 ++++++ 3 files changed, 71 insertions(+), 1 deletion(-)
base-commit: 47633099a672fc7bfe604ef454e4f116e2c954b1 change-id: 20240515-dma-buf-ecc-heap-28a311d2c94e prerequisite-message-id: 20250610131231.1724627-1-jkangas@redhat.com prerequisite-patch-id: bc44be5968feb187f2bc1b8074af7209462b18e7 prerequisite-patch-id: f02a91b723e5ec01fbfedf3c3905218b43d432da prerequisite-patch-id: e944d0a3e22f2cdf4d3b3906e5603af934696deb
Best regards,
On Wed, Jul 09, 2025 at 09:10:02AM -0400, Nicolas Dufresne wrote:
Hi Maxime,
Le mercredi 09 juillet 2025 à 14:44 +0200, Maxime Ripard a écrit :
Hi,
Here's another attempt at supporting user-space allocations from a specific carved-out reserved memory region.
The initial problem we were discussing was that I'm currently working on a platform which has a memory layout with ECC enabled. However, enabling the ECC has a number of drawbacks on that platform: lower performance, increased memory usage, etc. So for things like framebuffers, the trade-off isn't great and thus there's a memory region with ECC disabled to allocate from for such use cases.
After a suggestion from John, I chose to first start using heap allocations flags to allow for userspace to ask for a particular ECC setup. This is then backed by a new heap type that runs from reserved memory chunks flagged as such, and the existing DT properties to specify the ECC properties.
After further discussion, it was considered that flags were not the right solution, and relying on the names of the heaps would be enough to let userspace know the kind of buffer it deals with.
Thus, even though the uAPI part of it had been dropped in this second version, we still needed a driver to create heaps out of carved-out memory regions. In addition to the original usecase, a similar driver can be found in BSPs from most vendors, so I believe it would be a useful addition to the kernel.
Some extra discussion with Rob Herring [1] came to the conclusion that some specific compatible for this is not great either, and as such an new driver probably isn't called for either.
Some other discussions we had with John [2] also dropped some hints that multiple CMA heaps might be a good idea, and some vendors seem to do that too.
So here's another attempt that doesn't affect the device tree at all and will just create a heap for every CMA reserved memory region.
Does that means that if we carve-out memory for a co-processor operating system, that memory region is now available to userspace to allocate from ? Or is there a nuance to that ?
There is a nuance to that :)
You need to have the "reusable" property set which is documented as:
The operating system can use the memory in this region with the limitation that the device driver(s) owning the region need to be able to reclaim it back. Typically that means that the operating system can use that region to store volatile or cached data that can be otherwise regenerated or migrated elsewhere.
https://github.com/devicetree-org/dt-schema/blob/main/dtschema/schemas/reser...
If it's not set, it's not exposed, and I'd expect a coprocessor memory region wouldn't be flagged as such.
For other carveout, such as RK3588 HDMI receiver, that is clearly a win, giving user the ability to allocate using externally supplied constraints rather then having to convince the v4l2 driver to match these. While keeping the safety that this carveout will yield valid addresses for the IP.
Will there be a generic way to find out which driver/device this carveout belongs to ? In V4L2, only complex cameras have userspace drivers, everything else is generic code.
I believe it's a separate discussion, but the current stance is that the heap name is enough to identify in a platform-specific way where you allocate from. I've worked on documenting what a good name is so userspace can pick it up more easily here:
https://lore.kernel.org/r/20250616-dma-buf-heap-names-doc-v2-1-8ae43174cdbf@...
But it's not really what you expected
Maxime
Hi,
Le mercredi 09 juillet 2025 à 15:38 +0200, Maxime Ripard a écrit :
Will there be a generic way to find out which driver/device this carveout belongs to ? In V4L2, only complex cameras have userspace drivers, everything else is generic code.
I believe it's a separate discussion, but the current stance is that the heap name is enough to identify in a platform-specific way where you allocate from. I've worked on documenting what a good name is so userspace can pick it up more easily here:
https://lore.kernel.org/r/20250616-dma-buf-heap-names-doc-v2-1-8ae43174cdbf@...
But it's not really what you expected
From a dma-heap API, the naming rules seems necessary, but suggesting generic code to use "grep" style of search to match a heap is extremely fragile. The documentation you propose is (intentionally?) vague. For me, the naming is more like giving proper names to your function calls do devs can make sense out of it.
Stepping back a little, we already opened the door for in-driver use of heaps. So perhaps the way forward is to have V4L2 drivers utilize heaps from inside the kernel. Once driver are fully ported, additional APIs could be added so that userspace can read which heap(s) is going to be used for the active configuration, and which other heaps are known usable (enumerate them). There is no need to add properties in that context, since these will derives from the driver configuration you picked. If you told you driver you doing secure memory playback, the driver will filter-out what can't be used.
Examples out there often express simplified view of the problem. Your ECC video playback case is a good one. Let's say you have performance issue in both decoder and display due to ECC. You may think that you just allocate from a non- ECC heap, import these into the decoder, and once filled, import these into the display driver and you won.
But in reality, your display buffer might not be the reference buffers, and most of the memory bandwidth in a modern decoder goes into reading reference frames and the attached metadata (the later which may or may not be in the same allocation block).
Even once the reference frames get exposed to userspace (which is a long term goal), there will still be couple of buffers that just simply don't fit and must be kept hidden inside the driver.
My general conclusion is that once these heap exists, and that we guarantee platform specific unique names, we should probably build on top. Both userspace and driver become consumers of the heap. And for the case where the platform- specific knowledge lives inside the kernel, then heaps are selected by the kernel. Also, very little per-driver duplication will be needed, since 90% of the V4L2 driver share the allocator implementation.
Does that makes any sense to anyone ?
Nicolas
On Thu, Jul 10, 2025 at 11:21:02AM -0400, Nicolas Dufresne wrote:
Hi,
Le mercredi 09 juillet 2025 à 15:38 +0200, Maxime Ripard a écrit :
Will there be a generic way to find out which driver/device this carveout belongs to ? In V4L2, only complex cameras have userspace drivers, everything else is generic code.
I believe it's a separate discussion, but the current stance is that the heap name is enough to identify in a platform-specific way where you allocate from. I've worked on documenting what a good name is so userspace can pick it up more easily here:
https://lore.kernel.org/r/20250616-dma-buf-heap-names-doc-v2-1-8ae43174cdbf@...
But it's not really what you expected
From a dma-heap API, the naming rules seems necessary, but suggesting generic code to use "grep" style of search to match a heap is extremely fragile. The documentation you propose is (intentionally?) vague. For me, the naming is more like giving proper names to your function calls do devs can make sense out of it.
I agree, and made a proposal to implement some kind of heap capabilities discovery ioctl. The main concern at the time was that Android tried that with ION and it lead to a proliferation of poorly defined flags, and that names were enough to do so.
I still think that at some point we will need this, but I also don't have a good idea to address these concerns.
Stepping back a little, we already opened the door for in-driver use of heaps. So perhaps the way forward is to have V4L2 drivers utilize heaps from inside the kernel. Once driver are fully ported, additional APIs could be added so that userspace can read which heap(s) is going to be used for the active configuration, and which other heaps are known usable (enumerate them). There is no need to add properties in that context, since these will derives from the driver configuration you picked. If you told you driver you doing secure memory playback, the driver will filter-out what can't be used.
Examples out there often express simplified view of the problem. Your ECC video playback case is a good one. Let's say you have performance issue in both decoder and display due to ECC. You may think that you just allocate from a non- ECC heap, import these into the decoder, and once filled, import these into the display driver and you won.
But in reality, your display buffer might not be the reference buffers, and most of the memory bandwidth in a modern decoder goes into reading reference frames and the attached metadata (the later which may or may not be in the same allocation block).
Even once the reference frames get exposed to userspace (which is a long term goal), there will still be couple of buffers that just simply don't fit and must be kept hidden inside the driver.
My general conclusion is that once these heap exists, and that we guarantee platform specific unique names, we should probably build on top. Both userspace and driver become consumers of the heap. And for the case where the platform- specific knowledge lives inside the kernel, then heaps are selected by the kernel. Also, very little per-driver duplication will be needed, since 90% of the V4L2 driver share the allocator implementation.
Does that makes any sense to anyone ?
It does, and it's roughly what we have in mind for the cgroups support in KMS and v4l2. The main issue with it is that knowing if you allocate from a dedicated pool (which would use the dmem cgroup controller) or the main memory pool (which would use memcg) wasn't deterministic and thus you couldn't properly account.
The solution we have in mind right now is indeed to switch everyone to using heaps, and then exposing which cgroup that heap allocates from.
Your proposal here has a few extra steps, but the main idea is there still.
Maxime
linaro-mm-sig@lists.linaro.org
-
Andrew Davis -
Maxime Ripard -
Nicolas Dufresne