set_memory_decrypted() doesn't (currently) guarantee to preserve or zero memory, but both the GICv3 ITS driver and the system_cc_shared dma-buf heap currently allocate memory with __GFP_ZERO followed by calling set_memory_decrypted(). On an Arm CCA system with MEC this can cause ciphertext to be visible to the guest rather than the expected zeros.
Patches 1 and 3 fix this by zeroing after the set_memory_decrypted() call.
Patches 2 and 4 fix other related bugs that Sashiko found. Patch 2 fixes the issue that set_memory_decrypted() can be a sleeping call, so moves the allocation out of an atomic context.
Patch 4 deals with the situation where set_memory_decrypted() fails and the rollback path could attempt to re-encrypt memory which was never decrypted.
I've sorted the patches by area, but there's no (semantic) dependency between them.
Changes in v2: * Switched to use BIT(n) rather than 1 << n in the GICv3 change. * Added Jason's R-b. * Patches 2 and 4 are new.
v1: https://lore.kernel.org/r/20260820105026.53208-1-steven.price@arm.com
Steven Price (4): irqchip/gic-v3-its: Zero shared pages after conversion irqchip/gic-v3-its: Allocate VPE tables from sleepable context dma-buf: heaps: Zero system shared heap pages after conversion dma-buf: heaps: Fix shared system heap allocation rollback
drivers/dma-buf/heaps/system_heap.c | 24 +++++++++++++----- drivers/irqchip/irq-gic-v3-its.c | 39 ++++++++++++++++------------- 2 files changed, 40 insertions(+), 23 deletions(-)
its_alloc_pages_node() passes __GFP_ZERO to the page allocator before calling set_memory_decrypted(). This assumes that converting a page from private to shared preserves its contents.
For Arm CCA with MEC (Memory Encryption Contexts) the key used to access the page will change, and so by default the visible data will change. The host could ensure that it zeros the page, but rather than relying on the host's behaviour it's best if the guest simply zeros after the decryption rather than before. Specifically in this case the ITS tables are required to be zeroed.
Mask out __GFP_ZERO from the allocation request, and do the zeroing as a separate step after decryption.
Reviewed-by: Jason Gunthorpe jgg@nvidia.com Fixes: b08e2f42e86b ("irqchip/gic-v3-its: Share ITS tables with a non-trusted hypervisor") Signed-off-by: Steven Price steven.price@arm.com --- drivers/irqchip/irq-gic-v3-its.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c index 6f5811aae59c..a055837832bc 100644 --- a/drivers/irqchip/irq-gic-v3-its.c +++ b/drivers/irqchip/irq-gic-v3-its.c @@ -213,16 +213,18 @@ static gfp_t gfp_flags_quirk; static struct page *its_alloc_pages_node(int node, gfp_t gfp, unsigned int order) { + bool want_zero = gfp & __GFP_ZERO; struct page *page; int ret = 0;
- page = alloc_pages_node(node, gfp | gfp_flags_quirk, order); + page = alloc_pages_node(node, (gfp & ~__GFP_ZERO) | gfp_flags_quirk, + order);
if (!page) return NULL;
ret = set_memory_decrypted((unsigned long)page_address(page), - 1 << order); + BIT(order)); /* * If set_memory_decrypted() fails then we don't know what state the * page is in, so we can't free it. Instead we leak it. @@ -231,6 +233,9 @@ static struct page *its_alloc_pages_node(int node, gfp_t gfp, if (ret) return NULL;
+ if (want_zero) + clear_pages(page_address(page), BIT(order)); + return page; }
The VPE L1 table is allocated from the CPU-starting hotplug state, where interrupts are disabled. Although the page allocation uses GFP_ATOMIC, its_alloc_pages() subsequently calls set_memory_decrypted(), which can sleep while splitting the arm64 linear map.
Move the allocation to the existing CPU-online callback, which runs in sleepable context, and use GFP_KERNEL for both allocations performed there. Register the callback even without EFI, since it is now also responsible for VPE table allocation.
Fixes: b08e2f42e86b ("irqchip/gic-v3-its: Share ITS tables with a non-trusted hypervisor") Signed-off-by: Steven Price steven.price@arm.com --- drivers/irqchip/irq-gic-v3-its.c | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-)
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c index a055837832bc..71515dbff9ec 100644 --- a/drivers/irqchip/irq-gic-v3-its.c +++ b/drivers/irqchip/irq-gic-v3-its.c @@ -2932,7 +2932,7 @@ static int allocate_vpe_l1_table(void) if (val & GICR_VPROPBASER_4_1_VALID) goto out;
- gic_data_rdist()->vpe_table_mask = kzalloc_obj(cpumask_t, GFP_ATOMIC); + gic_data_rdist()->vpe_table_mask = kzalloc_obj(cpumask_t, GFP_KERNEL); if (!gic_data_rdist()->vpe_table_mask) return -ENOMEM;
@@ -2999,7 +2999,7 @@ static int allocate_vpe_l1_table(void)
pr_debug("np = %d, npg = %lld, psz = %d, epp = %d, esz = %d\n", np, npg, psz, epp, esz); - page = its_alloc_pages(GFP_ATOMIC | __GFP_ZERO, get_order(np * PAGE_SIZE)); + page = its_alloc_pages(GFP_KERNEL | __GFP_ZERO, get_order(np * PAGE_SIZE)); if (!page) return -ENOMEM;
@@ -3268,16 +3268,6 @@ static void its_cpu_init_lpis(void) val = its_clear_vpend_valid(vlpi_base, 0, 0); }
- if (allocate_vpe_l1_table()) { - /* - * If the allocation has failed, we're in massive trouble. - * Disable direct injection, and pray that no VM was - * already running... - */ - gic_rdists->has_rvpeid = false; - gic_rdists->has_vlpis = false; - } - /* Make sure the GIC has seen the above */ dsb(sy); gic_data_rdist()->flags |= RD_LOCAL_LPI_ENABLED; @@ -5452,6 +5442,19 @@ static int its_cpu_memreserve_lpi(unsigned int cpu) if (gic_data_rdist()->flags & RD_LOCAL_MEMRESERVE_DONE) return 0;
+ if (allocate_vpe_l1_table()) { + /* + * If the allocation has failed, we're in massive trouble. + * Disable direct injection, and pray that no VM was + * already running... + */ + gic_rdists->has_rvpeid = false; + gic_rdists->has_vlpis = false; + } + + if (!efi_enabled(EFI_CONFIG_TABLES)) + goto out; + pend_page = gic_data_rdist()->pend_page; if (WARN_ON(!pend_page)) { ret = -ENOMEM; @@ -5793,9 +5796,6 @@ int __init its_lpi_memreserve_init(void) { int state;
- if (!efi_enabled(EFI_CONFIG_TABLES)) - return 0; - if (list_empty(&its_nodes)) return 0;
On 20/08/2026 16:00, Steven Price wrote:
The VPE L1 table is allocated from the CPU-starting hotplug state, where interrupts are disabled. Although the page allocation uses GFP_ATOMIC, its_alloc_pages() subsequently calls set_memory_decrypted(), which can sleep while splitting the arm64 linear map.
Move the allocation to the existing CPU-online callback, which runs in sleepable context, and use GFP_KERNEL for both allocations performed there. Register the callback even without EFI, since it is now also responsible for VPE table allocation.
Ok, Sashiko pointed out this is bunk:
Does moving this allocation to the CPUHP_AP_ONLINE_DYN callback expose uninitialized GICv4.1 redistributor state to concurrent VPE mappings?
The CPUHP_AP_ONLINE_DYN state runs for its_cpu_memreserve_lpi() in the hotplug thread asynchronously after the CPU has already been marked online in cpu_online_mask.
Could a concurrent process changing IRQ affinities (like irqbalance) observe the CPU in cpu_online_mask and route a virtual interrupt to it via its_vpe_set_affinity() before the hotplug thread has executed this table allocation?
If a VMAPP command executes before allocate_vpe_l1_table() programs the GICR_VPROPBASER, it appears the hardware could use stale or uninitialized physical addresses, leading to unpredictable behavior or dropped interrupts.
Please ignore this patch. I thought this was a bit too simple to fix :( I guess some sort of preallocation might be the solution.
Thanks, Steve
Fixes: b08e2f42e86b ("irqchip/gic-v3-its: Share ITS tables with a non-trusted hypervisor") Signed-off-by: Steven Price steven.price@arm.com
drivers/irqchip/irq-gic-v3-its.c | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-)
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c index a055837832bc..71515dbff9ec 100644 --- a/drivers/irqchip/irq-gic-v3-its.c +++ b/drivers/irqchip/irq-gic-v3-its.c @@ -2932,7 +2932,7 @@ static int allocate_vpe_l1_table(void) if (val & GICR_VPROPBASER_4_1_VALID) goto out;
- gic_data_rdist()->vpe_table_mask = kzalloc_obj(cpumask_t, GFP_ATOMIC);
- gic_data_rdist()->vpe_table_mask = kzalloc_obj(cpumask_t, GFP_KERNEL); if (!gic_data_rdist()->vpe_table_mask) return -ENOMEM;
@@ -2999,7 +2999,7 @@ static int allocate_vpe_l1_table(void) pr_debug("np = %d, npg = %lld, psz = %d, epp = %d, esz = %d\n", np, npg, psz, epp, esz);
- page = its_alloc_pages(GFP_ATOMIC | __GFP_ZERO, get_order(np * PAGE_SIZE));
- page = its_alloc_pages(GFP_KERNEL | __GFP_ZERO, get_order(np * PAGE_SIZE)); if (!page) return -ENOMEM;
@@ -3268,16 +3268,6 @@ static void its_cpu_init_lpis(void) val = its_clear_vpend_valid(vlpi_base, 0, 0); }
- if (allocate_vpe_l1_table()) {
/** If the allocation has failed, we're in massive trouble.* Disable direct injection, and pray that no VM was* already running...*/gic_rdists->has_rvpeid = false;gic_rdists->has_vlpis = false;- }
- /* Make sure the GIC has seen the above */ dsb(sy); gic_data_rdist()->flags |= RD_LOCAL_LPI_ENABLED;
@@ -5452,6 +5442,19 @@ static int its_cpu_memreserve_lpi(unsigned int cpu) if (gic_data_rdist()->flags & RD_LOCAL_MEMRESERVE_DONE) return 0;
- if (allocate_vpe_l1_table()) {
/** If the allocation has failed, we're in massive trouble.* Disable direct injection, and pray that no VM was* already running...*/gic_rdists->has_rvpeid = false;gic_rdists->has_vlpis = false;- }
- if (!efi_enabled(EFI_CONFIG_TABLES))
goto out;- pend_page = gic_data_rdist()->pend_page; if (WARN_ON(!pend_page)) { ret = -ENOMEM;
@@ -5793,9 +5796,6 @@ int __init its_lpi_memreserve_init(void) { int state;
- if (!efi_enabled(EFI_CONFIG_TABLES))
return 0;- if (list_empty(&its_nodes)) return 0;
The system_cc_shared heap allocates pages with __GFP_ZERO before converting them from private to shared with set_memory_decrypted(). This assumes that the conversion preserves the contents of the pages.
For Arm CCA with MEC (Memory Encryption Contexts) the key used to access the page will change, and so by default the visible data will change. The host could ensure that it zeros the page after decryption, but rather than relying on the host's behaviour it's best if the guest simply zeros after the decryption rather than before.
For CC shared buffers, defer zeroing until each page has been converted successfully. For other buffers keep the existing behaviour.
Reviewed-by: Jason Gunthorpe jgg@nvidia.com Fixes: 78b30c50a7ac ("dma-buf: heaps: system: add system_cc_shared heap for explicitly shared memory") Signed-off-by: Steven Price steven.price@arm.com --- drivers/dma-buf/heaps/system_heap.c | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)
diff --git a/drivers/dma-buf/heaps/system_heap.c b/drivers/dma-buf/heaps/system_heap.c index c8959eadc71d..f14930904089 100644 --- a/drivers/dma-buf/heaps/system_heap.c +++ b/drivers/dma-buf/heaps/system_heap.c @@ -376,7 +376,8 @@ static const struct dma_buf_ops system_heap_buf_ops = { };
static struct page *alloc_largest_available(unsigned long size, - unsigned int max_order) + unsigned int max_order, + bool defer_zero) { struct page *page; int i; @@ -388,6 +389,9 @@ static struct page *alloc_largest_available(unsigned long size, if (max_order < orders[i]) continue; flags = order_flags[i]; + /* Decryption can change the contents, so clear it afterwards. */ + if (defer_zero) + flags &= ~__GFP_ZERO; if (mem_accounting) flags |= __GFP_ACCOUNT; page = alloc_pages(flags, orders[i]); @@ -438,7 +442,8 @@ static struct dma_buf *system_heap_allocate(struct dma_heap *heap, goto free_buffer; }
- page = alloc_largest_available(size_remaining, max_order); + page = alloc_largest_available(size_remaining, max_order, + cc_shared_buffer(buffer)); if (!page) goto free_buffer;
@@ -461,9 +466,12 @@ static struct dma_buf *system_heap_allocate(struct dma_heap *heap,
if (cc_shared_buffer(buffer)) { for_each_sgtable_sg(table, sg, i) { - ret = system_heap_set_page_decrypted(sg_page(sg)); + page = sg_page(sg); + ret = system_heap_set_page_decrypted(page); if (ret) goto free_pages; + + clear_pages(page_address(page), 1 << compound_order(page)); } }
If converting one of the allocated pages to shared memory fails, the cleanup path attempts to convert every page back to private memory. Pages after the failed page have not been converted yet, so attempting to convert them back can fail and cause otherwise reusable memory to be leaked.
Count the pages converted successfully and only convert those and the failed allocation back during cleanup. If converting the failed allocation back succeeds it can be freed safely; otherwise it is leaked because its state is unknown. Allocations that were not converted can be freed directly.
Fixes: 78b30c50a7ac ("dma-buf: heaps: system: add system_cc_shared heap for explicitly shared memory") Signed-off-by: Steven Price steven.price@arm.com --- drivers/dma-buf/heaps/system_heap.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
diff --git a/drivers/dma-buf/heaps/system_heap.c b/drivers/dma-buf/heaps/system_heap.c index f14930904089..8d3ffeb64e00 100644 --- a/drivers/dma-buf/heaps/system_heap.c +++ b/drivers/dma-buf/heaps/system_heap.c @@ -418,6 +418,7 @@ static struct dma_buf *system_heap_allocate(struct dma_heap *heap, struct scatterlist *sg; struct list_head pages; struct page *page, *tmp_page; + int nr_decrypted = 0; int i, ret = -ENOMEM;
buffer = kzalloc_obj(*buffer); @@ -472,6 +473,7 @@ static struct dma_buf *system_heap_allocate(struct dma_heap *heap, goto free_pages;
clear_pages(page_address(page), 1 << compound_order(page)); + nr_decrypted++; } }
@@ -496,9 +498,11 @@ static struct dma_buf *system_heap_allocate(struct dma_heap *heap, * Intentionally leak pages that cannot be re-encrypted * to prevent shared memory from being reused. */ - if (cc_shared_buffer(buffer) && - system_heap_set_page_encrypted(p)) - continue; + if (cc_shared_buffer(buffer)) { + if (i <= nr_decrypted && + system_heap_set_page_encrypted(p)) + continue; + } __free_pages(p, compound_order(p)); } sg_free_table(table);
linaro-mm-sig@lists.linaro.org
-
Steven Price