Hello everyone,
Moving the discussion from google doc [1] to email (Sorry, it was becoming impossible to follow and reply there).
FWIW, I am not the best person to answer all questions here, that would be Armelle as she understand the requirements and the end goal much better than I do. I can though try to answer from kernel's perspective, based on whatever implementation we have right now.
AFAIK, the broad idea is to implement two virtio communication paths from pVM, one to the Linux host (via virtio-pci) and another one to Trusty (via virtio-msg-ffa). In order to not take performance hit at runtime (to map io buffers), the idea is to map whatever amount of memory we can at the beginning and then keep allocating from there.
Current setup:
What we have achieved until now is virtio-msg communication between host and trusty. We have implemented FFA specific dma-hal [2] to perform FFA memory sharing with trusty. With "reserved-mem" and "memory-region" DT entries (not sure if that is the final solution), we are able to allocate memory the FFA device (which represents bus for all the enumerated devices between trusty/host). This memory is shared with trusty at probe time (from virtio-msg-ffa layer) and the DMA hal later allocates memory from there for coherent allocations and bounce buffers. This works just fine right now.
Now looking at "dynamic mapping" section in [1] we are not sure if that will work fine for the end use case, pVM to trusty. It looks like the coco implementation will always end up using dma encrypt/decrypt when a pVM is running and share the memory with host, even when all we want to do is share with trusty. Is that understanding correct ? We would also want to establish virtio-pci (existing tech) based communication between pVM and host, which should use mem decrypt path (?).
I am not sure if we need contiguous PA here, contiguous IPA should be sufficient, Armelle?
We are also looking for further suggestions to improve the design, as my understanding of memory mapping, dma hal etc. is limited, and maybe there are better ways to do this.
-- Viresh
[1] https://docs.google.com/document/d/1KyxclKngQ0MShX8Q1YPstZQVIDIctkxfj6lQ0V3R... [2] https://web.git.kernel.org/pub/scm/linux/kernel/git/vireshk/linux.git/tree/d...
On Fri, Mar 07, 2025 at 11:59:56AM +0530, Viresh Kumar wrote:
Hi,
Hello everyone,
Moving the discussion from google doc [1] to email (Sorry, it was becoming impossible to follow and reply there).
FWIW, I am not the best person to answer all questions here, that would be Armelle as she understand the requirements and the end goal much better than I do. I can though try to answer from kernel's perspective, based on whatever implementation we have right now.
AFAIK, the broad idea is to implement two virtio communication paths from pVM, one to the Linux host (via virtio-pci) and another one to Trusty (via virtio-msg-ffa).
We already have from pVM to Host (not over FF-A but with pKVM APIs). What is new here/what do we want to do ?
In order to not take performance hit at runtime (to map io buffers), the idea is to map whatever amount of memory we can at the beginning and then keep allocating from there.
Current setup:
What we have achieved until now is virtio-msg communication between host and trusty. We have implemented FFA specific dma-hal [2] to perform FFA memory sharing with trusty. With "reserved-mem" and "memory-region" DT entries (not sure if that is the final solution), we are able to allocate memory the FFA device (which represents bus for all the enumerated devices between trusty/host). This memory is shared with trusty at probe time (from virtio-msg-ffa layer) and the DMA hal later allocates memory from there for coherent allocations and bounce buffers. This works just fine right now.
Now looking at "dynamic mapping" section in [1] we are not sure if that will work fine for the end use case, pVM to trusty. It looks like the coco implementation will always end up using dma encrypt/decrypt when a pVM is running and share the memory with host, even when all we want to do is share with trusty.
We use coco to share pages from protected guest -> host (from the swiotlb region). This is to establish a channel with the host via bounce buffering and the virtio-pci communication is done through this window.
Now, I don't understand why do you need a special pool to allocate memory, what allocation requirements do you have ? Performance is impacted even if you use another pool of memory because you have to page in from host first.
Why can't you page_alloc & ffa_share with Trusty on demand ?
Is that understanding correct ? We would also want to establish virtio-pci (existing tech) based communication between pVM and host, which should use mem decrypt path (?).
I am not sure if we need contiguous PA here, contiguous IPA should be sufficient, Armelle?
I think this is super important to know what we need because having PA contiguous space in a guest is a whole new challenge by itself. I've been talking to Quentin on this and this would be more approachable (from an u-API perspective for the VMM) for pKVM when we switch over to memfd.
We are also looking for further suggestions to improve the design, as my understanding of memory mapping, dma hal etc. is limited, and maybe there are better ways to do this.
-- Viresh
Thanks, Sebastian
[1] https://docs.google.com/document/d/1KyxclKngQ0MShX8Q1YPstZQVIDIctkxfj6lQ0V3R... [2] https://web.git.kernel.org/pub/scm/linux/kernel/git/vireshk/linux.git/tree/d...
On 09-03-25, 21:59, Sebastian Ene wrote:
On Fri, Mar 07, 2025 at 11:59:56AM +0530, Viresh Kumar wrote:
AFAIK, the broad idea is to implement two virtio communication paths from pVM, one to the Linux host (via virtio-pci) and another one to Trusty (via virtio-msg-ffa).
We already have from pVM to Host (not over FF-A but with pKVM APIs). What is new here/what do we want to do ?
Yes, that's what we want to use for pVM to Host. The problem is in pVM to Trusty.
Now looking at "dynamic mapping" section in [1] we are not sure if that will work fine for the end use case, pVM to trusty. It looks like the coco implementation will always end up using dma encrypt/decrypt when a pVM is running and share the memory with host, even when all we want to do is share with trusty.
We use coco to share pages from protected guest -> host (from the swiotlb region). This is to establish a channel with the host via bounce buffering and the virtio-pci communication is done through this window.
Right. We want to use that for pVM to Host communication.
Now, I don't understand why do you need a special pool to allocate memory, what allocation requirements do you have ? Performance is impacted even if you use another pool of memory because you have to page in from host first.
What do you mean by "page in from host first" ?
I think there is some confusion here. The pool we are talking about is from pVM's address space. Basically, we are adding a "reserved-memory" region in Guest's DT, so guest can set aside some space for us and we will use that for bounce-buffering too via swiotlb.
Why can't you page_alloc & ffa_share with Trusty on demand ?
We have tested that already and it works. With reserved-mem thing we just want to pre-map the entire region for performance reasons.
Even if we leave that point aside, we will have the same problem with page_alloc and ffa_share. Won't the coco driver share that memory with the host too with dma decrypt callback? I haven't tested it yet (don't have a pVM setup), but it looked like that the dma decrypt callback will get called anyway and that's what we are looking to avoid here.
I am not sure if we need contiguous PA here, contiguous IPA should be sufficient, Armelle?
I think this is super important to know what we need because having PA contiguous space in a guest is a whole new challenge by itself. I've been talking to Quentin on this and this would be more approachable (from an u-API perspective for the VMM) for pKVM when we switch over to memfd.
I don't see why we would need PA continuous space, but I would leave that for Armelle to answer.
Armelle ?
Ping.
On 10-03-25, 10:36, Viresh Kumar wrote:
On 09-03-25, 21:59, Sebastian Ene wrote:
On Fri, Mar 07, 2025 at 11:59:56AM +0530, Viresh Kumar wrote:
AFAIK, the broad idea is to implement two virtio communication paths from pVM, one to the Linux host (via virtio-pci) and another one to Trusty (via virtio-msg-ffa).
We already have from pVM to Host (not over FF-A but with pKVM APIs). What is new here/what do we want to do ?
Yes, that's what we want to use for pVM to Host. The problem is in pVM to Trusty.
Now looking at "dynamic mapping" section in [1] we are not sure if that will work fine for the end use case, pVM to trusty. It looks like the coco implementation will always end up using dma encrypt/decrypt when a pVM is running and share the memory with host, even when all we want to do is share with trusty.
We use coco to share pages from protected guest -> host (from the swiotlb region). This is to establish a channel with the host via bounce buffering and the virtio-pci communication is done through this window.
Right. We want to use that for pVM to Host communication.
Now, I don't understand why do you need a special pool to allocate memory, what allocation requirements do you have ? Performance is impacted even if you use another pool of memory because you have to page in from host first.
What do you mean by "page in from host first" ?
I think there is some confusion here. The pool we are talking about is from pVM's address space. Basically, we are adding a "reserved-memory" region in Guest's DT, so guest can set aside some space for us and we will use that for bounce-buffering too via swiotlb.
Why can't you page_alloc & ffa_share with Trusty on demand ?
We have tested that already and it works. With reserved-mem thing we just want to pre-map the entire region for performance reasons.
Even if we leave that point aside, we will have the same problem with page_alloc and ffa_share. Won't the coco driver share that memory with the host too with dma decrypt callback? I haven't tested it yet (don't have a pVM setup), but it looked like that the dma decrypt callback will get called anyway and that's what we are looking to avoid here.
I am not sure if we need contiguous PA here, contiguous IPA should be sufficient, Armelle?
I think this is super important to know what we need because having PA contiguous space in a guest is a whole new challenge by itself. I've been talking to Quentin on this and this would be more approachable (from an u-API perspective for the VMM) for pKVM when we switch over to memfd.
I don't see why we would need PA continuous space, but I would leave that for Armelle to answer.
Armelle ?
-- viresh
wow you all were waiting for my answer?! I am not sure it was worth waiting one month for this answer, Viresh next time, please ping me on chat. The problem statement does not seem to be understood by all parties.
The goal we are seeking to achieve is for microdroid pVM to share virtqueues (and io buffers) with two isolated devices, the host and the Trutzone TEE. By isolated, we mean that the virtqueues shared between the pVM and TZ TEE shall not be shared with the Host.
However given: 1/ the linux kernel assumes devices using dma transfers don't support encrypted memory 2/ by default all dma transfers will trigger the swioltb bounce buffering to a decrypted buffer 3/ microdroid uses the coco driver for memory encryption / decryption control
That implies that any dma map from any virtio device driver (the one interacting with the host, as well from the one interacting with the TEE) will lead to using a "decrypted" bounce buffer. Given that the coco driver is such that all decrypted bounce buffers are shared with the host, the TEE virtqueues would also be shared with the host. This is what we'd like to avoid. There is as you know no restrictions for the TEE to use CMA, but this is besides the point.
Hope that clarifies, I'll set up a meeting so we can make progress. Thanks, Armellle
On Tue, Mar 18, 2025 at 3:52 AM Viresh Kumar viresh.kumar@linaro.org wrote:
Ping.
On 10-03-25, 10:36, Viresh Kumar wrote:
On 09-03-25, 21:59, Sebastian Ene wrote:
On Fri, Mar 07, 2025 at 11:59:56AM +0530, Viresh Kumar wrote:
AFAIK, the broad idea is to implement two virtio communication paths
from pVM,
one to the Linux host (via virtio-pci) and another one to Trusty (via virtio-msg-ffa).
We already have from pVM to Host (not over FF-A but with pKVM APIs). What is new here/what do we want to do ?
Yes, that's what we want to use for pVM to Host. The problem is in pVM to Trusty.
Now looking at "dynamic mapping" section in [1] we are not sure if
that will
work fine for the end use case, pVM to trusty. It looks like the coco implementation will always end up using dma encrypt/decrypt when a
pVM is
running and share the memory with host, even when all we want to do
is share
with trusty.
We use coco to share pages from protected guest -> host (from the
swiotlb region). This is
to establish a channel with the host via bounce buffering and the
virtio-pci communication
is done through this window.
Right. We want to use that for pVM to Host communication.
Now, I don't understand why do you need a special pool to allocate
memory, what allocation
requirements do you have ? Performance is impacted even if you use
another pool of memory
because you have to page in from host first.
What do you mean by "page in from host first" ?
I think there is some confusion here. The pool we are talking about is from pVM's address space. Basically, we are adding a "reserved-memory" region in Guest's DT, so guest can set aside some space for us and we will use that for bounce-buffering too via swiotlb.
Why can't you page_alloc & ffa_share with Trusty on demand ?
We have tested that already and it works. With reserved-mem thing we just want to pre-map the entire region for performance reasons.
Even if we leave that point aside, we will have the same problem with page_alloc and ffa_share. Won't the coco driver share that memory with the host too with dma decrypt callback? I haven't tested it yet (don't have a pVM setup), but it looked like that the dma decrypt callback will get called anyway and that's what we are looking to avoid here.
I am not sure if we need contiguous PA here, contiguous IPA should be sufficient, Armelle?
I think this is super important to know what we need because having PA
contiguous space in a
guest is a whole new challenge by itself. I've been talking to Quentin
on this
and this would be more approachable (from an u-API perspective for the
VMM) for pKVM when we switch over to memfd.
I don't see why we would need PA continuous space, but I would leave that for Armelle to answer.
Armelle ?
-- viresh
-- viresh
On Wed, Apr 23, 2025 at 11:12:53PM -0700, Armelle Laine wrote:
Hello Armelle, folks
wow you all were waiting for my answer?! I am not sure it was worth waiting one month for this answer, Viresh next time, please ping me on chat. The problem statement does not seem to be understood by all parties.
The goal we are seeking to achieve is for microdroid pVM to share virtqueues (and io buffers) with two isolated devices, the host and the Trutzone TEE. By isolated, we mean that the virtqueues shared between the pVM and TZ TEE shall not be shared with the Host.
However given: 1/ the linux kernel assumes devices using dma transfers don't support encrypted memory 2/ by default all dma transfers will trigger the swioltb bounce buffering to a decrypted buffer 3/ microdroid uses the coco driver for memory encryption / decryption control
That implies that any dma map from any virtio device driver (the one interacting with the host, as well from the one interacting with the TEE) will lead to using a "decrypted" bounce buffer. Given that the coco driver is such that all decrypted bounce buffers are shared with the host, the TEE virtqueues would also be shared with the host.
Is it because under the hood the virtio_pci_legacy_probe ends up calling ioremap ? How would you make the distinction in the virtio driver code between the device exposed by the host and the one device exposed by Trustzone ?
This is what we'd like to avoid.
Why can't you write a separate virtio_ffa.c that makes use of the arm ff-a driver and uses the sharing infrastructure exposed by it and avoid going on the ioremap path ?
There is as you know no restrictions for the TEE to use CMA, but this is besides the point.
Hope that clarifies,
Thanks for the clarification, looking forward to understanding the complete picture.
I'll set up a meeting so we can make progress. Thanks, Armellle
Thanks, Sebastian
On Tue, Mar 18, 2025 at 3:52 AM Viresh Kumar viresh.kumar@linaro.org wrote:
Ping.
On 10-03-25, 10:36, Viresh Kumar wrote:
On 09-03-25, 21:59, Sebastian Ene wrote:
On Fri, Mar 07, 2025 at 11:59:56AM +0530, Viresh Kumar wrote:
AFAIK, the broad idea is to implement two virtio communication paths
from pVM,
one to the Linux host (via virtio-pci) and another one to Trusty (via virtio-msg-ffa).
We already have from pVM to Host (not over FF-A but with pKVM APIs). What is new here/what do we want to do ?
Yes, that's what we want to use for pVM to Host. The problem is in pVM to Trusty.
Now looking at "dynamic mapping" section in [1] we are not sure if
that will
work fine for the end use case, pVM to trusty. It looks like the coco implementation will always end up using dma encrypt/decrypt when a
pVM is
running and share the memory with host, even when all we want to do
is share
with trusty.
We use coco to share pages from protected guest -> host (from the
swiotlb region). This is
to establish a channel with the host via bounce buffering and the
virtio-pci communication
is done through this window.
Right. We want to use that for pVM to Host communication.
Now, I don't understand why do you need a special pool to allocate
memory, what allocation
requirements do you have ? Performance is impacted even if you use
another pool of memory
because you have to page in from host first.
What do you mean by "page in from host first" ?
I think there is some confusion here. The pool we are talking about is from pVM's address space. Basically, we are adding a "reserved-memory" region in Guest's DT, so guest can set aside some space for us and we will use that for bounce-buffering too via swiotlb.
Why can't you page_alloc & ffa_share with Trusty on demand ?
We have tested that already and it works. With reserved-mem thing we just want to pre-map the entire region for performance reasons.
Even if we leave that point aside, we will have the same problem with page_alloc and ffa_share. Won't the coco driver share that memory with the host too with dma decrypt callback? I haven't tested it yet (don't have a pVM setup), but it looked like that the dma decrypt callback will get called anyway and that's what we are looking to avoid here.
I am not sure if we need contiguous PA here, contiguous IPA should be sufficient, Armelle?
I think this is super important to know what we need because having PA
contiguous space in a
guest is a whole new challenge by itself. I've been talking to Quentin
on this
and this would be more approachable (from an u-API perspective for the
VMM) for pKVM when we switch over to memfd.
I don't see why we would need PA continuous space, but I would leave that for Armelle to answer.
Armelle ?
-- viresh
-- viresh
Hey Sebastian,
Please see some more answers.
On Mon, Apr 28, 2025 at 3:50 AM Sebastian Ene sebastianene@google.com wrote:
On Wed, Apr 23, 2025 at 11:12:53PM -0700, Armelle Laine wrote:
Hello Armelle, folks
wow you all were waiting for my answer?! I am not sure it was worth waiting one month for this answer, Viresh next time, please ping me on chat. The problem statement does not seem to be understood by all parties.
The goal we are seeking to achieve is for microdroid pVM to share virtqueues (and io buffers) with two isolated devices, the host and the Trutzone TEE. By isolated, we mean that the virtqueues shared between the pVM and TZ
TEE
shall not be shared with the Host.
However given: 1/ the linux kernel assumes devices using dma transfers don't support encrypted memory 2/ by default all dma transfers will trigger the swioltb bounce buffering to a decrypted buffer 3/ microdroid uses the coco driver for memory encryption / decryption control
That implies that any dma map from any virtio device driver (the one interacting with the host, as well from the one interacting with the TEE) will lead to using a "decrypted" bounce buffer. Given that the coco driver is such that all decrypted bounce buffers are shared with the
host,
the TEE virtqueues would also be shared with the host.
Is it because under the hood the virtio_pci_legacy_probe ends up calling ioremap ?
IIUC that's because these memory regions are allocated via dma_dal, nothing to do with virtio. One a memory region is tied to a DMA transfer, the kernel would create a "decrypted" bounce buffer when it assumes that the device does not support memory encryption (which is typically the default).
How would you make the distinction in the virtio driver code between the device exposed by the host and the one device exposed by Trustzone ?
The initial idea was to use different memory regions hooked to different dma hal. However, the approach taken by pKVM to hook the share abi on the memory decryption handler is not easily disabled by region or device. Hence our email to seek for a solution.
This is what we'd like to avoid.
Why can't you write a separate virtio_ffa.c that makes use of the arm ff-a driver and uses the sharing infrastructure exposed by it and avoid going on the ioremap path ?
The integration of virtio with ffa has been done relying on the kernel's
DMA infrastructure. See what Viresh described:
*"We have implemented FFA specific dma-hal [2] to perform FFA memory* *sharing with trusty. With "reserved-mem" and "memory-region" DT entries (not* *sure if that is the final solution), we are able to allocate memory the FFA* *device (which represents bus for all the enumerated devices between* *trusty/host). This memory is shared with trusty at probe time (from* *virtio-msg-ffa layer) and the DMA hal later allocates memory from there for* *coherent allocations and bounce buffers. This works just fine right now. "*
The issue is that the pKVM memory decryption handler will trigger on all DMA transfers (because the kernel assumes devices do not support memory encryption by default).
There is as you know no restrictions for the TEE to use CMA, but this is
besides the point.
Hope that clarifies,
Thanks for the clarification, looking forward to understanding the complete picture.
I'll set up a meeting so we can make progress. Thanks, Armellle
Thanks, Sebastian
On Tue, Mar 18, 2025 at 3:52 AM Viresh Kumar viresh.kumar@linaro.org wrote:
Ping.
On 10-03-25, 10:36, Viresh Kumar wrote:
On 09-03-25, 21:59, Sebastian Ene wrote:
On Fri, Mar 07, 2025 at 11:59:56AM +0530, Viresh Kumar wrote:
AFAIK, the broad idea is to implement two virtio communication
paths
from pVM,
one to the Linux host (via virtio-pci) and another one to Trusty
(via
virtio-msg-ffa).
We already have from pVM to Host (not over FF-A but with pKVM
APIs).
What is new here/what do we want to do ?
Yes, that's what we want to use for pVM to Host. The problem is in
pVM
to Trusty.
Now looking at "dynamic mapping" section in [1] we are not sure
if
that will
work fine for the end use case, pVM to trusty. It looks like the
coco
implementation will always end up using dma encrypt/decrypt when
a
pVM is
running and share the memory with host, even when all we want to
do
is share
with trusty.
We use coco to share pages from protected guest -> host (from the
swiotlb region). This is
to establish a channel with the host via bounce buffering and the
virtio-pci communication
is done through this window.
Right. We want to use that for pVM to Host communication.
Now, I don't understand why do you need a special pool to allocate
memory, what allocation
requirements do you have ? Performance is impacted even if you use
another pool of memory
because you have to page in from host first.
What do you mean by "page in from host first" ?
I think there is some confusion here. The pool we are talking about
is
from pVM's address space. Basically, we are adding a
"reserved-memory"
region in Guest's DT, so guest can set aside some space for us and we will use that for bounce-buffering too via swiotlb.
Why can't you page_alloc & ffa_share with Trusty on demand ?
We have tested that already and it works. With reserved-mem thing we just want to pre-map the entire region for performance reasons.
Even if we leave that point aside, we will have the same problem with page_alloc and ffa_share. Won't the coco driver share that memory
with
the host too with dma decrypt callback? I haven't tested it yet
(don't
have a pVM setup), but it looked like that the dma decrypt callback will get called anyway and that's what we are looking to avoid here.
I am not sure if we need contiguous PA here, contiguous IPA
should be
sufficient, Armelle?
I think this is super important to know what we need because
having PA
contiguous space in a
guest is a whole new challenge by itself. I've been talking to
Quentin
on this
and this would be more approachable (from an u-API perspective for
the
VMM) for pKVM when we switch over to memfd.
I don't see why we would need PA continuous space, but I would leave that for Armelle to answer.
Armelle ?
-- viresh
-- viresh
On 28-04-25, 07:13, Armelle Laine wrote:
The initial idea was to use different memory regions hooked to different dma hal. However, the approach taken by pKVM to hook the share abi on the memory decryption handler is not easily disabled by region or device. Hence our email to seek for a solution.
The issue is that the pKVM memory decryption handler will trigger on all DMA transfers (because the kernel assumes devices do not support memory encryption by default).
Sebastian,
One thing I wanted to clarify is that we haven't tested this path yet as we don't have a pVM to Trusty setup for now. All we have for now is host to Trusty, which doesn't hit this path.
But looking at the code, dma_direct_alloc() calls dma_set_decrypted() for calls made from the pVM kernel. It looks like this will get called for both transfer types, pVM to host and pVM to Trusty.
The virtio layer, for example, calls dma_alloc_coherent() for virtqueue allocation, and that should lead us here for both virtio-msg and virtio-pci code-paths.
I may be wrong (as this isn't tested for now), but we thought you guys might have already thought about such cases when the code was written and so thought of asking in advance.
On Tue, Apr 29, 2025 at 12:39:34PM +0530, Viresh Kumar wrote:
On 28-04-25, 07:13, Armelle Laine wrote:
The initial idea was to use different memory regions hooked to different dma hal. However, the approach taken by pKVM to hook the share abi on the memory decryption handler is not easily disabled by region or device. Hence our email to seek for a solution.
The issue is that the pKVM memory decryption handler will trigger on all DMA transfers (because the kernel assumes devices do not support memory encryption by default).
Sebastian,
One thing I wanted to clarify is that we haven't tested this path yet as we don't have a pVM to Trusty setup for now. All we have for now is host to Trusty, which doesn't hit this path.
But looking at the code, dma_direct_alloc() calls dma_set_decrypted() for calls made from the pVM kernel. It looks like this will get called for both transfer types, pVM to host and pVM to Trusty.
Hello Viresh,
Looking at virtio_msg_dma_alloc you use dma_direct_alloc which in turn calls __dma_direct_alloc_pages which in turn calls dma_direct_alloc_swiotlb. I assume all the memory has already been shared with the host for this swiotlb by a call to set_memory_decrypted from the swiotlb init code.
Am I following it correctly ?
Why did we choose to use the swiotlb reserved memory region to bake this allocations from ? Can't we just alloc_page() from the implemented virtio_msg_ffa_dma_ops layer ?
The virtio layer, for example, calls dma_alloc_coherent() for virtqueue allocation, and that should lead us here for both virtio-msg and virtio-pci code-paths.
I may be wrong (as this isn't tested for now), but we thought you guys might have already thought about such cases when the code was written and so thought of asking in advance.
-- viresh
Thanks, Sebastian
On 29-04-25, 16:55, Sebastian Ene wrote:
Looking at virtio_msg_dma_alloc you use dma_direct_alloc which in turn calls __dma_direct_alloc_pages which in turn calls dma_direct_alloc_swiotlb. I assume all the memory has already been shared with the host for this swiotlb by a call to set_memory_decrypted from the swiotlb init code.
Am I following it correctly ?
Why did we choose to use the swiotlb reserved memory region to bake this allocations from ? Can't we just alloc_page() from the implemented virtio_msg_ffa_dma_ops layer ?
Lets try to simplify this a bit.
Yes we are going to likely use "reserved-memory" regions eventually along with swiotlbs (for bounce buffers) to avoid mapping memory on the fly, but that can be ignored for now.
Lets say we do on-demand mapping only, no advance mapping for buffers.
I earlier thought that the call chain would be like this:
vring_alloc_queue->dma_alloc_coherent->dma_direct_alloc->dma_set_decrypted
This will likely be the call path for pVM to Host transfers (with virtio-pci).
I thought the same may end up getting triggered for pVM to Trusty too, but now I wonder if it will go to dma_direct_alloc() or not ? With our own DMA HAL for virtio-msg-ffa, we may end up following ops->alloc() path instead. If dma_direct_alloc() isn't called in our case, we may not have the problem in pVM to Trusty case.
It would be more clear once we have a setup to check this :(
On Wed, Apr 30, 2025 at 10:31:19AM +0530, Viresh Kumar wrote:
On 29-04-25, 16:55, Sebastian Ene wrote:
Looking at virtio_msg_dma_alloc you use dma_direct_alloc which in turn calls __dma_direct_alloc_pages which in turn calls dma_direct_alloc_swiotlb. I assume all the memory has already been shared with the host for this swiotlb by a call to set_memory_decrypted from the swiotlb init code.
Am I following it correctly ?
Why did we choose to use the swiotlb reserved memory region to bake this allocations from ? Can't we just alloc_page() from the implemented virtio_msg_ffa_dma_ops layer ?
Lets try to simplify this a bit.
Yes we are going to likely use "reserved-memory" regions eventually along with swiotlbs (for bounce buffers) to avoid mapping memory on the fly, but that can be ignored for now.
Lets say we do on-demand mapping only, no advance mapping for buffers.
I earlier thought that the call chain would be like this:
vring_alloc_queue->dma_alloc_coherent->dma_direct_alloc->dma_set_decrypted
This will likely be the call path for pVM to Host transfers (with virtio-pci).
Looking at vring_alloc_queue, if VIRTIO_F_ACCESS_PLATFORM is not set in we don't end up on the dma_alloc APIs and we avoid the set_memory_decrypted path. Instead pages will be allocated with alloc_pages_exact which is what I suggested earlier.
The feature set seems to be returned by the device in response to the VIRTIO_MSG_GET_FEATURES call.
I thought the same may end up getting triggered for pVM to Trusty too, but now I wonder if it will go to dma_direct_alloc() or not ? With our own DMA HAL for virtio-msg-ffa, we may end up following ops->alloc() path instead. If dma_direct_alloc() isn't called in our case, we may not have the problem in pVM to Trusty case.
It would be more clear once we have a setup to check this :(
We have a qemu based setup with Trusty and I landed patches in android16-6.12 to allow a pVM guest to talk to TZ. Maybe Armelle can share this with you if you want to give it a spin ?
-- viresh
Hope this helps, thanks Sebastian
On 30-04-25, 10:54, Sebastian Ene wrote:
Looking at vring_alloc_queue, if VIRTIO_F_ACCESS_PLATFORM is not set in we don't end up on the dma_alloc APIs and we avoid the set_memory_decrypted path. Instead pages will be allocated with alloc_pages_exact which is what I suggested earlier.
The feature set seems to be returned by the device in response to the VIRTIO_MSG_GET_FEATURES call.
The VIRTIO_F_ACCESS_PLATFORM feature is enabled in our case. And we need to use the dma_alloc APIs, as we want our virtio-msg specific DMA HAL to take care of memory mapping over FFA. We do need dma_alloc_coherent() to be called here.
We have a qemu based setup with Trusty and I landed patches in android16-6.12 to allow a pVM guest to talk to TZ. Maybe Armelle can share this with you if you want to give it a spin ?
I won't be able to give it a try before June (vacations, travel, etc.), but if that is available maybe someone else can get it to work with Trusty. Armelle ?
On Wed, Apr 30, 2025 at 04:30:26PM +0530, Viresh Kumar wrote:
On 30-04-25, 10:54, Sebastian Ene wrote:
Looking at vring_alloc_queue, if VIRTIO_F_ACCESS_PLATFORM is not set in we don't end up on the dma_alloc APIs and we avoid the set_memory_decrypted path. Instead pages will be allocated with alloc_pages_exact which is what I suggested earlier.
The feature set seems to be returned by the device in response to the VIRTIO_MSG_GET_FEATURES call.
The VIRTIO_F_ACCESS_PLATFORM feature is enabled in our case. And we need to use the dma_alloc APIs, as we want our virtio-msg specific DMA HAL to take care of memory mapping over FFA. We do need dma_alloc_coherent() to be called here.
Right I see, in that case I wonder if it makese sense to have a new property in the DT for the reserved-memory region which informs the swiotlb driver to not call `set_memory_decrypted` upon region initialization. By doing this we don't end up with the memory shared with the host and you can continue using the DMA HAL which takes care of the FF-A mapping.
We have a qemu based setup with Trusty and I landed patches in android16-6.12 to allow a pVM guest to talk to TZ. Maybe Armelle can share this with you if you want to give it a spin ?
I won't be able to give it a try before June (vacations, travel, etc.), but if that is available maybe someone else can get it to work with Trusty. Armelle ?
-- viresh
Sebastian
Hi all,
On Wed, Apr 30, 2025 at 4:40 PM Sebastian Ene sebastianene@google.com wrote:
On Wed, Apr 30, 2025 at 04:30:26PM +0530, Viresh Kumar wrote:
On 30-04-25, 10:54, Sebastian Ene wrote:
Looking at vring_alloc_queue, if VIRTIO_F_ACCESS_PLATFORM is not set in we don't end up on the dma_alloc APIs and we avoid the set_memory_decrypted path. Instead pages will be allocated with alloc_pages_exact which is what I suggested earlier.
The feature set seems to be returned by the device in response to the VIRTIO_MSG_GET_FEATURES call.
The VIRTIO_F_ACCESS_PLATFORM feature is enabled in our case. And we need to use the dma_alloc APIs, as we want our virtio-msg specific DMA HAL to take care of memory mapping over FFA. We do need dma_alloc_coherent() to be called here.
Right I see, in that case I wonder if it makese sense to have a new property in the DT for the reserved-memory region which informs the swiotlb driver to not call `set_memory_decrypted` upon region initialization. By doing this we don't end up with the memory shared with the host and you can continue using the DMA HAL which takes care of the FF-A mapping.
Not my area of expertise, but by a quick eyeball at the code it seems that there is already per-device logic to skip the set_memory_decrypted() calls and such. Specifically, force_dma_unencrypted(dev) does that, although it seems to only be plumbed with CC-A specific logic. Could that be extended for this use-case?
On Wed, 30 Apr 2025 at 20:16, Quentin Perret qperret@google.com wrote:
Not my area of expertise, but by a quick eyeball at the code it seems that there is already per-device logic to skip the set_memory_decrypted() calls and such. Specifically, force_dma_unencrypted(dev) does that, although it seems to only be plumbed with CC-A specific logic. Could that be extended for this use-case?
Right, I also saw that yesterday and it can be the perfect place to add the check. It only checks for realm-world right now IIRC, we can also check if DMA ops are registered for a device, and skip DMA encryption/decryption in that case.
I just wasn't sure if that's a hack or the right way to solve this :)
-- Viresh
-
Armelle Laine -
Quentin Perret -
Sebastian Ene -
Viresh Kumar