Hello,
I have some questions / feedback on the Xen side of virtio-msg.
AFAIU virtio-msg defines a protocol in order to deal with discovery (optional) and configuration of the PV devices. But things are undefined regarding what is a "memory address".
In Xen memory model with grants, each guest has its own memory space. The frontend shares memory pages with the backend through grants pages identified by grant references.
So in a design based on grants, virtio addresses can't (or at least shouldn't) be guest physical address; but needs to be something derived on grants. A earlier design [1] was forging a address with the grant reference, but I feel it's not great, as it forces "map+unmap" cycles for temporary buffers thus has the same performance problem as Xen PV drivers (without persistent grants) where map+unmap cycles is a performance problem.
And would make the address space very fragmented and in often limited to 4KB buffers.
One idea that is already used by Xen displif (PV Display) is to have a "gref directory" and describe the address space on that. The gref directory is roughly a array of grant references (shared pages infos) that could describe a address space starting at 0, where each page is defined by a grant reference of the directory. That way, the backend can freely keep all or a part or the address space persistently mapped (or eventually map it all at once); and the address space is also contiguous which would help with >4KB buffers.
Any thoughts ?
[1] https://static.sched.com/hosted_files/xen2021/bf/Thursday_2021-Xen-Summit-vi...
-- Teddy Astie | Vates XCP-ng Developer
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
Hi Teddy,
On 5 Jan 2026, at 14:09, Teddy Astie teddy.astie@vates.tech wrote:
Hello,
I have some questions / feedback on the Xen side of virtio-msg.
AFAIU virtio-msg defines a protocol in order to deal with discovery (optional) and configuration of the PV devices. But things are undefined regarding what is a "memory address".
In Xen memory model with grants, each guest has its own memory space. The frontend shares memory pages with the backend through grants pages identified by grant references.
So in a design based on grants, virtio addresses can't (or at least shouldn't) be guest physical address; but needs to be something derived on grants. A earlier design [1] was forging a address with the grant reference, but I feel it's not great, as it forces "map+unmap" cycles for temporary buffers thus has the same performance problem as Xen PV drivers (without persistent grants) where map+unmap cycles is a performance problem.
And would make the address space very fragmented and in often limited to 4KB buffers.
One idea that is already used by Xen displif (PV Display) is to have a "gref directory" and describe the address space on that. The gref directory is roughly a array of grant references (shared pages infos) that could describe a address space starting at 0, where each page is defined by a grant reference of the directory. That way, the backend can freely keep all or a part or the address space persistently mapped (or eventually map it all at once); and the address space is also contiguous which would help with >4KB buffers.
Any thoughts ?
I think those problematics can be solved using a gref system by the Xen virtio-msg bus driver. We do something equivalent to define a "bus address" in the virtio-msg over FF-A specification and in any case a Xen based bus will have to exchange information between the backend and the frontend in addition to what is exchanged with the messages defined by Virtio message. We do the same for the FF-A bus to detect devices or exchange information on shared area. In the xen case sharing will be for sure based on grant tables and a gref directory could be created in a page of memory shared between both sides to optimize things.
Do you think such principles would need more definition in the Virtio Message specification or perhaps more guidance on what could be done on those aspects by the bus ?
Here is a link to the current virtio message over FF-A specification if you want to have a look first: https://documentation-service.arm.com/static/68f647791134f773ab3f0a7c
Happy to discuss further or answer your questions if you have any.
Regards Bertrand
[1] https://static.sched.com/hosted_files/xen2021/bf/Thursday_2021-Xen-Summit-vi...
-- Teddy Astie | Vates XCP-ng Developer
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
Virtio-msg mailing list -- virtio-msg@lists.linaro.org To unsubscribe send an email to virtio-msg-leave@lists.linaro.org
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Le 05/01/2026 à 14:42, Bertrand Marquis a écrit :
Hi Teddy,
On 5 Jan 2026, at 14:09, Teddy Astie teddy.astie@vates.tech wrote:
Hello,
I have some questions / feedback on the Xen side of virtio-msg.
AFAIU virtio-msg defines a protocol in order to deal with discovery (optional) and configuration of the PV devices. But things are undefined regarding what is a "memory address".
In Xen memory model with grants, each guest has its own memory space. The frontend shares memory pages with the backend through grants pages identified by grant references.
So in a design based on grants, virtio addresses can't (or at least shouldn't) be guest physical address; but needs to be something derived on grants. A earlier design [1] was forging a address with the grant reference, but I feel it's not great, as it forces "map+unmap" cycles for temporary buffers thus has the same performance problem as Xen PV drivers (without persistent grants) where map+unmap cycles is a performance problem.
And would make the address space very fragmented and in often limited to 4KB buffers.
One idea that is already used by Xen displif (PV Display) is to have a "gref directory" and describe the address space on that. The gref directory is roughly a array of grant references (shared pages infos) that could describe a address space starting at 0, where each page is defined by a grant reference of the directory. That way, the backend can freely keep all or a part or the address space persistently mapped (or eventually map it all at once); and the address space is also contiguous which would help with >4KB buffers.
Any thoughts ?
I think those problematics can be solved using a gref system by the Xen virtio-msg bus driver. We do something equivalent to define a "bus address" in the virtio-msg over FF-A specification and in any case a Xen based bus will have to exchange information between the backend and the frontend in addition to what is exchanged with the messages defined by Virtio message. We do the same for the FF-A bus to detect devices or exchange information on shared area. In the xen case sharing will be for sure based on grant tables and a gref directory could be created in a page of memory shared between both sides to optimize things.
On Xen, a "grant directory" could be either a way to describe a large shared area. But that doesn't really answer whether the grant reference number or something else describes what would be a "bus address" in Xen.
Do you think such principles would need more definition in the Virtio Message specification or perhaps more guidance on what could be done on those aspects by the bus ?
Here is a link to the current virtio message over FF-A specification
if you want to have a look first:
https://documentation-service.arm.com/static/68f647791134f773ab3f0a7c
Happy to discuss further or answer your questions if you have any.
One of the issues I have in mind and that we encounter in current Xen PV devices is that some shared buffers are often short-lived, and a lot of time is taken in mapping and unmapping those buffers.
For instance, if the frontend passes a new bus address for some grant, the backend would need to explicitly map it; unless it has already been mapped.
One solution is to reuse the buffers, but it wants to be structured. The virtio-msg FF-A appears to define explicit share/unshare operations. I guess the frontend is expected to not abuse it and to try to reuse already shared bus addresses if possible (i.e avoid short-lived buffers) ?
Regards Bertrand
[1] https://static.sched.com/hosted_files/xen2021/bf/Thursday_2021-Xen-Summit-vi...
-- Teddy Astie | Vates XCP-ng Developer
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
Virtio-msg mailing list -- virtio-msg@lists.linaro.org To unsubscribe send an email to virtio-msg-leave@lists.linaro.org
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. _______________________________________________ Virtio-msg mailing list -- virtio-msg@lists.linaro.org To unsubscribe send an email to virtio-msg-leave@lists.linaro.org
-- Teddy Astie | Vates XCP-ng Developer
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
Hi Teddy,
On 5 Jan 2026, at 16:43, Teddy Astie teddy.astie@vates.tech wrote:
Le 05/01/2026 à 14:42, Bertrand Marquis a écrit :
Hi Teddy,
On 5 Jan 2026, at 14:09, Teddy Astie teddy.astie@vates.tech wrote:
Hello,
I have some questions / feedback on the Xen side of virtio-msg.
AFAIU virtio-msg defines a protocol in order to deal with discovery (optional) and configuration of the PV devices. But things are undefined regarding what is a "memory address".
In Xen memory model with grants, each guest has its own memory space. The frontend shares memory pages with the backend through grants pages identified by grant references.
So in a design based on grants, virtio addresses can't (or at least shouldn't) be guest physical address; but needs to be something derived on grants. A earlier design [1] was forging a address with the grant reference, but I feel it's not great, as it forces "map+unmap" cycles for temporary buffers thus has the same performance problem as Xen PV drivers (without persistent grants) where map+unmap cycles is a performance problem.
And would make the address space very fragmented and in often limited to 4KB buffers.
One idea that is already used by Xen displif (PV Display) is to have a "gref directory" and describe the address space on that. The gref directory is roughly a array of grant references (shared pages infos) that could describe a address space starting at 0, where each page is defined by a grant reference of the directory. That way, the backend can freely keep all or a part or the address space persistently mapped (or eventually map it all at once); and the address space is also contiguous which would help with >4KB buffers.
Any thoughts ?
I think those problematics can be solved using a gref system by the Xen virtio-msg bus driver. We do something equivalent to define a "bus address" in the virtio-msg over FF-A specification and in any case a Xen based bus will have to exchange information between the backend and the frontend in addition to what is exchanged with the messages defined by Virtio message. We do the same for the FF-A bus to detect devices or exchange information on shared area. In the xen case sharing will be for sure based on grant tables and a gref directory could be created in a page of memory shared between both sides to optimize things.
On Xen, a "grant directory" could be either a way to describe a large shared area. But that doesn't really answer whether the grant reference number or something else describes what would be a "bus address" in Xen.
A bus address should be a grant reference + offset in page.
Do you think such principles would need more definition in the Virtio Message specification or perhaps more guidance on what could be done on those aspects by the bus ?
Here is a link to the current virtio message over FF-A specification
if you want to have a look first:
https://documentation-service.arm.com/static/68f647791134f773ab3f0a7c
Happy to discuss further or answer your questions if you have any.
One of the issues I have in mind and that we encounter in current Xen PV devices is that some shared buffers are often short-lived, and a lot of time is taken in mapping and unmapping those buffers.
For instance, if the frontend passes a new bus address for some grant, the backend would need to explicitly map it; unless it has already been mapped.
One solution is to reuse the buffers, but it wants to be structured. The virtio-msg FF-A appears to define explicit share/unshare operations. I guess the frontend is expected to not abuse it and to try to reuse already shared bus addresses if possible (i.e avoid short-lived buffers) ?
Nothing is forcing the frontend to unshare something once it is used. For example the frontend is free to share a big area in FF-A and never unshare to prevent share/unshare (google plans to use something like that where they will share bigger areas instead of doing page by page for example).
There is definitely some room to define something more optimal to prevent frequent share/unshare and try to reuse previously shared areas when possible and I would be more than happy to discuss this to find a solution.
In the beginning of the design of virtio message over ff-a, i designed something working more or less like this for FF-A: - one share page (or bigger) would contain a table which an FF-A handle and information for each shared area - frontend just add entries there - backend pulls information and maps on demand when it needs - frontend can ask to release through a message an entry - backend can signal it does not need anything anymore so that it can be retrieved by the frontend
I think we could design something working in an equivalent way to optimize things and maybe flag entries as "unmap once done" or "keep until i tell you"
This would allow the frontend to flag entries that are oneshot from the entries that it can reuse (user space vs kernel alloc mainly) so that the dma_ops in the frontend could provide an alloc function that would optimize the reuse of previously shared buffers.
Both FF-A and Xen could benefit of such a system but this needs a bit more thinking.
Definitely this is a bus internal part, not belonging in the Virtio Message transport itself but such a system could help optimize the performance.
What do you think ?
Cheers Bertrand
Regards Bertrand
[1] https://static.sched.com/hosted_files/xen2021/bf/Thursday_2021-Xen-Summit-vi...
-- Teddy Astie | Vates XCP-ng Developer
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
Virtio-msg mailing list -- virtio-msg@lists.linaro.org To unsubscribe send an email to virtio-msg-leave@lists.linaro.org
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. _______________________________________________ Virtio-msg mailing list -- virtio-msg@lists.linaro.org To unsubscribe send an email to virtio-msg-leave@lists.linaro.org
-- Teddy Astie | Vates XCP-ng Developer
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Le 05/01/2026 à 16:56, Bertrand Marquis a écrit :
Hi Teddy,
On 5 Jan 2026, at 16:43, Teddy Astie teddy.astie@vates.tech wrote:
Le 05/01/2026 à 14:42, Bertrand Marquis a écrit :
Hi Teddy,
On 5 Jan 2026, at 14:09, Teddy Astie teddy.astie@vates.tech wrote:
Hello,
I have some questions / feedback on the Xen side of virtio-msg.
AFAIU virtio-msg defines a protocol in order to deal with discovery (optional) and configuration of the PV devices. But things are undefined regarding what is a "memory address".
In Xen memory model with grants, each guest has its own memory space. The frontend shares memory pages with the backend through grants pages identified by grant references.
So in a design based on grants, virtio addresses can't (or at least shouldn't) be guest physical address; but needs to be something derived on grants. A earlier design [1] was forging a address with the grant reference, but I feel it's not great, as it forces "map+unmap" cycles for temporary buffers thus has the same performance problem as Xen PV drivers (without persistent grants) where map+unmap cycles is a performance problem.
And would make the address space very fragmented and in often limited to 4KB buffers.
One idea that is already used by Xen displif (PV Display) is to have a "gref directory" and describe the address space on that. The gref directory is roughly a array of grant references (shared pages infos) that could describe a address space starting at 0, where each page is defined by a grant reference of the directory. That way, the backend can freely keep all or a part or the address space persistently mapped (or eventually map it all at once); and the address space is also contiguous which would help with >4KB buffers.
Any thoughts ?
I think those problematics can be solved using a gref system by the Xen virtio-msg bus driver. We do something equivalent to define a "bus address" in the virtio-msg over FF-A specification and in any case a Xen based bus will have to exchange information between the backend and the frontend in addition to what is exchanged with the messages defined by Virtio message. We do the same for the FF-A bus to detect devices or exchange information on shared area. In the xen case sharing will be for sure based on grant tables and a gref directory could be created in a page of memory shared between both sides to optimize things.
On Xen, a "grant directory" could be either a way to describe a large shared area. But that doesn't really answer whether the grant reference number or something else describes what would be a "bus address" in Xen.
A bus address should be a grant reference + offset in page.
Do you think such principles would need more definition in the Virtio Message specification or perhaps more guidance on what could be done on those aspects by the bus ?
Here is a link to the current virtio message over FF-A specification
if you want to have a look first:
https://documentation-service.arm.com/static/68f647791134f773ab3f0a7c
Happy to discuss further or answer your questions if you have any.
One of the issues I have in mind and that we encounter in current Xen PV devices is that some shared buffers are often short-lived, and a lot of time is taken in mapping and unmapping those buffers.
For instance, if the frontend passes a new bus address for some grant, the backend would need to explicitly map it; unless it has already been mapped.
One solution is to reuse the buffers, but it wants to be structured. The virtio-msg FF-A appears to define explicit share/unshare operations. I guess the frontend is expected to not abuse it and to try to reuse already shared bus addresses if possible (i.e avoid short-lived buffers) ?
Nothing is forcing the frontend to unshare something once it is used. For example the frontend is free to share a big area in FF-A and never unshare to prevent share/unshare (google plans to use something like that where they will share bigger areas instead of doing page by page for example).
There is definitely some room to define something more optimal to prevent frequent share/unshare and try to reuse previously shared areas when possible and I would be more than happy to discuss this to find a solution.
So as I understand it, it is mostly a frontend implementation side.
In the beginning of the design of virtio message over ff-a, i designed something working more or less like this for FF-A:
- one share page (or bigger) would contain a table which an FF-A handle and
information for each shared area
- frontend just add entries there
- backend pulls information and maps on demand when it needs
- frontend can ask to release through a message an entry
- backend can signal it does not need anything anymore so that it can
be retrieved by the frontend
I think we could design something working in an equivalent way to optimize things and maybe flag entries as "unmap once done" or "keep until i tell you"
This would allow the frontend to flag entries that are oneshot from the entries that it can reuse (user space vs kernel alloc mainly) so that the dma_ops in the frontend could provide an alloc function that would optimize the reuse of previously shared buffers.
Both FF-A and Xen could benefit of such a system but this needs a bit more thinking.
Definitely this is a bus internal part, not belonging in the Virtio Message transport itself but such a system could help optimize the performance.
What do you think ?
Looks good to me.
Cheers Bertrand
Regards Bertrand
[1] https://static.sched.com/hosted_files/xen2021/bf/Thursday_2021-Xen-Summit-vi...
-- Teddy Astie | Vates XCP-ng Developer
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
Virtio-msg mailing list -- virtio-msg@lists.linaro.org To unsubscribe send an email to virtio-msg-leave@lists.linaro.org
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. _______________________________________________ Virtio-msg mailing list -- virtio-msg@lists.linaro.org To unsubscribe send an email to virtio-msg-leave@lists.linaro.org
-- Teddy Astie | Vates XCP-ng Developer
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
-- Teddy Astie | Vates XCP-ng Developer
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
-
Bertrand Marquis -
Teddy Astie