Hello,
I am trying to get feedback / ideas (specially from the people in
--to) before I reach out to LKML.
Problem statement:
Share memory (dma-buf/dma-heap fd) over VSock (over virtio-msg-ffa).
Current status:
I have implemented the first version [1]/[2]. The system-heap memory
is successfully shared with OPTEE, test setup available here [3].
Workflow:
Userspace:
open("/dev/dma_heap/system", O_RDWR)
+
ioctl(heap_fd, DMA_HEAP_IOCTL_ALLOC, &alloc)
+
sendmsg(sock, &msg, 0)
// msg: SCM_VSOCK_SHMEM / VSOCK_SHMEM_TYPE_FFA / VSOCK_SHMEM_SUBOP_OFFER
Kernel:
vsock_sendmsg_shmem(vsk, msg)
|
vsk->transport->map_dma_buf(dmabuf)
|
virtio_transport_map_dma_buf(dmabuf)
|
dma_buf_attach(dmabuf, dev); // Vsock's underlying device - virtio-msg device
+
dma_buf_map_attachment_unlocked(dbuf->attach, DMA_BIDIRECTIONAL);
|
system_heap_map_dma_buf(attach, dir);
|
dma_map_sgtable(attachment->dev, table, dir, attrs=0);
|
virtio_msg_dma_map_sg_dev(dev, sgl, nents, dir, attrs); // FFA dma-ops
|
ffa_dev->ops->mem_ops->memory_share(&args);
+
vmfdev->send(vmfdev, request, response, &vmfdev->idata);
The memory is shared with FFA SHARE type. This doesn't require a
custom heap and works with any DMA heap/buf, system heap is just an
example here.
FFA supports LEND-ing the memory as well and Google needs it. And this
is where I am _blocked_ right now and need help.
We need to support both SHARE (both sides can access memory) and LEND
(Linux doesn't access it any further - unmapped) mechanisms and I am
not sure how to move forward.
- We can pass the share-type from userspace and that can reach up to
vsock / dma-buf (with minor API change to
dma_buf_map_attachment_unlocked()) / system-heap.
- But after that the standard dma_map_sgtable() call takes only one
configurable parameter, `attributes` (which is set to 0 for now by
system heap). There is nothing else which I can use to pass the flag
to FFA specific DMA OPs.
- I thought maybe I can create a new DMA attribute:
`DMA_ATTR_MEM_LEND`, which can be passed in the `attr` field and
that will work I guess.
- But I am not sure if it would be acceptable upstream to add a new
attribute for this.
- And so I am looking for some ideas.
--
viresh
[1] https://git.kernel.org/pub/scm/linux/kernel/git/vireshk/linux.git/log/?h=vi…
[2] Patches of interest
- vsock: Add shared-memory support for virtio and loopback
- virtio-msg: ffa: Add device DMA-OPS
[3] https://linaro.atlassian.net/wiki/spaces/HVAC/pages/30372528129
This series adds the virtio-msg transport layer.
The individuals and organizations involved in this effort have had difficulty in
using the existing virtio-transports in various situations and desire to add one
more transport that performs its transport layer operations by sending and
receiving messages.
Implementations of virtio-msg will normally be done in multiple layers:
* common / device level
* bus level
The common / device level defines the messages exchanged between the driver
and a device. This common part should lead to a common driver holding most
of the virtio specifics and can be shared by all virtio-msg bus implementations.
The kernel implementation in [3] shows this separation. As with other transport
layers, virtio-msg should not require modifications to existing virtio device
implementations (virtio-net, virtio-blk etc). The common / device level is the
main focus of this version of the patch series.
The virtio-msg bus level implements the normal things a bus defines
(enumeration, dma operations, etc) but also implements the message send and
receive operations. A number of bus implementations are envisioned,
some of which will be reusable and general purpose. Other bus implementations
might be unique to a given situation, for example only used by a PCIe card
and its driver.
How much of the bus level should be described in the virtio spec is one item
we wish to discuss. This draft takes a middle approach by describing the bus
level and defining some standard bus level messages that MAY be used by the bus.
It also describes a range of bus messages that are implementation dependent.
The standard bus messages are an effort to avoid different bus implementations
doing the same thing in different ways for no good reason. However the
different environments will require different things. Instead of trying to
anticipate all needs and provide something very abstract, we think
implementation specific messages will be needed at the bus level. Over time,
if we see similar messages across multiple bus implementations, we will move to
standardize a bus level message for that.
We are working on a few reusable bus implementations:
* virtio-msg-ffa based on Arm FF-A interface for use between:
* normal world and secure world
* host and VM or VM to VM
* Can be used w/ or with out a hypervisor
* Any Hypervisor that implements FF-A can be used
* We have this working with pKVM and Xen
* We have this working with Trusty and OP-TEE
* virtio-msg-amp for use between heterogenous systems
* The main processors and its co-processors on an AMP SOC
* Two or more systems connected via PCIe
* Minimal requirements: bi-directional interrupts and
at least one shared memory area
* hvac-demo has 2 demos of this
* This is working on two hardware platforms
* virtio-msg-loopback for userspace implemented devices
* Allows user space to provide devices to its own kernel
* This is similar to fuse, cuse or loopback block devices but for virtio
* hvac-demo has a demo
We also anticipate a few more:
* virtio-msg-xen specific to Xen
* Usable on any Xen system (including x86 where FF-A does not exist)
* Using Xen events and page grants
* virtio-msg over admin virtqueues
* This allows any virtio-pci device that supports admin virtqueues to also
support a virtio-msg bus that supports sub devices
* [We are looking for collaborators for this work]
Changes since RFC2:
Spec Functional:
* Made the common message header 8 bytes and added a token for optional use
when parallel outstanding requests are possible
* Made the 8 byte fields align to 8 byte offsets.
This effects the {SET,GET}_VQUEUE messages
* Many conformance cases have been tightened.
Spec Editorial:
* Major restructure to better align with virtio spec
* Conformance model now matches other transports
* Use C structures to define messages instead of tables
* Added a section to describe how responses are matched to requests
This includes the use of the new token field
* Redefined / better described error handling between transport and bus
layers to eliminate the need for the bus to generate fake response messages
* Included editorial feedback from RFC2
Eco-system:
* Added virtio-msg-loopback demo
* Arm has published the first draft of the virtio-msg over FFA spec [6]
* virtio-msg over FFA has been demonstrated with both Trusty and OP-TEE
secure world
* LKML RFC has been sent [7]
* QEMU RFC has been sent [8]
This is the first non-RFC patch series. The known short comings have been
addressed. We ask for review in earnest on this series and thank you for
any feedback you can provide.
Background info and work in progress implementations:
* HVAC project page with intro slides [1]
* HVAC demo repo w/ instructions in README.md [2]
* Kernel w/ virtio-msg common level and ffa support [3]
* QEMU w/ support for one form of virtio-msg-amp [4]
* Portable RTOS library w/ one form of virtio-msg-amp [5]
In addition to the QEMU system based demos in the hvac-demo repo, we also have
two hardware systems running:
* AMD x86 + AMD Arm Versal connected via PCIe
* ST STM32MP157 A7 Linux using virtio-i2c provided by M4 Zephyr
Please note that although the demos work, a few have not been aligned
with this version of the spec.
[1] https://linaro.atlassian.net/wiki/spaces/HVAC/overview
[2] https://github.com/wmamills/hvac-demo
[3] https://git.kernel.org/pub/scm/linux/kernel/git/vireshk/linux.git/log/?h=vi…
[4] https://github.com/edgarigl/qemu/commits/edgar/virtio-msg-rfc/
[5] https://github.com/arnopo/open-amp/commits/virtio-msg/
[6] https://documentation-service.arm.com/static/68f647791134f773ab3f0a7c
[7] https://lore.kernel.org/all/cover.1753865268.git.viresh.kumar@linaro.org/
[8] https://mail.gnu.org/archive/html/qemu-devel/2025-10/msg07438.html
Bertrand Marquis (2):
virtio-msg: add new command for bus normative
virtio-msg: add conformance entries in conformance chapter
Bill Mills (2):
virtio-msg: Add virtio-msg, a message based virtio transport layer
virtio-msg: link virtio-msg content
commands.tex | 3 +-
conformance.tex | 105 ++-
content.tex | 1 +
transport-msg.tex | 1640 +++++++++++++++++++++++++++++++++++++++++++++
4 files changed, 1746 insertions(+), 3 deletions(-)
create mode 100644 transport-msg.tex
--
2.34.1
This series adds the virtio-msg transport layer.
The individuals and organizations involved in this effort have had difficulty in
using the existing virtio-transports in various situations and desire to add one
more transport that performs its transport layer operations by sending and
receiving messages.
Implementations of virtio-msg will normally be done in multiple layers:
* common / device level
* bus level
The common / device level defines the messages exchanged between the driver
and a device. This common part should lead to a common driver holding most
of the virtio specifics and can be shared by all virtio-msg bus implementations.
The kernel implementation in [3] shows this separation. As with other transport
layers, virtio-msg should not require modifications to existing virtio device
implementations (virtio-net, virtio-blk etc). The common / device level is the
main focus of this version of the patch series.
The virtio-msg bus level implements the normal things a bus defines
(enumeration, dma operations, etc) but also implements the message send and
receive operations. A number of bus implementations are envisioned,
some of which will be reusable and general purpose. Other bus implementations
might be unique to a given situation, for example only used by a PCIe card
and its driver.
How much of the bus level should be described in the virtio spec is one item
we wish to discuss. This draft takes a middle approach by describing the bus
level and defining some standard bus level messages that MAY be used by the bus.
It also describes a range of bus messages that are implementation dependent.
The standard bus messages are an effort to avoid different bus implementations
doing the same thing in different ways for no good reason. However the
different environments will require different things. Instead of trying to
anticipate all needs and provide something very abstract, we think
implementation specific messages will be needed at the bus level. Over time,
if we see similar messages across multiple bus implementations, we will move to
standardize a bus level message for that.
We are working on a few reusable bus implementations:
* virtio-msg-ffa based on Arm FF-A interface for use between:
* normal world and secure world
* host and VM or VM to VM
* Can be used w/ or with out a hypervisor
* Any Hypervisor that implements FF-A can be used
* We have this working with pKVM and Xen
* We have this working with Trusty and OP-TEE
* virtio-msg-amp for use between heterogenous systems
* The main processors and its co-processors on an AMP SOC
* Two or more systems connected via PCIe
* Minimal requirements: bi-directional interrupts and
at least one shared memory area
* hvac-demo has 2 demos of this
* This is working on two hardware platforms
* virtio-msg-loopback for userspace implemented devices
* Allows user space to provide devices to its own kernel
* This is similar to fuse, cuse or loopback block devices but for virtio
* hvac-demo has a demo
We also anticipate a few more:
* virtio-msg-xen specific to Xen
* Usable on any Xen system (including x86 where FF-A does not exist)
* Using Xen events and page grants
* virtio-msg over admin virtqueues
* This allows any virtio-pci device that supports admin virtqueues to also
support a virtio-msg bus that supports sub devices
* [We are looking for collaborators for this work]
Changes since RFC2:
Spec Functional:
* Made the common message header 8 bytes and added a token for optional use
when parallel outstanding requests are possible
* Made the 8 byte fields align to 8 byte offsets.
This effects the {SET,GET}_VQUEUE messages
* Many conformance cases have been tightened.
Spec Editorial:
* Major restructure to better align with virtio spec
* Conformance model now matches other transports
* Use C structures to define messages instead of tables
* Added a section to describe how responses are matched to requests
This includes the use of the new token field
* Redefined / better described error handling between transport and bus
layers to eliminate the need for the bus to generate fake response messages
* Included editorial feedback from RFC2
Eco-system:
* Added virtio-msg-loopback demo
* Arm has published the first draft of the virtio-msg over FFA spec [6]
* virtio-msg over FFA has been demonstrated with both Trusty and OP-TEE
secure world
* LKML RFC has been sent [7]
* QEMU RFC has been sent [8]
This is the first non-RFC patch series. The known short comings have been
addressed. We ask for review in earnest on this series and thank you for
any feedback you can provide.
Background info and work in progress implementations:
* HVAC project page with intro slides [1]
* HVAC demo repo w/ instructions in README.md [2]
* Kernel w/ virtio-msg common level and ffa support [3]
* QEMU w/ support for one form of virtio-msg-amp [4]
* Portable RTOS library w/ one form of virtio-msg-amp [5]
In addition to the QEMU system based demos in the hvac-demo repo, we also have
two hardware systems running:
* AMD x86 + AMD Arm Versal connected via PCIe
* ST STM32MP157 A7 Linux using virtio-i2c provided by M4 Zephyr
Please note that although the demos work, a few have not been aligned
with this version of the spec.
[1] https://linaro.atlassian.net/wiki/spaces/HVAC/overview
[2] https://github.com/wmamills/hvac-demo
[3] https://git.kernel.org/pub/scm/linux/kernel/git/vireshk/linux.git/log/?h=vi…
[4] https://github.com/edgarigl/qemu/commits/edgar/virtio-msg-rfc/
[5] https://github.com/arnopo/open-amp/commits/virtio-msg/
[6] https://documentation-service.arm.com/static/68f647791134f773ab3f0a7c
[7] https://lore.kernel.org/all/cover.1753865268.git.viresh.kumar@linaro.org/
[8] https://mail.gnu.org/archive/html/qemu-devel/2025-10/msg07438.html
Alex Bennée (1):
local: add local build help and github automation (!UPSTREAM)
Bertrand Marquis (2):
virtio-msg: add new command for bus normative
virtio-msg: add conformance entries in conformance chapter
Bill Mills (5):
email: Add script and cover letter for RFC1
email: adjust cover letter for RFC2
email: adjust cover letter for RFC3
virtio-msg: Add virtio-msg, a message based virtio transport layer
virtio-msg: link virtio-msg content
.github/PULL_REQUEST_TEMPLATE.md | 14 +-
.github/workflows/deploy.yaml | 38 +
.github/workflows/test.yml | 15 +
.prjinfo/sendmail/cover.txt | 125 +++
.prjinfo/sendmail/send.sh | 57 ++
Makefile | 33 +
REVISION | 2 +-
commands.tex | 3 +-
conformance.tex | 105 +-
content.tex | 1 +
make-setup-generated.sh | 31 +-
makeall.sh | 4 +-
makehtml.sh | 2 +-
makepdf.sh | 2 +-
transport-msg.tex | 1640 ++++++++++++++++++++++++++++++
15 files changed, 2052 insertions(+), 20 deletions(-)
create mode 100644 .github/workflows/deploy.yaml
create mode 100644 .github/workflows/test.yml
create mode 100644 .prjinfo/sendmail/cover.txt
create mode 100755 .prjinfo/sendmail/send.sh
create mode 100644 Makefile
create mode 100644 transport-msg.tex
--
2.34.1
This series adds the virtio-msg transport layer.
The individuals and organizations involved in this effort have had difficulty in
using the existing virtio-transports in various situations and desire to add one
more transport that performs its transport layer operations by sending and
receiving messages.
Implementations of virtio-msg will normally be done in multiple layers:
* common / device level
* bus level
The common / device level defines the messages exchanged between the driver
and a device. This common part should lead to a common driver holding most
of the virtio specifics and can be shared by all virtio-msg bus implementations.
The kernel implementation in [3] shows this separation. As with other transport
layers, virtio-msg should not require modifications to existing virtio device
implementations (virtio-net, virtio-blk etc). The common / device level is the
main focus of this version of the patch series.
The virtio-msg bus level implements the normal things a bus defines
(enumeration, dma operations, etc) but also implements the message send and
receive operations. A number of bus implementations are envisioned,
some of which will be reusable and general purpose. Other bus implementations
might be unique to a given situation, for example only used by a PCIe card
and its driver.
How much of the bus level should be described in the virtio spec is one item
we wish to discuss. This draft takes a middle approach by describing the bus
level and defining some standard bus level messages that MAY be used by the bus.
It also describes a range of bus messages that are implementation dependent.
The standard bus messages are an effort to avoid different bus implementations
doing the same thing in different ways for no good reason. However the
different environments will require different things. Instead of trying to
anticipate all needs and provide something very abstract, we think
implementation specific messages will be needed at the bus level. Over time,
if we see similar messages across multiple bus implementations, we will move to
standardize a bus level message for that.
We are working on a few reusable bus implementations:
* virtio-msg-ffa based on Arm FF-A interface for use between:
* normal world and secure world
* host and VM or VM to VM
* Can be used w/ or with out a hypervisor
* Any Hypervisor that implements FF-A can be used
* We have this working with pKVM and Xen
* We have this working with Trusty and OP-TEE
* virtio-msg-amp for use between heterogenous systems
* The main processors and its co-processors on an AMP SOC
* Two or more systems connected via PCIe
* Minimal requirements: bi-directional interrupts and
at least one shared memory area
* hvac-demo has 2 demos of this
* This is working on two hardware platforms
* virtio-msg-loopback for userspace implemented devices
* Allows user space to provide devices to its own kernel
* This is similar to fuse, cuse or loopback block devices but for virtio
* hvac-demo has a demo
We also anticipate a few more:
* virtio-msg-xen specific to Xen
* Usable on any Xen system (including x86 where FF-A does not exist)
* Using Xen events and page grants
* virtio-msg over admin virtqueues
* This allows any virtio-pci device that supports admin virtqueues to also
support a virtio-msg bus that supports sub devices
* [We are looking for collaborators for this work]
Changes since RFC2:
Spec Functional:
* Made the common message header 8 bytes and added a token for optional use
when parallel outstanding requests are possible
* Made the 8 byte fields align to 8 byte offsets.
This effects the {SET,GET}_VQUEUE messages
Spec Editorial:
* Major restructure to better align with virtio spec
* Conformance model now matches
* Use C structures to define messages instead of tables
* Added a section to describe how responses are matched to requests
This includes the use of the new token field
* Redefined / better described error handling between transport and bus
layers to eliminate the need for the bus to generate fake response messages
* Included editorial feedback from RFC2
Eco-system:
* Add virtio-msg-loopback demo
* Arm has published the first draft of the virtio-msg over FFA spec
* virtio-msg over FFA has been demonstrated with both Trusty and OP-TEE
secure world
* QEMU RFCv1 has been sent
* LKML RFCs have been sent
Background info and work in progress implementations:
* HVAC project page with intro slides [1]
* HVAC demo repo w/ instructions in README.md [2]
* Kernel w/ virtio-msg common level and ffa support [3]
* QEMU w/ support for one form of virtio-msg-amp [4]
* Portable RTOS library w/ one form of virtio-msg-amp [5]
In addition to the QEMU system based demos in the hvac-demo repo, we also have
two hardware systems running:
* AMD x86 + AMD Arm Versal connected via PCIe
* ST STM32MP157 A7 Linux using virtio-i2c provided by M4 Zephyr
Please note that although the demos work, not all of them have been aligned
with this version of the spec.
[1] https://linaro.atlassian.net/wiki/spaces/HVAC/overview
[2] https://github.com/wmamills/hvac-demo
[3] https://git.kernel.org/pub/scm/linux/kernel/git/vireshk/linux.git/log/?h=vi…
[4] https://github.com/edgarigl/qemu/commits/edgar/virtio-msg-new
[5] https://github.com/arnopo/open-amp/commits/virtio-msg/
Bertrand Marquis (2):
virtio-msg: add new command for bus normative
virtio-msg: add conformance entries in conformance chapter
Bill Mills (2):
virtio-msg: Add virtio-msg, a message based virtio transport layer
virtio-msg: link virtio-msg content
commands.tex | 3 +-
conformance.tex | 105 ++-
content.tex | 1 +
transport-msg.tex | 1640 +++++++++++++++++++++++++++++++++++++++++++++
4 files changed, 1746 insertions(+), 3 deletions(-)
create mode 100644 transport-msg.tex
--
2.34.1
Happy new year everyone,
I have a few questions regarding the design of the whole memory sharing over
Vsock work and looking to get some feedback before I spend time making the
changes.
The initial design (from Nov):
- Initially I implemented a FFA based DMA heap (no DMA ops used), which would
allocate memory and make direct FFA calls to send the memory to the other
endpoint.
- The userspace would open this heap, allocated dma-buf and pass its FD over
Vsock.
- The Vsock layer would then call ->shmem() (a new callback I added to
dma-buf/heap, which will send memory over FFA and return ffa bus address), to
get the metadata to be shared over Vsock.
The current design:
- In one of the calls Bertrand suggested to not create parallel paths for
sending memory over FFA. Instead we should use the exiting dma-ops for FFA
(used for virtqueue and reserved-mem) somehow.
- I created a platform device (as a child of the FFA device) and assigned a new
set of DMA ops to it (the only difference from reserved-mem ops is that we
don't do swiotlb here and allocate fresh instead of the reserved mem).
- This pdev is used by the DMA heap to allocate memory using
dma_alloc_coherent() and that made sure everything got mapped correctly.
- I still need a dma-buf helper to get the metadata to send over Vsock. The
existing helper was renamed as s/shmem/shmem_data.
The future design (that I have questions about):
- The FFA specific DMA heap I now have doesn't do anything special compared to
the system heap, mostly exactly same.
- Which made me realize that probably I shouldn't add a new heap (Google can add
one later if they really want) and the solution should work with any heap /
dma-buf.
- So, userspace should allocate heap from system-heap, get a dma-buf from it and
send its FD.
- The vsock layer then should attach this dma-buf to a `struct device` somehow
and then call map_dma_buf() for the dma-buf. This requires the dma-ops of the
device to be set to the FFA based dma-ops and then it should just work.
- The tricky point is finding that device struct (as Vsock can't get it from
dma-buf or usersapce).
- One way, I think (still needs exploring but should be possible) is to use the
struct device of the virtio-msg device over which Vsock is implemented. We can
set the dma-ops of the virtio-msg device accordingly.
- The system heap doesn't guarantee contiguous allocation though (which my FFA
heap did) and so we will be required to send a scatter-gather list over vsock,
instead of just one address and size (what I am doing right now).
- Does this make sense ? Or if there is a use-case that this won't solve, etc ?
--
viresh
Hello,
I have some questions / feedback on the Xen side of virtio-msg.
AFAIU virtio-msg defines a protocol in order to deal with discovery
(optional) and configuration of the PV devices. But things are undefined
regarding what is a "memory address".
In Xen memory model with grants, each guest has its own memory space.
The frontend shares memory pages with the backend through grants pages
identified by grant references.
So in a design based on grants, virtio addresses can't (or at least
shouldn't) be guest physical address; but needs to be something derived
on grants. A earlier design [1] was forging a address with the grant
reference, but I feel it's not great, as it forces "map+unmap" cycles
for temporary buffers thus has the same performance problem as Xen PV
drivers (without persistent grants) where map+unmap cycles is a
performance problem.
And would make the address space very fragmented and in often limited to
4KB buffers.
One idea that is already used by Xen displif (PV Display) is to have a
"gref directory" and describe the address space on that. The gref
directory is roughly a array of grant references (shared pages infos)
that could describe a address space starting at 0, where each page is
defined by a grant reference of the directory. That way, the backend can
freely keep all or a part or the address space persistently mapped (or
eventually map it all at once); and the address space is also contiguous
which would help with >4KB buffers.
Any thoughts ?
[1]
https://static.sched.com/hosted_files/xen2021/bf/Thursday_2021-Xen-Summit-v…
--
Teddy Astie | Vates XCP-ng Developer
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
