All,
I have pushed and tagged a new version of the hvac-demo repo.
I have also updated the pre-built container image at:
docker.io/wmills/hvac-demo
Remember you may need to pull the container image again to see the new version:
podman pull docker.io/wmills/hvac-demo
- or -
docker pull docker.io/wmills/hvac-demo
Changes include:
* added demo-loopback to show virtio-msg-loopback (based on Viresh's work)
* added demo-amp-qemu-pci with virtio-msg-amp on an emulated pcie card in QEMU
* renamed all the demos to describe the demo instead of numbers
* rebased demo-ffa-xen to use Viresh's latest kernel and rust code
* other demos are just renamed
Old name New name
----------------------------------
new demo-loopback
new demo-amp-qemu-pci
demo1 demoo-ffa-xen
demo2a (deleted)
demo2b demo-proxy-qemu
demo3 demo-amp-zephyr
demo4 demo-amp-dual-linux
The following issues are known:
1) demo-ffa-xen hangs or crashes 1 out of 5 or 10 times
Just run again if you see this
2) ./Build all does not build linux-virtio-msg-v2, use
./Build all linux-virtio-msg-amp-v2
You do not need to build the demos to run them
3) The timeout in demo-amp-dual-linux to too short for VERY slow systems
Most people won't see this issue. I was testing on a RPiv5
Fixes for the last two issues are in the wip-v0.6-fixes branch.
--
Bill Mills
Principal Technical Consultant, Linaro
+1-240-643-0836
TZ: US Eastern
Work Schedule: Tues/Wed/Thur
Hello,
I will try to send this upstream early next week.
V2:
- Don't limit get/set config to 8 bytes only.
- Improved direct/indirect message handling.
- Take care of EBUSY error during indirect message.
- Static assertions.
-------------------------8<-------------------------
Hello,
This RFC series introduces support for a new Virtio transport type:
"virtio-msg", as proposed in [1]. Unlike existing transport types like
virtio-mmio or virtio-pci which rely on memory-mapped registers, virtio-msg
implements transport operations via structured messages. Those messages can be
transported through different mechanisms such as mailboxes, shared memory based
FIFO or specific protocols such as FF-A on Arm.
This series includes:
- Core virtio-msg transport support.
- Two message transport bus implementations:
- virtio-msg-ffa: based on ARM's Firmware Framework for Arm (FF-A).
- virtio-msg-loopback: a loopback device for testing and validation.
The code is available here for reference: [3] and virtio-msg loopback test setup
is explained here: [2].
This series is based on v6.16-rc6 and depends on commit [4] from linux-next.
### Memory Mapping and Reserved Memory Usage
The first two patches enhance the reserved-memory subsystem to support attaching
struct device`s that do not originate from DT nodes—essential for virtual or
dynamically discovered devices like the FF-A or loopback buses.
This reserved-memory region enables:
- Restricting all DMA-coherent and streaming DMA memory to a controlled range.
- Allowing the remote endpoint to pre-map this memory, reducing runtime overhead.
- Preventing unintentional data leaks, since memory is typically shared at page
granularity.
- For the loopback bus, it restricts the portion of kernel memory that can be
mapped into userspace, improving safety.
Device association with reserved-memory regions is based on DT node naming
conventions, such as vmsglb@ or vmsgffa@, similar to the remoteproc framework’s
approach.
Feedback on the design, API, and approach is welcome.
--
Viresh
[1] https://lore.kernel.org/all/20250620224426.3923880-2-bill.mills@linaro.org/
[2] https://linaro.atlassian.net/wiki/spaces/HVAC/pages/30104092673
[3] git://git.kernel.org/pub/scm/linux/kernel/git/vireshk/linux.git virtio/msg-rfc-v1
[4] From linux-next: 5be53630b4f0 virtio-mmio: Remove virtqueue list from mmio device
Viresh Kumar (6):
of: reserved-memory: Add reserved_mem_device_init()
of: reserved-memory: Add of_reserved_mem_lookup_by_name
virtio: Add support for virtio-msg transport
virtio-msg: Add optional userspace interface for message I/O
virtio-msg: Add support for FF-A (Firmware Framework for Arm) bus
virtio-msg: Add support for loopback bus
MAINTAINERS | 7 +
drivers/of/of_reserved_mem.c | 91 ++--
drivers/virtio/Kconfig | 34 ++
drivers/virtio/Makefile | 5 +
drivers/virtio/virtio_msg.c | 611 +++++++++++++++++++++++++++
drivers/virtio/virtio_msg.h | 88 ++++
drivers/virtio/virtio_msg_ffa.c | 505 ++++++++++++++++++++++
drivers/virtio/virtio_msg_loopback.c | 323 ++++++++++++++
drivers/virtio/virtio_msg_user.c | 119 ++++++
include/linux/of_reserved_mem.h | 13 +
include/uapi/linux/virtio_msg.h | 221 ++++++++++
include/uapi/linux/virtio_msg_ffa.h | 94 +++++
include/uapi/linux/virtio_msg_lb.h | 22 +
13 files changed, 2109 insertions(+), 24 deletions(-)
create mode 100644 drivers/virtio/virtio_msg.c
create mode 100644 drivers/virtio/virtio_msg.h
create mode 100644 drivers/virtio/virtio_msg_ffa.c
create mode 100644 drivers/virtio/virtio_msg_loopback.c
create mode 100644 drivers/virtio/virtio_msg_user.c
create mode 100644 include/uapi/linux/virtio_msg.h
create mode 100644 include/uapi/linux/virtio_msg_ffa.h
create mode 100644 include/uapi/linux/virtio_msg_lb.h
--
2.31.1.272.g89b43f80a514
Hello,
This RFC series introduces support for a new Virtio transport type:
"virtio-msg", as proposed in [1]. Unlike existing transport types like
virtio-mmio or virtio-pci, which rely on memory-mapped registers, virtio-msg
implements transport operations via structured messages. Those messages can be
transported through different mechanisms such as mailboxes, shared memory based
FIFO or specific protocols such as FF-A on Arm.
This series includes:
- Core virtio-msg transport support.
- Two message transport bus implementations:
- virtio-msg-ffa: based on ARM's Firmware Framework for Arm (FF-A).
- virtio-msg-loopback: a loopback device for testing and validation.
The code is available here for reference: [2] and virtio-msg loopback and FF-A
test setups are explained here: [3] and [4].
This series is based on v6.16 and depends on commit [5].
### Memory Mapping and Reserved Memory Usage
The first two patches enhance the reserved-memory subsystem to support attaching
struct device`s that do not originate from DT nodes — essential for virtual or
dynamically discovered devices like the FF-A or loopback buses.
This reserved-memory region enables:
- Restricting all DMA-coherent and streaming DMA memory to a controlled range.
- Allowing the remote endpoint to pre-map this memory, reducing runtime overhead.
- Preventing unintentional data leaks, since memory is typically shared at page
granularity.
- For the loopback bus, it restricts the portion of kernel memory that can be
mapped into userspace, improving safety.
Device association with reserved-memory regions is based on DT node naming
conventions, such as vmsglb@ or vmsgffa@, similar to the remoteproc framework’s
approach.
Feedback on the design, API, and approach is welcome.
--
Viresh
[1] https://lore.kernel.org/all/20250620224426.3923880-2-bill.mills@linaro.org/
[2] git://git.kernel.org/pub/scm/linux/kernel/git/vireshk/linux.git virtio/msg-rfc-v1
[3] https://linaro.atlassian.net/wiki/spaces/HVAC/pages/30104092673
[4] https://linaro.atlassian.net/wiki/spaces/HVAC/pages/29657792513
[5] From linux-next: 5be53630b4f0 virtio-mmio: Remove virtqueue list from mmio device
Viresh Kumar (6):
of: reserved-memory: Add reserved_mem_device_init()
of: reserved-memory: Add of_reserved_mem_lookup_by_name
virtio: Add support for virtio-msg transport
virtio-msg: Add optional userspace interface for message I/O
virtio-msg: Add support for FF-A (Firmware Framework for Arm) bus
virtio-msg: Add support for loopback bus
MAINTAINERS | 7 +
drivers/of/of_reserved_mem.c | 91 +++-
drivers/virtio/Kconfig | 34 ++
drivers/virtio/Makefile | 5 +
drivers/virtio/virtio_msg.c | 655 +++++++++++++++++++++++++++
drivers/virtio/virtio_msg_ffa.c | 505 +++++++++++++++++++++
drivers/virtio/virtio_msg_internal.h | 88 ++++
drivers/virtio/virtio_msg_loopback.c | 323 +++++++++++++
drivers/virtio/virtio_msg_user.c | 140 ++++++
include/linux/of_reserved_mem.h | 13 +
include/uapi/linux/virtio_msg.h | 213 +++++++++
include/uapi/linux/virtio_msg_ffa.h | 94 ++++
include/uapi/linux/virtio_msg_lb.h | 22 +
13 files changed, 2166 insertions(+), 24 deletions(-)
create mode 100644 drivers/virtio/virtio_msg.c
create mode 100644 drivers/virtio/virtio_msg_ffa.c
create mode 100644 drivers/virtio/virtio_msg_internal.h
create mode 100644 drivers/virtio/virtio_msg_loopback.c
create mode 100644 drivers/virtio/virtio_msg_user.c
create mode 100644 include/uapi/linux/virtio_msg.h
create mode 100644 include/uapi/linux/virtio_msg_ffa.h
create mode 100644 include/uapi/linux/virtio_msg_lb.h
--
2.31.1.272.g89b43f80a514
Hi everyone,
Over the last few weeks, I have made numerous changes in the design and code to
get it ready for mainline. I have now prepared the initial RFC patchset to be
sent to LKML.
Please have a look and provide your valuable feedback.
-------------------------8<-------------------------
Hello,
This RFC series introduces support for a new Virtio transport type:
"virtio-msg", as proposed in [1]. Unlike existing transport types like
virtio-mmio or virtio-pci which rely on memory-mapped registers, virtio-msg
implements transport operations via structured message exchanges using standard
virtqueues.
This series includes:
- Core virtio-msg transport support.
- Two message transport bus implementations:
- virtio-msg-ffa: based on ARM's Firmware Framework for Arm (FF-A).
- virtio-msg-loopback: a loopback device for testing and validation.
The code is available here for reference: [3] and virtio-msg loopback test setup
is explained here: [2].
This series is based on v6.16-rc6 and depends on commit [4] from linux-next.
### Memory Mapping and Reserved Memory Usage
The first two patches enhance the reserved-memory subsystem to support attaching
`struct device`s that do not originate from DT nodes—essential for virtual or
dynamically discovered devices like the FF-A or loopback buses.
This reserved-memory region enables:
- Restricting all DMA-coherent and streaming DMA memory to a controlled range.
- Allowing the remote endpoint to pre-map this memory, reducing runtime overhead.
- Preventing unintentional data leaks, since memory is typically shared at page
granularity.
- For the loopback bus, it restricts the portion of kernel memory that can be
mapped into userspace, improving safety.
Feedback on the design, API, and approach is welcome.
--
Viresh
[1] https://lore.kernel.org/all/20250620224426.3923880-2-bill.mills@linaro.org/
[2] https://linaro.atlassian.net/wiki/spaces/HVAC/pages/30104092673
[3] git://git.kernel.org/pub/scm/linux/kernel/git/vireshk/linux.git virtio/msg-rfc-v1
[4] From linux-next: 5be53630b4f0 virtio-mmio: Remove virtqueue list from mmio device
Viresh Kumar (6):
of: reserved-memory: Add reserved_mem_device_init()
of: reserved-memory: Add of_reserved_mem_lookup_by_name
virtio: Add support for virtio-msg transport
virtio-msg: Add optional userspace interface for message I/O
virtio-msg: Add support for FF-A (Firmware Framework for Arm) bus
virtio-msg: Add support for loopback bus
MAINTAINERS | 7 +
drivers/of/of_reserved_mem.c | 91 +++--
drivers/virtio/Kconfig | 34 ++
drivers/virtio/Makefile | 5 +
drivers/virtio/virtio_msg.c | 546 +++++++++++++++++++++++++++
drivers/virtio/virtio_msg.h | 88 +++++
drivers/virtio/virtio_msg_ffa.c | 501 ++++++++++++++++++++++++
drivers/virtio/virtio_msg_loopback.c | 323 ++++++++++++++++
drivers/virtio/virtio_msg_user.c | 119 ++++++
include/linux/of_reserved_mem.h | 13 +
include/uapi/linux/virtio_msg.h | 221 +++++++++++
include/uapi/linux/virtio_msg_ffa.h | 94 +++++
include/uapi/linux/virtio_msg_lb.h | 22 ++
13 files changed, 2040 insertions(+), 24 deletions(-)
create mode 100644 drivers/virtio/virtio_msg.c
create mode 100644 drivers/virtio/virtio_msg.h
create mode 100644 drivers/virtio/virtio_msg_ffa.c
create mode 100644 drivers/virtio/virtio_msg_loopback.c
create mode 100644 drivers/virtio/virtio_msg_user.c
create mode 100644 include/uapi/linux/virtio_msg.h
create mode 100644 include/uapi/linux/virtio_msg_ffa.h
create mode 100644 include/uapi/linux/virtio_msg_lb.h
--
2.31.1.272.g89b43f80a514
Hello,
As discussed yesterday during the call, I have prepared the list of instructions
to reproduce my virtio-msg loopback setup. Please give a try and lemme know if
something doesn't work.
This is based on the latest version of the spec (from Bill's latest pull request
today).
--
Viresh
[1] https://linaro.atlassian.net/wiki/spaces/HVAC/pages/30104092673
--
viresh
This series adds the virtio-msg transport layer.
The individuals and organizations involved in this effort have had difficulty in
using the existing virtio-transports in various situations and desire to add one
more transport that performs its transport layer operations by sending and
receiving messages.
Implementations of virtio-msg will normally be done in multiple layers:
* common / device level
* bus level
The common / device level defines the messages exchanged between the driver
and a device. This common part should lead to a common driver holding most
of the virtio specifics and can be shared by all virtio-msg bus implementations.
The kernel implementation in [3] shows this separation. As with other transport
layers, virtio-msg should not require modifications to existing virtio device
implementations (virtio-net, virtio-blk etc). The common / device level is the
main focus of this version of the patch series.
The virtio-msg bus level implements the normal things a bus defines
(enumeration, dma operations, etc) but also implements the message send and
receive operations. A number of bus implementations are envisioned,
some of which will be reusable and general purpose. Other bus implementations
might be unique to a given situation, for example only used by a PCIe card
and its driver.
How much of the bus level should be described in the virtio spec is one item
we wish to discuss. This draft takes a middle approach by describing the bus
level and defining some standard bus level messages that MAY be used by the bus.
It also describes a range of bus messages that are implementation dependent.
The standard bus messages are an effort to avoid different bus implementations
doing the same thing in different ways for no good reason. However the
different environments will require different things. Instead of trying to
anticipate all needs and provide something very abstract, we think
implementation specific messages will be needed at the bus level. Over time,
if we see similar messages across multiple bus implementations, we will move to
standardize a bus level message for that.
We are working on two reusable bus implementations:
* virtio-msg-ffa based on Arm FF-A interface for use between:
* normal world and secure world
* host and VM or VM to VM
* Can be used w/ or with out a hypervisor
* Any Hypervisor that implements FF-A can be used
* virtio-msg-amp for use between heterogenous systems
* The main processors and its co-processors on an AMP SOC
* Two or more systems connected via PCIe
* Minimal requirements: bi-directional interrupts and
at least one shared memory area
We also anticipate a few more:
* virtio-msg-xen specific to Xen
* Usable on any Xen system (including x86 where FF-A does not exist)
* Using Xen events and page grants
* virtio-msg-loopback for userspace implemented devices
* Allows user space to provide devices to its own kernel
* This is similar to fuse, cuse or loopback block devices but for virtio
* Once developed this can provide a single kernel demo of virtio-msg
* [Work has begun on this]
* virtio-msg over admin virtqueues
* This allows any virtio-pci device that supports admin virtqueues to also
support a virtio-msg bus that supports sub devices
* [We are looking for collaborators for this work]
Changes since RFC1:
* reformated document to better conform to the virtio spec style
- created an introduction chapter
- created a basic concept chapter
- created bus operation and device initialization and operation chapters
- reworked description of transport and bus messages
- attempted a "compliance chapter"
- reused spec macros
- switched to MAY/MUST/SHALL/SHOULD wording
- eliminate the use of terms front-end and back-end and use driver and device
* made the maximum message size variable per bus instance
* use "device number" for virtio-msg device instances on the bus instead of
adding yet another meaning for "device ID"
* added the configuration generation count and its use
* described types of things that can be done with bus specific messages
such as setup of bus level shared memory and out of band notifications
* removed wording of notification being optional at transport level and
described bus level notifications of in-band, out-of-band, and polled
from driver side bus
* removed the ERROR message from transport level. Errors should be handled at
the bus level to better match virtio-pci and virtio-mmio
* removed bus level reset and status from standard bus messages
* replaced bus messages DEVICE_{ADDED,REMOVED} with EVENT_DEVICE
* changed names to GET_DEVICE_FEATURES and SET_DRIVER_FEATURES for clarity
* made SET_DEVICE_STATE return new state as it may not match
* Don't echo back the data in SET_VQUEUE (it cannot change)
* defined request/response vs event message id ranges
* match field size of next offset and wrap to virtio-{pci,mmio}
* added maximum number of virtqueues to DEVICE_INFO
* added admin virtqueue and SHM support
This series is a work in progress and we acknowledge at least the following
issues we need to work on:
* Better conformance documentation
* Publish an update to Arm FF-A spec that shows virtio-msg binding (work underway)
* Publish virtio-msg-amp data structures and messages somewhere
* Align implementations to this version and send PATCH v1 (non-rfc)
Background info and work in progress implementations:
* HVAC project page with intro slides [1]
* HVAC demo repo w/ instructions in README.md [2]
* Kernel w/ virtio-msg common level and ffa support [3]
* QEMU w/ support for one form of virtio-msg-amp [4]
* Portable RTOS library w/ one form of virtio-msg-amp [5]
In addition to the QEMU system based demos in the hvac-demo repo, we also have
two hardware systems running:
* AMD x86 + AMD Arm Versal connected via PCIe
* ST STM32MP157 A7 Linux using virtio-i2c provided by M4 Zephyr
Please note that although the demos work, they are not yet aligned with each
other nor this version of the spec.
[1] https://linaro.atlassian.net/wiki/spaces/HVAC/overview
[2] https://github.com/wmamills/hvac-demo
[3] https://git.kernel.org/pub/scm/linux/kernel/git/vireshk/linux.git/log/?h=vi…
[4] https://github.com/edgarigl/qemu/commits/edgar/virtio-msg-new
[5] https://github.com/arnopo/open-amp/commits/virtio-msg/
Bill Mills (1):
virtio-msg: Add virtio-msg, a message based virtio transport layer
content.tex | 1 +
transport-msg.tex | 1354 +++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 1355 insertions(+)
create mode 100644 transport-msg.tex
--
2.34.1
All,
The pull request referenced here covers the remaining items needed before
sending RFCv2. Please take a look and let me know if you see issues. You
can comment in the PR or send email to the list.
https://github.com/Linaro/virtio-msg-spec/pull/20
Thanks,
Bill
--
Bill Mills
Principal Technical Consultant, Linaro
+1-240-643-0836
TZ: US Eastern
Work Schedule: Tues/Wed/Thur
Hello everyone,
Moving the discussion from google doc [1] to email (Sorry, it was becoming
impossible to follow and reply there).
FWIW, I am not the best person to answer all questions here, that would be
Armelle as she understand the requirements and the end goal much better than I
do. I can though try to answer from kernel's perspective, based on whatever
implementation we have right now.
AFAIK, the broad idea is to implement two virtio communication paths from pVM,
one to the Linux host (via virtio-pci) and another one to Trusty (via
virtio-msg-ffa). In order to not take performance hit at runtime (to map io
buffers), the idea is to map whatever amount of memory we can at the beginning
and then keep allocating from there.
Current setup:
What we have achieved until now is virtio-msg communication between host and
trusty. We have implemented FFA specific dma-hal [2] to perform FFA memory
sharing with trusty. With "reserved-mem" and "memory-region" DT entries (not
sure if that is the final solution), we are able to allocate memory the FFA
device (which represents bus for all the enumerated devices between
trusty/host). This memory is shared with trusty at probe time (from
virtio-msg-ffa layer) and the DMA hal later allocates memory from there for
coherent allocations and bounce buffers. This works just fine right now.
Now looking at "dynamic mapping" section in [1] we are not sure if that will
work fine for the end use case, pVM to trusty. It looks like the coco
implementation will always end up using dma encrypt/decrypt when a pVM is
running and share the memory with host, even when all we want to do is share
with trusty. Is that understanding correct ? We would also want to establish
virtio-pci (existing tech) based communication between pVM and host, which
should use mem decrypt path (?).
I am not sure if we need contiguous PA here, contiguous IPA should be
sufficient, Armelle?
We are also looking for further suggestions to improve the design, as my
understanding of memory mapping, dma hal etc. is limited, and maybe there are
better ways to do this.
--
Viresh
[1] https://docs.google.com/document/d/1KyxclKngQ0MShX8Q1YPstZQVIDIctkxfj6lQ0V3…
[2] https://web.git.kernel.org/pub/scm/linux/kernel/git/vireshk/linux.git/tree/…
