Hello everyone,
Moving the discussion from google doc [1] to email (Sorry, it was becoming
impossible to follow and reply there).
FWIW, I am not the best person to answer all questions here, that would be
Armelle as she understand the requirements and the end goal much better than I
do. I can though try to answer from kernel's perspective, based on whatever
implementation we have right now.
AFAIK, the broad idea is to implement two virtio communication paths from pVM,
one to the Linux host (via virtio-pci) and another one to Trusty (via
virtio-msg-ffa). In order to not take performance hit at runtime (to map io
buffers), the idea is to map whatever amount of memory we can at the beginning
and then keep allocating from there.
Current setup:
What we have achieved until now is virtio-msg communication between host and
trusty. We have implemented FFA specific dma-hal [2] to perform FFA memory
sharing with trusty. With "reserved-mem" and "memory-region" DT entries (not
sure if that is the final solution), we are able to allocate memory the FFA
device (which represents bus for all the enumerated devices between
trusty/host). This memory is shared with trusty at probe time (from
virtio-msg-ffa layer) and the DMA hal later allocates memory from there for
coherent allocations and bounce buffers. This works just fine right now.
Now looking at "dynamic mapping" section in [1] we are not sure if that will
work fine for the end use case, pVM to trusty. It looks like the coco
implementation will always end up using dma encrypt/decrypt when a pVM is
running and share the memory with host, even when all we want to do is share
with trusty. Is that understanding correct ? We would also want to establish
virtio-pci (existing tech) based communication between pVM and host, which
should use mem decrypt path (?).
I am not sure if we need contiguous PA here, contiguous IPA should be
sufficient, Armelle?
We are also looking for further suggestions to improve the design, as my
understanding of memory mapping, dma hal etc. is limited, and maybe there are
better ways to do this.
--
Viresh
[1] https://docs.google.com/document/d/1KyxclKngQ0MShX8Q1YPstZQVIDIctkxfj6lQ0V3…
[2] https://web.git.kernel.org/pub/scm/linux/kernel/git/vireshk/linux.git/tree/…
Hi Everyone,
I pushed a new pull request to github on top of the current RFC status: https://github.com/Linaro/virtio-msg-spec/pull/13
This includes changes to handle some of the issues raised in github:
- have a version of the protocol
- have a variable message size
- have a configuration generation count
- give hint that bus implementation can be used for memory sharing or use of out of band notifications
Any comment is welcome :-)
Cheers
Bertrand
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
All,
The RFC went out a bit ago.
I updated it based on Bertrand's and Arnaud's suggestions (cover letter,
commit description, and s/request/requests/ only).
(I also fixed a s/massage/message/ in the cover letter.)
Going forward the new baseline for spec work will be
virtio-msg-rfc1. This branch is now the default at the linaro repo [1].
Please target all new pull requests to this branch.
The virtio-msg-alpha branch is now frozen but will be kept for reference.
Thanks,
Bill
[1] https://github.com/Linaro/virtio-msg-spec
--
Bill Mills
Principal Technical Consultant, Linaro
+1-240-643-0836
TZ: US Eastern
Work Schedule: Tues/Wed/Thur
All,
I have updated the hvac-demo repo [1] with a new version.
This version is tagged: v0.5
The following outlines the changes since Jan 27:
* The demos can now be run on an arm64 host
(I tested with an AWS EC2 m7g.2xlarge instance)
(x86_64 testing was done on my desktop and AWS EC2 m7i.2xlarge instances)
* Pre-setup container images (amd64 & arm64) are published at
docker.io/wmills/havc-demo
* A convenience script ./container was added to make it easier to run a
container with a mounted directory
(This is an alternative to using the published container images and
suggested for building the demos)
* The ./container flow now requires the user to run ./setup themselves
(The pre-built container images already have ./setup done for both "run"
and "build")
* Fixes found from Dan's testing
* Building demo1
* Running demo3
* Running demo4
* Fixes found from Alex's testing (running on Debian-12 w/o container and
using tmux already)
* Handle tmux server already running
* Isolate from users .tmux.conf and default server
* (These fixes are not needed to run in a container)
* I have tested with podman and docker
* Most of my testing was done on clean EC2 machines w/ Ubuntu 24.04, both
x86_64 and arm64
* I also used my x86_64 desktop w/ Ubuntu 22.04
* I tested running the docker.io/wmills/hvac-demo images and the
./container flow
* Updated README.MD with simplified instructions including above
Thanks,
Bill
[1] https://github.com/wmamills/hvac-demo.git
--
Bill Mills
Principal Technical Consultant, Linaro
+1-240-643-0836
TZ: US Eastern
Work Schedule: Tues/Wed/Thur
Hi,
I just tried to run demo1 from a docker setup.
I can run the demo1 using docker but the demo does not work (some error 95 in kernel transmitting FF-A messages, it went out to far so i could not copy paste it).
Cheers
Bertrand
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
This series adds the virtio-msg transport layer.
The individuals and organizations involved in this effort have had difficulty in
using the existing virtio-transports in various situations and desire to add one
more transport that performs its transport layer operations by sending and
receiving messages.
Implementations of virtio-msg will normally be done in multiple layers:
* common / device level
* bus level
The common / device level can be shared by all virtio-msg bus implementations
and defines the messages exchanged between the driver and a device. As with
other transport layers, virtio-msg should not require modifications to existing
virtio device implementations (virtio-net, virtio-blk etc). The common / device
level is the main focus of this version of the patch series.
The virtio-msg bus level implements the normal things a bus defines
(enumeration, dma operations, etc) but also implements the message send and
receive operations. A number of bus implementations are envisioned,
some of which will be reusable and general purpose. Other bus implementations
might be unique to a given situation, for example only used by a PCIe card
and its driver.
How much of the bus level should be described in the virtio spec is one item
we wish to discuss. This draft takes a middle approach by describing the bus
level and defining some standard bus level messages that MAY be used by the bus.
It also describes a range of bus messages that are implementation dependent.
The standard bus messages are an effort to avoid different bus implementations
doing the same thing in different ways for no good reason. However the
different environments will require different things. Instead of trying to
anticipate all needs and provide something very abstract, we think
implementation specific messages will be needed at the bus level. Over time,
if we see similar messages across multiple bus implementations, we will move to
standardize a bus level message for that.
We are working on two reusable bus implementations:
* virtio-msg-ffa based on Arm FF-A interface for use between:
* normal world and secure world
* host and VM or VM to VM
* Can be used w/ or with out a hypervisor
* Any Hypervisor that implements FF-A can be used
* virtio-msg-amp for use between heterogenous systems
* The main processors and its co-processors on an AMP SOC
* Two or more systems connected via PCIe
* Minimal requirements: bi-directional interrupts and
at least one shared memory area
We also anticipate a third:
* virtio-msg-xen specific to Xen
* Usable on any Xen system (including x86 where FF-A does not exist)
* Using Xen events and page grants
This series is a work in progress and we acknowledge at least the following
issues we need to work on:
* Conform to virtio spec nouns (device/driver vs frontend/backend)
and verbs (must/may)
* Perhaps move error definition elsewhere it the spec and align on its symbols
and numeric values
* Allow massage size to be greater than 40 bytes and allow bus implementations
to define their max message size
* Add a way to discover the protocol version
* Add a better description of the types of things a bus can do, specifically
including out-of-band notification and memory area sharing/discovery
* Maybe redo configuration generation handling
Background info and work in progress implementations:
* HVAC project page with intro slides [1]
* HVAC demo repo w/ instructions in README.md [2]
* Kernel w/ virtio-msg common level and ffa support [3]
* QEMU w/ support for one form of virtio-msg-amp [4]
* Portable RTOS library w/ one form of virtio-msg-amp [5]
In addition to the QEMU system based demos in the hvac-demo repo, we also have
two hardware systems running:
* AMD x86 + AMD Arm Versal connected via PCIe
* ST STM32MP157 A7 Linux using virtio-i2c provided by M4 Zephyr
Please note that although the demos work, they are not yet aligned with each
other nor this version of the spec.
[1] https://linaro.atlassian.net/wiki/spaces/HVAC/overview
[2] https://github.com/wmamills/hvac-demo
[3] https://git.kernel.org/pub/scm/linux/kernel/git/vireshk/linux.git/log/?h=vi…
[4] https://github.com/edgarigl/qemu/commits/edgar/virtio-msg-new
[5] https://github.com/arnopo/open-amp/commits/virtio-msg/
Bill Mills (1):
virtio-msg: Add virtio-msg, a message based virtio transport layer
content.tex | 1 +
transport-msg.tex | 680 ++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 681 insertions(+)
create mode 100644 transport-msg.tex
--
2.34.1
Hi Viresh,
I'll try to respond to your first question: indirect messages require FF-A
notifications, and Trusty does not support those either.
The problem isn't Trusty specifically, but the layers between it and Linux:
the hypervisor and SPMC.
To be as specific as I can: Hafnium supports notifications, but others
don't.
For now, we'll need to work around this in virtio-msg-ffa with direct
messages and polling.
Andrei Homescu
Hi Ayrton / Armelle,
I tried to investigate about how to make VIRTIO_MSG_EVENT_USED work without
indirect FFA messages.
- Firstly, I am still not sure if I fully understand why indirect messages can't
be implemented on Trustee side ? I know it was discussed a bit in the call,
but I don't remember any of it :(
- If we want to do polling of the virtqueues, then it needs to be done on the
driver side (vsock), I don't see how I can generalize it and do it from
virtio-msg-ffa layer. The driver needs to do something like this for the
virtqueue:
while (1) {
/* Try to get a buffer from the virtqueue */
buf = virtqueue_get_buf(vq, &len);
if (buf) {
// Handle the received buffer
break;
}
/* Avoid busy looping with a small delay */
cpu_relax();
}
- The other idea discussed on the call was about always queuing a direct message
for a FFA device (not virtio device) for EVENT_USED message. Once the trustee
side has an event to send, it can respond to this request.
I was thinking if this may block the CPU for ever with the SMC call, but I
don't think that is the case. The FFA layer makes the SMC call, checks its
return value and sleeps for a ms, and tries again. With this the CPU will
schedule other work as soon as it can. And looks like we support sending
multiple direct messages concurrently too, so while a thread is waiting for a
response to EVENT_USED message, we can keep sending other messages.
If we want to support this, will this be part of the spec ? Bertrand ?
- Any other ideas on how to get this solved ?
--
viresh
All,
You will soon receive an email with the RFC series in its current state.
This is a test run; a draft of RFC just for members of this list.
If you see anything in the cover letter or commit message that you wish
to change please let me know by Monday at 4pm US Eastern time.
If you wish to be added to the Signed-off-by list or just the CC list
please let me know by the same deadline. Please specify which. It would
be good to include someone from Google (even if only in the CC).
Changes to the actual patch content is out of scope for now.
On Tuesday I will explain more about the branching and PR strategy going
forward. For now I am sending the RFC from a branch not yet on the
Linaro github.
Thanks,
Bill
--
Bill Mills
Principal Technical Consultant, Linaro
+1-240-643-0836
TZ: US Eastern
Work Schedule: Tues/Wed/Thur
All,
Deadline for issues and PRs: Wednesday Feb 12
Discussion: Thursday Feb 13
RFC Target: Friday Feb 21
We want to publish the first draft of the virtio-msg changes to the
virtio spec list as an RFC. We want to make sure:
1) It describes what we are doing
2) Is complete enough for virtio knowledgeable people to understand
3) _We_ don't need big changes*
* Reviewers may ask for big changes and we need to be prepared for that.
But we need to have our thinking on the same page collectively as much
as possible.
The work-in-progress spec is here:
https://github.com/Linaro/virtio-msg-spec
The most recent rendered pdf is here:
https://github.com/Linaro/virtio-msg-spec/releases
In addition there is pull request #5 that adds most of the things we
spoke of in our Dublin sprint:
https://github.com/Linaro/virtio-msg-spec/pull/5
All changes are in the file transport-msg.tex
(There is one change to content.tex to include the new file)
Providing feedback:
Unlike the upstream spec, we can use github issues and github pull
requests to manage changes within this group. Then one of us (myself or
Bertrand most likely) will send to upstream virtio spec mailing list.
To provide your feedback please create an issue or a PR.
For small changes it is probibly easier to just create a PR with the
exact wording you want. You can bundle a number commits into one PR but
try to keep each commit to one topic or one section.
After the RFC most of the interaction will move to
virtio-comment(a)lists.linux.dev so please subscribe now.
See the upstream README for details:
https://github.com/oasis-tcs/virtio-spec/blob/master/README.md
Thanks,
Bill
--
Bill Mills
Principal Technical Consultant, Linaro
+1-240-643-0836
TZ: US Eastern
Work Schedule: Tues/Wed/Thur
