- Virtio-msg - lists.linaro.org

by Bill Mills

All, I have pushed and tagged a new version of the hvac-demo repo. I have also updated the pre-built container image at: docker.io/wmills/hvac-demo Remember you may need to pull the container image again to see the new version: podman pull docker.io/wmills/hvac-demo - or - docker pull docker.io/wmills/hvac-demo Changes include: * added demo-loopback to show virtio-msg-loopback (based on Viresh's work) * added demo-amp-qemu-pci with virtio-msg-amp on an emulated pcie card in QEMU * renamed all the demos to describe the demo instead of numbers * rebased demo-ffa-xen to use Viresh's latest kernel and rust code * other demos are just renamed Old name New name ---------------------------------- new demo-loopback new demo-amp-qemu-pci demo1 demoo-ffa-xen demo2a (deleted) demo2b demo-proxy-qemu demo3 demo-amp-zephyr demo4 demo-amp-dual-linux The following issues are known: 1) demo-ffa-xen hangs or crashes 1 out of 5 or 10 times Just run again if you see this 2) ./Build all does not build linux-virtio-msg-v2, use ./Build all linux-virtio-msg-amp-v2 You do not need to build the demos to run them 3) The timeout in demo-amp-dual-linux to too short for VERY slow systems Most people won't see this issue. I was testing on a RPiv5 Fixes for the last two issues are in the wip-v0.6-fixes branch. -- Bill Mills Principal Technical Consultant, Linaro +1-240-643-0836 TZ: US Eastern Work Schedule: Tues/Wed/Thur

4 months

1
0
0 0

[RFC PATCH 0/6] virtio: Add support for Virtio message transport

by Viresh Kumar

Hello, This RFC series introduces support for a new Virtio transport type: "virtio-msg", as proposed in [1]. Unlike existing transport types like virtio-mmio or virtio-pci, which rely on memory-mapped registers, virtio-msg implements transport operations via structured messages. Those messages can be transported through different mechanisms such as mailboxes, shared memory based FIFO or specific protocols such as FF-A on Arm. This series includes: - Core virtio-msg transport support. - Two message transport bus implementations: - virtio-msg-ffa: based on ARM's Firmware Framework for Arm (FF-A). - virtio-msg-loopback: a loopback device for testing and validation. The code is available here for reference: [2] and virtio-msg loopback and FF-A test setups are explained here: [3] and [4]. This series is based on v6.16 and depends on commit [5]. ### Memory Mapping and Reserved Memory Usage The first two patches enhance the reserved-memory subsystem to support attaching struct device`s that do not originate from DT nodes — essential for virtual or dynamically discovered devices like the FF-A or loopback buses. This reserved-memory region enables: - Restricting all DMA-coherent and streaming DMA memory to a controlled range. - Allowing the remote endpoint to pre-map this memory, reducing runtime overhead. - Preventing unintentional data leaks, since memory is typically shared at page granularity. - For the loopback bus, it restricts the portion of kernel memory that can be mapped into userspace, improving safety. Device association with reserved-memory regions is based on DT node naming conventions, such as vmsglb@ or vmsgffa@, similar to the remoteproc framework’s approach. Feedback on the design, API, and approach is welcome. -- Viresh [1] https://lore.kernel.org/all/20250620224426.3923880-2-bill.mills@linaro.org/ [2] git://git.kernel.org/pub/scm/linux/kernel/git/vireshk/linux.git virtio/msg-rfc-v1 [3] https://linaro.atlassian.net/wiki/spaces/HVAC/pages/30104092673 [4] https://linaro.atlassian.net/wiki/spaces/HVAC/pages/29657792513 [5] From linux-next: 5be53630b4f0 virtio-mmio: Remove virtqueue list from mmio device Viresh Kumar (6): of: reserved-memory: Add reserved_mem_device_init() of: reserved-memory: Add of_reserved_mem_lookup_by_name virtio: Add support for virtio-msg transport virtio-msg: Add optional userspace interface for message I/O virtio-msg: Add support for FF-A (Firmware Framework for Arm) bus virtio-msg: Add support for loopback bus MAINTAINERS | 7 + drivers/of/of_reserved_mem.c | 91 +++- drivers/virtio/Kconfig | 34 ++ drivers/virtio/Makefile | 5 + drivers/virtio/virtio_msg.c | 655 +++++++++++++++++++++++++++ drivers/virtio/virtio_msg_ffa.c | 505 +++++++++++++++++++++ drivers/virtio/virtio_msg_internal.h | 88 ++++ drivers/virtio/virtio_msg_loopback.c | 323 +++++++++++++ drivers/virtio/virtio_msg_user.c | 140 ++++++ include/linux/of_reserved_mem.h | 13 + include/uapi/linux/virtio_msg.h | 213 +++++++++ include/uapi/linux/virtio_msg_ffa.h | 94 ++++ include/uapi/linux/virtio_msg_lb.h | 22 + 13 files changed, 2166 insertions(+), 24 deletions(-) create mode 100644 drivers/virtio/virtio_msg.c create mode 100644 drivers/virtio/virtio_msg_ffa.c create mode 100644 drivers/virtio/virtio_msg_internal.h create mode 100644 drivers/virtio/virtio_msg_loopback.c create mode 100644 drivers/virtio/virtio_msg_user.c create mode 100644 include/uapi/linux/virtio_msg.h create mode 100644 include/uapi/linux/virtio_msg_ffa.h create mode 100644 include/uapi/linux/virtio_msg_lb.h -- 2.31.1.272.g89b43f80a514

4 months, 1 week

1
6
0 0

[RFC PATCH 0/6] virtio: Add support for Virtio message transport

by Viresh Kumar

Hi everyone, Over the last few weeks, I have made numerous changes in the design and code to get it ready for mainline. I have now prepared the initial RFC patchset to be sent to LKML. Please have a look and provide your valuable feedback. -------------------------8<------------------------- Hello, This RFC series introduces support for a new Virtio transport type: "virtio-msg", as proposed in [1]. Unlike existing transport types like virtio-mmio or virtio-pci which rely on memory-mapped registers, virtio-msg implements transport operations via structured message exchanges using standard virtqueues. This series includes: - Core virtio-msg transport support. - Two message transport bus implementations: - virtio-msg-ffa: based on ARM's Firmware Framework for Arm (FF-A). - virtio-msg-loopback: a loopback device for testing and validation. The code is available here for reference: [3] and virtio-msg loopback test setup is explained here: [2]. This series is based on v6.16-rc6 and depends on commit [4] from linux-next. ### Memory Mapping and Reserved Memory Usage The first two patches enhance the reserved-memory subsystem to support attaching `struct device`s that do not originate from DT nodes—essential for virtual or dynamically discovered devices like the FF-A or loopback buses. This reserved-memory region enables: - Restricting all DMA-coherent and streaming DMA memory to a controlled range. - Allowing the remote endpoint to pre-map this memory, reducing runtime overhead. - Preventing unintentional data leaks, since memory is typically shared at page granularity. - For the loopback bus, it restricts the portion of kernel memory that can be mapped into userspace, improving safety. Feedback on the design, API, and approach is welcome. -- Viresh [1] https://lore.kernel.org/all/20250620224426.3923880-2-bill.mills@linaro.org/ [2] https://linaro.atlassian.net/wiki/spaces/HVAC/pages/30104092673 [3] git://git.kernel.org/pub/scm/linux/kernel/git/vireshk/linux.git virtio/msg-rfc-v1 [4] From linux-next: 5be53630b4f0 virtio-mmio: Remove virtqueue list from mmio device Viresh Kumar (6): of: reserved-memory: Add reserved_mem_device_init() of: reserved-memory: Add of_reserved_mem_lookup_by_name virtio: Add support for virtio-msg transport virtio-msg: Add optional userspace interface for message I/O virtio-msg: Add support for FF-A (Firmware Framework for Arm) bus virtio-msg: Add support for loopback bus MAINTAINERS | 7 + drivers/of/of_reserved_mem.c | 91 +++-- drivers/virtio/Kconfig | 34 ++ drivers/virtio/Makefile | 5 + drivers/virtio/virtio_msg.c | 546 +++++++++++++++++++++++++++ drivers/virtio/virtio_msg.h | 88 +++++ drivers/virtio/virtio_msg_ffa.c | 501 ++++++++++++++++++++++++ drivers/virtio/virtio_msg_loopback.c | 323 ++++++++++++++++ drivers/virtio/virtio_msg_user.c | 119 ++++++ include/linux/of_reserved_mem.h | 13 + include/uapi/linux/virtio_msg.h | 221 +++++++++++ include/uapi/linux/virtio_msg_ffa.h | 94 +++++ include/uapi/linux/virtio_msg_lb.h | 22 ++ 13 files changed, 2040 insertions(+), 24 deletions(-) create mode 100644 drivers/virtio/virtio_msg.c create mode 100644 drivers/virtio/virtio_msg.h create mode 100644 drivers/virtio/virtio_msg_ffa.c create mode 100644 drivers/virtio/virtio_msg_loopback.c create mode 100644 drivers/virtio/virtio_msg_user.c create mode 100644 include/uapi/linux/virtio_msg.h create mode 100644 include/uapi/linux/virtio_msg_ffa.h create mode 100644 include/uapi/linux/virtio_msg_lb.h -- 2.31.1.272.g89b43f80a514

4 months, 2 weeks

3
20
0 0

Virtio message loopback

by Viresh Kumar

Hello, As discussed yesterday during the call, I have prepared the list of instructions to reproduce my virtio-msg loopback setup. Please give a try and lemme know if something doesn't work. This is based on the latest version of the spec (from Bill's latest pull request today). -- Viresh [1] https://linaro.atlassian.net/wiki/spaces/HVAC/pages/30104092673 -- viresh

5 months, 1 week

2
2
0 0

[PATCH RFC v2 0/1] virtio-msg transport layer

by Bill Mills

This series adds the virtio-msg transport layer. The individuals and organizations involved in this effort have had difficulty in using the existing virtio-transports in various situations and desire to add one more transport that performs its transport layer operations by sending and receiving messages. Implementations of virtio-msg will normally be done in multiple layers: * common / device level * bus level The common / device level defines the messages exchanged between the driver and a device. This common part should lead to a common driver holding most of the virtio specifics and can be shared by all virtio-msg bus implementations. The kernel implementation in [3] shows this separation. As with other transport layers, virtio-msg should not require modifications to existing virtio device implementations (virtio-net, virtio-blk etc). The common / device level is the main focus of this version of the patch series. The virtio-msg bus level implements the normal things a bus defines (enumeration, dma operations, etc) but also implements the message send and receive operations. A number of bus implementations are envisioned, some of which will be reusable and general purpose. Other bus implementations might be unique to a given situation, for example only used by a PCIe card and its driver. How much of the bus level should be described in the virtio spec is one item we wish to discuss. This draft takes a middle approach by describing the bus level and defining some standard bus level messages that MAY be used by the bus. It also describes a range of bus messages that are implementation dependent. The standard bus messages are an effort to avoid different bus implementations doing the same thing in different ways for no good reason. However the different environments will require different things. Instead of trying to anticipate all needs and provide something very abstract, we think implementation specific messages will be needed at the bus level. Over time, if we see similar messages across multiple bus implementations, we will move to standardize a bus level message for that. We are working on two reusable bus implementations: * virtio-msg-ffa based on Arm FF-A interface for use between: * normal world and secure world * host and VM or VM to VM * Can be used w/ or with out a hypervisor * Any Hypervisor that implements FF-A can be used * virtio-msg-amp for use between heterogenous systems * The main processors and its co-processors on an AMP SOC * Two or more systems connected via PCIe * Minimal requirements: bi-directional interrupts and at least one shared memory area We also anticipate a few more: * virtio-msg-xen specific to Xen * Usable on any Xen system (including x86 where FF-A does not exist) * Using Xen events and page grants * virtio-msg-loopback for userspace implemented devices * Allows user space to provide devices to its own kernel * This is similar to fuse, cuse or loopback block devices but for virtio * Once developed this can provide a single kernel demo of virtio-msg * [Work has begun on this] * virtio-msg over admin virtqueues * This allows any virtio-pci device that supports admin virtqueues to also support a virtio-msg bus that supports sub devices * [We are looking for collaborators for this work] Changes since RFC1: * reformated document to better conform to the virtio spec style - created an introduction chapter - created a basic concept chapter - created bus operation and device initialization and operation chapters - reworked description of transport and bus messages - attempted a "compliance chapter" - reused spec macros - switched to MAY/MUST/SHALL/SHOULD wording - eliminate the use of terms front-end and back-end and use driver and device * made the maximum message size variable per bus instance * use "device number" for virtio-msg device instances on the bus instead of adding yet another meaning for "device ID" * added the configuration generation count and its use * described types of things that can be done with bus specific messages such as setup of bus level shared memory and out of band notifications * removed wording of notification being optional at transport level and described bus level notifications of in-band, out-of-band, and polled from driver side bus * removed the ERROR message from transport level. Errors should be handled at the bus level to better match virtio-pci and virtio-mmio * removed bus level reset and status from standard bus messages * replaced bus messages DEVICE_{ADDED,REMOVED} with EVENT_DEVICE * changed names to GET_DEVICE_FEATURES and SET_DRIVER_FEATURES for clarity * made SET_DEVICE_STATE return new state as it may not match * Don't echo back the data in SET_VQUEUE (it cannot change) * defined request/response vs event message id ranges * match field size of next offset and wrap to virtio-{pci,mmio} * added maximum number of virtqueues to DEVICE_INFO * added admin virtqueue and SHM support This series is a work in progress and we acknowledge at least the following issues we need to work on: * Better conformance documentation * Publish an update to Arm FF-A spec that shows virtio-msg binding (work underway) * Publish virtio-msg-amp data structures and messages somewhere * Align implementations to this version and send PATCH v1 (non-rfc) Background info and work in progress implementations: * HVAC project page with intro slides [1] * HVAC demo repo w/ instructions in README.md [2] * Kernel w/ virtio-msg common level and ffa support [3] * QEMU w/ support for one form of virtio-msg-amp [4] * Portable RTOS library w/ one form of virtio-msg-amp [5] In addition to the QEMU system based demos in the hvac-demo repo, we also have two hardware systems running: * AMD x86 + AMD Arm Versal connected via PCIe * ST STM32MP157 A7 Linux using virtio-i2c provided by M4 Zephyr Please note that although the demos work, they are not yet aligned with each other nor this version of the spec. [1] https://linaro.atlassian.net/wiki/spaces/HVAC/overview [2] https://github.com/wmamills/hvac-demo [3] https://git.kernel.org/pub/scm/linux/kernel/git/vireshk/linux.git/log/?h=vi… [4] https://github.com/edgarigl/qemu/commits/edgar/virtio-msg-new [5] https://github.com/arnopo/open-amp/commits/virtio-msg/ Bill Mills (1): virtio-msg: Add virtio-msg, a message based virtio transport layer content.tex | 1 + transport-msg.tex | 1354 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 1355 insertions(+) create mode 100644 transport-msg.tex -- 2.34.1

5 months, 2 weeks

4
5
0 0

PR #20, fix-up pending changes before RFCv2

by Bill Mills

All, The pull request referenced here covers the remaining items needed before sending RFCv2. Please take a look and let me know if you see issues. You can comment in the PR or send email to the list. https://github.com/Linaro/virtio-msg-spec/pull/20 Thanks, Bill -- Bill Mills Principal Technical Consultant, Linaro +1-240-643-0836 TZ: US Eastern Work Schedule: Tues/Wed/Thur

5 months, 3 weeks

1
0
0 0

[PATCH] firmware: arm_ffa: Fix struct ffa_indirect_msg_hdr

by Viresh Kumar

As per the spec, one 32 bit reserved entry is missing here, add it. Signed-off-by: Viresh Kumar <viresh.kumar(a)linaro.org> --- include/linux/arm_ffa.h | 1 + 1 file changed, 1 insertion(+) diff --git a/include/linux/arm_ffa.h b/include/linux/arm_ffa.h index 5bded24dc24f..e1634897e159 100644 --- a/include/linux/arm_ffa.h +++ b/include/linux/arm_ffa.h @@ -283,6 +283,7 @@ struct ffa_indirect_msg_hdr { u32 offset; u32 send_recv_id; u32 size; + u32 res1; uuid_t uuid; }; -- 2.31.1.272.g89b43f80a514

6 months, 1 week

1
0
0 0

[PATCH] virtio-mmio: Remove virtqueue list from mmio device

by Viresh Kumar

The MMIO transport implementation creates a list of virtqueues for a virtio device, while the same is already available in the struct virtio_device. Don't create a duplicate list, and use the other one instead. While at it, fix the virtio_device_for_each_vq() macro to accept an argument like "&vm_dev->vdev" (which currently fails to build). Signed-off-by: Viresh Kumar <viresh.kumar(a)linaro.org> --- I have tested this with a Xen/Qemu based setup to test guest to host virtio-i2c communication. Works fine, not sure why this code was added in the first place. drivers/virtio/virtio_mmio.c | 52 +++--------------------------------- include/linux/virtio.h | 2 +- 2 files changed, 4 insertions(+), 50 deletions(-) diff --git a/drivers/virtio/virtio_mmio.c b/drivers/virtio/virtio_mmio.c index 5d78c2d572ab..b152a1eca05a 100644 --- a/drivers/virtio/virtio_mmio.c +++ b/drivers/virtio/virtio_mmio.c @@ -65,7 +65,6 @@ #include <linux/platform_device.h> #include <linux/pm.h> #include <linux/slab.h> -#include <linux/spinlock.h> #include <linux/virtio.h> #include <linux/virtio_config.h> #include <uapi/linux/virtio_mmio.h> @@ -88,22 +87,8 @@ struct virtio_mmio_device { void __iomem *base; unsigned long version; - - /* a list of queues so we can dispatch IRQs */ - spinlock_t lock; - struct list_head virtqueues; -}; - -struct virtio_mmio_vq_info { - /* the actual virtqueue */ - struct virtqueue *vq; - - /* the list node for the virtqueues list */ - struct list_head node; }; - - /* Configuration interface */ static u64 vm_get_features(struct virtio_device *vdev) @@ -300,9 +285,8 @@ static bool vm_notify_with_data(struct virtqueue *vq) static irqreturn_t vm_interrupt(int irq, void *opaque) { struct virtio_mmio_device *vm_dev = opaque; - struct virtio_mmio_vq_info *info; + struct virtqueue *vq; unsigned long status; - unsigned long flags; irqreturn_t ret = IRQ_NONE; /* Read and acknowledge interrupts */ @@ -315,10 +299,8 @@ static irqreturn_t vm_interrupt(int irq, void *opaque) } if (likely(status & VIRTIO_MMIO_INT_VRING)) { - spin_lock_irqsave(&vm_dev->lock, flags); - list_for_each_entry(info, &vm_dev->virtqueues, node) - ret |= vring_interrupt(irq, info->vq); - spin_unlock_irqrestore(&vm_dev->lock, flags); + virtio_device_for_each_vq(&vm_dev->vdev, vq) + ret |= vring_interrupt(irq, vq); } return ret; @@ -329,14 +311,8 @@ static irqreturn_t vm_interrupt(int irq, void *opaque) static void vm_del_vq(struct virtqueue *vq) { struct virtio_mmio_device *vm_dev = to_virtio_mmio_device(vq->vdev); - struct virtio_mmio_vq_info *info = vq->priv; - unsigned long flags; unsigned int index = vq->index; - spin_lock_irqsave(&vm_dev->lock, flags); - list_del(&info->node); - spin_unlock_irqrestore(&vm_dev->lock, flags); - /* Select and deactivate the queue */ writel(index, vm_dev->base + VIRTIO_MMIO_QUEUE_SEL); if (vm_dev->version == 1) { @@ -347,8 +323,6 @@ static void vm_del_vq(struct virtqueue *vq) } vring_del_virtqueue(vq); - - kfree(info); } static void vm_del_vqs(struct virtio_device *vdev) @@ -375,9 +349,7 @@ static struct virtqueue *vm_setup_vq(struct virtio_device *vdev, unsigned int in { struct virtio_mmio_device *vm_dev = to_virtio_mmio_device(vdev); bool (*notify)(struct virtqueue *vq); - struct virtio_mmio_vq_info *info; struct virtqueue *vq; - unsigned long flags; unsigned int num; int err; @@ -399,13 +371,6 @@ static struct virtqueue *vm_setup_vq(struct virtio_device *vdev, unsigned int in goto error_available; } - /* Allocate and fill out our active queue description */ - info = kmalloc(sizeof(*info), GFP_KERNEL); - if (!info) { - err = -ENOMEM; - goto error_kmalloc; - } - num = readl(vm_dev->base + VIRTIO_MMIO_QUEUE_NUM_MAX); if (num == 0) { err = -ENOENT; @@ -463,13 +428,6 @@ static struct virtqueue *vm_setup_vq(struct virtio_device *vdev, unsigned int in writel(1, vm_dev->base + VIRTIO_MMIO_QUEUE_READY); } - vq->priv = info; - info->vq = vq; - - spin_lock_irqsave(&vm_dev->lock, flags); - list_add(&info->node, &vm_dev->virtqueues); - spin_unlock_irqrestore(&vm_dev->lock, flags); - return vq; error_bad_pfn: @@ -481,8 +439,6 @@ static struct virtqueue *vm_setup_vq(struct virtio_device *vdev, unsigned int in writel(0, vm_dev->base + VIRTIO_MMIO_QUEUE_READY); WARN_ON(readl(vm_dev->base + VIRTIO_MMIO_QUEUE_READY)); } - kfree(info); -error_kmalloc: error_available: return ERR_PTR(err); } @@ -627,8 +583,6 @@ static int virtio_mmio_probe(struct platform_device *pdev) vm_dev->vdev.dev.release = virtio_mmio_release_dev; vm_dev->vdev.config = &virtio_mmio_config_ops; vm_dev->pdev = pdev; - INIT_LIST_HEAD(&vm_dev->virtqueues); - spin_lock_init(&vm_dev->lock); vm_dev->base = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(vm_dev->base)) { diff --git a/include/linux/virtio.h b/include/linux/virtio.h index 64cb4b04be7a..8b745ce0cf5f 100644 --- a/include/linux/virtio.h +++ b/include/linux/virtio.h @@ -196,7 +196,7 @@ int virtio_device_reset_done(struct virtio_device *dev); size_t virtio_max_dma_size(const struct virtio_device *vdev); #define virtio_device_for_each_vq(vdev, vq) \ - list_for_each_entry(vq, &vdev->vqs, list) + list_for_each_entry(vq, &(vdev)->vqs, list) /** * struct virtio_driver - operations for a virtio I/O driver -- 2.31.1.272.g89b43f80a514

6 months, 3 weeks

1
0
0 0

Re: Discussion Virtio with pKVM / FFA

by Viresh Kumar

Hello everyone, Moving the discussion from google doc [1] to email (Sorry, it was becoming impossible to follow and reply there). FWIW, I am not the best person to answer all questions here, that would be Armelle as she understand the requirements and the end goal much better than I do. I can though try to answer from kernel's perspective, based on whatever implementation we have right now. AFAIK, the broad idea is to implement two virtio communication paths from pVM, one to the Linux host (via virtio-pci) and another one to Trusty (via virtio-msg-ffa). In order to not take performance hit at runtime (to map io buffers), the idea is to map whatever amount of memory we can at the beginning and then keep allocating from there. Current setup: What we have achieved until now is virtio-msg communication between host and trusty. We have implemented FFA specific dma-hal [2] to perform FFA memory sharing with trusty. With "reserved-mem" and "memory-region" DT entries (not sure if that is the final solution), we are able to allocate memory the FFA device (which represents bus for all the enumerated devices between trusty/host). This memory is shared with trusty at probe time (from virtio-msg-ffa layer) and the DMA hal later allocates memory from there for coherent allocations and bounce buffers. This works just fine right now. Now looking at "dynamic mapping" section in [1] we are not sure if that will work fine for the end use case, pVM to trusty. It looks like the coco implementation will always end up using dma encrypt/decrypt when a pVM is running and share the memory with host, even when all we want to do is share with trusty. Is that understanding correct ? We would also want to establish virtio-pci (existing tech) based communication between pVM and host, which should use mem decrypt path (?). I am not sure if we need contiguous PA here, contiguous IPA should be sufficient, Armelle? We are also looking for further suggestions to improve the design, as my understanding of memory mapping, dma hal etc. is limited, and maybe there are better ways to do this. -- Viresh [1] https://docs.google.com/document/d/1KyxclKngQ0MShX8Q1YPstZQVIDIctkxfj6lQ0V3… [2] https://web.git.kernel.org/pub/scm/linux/kernel/git/vireshk/linux.git/tree/…

7 months, 1 week

4
14
0 0

My spec review list

by Bill Mills

Bertrand, I know you are on vacation but here is my full list. It took longer to flesh out from my pencil notes than I thought. While you are gone I will try to tackle all of these these in a PR w/ one commit per item. I will at least get to some of them. I will create issues for things I think need discussion. Thanks, Bill -------------- Global: * Eliminate all use of "frontend" and "backend", can use device side and driver side when just device or driver is not enough * Use device number (dev_num) instead of Device ID to avoid confusion with device type (called Device ID in spec) * Address out of band notification as an option, any polling that is required should be done by the bus implementation, not the common layer. * for all EVENT_ messages change "does not require a response" to "does not have a response". There is no optional response; a response is forbidden. 4.4.2.1 Bus level feature bits * are these local to the bus implementation or communicated to the virtio-msg transport layer? (I vote local) * what would be an example of a bus feature that would need to be known at the common layer that is NOT a device feature? 4.4.2.1.1 max message size definition. Does this include or exclude headers added by the bus implementation? I suggest it excludes it. 4.4.2.1.1 max message size of 64K * This seems a bit excessive. It may cause people to invent new uses for the message channel that we do not intend. * Should we lower the max message size? * Should we add a note that says no use of message sizes above 288 bytes is envisioned? (288 = 256 bytes of config data plus 32 bytes for message overhead. Right now message overhead is 18 for SET_CONFIG so 32 is plenty but 16 is too small.) 4.4.2.3 Config Generation Count * (self note: Config gen count only needs to be incremented once per *driver visible* change A device can do a whole set of changes and MAY/SHOULD only increment the gen count by one until the new gen count is published to the other side via EVENT_CONFIG or GET/SET_CONFIG However this is basically the same thing from the spec POV. ) * Should we say that Config gen count starts at 0 (SHOULD? MUST?) * Does config gen count reset at device reset or not? If not then maybe we don't say it starts at zero either and the driver MUST query it You do not need to know the config gen counter to GET_CONFIG, only to SET_CONFIG 4.4.2.5.1 Error codes * Are error codes aligned to admin queues yet? 4.4.2.6 Bus vs Transport messages * Mention shared memory setup and out of band notification in bus message overview * Mention bus implementation specific messages in bus message overview * Remove BUS_MSG_STATUS for now (or allow request/response and maybe add EVENT_ version w/o response) 4.4.2.8 dev_id to dev_num or dev_number 4.4.3.3 Resetting the BUS * We decided to drop this for now * When we do it graceful and abortive reset should be handled * For graceful device initiated reset, it should be handled by setting state to NEEDS_RESET 4.4.4.2 Device Info * add max number of virtqueues (or define a null response for GET_VQ) 4.4.4.3.1 error if *index* is beyond elsewhere it says pad with zeros (why do we need this just return zeros) 4.4.4.5 we need max number of virtqueue in DEVICE_INFO OR return maximum size of 0 for all unsupported indexes on GET and ERROR on set 4.4.4.6 Mention Device status change w/o config change as another possible reason to get 4.4.4.7 Actually virtqueues and come and go during the life of the device. The STATUS can (should?) be set to DRIVER OK before any virtqueues are set. 4.4.5.1 * what are NOTIFY_ON_AVAIL and NEGOCIATE_DATA (sp)? These appear no where else in the spec. * all existing transports have a way to notify driver -> device per virtqueue 4.4.5.1.1 * virtqueue id is always present * If VIRTIO_F_NOTIFICATION_DATA has not been negotiated then next_off and next_wrap should be 0 in message (Bertrand did not like using msg_length to exclude them). * We might want to update the text for a better description or just delete it and leave it to the VIRTIO_F_NOTIFICATION_DATA feature description. * Should we support VIRTIO_F_NOTIF_CONFIG_DATA? If so then virtqueue_index can be vq_notif_config_data instead 4.4.5.2 Device Notifications * Maybe here would be a place to mention OoB notifications and polling? * I don't think this is a MAY from the transport level. * The Bus MUST give these messages to the transport level. It can get them from messages on the bus, from OoB notifications, or generating them periodically to stimulate polling. 4.4.5.2.1 EVENT_CONFIG * sent on config OR STATUS change * we talked about a new version of this message with no data to use for OoB and polling stimulation 4.4.5.2.2 EVENT_USED * should we define virtqueue index of -1 as "all virtqueues? * we could put the loop in the bus instead, this would handle the case of 2 virtqueues on MSIX #2 and 1 one virtqueue on MSIX #3 4.4.5.3 Configuration Changes During Operation * Feature re-negotiation is not allowed by the spec. drop this part of the text From 2.2 Feature Bits: """ Each virtio device offers all the features it understands. During device initialization, the driver reads this and tells the device the subset that it accepts. The only way to renegotiate is to reset the device. """ 4.4.5.4 Device Rest and Shutdown * reword last sentence, it is valid to not re-init the device if your not going to use it again 4.4.5.4.1 Device-Initated Rest * match the other transports here. The device sends an async status change with DEVICE_NEEDS_RESET set and waits for driver to reset the device via a SET_STATUS of 0. 4.4.6.1.1 Bigger gap between request response messages and EVENT_ messages * Not a lot of space to add new request/response messages * Renumber to 0x80, 0x81, 0x82 4.4.6.2 VIRTIO_MSG_ERROR * no way to match an error to multiple outstanding messages of the same type ** Right now we assume we only have one outstanding request message per device. This will probibly be true in Linux. However a different OS could choose to do multiple SET_VQ in parrallel (for example). There is no way to coordinate which SET_VQ a VIRTIO_MSG_ERROR pairs with. ** Solutions 1) add a transaction id at the bus level and do all matching based on that 2) add a transaction id at the transport common header 3) add another field to the error message that would differenciate (index for VQ). This makes matching logic per msg_id 4) assume one message per device serialization 5) #4 UNLESS #1 4.4.6.3 VIRTIO_MSG_GET_DEVICE_INFO * add max num virtqueues * add number of Shared memory Regions (section 2.3) (do this in same patch that adds GET_SHM message) 4.4.6.4 VIRTIO_MSG_GET_FEATURES * Most transports have device features and driver features and allow both to be read. device features always reflect the capabilities of the device. driver features are written with the feastures the driver wants to use and read back the negotiated features It is clear that the SET_FEATURES writes the driver desired features and gets back the negotiated features What does GET_FEATURES read? I presume it reads device features before negotiation and negotiated features after negotiation * Should we add a param to specify DEVICE to DRIVER features to the GET_FEATURES message for parity with other transports? 4.4.6.4 & .5 * make it clear that a feature block is a set of 32 feature bits * feature bits beyond the number of max feature bits always read as zero 4.4.6.9 SET_DEVICE_STATUS * return the new status. It is vald for the device to refuse to set the FEATURES_OK bit. (3.1.1 number 6) A set status might also cause a DEVICE_NEEDS_RESET. 4.4.6.11 SET_VQUEUE * why is the info echoed back, can it ever change? 4.4.6.12 RESET_VQUEUE * Does this need to be negotiated? * VIRTIO_F_RING_RESET is the feature bit 4.4.6.13 VIRTIO_MSG_EVENT_CONFIG * see above about adding no data version of this 4.4.6.14 VIRTIO_MSG_EVENT_AVAIL * Next offset and next wrap should be 0 if not negotiated * should we add virtqueue index == -1 means all virtqueues (this would be use on device side between bus and transport for OoB or polling stimulation 4.4.6.15 VIRTIO_MSG_EVENT_USED * should we add virtqueue index == -1 means all virtqueues (this would be use on driver side bus and transport for OoB or polling stimulation and gives parity with virtio-mmio and base level of virtio-pci) 4.4.7.2 BUS_MSG_GET_DEVICES * needs better description of bit packing * offset is in bits, bytes, 32 bit words?? * number of device_nums requested, this is number of bits or number of packed data bytes / words. Must be multiple of * better description on how bits are packed into array 4.4.7.4 BUS_MSG_DEVICE_REMOVED * how do we handle a graceful eject vs an abortive one? * add indicator A) already removed, deal with it B) removing now, don't start new data transfers, I will finish existing ones C) removing soon, write any dirty data and then remove or reset device yourself. May escalate to B or A if driver does not take action D) ask to eject, can be refused Add BUS_MSG_DEVICE_REMOVE as driver to device MSG. Driver side say it is done with this device. 4.4.7.6 BUS_MSG_STATUS * lots of problems with this being a no response message * remove from spec for now 4.4.8.* Conformance (Initial thoughts. I did not review this section rigorously.) * Most of this is "do the right thing" without any specifics. * Other transports spread the requirements into the various sections. We should probibly do the same. One thing that needs to be specified is when to return ERROR and when to return "benign data". Ex: getting feature bits that don't exist, return zeros or return error Ex: setting features bit that don't exist to zero; OK or error Ex: setting feature bits that don't exist to 1; return negotiated 0 or return error The mmio transport will just return zeros for these cases. Why can't we? * We need to thing about notifications and events. I don't think they are optional. They are not optional in virtio-mmio and virtio-pci. 4.4.8.2.1 General Transport Requirements I don't see a precedent for common requirements. If no precedent is found, repeat the requirements in device and driver sections. 4.4.8.6 Versioning and Forward Compatibility "" If a bus instance or device does not support an advanced feature, it MUST reject or ignore those requests cleanly using VIRTIO_MSG_ERROR, rather than undefined behavior. """ Reject is via MSG_ERROR that is clear. If we negotiate a protocol version do we expect to send unknown messages? The option to Ignore messages we don't understand seems wrong as we don't know if they are important. -- Bill Mills Principal Technical Consultant, Linaro +1-240-643-0836 TZ: US Eastern Work Schedule: Tues/Wed/Thur

8 months, 1 week

1
0
0 0

2025

2024

Virtio-msg ----- 2025 ----- December 2025 November 2025 October 2025 September 2025 August 2025 July 2025 June 2025 May 2025 April 2025 March 2025 February 2025 January 2025 ----- 2024 ----- December 2024 November 2024 October 2024 September 2024

Virtio-msg