Add a new transport layer that is based on messages.
This transport layer still uses virtqueues as the other transport layers do but implements transport layer operations by sending and receiving messages instead of the "MMR" reads and writes used in virtio-mmio and virtio-pci.
This transport is useful when the device and driver are both implemented in software but the trap and emulate operations of virtio-mmio and virtio-pci can not be used.
This transport is intended to be used in many situations, including: * between a host processor and its co-processors * between two different systems (not SMP) connected via PCIe * between normal and secure worlds * host to vm * vm to vm
This is an RFC and not yet intended to be merged.
Signed-off-by: Bill Mills bill.mills@linaro.org Signed-off-by: Bertrand Marquis bertrand.marquis@arm.com Signed-off-by: Edgar E. Iglesias edgar.iglesias@amd.com Signed-off-by: Arnaud Pouliquen arnaud.pouliquen@foss.st.com --- content.tex | 1 + transport-msg.tex | 680 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 681 insertions(+) create mode 100644 transport-msg.tex
diff --git a/content.tex b/content.tex index 67b1bf3..510e306 100644 --- a/content.tex +++ b/content.tex @@ -638,6 +638,7 @@ \chapter{Virtio Transport Options}\label{sec:Virtio Transport Options} \input{transport-pci.tex} \input{transport-mmio.tex} \input{transport-ccw.tex} +\input{transport-msg.tex}
\chapter{Device Types}\label{sec:Device Types}
diff --git a/transport-msg.tex b/transport-msg.tex new file mode 100644 index 0000000..f0d5783 --- /dev/null +++ b/transport-msg.tex @@ -0,0 +1,680 @@ +\section{Virtio Over Messages}\label{sec:Virtio Transport Options / Virtio Over Messages} + +The goal of the messages transport is to have commands communicated between the +frontend (driver) and backend (device) sides over a message based conduit +allowing to reach destination endpoints such as another VM, a TEE or a remote +compute engine using different communication channels. + +The messages transport is designed to work efficiently between VMs on an hypervisor +based configuration or between heterogeneous systems by using messages to +reduce the number of context switches between the device and the driver sides +when each of them is in a different VM. In the same way, this is also reducing +cross system communications required in an heterogeneous system configuration. + +\subsection{Messages Format}\label{sec:Virtio Transport Options / Virtio Over Messages / Messages Format} + +The messages have the following properties: +\begin{itemize} +\item A message size is 40 bytes. +\item Most messages are driver/device initialization messages. +\item Messages are expected to be sent at low frequencies and quick round-trip is + not necessary and should not impact global performances (as communication + requiring performance should use virtqueues). +\end{itemize} + +The virtio-MSG messages are encoded as following: + +\begin{tabularx}{\textwidth}{|l|l|l|X|} +\hline +Name & Offset & Size (bytes) & Content \ +\hline \hline +VIRTIO_MSG_TYPE & 0 & 1 & Bit[0]: 0=Request, 1=Answer \newline Bit[1]: 0=virtio-msg, 1=bus-msg \newline Bit[2-7]: Reserved \ +\hline +VIRTIO_MSG_ID & 1 & 1 & Message ID \ +\hline +VIRTIO_MSG_DEV_ID & 2 & 2 & Device ID, must be set to 0 if VIRTIO_MSG_TYPE Bit[1]=1\ +\hline +VIRTIO_MSG_PAYLOAD & 4 & 36 & Payload \ +\hline +\end{tabularx} + +\subsection{Errors Definition}\label{sec::Virtio Transport Options / Virtio Over Messages / Errors Definition} + +The Virtio Message transport defines a list of error codes that are used in +the VIRTIO_MSG_ERROR message. + +\begin{tabular}{|l|l|l|} +\hline +Error Code & Name & Description \ +\hline \hline +1 & EINVAL & Invalid parameter or feature \ +2 & ENOTSUPP & Unsupported function or feature \ +3 & EBUSY & Device or ressource unavailable \ +4 & EABORT & Request was aborted \ +5 & EIO & Input/Output error \ +6 & ENOMEM & No memory available \ +7 & EPERM & Permission denied \ +8 & EFAULT & Bad address \ +9 & ENODEV & No such device \ +\hline +\end{tabular} + + +\subsection{Messages Definition}\label{sec:Virtio Transport Options / Virtio Over Messages / Messages Definition} + +The following table is listing the different Virtio-MSG messages and the sender +for each of them. + +\begin{tabular}{|l|l|l|} +\hline +Name & ID & Sender \ +\hline +\hline +Reserved & 0x0 & \ +\hline +VIRTIO_MSG_ERROR & 0x1 & Any \ +\hline +VIRTIO_MSG_GET_DEVICE_INFO & 0x2 & Driver \ +\hline +VIRTIO_MSG_GET_FEATURES & 0x3 & Driver \ +\hline +VIRTIO_MSG_SET_FEATURES & 0x4 & Driver \ +\hline +VIRTIO_MSG_GET_CONFIG & 0x5 & Driver \ +\hline +VIRTIO_MSG_SET_CONFIG & 0x6 & Driver \ +\hline +VIRTIO_MSG_GET_CONFIG_GEN & 0x7 & Driver \ +\hline +VIRTIO_MSG_GET_DEVICE_STATUS & 0x8 & Driver \ +\hline +VIRTIO_MSG_SET_DEVICE_STATUS & 0x9 & Driver \ +\hline +VIRTIO_MSG_GET_VQUEUE & 0xA & Driver \ +\hline +VIRTIO_MSG_SET_VQUEUE & 0xB & Driver \ +\hline +VIRTIO_MSG_RESET_VQUEUE & 0xC & Driver \ +\hline +VIRTIO_MSG_EVENT_CONFIG & 0x20 & Device \ +\hline +VIRTIO_MSG_EVENT_AVAIL & 0x21 & Driver \ +\hline +VIRTIO_MSG_EVENT_USED & 0x22 & Device \ +\hline +\end{tabular} + +The following sections are giving more details of the usage of each message +and the encoding of the payload for the request and answer (when applicable). + +\newcommand{\msgdef}[1]{\subsubsection{VIRTIO_MSG_#1}\label{sec:Virtio Transport Options / Virtio Over Messages / Messages Definition / VIRTIO_MSG_#1}} + +\msgdef{ERROR} + +The error message is used to signal to the driver or the device that a received +message is not valid or cannot be addressed. It can also be used to report an +internal error to the other entity. + +This message can be received by the driver or the device and can be sent as an +answer to an other message by the driver or the device. + +The message contains a Virtio Message error code and the Message ID of the +request that triggerred the error (or 0 if the error was not in answer to a +specific request). + +The message also contains a Data Type that can be set to STRING to use the +rest of the payload to give a NULL terminating string that can be used to carry +extra printable information. + +\begin{tabular}{|l|l|l|l|} +\hline +Type & Offset & Size (bytes) & Content \ +\hline \hline +Any & 0 & 4 & Virtio Message Error Code \ + & 4 & 1 & Message ID which generated an error or 0 if none \ + & 5 & 1 & Data Type: 0 (Reserved), 1 (STRING), 2-255 (Implementation Defined) \ + & 6 & 30 & Null Terminated String if Data Type is STRING else Implementation Defined \ +\hline +\end{tabular} + +\msgdef{GET_DEVICE_INFO} + +The get device information message is used by the driver to retrieve the device +version, device ID and vendor ID. + +This message is sent by the driver to the device and expects an answer from the +device. + +\begin{tabular}{|l|l|l|l|} +\hline +Type & Offset & Size (bytes) & Content \ +\hline \hline +Request & 0 & 36 & Reserved (MBZ) \ +\hline +Answer & 0 & 4 & Device version \ +& 4 & 4 & Device ID \ +& 8 & 4 & Vendor ID \ +& 12 & 24 & Reserved (MBZ) \ +\hline +\end{tabular} + +\msgdef{GET_FEATURES} + +The get features message is used to retrieve 256 bits of feature information +from the device. The driver request features at an index and the device answers +with the features bits at 256*index. + +If there are no features available at the requested index but there are +features available after the index, the backend must return an all-zero +response. +If there are no features available at and after the requested index, the +backend must return an ERROR message to inform the frontend that it does not +need to scan features further. + +This message is sent by the driver to the device and expects an answer from the +device. + +\begin{tabular}{|l|l|l|l|} +\hline +Type & Offset & Size (bytes) & Content \ +\hline \hline +Request & 0 & 4 & Feature index \ +& 4 & 32 & Reserved (MBZ) \ +\hline +Answer & 0 & 4 & Feature index \ +& 4 & 32 & Feature data \ +\hline +\end{tabular} + +\msgdef{SET_FEATURES} + +The set features message is used by the driver to configure the features of a +device. The driver configures 256 bits a time a given index. The device +provides in the answer the actual features value at the index after having +configured the bit requested giving an opportunity for the driver to detect if +some of the bits it tried to set were not accepted. + +This message is sent by the driver to the device and expects an answer from the +device. + +\begin{tabular}{|l|l|l|l|} +\hline +Type & Offset & Size (bytes) & Content \ +\hline \hline +Request & 0 & 4 & Feature index \ +& 4 & 32 & Feature data \ +\hline +Answer & 0 & 4 & Feature index \ +& 4 & 32 & Feature data \ +\hline +\end{tabular} + +\msgdef{GET_CONFIG} + +The get configuration message is used by the driver to retrieve a part of the +configuration values of the device. The driver can request up to 256 bits at +a given offset in the device configuration space. + +This message is sent by the driver to the device and expects an answer from the +device. + +\begin{tabular}{|l|l|l|l|} +\hline +Type & Offset & Size (bytes) & Content \ +\hline \hline +Request & 0 & 3 & Configuration offset \ +& 3 & 1 & Number of bytes (1-32) \ +& 4 & 32 & Reserved (MBZ) \ +\hline +Answer & 0 & 3 & Configuration offset \ +& 3 & 1 & Number of bytes (1-32) \ +& 4 & 32 & Configuration data \ +\hline +\end{tabular} + +\msgdef{SET_CONFIG} + +The set configuration message is used by the driver to modify a part of the +configuration values of the device. The driver can modify up to 256 bits at a +given offset in the device configuration. The device answers with the actual +configuration value at the offset after the modification to give a chance to the +driver to detect changes that have not been accepted by the device. + +This message is sent by the driver to the device and expects an answer from the +device. + +\begin{tabular}{|l|l|l|l|} +\hline +Type & Offset & Size (bytes) & Content \ +\hline \hline +Request & 0 & 3 & Configuration offset \ +& 3 & 1 & Number of bytes (1-32) \ +& 4 & 32 & Configuration data \ +\hline +Answer & 0 & 3 & Configuration offset \ +& 3 & 1 & Number of bytes (1-32) \ +& 4 & 32 & Configuration data \ +\hline +\end{tabular} + +\msgdef{GET_CONFIG_GEN} + +The get configuration generation message is used by the driver to retrieve the +device configuration atomicity value. + +TODO: need help to have a complete description here. + +This message is sent by the driver to the device and expects an answer from the +device. + +\begin{tabular}{|l|l|l|l|} +\hline +Type & Offset & Size (bytes) & Content \ +\hline \hline +Request & 0 & 36 & Reserved (MBZ) \ +\hline +Answer & 0 & 4 & Atomicity value \ +& 4 & 32 & Reserved (MBZ) \ +\hline +\end{tabular} + +\msgdef{GET_DEVICE_STATUS} + +The get device status message is used by the driver to retrieve the device +status fields. + +This message is sent by the driver to the device and expects an answer from the +device. + +\begin{tabular}{|l|l|l|l|} +\hline +Type & Offset & Size (bytes) & Content \ +\hline \hline +Request & 0 & 36 & Reserved (MBZ) \ +\hline +Answer & 0 & 4 & Device status \ +& 4 & 32 & Reserved (MBZ) \ +\hline +\end{tabular} + +\msgdef{SET_DEVICE_STATUS} + +The set device status message is used by the driver to modify the device status +fields. Writing a status of 0 triggers a device reset. + +This message is sent by the driver to the device and expects an answer from the +device. + +\begin{tabular}{|l|l|l|l|} +\hline +Type & Offset & Size (bytes) & Content \ +\hline \hline +Request & 0 & 4 & Device status \ +& 4 & 32 & Reserved (MBZ) \ +\hline +Answer & 0 & 36 & Reserved (MBZ) \ +\hline +\end{tabular} + +\msgdef{GET_VQUEUE} + +The get vqueue message is used to retrieve the maximum virtqueue size and +information about the vqueue if it was already configured (all information are +0 if the virtqueue was not configured). + +This message is sent by the driver to the device and expects an answer from the +device. + +\begin{tabular}{|l|l|l|l|} +\hline +Type & Offset & Size (bytes) & Content \ +\hline \hline +Request & 0 & 4 & Virtqueue index \ +& 4 & 32 & Reserved (MBZ) \ +\hline +Answer & 0 & 4 & Virtqueue index \ +& 4 & 4 & Max virtqueue size \ +& 8 & 4 & Virtqueue size \ +& 12 & 8 & Descriptor address \ +& 20 & 8 & Driver address \ +& 28 & 8 & Device address \ +\hline +\end{tabular} + +\msgdef{SET_VQUEUE} + +The set vqueue message is used to configure a virtqueue. +Setting the virtqueue size to 0 is disabling the virtqueue without modifying +the rest of the parameters (those should be ignored by the device). +If a driver needs to complete reset a virtqueue, the RESET_VQUEUE message +should be used instead. + +This message is sent by the driver to the device and expects an answer from the +device. + +\begin{tabular}{|l|l|l|l|} +\hline +Type & Offset & Size (bytes) & Content \ +\hline \hline +Request & 0 & 4 & Virtqueue index \ +& 4 & 4 & Reserved (MBZ) \ +& 8 & 4 & Virtqueue size \ +& 12 & 8 & Descriptor address \ +& 20 & 8 & Driver address \ +& 28 & 8 & Device address \ +\hline +Answer & 0 & 4 & Virtqueue index \ +& 4 & 4 & Reserved (MBZ) \ +& 8 & 4 & Virtqueue size \ +& 12 & 8 & Descriptor address \ +& 20 & 8 & Driver address \ +& 28 & 8 & Device address \ +\hline +\end{tabular} + +\msgdef{RESET_VQUEUE} + +The reset vqueue message is used to disable and reset a virtqueue. + +This message is sent by the driver to the device and expects an answer from the +device. + +\begin{tabular}{|l|l|l|l|} +\hline +Type & Offset & Size (bytes) & Content \ +\hline \hline +Request & 0 & 4 & Virtqueue index \ +& 4 & 32 & Reserved (MBZ) \ +\hline +Answer & 0 & 36 & Reserved (MBZ) \ +\hline +\end{tabular} + +\msgdef{EVENT_CONFIG} + +The event config message is sent by the device to signal to the driver that one +or several values in the device configuration have changed. The message +contains the current device status (as encoded in the GET_DEVICE_STATUS +answer) and optionally the part of the configuration that has been modified. If +this is not provided, the driver will have to use the GET_CONFIG to retrieve +the configuration and discover what has changed. + +This message is sent by the device to the driver and does not expect any +answer. + +\begin{tabular}{|l|l|l|l|} +\hline +Type & Offset & Size (bytes) & Content \ +\hline \hline +Request & 0 & 4 & Device status \ +& 4 & 3 & Configuration offset \ +& 7 & 1 & Number of bytes (1-16) \ +& 8 & 16 & Configuration data \ +& 24 & 12 & Reserved (MBZ) \ +\hline +\end{tabular} + +\msgdef{EVENT_AVAIL} + +The event available message is sent by the driver to the device to signal that +some data are available to process in a virtqueue. If the NOTIFICATION_DATA +was negotiated, the message can also include more details on what is actually +available. + +This message is sent by the driver to the device and does not expect any +answer. + +\begin{tabular}{|l|l|l|l|} +\hline +Type & Offset & Size (bytes) & Content \ +\hline \hline +Request & 0 & 4 & Virtqueue index \ +& 4 & 4 & Next offset \ +& 8 & 4 & Next wrap \ +& 12 & 24 & Reserved (MBZ) \ +\hline +\end{tabular} + +\msgdef{EVENT_USED} + +The event used message is sent by the device to the driver to signal that some +data is available or has been processed in the a virtqueue. + +This message is sent by the device to the driver and does not expect any +answer. + +\begin{tabular}{|l|l|l|l|} +\hline +Type & Offset & Size (bytes) & Content \ +\hline \hline +Request & 0 & 4 & Virtqueue index \ +& 4 & 32 & Reserved (MBZ) \ +\hline +\end{tabular} + +\subsection{Message Bus}\label{sec:Virtio Transport Options / Virtio Over Messages / Message Bus} + +The Message Bus acts as a communication channel between two endpoints, +providing a standardized abstraction to exchange messages. + +The Message Bus transmits the messages between the frontend and the backend in +both directions. + +A system can have several Message Buses and it must respect the following +principles: + +\begin{itemize} +\item One Message Bus has a backend and a frontend side located in 2 different + endpoints (an endpoint can be a VM, a secure Partition or a remote OS). +\item One Message Bus has one or more devices on the backend side handled by + drivers on the frontend side. +\item An endpoint can be connected to several Message Buses. +\end{itemize} + +A Message Bus implementation must provide interfaces to transmit and receive +messages with the other side. + +While the bus implementation and how communication of the messages is done +between the devices and the drivers is completely out of the scope of the +specification, this chapter provides a set of Bus messages to help +standardizing as much as possible a part of the bus implementation. + +\subsubsection{Discovery}\label{sec:Virtio Transport Options / Virtio Over Messages / Message Bus / Discovery} + +The Message Bus is responsible of listing the available buses and to enumerate +all devices available on each bus. + +The frontend may use a predefined virtio devices ID list (static way), or may +use the BUS_MSG_GET_DEVICES message to discover devices IDs available on the +backend (dynamic way), or a combination of the two methods. + +The backend may use a predefined virtio devices ID list that can be provided to +the driver on BUS_MSG_GET_DEVICES request, or may announce device IDs using +the BUS_MSG_DEVICE_ADDED and BUS_MSG_DEVICE_REMOVED messages, or a combination +of the two methods. + +\subsubsection{Message Definition}\label{sec:Virtio Transport Options / Virtio Over Messages / Message Bus / Message Definition} + +The Bus messages are transferred using the \emph{bus-msg} type by setting the +Bit[1] of the VIRTIO_MSG_TYPE field. + +The following table is listing the different Virtio-MSG Bus messages and the +sender for each of them. + +\begin{tabular}{|l|l|l|} +\hline +Name & ID & Sender \ +\hline +\hline +Reserved & 0x0 & \ +\hline +BUS_MSG_ERROR & 0x1 & Any \ +\hline +BUS_MSG_GET_DEVICES & 0x2 & Driver \ +\hline +BUS_MSG_DEVICE_ADDED & 0x3 & Device \ +\hline +BUS_MSG_DEVICE_REMOVED & 0x4 & Device \ +\hline +BUS_MSG_PING & 0x5 & Any \ +\hline +BUS_MSG_RESET & 0x6 & Driver \ +\hline +\end{tabular} + +Bus message ID 6-127 are reserved for future use in the specification and +message ID 128-255 can be used for implementation specific needs. + +The following sections are giving more details of the usage of each message +and the encoding of the payload for the request and answer (when applicable). + +\newcommand{\busdef}[1]{\paragraph{BUS_MSG_#1}\label{sec:Virtio Transport Options / Virtio Over Messages / Message Bus / Message Definition / BUS_MSG_#1}} + +\busdef{ERROR} + +The Error message is used by one side of the bus to inform the other side that +one message could not be processed correctly. The message contains a Virtio +Message error code. + +This message shall be used for Bus errors not specific to a particular device. +In case of a device specific error not impacting the complete bus, the +VIRTIO_MSG_ERROR shall be used. + +This message is using the same payload format as the VIRTIO_MSG_ERROR message. + +This message can be received by the frontend or the backend side and can be +send as answer to an other message or directly to signal an error. + +\busdef{GET_DEVICES} + +The Get Devices message is used by the frontend to retrieve the IDs of the +devices available on the backend of the bus. + +The frontend requests the availability of a group of 256 device IDs and the +backend returns a bitmap indicating which device IDs are available at the +requested device ID offset. + +A request at offset X gives the availability of device IDs X*256 to (X+1)*256-1. + +The backend also returns a next offset indicating to the frontend what device +ID offset should be requested next. This is to reduce the number of messages +required to scan all IDs when the backend has device IDs far from each other. + +This message shall be used in the following way: +\begin{itemize} +\item frontend request device IDs at offset X. +\item backend answers with the availability of device IDs X*256 to (X+1)*256-1 + and indicates to the frontend which offset to request after using the next + offset. +\end{itemize} +The frontend starts by doing a request with offset 0 and then iterates using +the next offset returned by the backend until the backend gives a next offset +of 0. + +This message is sent by the frontend side of the bus to the backend side and +expects an answer from the backend side. + +\begin{tabular}{|l|l|l|l|} +\hline +Type & Offset & Size (bytes) & Content \ +\hline \hline +Request & 0 & 2 & Offset \ + & 2 & 36 & Reserved (MBZ) \ +\hline +Answer & 0 & 2 & Offset \ + & 2 & 2 & Next offset or 0 if no devices after Offset \ + & 4 & 32 & Bit[n]: Device[offset*256 + n] not available(0)/available(1) \ +\hline +\end{tabular} + +\busdef{DEVICE_ADDED} + +This message is sent by the backend of the bus to signal to the frontend that +a new device appeared on the bus. + +This message is sent by the backend of the bus and does not expect any answer. + +\begin{tabular}{|l|l|l|l|} +\hline +Type & Offset & Size (bytes) & Content \ +\hline \hline +Request & 0 & 2 & Device ID \ + & 2 & 34 & Reserved (MBZ) \ +\hline +\end{tabular} + +\busdef{DEVICE_REMOVED} + +This message is sent by the backend of the bus to signal to the frontend that +a device was removed from the bus. + +This message is sent by the backend of the bus and does not expect any answer. + +\begin{tabular}{|l|l|l|l|} +\hline +Type & Offset & Size (bytes) & Content \ +\hline \hline +Request & 0 & 2 & Device ID \ + & 2 & 34 & Reserved (MBZ) \ +\hline +\end{tabular} + +\busdef{PING} + +The ping message can be sent by the backend or the frontend side of the bus +to check that the other side is responding properly. + +This message can be used by either side to monitor that the other side of the +bus is available and functional. + +The message contains a 32bit data field with a custom value that must be copied +back by the receiver in its answer. + +\begin{tabular}{|l|l|l|l|} +\hline +Type & Offset & Size (bytes) & Content \ +\hline \hline +Request & 0 & 4 & Data \ + & 4 & 32 & Reserved (MBZ) \ +\hline +Answer & 0 & 4 & Request Data \ + & 4 & 32 & Reserved (MBZ) \ +\hline +\end{tabular} + +\busdef{STATUS} + +This message can be sent by the backend or the frontend to signal to the other +side that its own status has changed which required the other side to take +global actions. + +For example this message can be send by the frontend when it starts to +request the backend to reset the bus and all devices on it or when the backend +stops to signal to the backend to shutdown everything on the bus. + +The backend can use it to signal to the frontend that for some reason it will +shutdown which shall trigger the frontend to disable all drivers. + +The message contains a New state field which gives to the receiving the new +state the sender is going into for which the following values are defined: +\begin{itemize} +\item 0: Reset. +\item 1: Shutdown. +\item 2: Suspend. +\item 4: Resume. +\item 5-127: Reserved. +\item 128-255: Implementation Defined. +\end{itemize} + +This message can be sent by the frontend or the backend side of the bus and +does not expect any answer. + +\begin{tabular}{|l|l|l|l|} +\hline +Type & Offset & Size (bytes) & Content \ +\hline \hline +Request & 0 & 1 & New state \ + & 1 & 35 & Reserved (MBZ) \ +\hline +\end{tabular} +