Hey all, new member here!
As the subject line suggests, our development hardware does not support UniPro so we were looking at a guest kernel solution with TCP/IP transport over gbridge.
However, one of the SDIO bus devices is a wifi module that requires DMA. Is this possible over the current gb-netlink/gbridge to your knowledge?
Cheers,
On Fri, Apr 16, 2021 at 09:13:18AM -0400, Kyle Harding wrote:
Hey all, new member here!
As the subject line suggests, our development hardware does not support UniPro so we were looking at a guest kernel solution with TCP/IP transport over gbridge.
However, one of the SDIO bus devices is a wifi module that requires DMA. Is this possible over the current gb-netlink/gbridge to your knowledge?
Where is the DMA happening here? Across the network? Or on the system that actually has the SDIO device hardware? If the latter, you should be fine, right?
thanks,
greg k-h
My understanding is that the system with the SDIO hardware will write packets directly to shared memory, and thus will be unavailable to be read by the networking stack on the remote/virtual system.
But I will also admit that I started learning about DMA recently so I could be misunderstanding something crucial.
On Fri, Apr 16, 2021 at 9:25 AM Greg KH gregkh@linuxfoundation.org wrote:
On Fri, Apr 16, 2021 at 09:13:18AM -0400, Kyle Harding wrote:
Hey all, new member here!
As the subject line suggests, our development hardware does not support UniPro so we were looking at a guest kernel solution with TCP/IP
transport
over gbridge.
However, one of the SDIO bus devices is a wifi module that requires DMA.
Is
this possible over the current gb-netlink/gbridge to your knowledge?
Where is the DMA happening here? Across the network? Or on the system that actually has the SDIO device hardware? If the latter, you should be fine, right?
thanks,
greg k-h
On Fri, Apr 16, 2021 at 09:34:01AM -0400, Kyle Harding wrote:
My understanding is that the system with the SDIO hardware will write packets directly to shared memory, and thus will be unavailable to be read by the networking stack on the remote/virtual system.
The SDIO hardware controller is off on the "device" side of a greybus connection, not on the host, so if any DMA needs to happen, that device side handles it.
The SDIO hardware is not on the "host" side, perhaps you are getting confused as to how a greybus protocol is supposed to be used?
Think of it as a SDIO device that is plugged into a system by a USB connection. The USB data traveling to the device doesn't care about SDIO dma stuff, as the whole SDIO controller hardware is in the USB device itself.
You can think of greybus as just a way to tell a device "hey, here's some SDIO data, can you read/write it please?" The transport on which that message goes on does not matter, which is why you can do this over a network connection.
hope this helps,
greg k-h
That does help, thank you Greg! It seems that in general the transport shouldn't be a problem.
What if the non-hardware side is a kvm guest kernel without IOMMU support? If we used greybus to expose the SDIO bus to the kvm would the DMA transfer garbage in Host Memory?
I understand this is a non-traditional approach, and I believe that with IOMMU this wouldn't be an issue.
Thanks, Kyle
On Sat, Apr 17, 2021 at 2:08 AM Greg KH gregkh@linuxfoundation.org wrote:
On Fri, Apr 16, 2021 at 09:34:01AM -0400, Kyle Harding wrote:
My understanding is that the system with the SDIO hardware will write packets directly to shared memory, and thus will be unavailable to be
read
by the networking stack on the remote/virtual system.
The SDIO hardware controller is off on the "device" side of a greybus connection, not on the host, so if any DMA needs to happen, that device side handles it.
The SDIO hardware is not on the "host" side, perhaps you are getting confused as to how a greybus protocol is supposed to be used?
Think of it as a SDIO device that is plugged into a system by a USB connection. The USB data traveling to the device doesn't care about SDIO dma stuff, as the whole SDIO controller hardware is in the USB device itself.
You can think of greybus as just a way to tell a device "hey, here's some SDIO data, can you read/write it please?" The transport on which that message goes on does not matter, which is why you can do this over a network connection.
hope this helps,
greg k-h
On Sun, Apr 18, 2021 at 01:31:23PM -0400, Kyle Harding wrote:
That does help, thank you Greg! It seems that in general the transport shouldn't be a problem.
What if the non-hardware side is a kvm guest kernel without IOMMU support? If we used greybus to expose the SDIO bus to the kvm would the DMA transfer garbage in Host Memory?
I understand this is a non-traditional approach, and I believe that with IOMMU this wouldn't be an issue.
Again, I think you are getting very confused as to what the greybus protocol stack is for.
You can use it for any type of bus you want, you just need to write a driver for that transport layer. But for talking to virtual machines, why not just use the virtio layer which is designed just for that? You could make a virtio transport for greybus, but really, why? Who would use that?
And again, IOMMUs have nothing to do with this at all.
thanks,
greg k-h
Thanks Greg, I've explained our solution poorly so I'll quote a colleague as I'm too close to the problem at this point!
We'd like to essentially run a virtualized vendor kernel to manage a hardware device such as an SDIO wireless card passed through from the host, running a mainline kernel. Network traffic would be routed from host to guest over a virtio interface.
Is it possible to use Greybus in this way, or am I misunderstanding the use case of this subsystem?
On Mon, Apr 19, 2021 at 3:38 AM Greg KH gregkh@linuxfoundation.org wrote:
On Sun, Apr 18, 2021 at 01:31:23PM -0400, Kyle Harding wrote:
That does help, thank you Greg! It seems that in general the transport shouldn't be a problem.
What if the non-hardware side is a kvm guest kernel without IOMMU
support?
If we used greybus to expose the SDIO bus to the kvm would the DMA
transfer
garbage in Host Memory?
I understand this is a non-traditional approach, and I believe that with IOMMU this wouldn't be an issue.
Again, I think you are getting very confused as to what the greybus protocol stack is for.
You can use it for any type of bus you want, you just need to write a driver for that transport layer. But for talking to virtual machines, why not just use the virtio layer which is designed just for that? You could make a virtio transport for greybus, but really, why? Who would use that?
And again, IOMMUs have nothing to do with this at all.
thanks,
greg k-h
A: http://en.wikipedia.org/wiki/Top_post Q: Were do I find info about this thing called top-posting? A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? A: Top-posting. Q: What is the most annoying thing in e-mail?
A: No. Q: Should I include quotations after my reply?
http://daringfireball.net/2007/07/on_top
On Mon, Apr 19, 2021 at 02:59:56PM -0400, Kyle Harding wrote:
Thanks Greg, I've explained our solution poorly so I'll quote a colleague as I'm too close to the problem at this point!
We'd like to essentially run a virtualized vendor kernel to manage a hardware device such as an SDIO wireless card passed through from the host, running a mainline kernel. Network traffic would be routed from host to guest over a virtio interface.
Is it possible to use Greybus in this way, or am I misunderstanding the use case of this subsystem?
You could abuse greybus to do something like this, but that feels like a ton of extra work when there are already solutions for this type of thing today in Linux. Why not just use some of the existing hardware pass-through solutions that are out there that allow virtual kernels to have access to hardware directly? Why try to create yet-another way to do this?
But hey, if you want to take the time to write such kernel code, all the best! :)
greg k-h
On Tue, Apr 20, 2021 at 3:51 AM Greg KH gregkh@linuxfoundation.org wrote:
A: http://en.wikipedia.org/wiki/Top_post Q: Were do I find info about this thing called top-posting? A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? A: Top-posting. Q: What is the most annoying thing in e-mail?
A: No. Q: Should I include quotations after my reply?
TIL about top-posting. Thanks for enlightening me!
On Mon, Apr 19, 2021 at 02:59:56PM -0400, Kyle Harding wrote:
Thanks Greg, I've explained our solution poorly so I'll quote a
colleague
as I'm too close to the problem at this point!
We'd like to essentially run a virtualized vendor kernel to manage a hardware device such as an SDIO wireless card passed through from the
host,
running a mainline kernel. Network traffic would be routed from host to guest over a virtio interface.
Is it possible to use Greybus in this way, or am I misunderstanding the
use
case of this subsystem?
You could abuse greybus to do something like this, but that feels like a ton of extra work when there are already solutions for this type of thing today in Linux. Why not just use some of the existing hardware pass-through solutions that are out there that allow virtual kernels to have access to hardware directly? Why try to create yet-another way to do this?
But hey, if you want to take the time to write such kernel code, all the best! :)
greg k-h
We've explored options such as VFIO for this use case. Although VFIO works great on PCs with hardware peripherals on an I/O bus such as PCI behind an IOMMU, we're looking to passthrough a platform device (SDIO wireless specifically, but potentially others as well) on an embedded system, such as a Raspberry Pi 4. We've investigated vfio-platform intended for this, but based on a paper [1] describing the architecture and use cases, it seems to require specific hardware that implements DMA translation functionality, such as an IOMMU or SMMU, along with the appropriate driver. As we understand, a lot of the platforms we support don't have an IOMMU.
The kernel docs [2] describe the programming interface of devices as generally composed of I/O access, interrupts, and DMA.
Devices are the main target of any I/O driver. Devices typically create a programming interface made up of I/O access, interrupts, and DMA. Without going into the details of each of these, DMA is by far the most critical aspect for maintaining a secure environment as allowing a device read-write access to system memory imposes the greatest risk to the overall system integrity.
This seems to explain the need for an IOMMU to passthrough a device to a virtualized guest, as any DMA done by a device would need guest physical-addresses translated to host-physical addresses. However, we don't know for certain if the driver for our wireless adapter (mwifiex) uses DMA. Grepping the source shows references to DMA, but the fact that SDIO devices can be connected over a USB bridge seems to indicate it either doesn't, or we're not completely understanding.
We really don't know how feasible this idea is given our hardware constraints. Greybus may very likely be solving an entirely different issue, but it seemed to have enough overlap to be worth investigating. We're looking to understand if this is possible given our hardware constraints, or if we're barking up the wrong tree.
[1]: http://www.fp7-save.eu/papers/EUC2014B.pdf [2]: https://www.kernel.org/doc/Documentation/driver-api/vfio.rst
Cheers, Kyle
On Tue, Apr 20, 2021 at 08:01:06PM -0400, Kyle Harding wrote:
On Tue, Apr 20, 2021 at 3:51 AM Greg KH gregkh@linuxfoundation.org wrote:
We'd like to essentially run a virtualized vendor kernel to manage a hardware device such as an SDIO wireless card passed through from the
host,
running a mainline kernel. Network traffic would be routed from host to guest over a virtio interface.
Is it possible to use Greybus in this way, or am I misunderstanding the
use
case of this subsystem?
You could abuse greybus to do something like this, but that feels like a ton of extra work when there are already solutions for this type of thing today in Linux. Why not just use some of the existing hardware pass-through solutions that are out there that allow virtual kernels to have access to hardware directly? Why try to create yet-another way to do this?
But hey, if you want to take the time to write such kernel code, all the best! :)
greg k-h
We've explored options such as VFIO for this use case. Although VFIO works great on PCs with hardware peripherals on an I/O bus such as PCI behind an IOMMU, we're looking to passthrough a platform device (SDIO wireless specifically, but potentially others as well) on an embedded system, such as a Raspberry Pi 4.
If you are using hardware as cheap and broken as a rpi, sorry, you are on your own here. The processor in that thing is not set up for any type of virtualization, so to try to make it work that way is a lot of work for no gain.
We've investigated vfio-platform intended for this, but based on a paper [1] describing the architecture and use cases, it seems to require specific hardware that implements DMA translation functionality, such as an IOMMU or SMMU, along with the appropriate driver. As we understand, a lot of the platforms we support don't have an IOMMU.
Try it and see, who knows? You know your use case better than anyone else, and only by trying stuff will you determine what actually needs to be done.
The kernel docs [2] describe the programming interface of devices as generally composed of I/O access, interrupts, and DMA.
Devices are the main target of any I/O driver. Devices typically create a programming interface made up of I/O access, interrupts, and DMA. Without going into the details of each of these, DMA is by far the most critical aspect for maintaining a secure environment as allowing a device read-write access to system memory imposes the greatest risk to the overall system integrity.
This seems to explain the need for an IOMMU to passthrough a device to a virtualized guest, as any DMA done by a device would need guest physical-addresses translated to host-physical addresses. However, we don't know for certain if the driver for our wireless adapter (mwifiex) uses DMA. Grepping the source shows references to DMA, but the fact that SDIO devices can be connected over a USB bridge seems to indicate it either doesn't, or we're not completely understanding.
Odds are the driver itself does not require DMA, that comes from the layers below the driver that talks to the specific hardware on the platform.
And if your device is connected over a USB bridge, that shows that the driver doesn't matter here, it's up to the bus controller to do that type of work for you.
We really don't know how feasible this idea is given our hardware constraints. Greybus may very likely be solving an entirely different issue, but it seemed to have enough overlap to be worth investigating. We're looking to understand if this is possible given our hardware constraints, or if we're barking up the wrong tree.
No idea, try it and see, good luck! You have a long road ahead of you, I really would recommend using sane hardware...
greg k-h
-
Greg KH -
Kyle Harding