On Fri, Sep 27, 2024 at 3:58 PM Jason Gunthorpe jgg@nvidia.com wrote:
On Fri, Sep 27, 2024 at 02:22:20PM +0000, Mostafa Saleh wrote:
We don't support multi SID through this interface, at least in this version.
To do multi-sid you have to inform the VM of all the different pSIDs the device has and then setup the vSID/pSID translation to map them all to the HW invalidation logic.
Why would the VM need to know the pSID?
It doesn't need to know the pSID exactly, but it needs to know all the pSIDs that exist and have them be labeled with vSIDs.
With cmdq direct assignment the VM has to issue an ATS invalidation for each and every physical device using its vSID. There is no HW path to handle a 1:N v/p SID relationship.
I see, that's for the cmdq assignment.
Ah, I thought IOMMUFD would be used instead of VFIO_TYPE1*, which should cover platform devices (VFIO-platform) or am I missing something?
It does work with platform, but AFAIK nobody is interested in that so it hasn't been any focus. Enabling multi-sid nesting, sub stream ids, etc is some additional work I think.
But what I mean is the really hard use case for the vSID/pSID mapping is ATS invalidation and you won't use ATS invalidation on platform so multi-sid likely doesn't matter.
STE/CD invalidation could possibly be pushed down through the per-domain ioctl and replicated to all domain attachments. We don't have code in the series to do that, but it could work from a uAPI perspective.
If possible, can the UAPI be designed with this in mind, even if not implemented now?
It is reasonable to ask. I think things are extensible enough. I'd imagine we can add a flag 'secondary ID' and then a new field 'secondary ID index' to the vdev operations when someone wants to take this on.
Makes sense, I can take this when I start doing the pKVM work with IOMMUFD, in case it wasn't supported by then.
Thanks, Mostafa
Jason