On 2025-08-05 05:08 PM, Joel Granados wrote:
On Fri, Jun 20, 2025 at 11:19:58PM +0000, David Matlack wrote:
This series introduces VFIO selftests, located in tools/testing/selftests/vfio/.
Sorry for coming late to the party. Only recently got some cycles to go through this. This seems very similar to what we are trying to do with iommutests [3].
VFIO selftests aim to enable kernel developers to write and run tests that take the form of userspace programs that interact with VFIO and IOMMUFD uAPIs. VFIO selftests can be used to write functional tests for new features, regression tests for bugs, and performance tests for optimizations.
Have you considered implementing something outside the kernel sources? Something similar to fstests [1] or blktests [2]?
I ask because I have always seen it as a suit that tests (regression and performance) the Linux kernel as well as the hardware. By hardware I mean IOMMU devices, peripherals as well as their QEMU implementations. Since the scope is quite big, seemed (to me) like a good idea to keep it out of the kernel sources.
Can you speak to the pros/cons of having it in selftests? (sorry if this was already answered)
I talked about this a bit in the RFC cover letter, but did not copy it over to the v1 cover letter. Relevant excerpts below:
: We chose selftests to host these tests primarily to enable integration : with the existing KVM selftests. As explained in the next section, : enabling KVM developers to test the interaction between VFIO and KVM is : one of the motivators of this series. : : [...] : : - It enables testing the interaction between VFIO and KVM. There are : some paths in KVM that are only exercised through VFIO, such as IRQ : bypass. VFIO selftests provides a helper library to enable KVM : developers to write KVM selftests to test those interactions [3]. : : [...] : : To support testing the interactions between VFIO and KVM, the VFIO : selftests support sharing its library with the KVM selftest. The patches : at the end of this series demonstrate how that works. : : Essentially, we allow the KVM selftests to build their own copy of : tools/testing/selftests/vfio/lib/ and link it into KVM selftests : binaries. This requires minimal changes to the KVM selftests Makefile. : : [3] https://lore.kernel.org/kvm/20250404193923.1413163-68-seanjc@google.com/
If we had these tests out of tree, then I dn't see a clear path to being able to test the interaction between VFIO and KVM.
I am also probably biased/influenced coming from KVM development, where we use KVM selftests. I have found it very valuable as a developer to be able to send code changes along with the tests for those changes in a single series, without having to work across multiple repositories. And I am very used to that workflow.
Lastly, since the goal of my series is only to enable testing the kernel (VFIO, IOMMUFD, KVM, IOMMU drivers, etc.), co-locating with the kernel makes sense. I was not looking test hardware or emulators for correctness.
I do see that there is a lot of overlap between testing the kernel and testing hardware/emulators, and that there's likely code we can share. But I also want to be practical and make sure we solve the problem of testing the kernel.
These tests are designed to interact with real PCI devices, i.e. they do not rely on mocking out or faking any behavior in the kernel. This allows the tests to exercise not only VFIO but also IOMMUFD, the IOMMU driver, interrupt remapping, IRQ handling, etc.
And depending on how you execute them, you might also be exercising the QEMU emulation paths. You could even test different HW firmwares if you wanted to :)
That's true, but not a problem I'm looking to solve (testing hardware or testing QEMU). My only goal is testing the kernel.
For more background on the motivation and design of this series, please see the RFC:
https://lore.kernel.org/kvm/20250523233018.1702151-1-dmatlack@google.com/
This series can also be found on GitHub:
...
Instructions
Running VFIO selftests requires at a PCI device bound to vfio-pci for the tests to use. The address of this device is passed to the test as a segment:bus:device.function string, which must match the path to the device in /sys/bus/pci/devices/ (e.g. 0000:00:04.0).
Would you be able to autodetect the devices that are vfio-testable? I saw this question in the thread, but did not see the answer (sorry if I missed it).
Eventually I think that could be possible. But for now it is up to the user to deide which devices to use.
Once you have chosen a device, there is a helper script provided to unbind the device from its current driver, bind it to vfio-pci, export the environment variable $VFIO_SELFTESTS_BDF, and launch a shell:
If I'm reading the series correctly there is a fair amount of helper code needed: device (un)binding, checking capabilities, setting up DMAable memory.... Have you considered a library like libvfn [4] (is there any other?) to take care of all this?
I didn't consider using an external library since it seemed like there would be several downsides, and the amount of helper code here was quite small IMO:
.../selftests/vfio/lib/include/vfio_util.h | 295 +++++++++ .../selftests/vfio/lib/vfio_pci_device.c | 594 ++++++++++++++++++ .../selftests/vfio/lib/vfio_pci_driver.c | 126 ++++ tools/testing/selftests/vfio/run.sh | 109 ++++
My concerns with using an external helper library are:
- It will make writing tests more difficult because it will require making changes both to the external library and to selftests. This means dealing with 2 different repositories, 2 different code review processes, 2 different maintaineres, etc. Not to mention needing to first change the library before writing the test.
The harder it is to write tests, the less tests will be written IMO.
- It will make maintaining VFIO selftests more difficult as we have to maintain compatability with the library and deal with breaking changes (or never make breaking changes to the library, which makes it harder to maintain and extend).
Contrast this to having the helper code co-located with the tests we can make whatever changes we want as needed and keep things as simple as possible.
It can be good to avoid duplicate code across projects, but in this case it doesn't seem like it enhances our ability to test the kernel or solves any specific problem.
One place where libvfn could make sense is if someone wants to support running VFIO selftests that require a driver with NVMe devices. Implementing that driver with the help of libvfn may be simpler than implementing one from scratch. But I think we should cross that bridge when we get there and compare the 2 approaches. The VFIO selftests driver API is pretty simple so implementing an NVMe driver in VFIO selftests might still be the simpler option.
One of the good things about having all this in a library is that it can be used in other contexts besides testing.
True but if it comes at the cost of making VFIO tests harder to write and maintain, I don't think it's necessarily a good idea.
$ tools/testing/selftests/vfio/run.sh -d 0000:00:04.0 -s
The -d option tells the script which device to use and the -s option tells the script to launch a shell.
Additionally, the VFIO selftest vfio_dma_mapping_test has test cases that rely on HugeTLB pages being available, otherwise they are skipped. To enable those tests make sure at least 1 2MB and 1 1GB HugeTLB pages are available.
$ echo 1 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages $ echo 1 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
Should this be automatic? You can just modify nr_hugepages everytime you execute a test that requires it.
Yeah I think we could probably do this automatically. As you can probably tell, I left most of the environment setup to the user for now but a lot of this can be automated in the seltests themselves.
To run all VFIO selftests using make:
$ make -C tools/testing/selftests/vfio run_tests
This ties back to having the test suit outside the Linux Kernel sources. I might not always want/have a Linux Kernel selftests. Like if I would want to test the Intel/AMD IOMMU implementation in QEMU.
Agreed, but the goal of this series is not to test the correctness of QEMU or hardware. It's to enable testing the kernel code.
If you want to test the correctness of QEMU, testing from the VFIO/userspace level might not be the right approach anyway. You probably want to be able to interact directly with the IOMMU registers, interrupt handlers, etc. e.g. Something like intel-iommu.c in kvm-unit-tests[*], combined with higher-level application testing.
VFIO selftests would be kind of a weird middle-ground where you can't actually control the interactions with the hardware/QEMU, (since the test is running in userspace and going through the kernel), and at the same time it's not a realistic application.
[*] https://gitlab.com/kvm-unit-tests/kvm-unit-tests/-/blob/master/x86/intel-iom...
To run individual tests:
$ tools/testing/selftests/vfio/vfio_dma_mapping_test $ tools/testing/selftests/vfio/vfio_dma_mapping_test -v iommufd_anonymous_hugetlb_2mb $ tools/testing/selftests/vfio/vfio_dma_mapping_test -r vfio_dma_mapping_test.iommufd_anonymous_hugetlb_2mb.dma_map_unmap
The environment variable $VFIO_SELFTESTS_BDF can be overridden for a specific test by passing in the BDF on the command line as the last positional argument.
$ tools/testing/selftests/vfio/vfio_dma_mapping_test 0000:00:04.0 $ tools/testing/selftests/vfio/vfio_dma_mapping_test -v iommufd_anonymous_hugetlb_2mb 0000:00:04.0 $ tools/testing/selftests/vfio/vfio_dma_mapping_test -r vfio_dma_mapping_test.iommufd_anonymous_hugetlb_2mb.dma_map_unmap 0000:00:04.0
When you are done, free the HugeTLB pages and exit the shell started by run.sh. Exiting the shell will cause the device to be unbound from vfio-pci and bound back to its original driver.
$ echo 0 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages $ echo 0 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages $ exit
As before: Can this be done automatically?
Yeah, most likely this can be done automatically.
It's also possible to use run.sh to run just a single test hermetically, rather than dropping into a shell:
$ tools/testing/selftests/vfio/run.sh -d 0000:00:04.0 -- tools/testing/selftests/vfio/vfio_dma_mapping_test -v iommufd_anonymous
Tests
There are 5 tests in this series, mostly to demonstrate as a proof-of-concept:
- tools/testing/selftests/vfio/vfio_pci_device_test.c
- tools/testing/selftests/vfio/vfio_pci_driver_test.c
- tools/testing/selftests/vfio/vfio_iommufd_setup_test.c
- tools/testing/selftests/vfio/vfio_dma_mapping_test.c
- tools/testing/selftests/kvm/vfio_pci_device_irq_test.c
Future Areas of Development
Library:
- Driver support for devices that can be used on AMD, ARM, and other platforms (e.g. mlx5).
- Driver support for a device available in QEMU VMs (e.g. pcie-ats-testdev [1])
- Support for tests that use multiple devices.
- Support for IOMMU groups with multiple devices.
- Support for multiple devices sharing the same container/iommufd.
- Sharing TEST_ASSERT() macros and other common code between KVM and VFIO selftests.
Same as before: How about a lib?
That is how I plan to approach it. Just like tools/testing/selftests/vfio/lib can be shared with KVM selftests, we can create a selftests library for this common stuff and share it between VFIO and KVM.
But if you are referring to an external library that is outside of the kernel tree, see my comments above.
Tests:
- DMA mapping performance tests for BARs/HugeTLB/etc.
- Porting tests from https://github.com/awilliam/tests/commits/for-clg/ to selftests.
- Live Update selftests.
- Porting Sean's KVM selftest for posted interrupts to use the VFIO selftests library [2]
...
base-commit: e271ed52b344ac02d4581286961d0c40acc54c03 prerequisite-patch-id: c1decca4653262d3d2451e6fd4422ebff9c0b589 -- 2.50.0.rc2.701.gf1e915cc24-goog
Best
Thanks for taking a look!
[1] https://github.com/kdave/xfstests [2] https://github.com/linux-blktests/blktests [3] https://github.com/SamsungDS/iommutests [4] https://github.com/SamsungDS/libvfn
--
Joel Granados