irqfd_test on arm triggers the following assertion: ==== Test Assertion Failure ==== include/kvm_util.h:527: !ret pid=3643 tid=3643 errno=11 - Resource temporarily unavailable 1 0x00000000004026d7: kvm_irqfd at kvm_util.h:527 2 0x0000000000402083: main at irqfd_test.c:100 3 0x0000ffffa5aab587: ?? ??:0 4 0x0000ffffa5aab65f: ?? ??:0 5 0x000000000040236f: _start at ??:? KVM_IRQFD failed, rc: -1 errno: 11 (Resource temporarily unavailable)
Fix this by setting up a vgic for the vm.
Signed-off-by: Sebastian Ott sebott@redhat.com --- tools/testing/selftests/kvm/irqfd_test.c | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-)
diff --git a/tools/testing/selftests/kvm/irqfd_test.c b/tools/testing/selftests/kvm/irqfd_test.c index 7c301b4c7005..f7b8766e9d42 100644 --- a/tools/testing/selftests/kvm/irqfd_test.c +++ b/tools/testing/selftests/kvm/irqfd_test.c @@ -8,7 +8,11 @@ #include <stdint.h> #include <sys/sysinfo.h>
+#include "processor.h" #include "kvm_util.h" +#ifdef __aarch64__ +#include "vgic.h" +#endif
static struct kvm_vm *vm1; static struct kvm_vm *vm2; @@ -86,14 +90,30 @@ static void juggle_eventfd_primary(struct kvm_vm *vm, int eventfd) kvm_irqfd(vm, GSI_BASE_PRIMARY + 1, eventfd, KVM_IRQFD_FLAG_DEASSIGN); }
+static struct kvm_vm *test_vm_create(void) +{ +#ifdef __aarch64__ + struct kvm_vm *vm; + struct kvm_vcpu *vcpu; + int gic_fd; + + vm = vm_create_with_one_vcpu(&vcpu, NULL); + gic_fd = vgic_v3_setup(vm, 1, 64); + __TEST_REQUIRE(gic_fd >= 0, "Failed to create vgic-v3"); + + return vm; +#endif + return vm_create(1); +} + int main(int argc, char *argv[]) { pthread_t racing_thread; int r, i;
/* Create "full" VMs, as KVM_IRQFD requires an in-kernel IRQ chip. */ - vm1 = vm_create(1); - vm2 = vm_create(1); + vm1 = test_vm_create(); + vm2 = test_vm_create();
WRITE_ONCE(__eventfd, kvm_new_eventfd());
On Mon, Aug 25, 2025, Sebastian Ott wrote:
irqfd_test on arm triggers the following assertion: ==== Test Assertion Failure ==== include/kvm_util.h:527: !ret pid=3643 tid=3643 errno=11 - Resource temporarily unavailable 1 0x00000000004026d7: kvm_irqfd at kvm_util.h:527 2 0x0000000000402083: main at irqfd_test.c:100 3 0x0000ffffa5aab587: ?? ??:0 4 0x0000ffffa5aab65f: ?? ??:0 5 0x000000000040236f: _start at ??:? KVM_IRQFD failed, rc: -1 errno: 11 (Resource temporarily unavailable)
Fix this by setting up a vgic for the vm.
Signed-off-by: Sebastian Ott sebott@redhat.com
@@ -86,14 +90,30 @@ static void juggle_eventfd_primary(struct kvm_vm *vm, int eventfd) kvm_irqfd(vm, GSI_BASE_PRIMARY + 1, eventfd, KVM_IRQFD_FLAG_DEASSIGN); } +static struct kvm_vm *test_vm_create(void) +{ +#ifdef __aarch64__
- struct kvm_vm *vm;
- struct kvm_vcpu *vcpu;
- int gic_fd;
- vm = vm_create_with_one_vcpu(&vcpu, NULL);
- gic_fd = vgic_v3_setup(vm, 1, 64);
- __TEST_REQUIRE(gic_fd >= 0, "Failed to create vgic-v3");
I don't think this test requires v3+, any GIC will do.
Is there a sane way to handle vGIC creation in kvm_arch_vm_post_create()? E.g. could we create a v3 GIC when possible, and fall back to v2? And then provide a way for tests to express a hard v3 GIC dependency?
Having to worry about things like this in fairly generic code is quite burdensome.
- return vm;
+#endif
- return vm_create(1);
+}
On Mon, 25 Aug 2025 20:52:21 +0100, Sean Christopherson seanjc@google.com wrote:
On Mon, Aug 25, 2025, Sebastian Ott wrote:
irqfd_test on arm triggers the following assertion: ==== Test Assertion Failure ==== include/kvm_util.h:527: !ret pid=3643 tid=3643 errno=11 - Resource temporarily unavailable 1 0x00000000004026d7: kvm_irqfd at kvm_util.h:527 2 0x0000000000402083: main at irqfd_test.c:100 3 0x0000ffffa5aab587: ?? ??:0 4 0x0000ffffa5aab65f: ?? ??:0 5 0x000000000040236f: _start at ??:? KVM_IRQFD failed, rc: -1 errno: 11 (Resource temporarily unavailable)
Fix this by setting up a vgic for the vm.
Signed-off-by: Sebastian Ott sebott@redhat.com
@@ -86,14 +90,30 @@ static void juggle_eventfd_primary(struct kvm_vm *vm, int eventfd) kvm_irqfd(vm, GSI_BASE_PRIMARY + 1, eventfd, KVM_IRQFD_FLAG_DEASSIGN); } +static struct kvm_vm *test_vm_create(void) +{ +#ifdef __aarch64__
- struct kvm_vm *vm;
- struct kvm_vcpu *vcpu;
- int gic_fd;
- vm = vm_create_with_one_vcpu(&vcpu, NULL);
- gic_fd = vgic_v3_setup(vm, 1, 64);
- __TEST_REQUIRE(gic_fd >= 0, "Failed to create vgic-v3");
I don't think this test requires v3+, any GIC will do.
There is no such thing as "any GIC". You need to know what is available, and ask for something that actually exists. So while the above is wrong on the ground that this doesn't work on v2 or v5, the selection has to be explicit.
Is there a sane way to handle vGIC creation in kvm_arch_vm_post_create()? E.g. could we create a v3 GIC when possible, and fall back to v2? And then provide a way for tests to express a hard v3 GIC dependency?
You can ask KVM what's available. Like an actual VMM does. There is no shortage of examples in the current code base.
And ideally, this should be made an integral part of creating a viable VM, which the current VM creation hack makes a point in not providing.
M.
On Mon, Aug 25, 2025, Marc Zyngier wrote:
On Mon, 25 Aug 2025 20:52:21 +0100, Sean Christopherson seanjc@google.com wrote:
Is there a sane way to handle vGIC creation in kvm_arch_vm_post_create()? E.g. could we create a v3 GIC when possible, and fall back to v2? And then provide a way for tests to express a hard v3 GIC dependency?
You can ask KVM what's available. Like an actual VMM does. There is no shortage of examples in the current code base.
Right, by "sane" I meant: is there a way to instantiate a supported GIC without making it hard/painful to write tests, and without having to plumb in arm64 specific requirements to common APIs?
E.g. are there tests that use the common vm_create() APIs and rely on NOT having a GIC?
And ideally, this should be made an integral part of creating a viable VM, which the current VM creation hack makes a point in not providing.
On Mon, Aug 25, 2025 at 02:11:30PM -0700, Sean Christopherson wrote:
On Mon, Aug 25, 2025, Marc Zyngier wrote:
On Mon, 25 Aug 2025 20:52:21 +0100, Sean Christopherson seanjc@google.com wrote:
Is there a sane way to handle vGIC creation in kvm_arch_vm_post_create()? E.g. could we create a v3 GIC when possible, and fall back to v2? And then provide a way for tests to express a hard v3 GIC dependency?
You can ask KVM what's available. Like an actual VMM does. There is no shortage of examples in the current code base.
Right, by "sane" I meant: is there a way to instantiate a supported GIC without making it hard/painful to write tests, and without having to plumb in arm64 specific requirements to common APIs?
E.g. are there tests that use the common vm_create() APIs and rely on NOT having a GIC?
Instead of stuffing a GIC in behind vm_create(), I'd rather we have a specific helper for creating a VM with an irqchip. There's tests in arm64 that rely on all this generic infrastructure and also need to select / dimension a GIC appropriately for the test.
The majority of selftests don't even need an irqchip anyway.
Thanks, Oliver
On Mon, Aug 25, 2025, Oliver Upton wrote:
On Mon, Aug 25, 2025 at 02:11:30PM -0700, Sean Christopherson wrote:
On Mon, Aug 25, 2025, Marc Zyngier wrote:
On Mon, 25 Aug 2025 20:52:21 +0100, Sean Christopherson seanjc@google.com wrote:
Is there a sane way to handle vGIC creation in kvm_arch_vm_post_create()? E.g. could we create a v3 GIC when possible, and fall back to v2? And then provide a way for tests to express a hard v3 GIC dependency?
You can ask KVM what's available. Like an actual VMM does. There is no shortage of examples in the current code base.
Right, by "sane" I meant: is there a way to instantiate a supported GIC without making it hard/painful to write tests, and without having to plumb in arm64 specific requirements to common APIs?
E.g. are there tests that use the common vm_create() APIs and rely on NOT having a GIC?
Instead of stuffing a GIC in behind vm_create(), I'd rather we have a specific helper for creating a VM with an irqchip. There's tests in arm64 that rely on all this generic infrastructure and also need to select / dimension a GIC appropriately for the test.
I've no objection to arm64 providing an API for tests that need a specific GIC configuration (I think it's a great idea), but I don't think that needs to be mutually exclusive with having the common APIs instantiate _a_ GIC by default.
What if we use a global flag with thread local storage (because paranoia is basically free, I think) to communicate that the test wants to create a v3 vGIC?
E.g.
--- static __thread bool vm_wants_custom_vgic;
void kvm_arch_vm_post_create(struct kvm_vm *vm) { if (!vm_wants_custom_vgic) { <setup default vgic> } }
struct kvm_vm *vm_create_with_vgic_v3(uint32_t nr_vcpus, void *guest_code, uint32_t nr_irqs, struct kvm_vcpu **vcpus) { TEST_ASSERT(!vm_wants_custom_vgic, "blah blah blah"); vm_wants_custom_vgic = true; vm = vm_create_with_vcpus(nr_vcpus, guest_code, vcpus) vm_wants_custom_vgic = false;
vgic_v3_setup(vm, nr_vcpus, nr_irqs); }
struct kvm_vm *vm_create_with_one_vcpu_and_vgic_v3(struct kvm_vcpu **vcpu, void *guest_code, uint32_t nr_irqs) { struct kvm_vcpu *vcpus[1]; struct kvm_vm *vm;
vm = vm_create_with_vgic_v3(1, guest_code, nr_irqs, vcpus);
*vcpu = vcpus[0]; return vm; } ---
The majority of selftests don't even need an irqchip anyway.
But it's really, really nice for developers if they can assume a certain level of configuration is done by the infrastructure, i.e. don't have worry about doing what is effectively "basic" VM setup.
E.g. x86 selftests creates an IRQCHIP, sets up descriptor tables and exception handlers, and a handful of other "basic" things, and that has eliminated soooo much boilerplate code and the associated friction with having to know/discover that e.g. sending IRQs in a test requires additional setup beyond the obvious steps like wiring up a handler.
On Tue, Aug 26, 2025 at 11:51:18AM -0700, Sean Christopherson wrote:
On Mon, Aug 25, 2025, Oliver Upton wrote:
The majority of selftests don't even need an irqchip anyway.
But it's really, really nice for developers if they can assume a certain level of configuration is done by the infrastructure, i.e. don't have worry about doing what is effectively "basic" VM setup.
The more we pile behind what a "basic" VM configuration is the less expressive the tests become. Being able to immediately grok the *intent* of a test from reading it first pass is a very good thing. Otherwise I need to go figure out what the definition of "basic" means when I need to write a test and decide if that is compatible with what I'm trying to do.
vm_create_with_irqchip() is delightfully unambiguous.
E.g. x86 selftests creates an IRQCHIP, sets up descriptor tables and exception handlers, and a handful of other "basic" things, and that has eliminated soooo much boilerplate code and the associated friction with having to know/discover that e.g. sending IRQs in a test requires additional setup beyond the obvious steps like wiring up a handler.
That simply isn't going to happen on arm64. On top of the fact that the irqchip configuration depends on the intent of the test (e.g. wired IRQs v. MSIs), there's a bunch of guest-side initialization that needs to happen too.
We can add an extremely barebones GIC when asked for (although guest init isn't addressed) but batteries are not included on this architecture and I'd rather not attempt to abstract that.
Thanks, Oliver
On Tue, Aug 26, 2025, Oliver Upton wrote:
On Tue, Aug 26, 2025 at 11:51:18AM -0700, Sean Christopherson wrote:
On Mon, Aug 25, 2025, Oliver Upton wrote:
The majority of selftests don't even need an irqchip anyway.
But it's really, really nice for developers if they can assume a certain level of configuration is done by the infrastructure, i.e. don't have worry about doing what is effectively "basic" VM setup.
The more we pile behind what a "basic" VM configuration is the less expressive the tests become. Being able to immediately grok the *intent* of a test from reading it first pass is a very good thing. Otherwise I need to go figure out what the definition of "basic" means when I need to write a test and decide if that is compatible with what I'm trying to do.
Eh, I don't buy that argument, not as a blanket statement.
The existence of code doesn't always communicate intent, e.g. the _only_ instance I can think of where doing more setup by default caused problems was a few crusty x86 tests that relied on an int3 to cause SHUTDOWN due to lack of an IDT. OMG was I increduluous when I figured out what those tests were doing.
And in that case, _not_ doing the "basic" setup hid the intent of the test. Aside from the fact that deliberately triggering SHUTDOWN was completely unnecessary in those tests, IMO forcing such a test to use vm_create_barebones() would better capture that the test is doing something odd, i.e. has unusual intent.
And explicitly doing something doesn't necessarily communicate the intent of the test. E.g. the intent of the irqfd_test is to verify that KVM_IRQFD assign and deassign behaves as expected. The test never generates IRQs, i.e. doesn't actually need an IRQCHIP beyond satisfying KVM's requirements for KVM_IRQFD.
There are undoubtedly other tests that have similar "intent". E.g. the in-progress mediated PMU support for x86 requires an in-kernel local APIC, and so tests like pmu_counters_test.c, pmu_event_filter_test.c, and vmx_pmu_caps_test.c will need to instantiate an IRQCHIP. None of those tests actually touch the local APIC in any way, e.g. don't generate PMU interrupts, so creating an IRQCHIP is once again nothing more than a means to an end, and not indicative of the test's main intent.
I think the use of vgic_v3_setup() in dirty_log_perf_test.c is also a case where the existence of code fails to communicate intent. Without the comment in arch_setup_vm() to explain that having GICv3 somehow reduces the number of exits, I would be very confused as to why the test cares about GICv3.
I agree there's a balance to be had in terms of doing too much. Unfortunately in this case, it sounds like the fundamental problem is that the balance is simply different for x86 versus arm64. Having an in-kernel local APIC is tables stakes for x86, to the point where I'm looking for any excuse to have KVM create a local APIC by default. But for arm64, there's tremendous value in having tests do the lifting.
vm_create_with_irqchip() is delightfully unambiguous.
E.g. x86 selftests creates an IRQCHIP, sets up descriptor tables and exception handlers, and a handful of other "basic" things, and that has eliminated soooo much boilerplate code and the associated friction with having to know/discover that e.g. sending IRQs in a test requires additional setup beyond the obvious steps like wiring up a handler.
That simply isn't going to happen on arm64. On top of the fact that the irqchip configuration depends on the intent of the test (e.g. wired IRQs v. MSIs), there's a bunch of guest-side initialization that needs to happen too.
We can add an extremely barebones GIC when asked for (although guest init isn't addressed) but batteries are not included on this architecture and I'd rather not attempt to abstract that.
What about providing an API to do exactly that, instantiate and initialize a barebones GIC? E.g.
void kvm_arch_init_barebones_irqchip(struct kvm_vm *vm)
Hmm, then we'd also need
void kvm_arch_vm_free(struct kvm_vm *vm)
to gracefully free the GIC, as done by dirty_log_perf_test.c. Blech. Though maybe we'll end up with that hook sooner or later?
All in all, I have no strong preference at this point.
linux-kselftest-mirror@lists.linaro.org
-
Marc Zyngier -
Oliver Upton -
Sean Christopherson -
Sebastian Ott