Implement save/restore of the VGIC state using the newer KVM Device
Control API. This requries some number of changes to existing code in
addition to actually supporting save/restore of the necessary state.
The first patches (01-03) support creating the VGIC using the Device
Control API. This change is necessary because there are no other
suitable KVM APIs that we can leverage to access the VGIC state from
user space and the device control API was crafted exactly for this
purpose.
Subsequent patches add the missing infrastructure and user space API
pieces necessary to actually save and restore the VGIC state. The GIC
v2.0 architecture specification already specifies registers that can be
used to save and restore the complete VGIC state for suspend/resume
purposes on real hardware, and we can reuse this interface for the
VGIC. The API is therefore based on the memory-mapped register accesses
defined in the specs. See the individual patches for details.
The patches are based on kvm-arm-next with the arch timers save/restore
patches applied:
git://git.linaro.org/people/cdall/linux-kvm-arm.git timer-migrate-v4
This patch series based on the above can be cloned from:
git://git.linaro.org/people/cdall/linux-kvm-arm.git vgic-migrate-v5
User space patches for QEMU have also been posted on the list, but an
updated version is underway. Tested on Versatile Express TC2.
Changelogs in the individual patches.
Christoffer Dall (10):
ARM: KVM: Allow creating the VGIC after VCPUs
KVM: arm-vgic: Support KVM_CREATE_DEVICE for VGIC
KVM: arm-vgic: Set base addr through device API
irqchip: arm-gic: Define additional MMIO offsets and masks
KVM: arm-vgic: Make vgic mmio functions more generic
arm/arm64: kvm: Set vcpu->cpu to -1 on vcpu_put
KVM: arm-vgic: Add vgic reg access from dev attr
KVM: arm-vgic: Support unqueueing of LRs to the dist
KVM: arm-vgic: Add GICD_SPENDSGIR and GICD_CPENDSGIR handlers
KVM: arm-vgic: Support CPU interface reg access
Documentation/virtual/kvm/api.txt | 7 +-
Documentation/virtual/kvm/devices/arm-vgic.txt | 73 ++++
arch/arm/include/uapi/asm/kvm.h | 8 +
arch/arm/kvm/arm.c | 17 +-
include/kvm/arm_vgic.h | 2 +-
include/linux/irqchip/arm-gic.h | 12 +
include/linux/kvm_host.h | 1 +
include/uapi/linux/kvm.h | 1 +
virt/kvm/arm/vgic.c | 583 +++++++++++++++++++++++--
virt/kvm/kvm_main.c | 5 +
10 files changed, 672 insertions(+), 37 deletions(-)
create mode 100644 Documentation/virtual/kvm/devices/arm-vgic.txt
--
1.8.5
The current KVM implementation of PSCI returns INVALID_PARAMETERS if the
waitqueue for the corresponding CPU is not active. This does not seem
correct, since KVM should not care what the specific thread is doing,
for example, user space may not have called KVM_RUN on this VCPU yet or
the thread may be busy looping to user space because it received a
signal; this is really up to the user space implementation. Instead we
should check specifically that the CPU is marked as being turned off,
regardless of the VCPU thread state, and if it is, we shall
simply clear the pause flag on the CPU and wake up the thread if it
happens to be blocked for us.
Further, the implementation seems to be racy when executing multiple
VCPU threads. There really isn't a reasonable user space programming
scheme to ensure all secondary CPUs have reached kvm_vcpu_first_run_init
before turning on the boot CPU.
Therefore, set the pause flag on the vcpu at VCPU init time (which can
reasonably be expected to be completed for all CPUs by user space before
running any VCPUs) and clear both this flag and the feature (in case the
feature can somehow get set again in the future) and ping the waitqueue
on turning on a VCPU using PSCI.
Reported-by: Peter Maydell <peter.maydell(a)linaro.org>
Signed-off-by: Christoffer Dall <christoffer.dall(a)linaro.org>
---
Changes[v2]:
- Use non-atomic version of test_and_clear_bit instead
- Check if vcpu is paused and return KVM_PSCI_RET_INVAL if not
- Remove unnecessary feature bit clear
arch/arm/kvm/arm.c | 30 +++++++++++++++++++-----------
arch/arm/kvm/psci.c | 11 ++++++-----
2 files changed, 25 insertions(+), 16 deletions(-)
diff --git a/arch/arm/kvm/arm.c b/arch/arm/kvm/arm.c
index 2a700e0..151eb91 100644
--- a/arch/arm/kvm/arm.c
+++ b/arch/arm/kvm/arm.c
@@ -478,15 +478,6 @@ static int kvm_vcpu_first_run_init(struct kvm_vcpu *vcpu)
return ret;
}
- /*
- * Handle the "start in power-off" case by calling into the
- * PSCI code.
- */
- if (test_and_clear_bit(KVM_ARM_VCPU_POWER_OFF, vcpu->arch.features)) {
- *vcpu_reg(vcpu, 0) = KVM_PSCI_FN_CPU_OFF;
- kvm_psci_call(vcpu);
- }
-
return 0;
}
@@ -700,6 +691,24 @@ int kvm_vm_ioctl_irq_line(struct kvm *kvm, struct kvm_irq_level *irq_level,
return -EINVAL;
}
+static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu,
+ struct kvm_vcpu_init *init)
+{
+ int ret;
+
+ ret = kvm_vcpu_set_target(vcpu, init);
+ if (ret)
+ return ret;
+
+ /*
+ * Handle the "start in power-off" case by marking the VCPU as paused.
+ */
+ if (__test_and_clear_bit(KVM_ARM_VCPU_POWER_OFF, vcpu->arch.features))
+ vcpu->arch.pause = true;
+
+ return 0;
+}
+
long kvm_arch_vcpu_ioctl(struct file *filp,
unsigned int ioctl, unsigned long arg)
{
@@ -713,8 +722,7 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
if (copy_from_user(&init, argp, sizeof(init)))
return -EFAULT;
- return kvm_vcpu_set_target(vcpu, &init);
-
+ return kvm_arch_vcpu_ioctl_vcpu_init(vcpu, &init);
}
case KVM_SET_ONE_REG:
case KVM_GET_ONE_REG: {
diff --git a/arch/arm/kvm/psci.c b/arch/arm/kvm/psci.c
index 0881bf1..448f60e 100644
--- a/arch/arm/kvm/psci.c
+++ b/arch/arm/kvm/psci.c
@@ -54,15 +54,15 @@ static unsigned long kvm_psci_vcpu_on(struct kvm_vcpu *source_vcpu)
}
}
- if (!vcpu)
+ /*
+ * Make sure the caller requested a valid CPU and that the CPU is
+ * turned off.
+ */
+ if (!vcpu || !vcpu->arch.pause)
return KVM_PSCI_RET_INVAL;
target_pc = *vcpu_reg(source_vcpu, 2);
- wq = kvm_arch_vcpu_wq(vcpu);
- if (!waitqueue_active(wq))
- return KVM_PSCI_RET_INVAL;
-
kvm_reset_vcpu(vcpu);
/* Gracefully handle Thumb2 entry point */
@@ -79,6 +79,7 @@ static unsigned long kvm_psci_vcpu_on(struct kvm_vcpu *source_vcpu)
vcpu->arch.pause = false;
smp_mb(); /* Make sure the above is visible */
+ wq = kvm_arch_vcpu_wq(vcpu);
wake_up_interruptible(wq);
return KVM_PSCI_RET_SUCCESS;
--
1.8.5