On Tue, Aug 23, 2011 at 4:52 PM, Ian Jackson Ian.Jackson@eu.citrix.comwrote:
We're looking into porting Xen to the ARM A15 architecture. In this context, it's necessary to arrange for the Xen hypervisor to be entered from the boot loader in an appropriate processor mode.
KVM needs to deal with the same problem, of course. And any future Linux kernel feature which uses the Secure State does too.
We are currently working with ARM's "Fast Model" of the Cortex A15, a software emulator. We're using it in the mode where you feed it an ELF and it loads it into simulated RAM and starts executing at the ELF entrypoint. It does this in the CPU's defined startup mode, which is Kernel mode in the Secure state.
It seems that this environment is what the nascent KVM-on-A15 developers [1] are using too. There is modified version of the tiny boot wrapper (the normal version of which just emulates the proper calling API for the kernel); it sets up a trampoline security monitor.
Yes, this is what we do for now, when the preferred method from SoC vendors is unclear.
We need to define a correct calling convention for the kernel which is compatible with old systems, but which also allows the booted kernel (Linux, perhaps KVM-enabled, or indeed a hypervisor like Xen) use of all the available facilities. The correct approach does seem to be to have Linux set itself up a trivial trampoline which allows the kernel to later regain the elevated privilege.
There are a couple of things with the existing KVM ARM approach with the trivial boot wrapper which need to be fixed, though: firstly, there should be separate trampolines for hypervisor mode and for secure state. That allows the two features to be used independently. Secondly, the trivial trampolines should be part of the kernel proper and their lifetime should not extend across the bootloader interface.
How are you imagining the first point? To have a fixed address and being able to issue an HVC instead of SMC from within the kernel if booted in non-secure?
At first I thought that the best thing to do would be to boot the kernel in any suitable mode, and have the kernel automatically detect the starting mode. I started writing code in linux's head.S to do this. However, detecting whether we are in secure state is very difficult: it involves deliberately risking an undefined instruction trap. The code for this was getting rather long and involved.
As it is now, I thought the kernel could run in both secure and non-secure mode. If this is true, is it not a simple and useful feature that should be preserved?
Also, unconditionally starting a kernel in hypervisor mode seems rather unfriendly.
Agreed, this seems wrong.
At the moment we unconditionally start it in the
secure state and indeed in the current setup it seems to run entirely in secure state. It seems to me that the kernel should mostly run in non-secure state.
So I propose the following approach:
- The kernel will advertise via ELF notes what modes it may be
started in. The possible modes will be: (i) secure monitor mode (ii) non-secure hypervisor mode (iii) non-secure kernel mode
hmm, so you would have to choose _one_ of these at compile time and your image is bound to be run in the specific mode? How about when you're actually running in a VM, should it then be in mode (iii)?
What are the benefits from the kernel being aware of its mode if we ignore initialization and handle that somehow else?
- The bootloader will select the first mode from the three listed
above which is supported by both the processor and the kernel to be loaded, and transition the processor to that mode. If this involves dropping out of secure or hypervisor mode, it will put those modes permanently beyond use.
- The kernel will examine CPSR to determine which of the three
possibilities above has happened, and: (a) If started in monitor mode: * Grant access to everything to non-secure state * Set the non-secure copies of the various CP15 registers which don't have a sane value on cpu reset * Install a trivial monitor vector which unconditionally copies r0 to MVBAR and returns * Switch to non-secure Hyp mode (if available) and do (b), or non-secure Kernel mode (if Hyp mode not supported).
so you're suggesting a method that causes the kernel never to be able to run isolated in secure mode?
(b) If started in Hyp mode: * Install a trivial hypervisor vector which unconditionally copies r0 to HVBAR and returns (c) Rest of startup.
Questions:
- What do people think ? If this seems plausible I will prepare
an RFC patch for Linux and the boot-wrapper.git.
- This cpu startup process must happen very early - before paging
is enabled, in any case, so before RAM is really available. However, it produces two bits of information: 1. does the kernel own secure state; 2. does the kernel own hyp mode. Where should this information be stored ?
so, per (1) I would think no. (2) if the kernel uses KVM, yes.
- Is Linux allowed to assume that the secondary CPUs have the
same properties as the boot CPU ? If not, where do I store the availability of the secure/hyp modes for the secondary cpus ? Perhaps that ought to be in the device tree.
got me.
- I'm not very familiar with the KVM on ARM code. How much would
have to be changed to existing KVM on ARM to make it conform to the above scheme ? In particular it would have to not use SMC to adjust the HVBAR; instead, it would have to take control of HVBAR once per CPU.
This is a tiny adjustment - we were planning to fit this with the community wishes anyhow.
Thanks for your suggestions to get this rolling.
Opinions welcome.
Thanks, Ian.
[1] http://wiki.ncl.cs.columbia.edu/wiki/KVMARM:Guides:Development_Environment _______________________________________________ Android-virt mailing list Android-virt@lists.cs.columbia.edu https://lists.cs.columbia.edu/cucslists/listinfo/android-virt