We're looking into porting Xen to the ARM A15 architecture. In this context, it's necessary to arrange for the Xen hypervisor to be entered from the boot loader in an appropriate processor mode.
KVM needs to deal with the same problem, of course. And any future Linux kernel feature which uses the Secure State does too.
We are currently working with ARM's "Fast Model" of the Cortex A15, a software emulator. We're using it in the mode where you feed it an ELF and it loads it into simulated RAM and starts executing at the ELF entrypoint. It does this in the CPU's defined startup mode, which is Kernel mode in the Secure state.
It seems that this environment is what the nascent KVM-on-A15 developers [1] are using too. There is modified version of the tiny boot wrapper (the normal version of which just emulates the proper calling API for the kernel); it sets up a trampoline security monitor.
We need to define a correct calling convention for the kernel which is compatible with old systems, but which also allows the booted kernel (Linux, perhaps KVM-enabled, or indeed a hypervisor like Xen) use of all the available facilities. The correct approach does seem to be to have Linux set itself up a trivial trampoline which allows the kernel to later regain the elevated privilege.
There are a couple of things with the existing KVM ARM approach with the trivial boot wrapper which need to be fixed, though: firstly, there should be separate trampolines for hypervisor mode and for secure state. That allows the two features to be used independently. Secondly, the trivial trampolines should be part of the kernel proper and their lifetime should not extend across the bootloader interface.
At first I thought that the best thing to do would be to boot the kernel in any suitable mode, and have the kernel automatically detect the starting mode. I started writing code in linux's head.S to do this. However, detecting whether we are in secure state is very difficult: it involves deliberately risking an undefined instruction trap. The code for this was getting rather long and involved.
Also, unconditionally starting a kernel in hypervisor mode seems rather unfriendly. At the moment we unconditionally start it in the secure state and indeed in the current setup it seems to run entirely in secure state. It seems to me that the kernel should mostly run in non-secure state.
So I propose the following approach:
1. The kernel will advertise via ELF notes what modes it may be started in. The possible modes will be: (i) secure monitor mode (ii) non-secure hypervisor mode (iii) non-secure kernel mode
2. The bootloader will select the first mode from the three listed above which is supported by both the processor and the kernel to be loaded, and transition the processor to that mode. If this involves dropping out of secure or hypervisor mode, it will put those modes permanently beyond use.
3. The kernel will examine CPSR to determine which of the three possibilities above has happened, and: (a) If started in monitor mode: * Grant access to everything to non-secure state * Set the non-secure copies of the various CP15 registers which don't have a sane value on cpu reset * Install a trivial monitor vector which unconditionally copies r0 to MVBAR and returns * Switch to non-secure Hyp mode (if available) and do (b), or non-secure Kernel mode (if Hyp mode not supported). (b) If started in Hyp mode: * Install a trivial hypervisor vector which unconditionally copies r0 to HVBAR and returns (c) Rest of startup.
Questions:
1. What do people think ? If this seems plausible I will prepare an RFC patch for Linux and the boot-wrapper.git.
2. This cpu startup process must happen very early - before paging is enabled, in any case, so before RAM is really available. However, it produces two bits of information: 1. does the kernel own secure state; 2. does the kernel own hyp mode. Where should this information be stored ?
3. Is Linux allowed to assume that the secondary CPUs have the same properties as the boot CPU ? If not, where do I store the availability of the secure/hyp modes for the secondary cpus ? Perhaps that ought to be in the device tree.
4. I'm not very familiar with the KVM on ARM code. How much would have to be changed to existing KVM on ARM to make it conform to the above scheme ? In particular it would have to not use SMC to adjust the HVBAR; instead, it would have to take control of HVBAR once per CPU.
Opinions welcome.
Thanks, Ian.
[1] http://wiki.ncl.cs.columbia.edu/wiki/KVMARM:Guides:Development_Environment
On Tue, Aug 23, 2011 at 4:52 PM, Ian Jackson Ian.Jackson@eu.citrix.comwrote:
We're looking into porting Xen to the ARM A15 architecture. In this context, it's necessary to arrange for the Xen hypervisor to be entered from the boot loader in an appropriate processor mode.
KVM needs to deal with the same problem, of course. And any future Linux kernel feature which uses the Secure State does too.
We are currently working with ARM's "Fast Model" of the Cortex A15, a software emulator. We're using it in the mode where you feed it an ELF and it loads it into simulated RAM and starts executing at the ELF entrypoint. It does this in the CPU's defined startup mode, which is Kernel mode in the Secure state.
It seems that this environment is what the nascent KVM-on-A15 developers [1] are using too. There is modified version of the tiny boot wrapper (the normal version of which just emulates the proper calling API for the kernel); it sets up a trampoline security monitor.
Yes, this is what we do for now, when the preferred method from SoC vendors is unclear.
We need to define a correct calling convention for the kernel which is compatible with old systems, but which also allows the booted kernel (Linux, perhaps KVM-enabled, or indeed a hypervisor like Xen) use of all the available facilities. The correct approach does seem to be to have Linux set itself up a trivial trampoline which allows the kernel to later regain the elevated privilege.
There are a couple of things with the existing KVM ARM approach with the trivial boot wrapper which need to be fixed, though: firstly, there should be separate trampolines for hypervisor mode and for secure state. That allows the two features to be used independently. Secondly, the trivial trampolines should be part of the kernel proper and their lifetime should not extend across the bootloader interface.
How are you imagining the first point? To have a fixed address and being able to issue an HVC instead of SMC from within the kernel if booted in non-secure?
At first I thought that the best thing to do would be to boot the kernel in any suitable mode, and have the kernel automatically detect the starting mode. I started writing code in linux's head.S to do this. However, detecting whether we are in secure state is very difficult: it involves deliberately risking an undefined instruction trap. The code for this was getting rather long and involved.
As it is now, I thought the kernel could run in both secure and non-secure mode. If this is true, is it not a simple and useful feature that should be preserved?
Also, unconditionally starting a kernel in hypervisor mode seems rather unfriendly.
Agreed, this seems wrong.
At the moment we unconditionally start it in the
secure state and indeed in the current setup it seems to run entirely in secure state. It seems to me that the kernel should mostly run in non-secure state.
So I propose the following approach:
- The kernel will advertise via ELF notes what modes it may be
started in. The possible modes will be: (i) secure monitor mode (ii) non-secure hypervisor mode (iii) non-secure kernel mode
hmm, so you would have to choose _one_ of these at compile time and your image is bound to be run in the specific mode? How about when you're actually running in a VM, should it then be in mode (iii)?
What are the benefits from the kernel being aware of its mode if we ignore initialization and handle that somehow else?
- The bootloader will select the first mode from the three listed
above which is supported by both the processor and the kernel to be loaded, and transition the processor to that mode. If this involves dropping out of secure or hypervisor mode, it will put those modes permanently beyond use.
- The kernel will examine CPSR to determine which of the three
possibilities above has happened, and: (a) If started in monitor mode: * Grant access to everything to non-secure state * Set the non-secure copies of the various CP15 registers which don't have a sane value on cpu reset * Install a trivial monitor vector which unconditionally copies r0 to MVBAR and returns * Switch to non-secure Hyp mode (if available) and do (b), or non-secure Kernel mode (if Hyp mode not supported).
so you're suggesting a method that causes the kernel never to be able to run isolated in secure mode?
(b) If started in Hyp mode: * Install a trivial hypervisor vector which unconditionally copies r0 to HVBAR and returns (c) Rest of startup.
Questions:
- What do people think ? If this seems plausible I will prepare
an RFC patch for Linux and the boot-wrapper.git.
- This cpu startup process must happen very early - before paging
is enabled, in any case, so before RAM is really available. However, it produces two bits of information: 1. does the kernel own secure state; 2. does the kernel own hyp mode. Where should this information be stored ?
so, per (1) I would think no. (2) if the kernel uses KVM, yes.
- Is Linux allowed to assume that the secondary CPUs have the
same properties as the boot CPU ? If not, where do I store the availability of the secure/hyp modes for the secondary cpus ? Perhaps that ought to be in the device tree.
got me.
- I'm not very familiar with the KVM on ARM code. How much would
have to be changed to existing KVM on ARM to make it conform to the above scheme ? In particular it would have to not use SMC to adjust the HVBAR; instead, it would have to take control of HVBAR once per CPU.
This is a tiny adjustment - we were planning to fit this with the community wishes anyhow.
Thanks for your suggestions to get this rolling.
Opinions welcome.
Thanks, Ian.
[1] http://wiki.ncl.cs.columbia.edu/wiki/KVMARM:Guides:Development_Environment _______________________________________________ Android-virt mailing list Android-virt@lists.cs.columbia.edu https://lists.cs.columbia.edu/cucslists/listinfo/android-virt
Christoffer Dall writes ("Re: [Android-virt] ARM processor mode, kernel startup, Hyp / secure state"):
There are a couple of things with the existing KVM ARM approach with the trivial boot wrapper which need to be fixed, though: firstly, there should be separate trampolines for hypervisor mode and for secure state. That allows the two features to be used independently. Secondly, the trivial trampolines should be part of the kernel proper and their lifetime should not extend across the bootloader interface.
How are you imagining the first point? To have a fixed address and being able to issue an HVC instead of SMC from within the kernel if booted in non-secure?
I was thinking that if the kernel is booted in hyp mode (non-secure obviously, then), the kernel would set HVBAR to point to a little pretend hypervisor vector table whose only functionality was to be able to set HVBAR. Then later, when you load the kvm module (say), this would be used (once per cpu) to set HVBAR to the actual vector table.
This is the only way to make things work if secure state is in use for something else (whether by the firmware, or by some other kernel feature).
As it is now, I thought the kernel could run in both secure and non-secure mode. If this is true, is it not a simple and useful feature that should be preserved?
This is a good point. Yes, the kernel can run in both secure and non-secure states. However, if you want to use kvm, this is no longer the case. kvm wants to use hyp mode and hyp mode is only available in the non-secure state. And if you let the kernel get properly booted in secure state, switching it to non-secure state while it is running would be quite painful - you'd have to do a very complicated dance to make sure the non-secure state was set up just right, etc.
- The kernel will advertise via ELF notes what modes it may be
started in. The possible modes will be: (i) secure monitor mode (ii) non-secure hypervisor mode (iii) non-secure kernel mode
hmm, so you would have to choose _one_ of these at compile time and your image is bound to be run in the specific mode? How about when you're actually running in a VM, should it then be in mode (iii)?
No, I'm proposing that you'd choose one _or more_ of these at compile time. Most modern Linux kernels would support all three. The Xen hypervisor (which has to be booted via the same interface) would support only (ii) and (iii).
What are the benefits from the kernel being aware of its mode if we ignore initialization and handle that somehow else?
I'm not sure what you mean by "being aware of its mode". If you mean "being aware of the security state", then there is no direct benefit just of that /awareness/. However, a kernel which knows that it is running in non-secure state but which has also set up a hook to allow it to use secure state, could later (eg when a kernel module is loaded) use the secure state for exciting functionality (eg, crypto key storage, or the like).
If you mean "being aware of kernel mode vs. hyp mode", then the answer is that the kernel cannot generally run in hyp mode, but if you want to use kvm it needs access to hyp mode. So the startup arrangements need to arrange that the kvm kernel module can get back into hyp mode when it wants to.
so you're suggesting a method that causes the kernel never to be able to run isolated in secure mode?
You're right that that was the implication of my proposal. However, if that's something that people want to do then my proposal needs to be modified. But to understand that better, I'd like to understand why you would want to run the whole kernel in secure state. What would you use the non-secure state for ?
If this is indeed a requirement, one option would be to simply say that a kernel may be started in secure kernel state. But there is a difficulty: the kernel needs to be told somehow that it is in secure kernel state, so that it knows that it can use the non-secure state.
- This cpu startup process must happen very early - before paging
is enabled, in any case, so before RAM is really available. However, it produces two bits of information: 1. does the kernel own secure state; 2. does the kernel own hyp mode. Where should this information be stored ?
so, per (1) I would think no. (2) if the kernel uses KVM, yes.
Um. What I meant is: these two bits of information vary according to how the kernel has been booted. They don't have fixed answers.
- I'm not very familiar with the KVM on ARM code. How much would
have to be changed to existing KVM on ARM to make it conform to the above scheme ? In particular it would have to not use SMC to adjust the HVBAR; instead, it would have to take control of HVBAR once per CPU.
This is a tiny adjustment - we were planning to fit this with the community wishes anyhow.
Right.
Thanks, Ian.
On Tue, Aug 23, 2011 at 03:52:25PM +0100, Ian Jackson wrote:
We're looking into porting Xen to the ARM A15 architecture. In this context, it's necessary to arrange for the Xen hypervisor to be entered from the boot loader in an appropriate processor mode.
KVM needs to deal with the same problem, of course. And any future Linux kernel feature which uses the Secure State does too.
We already have kernels which boot in both secure and non-secure mode CPUs. There's very little difference between the two from the core ARM support code - except that we have expectations that appropriate hardware work-arounds will have already been initialized by the secure monitor (as there is no way for the kernel to be able to call out to the secure monitor in a platform independent way.)
That's unfortunately one of the necessary evils of having the secure stuff, which doesn't have any kind of defined API, getting in the way.
So, basically, which side of the secure division the kernel runs on depends highly on the platform itself - and as the secure monitor APIs are highly platform specific, there's no getting away from that.
If the kernel does not contain the appropriate platform specific secure monitor API then it can't run in non-secure mode (or it can but the functionality will be severely limited - no power management, and in some cases buggy L2 cache.) So you wouldn't _want_ to run such a kernel if you value your data.
As far as I know, the kernel runs non-secure on all real silicon out in the wild. It's only ARMs evaluation boards and models where the kernel runs on the secure side.
Given all that, there's not much point to publishing whether a kernel is built for secure side or not. It's tied extremely closely to the platform itself, which is where most of the secure API calls already live.
Russell King - ARM Linux writes ("Re: ARM processor mode, kernel startup, Hyp / secure state"):
We already have kernels which boot in both secure and non-secure mode CPUs. There's very little difference between the two from the core ARM support code - except that we have expectations that appropriate hardware work-arounds will have already been initialized by the secure monitor (as there is no way for the kernel to be able to call out to the secure monitor in a platform independent way.)
Right.
So, basically, which side of the secure division the kernel runs on depends highly on the platform itself - and as the secure monitor APIs are highly platform specific, there's no getting away from that.
It seems to me that from the PoV of Linux there are two kind of platform: ones where the kernel is expected to own the secure state, and ones where it is expected to run only in non-secure state.
But just because Linux owns the secure state doesn't mean that it should run in that state most of the time. The question is really whether Linux would like to use the secure/non-secure division to provide a more secure facility which other parts of the system can use, or to use it to provide a more constrained/contained environment in which to run less-trusted code.
In the latter case Linux would want to run in secure state all of the time. In the former it would want to run mostly in non-secure state. But it is difficult to choose between these later than boot time because switching a running system from the secure to non-secure state is so complex.
If the kernel does not contain the appropriate platform specific secure monitor API then it can't run in non-secure mode (or it can but the functionality will be severely limited - no power management, and in some cases buggy L2 cache.) So you wouldn't _want_ to run such a kernel if you value your data.
So is the secure state used for the same kind of things as SMM is on Intel, then ? Perhaps I haven't understood the usual setups well enough.
Given all that, there's not much point to publishing whether a kernel is built for secure side or not. It's tied extremely closely to the platform itself, which is where most of the secure API calls already live.
The point of my proposed ELF flag isn't to advertise that a kernel is capable of running in the secure state. As you say, where this is possible, we don't need to advertise it.
The point of my ELF flag is to advertise that the kernel would like to _know_ that it has control of the secure state; the kernel is told this by the fact that it enters in _monitor mode_ rather than kernel mode. Obviously you shouldn't enter an unconsenting kernel in monitor mode.
If the kernel is to have control of the secure state it needs to know this, and to have a way of making use of it. It looked to me like the easiest way of doing this was entering it in monitor mode. (Since a kernel entered in kernel mode can't tell easily whether it's in secure or non-secure state.)
Likewise, if the kernel is to have control of the hyp mode, it needs to know that and have a way to use it. So entering the kernel in hyp mode seemed to me the easiest way.
I think I know what the hyp mode is for and how people will use it :-). But I don't really know what the secure state is for.
Thanks, Ian.
On Tue, Aug 23, 2011 at 05:50:10PM +0100, Ian Jackson wrote:
Russell King - ARM Linux writes ("Re: ARM processor mode, kernel startup, Hyp / secure state"):
We already have kernels which boot in both secure and non-secure mode CPUs. There's very little difference between the two from the core ARM support code - except that we have expectations that appropriate hardware work-arounds will have already been initialized by the secure monitor (as there is no way for the kernel to be able to call out to the secure monitor in a platform independent way.)
Right.
So, basically, which side of the secure division the kernel runs on depends highly on the platform itself - and as the secure monitor APIs are highly platform specific, there's no getting away from that.
It seems to me that from the PoV of Linux there are two kind of platform: ones where the kernel is expected to own the secure state, and ones where it is expected to run only in non-secure state.
The distinction is a highly technical one, and it comes from the lack of any kind of secure monitor API, and the fact that the non-secure world is prevented from accessing various registers.
A secure side kernel contains no calls to the secure monitor. Calls to the secure monitor are required to reprogram critical registers such as L2 cache configuration registers and the like. For example, on OMAP, it is impossible to reliably invalidate the contents of the L2 cache without making secure mode calls to reprogram various control registers.
Not having that register reprogrammed means you're running against data corrupting errata bugs.
So, basically, a kernel built for non-secure will only run on the non-secure side. A kernel built for secure will only run reliably on the secure side.
But just because Linux owns the secure state doesn't mean that it should run in that state most of the time. The question is really whether Linux would like to use the secure/non-secure division to provide a more secure facility which other parts of the system can use, or to use it to provide a more constrained/contained environment in which to run less-trusted code.
What you're talking about is using Linux as the secure monitor, and effectively matching the dividing line between Linux and userspace with the secure/non-secure division.
Remember that Linux is not a micro-kernel, so any other partitioning doesn't make sense. (To place the partition elsewhere implies that you have to carve the kernel data structures up and contain them within each world - with separate memory allocators etc.)
If the kernel does not contain the appropriate platform specific secure monitor API then it can't run in non-secure mode (or it can but the functionality will be severely limited - no power management, and in some cases buggy L2 cache.) So you wouldn't _want_ to run such a kernel if you value your data.
So is the secure state used for the same kind of things as SMM is on Intel, then ? Perhaps I haven't understood the usual setups well enough.
You're assuming I have deep knowledge of x86. I don't, sorry.
I think I know what the hyp mode is for and how people will use it :-). But I don't really know what the secure state is for.
Secure state is used to run the secure monitor, which is a chunk of code provided by the SoC manufacturer and normally programmed into ROM on real SoCs.
What is does is up to the SoC manufacturer. What services it provides is up to the SoC manufacturer.
The implication is that once the secure monitor has set things up, it will start execution of things like boot loaders, etc in non-secure mode. Moreover, there's usually no way to get back into secure mode to run your own code (otherwise it wouldn't be secure...)
A SoC manufacturer may provide a secure monitor to deal with storing secret keys, and providing some kind of encryption services. Or it may provide nothing more than a way to access system registers which are only accessible from secure mode.
Russell King - ARM Linux writes ("Re: ARM processor mode, kernel startup, Hyp / secure state"):
A secure side kernel contains no calls to the secure monitor. Calls to the secure monitor are required to reprogram critical registers such as L2 cache configuration registers and the like. For example, on OMAP, it is impossible to reliably invalidate the contents of the L2 cache without making secure mode calls to reprogram various control registers.
Not having that register reprogrammed means you're running against data corrupting errata bugs.
So, basically, a kernel built for non-secure will only run on the non-secure side. A kernel built for secure will only run reliably on the secure side.
It's not clear to me from reading the ARM ARM that a kernel written for the non-secure side couldn't make the same monitor calls, but I haven't seen any of the OMAP manuals and I haven't read any of the errata (which you obviously have).
If there are kernels intended to run on the secure side, a difficulty arises: if KVM wants to use Hyp mode and the associated features, how is that to be done ?
But just because Linux owns the secure state doesn't mean that it should run in that state most of the time. The question is really whether Linux would like to use the secure/non-secure division to provide a more secure facility which other parts of the system can use, or to use it to provide a more constrained/contained environment in which to run less-trusted code.
What you're talking about is using Linux as the secure monitor, and effectively matching the dividing line between Linux and userspace with the secure/non-secure division.
Well, yes, that would be one way of looking at it. I'm not sure why you would want to do that, but it was just one of the two options if Linux is trying to own the secure/non-secure distinction.
On the other hand if Linux wants to own that boundary and to use it to provide a /more/ privileged facility, that would involve some Linux kernel module hoisting a pile of its code (or, more likely, code provided from userspace) into secure state, which code then pulled the ladder up after itself. The resulting thing wouldn't look much like a user process and wouldn't (afterwards) be subject to unwanted interference. But I don't think anyone is doing anything like this ?
So is the secure state used for the same kind of things as SMM is on Intel, then ? Perhaps I haven't understood the usual setups well enough.
You're assuming I have deep knowledge of x86. I don't, sorry.
Fair enough. (My own x86 knowledge is pretty shallow too TBH.)
Secure state is used to run the secure monitor, which is a chunk of code provided by the SoC manufacturer and normally programmed into ROM on real SoCs.
If this is always (or generally) the case then Linux needs to run on only one side of the secure/non-secure barrier. Since Linux might want to run kvm, which requires Hyp mode, which is only on the non-secure side, that means it needs to run on the non-secure side.
What have I missed ?
Ian.
On Tue, 23 Aug 2011, Ian Jackson wrote:
If there are kernels intended to run on the secure side, a difficulty arises: if KVM wants to use Hyp mode and the associated features, how is that to be done ?
I think that you can safely presume that the secure world is not open to Open Source in practice. As Russell said, all the actual implementations meant for products out there have the secure mode effectively burried behind ROM code that the SOC vendor controls.
Nicolas
On Tue, Aug 23, 2011 at 3:52 PM, Ian Jackson Ian.Jackson@eu.citrix.com wrote:
We're looking into porting Xen to the ARM A15 architecture. In this context, it's necessary to arrange for the Xen hypervisor to be entered from the boot loader in an appropriate processor mode.
KVM needs to deal with the same problem, of course. And any future Linux kernel feature which uses the Secure State does too.
We are currently working with ARM's "Fast Model" of the Cortex A15, a software emulator. We're using it in the mode where you feed it an ELF and it loads it into simulated RAM and starts executing at the ELF entrypoint. It does this in the CPU's defined startup mode, which is Kernel mode in the Secure state.
It seems that this environment is what the nascent KVM-on-A15 developers [1] are using too. There is modified version of the tiny boot wrapper (the normal version of which just emulates the proper calling API for the kernel); it sets up a trampoline security monitor.
We need to define a correct calling convention for the kernel which is compatible with old systems, but which also allows the booted kernel (Linux, perhaps KVM-enabled, or indeed a hypervisor like Xen) use of all the available facilities. The correct approach does seem to be to have Linux set itself up a trivial trampoline which allows the kernel to later regain the elevated privilege.
There are a couple of things with the existing KVM ARM approach with the trivial boot wrapper which need to be fixed, though: firstly, there should be separate trampolines for hypervisor mode and for secure state. That allows the two features to be used independently. Secondly, the trivial trampolines should be part of the kernel proper and their lifetime should not extend across the bootloader interface.
At first I thought that the best thing to do would be to boot the kernel in any suitable mode, and have the kernel automatically detect the starting mode. I started writing code in linux's head.S to do this. However, detecting whether we are in secure state is very difficult: it involves deliberately risking an undefined instruction trap. The code for this was getting rather long and involved.
There may be a safe way to do this check -- for example, on ARM1176 and Cortex-A8 there is a CP14 debug status/control register that you can read which includes a flag indicating which world you're in. This isn't part of the architecture though and may be different/not possible on some CPUs.
All in all, it's better to engineer things so that the check doesn't need to be done at all.
Also, unconditionally starting a kernel in hypervisor mode seems rather unfriendly. At the moment we unconditionally start it in the secure state and indeed in the current setup it seems to run entirely in secure state. It seems to me that the kernel should mostly run in non-secure state.
So I propose the following approach:
1. The kernel will advertise via ELF notes what modes it may be started in. The possible modes will be: (i) secure monitor mode (ii) non-secure hypervisor mode (iii) non-secure kernel mode
Note that in real deployments, the kernel is not an ELF image and therefore cannot have notes.
Currently, I think the kernel doesn't really have any metadata readable by the bootloader at all.
I think the Secure World / Normal World distinction may be a red herring. The kernel currently just stays in whatever world it was started in, and doesn't have support for being the bridge between worlds. If the kernel is started in the Normal World, this will be because something else occupies the Secure World, and so Linux won't have direct access to that anyway. Conversely, if Linux is started in the Secure World, it can always access the monitor state if this is really desired -- the monitor is at the same level of privilege as the other Secure World privileged modes.
I haven't seen a compelling reason why this would need to change; do you have a particular scenario in mind?
2. The bootloader will select the first mode from the three listed above which is supported by both the processor and the kernel to be loaded, and transition the processor to that mode. If this involves dropping out of secure or hypervisor mode, it will put those modes permanently beyond use.
3. The kernel will examine CPSR to determine which of the three possibilities above has happened, and: (a) If started in monitor mode: * Grant access to everything to non-secure state * Set the non-secure copies of the various CP15 registers which don't have a sane value on cpu reset * Install a trivial monitor vector which unconditionally copies r0 to MVBAR and returns * Switch to non-secure Hyp mode (if available) and do (b), or non-secure Kernel mode (if Hyp mode not supported). (b) If started in Hyp mode: * Install a trivial hypervisor vector which unconditionally copies r0 to HVBAR and returns
Starting the kernel in Hyp mode may be reasonable in this scenario.
I think this is actually backwards-compatible, because in both an uncompressed kernel and a zImage, pretty much the first thing that happens is a switch to SVC mode at present. So an older kernel will just transparently end up in SVC mode as if nothing unusual had happened.
The boot procotol would need to define the initial hypervisor state: basically, a dormant identity configuration with all traps and fancy features turned off.
With only minor changes, I think a kernel supporting this boot protocol could run the zImage decompressor in Hyp mode, and then set up a stub HVC handler when reaching the main kernel entry point. It's certainly worth investigating.
There may be architectural reasons why you can't run the decompressor in Hyp mode, but I'm not aware of any off the top of my head.
(c) Rest of startup.
Questions:
1. What do people think ? If this seems plausible I will prepare an RFC patch for Linux and the boot-wrapper.git.
2. This cpu startup process must happen very early - before paging is enabled, in any case, so before RAM is really available. However, it produces two bits of information: 1. does the kernel own secure state; 2. does the kernel own hyp mode. Where should this information be stored ?
3. Is Linux allowed to assume that the secondary CPUs have the same properties as the boot CPU ? If not, where do I store the availability of the secure/hyp modes for the secondary cpus ? Perhaps that ought to be in the device tree.
I think yes, for now. Having a different level of access on different CPUs is either an unnecessary hindrance or a security hole (depending on your point of view), and such a system would not be SMP.
Whether Linux makes use of those facilities on all CPUs is up to Linux though.
It's up to the Secure firmware (if any) and the bootloader to get the secondary CPUs into the appropriate state.
4. I'm not very familiar with the KVM on ARM code. How much would have to be changed to existing KVM on ARM to make it conform to the above scheme ? In particular it would have to not use SMC to adjust the HVBAR; instead, it would have to take control of HVBAR once per CPU.
Opinions welcome.
Thanks, Ian.
[1] http://wiki.ncl.cs.columbia.edu/wiki/KVMARM:Guides:Development_Environment
boot-architecture mailing list boot-architecture@lists.linaro.org http://lists.linaro.org/mailman/listinfo/boot-architecture
Dave Martin writes ("Re: ARM processor mode, kernel startup, Hyp / secure state"):
I think the Secure World / Normal World distinction may be a red herring. The kernel currently just stays in whatever world it was started in, and doesn't have support for being the bridge between worlds. If the kernel is started in the Normal World, this will be because something else occupies the Secure World, and so Linux won't have direct access to that anyway. Conversely, if Linux is started in the Secure World, it can always access the monitor state if this is really desired -- the monitor is at the same level of privilege as the other Secure World privileged modes.
If Linux is booted in the secure world, it will have to switch to the non-secure world at boot if it is to use kvm.
I haven't seen a compelling reason why this would need to change; do you have a particular scenario in mind?
The non-secure world doesn't have hyp mode, which is needed by kvm.
So if kvm might be loaded later (and we don't really want to rule that out at boot time unless we can help it) then the kernel needs either - to be started in the secure world, to know this, and switch to the non-secure world while installing appropriate trampoline(s) - to be started in hyp mode, and switch to kernel mode while installing an appropriate trampoline - something more horrible (eg the bootloader provides the trampoline; or switch the running system to the non-secure world when the desire for kvm becomes evident)
Starting the kernel in Hyp mode may be reasonable in this scenario.
I think this is actually backwards-compatible, because in both an uncompressed kernel and a zImage, pretty much the first thing that happens is a switch to SVC mode at present. So an older kernel will just transparently end up in SVC mode as if nothing unusual had happened.
Do all relevant existing kernels unconditionally switch to kernel mode at the start ?
The boot procotol would need to define the initial hypervisor state: basically, a dormant identity configuration with all traps and fancy features turned off.
Right. Helpfully, that is (more or less) the processor reset state.
With only minor changes, I think a kernel supporting this boot protocol could run the zImage decompressor in Hyp mode, and then set up a stub HVC handler when reaching the main kernel entry point. It's certainly worth investigating.
I haven't checked the decompressor but I don't see why it wouldn't work in hyp mode. I can try it easily enough.
Ian.
I wrote:
I haven't checked the decompressor but I don't see why it wouldn't work in hyp mode. I can try it easily enough.
I tried booting linux in non-secure hyp mode [1] and it seemed to work just fine. The kernel decompressed and booted up to trying to mount its root fs, just like it does if you start it in secure kernel mode.
Ian.
[1] git://git.kernel.org/pub/scm/linux/kernel/git/cmarinas/linux-2.6-cm.git a104d064b95e4981619c66bb94840ab3cb67c75d
On Tue, Aug 23, 2011 at 5:59 PM, Ian Jackson Ian.Jackson@eu.citrix.com wrote:
Dave Martin writes ("Re: ARM processor mode, kernel startup, Hyp / secure state"):
I think the Secure World / Normal World distinction may be a red herring. The kernel currently just stays in whatever world it was started in, and doesn't have support for being the bridge between worlds. If the kernel is started in the Normal World, this will be because something else occupies the Secure World, and so Linux won't have direct access to that anyway. Conversely, if Linux is started in the Secure World, it can always access the monitor state if this is really desired -- the monitor is at the same level of privilege as the other Secure World privileged modes.
If Linux is booted in the secure world, it will have to switch to the non-secure world at boot if it is to use kvm.
I haven't seen a compelling reason why this would need to change; do you have a particular scenario in mind?
The non-secure world doesn't have hyp mode, which is needed by kvm.
You mean "the secure world doesn't have hyp mode"? Fair point if so; I was forgetting this.
So if kvm might be loaded later (and we don't really want to rule that out at boot time unless we can help it) then the kernel needs either - to be started in the secure world, to know this, and switch to the non-secure world while installing appropriate trampoline(s) - to be started in hyp mode, and switch to kernel mode while installing an appropriate trampoline
This feels like the preferable approach -- it makes no sense to require the kernel to be started in the Secure World if we want to install a hypervisor.
There might be some sense in *requiring* the bootloader to start the kernel in the Normal World and in hyp mode if kvm (or other kernel-hosted hypervisor) is to be used. If you weren't booted this way then tough, you can't use kvm.
If you allow the kernel to be started in the Secure World, it can switch to the Normal World on boot, but this is in practice a complex, platform-specific procecure: you need to reprogram protection controllers etc. to move all the peripherals and RAM into the Normal World,
Currently, I believe none of the needed code to accomplish this is in the kernel, because there's been no prior need for it. Implemting this would potentially require work for every board >= ARMv6Z. Bootloader maintainers might argue with that though: general-purpose bootloaders probably won't have this code either yet, for the same reasons. If there is real secure firmware, it will of course set all this up before invoking the bootloader. But otherwise, _someone_ has to do it...
- something more horrible (eg the bootloader provides the trampoline; or switch the running system to the non-secure world when the desire for kvm becomes evident)
I agree with "horrible" :)
In particular, switching to the Normal World after boot is going to be extremely horrible given the need to move peripherals and RAM into the Normal world while somehow not pulling the floor from under your feet...
Starting the kernel in Hyp mode may be reasonable in this scenario.
I think this is actually backwards-compatible, because in both an uncompressed kernel and a zImage, pretty much the first thing that happens is a switch to SVC mode at present. So an older kernel will just transparently end up in SVC mode as if nothing unusual had happened.
Do all relevant existing kernels unconditionally switch to kernel mode at the start ?
In practice, I think so. Take a look at arch/arm/kernel/head.S and arch/arm/boot/compressed/head.S. Really, it's interrupts which get disabled at these entry points -- but prior to ARMv6 it used to be simpler to set the mode too since you couldn't force the interrupt mask bits using a single MSR instruction without also setting the processor mode.
Looking at compressed/head.S, it looks like the CPSR is only touched by read-modify-write on the interrupt bits, or not at all. So this code is probably transparent to the initial CPU mode without modification.
The boot procotol would need to define the initial hypervisor state: basically, a dormant identity configuration with all traps and fancy features turned off.
Right. Helpfully, that is (more or less) the processor reset state.
Even if it isn't, I think it makes sense to place this requirement on the bootloader. But you're probably right.
Cheers ---Dave
Dave Martin writes ("Re: ARM processor mode, kernel startup, Hyp / secure state"):
There may be a safe way to do this check -- for example, on ARM1176 and Cortex-A8 there is a CP14 debug status/control register that you can read which includes a flag indicating which world you're in. This isn't part of the architecture though and may be different/not possible on some CPUs.
Yes. I don't think we can write this into a boot protocol spec unless we have some architectural way of doing it.
All in all, it's better to engineer things so that the check doesn't need to be done at all.
Yes, in principle.
Note that in real deployments, the kernel is not an ELF image and therefore cannot have notes.
Yes. But in the future it will do, surely ? I mean, we're working on bootloader protocols and this part (to do with startup cpu states and ownership of the various privilege levels) is only part of it.
Ian.
On Tue, Aug 23, 2011 at 06:18:37PM +0100, Ian Jackson wrote:
Yes. But in the future it will do, surely ? I mean, we're working on bootloader protocols and this part (to do with startup cpu states and ownership of the various privilege levels) is only part of it.
No - at least not yet, and I don't see any reason why we'd ever want to use raw ELF.
The problem with ELF is that it carries lots of baggage around which platforms just don't need. Fine if it's on a hard disk and you're booting a desktop, but not for embedded where space tends to be a premium.
And most of that additional data just isn't required.
Plus, I'd point out that _no one_ is talking about boot protocols. Not outside linaro at any rate. Which means that they're doing effective closed-door discussion.
Russell King - ARM Linux writes ("Re: ARM processor mode, kernel startup, Hyp / secure state"):
On Tue, Aug 23, 2011 at 06:18:37PM +0100, Ian Jackson wrote:
Yes. But in the future it will do, surely ? I mean, we're working on bootloader protocols and this part (to do with startup cpu states and ownership of the various privilege levels) is only part of it.
No - at least not yet, and I don't see any reason why we'd ever want to use raw ELF.
The problem with ELF is that it carries lots of baggage around which platforms just don't need. Fine if it's on a hard disk and you're booting a desktop, but not for embedded where space tends to be a premium.
Hrm, yes.
Plus, I'd point out that _no one_ is talking about boot protocols. Not outside linaro at any rate. Which means that they're doing effective closed-door discussion.
I'm not sure whether by "outside linaro" you mean "outside some kind of top secret Linaro meetings". I went to an open meeting of Linaro developers touching on exactly these kind of topics, and everything was public and there was substantial interest.
I'm not a Linaro developer; but I have CC'd their boot-architecture list which AIUI is where some of the Linaro folks were thinking of discussing boot protocols.
Just to be clear: I don't have a particular axe to grind, except that I would like there to be some kind of defined and reasonably stable interface, and that obviously we (Xen.org) need that interface to support access to Hyp mode (which is something that Linux needs too if we are to have a properly supported KVM on ARM).
Are linux-arm-kernel@infradead and boot-architecture@linaro the right places to try to have this conversation ?
Ian.
On Tue, 23 Aug 2011, Ian Jackson wrote:
Russell King - ARM Linux writes ("Re: ARM processor mode, kernel startup, Hyp / secure state"):
Plus, I'd point out that _no one_ is talking about boot protocols. Not outside linaro at any rate. Which means that they're doing effective closed-door discussion.
I'm not sure whether by "outside linaro" you mean "outside some kind of top secret Linaro meetings". I went to an open meeting of Linaro developers touching on exactly these kind of topics, and everything was public and there was substantial interest.
For the record, this Linaro topic and the progress done is documented here: https://blueprints.launchpad.net/linaro/+spec/linaro-kernel-o-bootarchitectu...
Nothing (yet) in there that would highly passionate you Russell I'm afraid. This is probably why you didn't hear of it before. But the discussion and the mailing list is public and open to anyone interested.
I'm not a Linaro developer; but I have CC'd their boot-architecture list which AIUI is where some of the Linaro folks were thinking of discussing boot protocols.
Just to be clear: I don't have a particular axe to grind, except that I would like there to be some kind of defined and reasonably stable interface, and that obviously we (Xen.org) need that interface to support access to Hyp mode (which is something that Linux needs too if we are to have a properly supported KVM on ARM).
Are linux-arm-kernel@infradead and boot-architecture@linaro the right places to try to have this conversation ?
Absolutely.
Nicolas
Dave,
On Tue, Aug 23, 2011 at 05:50:19PM +0100, Dave Martin wrote:
On Tue, Aug 23, 2011 at 3:52 PM, Ian Jackson Ian.Jackson@eu.citrix.com wrote:
At first I thought that the best thing to do would be to boot the kernel in any suitable mode, and have the kernel automatically detect the starting mode. I started writing code in linux's head.S to do this. However, detecting whether we are in secure state is very difficult: it involves deliberately risking an undefined instruction trap. The code for this was getting rather long and involved.
There may be a safe way to do this check -- for example, on ARM1176 and Cortex-A8 there is a CP14 debug status/control register that you can read which includes a flag indicating which world you're in. This isn't part of the architecture though and may be different/not possible on some CPUs.
Please don't do this! Accessing the debug registers via the CP14 registers is like playing russian roulette with a machine gun, especially when you have various hypervisor registers and hardware lock registers to contend with. For 3.2, I will be guarding all of the hw_breakpoint init debug probing with an undef_hook because I'm sick of blowing my head off when systems are configured to keep debug out.
All in all, it's better to engineer things so that the check doesn't need to be done at all
Agreed. I think it's better to assume that you can't detect whether you're running in secure state or not.
Will
On Tue, Aug 23, 2011 at 11:18:00PM +0100, Will Deacon wrote:
Dave,
On Tue, Aug 23, 2011 at 05:50:19PM +0100, Dave Martin wrote:
On Tue, Aug 23, 2011 at 3:52 PM, Ian Jackson Ian.Jackson@eu.citrix.com wrote:
At first I thought that the best thing to do would be to boot the kernel in any suitable mode, and have the kernel automatically detect the starting mode. I started writing code in linux's head.S to do this. However, detecting whether we are in secure state is very difficult: it involves deliberately risking an undefined instruction trap. The code for this was getting rather long and involved.
There may be a safe way to do this check -- for example, on ARM1176 and Cortex-A8 there is a CP14 debug status/control register that you can read which includes a flag indicating which world you're in. This isn't part of the architecture though and may be different/not possible on some CPUs.
Please don't do this! Accessing the debug registers via the CP14 registers is like playing russian roulette with a machine gun, especially when you have various hypervisor registers and hardware lock registers to contend with. For 3.2, I will be guarding all of the hw_breakpoint init debug probing with an undef_hook because I'm sick of blowing my head off when systems are configured to keep debug out.
Heh -- I didn't think it was quite that scary, but point taken.
All in all, it's better to engineer things so that the check doesn't need to be done at all
Agreed. I think it's better to assume that you can't detect whether you're running in secure state or not.
Will
Ian,
(I'll be on holiday from tomorrow, so not able to follow up until September).
On 23 August 2011 15:52, Ian Jackson Ian.Jackson@eu.citrix.com wrote:
We're looking into porting Xen to the ARM A15 architecture. In this context, it's necessary to arrange for the Xen hypervisor to be entered from the boot loader in an appropriate processor mode.
We had some discussions both with ARM partners and Christoffer and it is not clear how we go about initialising the Hyp mode. Most SoC vendors would probably just run Linux in non-secure SVC mode and would not touch the Hyp mode at all. Linux would issue an SMC to set the HVBAR (could do HVC but I expect that there is no secure monitor code). Once this is set, assuming that the Hyp MMU is disabled, Linux can invoke an HVC and initialise the Hyp mode.
The problem is that the SMC API is specific to a secure monitor implementation. An alternative would be to ask the vendors to get some simple Hyp code which would trap access to HVBAR for example and set it with that value.
So I propose the following approach:
1. The kernel will advertise via ELF notes what modes it may be started in. The possible modes will be: (i) secure monitor mode (ii) non-secure hypervisor mode (iii) non-secure kernel mode
From the above, I would just allow (iii). As Dave pointed out, we
don't have the ELF information at this point.
2. The bootloader will select the first mode from the three listed above which is supported by both the processor and the kernel to be loaded, and transition the processor to that mode. If this involves dropping out of secure or hypervisor mode, it will put those modes permanently beyond use.
I'm not considering this step because of my point above.
3. The kernel will examine CPSR to determine which of the three possibilities above has happened, and: (a) If started in monitor mode: * Grant access to everything to non-secure state * Set the non-secure copies of the various CP15 registers which don't have a sane value on cpu reset * Install a trivial monitor vector which unconditionally copies r0 to MVBAR and returns * Switch to non-secure Hyp mode (if available) and do (b), or non-secure Kernel mode (if Hyp mode not supported). (b) If started in Hyp mode: * Install a trivial hypervisor vector which unconditionally copies r0 to HVBAR and returns (c) Rest of startup.
Maybe we could allow (b) if SoC vendors don't provide a different API to set the HVBAR. But it means that kernels not aware of this feature would fail.
Catalin Marinas writes ("Re: ARM processor mode, kernel startup, Hyp / secure state"):
(I'll be on holiday from tomorrow, so not able to follow up until September).
Right. Let's try to be quick :-).
We had some discussions both with ARM partners and Christoffer and it is not clear how we go about initialising the Hyp mode. Most SoC vendors would probably just run Linux in non-secure SVC mode and would not touch the Hyp mode at all. Linux would issue an SMC to set the HVBAR (could do HVC but I expect that there is no secure monitor code). Once this is set, assuming that the Hyp MMU is disabled, Linux can invoke an HVC and initialise the Hyp mode.
Well, yes. Do SoC vendors all expect to own the secure state ?
If so then the answer is perhaps simply that they should boot Linux in Hyp mode (and expect Linux to drop to ordinary svc mode).
The problem is that the SMC API is specific to a secure monitor implementation. An alternative would be to ask the vendors to get some simple Hyp code which would trap access to HVBAR for example and set it with that value.
That's seems to be extending the range of ways in which the firmware can make things go wrong, and broadening the interface to the firmware.
From the above, I would just allow (iii). As Dave pointed out, we don't have the ELF information at this point.
If we had a more functional bootloader setup (which AIUI is definitely planned) we could do.
(b) If started in Hyp mode: * Install a trivial hypervisor vector which unconditionally copies r0 to HVBAR and returns (c) Rest of startup.
Maybe we could allow (b) if SoC vendors don't provide a different API to set the HVBAR. But it means that kernels not aware of this feature would fail.
It turns out that at least your kernel boots just fine if started in Hyp mode :-).
Ian.
On 23 August 2011 18:23, Ian Jackson Ian.Jackson@eu.citrix.com wrote:
Catalin Marinas writes ("Re: ARM processor mode, kernel startup, Hyp / secure state"):
We had some discussions both with ARM partners and Christoffer and it is not clear how we go about initialising the Hyp mode. Most SoC vendors would probably just run Linux in non-secure SVC mode and would not touch the Hyp mode at all. Linux would issue an SMC to set the HVBAR (could do HVC but I expect that there is no secure monitor code). Once this is set, assuming that the Hyp MMU is disabled, Linux can invoke an HVC and initialise the Hyp mode.
Well, yes. Do SoC vendors all expect to own the secure state ?
Maybe some parts of it, that's why it's hard to standardise some kind of API. Since the platform boots in secure mode, they need at least some code which they own, in addition to other stuff like secure transactions.
If so then the answer is perhaps simply that they should boot Linux in Hyp mode (and expect Linux to drop to ordinary svc mode).
That would be a solution as it is relatively simple for Linux to check the mode and switch to SVC. The issue I see is that Linux is not the first non-secure code to run but you usually get some boot loader like U-Boot. You would have to make the latter aware of the Hyp mode as well and many SoC vendors may not be willing to put the effort in (we can help by providing boot loader fixes but the boot loader may not always be open source).
(I may get back online towards the weekend)
On Tue, Aug 23, 2011 at 11:12 PM, Catalin Marinas catalin.marinas@arm.comwrote:
On 23 August 2011 18:23, Ian Jackson Ian.Jackson@eu.citrix.com wrote:
Catalin Marinas writes ("Re: ARM processor mode, kernel startup, Hyp /
secure state"):
We had some discussions both with ARM partners and Christoffer and it is not clear how we go about initialising the Hyp mode. Most SoC vendors would probably just run Linux in non-secure SVC mode and would not touch the Hyp mode at all. Linux would issue an SMC to set the HVBAR (could do HVC but I expect that there is no secure monitor code). Once this is set, assuming that the Hyp MMU is disabled, Linux can invoke an HVC and initialise the Hyp mode.
Well, yes. Do SoC vendors all expect to own the secure state ?
Maybe some parts of it, that's why it's hard to standardise some kind of API. Since the platform boots in secure mode, they need at least some code which they own, in addition to other stuff like secure transactions.
If so then the answer is perhaps simply that they should boot Linux in Hyp mode (and expect Linux to drop to ordinary svc mode).
That would be a solution as it is relatively simple for Linux to check the mode and switch to SVC. The issue I see is that Linux is not the first non-secure code to run but you usually get some boot loader like U-Boot. You would have to make the latter aware of the Hyp mode as well and many SoC vendors may not be willing to put the effort in (we can help by providing boot loader fixes but the boot loader may not always be open source).
I second that booting in Hyp mode seems like a clean solution, although I'm not an export on U-Boot. What I think we want to avoid is being unable to use KVM on a platform, that has the architecture virtualization support, due to the boot process and an incompatible API.
However, there's no way to use Hyp mode without _some_ support from the earlier stages in the boot process am I right? In that case, the question is simply whether we should expect platforms to boot the kernel in Hyp mode or if we can expect vendors to agree on a stable secure monitor API for initializing Hyp mode.
Personally I don't care either way and it won't effect the implementation of KVM in any significant way. Worst case, we can have a common in-kernel API that sets the HVBAR through either method depending on the platform, but would be great if this could be avoided.
Hi Ian,
On Tue, Aug 23, 2011 at 06:23:32PM +0100, Ian Jackson wrote:
Catalin Marinas writes ("Re: ARM processor mode, kernel startup, Hyp / secure state"):
(I'll be on holiday from tomorrow, so not able to follow up until September).
Right. Let's try to be quick :-).
Ha, you've got enough people interested now that there's no need to rush!
We had some discussions both with ARM partners and Christoffer and it is not clear how we go about initialising the Hyp mode. Most SoC vendors would probably just run Linux in non-secure SVC mode and would not touch the Hyp mode at all. Linux would issue an SMC to set the HVBAR (could do HVC but I expect that there is no secure monitor code). Once this is set, assuming that the Hyp MMU is disabled, Linux can invoke an HVC and initialise the Hyp mode.
Well, yes. Do SoC vendors all expect to own the secure state ?
If so then the answer is perhaps simply that they should boot Linux in Hyp mode (and expect Linux to drop to ordinary svc mode).
I think it's important to separate the problems of secure boot with the problems of installing a hypervisor. Whatever happens in secure world, we can expect to be dropped at either HYP mode or non-secure SVC mode. Sure, on a dev board you might run directly in the secure world so there's a bit of extra work to do to get out of that but then we can just drop into HVC mode and forget about it.
If we had a more functional bootloader setup (which AIUI is definitely planned) we could do.
(b) If started in Hyp mode: * Install a trivial hypervisor vector which unconditionally copies r0 to HVBAR and returns (c) Rest of startup.
From discussions that I've been having with the technical architects at ARM
and various bootloader people, it looks like we have two scenarios [this is reiterating a lot of what you've said but I think it's important]:
After the system has taken care of whatever secure initialisation was required, it moves into HYP mode. At this point there are two things that can happen:
1.0: HYP mode boot loader (uboot, UEFI) runs and installs Linux at the same privilege level 1.1: Raw Linux boots, and detects HYP mode 1.2: Sets up basic init, with HVC trampoline for installing KVM/Xen later 1.3: Switch to SVC mode 1.4: Continue booting Linux as normal
Or:
2.0: HYP layer installed as part of system firmware 2.1: Hypervisor system init, HVC handlers installed 2.2: Switch to SVC mode 2.3: SVC mode boot loader (uboot, UEFI) runs and installs Linux 2.4: Linux boots as normal
Obviously, as kernel developers, we prefer the first approach because it gives us control over the HYP interface. In practice, I expect to see some platforms that take the second approach and I think that we have to support it. The best we can hope for in this case is that the HVC layer has an API for installing another hypervisor. If that API is standardised, then even better.
Unlike the fragmented secure monitor API that currently exists across different platforms, it's really in the interests of the vendor to standardise on the HYP interface and provide calls to install code, otherwise they run the risk of producing what is essentially a closed system.
Will
Will Deacon writes ("Re: ARM processor mode, kernel startup, Hyp / secure state"):
From discussions that I've been having with the technical architects at ARM and various bootloader people, it looks like we have two scenarios [this is reiterating a lot of what you've said but I think it's important]:
Right.
After the system has taken care of whatever secure initialisation was required, it moves into HYP mode. At this point there are two things that can happen:
1.0: HYP mode boot loader (uboot, UEFI) runs and installs Linux at the same privilege level 1.1: Raw Linux boots, and detects HYP mode 1.2: Sets up basic init, with HVC trampoline for installing KVM/Xen later 1.3: Switch to SVC mode 1.4: Continue booting Linux as normal
Since we have discovered that this approach will work fine with existing kernels, even ones which are completely unaware of Hyp mode, I think this is the right approach. It can be implemented by boot loaders etc. right away.
Ian.
On Wed, Aug 24, 2011 at 04:45:48PM +0100, Ian Jackson wrote:
Will Deacon writes ("Re: ARM processor mode, kernel startup, Hyp / secure state"):
After the system has taken care of whatever secure initialisation was required, it moves into HYP mode. At this point there are two things that can happen:
1.0: HYP mode boot loader (uboot, UEFI) runs and installs Linux at the same privilege level 1.1: Raw Linux boots, and detects HYP mode 1.2: Sets up basic init, with HVC trampoline for installing KVM/Xen later 1.3: Switch to SVC mode 1.4: Continue booting Linux as normal
Since we have discovered that this approach will work fine with existing kernels, even ones which are completely unaware of Hyp mode, I think this is the right approach. It can be implemented by boot loaders etc. right away.
It's certainly the simplest method from our point of view, but there is still the necessity to support the other situation I described rather than retrofit it when hardware that does it turns up [which it will].
The next step is to define our interface to the HVC trampoline. I think it would be good if this interface could be reused by vendors who provide their own hypervisor layer as this would reduce the need for lots of distinct HVC APIs that essentially all do the same thing. It might be a good idea to `reserve' a high region of the HVC space for use by Linux rather than index the calls from 0, for example, which is likely to conflict with other hypervisor implementations. I think the trampoline only needs a handful of HVC calls (if that) anyway.
What do you reckon?
Will
Will Deacon writes ("Re: ARM processor mode, kernel startup, Hyp / secure state"):
The next step is to define our interface to the HVC trampoline. I think it would be good if this interface could be reused by vendors who provide their own hypervisor layer as this would reduce the need for lots of distinct HVC APIs that essentially all do the same thing. It might be a good idea to `reserve' a high region of the HVC space for use by Linux rather than index the calls from 0, for example, which is likely to conflict with other hypervisor implementations. I think the trampoline only needs a handful of HVC calls (if that) anyway.
What do you reckon?
I think the trampoline needs to have one hypercall: "set HVBAR". (And, implicitly, it needs to be specified that the resulting vectors will run without physical-to-virtual page translation, etc.) That is sufficient to bootstrap all the rest of the way up - and that other bootstrapping code would have to exist anyway.
So I'm imagining that the kvm kernel module would, when loaded, make that "set HVBAR" once on each cpu, replacing the trampoline hypervisor with its own set of hypercall vectors.
Ian.
On Wed, Aug 24, 2011 at 05:09:01PM +0100, Ian Jackson wrote:
Will Deacon writes ("Re: ARM processor mode, kernel startup, Hyp / secure state"):
The next step is to define our interface to the HVC trampoline. I think it would be good if this interface could be reused by vendors who provide their own hypervisor layer as this would reduce the need for lots of distinct HVC APIs that essentially all do the same thing. It might be a good idea to `reserve' a high region of the HVC space for use by Linux rather than index the calls from 0, for example, which is likely to conflict with other hypervisor implementations. I think the trampoline only needs a handful of HVC calls (if that) anyway.
What do you reckon?
I think the trampoline needs to have one hypercall: "set HVBAR". (And, implicitly, it needs to be specified that the resulting vectors will run without physical-to-virtual page translation, etc.) That is sufficient to bootstrap all the rest of the way up - and that other bootstrapping code would have to exist anyway.
I also thought you might want a hypercall that gives you a version number (i.e. a way to identify the hypervisor trampoline in case it changes) and we might also want something for doing a system reset (for example, if a kernel that hasn't installed a hypervisor wants to kexec another kernel image).
So I'm imagining that the kvm kernel module would, when loaded, make that "set HVBAR" once on each cpu, replacing the trampoline hypervisor with its own set of hypercall vectors.
That's the plan!
Will
Will Deacon writes ("Re: ARM processor mode, kernel startup, Hyp / secure state"):
On Wed, Aug 24, 2011 at 05:09:01PM +0100, Ian Jackson wrote:
I think the trampoline needs to have one hypercall: "set HVBAR". (And, implicitly, it needs to be specified that the resulting vectors will run without physical-to-virtual page translation, etc.) That is sufficient to bootstrap all the rest of the way up - and that other bootstrapping code would have to exist anyway.
I also thought you might want a hypercall that gives you a version number (i.e. a way to identify the hypervisor trampoline in case it changes)
Since the hypervisor trampoline was installed by the very same kernel which is running, there is no need for any defined interface or version numbers.
and we might also want something for doing a system reset (for example, if a kernel that hasn't installed a hypervisor wants to kexec another kernel image).
No, if a kernel has installed a hypervisor and wishes to kexec then it must start the new kernel in Hyp mode - since that is what we are proposing to specify as the cpu mode to use when booting the kernel.
So the outgoing kernel simply switches to Hyp mode, tears down Hyp mode's paging (and probably the rest of the paging too), and then enters the new kernel in Hyp mode with paging off. The new kernel then installs its own hypervisor trampoline.
If the outgoing kernel hasn't installed a hypervisor it's even easier: simply do the usual teardown, and then switch to Hyp mode using its existing trampoline and then enter the new kernel.
Ian.
On Thu, Aug 25, 2011 at 11:13:40AM +0100, Ian Jackson wrote:
Will Deacon writes ("Re: ARM processor mode, kernel startup, Hyp / secure state"):
On Wed, Aug 24, 2011 at 05:09:01PM +0100, Ian Jackson wrote:
I think the trampoline needs to have one hypercall: "set HVBAR". (And, implicitly, it needs to be specified that the resulting vectors will run without physical-to-virtual page translation, etc.) That is sufficient to bootstrap all the rest of the way up - and that other bootstrapping code would have to exist anyway.
I also thought you might want a hypercall that gives you a version number (i.e. a way to identify the hypervisor trampoline in case it changes)
Since the hypervisor trampoline was installed by the very same kernel which is running, there is no need for any defined interface or version numbers.
Sure, but when we are confronted with a hypervisor that we didn't write ourselves it might be nice if it looks the same as our Linux trampoline. If we don't define this interface, that can never happen. So at the very least, let's define the HVC number that we want to use for our single hypercall.
and we might also want something for doing a system reset (for example, if a kernel that hasn't installed a hypervisor wants to kexec another kernel image).
No, if a kernel has installed a hypervisor and wishes to kexec then it must start the new kernel in Hyp mode - since that is what we are proposing to specify as the cpu mode to use when booting the kernel.
Agreed.
So the outgoing kernel simply switches to Hyp mode, tears down Hyp mode's paging (and probably the rest of the paging too), and then enters the new kernel in Hyp mode with paging off. The new kernel then installs its own hypervisor trampoline.
If the outgoing kernel hasn't installed a hypervisor it's even easier: simply do the usual teardown, and then switch to Hyp mode using its existing trampoline and then enter the new kernel.
Thanks for the explanation. I was concerned about losing state across the HVC but if it just returns back to you in the higher privilege level then that's not a problem.
Will
On Wed, Aug 24, 2011 at 02:51:09PM +0100, Will Deacon wrote:
I think it's important to separate the problems of secure boot with the problems of installing a hypervisor. Whatever happens in secure world, we can expect to be dropped at either HYP mode or non-secure SVC mode. Sure, on a dev board you might run directly in the secure world so there's a bit of extra work to do to get out of that but then we can just drop into HVC mode and forget about it.
I think you're painting a very simple picture there.
If the kernel has been handed over to while in secure mode, that's probably because there is no secure monitor implemented. If there's no secure monitor implemented, there's no code in place to handle SMC instructions. To make things worse, if we drop out into non-secure mode, due to the lack of working SMC instructions, we have no way to access the various registers we need to.
So, if the kernel is booted in secure mode and wants to drop into non- secure mode, we need a separate blob of code to install as our own secure monitor to provide these services.
Unlike the fragmented secure monitor API that currently exists across different platforms, it's really in the interests of the vendor to standardise on the HYP interface and provide calls to install code, otherwise they run the risk of producing what is essentially a closed system.
On the other hand, I'm sure vendors are already thinking along those lines - the precident has been set by the secure monitor API mess, so if it can be made to "work" there...
On Wed, Aug 24, 2011 at 06:09:53PM +0100, Russell King - ARM Linux wrote:
On Wed, Aug 24, 2011 at 02:51:09PM +0100, Will Deacon wrote:
I think it's important to separate the problems of secure boot with the problems of installing a hypervisor. Whatever happens in secure world, we can expect to be dropped at either HYP mode or non-secure SVC mode. Sure, on a dev board you might run directly in the secure world so there's a bit of extra work to do to get out of that but then we can just drop into HVC mode and forget about it.
I think you're painting a very simple picture there.
Yes, I was trying to avoid derailing the discussion because the secure - non-secure transition should be treated separately to the hypervisor stuff.
If the kernel has been handed over to while in secure mode, that's probably because there is no secure monitor implemented. If there's no secure monitor implemented, there's no code in place to handle SMC instructions. To make things worse, if we drop out into non-secure mode, due to the lack of working SMC instructions, we have no way to access the various registers we need to.
So, if the kernel is booted in secure mode and wants to drop into non- secure mode, we need a separate blob of code to install as our own secure monitor to provide these services.
Indeed. I'm not aware of any open-source monitor implementations and I'm not sure whether it's something worth investing effort in.
Unlike the fragmented secure monitor API that currently exists across different platforms, it's really in the interests of the vendor to standardise on the HYP interface and provide calls to install code, otherwise they run the risk of producing what is essentially a closed system.
On the other hand, I'm sure vendors are already thinking along those lines - the precident has been set by the secure monitor API mess, so if it can be made to "work" there...
That's partly why I'm trying to keep the two problems separate (but I doubt it will make any difference!). That said, I firmly believe that whatever we do, we *will* see platforms which install a hypervisor before dropping to NS-SVC and booting Linux. We can either give up in these cases (which is fine until the hardware becomes available and people want to use it) or we can define a simple HVC interface that the kernel can use and hope people implement that. If they don't, we'll end up in the same situation where we are for the secure stuff but at least we tried. Unlike the SVC mess, you can get away with a single (custom) HVC to bootstrap the thing with an interface that we understand.
Will
On Wed, Aug 24, 2011 at 6:09 PM, Russell King - ARM Linux linux@arm.linux.org.uk wrote:
On Wed, Aug 24, 2011 at 02:51:09PM +0100, Will Deacon wrote:
I think it's important to separate the problems of secure boot with the problems of installing a hypervisor. Whatever happens in secure world, we can expect to be dropped at either HYP mode or non-secure SVC mode. Sure, on a dev board you might run directly in the secure world so there's a bit of extra work to do to get out of that but then we can just drop into HVC mode and forget about it.
I think you're painting a very simple picture there.
If the kernel has been handed over to while in secure mode, that's probably because there is no secure monitor implemented. If there's no secure monitor implemented, there's no code in place to handle SMC instructions. To make things worse, if we drop out into non-secure mode, due to the lack of working SMC instructions, we have no way to access the various registers we need to.
So, if the kernel is booted in secure mode and wants to drop into non- secure mode, we need a separate blob of code to install as our own secure monitor to provide these services.
Unlike the fragmented secure monitor API that currently exists across different platforms, it's really in the interests of the vendor to standardise on the HYP interface and provide calls to install code, otherwise they run the risk of producing what is essentially a closed system.
On the other hand, I'm sure vendors are already thinking along those lines - the precident has been set by the secure monitor API mess, so if it can be made to "work" there...
A few thoughts from my end...
The kernel potentially has a lot of work to do if it is to support virtualisation when booted in the Secure World, because the kernel needs to switch to the Normal World -- security controllers on the bus will need to be reprogrammed etc., and the kernel would also need to retain some Secure memory somewhere for handling SMC -- i.e., per-SoC code and configuration which just doesn't exist in the kernel yet. The kernel does not need a "real" monitor in this scenario, but it will still need to get back into the Secure World to do certain low-level operations as we observe on e.g. OMAP. This is not trivial since SMC causes a change of memory space as well as privilege.
This potentially gets quite complex and messy, and the kernel doesn't really care; my gut feeling is that we probably don't want to go there -- it's a fair amount of extra code from which we don't really get much benefit.
So, it could be reasonable to _require_ linux to be started in the Normal World if linux will install its own hypervisor, with suitable (possibly minimal) SMC and HVC handlers/stubs provided by the bootloader or (more probably) firmware.
Hopefully, those SMC/HVC interfaces will get reasonably standardised, but in the meantime a bit of SoC-specific code in linux should be adequate to smooth any differences. Vendors likely have a strong incentive to provide adequate and working functionality at these interfaces because failure to do so could be a significant market disadvantage.
Cheers ---Dave
On Tue, Aug 23, 2011 at 06:17:04PM +0100, Catalin Marinas wrote:
Maybe we could allow (b) if SoC vendors don't provide a different API to set the HVBAR. But it means that kernels not aware of this feature would fail.
Oh, does this mean that ARM Ltd are starting to see the light in having a common defined secure API at last - at least a subset - something which I've mentioned several times...
On Tue, Aug 23, 2011 at 7:42 PM, Russell King - ARM Linux linux@arm.linux.org.uk wrote:
On Tue, Aug 23, 2011 at 06:17:04PM +0100, Catalin Marinas wrote:
Maybe we could allow (b) if SoC vendors don't provide a different API to set the HVBAR. But it means that kernels not aware of this feature would fail.
Oh, does this mean that ARM Ltd are starting to see the light in having a common defined secure API at last - at least a subset - something which I've mentioned several times...
I've discussed this internally at ST-Ericsson at times.
There is something called the Trusted Execution Environment (TEE) which is standardized by Global Platform, a consortia where ARM are members, as are ST, Ericsson, NXP, Renesas, and TI (etc): http://www.globalplatform.org/membershipcurrentfull.asp
For Ux500 we have a driver for this API which can be found in an out-of-tree git, basically for Ux500 the ROM implements the service tunnel, and we call into the ROM to perform them. The ROM in turn will issue an SMC to execute the actual call to the secure world.
The TEE defines Trusted Applications (TA:s) that can be called, they use a GUID identifier on each side as handle. I think of it as some exec() call which returns the result in a pipe ... (I don't know if this analogy is correct)
Right now these TEE calls are mostly for performing this or that cryptographic operation on some buffer of data, keeping the secret crypto keys in the secure world. These calls are done from userspace with the kernel driver simply mediating them.
The basic question to which I have no answer is whether TEE is or is not intended to be used for low-level services like l2x0 on/off or so. Or if the idea never even crossed the minds of the people devising TEE, like say if they were having totally different use cases in their mind when they cooked it up.
If it was intended for such stuff, we/GlobalPlatform could define a few standard "TAs" that all platforms would implement, unless it's by nature too heavyweight for that.
Would be fun if someone with insight could fill in on this...
[TEE experts, smack my fingers if I got some detail wrong in here.]
Yours, Linus Walleij
Hi,
The GlobalPlatform TEE spec do not address this kinds of functions. The Trusted Applications (TAs) defined by this initiative runs in user space and do not have the rights to do poke in hw. An extension of the GP framework to support applications running in the TEE Core (supervisor mode) would be needed. This extension could be what we have running on the 8500 today. Basically we use the same GP Client API (Linux side) with the addition to the secure side OS to access functions running in supervisor mode. I.e. the secure side implementation is extended without creating a conflict or changing the GP Client API in Linux.
Regards /Martin
-----Original Message----- From: Linus Walleij [mailto:linus.walleij@linaro.org] Sent: den 24 augusti 2011 13:43 To: Russell King - ARM Linux Cc: Catalin Marinas; boot-architecture@lists.linaro.org; Ian Jackson; linux-arm-kernel@lists.infradead.org; android-virt@lists.cs.columbia.edu; Hakan ENGLUND; Martin HOVANG Subject: Re: ARM processor mode, kernel startup, Hyp / secure state
On Tue, Aug 23, 2011 at 7:42 PM, Russell King - ARM Linux linux@arm.linux.org.uk wrote:
On Tue, Aug 23, 2011 at 06:17:04PM +0100, Catalin Marinas wrote:
Maybe we could allow (b) if SoC vendors don't provide a different API to set the HVBAR. But it means that kernels not aware of this feature would fail.
Oh, does this mean that ARM Ltd are starting to see the light in having a common defined secure API at last - at least a subset - something which I've mentioned several times...
I've discussed this internally at ST-Ericsson at times.
There is something called the Trusted Execution Environment (TEE) which is standardized by Global Platform, a consortia where ARM are members, as are ST, Ericsson, NXP, Renesas, and TI (etc): http://www.globalplatform.org/membershipcurrentfull.asp
For Ux500 we have a driver for this API which can be found in an out-of-tree git, basically for Ux500 the ROM implements the service tunnel, and we call into the ROM to perform them. The ROM in turn will issue an SMC to execute the actual call to the secure world.
The TEE defines Trusted Applications (TA:s) that can be called, they use a GUID identifier on each side as handle. I think of it as some exec() call which returns the result in a pipe ... (I don't know if this analogy is correct)
Right now these TEE calls are mostly for performing this or that cryptographic operation on some buffer of data, keeping the secret crypto keys in the secure world. These calls are done from userspace with the kernel driver simply mediating them.
The basic question to which I have no answer is whether TEE is or is not intended to be used for low-level services like l2x0 on/off or so. Or if the idea never even crossed the minds of the people devising TEE, like say if they were having totally different use cases in their mind when they cooked it up.
If it was intended for such stuff, we/GlobalPlatform could define a few standard "TAs" that all platforms would implement, unless it's by nature too heavyweight for that.
Would be fun if someone with insight could fill in on this...
[TEE experts, smack my fingers if I got some detail wrong in here.]
Yours, Linus Walleij
boot-architecture@lists.linaro.org
-
Catalin Marinas -
Christoffer Dall -
Dave Martin -
Ian Jackson -
Linus Walleij -
Martin HOVANG -
Nicolas Pitre -
Russell King - ARM Linux -
Will Deacon