Hi all, a few weeks ago I (and a few others) started hacking on a proof-of-concept hypervisor port to Cortex-A15 which uses and requires ARMv7 virtualization extensions. The intention of this work was to find out how to best support ARM v7+ on Xen. See http://old-list-archives.xen.org/archives/html/xen-arm/2011-09/msg00013.html for more details.
I am pleased to announce that significant progress has been made, and that we now have a nascent Xen port for Cortex-A15. The port is based on xen-unstable (HG CS 8d6edc3d26d2) and written from scratch exploiting the latest virtualization, LPAE, GIC and generic timer support in hardware.
We started the work less than three months ago, but the port is already capable of booting a Linux 3.0 based virtual machine (dom0) up to a shell prompt on an ARM Architecture Envelope Model, configured to emulate an A15-based Versatile Express. In this context, we wanted to thank ARM for making the model available to us. Now we are looking forward to porting the tools and running multiple guests.
The code requires virtualization, LPAE and GIC support and therefore it won't be able to run on anything older than a Cortex-A15. On the other hand, thanks to this, it is very small and easy to read, write and understand. The implementation does not distinguish between PV and HVM guests: there is just one type of guests that would be comparable to Linux PV on HVM in the Xen X86 world, but with no need for Qemu emulation. The code only requires minimal changes to the Linux kernel: just enough to support PV drivers.
Even though we are currently targeting Versatile Express and Cortex-A15 we do intend to support other machines and other ARMv7 with virtualization extensions CPUs. We are also looking forward to ARMv8 and 64 bits support.
Given that porting Xen to Cortex-A15 could be done with so little code, we believe that the best course of action is to merge it into xen-unstable as quickly as possible. There are still few rough edges to sort out but we should be able to produce a clean and digestible patch series for submission to xen-devel within the next couple of months. I hope to see the first patches going to the list as soon as possible. We would very welcome any contributions, in the form of testing, code reviews and, of course, patches!
A git tree is available here:
git://xenbits.xen.org/people/sstabellini/xen-unstable.git arm
the gitweb url is the following:
http://xenbits.xen.org/gitweb/?p=people/sstabellini/xen-unstable.git/.git%3B...
And here is the full diff:
http://xenbits.xen.org/people/sstabellini/diff
We want to point out that this effort is in addition to Samsung's ongoing efforts to upstream Xen ARM to xen-unstable. Samsung's XenARM port allows virtualization of Xen on ARM CPUs prior to virtualization extensions and supports traditional PV guests.
I would like to thank Tim Deegan and Ian Campbell: if you spend some time reading the history, you'll see that this project wouldn't have been possible in such a short time without great contributions from them.
Cheers, Stefano
On Tuesday 29 November 2011, Stefano Stabellini wrote:
Hi all, a few weeks ago I (and a few others) started hacking on a proof-of-concept hypervisor port to Cortex-A15 which uses and requires ARMv7 virtualization extensions. The intention of this work was to find out how to best support ARM v7+ on Xen. See http://old-list-archives.xen.org/archives/html/xen-arm/2011-09/msg00013.html for more details.
I am pleased to announce that significant progress has been made, and that we now have a nascent Xen port for Cortex-A15. The port is based on xen-unstable (HG CS 8d6edc3d26d2) and written from scratch exploiting the latest virtualization, LPAE, GIC and generic timer support in hardware.
Very nice!
Do you have a pointer to the kernel sources for the Linux guest? Since Xen and KVM are both in an early working state right now, it would be very nice if we could agree on the guest model to make sure that it's always possible to run the same kernel in both (and potentially other future) hypervisors without modifications.
Arnd
Hi all,
I wanted to know how Xen-ARM for A15 will address following concerns:
- How will Xen-ARM for A15 support legacy guest environment like ARMv5 or ARMv6 ? - What if my Cortex-A15 board does not have a GIC with virtualization support ?
Best Regards, Anup Patel
On Wed, Nov 30, 2011 at 2:59 AM, Arnd Bergmann arnd@arndb.de wrote:
On Tuesday 29 November 2011, Stefano Stabellini wrote:
Hi all, a few weeks ago I (and a few others) started hacking on a proof-of-concept hypervisor port to Cortex-A15 which uses and requires ARMv7 virtualization extensions. The intention of this work was to find out how to best support ARM v7+ on Xen. See
http://old-list-archives.xen.org/archives/html/xen-arm/2011-09/msg00013.html
for more details.
I am pleased to announce that significant progress has been made, and that we now have a nascent Xen port for Cortex-A15. The port is based on xen-unstable (HG CS 8d6edc3d26d2) and written from scratch exploiting the latest virtualization, LPAE, GIC and generic timer support in hardware.
Very nice!
Do you have a pointer to the kernel sources for the Linux guest? Since Xen and KVM are both in an early working state right now, it would be very nice if we could agree on the guest model to make sure that it's always possible to run the same kernel in both (and potentially other future) hypervisors without modifications.
Arnd-- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
On Wed, 30 Nov 2011, Anup Patel wrote:
Hi all,
I wanted to know how Xen-ARM for A15 will address following concerns:
- How will Xen-ARM for A15 support legacy guest environment like ARMv5 or ARMv6 ?
It is not our focus at the moment; we are targeting operating systems that support a modern ARMv7 machine with GIC support. That said, it might be possible to run legacy guests in the future, introducing more emulation to the hypervisor.
- What if my Cortex-A15 board does not have a GIC with virtualization support ?
We expect most hardware vendors to provide a GIC with virtualization support. However if they do not, in order to support their boards we'll have to do more emulation in the hypervisor.
On Tue, 29 Nov 2011, Arnd Bergmann wrote:
On Tuesday 29 November 2011, Stefano Stabellini wrote:
Hi all, a few weeks ago I (and a few others) started hacking on a proof-of-concept hypervisor port to Cortex-A15 which uses and requires ARMv7 virtualization extensions. The intention of this work was to find out how to best support ARM v7+ on Xen. See http://old-list-archives.xen.org/archives/html/xen-arm/2011-09/msg00013.html for more details.
I am pleased to announce that significant progress has been made, and that we now have a nascent Xen port for Cortex-A15. The port is based on xen-unstable (HG CS 8d6edc3d26d2) and written from scratch exploiting the latest virtualization, LPAE, GIC and generic timer support in hardware.
Very nice!
Do you have a pointer to the kernel sources for the Linux guest?
We have very few changes to the Linux kernel at the moment (only 3 commits!), just enough to be able to issue hypercalls and start a PV console.
A git branch is available here (not ready for submission):
git://xenbits.xen.org/people/sstabellini/linux-pvhvm.git arm
the branch above is based on git://linux-arm.org/linux-2.6.git arm-lpae, even though guests don't really need lpae support to run on Xen.
Since Xen and KVM are both in an early working state right now, it would be very nice if we could agree on the guest model to make sure that it's always possible to run the same kernel in both (and potentially other future) hypervisors without modifications.
Yes, that would be ideal. We don't plan on making many changes other than enabling PV frontends and backends.
On Wednesday 30 November 2011, Stefano Stabellini wrote:
On Tue, 29 Nov 2011, Arnd Bergmann wrote:
On Tuesday 29 November 2011, Stefano Stabellini wrote:
Do you have a pointer to the kernel sources for the Linux guest?
We have very few changes to the Linux kernel at the moment (only 3 commits!), just enough to be able to issue hypercalls and start a PV console.
A git branch is available here (not ready for submission):
git://xenbits.xen.org/people/sstabellini/linux-pvhvm.git arm
Ok, interesting. There really isn't much of the platform support that I was expecting there. I finally found the information I was looking for in the xen construct_dom0() function:
167 regs->r0 = 0; /* SBZ */ 168 regs->r1 = 2272; /* Machine NR: Versatile Express */ 169 regs->r2 = 0xc0000100; /* ATAGS */
What this means is that you are emulating the current ARM/Keil reference board, at least to the degree that is necessary to get the guest started.
This is the same choice people have made for KVM, but it's not necessarily the best option in the long run. In particular, this board has a lot of hardware that you claim to have by putting the machine number there, when you don't really want to emulate it.
Pawell Moll is working on a variant of the vexpress code that uses the flattened device tree to describe the present hardware [1], and I think that would be a much better target for an official release. Ideally, the hypervisor should provide the device tree binary (dtb) to the guest OS describing the hardware that is actually there.
This would also be the place where you tell the guest that it should look for PV devices. I'm not familiar with how Xen announces PV devices to the guest on other architectures, but you have the choice between providing a full "binding", i.e. a formal specification in device tree format for the guest to detect PV devices in the same way as physical or emulated devices, or just providing a single place in the device tree in which the guest detects the presence of a xen device bus and then uses hcalls to find the devices on that bus.
Another topic is the question whether there are any hcalls that we should try to standardize before we get another architecture with multiple conflicting hcall APIs as we have on x86 and powerpc.
Arnd
On Wed, 2011-11-30 at 13:03 +0000, Arnd Bergmann wrote:
On Wednesday 30 November 2011, Stefano Stabellini wrote:
On Tue, 29 Nov 2011, Arnd Bergmann wrote:
On Tuesday 29 November 2011, Stefano Stabellini wrote:
Do you have a pointer to the kernel sources for the Linux guest?
We have very few changes to the Linux kernel at the moment (only 3 commits!), just enough to be able to issue hypercalls and start a PV console.
A git branch is available here (not ready for submission):
git://xenbits.xen.org/people/sstabellini/linux-pvhvm.git arm
Ok, interesting. There really isn't much of the platform support that I was expecting there. I finally found the information I was looking for in the xen construct_dom0() function:
167 regs->r0 = 0; /* SBZ */ 168 regs->r1 = 2272; /* Machine NR: Versatile Express */ 169 regs->r2 = 0xc0000100; /* ATAGS */
What this means is that you are emulating the current ARM/Keil reference board, at least to the degree that is necessary to get the guest started.
This is the same choice people have made for KVM, but it's not necessarily the best option in the long run. In particular, this board has a lot of hardware that you claim to have by putting the machine number there, when you don't really want to emulate it.
This code is actually setting up dom0 which (for the most part) sees the real hardware.
The hardcoding of the platform is just a short term hack.
Pawell Moll is working on a variant of the vexpress code that uses the flattened device tree to describe the present hardware [1], and I think that would be a much better target for an official release. Ideally, the hypervisor should provide the device tree binary (dtb) to the guest OS describing the hardware that is actually there.
Agreed. Our intention was to use DT so this fits perfectly with our plans.
For dom0 we would expose a (possibly filtered) version of the DT given to us by the firmware (e.g. we might hide a serial port to reserve it for Xen's use, we'd likely fiddle with the memory map etc).
For domU the DT would presumably be constructed by the toolstack (in dom0 userspace) as appropriate for the guest configuration. I guess this needn't correspond to any particular "real" hardware platform.
This would also be the place where you tell the guest that it should look for PV devices. I'm not familiar with how Xen announces PV devices to the guest on other architectures, but you have the choice between providing a full "binding", i.e. a formal specification in device tree format for the guest to detect PV devices in the same way as physical or emulated devices, or just providing a single place in the device tree in which the guest detects the presence of a xen device bus and then uses hcalls to find the devices on that bus.
On x86 there is an emulated PCI device which serves as the hooking point for the PV drivers. For ARM I don't think it would be unreasonable to have a DT entry instead. I think it would be fine just represent the root of the "xenbus" and further discovery would occur using the normal xenbus mechanisms (so not a full binding). AIUI for buses which are enumerable this is the preferred DT scheme to use.
Another topic is the question whether there are any hcalls that we should try to standardize before we get another architecture with multiple conflicting hcall APIs as we have on x86 and powerpc.
The hcall API we are currently targeting is the existing Xen API (at least the generic parts of it). These generally deal with fairly Xen specific concepts like grant tables etc.
Ian.
Arnd
[1] http://www.spinics.net/lists/arm-kernel/msg149604.html
Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wednesday 30 November 2011, Ian Campbell wrote:
On Wed, 2011-11-30 at 13:03 +0000, Arnd Bergmann wrote:
On Wednesday 30 November 2011, Stefano Stabellini wrote: This is the same choice people have made for KVM, but it's not necessarily the best option in the long run. In particular, this board has a lot of hardware that you claim to have by putting the machine number there, when you don't really want to emulate it.
This code is actually setting up dom0 which (for the most part) sees the real hardware.
Ok, I see.
Pawell Moll is working on a variant of the vexpress code that uses the flattened device tree to describe the present hardware [1], and I think that would be a much better target for an official release. Ideally, the hypervisor should provide the device tree binary (dtb) to the guest OS describing the hardware that is actually there.
Agreed. Our intention was to use DT so this fits perfectly with our plans.
For dom0 we would expose a (possibly filtered) version of the DT given to us by the firmware (e.g. we might hide a serial port to reserve it for Xen's use, we'd likely fiddle with the memory map etc).
Ah, very good.
For domU the DT would presumably be constructed by the toolstack (in dom0 userspace) as appropriate for the guest configuration. I guess this needn't correspond to any particular "real" hardware platform.
Correct, but it needs to correspond to some platform that is supported by the guest OS, which leaves the choice between emulating a real hardware platform, adding a completely new platform specifically for virtual machines, or something in between the two.
What I suggested to the KVM developers is to start out with the vexpress platform, but then generalize it to the point where it fits your needs. All hardware that one expects a guest to have (GIC, timer, ...) will still show up in the same location as on a real vexpress, while anything that makes no sense or is better paravirtualized (LCD, storage, ...) just becomes optional and has to be described in the device tree if it's actually there.
This would also be the place where you tell the guest that it should look for PV devices. I'm not familiar with how Xen announces PV devices to the guest on other architectures, but you have the choice between providing a full "binding", i.e. a formal specification in device tree format for the guest to detect PV devices in the same way as physical or emulated devices, or just providing a single place in the device tree in which the guest detects the presence of a xen device bus and then uses hcalls to find the devices on that bus.
On x86 there is an emulated PCI device which serves as the hooking point for the PV drivers. For ARM I don't think it would be unreasonable to have a DT entry instead. I think it would be fine just represent the root of the "xenbus" and further discovery would occur using the normal xenbus mechanisms (so not a full binding). AIUI for buses which are enumerable this is the preferred DT scheme to use.
In general that is the case, yes. One could argue that any software protocol between Xen and the guest is as good as any other, so it makes sense to use the device tree to describe all devices here. The counterargument to that is that Linux and other OSs already support Xenbus, so there is no need to come up with a new binding.
I don't care much either way, but I think it would be good to use similar solutions across all hypervisors. The two options that I've seen discussed for KVM were to use either a virtual PCI bus with individual virtio-pci devices as on the PC, or to use the new virtio-mmio driver and individually put virtio devices into the device tree.
Another topic is the question whether there are any hcalls that we should try to standardize before we get another architecture with multiple conflicting hcall APIs as we have on x86 and powerpc.
The hcall API we are currently targeting is the existing Xen API (at least the generic parts of it). These generally deal with fairly Xen specific concepts like grant tables etc.
Ok. It would of course still be possible to agree on an argument passing convention so that we can share the macros used to issue the hcalls, even if the individual commands are all different. I think I also remember talk about the need for a set of hypervisor independent calls that everyone should implement, but I can't remember what those were. Maybe we can split the number space into a range of some generic and some vendor specific hcalls?
Arnd
On Wed, 2011-11-30 at 14:32 +0000, Arnd Bergmann wrote:
I don't care much either way, but I think it would be good to use similar solutions across all hypervisors. The two options that I've seen discussed for KVM were to use either a virtual PCI bus with individual virtio-pci devices as on the PC, or to use the new virtio-mmio driver and individually put virtio devices into the device tree.
Let me just add that the virtio-mmio devices can already be instantiated from DT (see Documentation/devicetree/bindings/virtio/mmio.txt).
For A9-based VE I'd suggest placing them around 0x1001e000, eg.:
virtio_block@1001e000 { compatible = "virtio,mmio"; reg = <0x1001e000 0x100>; interrupts = <41>; }
Cheers!
Paweł
On Wed, 2011-11-30 at 14:32 +0000, Arnd Bergmann wrote:
On Wednesday 30 November 2011, Ian Campbell wrote:
On Wed, 2011-11-30 at 13:03 +0000, Arnd Bergmann wrote: For domU the DT would presumably be constructed by the toolstack (in dom0 userspace) as appropriate for the guest configuration. I guess this needn't correspond to any particular "real" hardware platform.
Correct, but it needs to correspond to some platform that is supported by the guest OS, which leaves the choice between emulating a real hardware platform, adding a completely new platform specifically for virtual machines, or something in between the two.
What I suggested to the KVM developers is to start out with the vexpress platform, but then generalize it to the point where it fits your needs. All hardware that one expects a guest to have (GIC, timer, ...) will still show up in the same location as on a real vexpress, while anything that makes no sense or is better paravirtualized (LCD, storage, ...) just becomes optional and has to be described in the device tree if it's actually there.
That's along the lines of what I was thinking as well.
The DT contains the address of GIC, timer etc as well right? So at least in principal we needn't provide e.g. the GIC at the same address as any real platform but in practice I expect we will.
In principal we could also offer the user options as to which particular platform a guest looks like.
This would also be the place where you tell the guest that it should look for PV devices. I'm not familiar with how Xen announces PV devices to the guest on other architectures, but you have the choice between providing a full "binding", i.e. a formal specification in device tree format for the guest to detect PV devices in the same way as physical or emulated devices, or just providing a single place in the device tree in which the guest detects the presence of a xen device bus and then uses hcalls to find the devices on that bus.
On x86 there is an emulated PCI device which serves as the hooking point for the PV drivers. For ARM I don't think it would be unreasonable to have a DT entry instead. I think it would be fine just represent the root of the "xenbus" and further discovery would occur using the normal xenbus mechanisms (so not a full binding). AIUI for buses which are enumerable this is the preferred DT scheme to use.
In general that is the case, yes. One could argue that any software protocol between Xen and the guest is as good as any other, so it makes sense to use the device tree to describe all devices here. The counterargument to that is that Linux and other OSs already support Xenbus, so there is no need to come up with a new binding.
Right.
I don't care much either way, but I think it would be good to use similar solutions across all hypervisors. The two options that I've seen discussed for KVM were to use either a virtual PCI bus with individual virtio-pci devices as on the PC, or to use the new virtio-mmio driver and individually put virtio devices into the device tree.
Another topic is the question whether there are any hcalls that we should try to standardize before we get another architecture with multiple conflicting hcall APIs as we have on x86 and powerpc.
The hcall API we are currently targeting is the existing Xen API (at least the generic parts of it). These generally deal with fairly Xen specific concepts like grant tables etc.
Ok. It would of course still be possible to agree on an argument passing convention so that we can share the macros used to issue the hcalls, even if the individual commands are all different.
I think it likely that we can all agree on a common calling convention for N-argument hypercalls. It doubt there are that many useful choices with conflicting requirements yet strongly compelling advantages.
I think I also remember talk about the need for a set of hypervisor independent calls that everyone should implement, but I can't remember what those were.
I'd not heard of this, maybe I just wasn't looking the right way though.
Maybe we can split the number space into a range of some generic and some vendor specific hcalls?
Ian.
On Wednesday 30 November 2011, Ian Campbell wrote:
On Wed, 2011-11-30 at 14:32 +0000, Arnd Bergmann wrote:
On Wednesday 30 November 2011, Ian Campbell wrote: What I suggested to the KVM developers is to start out with the vexpress platform, but then generalize it to the point where it fits your needs. All hardware that one expects a guest to have (GIC, timer, ...) will still show up in the same location as on a real vexpress, while anything that makes no sense or is better paravirtualized (LCD, storage, ...) just becomes optional and has to be described in the device tree if it's actually there.
That's along the lines of what I was thinking as well.
The DT contains the address of GIC, timer etc as well right? So at least in principal we needn't provide e.g. the GIC at the same address as any real platform but in practice I expect we will.
Yes.
In principal we could also offer the user options as to which particular platform a guest looks like.
At least when using a qemu based simulation. Most platforms have some characteristics that are not meaningful in a classic virtualization scenario, but it would certainly be helpful to use the virtualization extensions to run a kernel that was built for a particular platform faster than with pure qemu, when you want to test that kernel image.
It has been suggested in the past that it would be nice to run the guest kernel built for the same platform as the host kernel by default, but I think it would be much better to have just one platform that we end up using for guests on any host platform, unless there is a strong reason to do otherwise.
There is also ongoing restructuring in the ARM Linux kernel to allow running the same kernel binary on multiple platforms. While there is still a lot of work to be done, you should assume that we will finish it before you see lots of users in production, there is no need to plan for the current one-kernel-per-board case.
Ok. It would of course still be possible to agree on an argument passing convention so that we can share the macros used to issue the hcalls, even if the individual commands are all different.
I think it likely that we can all agree on a common calling convention for N-argument hypercalls. It doubt there are that many useful choices with conflicting requirements yet strongly compelling advantages.
Exactly. I think it's only lack of communication that has resulted in different interfaces for each hypervisor on the other architectures.
KVM and Xen at least both fall into the single-return-value category, so we should be able to agree on a calling conventions. KVM does not have an hcall API on ARM yet, and I see no reason not to use the same implementation that you have in the Xen guest.
Stefano, can you split out the generic parts of your asm/xen/hypercall.h file into a common asm/hypercall.h and submit it for review to the arm kernel list?
Arnd
On Wed, 30 Nov 2011, Arnd Bergmann wrote:
In principal we could also offer the user options as to which particular platform a guest looks like.
At least when using a qemu based simulation. Most platforms have some characteristics that are not meaningful in a classic virtualization scenario, but it would certainly be helpful to use the virtualization extensions to run a kernel that was built for a particular platform faster than with pure qemu, when you want to test that kernel image.
It has been suggested in the past that it would be nice to run the guest kernel built for the same platform as the host kernel by default, but I think it would be much better to have just one platform that we end up using for guests on any host platform, unless there is a strong reason to do otherwise.
There is also ongoing restructuring in the ARM Linux kernel to allow running the same kernel binary on multiple platforms. While there is still a lot of work to be done, you should assume that we will finish it before you see lots of users in production, there is no need to plan for the current one-kernel-per-board case.
It is very good to hear, I am counting on it.
Ok. It would of course still be possible to agree on an argument passing convention so that we can share the macros used to issue the hcalls, even if the individual commands are all different.
I think it likely that we can all agree on a common calling convention for N-argument hypercalls. It doubt there are that many useful choices with conflicting requirements yet strongly compelling advantages.
Exactly. I think it's only lack of communication that has resulted in different interfaces for each hypervisor on the other architectures.
It is also due to history: on X86 it was possible to issue hypercalls to Xen before VMCALL (the X86 version of HVC) was available.
KVM and Xen at least both fall into the single-return-value category, so we should be able to agree on a calling conventions. KVM does not have an hcall API on ARM yet, and I see no reason not to use the same implementation that you have in the Xen guest.
Stefano, can you split out the generic parts of your asm/xen/hypercall.h file into a common asm/hypercall.h and submit it for review to the arm kernel list?
Sure, I can do that. Usually the hypercall calling convention is very hypervisor specific, but if it turns out that we have the same requirements I happy to design a common interface.
On Wed, 2011-11-30 at 18:32 +0000, Stefano Stabellini wrote:
On Wed, 30 Nov 2011, Arnd Bergmann wrote:
KVM and Xen at least both fall into the single-return-value category, so we should be able to agree on a calling conventions. KVM does not have an hcall API on ARM yet, and I see no reason not to use the same implementation that you have in the Xen guest.
Stefano, can you split out the generic parts of your asm/xen/hypercall.h file into a common asm/hypercall.h and submit it for review to the arm kernel list?
Sure, I can do that. Usually the hypercall calling convention is very hypervisor specific, but if it turns out that we have the same requirements I happy to design a common interface.
I expect the only real decision to be made is hypercall page vs. raw hvc instruction.
The page was useful on x86 where there is a variety of instructions which could be used (at least for PV there was systenter/syscall/int, I think vmcall instruction differs between AMD and Intel also) and gives some additional flexibility. It's hard to predict but I don't think I'd expect that to be necessary on ARM.
Another reason for having a hypercall page instead of a raw instruction might be wanting to support 32 bit guests (from ~today) on a 64 bit hypervisor in the future and perhaps needing to do some shimming/arg translation. It would be better to aim for having the interface just be 32/64 agnostic but mistakes do happen.
Ian.
On Thu, Dec 01, 2011 at 10:26:37AM +0000, Ian Campbell wrote:
On Wed, 2011-11-30 at 18:32 +0000, Stefano Stabellini wrote:
On Wed, 30 Nov 2011, Arnd Bergmann wrote:
KVM and Xen at least both fall into the single-return-value category, so we should be able to agree on a calling conventions. KVM does not have an hcall API on ARM yet, and I see no reason not to use the same implementation that you have in the Xen guest.
Stefano, can you split out the generic parts of your asm/xen/hypercall.h file into a common asm/hypercall.h and submit it for review to the arm kernel list?
Sure, I can do that. Usually the hypercall calling convention is very hypervisor specific, but if it turns out that we have the same requirements I happy to design a common interface.
I expect the only real decision to be made is hypercall page vs. raw hvc instruction.
The page was useful on x86 where there is a variety of instructions which could be used (at least for PV there was systenter/syscall/int, I think vmcall instruction differs between AMD and Intel also) and gives some additional flexibility. It's hard to predict but I don't think I'd expect that to be necessary on ARM.
Another reason for having a hypercall page instead of a raw instruction might be wanting to support 32 bit guests (from ~today) on a 64 bit hypervisor in the future and perhaps needing to do some shimming/arg translation. It would be better to aim for having the interface just be 32/64 agnostic but mistakes do happen.
Given the way register banking is done on AArch64, issuing an HVC on a 32-bit guest OS doesn't require translation on a 64-bit hypervisor. We have a similar implementation at the SVC level (for 32-bit user apps on a 64-bit kernel), the only modification was where a 32-bit SVC takes a 64-bit parameter in two separate 32-bit registers, so packing needs to be done in a syscall wrapper.
I'm not closely involved with any of the Xen or KVM work but I would vote for using HVC than a hypercall page.
On Thursday 01 December 2011, Catalin Marinas wrote:
Given the way register banking is done on AArch64, issuing an HVC on a 32-bit guest OS doesn't require translation on a 64-bit hypervisor. We have a similar implementation at the SVC level (for 32-bit user apps on a 64-bit kernel), the only modification was where a 32-bit SVC takes a 64-bit parameter in two separate 32-bit registers, so packing needs to be done in a syscall wrapper.
How do you deal with signed integer arguments passed into SVC or HVC from a caller? If I understand the architecture correctly, the upper halves of the argument register end up zero-padded, while the callee expects sign-extension.
Arnd
On Thu, Dec 01, 2011 at 03:42:19PM +0000, Arnd Bergmann wrote:
On Thursday 01 December 2011, Catalin Marinas wrote:
Given the way register banking is done on AArch64, issuing an HVC on a 32-bit guest OS doesn't require translation on a 64-bit hypervisor. We have a similar implementation at the SVC level (for 32-bit user apps on a 64-bit kernel), the only modification was where a 32-bit SVC takes a 64-bit parameter in two separate 32-bit registers, so packing needs to be done in a syscall wrapper.
How do you deal with signed integer arguments passed into SVC or HVC from a caller? If I understand the architecture correctly, the upper halves of the argument register end up zero-padded, while the callee expects sign-extension.
If you treat it as an "int" (32-bit) and function prototype defined accordingly, then the generated code only accesses it as a W (rather than X) register and the top 32-bit part is ignored (no need for sign-extension). If it is defined as a "long" in the 32-bit world, then it indeed needs explicit conversion given the different sizes for long (for example sys_lseek, the second argument is a 'long' and we do explicit sign extension in the wrapper).
On Thursday 01 December 2011, Catalin Marinas wrote:
On Thu, Dec 01, 2011 at 03:42:19PM +0000, Arnd Bergmann wrote:
On Thursday 01 December 2011, Catalin Marinas wrote: How do you deal with signed integer arguments passed into SVC or HVC from a caller? If I understand the architecture correctly, the upper halves of the argument register end up zero-padded, while the callee expects sign-extension.
If you treat it as an "int" (32-bit) and function prototype defined accordingly, then the generated code only accesses it as a W (rather than X) register and the top 32-bit part is ignored (no need for sign-extension). If it is defined as a "long" in the 32-bit world, then it indeed needs explicit conversion given the different sizes for long (for example sys_lseek, the second argument is a 'long' and we do explicit sign extension in the wrapper).
Ok, so it's actually different from most other 64 bit architectures, which normally operate on 64-bit registers and expect the caller to do the correct sign-extension.
Doing the sign-extension for long arguments then falls into the same category as long long and unsigned long long arguments, which also need a wrapper, as you mentioned. Strictly speaking, we only need to do it for those where the long argument has a meaning outside of the 0..2^31 range, e.g. io_submit can only take small positive numbers although the type is 'long'.
What about unsigned long and pointer? Can we always rely on the upper half of the register to be zero-filled when we get an exception from 32 bit into 64 bit state, or do we also have to zero-extend those?
Arnd
On Thu, Dec 01, 2011 at 04:44:40PM +0000, Arnd Bergmann wrote:
On Thursday 01 December 2011, Catalin Marinas wrote:
On Thu, Dec 01, 2011 at 03:42:19PM +0000, Arnd Bergmann wrote:
On Thursday 01 December 2011, Catalin Marinas wrote: How do you deal with signed integer arguments passed into SVC or HVC from a caller? If I understand the architecture correctly, the upper halves of the argument register end up zero-padded, while the callee expects sign-extension.
If you treat it as an "int" (32-bit) and function prototype defined accordingly, then the generated code only accesses it as a W (rather than X) register and the top 32-bit part is ignored (no need for sign-extension). If it is defined as a "long" in the 32-bit world, then it indeed needs explicit conversion given the different sizes for long (for example sys_lseek, the second argument is a 'long' and we do explicit sign extension in the wrapper).
...
What about unsigned long and pointer? Can we always rely on the upper half of the register to be zero-filled when we get an exception from 32 bit into 64 bit state, or do we also have to zero-extend those?
They are also fine, no need for zero-extension.
On Thu, 2011-12-01 at 15:10 +0000, Catalin Marinas wrote:
On Thu, Dec 01, 2011 at 10:26:37AM +0000, Ian Campbell wrote:
On Wed, 2011-11-30 at 18:32 +0000, Stefano Stabellini wrote:
On Wed, 30 Nov 2011, Arnd Bergmann wrote:
KVM and Xen at least both fall into the single-return-value category, so we should be able to agree on a calling conventions. KVM does not have an hcall API on ARM yet, and I see no reason not to use the same implementation that you have in the Xen guest.
Stefano, can you split out the generic parts of your asm/xen/hypercall.h file into a common asm/hypercall.h and submit it for review to the arm kernel list?
Sure, I can do that. Usually the hypercall calling convention is very hypervisor specific, but if it turns out that we have the same requirements I happy to design a common interface.
I expect the only real decision to be made is hypercall page vs. raw hvc instruction.
The page was useful on x86 where there is a variety of instructions which could be used (at least for PV there was systenter/syscall/int, I think vmcall instruction differs between AMD and Intel also) and gives some additional flexibility. It's hard to predict but I don't think I'd expect that to be necessary on ARM.
Another reason for having a hypercall page instead of a raw instruction might be wanting to support 32 bit guests (from ~today) on a 64 bit hypervisor in the future and perhaps needing to do some shimming/arg translation. It would be better to aim for having the interface just be 32/64 agnostic but mistakes do happen.
Given the way register banking is done on AArch64, issuing an HVC on a 32-bit guest OS doesn't require translation on a 64-bit hypervisor.
The issue I was thinking about was struct packing for arguments passed as pointers etc rather than the argument registers themselves. Since the preference appears to be for raw hvc we should just be careful that they are agnostic in these.
Ian.
We have a similar implementation at the SVC level (for 32-bit user apps on a 64-bit kernel), the only modification was where a 32-bit SVC takes a 64-bit parameter in two separate 32-bit registers, so packing needs to be done in a syscall wrapper.
I'm not closely involved with any of the Xen or KVM work but I would vote for using HVC than a hypercall page.
On Thu, 1 Dec 2011, Ian Campbell wrote:
On Wed, 2011-11-30 at 18:32 +0000, Stefano Stabellini wrote:
On Wed, 30 Nov 2011, Arnd Bergmann wrote:
KVM and Xen at least both fall into the single-return-value category, so we should be able to agree on a calling conventions. KVM does not have an hcall API on ARM yet, and I see no reason not to use the same implementation that you have in the Xen guest.
Stefano, can you split out the generic parts of your asm/xen/hypercall.h file into a common asm/hypercall.h and submit it for review to the arm kernel list?
Sure, I can do that. Usually the hypercall calling convention is very hypervisor specific, but if it turns out that we have the same requirements I happy to design a common interface.
I expect the only real decision to be made is hypercall page vs. raw hvc instruction.
The page was useful on x86 where there is a variety of instructions which could be used (at least for PV there was systenter/syscall/int, I think vmcall instruction differs between AMD and Intel also) and gives some additional flexibility. It's hard to predict but I don't think I'd expect that to be necessary on ARM.
Another reason for having a hypercall page instead of a raw instruction might be wanting to support 32 bit guests (from ~today) on a 64 bit hypervisor in the future and perhaps needing to do some shimming/arg translation. It would be better to aim for having the interface just be 32/64 agnostic but mistakes do happen.
I always like to keep things as simple as possible, so I am in favor of using the raw hvc instruction. Besides with the bulk of mmu hypercalls gone, it should not be difficult to design a 32/64 bit agnostic interface.
On Wed, 2011-11-30 at 18:15 +0000, Arnd Bergmann wrote:
On Wednesday 30 November 2011, Ian Campbell wrote:
On Wed, 2011-11-30 at 14:32 +0000, Arnd Bergmann wrote:
On Wednesday 30 November 2011, Ian Campbell wrote: What I suggested to the KVM developers is to start out with the vexpress platform, but then generalize it to the point where it fits your needs. All hardware that one expects a guest to have (GIC, timer, ...) will still show up in the same location as on a real vexpress, while anything that makes no sense or is better paravirtualized (LCD, storage, ...) just becomes optional and has to be described in the device tree if it's actually there.
That's along the lines of what I was thinking as well.
The DT contains the address of GIC, timer etc as well right? So at least in principal we needn't provide e.g. the GIC at the same address as any real platform but in practice I expect we will.
Yes.
In principal we could also offer the user options as to which particular platform a guest looks like.
At least when using a qemu based simulation. Most platforms have some characteristics that are not meaningful in a classic virtualization scenario, but it would certainly be helpful to use the virtualization extensions to run a kernel that was built for a particular platform faster than with pure qemu, when you want to test that kernel image.
It has been suggested in the past that it would be nice to run the guest kernel built for the same platform as the host kernel by default, but I think it would be much better to have just one platform that we end up using for guests on any host platform, unless there is a strong reason to do otherwise.
Yes, I agree, certainly that is what we were planning to target in the first instance. Doing this means that we can get away with minimal emulation of actual hardware, relying instead on PV drivers or hardware virtualisation features.
Supporting specific board platforms as guests would be nice to have eventually. We would need to do more emulation (e.g. running qemu as a device model) for that case.
There is also ongoing restructuring in the ARM Linux kernel to allow running the same kernel binary on multiple platforms. While there is still a lot of work to be done, you should assume that we will finish it before you see lots of users in production, there is no need to plan for the current one-kernel-per-board case.
We were absolutely banking on targeting the results of this work, so that's good ;-)
Ian.
On 30 November 2011 11:39, Stefano Stabellini stefano.stabellini@eu.citrix.com wrote:
A git branch is available here (not ready for submission):
git://xenbits.xen.org/people/sstabellini/linux-pvhvm.git arm
the branch above is based on git://linux-arm.org/linux-2.6.git arm-lpae, even though guests don't really need lpae support to run on Xen.
Indeed, you don't really need LPAE. What you may need though is generic timers support for A15, it would allow less Hypervisor traps. For up-to-date architecture patches (well, development tree, not guaranteed to be stable), I would recommend this (they get into mainline at some point):
http://git.kernel.org/?p=linux/kernel/git/cmarinas/linux-arm-arch.git%3Ba=su...
Either use master or just cherry-pick the branches that you are interested in.
On Wed, 30 Nov 2011, Catalin Marinas wrote:
On 30 November 2011 11:39, Stefano Stabellini stefano.stabellini@eu.citrix.com wrote:
A git branch is available here (not ready for submission):
git://xenbits.xen.org/people/sstabellini/linux-pvhvm.git arm
the branch above is based on git://linux-arm.org/linux-2.6.git arm-lpae, even though guests don't really need lpae support to run on Xen.
Indeed, you don't really need LPAE. What you may need though is generic timers support for A15, it would allow less Hypervisor traps. For up-to-date architecture patches (well, development tree, not guaranteed to be stable), I would recommend this (they get into mainline at some point):
http://git.kernel.org/?p=linux/kernel/git/cmarinas/linux-arm-arch.git%3Ba=su...
Either use master or just cherry-pick the branches that you are interested in.
Thanks, I'll rebase on that.
-
Anup Patel -
Arnd Bergmann -
Catalin Marinas -
Ian Campbell -
Pawel Moll -
Stefano Stabellini