This email is just a quick summary of what we (Linaro) are planning in the way of QEMU work to support KVM on ARM Cortex-A15. The idea is to let people know what's coming up, find out if we've forgotten anything, and avoid people duplicating work unnecessarily. Most of this is based on a useful session at the recent 'ARM server mini-summit' in Orlando (UDS/Linaro Connect) at the beginning of this month.
The work we're currently proposing to do falls into three parts:
* refactor QEMU's cp15 register handling
At the moment QEMU handles cp15 accesses by calling out to a single helper function which is an enormous set of nested switch statements to handle the different coprocessor registers. Access permissions are checked separately at translate time. This design makes specifying board-dependent or cpu-dependent registers somewhat painful; it's also easy for the access permission checks to be out of sync. There is no support for banked cp15 registers either (needed for trustzone and virtualisation). We need a better design which lets a board or core register handler routines for cp15 registers. This will make the code cleaner and more maintainable as a base for new features.
This isn't strictly a requirement for KVM, but we're going to want KVM to be able to hand off cp15 accesses to QEMU, and I don't think that's going to be maintainable or reliable without this refactoring.
(https://blueprints.launchpad.net/qemu-linaro/+spec/cp15-rework)
* A15 system model
Basically a QEMU model of a Versatile-Express with a Cortex-A15 minus the virtualization and LPAE extensions. This needs the A15 private peripherals (just the GIC in the right place in the memory map, really; generic timer not required) and the new memory map version of the vexpress board model, plus some new cp15 registers. (Bill Carson has already done some patches in this area but they need a little rework and may have minor missing pieces.)
https://blueprints.launchpad.net/qemu-linaro/+spec/initial-a15-system-model
* miscellaneous integration work
We're aiming for a reasonable working prototype of A15 guest on an A15 Fast Model host here; we need to fix at least some of the bugs which currently mean upstream QEMU doesn't work on ARM hosts, sort out which kernel and qemu trees we are developing from, and get things running in our validation lab's continuous integration setup.
https://blueprints.launchpad.net/qemu-linaro/+spec/qemu-kvm-getting-started
Also on the radar is a fourth piece of work:
* QEMU virtio-mmio support
This is adding support for the 'mmio' virtio transport, which will allow virtio support in a versatile-express model. We're going to need this at some point but the current thought is that we want to do the above listed more important bits of work first... (The exception would probably be if it turned out that this was sufficiently useful for making early KVM development easier)
https://blueprints.launchpad.net/qemu-linaro/+spec/add-amba-virtio-support
So, questions: (1) did we forget something important? (2) is anybody else already planning to do any of this (or would like to start)? if so we should coordinate... (3) is there anything that the kernel folk need/want earlier rather than later?
thanks -- PMM
On 22.11.2011, at 15:44, Peter Maydell peter.maydell@linaro.org wrote:
This email is just a quick summary of what we (Linaro) are planning in the way of QEMU work to support KVM on ARM Cortex-A15. The idea is to let people know what's coming up, find out if we've forgotten anything, and avoid people duplicating work unnecessarily. Most of this is based on a useful session at the recent 'ARM server mini-summit' in Orlando (UDS/Linaro Connect) at the beginning of this month.
The work we're currently proposing to do falls into three parts:
- refactor QEMU's cp15 register handling
At the moment QEMU handles cp15 accesses by calling out to a single helper function which is an enormous set of nested switch statements to handle the different coprocessor registers. Access permissions are checked separately at translate time. This design makes specifying board-dependent or cpu-dependent registers somewhat painful; it's also easy for the access permission checks to be out of sync. There is no support for banked cp15 registers either (needed for trustzone and virtualisation). We need a better design which lets a board or core register handler routines for cp15 registers. This will make the code cleaner and more maintainable as a base for new features.
This isn't strictly a requirement for KVM, but we're going to want KVM to be able to hand off cp15 accesses to QEMU, and I don't think that's going to be maintainable or reliable without this refactoring.
(https://blueprints.launchpad.net/qemu-linaro/+spec/cp15-rework)
- A15 system model
Basically a QEMU model of a Versatile-Express with a Cortex-A15 minus the virtualization and LPAE extensions. This needs the A15 private peripherals (just the GIC in the right place in the memory map, really; generic timer not required) and the new memory map version of the vexpress board model, plus some new cp15 registers. (Bill Carson has already done some patches in this area but they need a little rework and may have minor missing pieces.)
https://blueprints.launchpad.net/qemu-linaro/+spec/initial-a15-system-model
- miscellaneous integration work
We're aiming for a reasonable working prototype of A15 guest on an A15 Fast Model host here; we need to fix at least some of the bugs which currently mean upstream QEMU doesn't work on ARM hosts,
I thought there was upstream tcg support for arm now? What specifically doesn't work?
Alex
sort out which kernel and qemu trees we are developing from, and get things running in our validation lab's continuous integration setup.
https://blueprints.launchpad.net/qemu-linaro/+spec/qemu-kvm-getting-started
Also on the radar is a fourth piece of work:
- QEMU virtio-mmio support
This is adding support for the 'mmio' virtio transport, which will allow virtio support in a versatile-express model. We're going to need this at some point but the current thought is that we want to do the above listed more important bits of work first... (The exception would probably be if it turned out that this was sufficiently useful for making early KVM development easier)
https://blueprints.launchpad.net/qemu-linaro/+spec/add-amba-virtio-support
So, questions: (1) did we forget something important? (2) is anybody else already planning to do any of this (or would like to start)? if so we should coordinate... (3) is there anything that the kernel folk need/want earlier rather than later?
thanks -- PMM _______________________________________________ Android-virt mailing list Android-virt@lists.cs.columbia.edu https://lists.cs.columbia.edu/cucslists/listinfo/android-virt
On 22 November 2011 15:05, Alexander Graf agraf@suse.de wrote:
On 22.11.2011, at 15:44, Peter Maydell peter.maydell@linaro.org wrote:
We're aiming for a reasonable working prototype of A15 guest on an A15 Fast Model host here; we need to fix at least some of the bugs which currently mean upstream QEMU doesn't work on ARM hosts,
I thought there was upstream tcg support for arm now? What specifically doesn't work?
* coroutine implementation doesn't cope if libc provides a makecontext() that returns ENOSYS (so we abort on startup) * the MAP_FIXED static codegen buffer is put at an address which overlaps the glibc heap so we crash about 50% of the time * if you try to start a standard i386 debian image which works fine on x86 host, the guest reboots just before displaying the blue-background grub image -- clearly something wrong here, not tracked down what yet * if the C code was compiled -mthumb, our choice of TCG_AREG0 clashes with the frame pointer (trivial fix but could do with a quick benchmark run to see whether making AREG0 r6 or r10 (ie thumb lowreg or high) has any effect on performance)
Those are just the ones I know about... The coroutine one is the only one that KVM specifically will require a fix for, but being able to cross-compare "run with KVM" vs "run with TCG" will be handy for debugging I suspect.
ARM host support is documented as "known not to work" in the 1.0 release notes (see earlier mail to qemu-devel).
-- PMM
Thanks for sharing this information. When is QEMU support for ARM Cortex A15 (with LPAE) expected?
Jubi
On Tue, Nov 22, 2011 at 8:14 PM, Peter Maydell peter.maydell@linaro.orgwrote:
This email is just a quick summary of what we (Linaro) are planning in the way of QEMU work to support KVM on ARM Cortex-A15. The idea is to let people know what's coming up, find out if we've forgotten anything, and avoid people duplicating work unnecessarily. Most of this is based on a useful session at the recent 'ARM server mini-summit' in Orlando (UDS/Linaro Connect) at the beginning of this month.
The work we're currently proposing to do falls into three parts:
- refactor QEMU's cp15 register handling
At the moment QEMU handles cp15 accesses by calling out to a single helper function which is an enormous set of nested switch statements to handle the different coprocessor registers. Access permissions are checked separately at translate time. This design makes specifying board-dependent or cpu-dependent registers somewhat painful; it's also easy for the access permission checks to be out of sync. There is no support for banked cp15 registers either (needed for trustzone and virtualisation). We need a better design which lets a board or core register handler routines for cp15 registers. This will make the code cleaner and more maintainable as a base for new features.
This isn't strictly a requirement for KVM, but we're going to want KVM to be able to hand off cp15 accesses to QEMU, and I don't think that's going to be maintainable or reliable without this refactoring.
(https://blueprints.launchpad.net/qemu-linaro/+spec/cp15-rework)
- A15 system model
Basically a QEMU model of a Versatile-Express with a Cortex-A15 minus the virtualization and LPAE extensions. This needs the A15 private peripherals (just the GIC in the right place in the memory map, really; generic timer not required) and the new memory map version of the vexpress board model, plus some new cp15 registers. (Bill Carson has already done some patches in this area but they need a little rework and may have minor missing pieces.)
https://blueprints.launchpad.net/qemu-linaro/+spec/initial-a15-system-model
- miscellaneous integration work
We're aiming for a reasonable working prototype of A15 guest on an A15 Fast Model host here; we need to fix at least some of the bugs which currently mean upstream QEMU doesn't work on ARM hosts, sort out which kernel and qemu trees we are developing from, and get things running in our validation lab's continuous integration setup.
https://blueprints.launchpad.net/qemu-linaro/+spec/qemu-kvm-getting-started
Also on the radar is a fourth piece of work:
- QEMU virtio-mmio support
This is adding support for the 'mmio' virtio transport, which will allow virtio support in a versatile-express model. We're going to need this at some point but the current thought is that we want to do the above listed more important bits of work first... (The exception would probably be if it turned out that this was sufficiently useful for making early KVM development easier)
https://blueprints.launchpad.net/qemu-linaro/+spec/add-amba-virtio-support
So, questions: (1) did we forget something important? (2) is anybody else already planning to do any of this (or would like to start)? if so we should coordinate... (3) is there anything that the kernel folk need/want earlier rather than later?
thanks -- PMM
linaro-toolchain mailing list linaro-toolchain@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-toolchain
On 11/22/2011 04:19 PM, Peter Maydell wrote:
- coroutine implementation doesn't cope if libc provides a makecontext() that returns ENOSYS (so we abort on startup)
I have a patch that replaces makecontext() with sigaltstack(), i.e. going from nice code that does not work to hideous code that works.
I had not submitted it yet because it does not help for OpenBSD so we would need anyway coroutine-gthread.c, and because I would like to test it more thoroughly on non-Linux hosts. If anyone can help with the testing, it is at git://github.com/bonzini/qemu.git branch coroutine-fix.
- the MAP_FIXED static codegen buffer is put at an address which overlaps the glibc heap so we crash about 50% of the time
For 1.1 we can just use USE_STATIC_CODE_GEN_BUFFER. It should not have major problems on Linux, since it is already tested for linux-user. That leaves out Windows of course.
Paolo
On 23 November 2011 08:45, Paolo Bonzini pbonzini@redhat.com wrote:
On 11/22/2011 04:19 PM, Peter Maydell wrote:
* the MAP_FIXED static codegen buffer is put at an address which overlaps the glibc heap so we crash about 50% of the time
For 1.1 we can just use USE_STATIC_CODE_GEN_BUFFER. It should not have major problems on Linux, since it is already tested for linux-user. That leaves out Windows of course.
This raises the question of why we don't use the static buffer everywhere. Presumably there was a disadvantage that caused us to move to the dynamic mmap for system mode...
(I don't actually think it should be very hard to make the ARM code cope with the idea that jumps into host code might be long range.)
-- PMM
On 23 November 2011 07:50, Jubi Taneja jubitaneja@gmail.com wrote:
Thanks for sharing this information. When is QEMU support for ARM Cortex A15 (with LPAE) expected?
Depends what you mean. LPAE is on the list of bits we currently hope we won't have to implement for KVM's benefit, but which we might have to. (Basically if we start with a guest kernel not compiled for LPAE then there's no need.) So adding LPAE on top of the "A15-without-virtualization" QEMU system model isn't in this planned work. If we need it it'll probably be done some time in the first quarter of next year.
A full QEMU system emulation including the virtualization extensions (ie what you would need to run a hypervisor inside QEMU) is not currently planned.
-- PMM
This isn't strictly a requirement for KVM, but we're going to want KVM to be able to hand off cp15 accesses to QEMU, and I don't think that's going to be maintainable or reliable without this refactoring.
Why do we need KVM to hand off cp15 accesses to QEMU? As of now almost all of this is supported in-kernel. Do the CP15 behavior vary that much according to the board config? I would think that other co-processor accesses would actually be more important if they're used as interfaces for DSPs etc. But in that case, yes, we need a way to handle them at least.
[snip]
Also on the radar is a fourth piece of work:
* QEMU virtio-mmio support
This is adding support for the 'mmio' virtio transport, which will allow virtio support in a versatile-express model. We're going to need this at some point but the current thought is that we want to do the above listed more important bits of work first... (The exception would probably be if it turned out that this was sufficiently useful for making early KVM development easier)
I don't really see why it would be. Is this not merely a question of performance?
So, questions: (1) did we forget something important? (2) is anybody else already planning to do any of this (or would like to start)? if so we should coordinate... (3) is there anything that the kernel folk need/want earlier rather than later?
Thanks for sending this out.
I would like some clarity (if possible) of which branch the KVM support for ARM should be based on. Is it the Linaro version of QEMU and basically just keep rebasing the changes off there until someone accepts them or?
What would be helpful from the point of view of KVM is to have basic ARM host support and the A15 system model upstream.
Currently there are a number of changes to the configure script to make things work and we link against a prebuilt zlib library that we keep distributing to people who wish to build QEMU for KVM/ARM. I am not an expert on QEMU and would definitely like some help on doing this the right way. What's the recommended procedure? Should I send out the hack-ish patch we have now, and if so, what should be the base of such a patch.
Best, Christoffer
On 24 November 2011 20:11, Christoffer Dall cdall@cs.columbia.edu wrote:
This isn't strictly a requirement for KVM, but we're going to want KVM to be able to hand off cp15 accesses to QEMU, and I don't think that's going to be maintainable or reliable without this refactoring.
Why do we need KVM to hand off cp15 accesses to QEMU? As of now almost all of this is supported in-kernel. Do the CP15 behavior vary that much according to the board config? I would think that other co-processor accesses would actually be more important if they're used as interfaces for DSPs etc. But in that case, yes, we need a way to handle them at least.
I think it was mostly the generic timer that inclined me that way: it's a device that just happens to be hung off the cp15 interface rather than being memory mapped.
My assumption was that initial default would be "hand everything off to qemu" with the kernel then providing implementations of things like timers for performance reasons later.
Also, there are lots of cp15 registers, and they vary between implementations, so (a) it would be useful to share the emulation implementation between qemu and kvm somehow and (b) are you really going to implement cp15 emulation for half a dozen CPU implementations in the kernel?
If it doesn't make sense to hand off cp15 accesses that's fine, though -- I want to do this refactoring for the A15 system mode implementation in qemu anyway, because I really don't think we should try to shoehorn in yet another cpu implementation into the current set of switch statements...
It looks (from a brief glance at the code) like ppc kvm does handoff-to-qemu with the DCRs, incidentally.
Also on the radar is a fourth piece of work:
* QEMU virtio-mmio support
This is adding support for the 'mmio' virtio transport, which will allow virtio support in a versatile-express model. We're going to need this at some point but the current thought is that we want to do the above listed more important bits of work first... (The exception would probably be if it turned out that this was sufficiently useful for making early KVM development easier)
I don't really see why it would be. Is this not merely a question of performance?
Yes; Marc Z was complaining at me earlier this week about SD card emulated performance being particularly dire, though :-) So this is kind of down here as a "if anybody wants this sooner than some-time-late-next-Q1 now is a good time to scream" marker...
I would like some clarity (if possible) of which branch the KVM support for ARM should be based on. Is it the Linaro version of QEMU and basically just keep rebasing the changes off there until someone accepts them or?
What would be helpful from the point of view of KVM is to have basic ARM host support and the A15 system model upstream.
My plan here was that the bits like A15 system model which are clearly upstreamable I would push directly upstream (and just keep in qemu-linaro for the typically week-or-three things take to go through upstream patch review). For the KVM support patch itself, my thought was simply to carry that in qemu-linaro as an "unsupported work in progress" patch until we think it's ready to commit upstream. [I'm suggesting qemu-linaro here mostly because it's convenient to me: it's a tree I already maintain and track upstream trunk with. If that would be awkward for other people we can do something else.]
Pretty high up my todo list was rebasing your kvm patch on to master / qemu-linaro (the two are more or less the same for this purpose).
Currently there are a number of changes to the configure script to make things work and we link against a prebuilt zlib library that we keep distributing to people who wish to build QEMU for KVM/ARM.
I have some instructions for Ubuntu oneiric hosts that work by using the stock oneiric ARM zlib (installed via dpkg-cross): see the section at the bottom of https://wiki.linaro.org/PeterMaydell/A15OnFastModels (that's "how to cross build upstream qemu master", not your qemu with the kvm patch.)
You'll find that when we update to QEMU 1.0 you'll also need a cross version of the glib/gthread libraries, which may make you more reluctant to stick to the "hand out prebuilt cross library" approach.
-- PMM
On Thu, Nov 24, 2011 at 4:27 PM, Peter Maydell peter.maydell@linaro.org wrote:
On 24 November 2011 20:11, Christoffer Dall cdall@cs.columbia.edu wrote:
This isn't strictly a requirement for KVM, but we're going to want KVM to be able to hand off cp15 accesses to QEMU, and I don't think that's going to be maintainable or reliable without this refactoring.
Why do we need KVM to hand off cp15 accesses to QEMU? As of now almost all of this is supported in-kernel. Do the CP15 behavior vary that much according to the board config? I would think that other co-processor accesses would actually be more important if they're used as interfaces for DSPs etc. But in that case, yes, we need a way to handle them at least.
I think it was mostly the generic timer that inclined me that way: it's a device that just happens to be hung off the cp15 interface rather than being memory mapped.
My assumption was that initial default would be "hand everything off to qemu" with the kernel then providing implementations of things like timers for performance reasons later.
hmm yeah, that could have been an approach. What happened was that before a15, I had to take care of stuff like cache flushes inside the kernel, so it wasn't really an option anyway.
Also, there are lots of cp15 registers, and they vary between implementations, so (a) it would be useful to share the emulation implementation between qemu and kvm somehow and (b) are you really going to implement cp15 emulation for half a dozen CPU implementations in the kernel?
regarding (a), yes most certainly and (b), not if I can avoid it :)
If it doesn't make sense to hand off cp15 accesses that's fine, though -- I want to do this refactoring for the A15 system mode implementation in qemu anyway, because I really don't think we should try to shoehorn in yet another cpu implementation into the current set of switch statements...
It looks (from a brief glance at the code) like ppc kvm does handoff-to-qemu with the DCRs, incidentally.
for the record, I wasn't questioning that the refactoring was a good idea, I'm sure it is.
Also on the radar is a fourth piece of work:
* QEMU virtio-mmio support
This is adding support for the 'mmio' virtio transport, which will allow virtio support in a versatile-express model. We're going to need this at some point but the current thought is that we want to do the above listed more important bits of work first... (The exception would probably be if it turned out that this was sufficiently useful for making early KVM development easier)
I don't really see why it would be. Is this not merely a question of performance?
Yes; Marc Z was complaining at me earlier this week about SD card emulated performance being particularly dire, though :-) So this is kind of down here as a "if anybody wants this sooner than some-time-late-next-Q1 now is a good time to scream" marker...
ok. I didn't do much performance tuning yet overall, but I do see virtio and generic timers and being pretty high on the list once we get there, so yes, let's start thinking about it.
I would like some clarity (if possible) of which branch the KVM support for ARM should be based on. Is it the Linaro version of QEMU and basically just keep rebasing the changes off there until someone accepts them or?
What would be helpful from the point of view of KVM is to have basic ARM host support and the A15 system model upstream.
My plan here was that the bits like A15 system model which are clearly upstreamable I would push directly upstream (and just keep in qemu-linaro for the typically week-or-three things take to go through upstream patch review). For the KVM support patch itself, my thought was simply to carry that in qemu-linaro as an "unsupported work in progress" patch until we think it's ready to commit upstream. [I'm suggesting qemu-linaro here mostly because it's convenient to me: it's a tree I already maintain and track upstream trunk with. If that would be awkward for other people we can do something else.]
Pretty high up my todo list was rebasing your kvm patch on to master / qemu-linaro (the two are more or less the same for this purpose).
if you could take charge on that it would be awesome from my point of view. I will send you a (slightly) cleaned up patch that you can use or throw away.
Currently there are a number of changes to the configure script to make things work and we link against a prebuilt zlib library that we keep distributing to people who wish to build QEMU for KVM/ARM.
I have some instructions for Ubuntu oneiric hosts that work by using the stock oneiric ARM zlib (installed via dpkg-cross): see the section at the bottom of https://wiki.linaro.org/PeterMaydell/A15OnFastModels (that's "how to cross build upstream qemu master", not your qemu with the kvm patch.)
You'll find that when we update to QEMU 1.0 you'll also need a cross version of the glib/gthread libraries, which may make you more reluctant to stick to the "hand out prebuilt cross library" approach.
I'm already reluctant, so I'm happy to hear there will be some "official" instructions available. I actually think it's important for the adoption of KVM that things are somewhat possible to build without too much jumping through hoops. A remaining item is to test this setup with the KVM stuff, but it should be a minor thing as long as the kernel headers for KVM/ARM can be integrated with the build process somehow. (Or is there an alternative?).
-Christoffer
On 24 November 2011 22:02, Christoffer Dall cdall@cs.columbia.edu wrote:
On Thu, Nov 24, 2011 at 4:27 PM, Peter Maydell peter.maydell@linaro.org wrote:
Pretty high up my todo list was rebasing your kvm patch on to master / qemu-linaro (the two are more or less the same for this purpose).
if you could take charge on that it would be awesome from my point of view. I will send you a (slightly) cleaned up patch that you can use or throw away.
That would be good, thanks. I'll try to get that rebasing started tomorrow and done early next week.
I'm already reluctant, so I'm happy to hear there will be some "official" instructions available. I actually think it's important for the adoption of KVM that things are somewhat possible to build without too much jumping through hoops.
Well, once we've got real hardware it'll be more straightforward because building QEMU on the hardware won't be quite so slow... Most of this is just because crosscompiling is and remains painful. (Alas, you can't compile QEMU in a QEMU arm-linux-user chroot, because gcc segfaults. I blame our mmap emulation layer.)
A remaining item is to test this setup with the KVM stuff, but it should be a minor thing as long as the kernel headers for KVM/ARM can be integrated with the build process somehow. (Or is there an alternative?).
Ah, kernel headers, good point. QEMU now carries the KVM kernel headers in its own git tree (this has changed since the QEMU version you based your original kvm patches on, I think). So we'll have to carry the ARM header changes in qemu-linaro too (I think). I guess that should be OK, presumably we won't be revising the kernel-userspace ABI at a rate of knots. (when the headers get upstream in the kernel upstream qemu can update to get them from there and we can drop the temporary copies in qemu-linaro). [that's kind of off-the-top-of-my-head, yell if it sounds lunatic.]
-- PMM
On 24.11.2011, at 23:02, Christoffer Dall wrote:
On Thu, Nov 24, 2011 at 4:27 PM, Peter Maydell peter.maydell@linaro.org wrote:
On 24 November 2011 20:11, Christoffer Dall cdall@cs.columbia.edu wrote:
[...]
I would like some clarity (if possible) of which branch the KVM support for ARM should be based on. Is it the Linaro version of QEMU and basically just keep rebasing the changes off there until someone accepts them or?
What would be helpful from the point of view of KVM is to have basic ARM host support and the A15 system model upstream.
My plan here was that the bits like A15 system model which are clearly upstreamable I would push directly upstream (and just keep in qemu-linaro for the typically week-or-three things take to go through upstream patch review). For the KVM support patch itself, my thought was simply to carry that in qemu-linaro as an "unsupported work in progress" patch until we think it's ready to commit upstream. [I'm suggesting qemu-linaro here mostly because it's convenient to me: it's a tree I already maintain and track upstream trunk with. If that would be awkward for other people we can do something else.]
Pretty high up my todo list was rebasing your kvm patch on to master / qemu-linaro (the two are more or less the same for this purpose).
if you could take charge on that it would be awesome from my point of view. I will send you a (slightly) cleaned up patch that you can use or throw away.
Please make sure that once there is upstream code in the kernel to actually implement A15 KVM, there should also be the patches ready for QEMU upstream inclusion, as that's the point where the ABI is stable :).
Currently there are a number of changes to the configure script to make things work and we link against a prebuilt zlib library that we keep distributing to people who wish to build QEMU for KVM/ARM.
I have some instructions for Ubuntu oneiric hosts that work by using the stock oneiric ARM zlib (installed via dpkg-cross): see the section at the bottom of https://wiki.linaro.org/PeterMaydell/A15OnFastModels (that's "how to cross build upstream qemu master", not your qemu with the kvm patch.)
You'll find that when we update to QEMU 1.0 you'll also need a cross version of the glib/gthread libraries, which may make you more reluctant to stick to the "hand out prebuilt cross library" approach.
I'm already reluctant, so I'm happy to hear there will be some "official" instructions available. I actually think it's important for the adoption of KVM that things are somewhat possible to build without too much jumping through hoops. A remaining item is to test this setup with the KVM stuff, but it should be a minor thing as long as the kernel headers for KVM/ARM can be integrated with the build process somehow. (Or is there an alternative?).
The kernel headers now all reside inside the QEMU tree, so you just sync them up to Linux's headers using a script and then have them around. That way building against KVM should be trivial :).
Alex
On 25.11.2011, at 00:06, Peter Maydell wrote:
On 24 November 2011 22:02, Christoffer Dall cdall@cs.columbia.edu wrote:
On Thu, Nov 24, 2011 at 4:27 PM, Peter Maydell peter.maydell@linaro.org wrote:
Pretty high up my todo list was rebasing your kvm patch on to master / qemu-linaro (the two are more or less the same for this purpose).
if you could take charge on that it would be awesome from my point of view. I will send you a (slightly) cleaned up patch that you can use or throw away.
That would be good, thanks. I'll try to get that rebasing started tomorrow and done early next week.
I'm already reluctant, so I'm happy to hear there will be some "official" instructions available. I actually think it's important for the adoption of KVM that things are somewhat possible to build without too much jumping through hoops.
Well, once we've got real hardware it'll be more straightforward because building QEMU on the hardware won't be quite so slow... Most of this is just because crosscompiling is and remains painful. (Alas, you can't compile QEMU in a QEMU arm-linux-user chroot, because gcc segfaults. I blame our mmap emulation layer.)
Just throw in MAP_32BIT in all mmaps and it should work like a charm :).
A remaining item is to test this setup with the KVM stuff, but it should be a minor thing as long as the kernel headers for KVM/ARM can be integrated with the build process somehow. (Or is there an alternative?).
Ah, kernel headers, good point. QEMU now carries the KVM kernel headers in its own git tree (this has changed since the QEMU version you based your original kvm patches on, I think). So we'll have to carry the ARM header changes in qemu-linaro too (I think). I guess that should be OK, presumably we won't be revising the kernel-userspace ABI at a rate of knots. (when the headers get upstream in the kernel upstream qemu can update to get them from there and we can drop the temporary copies in qemu-linaro). [that's kind of off-the-top-of-my-head, yell if it sounds lunatic.]
I fully agree. Treat the linaro tree as the temporary straighten-out-the-ABI-tree and then move to upstream for further development :)
Alex
On 24 November 2011 23:10, Alexander Graf agraf@suse.de wrote:
On 25.11.2011, at 00:06, Peter Maydell wrote:
Well, once we've got real hardware it'll be more straightforward because building QEMU on the hardware won't be quite so slow... Most of this is just because crosscompiling is and remains painful. (Alas, you can't compile QEMU in a QEMU arm-linux-user chroot, because gcc segfaults. I blame our mmap emulation layer.)
Just throw in MAP_32BIT in all mmaps and it should work like a charm :)
Want to submit a patch that does that if the host is x86-64? I think I'd rather pragmatically do that and fix the issues in the actually common case rather than hold out for a larger scale reimplementation of the mmap layer which we aren't in practice going to do...
-- PMM
On 25.11.2011, at 00:11, Peter Maydell wrote:
On 24 November 2011 23:10, Alexander Graf agraf@suse.de wrote:
On 25.11.2011, at 00:06, Peter Maydell wrote:
Well, once we've got real hardware it'll be more straightforward because building QEMU on the hardware won't be quite so slow... Most of this is just because crosscompiling is and remains painful. (Alas, you can't compile QEMU in a QEMU arm-linux-user chroot, because gcc segfaults. I blame our mmap emulation layer.)
Just throw in MAP_32BIT in all mmaps and it should work like a charm :)
Want to submit a patch that does that if the host is x86-64? I think I'd rather pragmatically do that and fix the issues in the actually common case rather than hold out for a larger scale reimplementation of the mmap layer which we aren't in practice going to do...
Happily, yes :).
Alex
[snip]
A remaining item is to test this setup with the KVM stuff, but it should be a minor thing as long as the kernel headers for KVM/ARM can be integrated with the build process somehow. (Or is there an alternative?).
Ah, kernel headers, good point. QEMU now carries the KVM kernel headers in its own git tree (this has changed since the QEMU version you based your original kvm patches on, I think). So we'll have to carry the ARM header changes in qemu-linaro too (I think). I guess that should be OK, presumably we won't be revising the kernel-userspace ABI at a rate of knots. (when the headers get upstream in the kernel upstream qemu can update to get them from there and we can drop the temporary copies in qemu-linaro). [that's kind of off-the-top-of-my-head, yell if it sounds lunatic.]
sounds good to me.
On 11/24/2011 03:27 PM, Peter Maydell wrote:
If it doesn't make sense to hand off cp15 accesses that's fine, though -- I want to do this refactoring for the A15 system mode implementation in qemu anyway, because I really don't think we should try to shoehorn in yet another cpu implementation into the current set of switch statements...
It looks (from a brief glance at the code) like ppc kvm does handoff-to-qemu with the DCRs, incidentally.
Not for SPRs, though, which may be more analogous to ARM's cp15 (and are where timers are implemented).
DCRs are more like the the PPC 4xx family's version of x86 I/O ports.
-Scott
On 24 November 2011 23:06, Peter Maydell peter.maydell@linaro.org wrote:
On 24 November 2011 22:02, Christoffer Dall cdall@cs.columbia.edu wrote:
On Thu, Nov 24, 2011 at 4:27 PM, Peter Maydell peter.maydell@linaro.org wrote:
Pretty high up my todo list was rebasing your kvm patch on to master / qemu-linaro (the two are more or less the same for this purpose).
if you could take charge on that it would be awesome from my point of view. I will send you a (slightly) cleaned up patch that you can use or throw away.
That would be good, thanks. I'll try to get that rebasing started tomorrow and done early next week.
I've now done this rebasing, fixed a few other minor issues and put the resulting patches into qemu-linaro as "experimental work in progress":
http://git.linaro.org/gitweb?p=qemu/qemu-linaro.git%3Ba=summary
Note that I've defaulted KVM to off for ARM hosts (for the benefit of other users of qemu-linaro who don't care about KVM yet), so run configure with "--enable-kvm".
A note about the branches here: the 'rebasing' branch is the patch stack, and (as the name suggests) it rebases on qemu upstream fairly frequently. 'master' is a never-rebases branch which always has identical contents to the head of 'rebasing' (generated by stg publish). Pick whichever branch suits your use case best.
Instructions on how to cross-compile it are here: https://wiki.linaro.org/PeterMaydell/A15OnFastModels (if you're moving from Christoffer's qemu tree these are worth checking, because QEMU now requires glib/gthread so a cross compile setup that could compile Christoffer's tree won't necessarily build this one.)
-- PMM
On Tue, Nov 29, 2011 at 1:16 PM, Peter Maydell peter.maydell@linaro.org wrote:
On 24 November 2011 23:06, Peter Maydell peter.maydell@linaro.org wrote:
On 24 November 2011 22:02, Christoffer Dall cdall@cs.columbia.edu wrote:
On Thu, Nov 24, 2011 at 4:27 PM, Peter Maydell peter.maydell@linaro.org wrote:
Pretty high up my todo list was rebasing your kvm patch on to master / qemu-linaro (the two are more or less the same for this purpose).
if you could take charge on that it would be awesome from my point of view. I will send you a (slightly) cleaned up patch that you can use or throw away.
That would be good, thanks. I'll try to get that rebasing started tomorrow and done early next week.
I've now done this rebasing, fixed a few other minor issues and put the resulting patches into qemu-linaro as "experimental work in progress":
http://git.linaro.org/gitweb?p=qemu/qemu-linaro.git%3Ba=summary
Note that I've defaulted KVM to off for ARM hosts (for the benefit of other users of qemu-linaro who don't care about KVM yet), so run configure with "--enable-kvm".
A note about the branches here: the 'rebasing' branch is the patch stack, and (as the name suggests) it rebases on qemu upstream fairly frequently. 'master' is a never-rebases branch which always has identical contents to the head of 'rebasing' (generated by stg publish). Pick whichever branch suits your use case best.
Instructions on how to cross-compile it are here: https://wiki.linaro.org/PeterMaydell/A15OnFastModels (if you're moving from Christoffer's qemu tree these are worth checking, because QEMU now requires glib/gthread so a cross compile setup that could compile Christoffer's tree won't necessarily build this one.)
this is great, thanks!
looking forward to trying it out later this week.
linaro-toolchain@lists.linaro.org
-
Alexander Graf -
Christoffer Dall -
Jubi Taneja -
Paolo Bonzini -
Peter Maydell -
Scott Wood