Update (sorry to steal the thread, google mail is not nice in this regard).
Prompted by Paul I've decided to try a bigLITTLE config. My test did not end up very good: 1) I've tried the RealView Express A15x1-A7x1 model 2) It has different options that can be passed to model shell, needles to say they are not compatible. I tweaked my semihosting line to look like this:
-C coretile.cluster0.cpu0.semihosting-cmd_line="--kernel $KERNEL -- console=ttyAMA0 mem=512M ip=dhcp root=/dev/nfs nfsroot=$NFSROOT,tcp rw earlyprintk $@"
3) I did not get the kernel to boot at all, I suspect that our boot wrapper is not compatible or I mis-configured the kernel or something in between. I did not even get a X11 terminal window as I usually always get. 4) I feel lame for not trying earlier
Ideas welcome ZK
On Wed, Feb 15, 2012 at 12:31 AM, Michael Hudson-Doyle michael.hudson@canonical.com wrote:
On Tue, 14 Feb 2012 20:24:51 +0100, Zygmunt Krynicki zygmunt.krynicki@linaro.org wrote:
On Tue, Feb 14, 2012 at 3:26 AM, Michael Hudson-Doyle michael.hudson@canonical.com wrote:
On Mon, 13 Feb 2012 22:27:25 +0100, Zygmunt Krynicki zygmunt.krynicki@linaro.org wrote:
Hi.
Fast model support is getting better. It seems that with the excellent patches by Peter Maydell we can now boot some kernels (I've only tried one tree, additional trees welcome :-). I'm currently building a from-scratch environment to ensure everything is accounted for and I understand how pieces interact.
Having said that I'd like to summarize how LAVA handles fast models:
Technically the solution is not unlike QEMU which you are all familiar with. The key differences are:
- Only NFS boot makes sense. There are no other sensible method that
I know of. We may also use a SD card (virtual obviously) but it is constrained to two gigabytes of data.
As mentioned in the other thread, it would be good to at least let ARM know that removing this limit would help us (if we can figure out how to do this).
We may figure out how to do this by reading the LISA source code that came with the model. That's a big task though (maybe grepping for mmc0 is a low hanging fruit, I did not check)
That's not what I was suggesting! We should try to persuade ARM to do that. It may be that they can't do it in a reasonable timeframe, or maybe it's simply not a need that has been explained to them yet and is something they can do in a week.
- The way we actually boot is complicated. There is no uboot, fast
model interpreter actually starts an .axf file that can do anything (some examples include running tests and benchmarks without actually setting the kernel or anything like that). There is no way to easily load the kernel and pass a command line. To work around that we're using a special axf file that uses fast model semihosting features to load the kernel/initrd from a host filesystem as well as to setup the command line that will be passed to the booting kernel. This allows us to freely configure NFS services and point our virtual kernel at appropriate IP addresses and pathnames.
So I guess I'd like to understand how this works in a bit more detail. Can you brain dump on the topic for a few minutes? :) What is "fast model semihosting"?
It's a way to have "syscalls" that connect the "bare hardware" (be it physical or emulated) to an external debugger or other monitor. You can find a short introduction in this blog [1]. For us it means we get to write bare-metal assembly that does the equivalent of open(), read(), write() and close(). The files are being opened are on the machine that runs the fast model. You can also print debugging statements straight to the console this way (we probably could write semihosting console driver if there is no such code yet) to get all of the output to the same tty that runs the model (model_shell). A more detailed explanation of this topic can be found in [2]
Fast model semihosting simply refers to using semihosting facilities in a fast model interpreter.
Thanks for that. Sounds a tiny little bit like it's using a JTAG-for-fast-models type facility?
- After the machine starts up we immediately open a TPC/IP connection
to a local TCP socket. We know which port is being used so we can easily allocate them up front. This port is now the traditional LAVA serial console.
I guess there is a risk here that we will miss early boot messages? This might not matter much.
There are other options but currently this seems to work quite okay.
Fair enough.
Once we've found someone at ARM we can officially complain at about fast models, an option to have serial comms happen on the process stdin/stdout would be nice.
I think the reason they don't happen in the console is that by default we get four telnet ports to connect to (definitely more than one) so the logical question they'll ask is "which port should we redirect". Maybe there is an option buried somewhere to make that happen but so far I have not found it.
Again, I'm not saying that this is something we should do...
The rest of this looks like QEMU:
- you can access the filesystem easily (to gather results)
- we can use QEMU to chroot into the NFS root to install additional
software (emulation via a fast model is extremely slow)
In my testing, the pip install bzr+lp:lava-test step did not really work under QEMU. Maybe it does now, or maybe we can install a tarball or something.
I installed lava-test using release tarball. That has worked pretty well.
OK. That makes sense.
In general I think that:
- We need to reconsider how to do testing on very slow machines
You mean a "drive from the outside" approach like lava-android-test uses may make sense?
- What can be invoked on the host (part of installation, unless that
wants to build stuff, result parsing and tracking)
Yeah, this sort of thing is a grey area currently. More below.
- What has to be invoked on the target (test code, system probes)
It's important to make the intent very clear. If we define that cmd_install installs something while in "master image" on the "target" then we should not break that.
Well. We can't avoid breaking that if there *is no master image*.
Currently the dispatcher has the concept of a "reliable session", which is meant to be a target-like environment where things like compilation are possible. For master image based deployments, this is "booted into the master image, chrooted into a mounted testrootfs". For qemu, it is currently "boot the test image and hope that works", but it could be "chrooted into the testrootfs mounted on the host with qemu-arm-static in the right place", but that was less reliable than the other approach when I was testing this.
I think that it would be sensible to add "host_chroot" mode that applies nicely to qemu and fast models. Very slow things that don't care about the architecture could be invoked in that mode without sacrificing performance.
This code exists already. See _chroot_into_rootfs_session in lava_dispatcher.client.qemu.LavaQEMUClient and surrounds. The problem is that qemu is some distance from perfect...
Maybe we can limit the things that lava-test install does to things that work under qemu -- I guess installing via dpkg usually works (unless it's something like mono) and gcc probably works ok? Maybe we can do something like scratchbox where gcc is magically a cross compiler running directly on the host?
Cheers, mwh
On 15 February 2012 00:33, Zygmunt Krynicki zygmunt.krynicki@linaro.org wrote:
Prompted by Paul I've decided to try a bigLITTLE config. My test did not end up very good:
- I've tried the RealView Express A15x1-A7x1 model
NB that I think the plan for big.LITTLE models is to use a prebuilt RTSM, which should look the same as a binary built from the example models, it's just a pre-cooked binary supplied by ARM rather than built by us.
- It has different options that can be passed to model shell, needles
to say they are not compatible. I tweaked my semihosting line to look like this:
-C coretile.cluster0.cpu0.semihosting-cmd_line="--kernel $KERNEL -- console=ttyAMA0 mem=512M ip=dhcp root=/dev/nfs nfsroot=$NFSROOT,tcp rw earlyprintk $@"
Yeah, the parameters are specific to the components within the model, so they vary depending on the structure of the model.
- I did not get the kernel to boot at all, I suspect that our boot
wrapper is not compatible or I mis-configured the kernel or something in between. I did not even get a X11 terminal window as I usually always get.
Yes, we've been clear for some time that the bootwrapper needs changes to work with big.LITTLE (in fact I'm not entirely sure whether the switcher code counts as part of the 'kernel' or not). Dave probably knows more about the status here (I suspect there's some working code somewhere, just maybe not in the git.linaro.org boot-wrapper or whichever kernel you tried).
-- PMM
Thanks for the update Peter, that clears things up.
Dave, if you have anything that works and can be used to evaluate how LAVA support for fast models needs to change to get bigLITTLE working, could you share that with me? Pre-built binaries, alpha git trees, anything that does more than I currently have.
Thanks ZK
On Wed, Feb 15, 2012 at 1:44 AM, Peter Maydell peter.maydell@linaro.org wrote:
On 15 February 2012 00:33, Zygmunt Krynicki zygmunt.krynicki@linaro.org wrote:
Prompted by Paul I've decided to try a bigLITTLE config. My test did not end up very good:
- I've tried the RealView Express A15x1-A7x1 model
NB that I think the plan for big.LITTLE models is to use a prebuilt RTSM, which should look the same as a binary built from the example models, it's just a pre-cooked binary supplied by ARM rather than built by us.
- It has different options that can be passed to model shell, needles
to say they are not compatible. I tweaked my semihosting line to look like this:
-C coretile.cluster0.cpu0.semihosting-cmd_line="--kernel $KERNEL -- console=ttyAMA0 mem=512M ip=dhcp root=/dev/nfs nfsroot=$NFSROOT,tcp rw earlyprintk $@"
Yeah, the parameters are specific to the components within the model, so they vary depending on the structure of the model.
- I did not get the kernel to boot at all, I suspect that our boot
wrapper is not compatible or I mis-configured the kernel or something in between. I did not even get a X11 terminal window as I usually always get.
Yes, we've been clear for some time that the bootwrapper needs changes to work with big.LITTLE (in fact I'm not entirely sure whether the switcher code counts as part of the 'kernel' or not). Dave probably knows more about the status here (I suspect there's some working code somewhere, just maybe not in the git.linaro.org boot-wrapper or whichever kernel you tried).
-- PMM
On Wed, Feb 15, 2012 at 12:44 AM, Peter Maydell peter.maydell@linaro.org wrote:
On 15 February 2012 00:33, Zygmunt Krynicki zygmunt.krynicki@linaro.org wrote:
Prompted by Paul I've decided to try a bigLITTLE config. My test did not end up very good:
- I've tried the RealView Express A15x1-A7x1 model
NB that I think the plan for big.LITTLE models is to use a prebuilt RTSM, which should look the same as a binary built from the example models, it's just a pre-cooked binary supplied by ARM rather than built by us.
I'm not actually sure exactly why we need a separate model for the b.L work -- but that's what we were given. If I understand correctly, the b.L platform can be built from the generic 7.0 fast models example platforms we have (this is what I was doing before the special pre-built model was available), but it's possible that the two models aren't exactly the same. That said, it shouldn't really matter. Launching the standalone model is almost exactly the same in practice as launching via model_shell.
- It has different options that can be passed to model shell, needles
to say they are not compatible. I tweaked my semihosting line to look like this:
-C coretile.cluster0.cpu0.semihosting-cmd_line="--kernel $KERNEL -- console=ttyAMA0 mem=512M ip=dhcp root=/dev/nfs nfsroot=$NFSROOT,tcp rw earlyprintk $@"
Yeah, the parameters are specific to the components within the model, so they vary depending on the structure of the model.
...but a generic-sounding option like this is highly likely to work on both models. (You mught need a different cluster* prefix, but it will be coretile.*.cpu0.semihosting-cmd_line -- we just need to parameterise it.)
- I did not get the kernel to boot at all, I suspect that our boot
wrapper is not compatible or I mis-configured the kernel or something in between. I did not even get a X11 terminal window as I usually always get.
Yes, we've been clear for some time that the bootwrapper needs changes to work with big.LITTLE (in fact I'm not entirely sure whether the switcher code counts as part of the 'kernel' or not). Dave probably
The generic bootwrapper and a vanilla VE+A15 kernel should work on the big.LITTLE model. I tried this before trying to get the switcher working.
I'm not sure how generic the bootwrapper used for KVM actually is, at this point (...?)
knows more about the status here (I suspect there's some working code somewhere, just maybe not in the git.linaro.org boot-wrapper or whichever kernel you tried).
The vanilla bootwrapper from linux-arm.org ought to work -- it's possible that the KVM version tries to poke actual hardware which isn't there in the b.L model. I was also using a separate kernel tree (git://git.linaro.org/people/dmart/linux-3-arm arm/vexpressdt-rtsm, built with the vexpress_rtsm_defconfig. This kernel needs a device tree, which you must pass using zImage dtb append, since the bootwrapper doesn't support DT yet. However, there is no special reason why we should not be able to use the same tree.)
Cheers ---Dave
On 15 February 2012 13:38, Dave Martin dave.martin@linaro.org wrote:
On Wed, Feb 15, 2012 at 12:44 AM, Peter Maydell peter.maydell@linaro.org wrote:
On 15 February 2012 00:33, Zygmunt Krynicki zygmunt.krynicki@linaro.org wrote:
Prompted by Paul I've decided to try a bigLITTLE config. My test did not end up very good:
- I've tried the RealView Express A15x1-A7x1 model
NB that I think the plan for big.LITTLE models is to use a prebuilt RTSM, which should look the same as a binary built from the example models, it's just a pre-cooked binary supplied by ARM rather than built by us.
I'm not actually sure exactly why we need a separate model for the b.L work -- but that's what we were given. If I understand correctly, the b.L platform can be built from the generic 7.0 fast models example platforms we have (this is what I was doing before the special pre-built model was available), but it's possible that the two models aren't exactly the same. That said, it shouldn't really matter. Launching the standalone model is almost exactly the same in practice as launching via model_shell.
My current recommendation for A15-for-KVM is to build and use a standalone ("isim system") model anyway, as it avoids the need to have model_shell available on the systems you want to run the model.
- It has different options that can be passed to model shell, needles
to say they are not compatible. I tweaked my semihosting line to look like this:
-C coretile.cluster0.cpu0.semihosting-cmd_line="--kernel $KERNEL -- console=ttyAMA0 mem=512M ip=dhcp root=/dev/nfs nfsroot=$NFSROOT,tcp rw earlyprintk $@"
Yeah, the parameters are specific to the components within the model, so they vary depending on the structure of the model.
...but a generic-sounding option like this is highly likely to work on both models. (You mught need a different cluster* prefix, but it will be coretile.*.cpu0.semihosting-cmd_line -- we just need to parameterise it.)
Yes, that is Zygmunt's point: on the A15 model it is cluster.cpu0.semihosting-cmd_line.
- I did not get the kernel to boot at all, I suspect that our boot
wrapper is not compatible or I mis-configured the kernel or something in between. I did not even get a X11 terminal window as I usually always get.
Yes, we've been clear for some time that the bootwrapper needs changes to work with big.LITTLE (in fact I'm not entirely sure whether the switcher code counts as part of the 'kernel' or not). Dave probably
The generic bootwrapper and a vanilla VE+A15 kernel should work on the big.LITTLE model. I tried this before trying to get the switcher working.
Yes, but that's not very exciting. I assumed Zygmunt was actually trying to run a big.LITTLE switcher+kernel, not just "boot some random kernel". Otherwise you're not really running big.LITTLE, you're just treating the system as an A15 based one.
I'm not sure how generic the bootwrapper used for KVM actually is, at this point (...?)
We haven't deliberately broken anything, but it is missing some features you're probably going to need, with "start kernel in Hyp mode" being the most obvious one.
We can add device tree support as well, I expect (insert complaint here about how you need to be able to understand the DTB blob just to pass the kernel its command line arguments).
-- PMM
On Wed, Feb 15, 2012 at 2:47 PM, Peter Maydell peter.maydell@linaro.org wrote:
On 15 February 2012 13:38, Dave Martin dave.martin@linaro.org wrote:
On Wed, Feb 15, 2012 at 12:44 AM, Peter Maydell peter.maydell@linaro.org wrote:
On 15 February 2012 00:33, Zygmunt Krynicki zygmunt.krynicki@linaro.org wrote:
Prompted by Paul I've decided to try a bigLITTLE config. My test did not end up very good:
- I've tried the RealView Express A15x1-A7x1 model
NB that I think the plan for big.LITTLE models is to use a prebuilt RTSM, which should look the same as a binary built from the example models, it's just a pre-cooked binary supplied by ARM rather than built by us.
I'm not actually sure exactly why we need a separate model for the b.L work -- but that's what we were given. If I understand correctly, the b.L platform can be built from the generic 7.0 fast models example platforms we have (this is what I was doing before the special pre-built model was available), but it's possible that the two models aren't exactly the same. That said, it shouldn't really matter. Launching the standalone model is almost exactly the same in practice as launching via model_shell.
My current recommendation for A15-for-KVM is to build and use a standalone ("isim system") model anyway, as it avoids the need to have model_shell available on the systems you want to run the model.
- It has different options that can be passed to model shell, needles
to say they are not compatible. I tweaked my semihosting line to look like this:
-C coretile.cluster0.cpu0.semihosting-cmd_line="--kernel $KERNEL -- console=ttyAMA0 mem=512M ip=dhcp root=/dev/nfs nfsroot=$NFSROOT,tcp rw earlyprintk $@"
Yeah, the parameters are specific to the components within the model, so they vary depending on the structure of the model.
...but a generic-sounding option like this is highly likely to work on both models. (You mught need a different cluster* prefix, but it will be coretile.*.cpu0.semihosting-cmd_line -- we just need to parameterise it.)
Yes, that is Zygmunt's point: on the A15 model it is cluster.cpu0.semihosting-cmd_line.
- I did not get the kernel to boot at all, I suspect that our boot
wrapper is not compatible or I mis-configured the kernel or something in between. I did not even get a X11 terminal window as I usually always get.
Yes, we've been clear for some time that the bootwrapper needs changes to work with big.LITTLE (in fact I'm not entirely sure whether the switcher code counts as part of the 'kernel' or not). Dave probably
The generic bootwrapper and a vanilla VE+A15 kernel should work on the big.LITTLE model. I tried this before trying to get the switcher working.
Yes, but that's not very exciting. I assumed Zygmunt was actually trying to run a big.LITTLE switcher+kernel, not just "boot some random kernel". Otherwise you're not really running big.LITTLE, you're just treating the system as an A15 based one.
Actually I just wanted to boot a naive kernel (the same one that works on plain A15) to see if it still works on a hybrid A14/A7 and it did not. I don't know how the switcher works, where to get it etc.
One thing that I find confusing is where to turn to for all the interesting ARM bleeding edge stuff. The kernel, the switcher (?), configuration, device tree, etc. Then there are KVM topics which I did not want to talk about here (as they are a next step after getting fast models to work).
I'm not sure how generic the bootwrapper used for KVM actually is, at this point (...?)
We haven't deliberately broken anything, but it is missing some features you're probably going to need, with "start kernel in Hyp mode" being the most obvious one.
We can add device tree support as well, I expect (insert complaint here about how you need to be able to understand the DTB blob just to pass the kernel its command line arguments).
-- PMM
On Wed, Feb 15, 2012 at 2:21 PM, Zygmunt Krynicki zygmunt.krynicki@linaro.org wrote:
On Wed, Feb 15, 2012 at 2:47 PM, Peter Maydell peter.maydell@linaro.org wrote:
On 15 February 2012 13:38, Dave Martin dave.martin@linaro.org wrote:
On Wed, Feb 15, 2012 at 12:44 AM, Peter Maydell peter.maydell@linaro.org wrote:
On 15 February 2012 00:33, Zygmunt Krynicki zygmunt.krynicki@linaro.org wrote:
Prompted by Paul I've decided to try a bigLITTLE config. My test did not end up very good:
- I've tried the RealView Express A15x1-A7x1 model
NB that I think the plan for big.LITTLE models is to use a prebuilt RTSM, which should look the same as a binary built from the example models, it's just a pre-cooked binary supplied by ARM rather than built by us.
I'm not actually sure exactly why we need a separate model for the b.L work -- but that's what we were given. If I understand correctly, the b.L platform can be built from the generic 7.0 fast models example platforms we have (this is what I was doing before the special pre-built model was available), but it's possible that the two models aren't exactly the same. That said, it shouldn't really matter. Launching the standalone model is almost exactly the same in practice as launching via model_shell.
My current recommendation for A15-for-KVM is to build and use a standalone ("isim system") model anyway, as it avoids the need to have model_shell available on the systems you want to run the model.
- It has different options that can be passed to model shell, needles
to say they are not compatible. I tweaked my semihosting line to look like this:
-C coretile.cluster0.cpu0.semihosting-cmd_line="--kernel $KERNEL -- console=ttyAMA0 mem=512M ip=dhcp root=/dev/nfs nfsroot=$NFSROOT,tcp rw earlyprintk $@"
Yeah, the parameters are specific to the components within the model, so they vary depending on the structure of the model.
...but a generic-sounding option like this is highly likely to work on both models. (You mught need a different cluster* prefix, but it will be coretile.*.cpu0.semihosting-cmd_line -- we just need to parameterise it.)
Yes, that is Zygmunt's point: on the A15 model it is cluster.cpu0.semihosting-cmd_line.
- I did not get the kernel to boot at all, I suspect that our boot
wrapper is not compatible or I mis-configured the kernel or something in between. I did not even get a X11 terminal window as I usually always get.
Yes, we've been clear for some time that the bootwrapper needs changes to work with big.LITTLE (in fact I'm not entirely sure whether the switcher code counts as part of the 'kernel' or not). Dave probably
The generic bootwrapper and a vanilla VE+A15 kernel should work on the big.LITTLE model. I tried this before trying to get the switcher working.
Yes, but that's not very exciting. I assumed Zygmunt was actually trying to run a big.LITTLE switcher+kernel, not just "boot some random kernel". Otherwise you're not really running big.LITTLE, you're just treating the system as an A15 based one.
Actually I just wanted to boot a naive kernel (the same one that works on plain A15) to see if it still works on a hybrid A14/A7 and it did not. I don't know how the switcher works, where to get it etc.
One thing that I find confusing is where to turn to for all the interesting ARM bleeding edge stuff. The kernel, the switcher (?), configuration, device tree, etc. Then there are KVM topics which I did not want to talk about here (as they are a next step after getting fast models to work).
I'm not sure how generic the bootwrapper used for KVM actually is, at this point (...?)
We haven't deliberately broken anything, but it is missing some features you're probably going to need, with "start kernel in Hyp mode" being the most obvious one.
We can add device tree support as well, I expect (insert complaint here about how you need to be able to understand the DTB blob just to pass the kernel its command line arguments).
There is no single location, and in fact I don't have a pair of trees which works together right now.
For integration purposes, it should not matter. You're booting some model with some bootwrapper and some kernel. When we do have something working, we can ping you -- but for now, it's probably best to assume that the integration will be just like the KVM case on your side. If not, that is most likely a bug, and not an integration problem..
Cheers ---Dave
linaro-validation@lists.linaro.org
-
Dave Martin -
Peter Maydell -
Zygmunt Krynicki