W dniu 30.07.2012 16:10, Alexander Sack pisze:
Hi,
big items we need to sort out:
- linaro-media-create install support
- lava-test support
- lava support
I had some thought on it (OpenEmbedded builds and LAVA) and due to that here are some questions.
1. Do tested image needs to have LAVA client code installed?
If it does then I would have to add LAVA client components into OpenEmbedded because I can not use Ubuntu packages as they are not compatible with each other.
2. Does lava build require rootfs + hwpack or can boot with just rootfs?
Rootfs will contain kernel in /boot/uImage and u-boot configuration in /boot/boot.scr (or other defined location). If hwpack is required then I will have to create one with OE and add support for them in l-m-c (as we can not use Ubuntu packages for things other than kernel/bootloader cause there is no warranty about binary compatibility).
I think if the fast model route is blocked we should take the pain to work on real board.
For now using real board is less pain then using fast model. I have real boards available at home so can test images on them.
On Tue, Jul 31, 2012 at 8:40 AM, Marcin Juszkiewicz < marcin.juszkiewicz@linaro.org> wrote:
W dniu 30.07.2012 16:10, Alexander Sack pisze:
Hi,
big items we need to sort out:
- linaro-media-create install support
- lava-test support
- lava support
I had some thought on it (OpenEmbedded builds and LAVA) and due to that here are some questions.
I haven't heard much about our OE plans, but would like to... are there further details of what we're looking to do here somewhere? Is this just for v8, or for other things too?
- Do tested image needs to have LAVA client code installed?
If it does then I would have to add LAVA client components into OpenEmbedded because I can not use Ubuntu packages as they are not compatible with each other.
Not really, lava-test can be installed easily on the target once it's deployed, but to the best of my knowledge, I don't think anyone's really tried running lava-test under OE. We would need pip to install it, and possibly a few other python dependencies. The bigger problem here is that lava-test was always fairly dependent on apt for a number of things. This can be worked around telling it not to gather a list of packages installed when it runs, but many of the tests rely on dependency installation via packages.
One of the things I've raised in the past that we should look at is consolidation of lava-test and lava-android-test by means of a host driven test framework, and this is one reason why. Looking to the future, I think we want to support more than ubuntu and android, and maybe even more than OE. Adding a lava-something-test every time is not going to be the best way to go about it.
- Does lava build require rootfs + hwpack or can boot with just rootfs?
Rootfs will contain kernel in /boot/uImage and u-boot configuration in /boot/boot.scr (or other defined location). If hwpack is required then I will have to create one with OE and add support for them in l-m-c (as we can not use Ubuntu packages for things other than kernel/bootloader cause there is no warranty about binary compatibility).
Certainly the dispatcher will have to support another means of installation, just like we have to for Android images. If linaro-media-create is used, then this makes things slightly better, but I'm sure it'll need at least a bit of special handling in the dispatcher, though probably not as bad as android was. I don't foresee a big problem here.
I think if the fast model route is blocked we should take the pain to work on real board.
For now using real board is less pain then using fast model. I have real boards available at home so can test images on them.
linaro-validation mailing list linaro-validation@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-validation
On 07/31/2012 01:33 PM, Paul Larson wrote:
One of the things I've raised in the past that we should look at is consolidation of lava-test and lava-android-test by means of a host driven test framework, and this is one reason why. Looking to the future, I think we want to support more than ubuntu and android, and maybe even more than OE. Adding a lava-something-test every time is not going to be the best way to go about it.
I brought up the same type of idea during my 1x1 with Alexander today. I'd even go 1 step beyond this and say this is where the black box concept would fit well.
On Tue, Jul 31, 2012 at 2:27 PM, Andy Doan andy.doan@linaro.org wrote:
On 07/31/2012 01:33 PM, Paul Larson wrote:
One of the things I've raised in the past that we should look at is consolidation of lava-test and lava-android-test by means of a host driven test framework, and this is one reason why. Looking to the future, I think we want to support more than ubuntu and android, and maybe even more than OE. Adding a lava-something-test every time is not going to be the best way to go about it.
I brought up the same type of idea during my 1x1 with Alexander today. I'd even go 1 step beyond this and say this is where the black box concept would fit well.
Yes, indeed. It requires rethinking things a bit, but I think provides the most overall flexibility. Of course, the case could also be made that this functionality could be merged into the dispatcher itself. But I don't want to digress this thread too far. We should start another thread for this, and perhaps take up the discussion at connect or similar.
Andy Doan andy.doan@linaro.org writes:
On 07/31/2012 01:33 PM, Paul Larson wrote:
One of the things I've raised in the past that we should look at is consolidation of lava-test and lava-android-test by means of a host driven test framework, and this is one reason why. Looking to the future, I think we want to support more than ubuntu and android, and maybe even more than OE. Adding a lava-something-test every time is not going to be the best way to go about it.
I brought up the same type of idea during my 1x1 with Alexander today. I'd even go 1 step beyond this and say this is where the black box concept would fit well.
Yes please.
Cheers, mwh
On Tue, Jul 31, 2012 at 3:33 PM, Paul Larson paul.larson@linaro.org wrote:
On Tue, Jul 31, 2012 at 8:40 AM, Marcin Juszkiewicz marcin.juszkiewicz@linaro.org wrote:
W dniu 30.07.2012 16:10, Alexander Sack pisze:
Hi,
big items we need to sort out:
- linaro-media-create install support
- lava-test support
- lava support
I had some thought on it (OpenEmbedded builds and LAVA) and due to that here are some questions.
I haven't heard much about our OE plans, but would like to... are there further details of what we're looking to do here somewhere? Is this just for v8, or for other things too?
Currently it's mainly for ARMv8, but who knows what might be coming later on :-) If it's successful, we'll probably have more requests to develop projects with it.
Also, the ARMv8 work should really be starting in the following weeks.
- Do tested image needs to have LAVA client code installed?
If it does then I would have to add LAVA client components into OpenEmbedded because I can not use Ubuntu packages as they are not compatible with each other.
Not really, lava-test can be installed easily on the target once it's deployed, but to the best of my knowledge, I don't think anyone's really tried running lava-test under OE. We would need pip to install it, and possibly a few other python dependencies. The bigger problem here is that lava-test was always fairly dependent on apt for a number of things. This can be worked around telling it not to gather a list of packages installed when it runs, but many of the tests rely on dependency installation via packages.
The first idea I had was to basically have a way at lava-tests to skip the step which it installs the test dependencies. At least with that approach, we could already have the dependencies in place for the test cases we care about.
Otherwise, we'd need a way to dynamically install the dependencies, which can be quite complicated with OE as we'd need to create and export a repository.
One of the things I've raised in the past that we should look at is consolidation of lava-test and lava-android-test by means of a host driven test framework, and this is one reason why.
Can you explain more about how this host driven test framework would work?
Looking to the future, I think we want to support more than ubuntu and android, and maybe even more than OE. Adding a lava-something-test every time is not going to be the best way to go about it.
I agree, and this is something that most test suites have to deal as well. The only easy way to be generic enough, is to build the test cases and the dependencies natively at the image (or at least make sure the image already delivers a basic set of dependencies by default).
- Does lava build require rootfs + hwpack or can boot with just rootfs?
Rootfs will contain kernel in /boot/uImage and u-boot configuration in /boot/boot.scr (or other defined location). If hwpack is required then I will have to create one with OE and add support for them in l-m-c (as we can not use Ubuntu packages for things other than kernel/bootloader cause there is no warranty about binary compatibility).
Certainly the dispatcher will have to support another means of installation, just like we have to for Android images. If linaro-media-create is used, then this makes things slightly better, but I'm sure it'll need at least a bit of special handling in the dispatcher, though probably not as bad as android was. I don't foresee a big problem here.
I believe we could even use the hwpacks we produce for Ubuntu here, but only dealing with the kernel and bootloader (skipping all other packages). As we don't care much about upgradability, we could have a logic at linaro-media-create to just extract the debian packages related with the kernel and bootloader (something that happens in some way anyway).
With that logic in place, we could support any rootfs, even Android (and share the same basic set of hwpacks we also use at Ubuntu).
Cheers,
Ricardo Salveti ricardo.salveti@linaro.org writes:
On Tue, Jul 31, 2012 at 3:33 PM, Paul Larson paul.larson@linaro.org wrote:
On Tue, Jul 31, 2012 at 8:40 AM, Marcin Juszkiewicz marcin.juszkiewicz@linaro.org wrote:
W dniu 30.07.2012 16:10, Alexander Sack pisze:
Hi,
big items we need to sort out:
- linaro-media-create install support
- lava-test support
- lava support
I had some thought on it (OpenEmbedded builds and LAVA) and due to that here are some questions.
I haven't heard much about our OE plans, but would like to... are there further details of what we're looking to do here somewhere? Is this just for v8, or for other things too?
Currently it's mainly for ARMv8, but who knows what might be coming later on :-) If it's successful, we'll probably have more requests to develop projects with it.
Also, the ARMv8 work should really be starting in the following weeks.
Can we start by only testing things we actually care about for this (and other new) work? :-)
- Do tested image needs to have LAVA client code installed?
If it does then I would have to add LAVA client components into OpenEmbedded because I can not use Ubuntu packages as they are not compatible with each other.
Not really, lava-test can be installed easily on the target once it's deployed, but to the best of my knowledge, I don't think anyone's really tried running lava-test under OE. We would need pip to install it, and possibly a few other python dependencies. The bigger problem here is that lava-test was always fairly dependent on apt for a number of things. This can be worked around telling it not to gather a list of packages installed when it runs, but many of the tests rely on dependency installation via packages.
The first idea I had was to basically have a way at lava-tests to skip the step which it installs the test dependencies. At least with that approach, we could already have the dependencies in place for the test cases we care about.
I think I'm coming around to the android team's POV here: tests don't have dependencies -- you test things in the base system (for a while we were running the ubuntu-leb-graphics tests on a nano image, which installed X and everything else first -- madness!).
Otherwise, we'd need a way to dynamically install the dependencies, which can be quite complicated with OE as we'd need to create and export a repository.
One of the things I've raised in the past that we should look at is consolidation of lava-test and lava-android-test by means of a host driven test framework, and this is one reason why.
Can you explain more about how this host driven test framework would work?
So the idea is that "test installation" arranges for the test to run on next boot of the test system, putting the result output in a known location. Then we boot the device, wait for the test to complete (hand waving here), fetch the results from the known location and parse them into a bundle. There will be a system dependent part around how the tests are started -- for ubuntu you might create an upstart job, for OE I guess you'll do something else -- but that feels like a small amount of variation.
How you get the result output after the tests have run will also vary by deployment approach, but we already handle this today to get the bundles off the device.
Looking to the future, I think we want to support more than ubuntu and android, and maybe even more than OE. Adding a lava-something-test every time is not going to be the best way to go about it.
I agree, and this is something that most test suites have to deal as well. The only easy way to be generic enough, is to build the test cases and the dependencies natively at the image (or at least make sure the image already delivers a basic set of dependencies by default).
Ah heh, maybe I should have read further before hitting reply :)
- Does lava build require rootfs + hwpack or can boot with just rootfs?
Rootfs will contain kernel in /boot/uImage and u-boot configuration in /boot/boot.scr (or other defined location). If hwpack is required then I will have to create one with OE and add support for them in l-m-c (as we can not use Ubuntu packages for things other than kernel/bootloader cause there is no warranty about binary compatibility).
Certainly the dispatcher will have to support another means of installation, just like we have to for Android images. If linaro-media-create is used, then this makes things slightly better, but I'm sure it'll need at least a bit of special handling in the dispatcher, though probably not as bad as android was. I don't foresee a big problem here.
I believe we could even use the hwpacks we produce for Ubuntu here, but only dealing with the kernel and bootloader (skipping all other packages). As we don't care much about upgradability, we could have a logic at linaro-media-create to just extract the debian packages related with the kernel and bootloader (something that happens in some way anyway).
I guess. I don't think the way it happens today is at all reusable fwiw: it looks like it looks it depends on linaro-hwpack-install installing the a package that creates a path such as boot/vmlinuz-*-linaro-omap. I presume it wouldn't be very hard to make a guess at which deb creates this file and use ar/tar to "install" it, but it would be new code I think.
With that logic in place, we could support any rootfs, even Android (and share the same basic set of hwpacks we also use at Ubuntu).
Cheers, mwh
On Sun, Aug 5, 2012 at 6:50 PM, Michael Hudson-Doyle michael.hudson@linaro.org wrote:
Ricardo Salveti ricardo.salveti@linaro.org writes:
On Tue, Jul 31, 2012 at 3:33 PM, Paul Larson paul.larson@linaro.org wrote:
On Tue, Jul 31, 2012 at 8:40 AM, Marcin Juszkiewicz marcin.juszkiewicz@linaro.org wrote:
W dniu 30.07.2012 16:10, Alexander Sack pisze:
Hi,
big items we need to sort out:
- linaro-media-create install support
- lava-test support
- lava support
I had some thought on it (OpenEmbedded builds and LAVA) and due to that here are some questions.
I haven't heard much about our OE plans, but would like to... are there further details of what we're looking to do here somewhere? Is this just for v8, or for other things too?
Currently it's mainly for ARMv8, but who knows what might be coming later on :-) If it's successful, we'll probably have more requests to develop projects with it.
Also, the ARMv8 work should really be starting in the following weeks.
Can we start by only testing things we actually care about for this (and other new) work? :-)
- Do tested image needs to have LAVA client code installed?
If it does then I would have to add LAVA client components into OpenEmbedded because I can not use Ubuntu packages as they are not compatible with each other.
Not really, lava-test can be installed easily on the target once it's deployed, but to the best of my knowledge, I don't think anyone's really tried running lava-test under OE. We would need pip to install it, and possibly a few other python dependencies. The bigger problem here is that lava-test was always fairly dependent on apt for a number of things. This can be worked around telling it not to gather a list of packages installed when it runs, but many of the tests rely on dependency installation via packages.
The first idea I had was to basically have a way at lava-tests to skip the step which it installs the test dependencies. At least with that approach, we could already have the dependencies in place for the test cases we care about.
I think I'm coming around to the android team's POV here: tests don't have dependencies -- you test things in the base system (for a while we were running the ubuntu-leb-graphics tests on a nano image, which installed X and everything else first -- madness!).
It can have, but if they are part of the image itself, we can at least cover that by default.
The problem is that supporting more test cases is directly connected on producing new kinds of images (with extra packages and such), which can be painful.
Otherwise, we'd need a way to dynamically install the dependencies, which can be quite complicated with OE as we'd need to create and export a repository.
One of the things I've raised in the past that we should look at is consolidation of lava-test and lava-android-test by means of a host driven test framework, and this is one reason why.
Can you explain more about how this host driven test framework would work?
So the idea is that "test installation" arranges for the test to run on next boot of the test system, putting the result output in a known location. Then we boot the device, wait for the test to complete (hand waving here), fetch the results from the known location and parse them into a bundle. There will be a system dependent part around how the tests are started -- for ubuntu you might create an upstart job, for OE I guess you'll do something else -- but that feels like a small amount of variation.
How you get the result output after the tests have run will also vary by deployment approach, but we already handle this today to get the bundles off the device.
Sounds fine, while you don't have total progress in real time, it's a lot simpler to follow up the execution and parsing.
While it can help depending on the environment you have, it'll not improve the current situation a lot (it'll probably just make the maintenance a bit easier). Unless the serial communication is an issue, or unreliable enough to break the test executions, it can for sure be one of the supported methods across images and platforms.
Looking to the future, I think we want to support more than ubuntu and android, and maybe even more than OE. Adding a lava-something-test every time is not going to be the best way to go about it.
I agree, and this is something that most test suites have to deal as well. The only easy way to be generic enough, is to build the test cases and the dependencies natively at the image (or at least make sure the image already delivers a basic set of dependencies by default).
Ah heh, maybe I should have read further before hitting reply :)
- Does lava build require rootfs + hwpack or can boot with just rootfs?
Rootfs will contain kernel in /boot/uImage and u-boot configuration in /boot/boot.scr (or other defined location). If hwpack is required then I will have to create one with OE and add support for them in l-m-c (as we can not use Ubuntu packages for things other than kernel/bootloader cause there is no warranty about binary compatibility).
Certainly the dispatcher will have to support another means of installation, just like we have to for Android images. If linaro-media-create is used, then this makes things slightly better, but I'm sure it'll need at least a bit of special handling in the dispatcher, though probably not as bad as android was. I don't foresee a big problem here.
I believe we could even use the hwpacks we produce for Ubuntu here, but only dealing with the kernel and bootloader (skipping all other packages). As we don't care much about upgradability, we could have a logic at linaro-media-create to just extract the debian packages related with the kernel and bootloader (something that happens in some way anyway).
I guess. I don't think the way it happens today is at all reusable fwiw: it looks like it looks it depends on linaro-hwpack-install installing the a package that creates a path such as boot/vmlinuz-*-linaro-omap. I presume it wouldn't be very hard to make a guess at which deb creates this file and use ar/tar to "install" it, but it would be new code I think.
With that logic in place, we could support any rootfs, even Android (and share the same basic set of hwpacks we also use at Ubuntu).
The code for such usage is mostly done at https://code.launchpad.net/~rsalveti/linaro-image-tools/generic-oe-support/, and I was able to test with the current OE images I produced locally. I'll be testing it properly in different use cases in the next following days, but should be ready to merge/review soon.
The idea is basically to use linaro-media-create the same way as it's done with Ubuntu's images, so we avoid changing much on the interface of LAVA or any other tool that relies on lmc.
The only problem I see in the future, which will require a bit of re-work again (on lmc), is when we start bootstrapping a new architecture, as we'll not have any qemu support to run native code (post installs or similar). While it's not that complicated, it can get messy because of the current way lmc is flashing up the images.
Cheers,
Ricardo Salveti ricardo.salveti@linaro.org writes:
The first idea I had was to basically have a way at lava-tests to skip the step which it installs the test dependencies. At least with that approach, we could already have the dependencies in place for the test cases we care about.
I think I'm coming around to the android team's POV here: tests don't have dependencies -- you test things in the base system (for a while we were running the ubuntu-leb-graphics tests on a nano image, which installed X and everything else first -- madness!).
It can have, but if they are part of the image itself, we can at least cover that by default.
I don't quite understand here, I'm afraid.
The problem is that supporting more test cases is directly connected on producing new kinds of images (with extra packages and such), which can be painful.
Maybe we should make that easier then?
Otherwise, we'd need a way to dynamically install the dependencies, which can be quite complicated with OE as we'd need to create and export a repository.
One of the things I've raised in the past that we should look at is consolidation of lava-test and lava-android-test by means of a host driven test framework, and this is one reason why.
Can you explain more about how this host driven test framework would work?
So the idea is that "test installation" arranges for the test to run on next boot of the test system, putting the result output in a known location. Then we boot the device, wait for the test to complete (hand waving here), fetch the results from the known location and parse them into a bundle. There will be a system dependent part around how the tests are started -- for ubuntu you might create an upstart job, for OE I guess you'll do something else -- but that feels like a small amount of variation.
How you get the result output after the tests have run will also vary by deployment approach, but we already handle this today to get the bundles off the device.
Sounds fine, while you don't have total progress in real time, it's a lot simpler to follow up the execution and parsing.
We could spam the output to the serial console as well while the test is running -- probably a good idea, in fact.
While it can help depending on the environment you have, it'll not improve the current situation a lot (it'll probably just make the maintenance a bit easier).
One of the nice things is that it means that we don't have to install (and have working) things like lava-test on the DUT, which is nice for things like fast models of new architectures :-)
Certainly the dispatcher will have to support another means of installation, just like we have to for Android images. If linaro-media-create is used, then this makes things slightly better, but I'm sure it'll need at least a bit of special handling in the dispatcher, though probably not as bad as android was. I don't foresee a big problem here.
I believe we could even use the hwpacks we produce for Ubuntu here, but only dealing with the kernel and bootloader (skipping all other packages). As we don't care much about upgradability, we could have a logic at linaro-media-create to just extract the debian packages related with the kernel and bootloader (something that happens in some way anyway).
I guess. I don't think the way it happens today is at all reusable fwiw: it looks like it looks it depends on linaro-hwpack-install installing the a package that creates a path such as boot/vmlinuz-*-linaro-omap. I presume it wouldn't be very hard to make a guess at which deb creates this file and use ar/tar to "install" it, but it would be new code I think.
With that logic in place, we could support any rootfs, even Android (and share the same basic set of hwpacks we also use at Ubuntu).
The code for such usage is mostly done at https://code.launchpad.net/~rsalveti/linaro-image-tools/generic-oe-support/, and I was able to test with the current OE images I produced locally. I'll be testing it properly in different use cases in the next following days, but should be ready to merge/review soon.
Ah, cool.
The idea is basically to use linaro-media-create the same way as it's done with Ubuntu's images, so we avoid changing much on the interface of LAVA or any other tool that relies on lmc.
The only problem I see in the future, which will require a bit of re-work again (on lmc), is when we start bootstrapping a new architecture, as we'll not have any qemu support to run native code (post installs or similar).
Yeah, that's another reason to do blackbox-style/host-side testing...
While it's not that complicated, it can get messy because of the current way lmc is flashing up the images.
Cheers, mwh
linaro-validation@lists.linaro.org
-
Andy Doan -
Marcin Juszkiewicz -
Michael Hudson-Doyle -
Paul Larson -
Ricardo Salveti