Ricardo Salveti ricardo.salveti@linaro.org writes:
The first idea I had was to basically have a way at lava-tests to skip the step which it installs the test dependencies. At least with that approach, we could already have the dependencies in place for the test cases we care about.
I think I'm coming around to the android team's POV here: tests don't have dependencies -- you test things in the base system (for a while we were running the ubuntu-leb-graphics tests on a nano image, which installed X and everything else first -- madness!).
It can have, but if they are part of the image itself, we can at least cover that by default.
I don't quite understand here, I'm afraid.
The problem is that supporting more test cases is directly connected on producing new kinds of images (with extra packages and such), which can be painful.
Maybe we should make that easier then?
Otherwise, we'd need a way to dynamically install the dependencies, which can be quite complicated with OE as we'd need to create and export a repository.
One of the things I've raised in the past that we should look at is consolidation of lava-test and lava-android-test by means of a host driven test framework, and this is one reason why.
Can you explain more about how this host driven test framework would work?
So the idea is that "test installation" arranges for the test to run on next boot of the test system, putting the result output in a known location. Then we boot the device, wait for the test to complete (hand waving here), fetch the results from the known location and parse them into a bundle. There will be a system dependent part around how the tests are started -- for ubuntu you might create an upstart job, for OE I guess you'll do something else -- but that feels like a small amount of variation.
How you get the result output after the tests have run will also vary by deployment approach, but we already handle this today to get the bundles off the device.
Sounds fine, while you don't have total progress in real time, it's a lot simpler to follow up the execution and parsing.
We could spam the output to the serial console as well while the test is running -- probably a good idea, in fact.
While it can help depending on the environment you have, it'll not improve the current situation a lot (it'll probably just make the maintenance a bit easier).
One of the nice things is that it means that we don't have to install (and have working) things like lava-test on the DUT, which is nice for things like fast models of new architectures :-)
Certainly the dispatcher will have to support another means of installation, just like we have to for Android images. If linaro-media-create is used, then this makes things slightly better, but I'm sure it'll need at least a bit of special handling in the dispatcher, though probably not as bad as android was. I don't foresee a big problem here.
I believe we could even use the hwpacks we produce for Ubuntu here, but only dealing with the kernel and bootloader (skipping all other packages). As we don't care much about upgradability, we could have a logic at linaro-media-create to just extract the debian packages related with the kernel and bootloader (something that happens in some way anyway).
I guess. I don't think the way it happens today is at all reusable fwiw: it looks like it looks it depends on linaro-hwpack-install installing the a package that creates a path such as boot/vmlinuz-*-linaro-omap. I presume it wouldn't be very hard to make a guess at which deb creates this file and use ar/tar to "install" it, but it would be new code I think.
With that logic in place, we could support any rootfs, even Android (and share the same basic set of hwpacks we also use at Ubuntu).
The code for such usage is mostly done at https://code.launchpad.net/~rsalveti/linaro-image-tools/generic-oe-support/, and I was able to test with the current OE images I produced locally. I'll be testing it properly in different use cases in the next following days, but should be ready to merge/review soon.
Ah, cool.
The idea is basically to use linaro-media-create the same way as it's done with Ubuntu's images, so we avoid changing much on the interface of LAVA or any other tool that relies on lmc.
The only problem I see in the future, which will require a bit of re-work again (on lmc), is when we start bootstrapping a new architecture, as we'll not have any qemu support to run native code (post installs or similar).
Yeah, that's another reason to do blackbox-style/host-side testing...
While it's not that complicated, it can get messy because of the current way lmc is flashing up the images.
Cheers, mwh