Multiple conversations over the last week have convinced me that
lava-test, as it currently is, is not well suited to the way LAVA is
changing.

I should say that I'm writing this email more to start us thinking
about where we're going rather han any immediate plans to start
coding.

The fundamental problem is that it runs solely on the device under
test (DUT).  This has two problems:

 1) It seems ill-suited to tests where the not all of the data
   produced by the test originates from the device being tested
   (think power measurement or hdmi capture here).

 2) We do too much work on the DUT.  As Zygmunt can tell you, just
   installing lava-test on a fast model is quite a trial; doing the
   test result parsing and bundle formatting there is just silly.

I think that both of these things suggest that the 'brains' of the
test running process should run on the host side, somewhat as
lava-android-test does already.

Surprisingly enough, I don't think this necessarily requires changing
much at all about how we specify the tests.  At the end of the day, a
test definition defines a bunch of shell commands to run, and we could
move to a model where lava-test sends these to the board[1] to be
executed rather than running them through os.system or whatever it
runs now (parsing is different I guess, but if we can get the output
onto the the host, we can just run parsing there).

To actually solve the problems of 1 and 2 above though we will want
some extensions I think.

For point 1, we clearly need some way to specify how to get the data
from the other data source.  I don't have any bright ideas here :-)

In the theme of point 2, if we can specify installation in a more
declarative way than "run these shell commands" there is a change we
can perform some of these steps on the host -- for example, stream
installation could really just drop a pre-compiled binary at a
particular location on the testrootfs before flashing it to the SD
card.  Tests can already depend on debian packages to be installed,
which I guess is a particular case of this (and "apt-get install"
usually works fine when chrooted into a armel or armhf rootfs with
qemu-arm-static in the right place).

We might want to take different approaches for different backends --
for example, running the install steps on real hardware might not be
any slower and certainly parallizes better than running them on the
host via qemu, but the same is emphatically not the case for fast
models.

Comments?  Thoughts?

Cheers,
mwh

[1] One way of doing this would be to create (on the testrootfs) a
   shell script that runs all the tests and an upstart job that runs
   the tests on boot -- this would avoid depending on a reliable
   network or serial console in the test image (although producing
   output on the serial console would still be useful for people
   watching the job).

_______________________________________________
linaro-validation mailing list
linaro-validation@lists.linaro.org
http://lists.linaro.org/mailman/listinfo/linaro-validation