Multiple conversations over the last week have convinced me that lava-test, as it currently is, is not well suited to the way LAVA is changing.
I should say that I'm writing this email more to start us thinking about where we're going rather han any immediate plans to start coding.
The fundamental problem is that it runs solely on the device under test (DUT). This has two problems:
1) It seems ill-suited to tests where the not all of the data produced by the test originates from the device being tested (think power measurement or hdmi capture here).
2) We do too much work on the DUT. As Zygmunt can tell you, just installing lava-test on a fast model is quite a trial; doing the test result parsing and bundle formatting there is just silly.
I think that both of these things suggest that the 'brains' of the test running process should run on the host side, somewhat as lava-android-test does already.
Surprisingly enough, I don't think this necessarily requires changing much at all about how we specify the tests. At the end of the day, a test definition defines a bunch of shell commands to run, and we could move to a model where lava-test sends these to the board[1] to be executed rather than running them through os.system or whatever it runs now (parsing is different I guess, but if we can get the output onto the the host, we can just run parsing there).
To actually solve the problems of 1 and 2 above though we will want some extensions I think.
For point 1, we clearly need some way to specify how to get the data from the other data source. I don't have any bright ideas here :-)
In the theme of point 2, if we can specify installation in a more declarative way than "run these shell commands" there is a change we can perform some of these steps on the host -- for example, stream installation could really just drop a pre-compiled binary at a particular location on the testrootfs before flashing it to the SD card. Tests can already depend on debian packages to be installed, which I guess is a particular case of this (and "apt-get install" usually works fine when chrooted into a armel or armhf rootfs with qemu-arm-static in the right place).
We might want to take different approaches for different backends -- for example, running the install steps on real hardware might not be any slower and certainly parallizes better than running them on the host via qemu, but the same is emphatically not the case for fast models.
Comments? Thoughts?
Cheers, mwh
[1] One way of doing this would be to create (on the testrootfs) a shell script that runs all the tests and an upstart job that runs the tests on boot -- this would avoid depending on a reliable network or serial console in the test image (although producing output on the serial console would still be useful for people watching the job).