On Fri, Mar 9, 2012 at 8:05 AM, Paul Larson paul.larson@linaro.org wrote:
On Fri, Mar 9, 2012 at 12:16 AM, Zach Pfeffer zach.pfeffer@linaro.orgwrote:
On 8 March 2012 23:44, Paul Larson paul.larson@linaro.org wrote:
On Thu, Mar 8, 2012 at 4:07 PM, Zach Pfeffer zach.pfeffer@linaro.org wrote:
Right. If a command in the command log causes the unit-under-test to do any of those things, then the unit should be rebooted (in the case of a hang) or the reboot should be sensed (in case the command caused a reboot) and when the unit boots back up, LAVA would continue the test on the command after the one that caused it to hang, reboot, freeze, etc. LAVA should save the logs for the entire command file run, including all hangs and reboots.
As mentioned when we talked about this on the phone, I don't think this
is
the best way to approach the problem. Currently, if a test hangs, times
out,
reboots improperly (and thus times out because it won't get to the
expected
place), lava will reboot the system back into the test image. However,
at
this point, it will mark the previous testsuite it tried to run as a
failed
run. Then, if there are other tests queued up to run, it will continue running with the next one - NOT try to re-enter the existing test and continue where it left off. This is not a capability currently
supported in
lava-android-test.
The good news is, I think there's a much more straightforward way to do this. I haven't really seen an example of the kinds of things you want to
run, but
it seems that it's just a list of several tests, possibly with multiple steps in each test. And if it reboots, or has a problem, you want it to fail that one and continue to the next. This is very simple to do, if
you
simply define each of those tests as lava-android-tests. Then you can
run
them all in your test job, and I think it will do exactly what you are looking for here. Also, they can then be easily reused in other test
jobs.
Hmm...
Here's what I'd like to do.
I'd like to pass a simple command file to LAVA. I'd like LAVA to execute each test and if that test hangs or causes a reboot I'd like LAVA to pick back up at the next test case. I'd like LAVA to collect all the logs and send them either to LAVA or back to us for post processing.
I'd like to do this, because I'd like to be able to include these command files in our builds so that people can run them themselves and include the post processing commands for people to see what passed and failed.
The text file also gives me a very easy way to add and remove tests on a per-build basis since it goes along with the build.
I get that, but I don't see why you can't have it both ways. If you want a simple set of instructions for someone to manually type at a command line to run through the tests, there's nothing preventing you from including that in a readme, script, or however you want to do that. But those tests described in there would be really nice to convert into reusable tests in lava. Once you have those building blocks, you can arrange them in future builds however you like.
I think what Zach is saying here is that he wants to use a single file to maintain the list of tests to run for a build. That file is then sourced both by a tool used by developers for local testing as well as by LAVA.
He most likely also would want the local tool used by devs to be a standard solution provided by the lava test framework...