If you don't follow dispatcher commit or review mail, you might not be aware than Andy and I have been working an implementation of the "black box" tests we've been talking about for a while. This mail is just a summary of the things that have been mentioned in review that would be good to fix "in the next branch" so that we don't forget about them. The list probably doesn't make a great deal of sense if you haven't read the review mail, sorry about that.
In no particular order:
* explode if an android test has deps rather than trying to install via apt-get
* need to have lava-test-runner be a proper service on android
* make power_off() == 'in the master image' consistently
* make cmd_android_install_binaries work for !master deployments
* think about suppressing the root shell on console on ubuntu
* remove separation between LavaClient and TargetBasedClient, consider slimming (and renaming?) interface
* fix retreive_results return interface
* rename download_image
* look at how boot_options works
* define common target superclass for FastModelTarget and QEMUTarget
* set up automated dispatcher validation
I've just thought of one more thing, which is that running a lava_test_shell action followed by a lava_test_run action in the same job will likely not work well because the lava test shell test will be run again when the lava_test_run action boots the image -- maybe the lava-test-runner should clean up after itself so that the tests only run on *the* next boot rather than every following boot? If we do do the "disable auto-login-shell" action, then it should probably reenable that too...
Cheers, mwh
On 10/04/2012 02:17 PM, Michael Hudson-Doyle wrote:
If you don't follow dispatcher commit or review mail, you might not be aware than Andy and I have been working an implementation of the "black box" tests we've been talking about for a while. This mail is just a summary of the things that have been mentioned in review that would be good to fix "in the next branch" so that we don't forget about them. The list probably doesn't make a great deal of sense if you haven't read the review mail, sorry about that.
In no particular order:
explode if an android test has deps rather than trying to install via apt-get
need to have lava-test-runner be a proper service on android
make power_off() == 'in the master image' consistently
make cmd_android_install_binaries work for !master deployments
think about suppressing the root shell on console on ubuntu
remove separation between LavaClient and TargetBasedClient, consider slimming (and renaming?) interface
fix retreive_results return interface
rename download_image
look at how boot_options works
define common target superclass for FastModelTarget and QEMUTarget
set up automated dispatcher validation
I'm a +1 for all of that.
I've just thought of one more thing, which is that running a lava_test_shell action followed by a lava_test_run action in the same job will likely not work well because the lava test shell test will be run again when the lava_test_run action boots the image -- maybe the lava-test-runner should clean up after itself so that the tests only run on *the* next boot rather than every following boot? If we do do the "disable auto-login-shell" action, then it should probably reenable that too...
Good idea. We should do definitely do this.
Cheers, mwh
linaro-validation mailing list linaro-validation@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-validation
Michael Hudson-Doyle michael.hudson@linaro.org writes:
If you don't follow dispatcher commit or review mail, you might not be aware than Andy and I have been working an implementation of the "black box" tests we've been talking about for a while. This mail is just a summary of the things that have been mentioned in review that would be good to fix "in the next branch" so that we don't forget about them. The list probably doesn't make a great deal of sense if you haven't read the review mail, sorry about that.
Watching an execution of the dispatcher running a lava-test-shell carefully, I came up with another one:
* Failing to boot when we should be running lava-test-shell looks like it will just timeout looking for "<LAVA_TEST_RUNNER>: exiting". I think we should at least look for "<LAVA_TEST_RUNNER>: starting" and report a failure to see that differently. I'm not sure if we can reliably tell the difference between the device failing to boot and lava-test-runner failing to run because we goofed on installing the upstart job. Would be nice if we could.
Cheers, mwh
On 10/08/2012 09:22 PM, Michael Hudson-Doyle wrote:
Michael Hudson-Doyle michael.hudson@linaro.org writes:
If you don't follow dispatcher commit or review mail, you might not be aware than Andy and I have been working an implementation of the "black box" tests we've been talking about for a while. This mail is just a summary of the things that have been mentioned in review that would be good to fix "in the next branch" so that we don't forget about them. The list probably doesn't make a great deal of sense if you haven't read the review mail, sorry about that.
Watching an execution of the dispatcher running a lava-test-shell carefully, I came up with another one:
- Failing to boot when we should be running lava-test-shell looks like it will just timeout looking for "<LAVA_TEST_RUNNER>: exiting". I think we should at least look for "<LAVA_TEST_RUNNER>: starting" and report a failure to see that differently. I'm not sure if we can reliably tell the difference between the device failing to boot and lava-test-runner failing to run because we goofed on installing the upstart job. Would be nice if we could.
I don't know if we can differentiate between boot failure and test-runner failure. But we could check for the "starting" message and then be able to log a more accurate error upon a timeout.
+1
linaro-validation@lists.linaro.org
-
Andy Doan -
Michael Hudson-Doyle