On 22 Jun 2012, at 17:11, Andy Doan wrote:
On 06/20/2012 09:40 AM, Zach Pfeffer wrote:
On 20 June 2012 09:34, Andy Doanandy.doan@linaro.org wrote:
On 06/20/2012 12:01 AM, Fathi Boudra wrote:
cc LAVA mailing list.
On 20 June 2012 06:19, Zach Pfeffer wrote:
Hey Andy,
During our call with ARM (invite on the way), the fact that LAVA has a network dependency came up again.
Could you expand on the network dependency issue? Filing a bug is even better.
We also have an issue with Connectivity Manager testing. Is there anything that can be done in 12.07 (SD mux) to get over this?
I don't think the SD mux will be available or deployed in 12.07, so we have to think about this another way.
I think the "black box" idea we spoke about at Connect might be the path to solve this. For those who weren't at the bar at midnight in Hong Kong: The "black box" idea is a change to how LAVA would run tests. Instead of having the LAVA dispatcher drive everything host side, we'd boot into a test image, let it do stuff, pull files from a know partition, analyze and report results.
Hehe...
If we were to do that, decisions like networking would be completely up to you(the test image). This concept works best with an SD-mux set up. However, it could work with our master images also.
I still worry 12.07 might be a little ambitious for such a fundamental change, but I think we should discuss.
Yeah, that's cool - though I think to hit our benchmark variance numbers and to test things like Connectivity Manager we'll need this. It should help LAVA to because it'll take a failure point out of the system.
Andy, what are you thinking are the big blockers for this with the master image approach?
-andy
I've put together a rough idea for how this could work. The blueprint is: https://blueprints.launchpad.net/lava-dispatcher/+spec/black-box-test-actions
However, the more interesting piece is the specification: https://wiki.linaro.org/Platform/Validation/Specs/BlackBoxActions
This still needs more thought and planning, but I think its something concrete from which we can start a more targetted dialog.
-andy
In broad principal (and given my unusual absence at the bar at midnight) I think this aligns nicely with the idea that LAVA should just provide the framework into which to run tests, but it does put the onus on test writers to create special tailored images and test runners that conform to a LAVA spec, unless I'm missing something.
Questions:
How does the test start running? Who initiates it, and how? Over a serial connection that LAVA provides? Or as a process that gets auto run on bootup?
I'm going to think more about this over the weekend. As I say, I like the principal.
Thanks
Dave