Hey Andy,
During our call with ARM (invite on the way), the fact that LAVA has a network dependency came up again. We also have an issue with Connectivity Manager testing. Is there anything that can be done in 12.07 (SD mux) to get over this?
cc LAVA mailing list.
On 20 June 2012 06:19, Zach Pfeffer wrote:
Hey Andy,
During our call with ARM (invite on the way), the fact that LAVA has a network dependency came up again.
Could you expand on the network dependency issue? Filing a bug is even better.
We also have an issue with Connectivity Manager testing. Is there anything that can be done in 12.07 (SD mux) to get over this?
On 20 June 2012 13:01, Fathi Boudra fathi.boudra@linaro.org wrote:
cc LAVA mailing list.
On 20 June 2012 06:19, Zach Pfeffer wrote:
Hey Andy,
During our call with ARM (invite on the way), the fact that LAVA has a network dependency came up again.
Could you expand on the network dependency issue? Filing a bug is even better.
Yes, could you details the issue with ConnectivityManager testing?
and I have verified with panda that the both the eth and wifi can be enabled at simultaneously.
We also have an issue with Connectivity Manager testing. Is there anything that can be done in
12.07 (SD mux) to get over this?
Thanks,
Yongqin Liu
On 06/20/2012 12:01 AM, Fathi Boudra wrote:
cc LAVA mailing list.
On 20 June 2012 06:19, Zach Pfeffer wrote:
Hey Andy,
During our call with ARM (invite on the way), the fact that LAVA has a network dependency came up again.
Could you expand on the network dependency issue? Filing a bug is even better.
We also have an issue with Connectivity Manager testing. Is there anything that can be done in 12.07 (SD mux) to get over this?
I don't think the SD mux will be available or deployed in 12.07, so we have to think about this another way.
I think the "black box" idea we spoke about at Connect might be the path to solve this. For those who weren't at the bar at midnight in Hong Kong: The "black box" idea is a change to how LAVA would run tests. Instead of having the LAVA dispatcher drive everything host side, we'd boot into a test image, let it do stuff, pull files from a know partition, analyze and report results.
If we were to do that, decisions like networking would be completely up to you(the test image). This concept works best with an SD-mux set up. However, it could work with our master images also.
I still worry 12.07 might be a little ambitious for such a fundamental change, but I think we should discuss.
-andy
On 20 June 2012 09:34, Andy Doan andy.doan@linaro.org wrote:
On 06/20/2012 12:01 AM, Fathi Boudra wrote:
cc LAVA mailing list.
On 20 June 2012 06:19, Zach Pfeffer wrote:
Hey Andy,
During our call with ARM (invite on the way), the fact that LAVA has a network dependency came up again.
Could you expand on the network dependency issue? Filing a bug is even better.
We also have an issue with Connectivity Manager testing. Is there anything that can be done in 12.07 (SD mux) to get over this?
I don't think the SD mux will be available or deployed in 12.07, so we have to think about this another way.
I think the "black box" idea we spoke about at Connect might be the path to solve this. For those who weren't at the bar at midnight in Hong Kong: The "black box" idea is a change to how LAVA would run tests. Instead of having the LAVA dispatcher drive everything host side, we'd boot into a test image, let it do stuff, pull files from a know partition, analyze and report results.
Hehe...
If we were to do that, decisions like networking would be completely up to you(the test image). This concept works best with an SD-mux set up. However, it could work with our master images also.
I still worry 12.07 might be a little ambitious for such a fundamental change, but I think we should discuss.
Yeah, that's cool - though I think to hit our benchmark variance numbers and to test things like Connectivity Manager we'll need this. It should help LAVA to because it'll take a failure point out of the system.
Andy, what are you thinking are the big blockers for this with the master image approach?
-andy
On 06/20/2012 09:40 AM, Zach Pfeffer wrote:
On 20 June 2012 09:34, Andy Doanandy.doan@linaro.org wrote:
On 06/20/2012 12:01 AM, Fathi Boudra wrote:
cc LAVA mailing list.
On 20 June 2012 06:19, Zach Pfeffer wrote:
Hey Andy,
During our call with ARM (invite on the way), the fact that LAVA has a network dependency came up again.
Could you expand on the network dependency issue? Filing a bug is even better.
We also have an issue with Connectivity Manager testing. Is there anything that can be done in 12.07 (SD mux) to get over this?
I don't think the SD mux will be available or deployed in 12.07, so we have to think about this another way.
I think the "black box" idea we spoke about at Connect might be the path to solve this. For those who weren't at the bar at midnight in Hong Kong: The "black box" idea is a change to how LAVA would run tests. Instead of having the LAVA dispatcher drive everything host side, we'd boot into a test image, let it do stuff, pull files from a know partition, analyze and report results.
Hehe...
If we were to do that, decisions like networking would be completely up to you(the test image). This concept works best with an SD-mux set up. However, it could work with our master images also.
I still worry 12.07 might be a little ambitious for such a fundamental change, but I think we should discuss.
Yeah, that's cool - though I think to hit our benchmark variance numbers and to test things like Connectivity Manager we'll need this. It should help LAVA to because it'll take a failure point out of the system.
Andy, what are you thinking are the big blockers for this with the master image approach?
Michael and I had a talk about this earlier. We think we've come up with a way to do this for Android (and a targeted subset of tests) that could be done in a month. The work would also serve as the foundation for extending things further into the total 'black box' concept.
The biggest obstacle to get this done will be finding a person to do the work. Here's the actions I see to kickstart this:
1) Michael or I will draft a BP this week. 2) Zach and I discuss resources
-andy
On 06/20/2012 09:40 AM, Zach Pfeffer wrote:
On 20 June 2012 09:34, Andy Doanandy.doan@linaro.org wrote:
On 06/20/2012 12:01 AM, Fathi Boudra wrote:
cc LAVA mailing list.
On 20 June 2012 06:19, Zach Pfeffer wrote:
Hey Andy,
During our call with ARM (invite on the way), the fact that LAVA has a network dependency came up again.
Could you expand on the network dependency issue? Filing a bug is even better.
We also have an issue with Connectivity Manager testing. Is there anything that can be done in 12.07 (SD mux) to get over this?
I don't think the SD mux will be available or deployed in 12.07, so we have to think about this another way.
I think the "black box" idea we spoke about at Connect might be the path to solve this. For those who weren't at the bar at midnight in Hong Kong: The "black box" idea is a change to how LAVA would run tests. Instead of having the LAVA dispatcher drive everything host side, we'd boot into a test image, let it do stuff, pull files from a know partition, analyze and report results.
Hehe...
If we were to do that, decisions like networking would be completely up to you(the test image). This concept works best with an SD-mux set up. However, it could work with our master images also.
I still worry 12.07 might be a little ambitious for such a fundamental change, but I think we should discuss.
Yeah, that's cool - though I think to hit our benchmark variance numbers and to test things like Connectivity Manager we'll need this. It should help LAVA to because it'll take a failure point out of the system.
Andy, what are you thinking are the big blockers for this with the master image approach?
-andy
I've put together a rough idea for how this could work. The blueprint is:
https://blueprints.launchpad.net/lava-dispatcher/+spec/black-box-test-actions
However, the more interesting piece is the specification: https://wiki.linaro.org/Platform/Validation/Specs/BlackBoxActions
This still needs more thought and planning, but I think its something concrete from which we can start a more targetted dialog.
-andy
Andy Doan andy.doan@linaro.org writes:
On 06/20/2012 09:40 AM, Zach Pfeffer wrote:
Andy, what are you thinking are the big blockers for this with the master image approach?
I've put together a rough idea for how this could work. The blueprint is: https://blueprints.launchpad.net/lava-dispatcher/+spec/black-box-test-actions
However, the more interesting piece is the specification: https://wiki.linaro.org/Platform/Validation/Specs/BlackBoxActions
This still needs more thought and planning, but I think its something concrete from which we can start a more targetted dialog.
I'm not sure describing partitions as "sdb2" and similar is really very useful. We could use partition numbers, but that seems fragile -- maybe just specifying "system" or "data" or whatever would be best.
Otherwise, well, it seems pretty easy really, assuming the android team are willing to do whatever it takes to make the presence of e.g. /etc/indicators/01-wifi Do Stuff on boot...
I guess one thing that seems a little weird about the proposal is that, say, results_directory reads like an instruction to the dispatcher ("put the results here") whereas it's really more of a description of how the image behaves ("the results will end up in this directory, so please look there!"). I'm not sure if this is anything more than the vaguest of concerns though.
Cheers, mwh
On 06/25/2012 10:27 PM, Michael Hudson-Doyle wrote:
Andy Doan andy.doan@linaro.org writes:
On 06/20/2012 09:40 AM, Zach Pfeffer wrote:
Andy, what are you thinking are the big blockers for this with the master image approach?
I've put together a rough idea for how this could work. The blueprint is:
https://blueprints.launchpad.net/lava-dispatcher/+spec/black-box-test-actions
However, the more interesting piece is the specification: https://wiki.linaro.org/Platform/Validation/Specs/BlackBoxActions
This still needs more thought and planning, but I think its something concrete from which we can start a more targetted dialog.
I'm not sure describing partitions as "sdb2" and similar is really very useful. We could use partition numbers, but that seems fragile -- maybe just specifying "system" or "data" or whatever would be best.
I changed this as you suggested. Makes sense to me. Then we just do find the partition via "/dev/disk/by-label/XXX"
Otherwise, well, it seems pretty easy really, assuming the android team are willing to do whatever it takes to make the presence of e.g. /etc/indicators/01-wifi Do Stuff on boot...
Zach - this is where we need your input to make sure you are okay with the logic.
I guess one thing that seems a little weird about the proposal is that, say, results_directory reads like an instruction to the dispatcher ("put the results here") whereas it's really more of a description of how the image behaves ("the results will end up in this directory, so please look there!"). I'm not sure if this is anything more than the vaguest of concerns though.
I'm open to any changes in wording
On 27 June 2012 10:55, Andy Doan andy.doan@linaro.org wrote:
On 06/25/2012 10:27 PM, Michael Hudson-Doyle wrote:
Andy Doan andy.doan@linaro.org writes:
On 06/20/2012 09:40 AM, Zach Pfeffer wrote:
Andy, what are you thinking are the big blockers for this with the master image approach?
I've put together a rough idea for how this could work. The blueprint is:
https://blueprints.launchpad.net/lava-dispatcher/+spec/black-box-test-actions
However, the more interesting piece is the specification: https://wiki.linaro.org/Platform/Validation/Specs/BlackBoxActions
This still needs more thought and planning, but I think its something concrete from which we can start a more targetted dialog.
I'm not sure describing partitions as "sdb2" and similar is really very useful. We could use partition numbers, but that seems fragile -- maybe just specifying "system" or "data" or whatever would be best.
I changed this as you suggested. Makes sense to me. Then we just do find the partition via "/dev/disk/by-label/XXX"
Otherwise, well, it seems pretty easy really, assuming the android team are willing to do whatever it takes to make the presence of e.g. /etc/indicators/01-wifi Do Stuff on boot...
Zach - this is where we need your input to make sure you are okay with the logic.
Why not just make the /system partition writeable and write all the results there? That should always be present and LAVA can read the test results there.
I guess one thing that seems a little weird about the proposal is that, say, results_directory reads like an instruction to the dispatcher ("put the results here") whereas it's really more of a description of how the image behaves ("the results will end up in this directory, so please look there!"). I'm not sure if this is anything more than the vaguest of concerns though.
I'm open to any changes in wording
On 06/27/2012 11:16 AM, Zach Pfeffer wrote:
Why not just make the /system partition writeable and write all the results there? That should always be present and LAVA can read the test results there.
That works with what we've described. Your action would just have something like:
"results_partition": "system", "results_directory": "lava_results",
I'd rather keep it explicit, that way if things do ever change, its a configuration knob and not a code update and deployment.
On 27 June 2012 12:03, Andy Doan andy.doan@linaro.org wrote:
On 06/27/2012 11:16 AM, Zach Pfeffer wrote:
Why not just make the /system partition writeable and write all the results there? That should always be present and LAVA can read the test results there.
That works with what we've described. Your action would just have something like:
"results_partition": "system", "results_directory": "lava_results",
I'm cool with that. 0xbench does the same thing.
I'd rather keep it explicit, that way if things do ever change, its a configuration knob and not a code update and deployment.
On 22 Jun 2012, at 17:11, Andy Doan wrote:
On 06/20/2012 09:40 AM, Zach Pfeffer wrote:
On 20 June 2012 09:34, Andy Doanandy.doan@linaro.org wrote:
On 06/20/2012 12:01 AM, Fathi Boudra wrote:
cc LAVA mailing list.
On 20 June 2012 06:19, Zach Pfeffer wrote:
Hey Andy,
During our call with ARM (invite on the way), the fact that LAVA has a network dependency came up again.
Could you expand on the network dependency issue? Filing a bug is even better.
We also have an issue with Connectivity Manager testing. Is there anything that can be done in 12.07 (SD mux) to get over this?
I don't think the SD mux will be available or deployed in 12.07, so we have to think about this another way.
I think the "black box" idea we spoke about at Connect might be the path to solve this. For those who weren't at the bar at midnight in Hong Kong: The "black box" idea is a change to how LAVA would run tests. Instead of having the LAVA dispatcher drive everything host side, we'd boot into a test image, let it do stuff, pull files from a know partition, analyze and report results.
Hehe...
If we were to do that, decisions like networking would be completely up to you(the test image). This concept works best with an SD-mux set up. However, it could work with our master images also.
I still worry 12.07 might be a little ambitious for such a fundamental change, but I think we should discuss.
Yeah, that's cool - though I think to hit our benchmark variance numbers and to test things like Connectivity Manager we'll need this. It should help LAVA to because it'll take a failure point out of the system.
Andy, what are you thinking are the big blockers for this with the master image approach?
-andy
I've put together a rough idea for how this could work. The blueprint is: https://blueprints.launchpad.net/lava-dispatcher/+spec/black-box-test-actions
However, the more interesting piece is the specification: https://wiki.linaro.org/Platform/Validation/Specs/BlackBoxActions
This still needs more thought and planning, but I think its something concrete from which we can start a more targetted dialog.
-andy
In broad principal (and given my unusual absence at the bar at midnight) I think this aligns nicely with the idea that LAVA should just provide the framework into which to run tests, but it does put the onus on test writers to create special tailored images and test runners that conform to a LAVA spec, unless I'm missing something.
Questions:
How does the test start running? Who initiates it, and how? Over a serial connection that LAVA provides? Or as a process that gets auto run on bootup?
I'm going to think more about this over the weekend. As I say, I like the principal.
Thanks
Dave
On 06/30/2012 03:53 AM, Dave Pigott wrote:
Questions:
How does the test start running?
We really put that burden on Android (or some day Ubuntu). The "indicators" we create on the filesystem are supposed to instruct them what expect them to test. As per the mechanics of it all from LAVA, we'll just boot the board into the test image, and then wait for a timeout or a completion signal (via serial line) that the test is done.
Who initiates it, and how?
The test job JSON will tell LAVA via these new actions. We'll then set up the proper "indicators", boot to the test image, etc.
Over a serial connection that LAVA provides?
LAVA would now only communicate via serial to the test image, and the data we communicate will be pretty minimal.
Or as a process that gets auto run on bootup?
I think my other answers explain this, but it will be up to the test image itself to understand what it needs to do at start up.
On 12-06-20 10:34 AM, Andy Doan wrote:
On 06/20/2012 12:01 AM, Fathi Boudra wrote:
cc LAVA mailing list.
On 20 June 2012 06:19, Zach Pfeffer wrote:
Hey Andy,
During our call with ARM (invite on the way), the fact that LAVA has a network dependency came up again.
Could you expand on the network dependency issue? Filing a bug is even better.
We also have an issue with Connectivity Manager testing. Is there anything that can be done in 12.07 (SD mux) to get over this?
I don't think the SD mux will be available or deployed in 12.07, so we have to think about this another way.
I think the "black box" idea we spoke about at Connect might be the path to solve this. For those who weren't at the bar at midnight in Hong Kong: The "black box" idea is a change to how LAVA would run tests. Instead of having the LAVA dispatcher drive everything host side, we'd boot into a test image, let it do stuff, pull files from a know partition, analyze and report results.
If we were to do that, decisions like networking would be completely up to you(the test image). This concept works best with an SD-mux set up. However, it could work with our master images also.
I still worry 12.07 might be a little ambitious for such a fundamental change, but I think we should discuss.
Agreed, it is a little too ambitious. Might be better to add it as a second mode of operation LAVA supports, and indicate the mode in the job JSON.
Scott
-andy
linaro-android mailing list linaro-android@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-android
On 06/20/2012 01:59 PM, Scott Bambrough wrote:
Agreed, it is a little too ambitious. Might be better to add it as a second mode of operation LAVA supports, and indicate the mode in the job JSON.
I probably wasn't clear about one thing: This would definitely be a new mode for LAVA that people would have to opt-in for when defining their jobs.
If I had to guess, I'd say that the network dependency issue ARM is talking about is the fact that we install an image over the wire, and that they probably want to do bare metal testing on systems that may not even have network support at all. I know that they've mentioned such things in the past, and the response has basically been that we chose this method of deployment based on the boards we care about testing. If they need to deploy things in a different way (usb, jtag, carrier pigeons...) a different deployment handler would need to be written to cover that.
On Wed, Jun 20, 2012 at 2:12 PM, Andy Doan andy.doan@linaro.org wrote:
On 06/20/2012 01:59 PM, Scott Bambrough wrote:
Agreed, it is a little too ambitious. Might be better to add it as a second mode of operation LAVA supports, and indicate the mode in the job JSON.
I probably wasn't clear about one thing: This would definitely be a new mode for LAVA that people would have to opt-in for when defining their jobs.
______________________________**_________________ linaro-validation mailing list linaro-validation@lists.**linaro.org linaro-validation@lists.linaro.org http://lists.linaro.org/**mailman/listinfo/linaro-**validationhttp://lists.linaro.org/mailman/listinfo/linaro-validation
On 06/20/2012 02:50 PM, Paul Larson wrote:
If I had to guess, I'd say that the network dependency issue ARM is talking about is the fact that we install an image over the wire, and that they probably want to do bare metal testing on systems that may not even have network support at all.
I suspect it could be a bit of this and a bit of what Zach is saying. Benchmark guys work very hard to make sure the system has as little unpredictable disruptions as possible. Having an active network service doesn't help their cause.
On 21 June 2012 10:39, Andy Doan andy.doan@linaro.org wrote:
On 06/20/2012 02:50 PM, Paul Larson wrote:
If I had to guess, I'd say that the network dependency issue ARM is talking about is the fact that we install an image over the wire, and that they probably want to do bare metal testing on systems that may not even have network support at all.
I suspect it could be a bit of this and a bit of what Zach is saying. Benchmark guys work very hard to make sure the system has as little unpredictable disruptions as possible. Having an active network service doesn't help their cause.
Yes its kind of both. We would want : 1.Avoid network connectivity in few cases which will help in reducing the variance. (which may not be possible in LAVA) 2.Have Wifi connectivity on the device since few third party benchmarking apps won't run without wifi connectivity.
______________________________**_________________ linaro-android mailing list linaro-android@lists.linaro.**org linaro-android@lists.linaro.org http://lists.linaro.org/**mailman/listinfo/linaro-**androidhttp://lists.linaro.org/mailman/listinfo/linaro-android
On Wed, Jun 20, 2012 at 9:50 PM, Paul Larson paul.larson@linaro.org wrote:
If I had to guess, I'd say that the network dependency issue ARM is talking about is the fact that we install an image over the wire, and that they probably want to do bare metal testing on systems that may not even have network support at all. I know that they've mentioned such things in the past, and the response has basically been that we chose this method of deployment based on the boards we care about testing. If they need to deploy things in a different way (usb, jtag, carrier pigeons...) a different deployment handler would need to be written to cover that.
Yes, we need networking for master image, but for android we also need it for the TARGET (adb) ... thats a pretty high requirement for the market we aim at :). So yes, LAVAv2 will (have to) cover blackbox testing and that is part of phase 1 or 2 of it which could mean Q3 or Q4 according to the roadmap I currently have in my mind.
linaro-android@lists.linaro.org
-
Alexander Sack -
Andy Doan -
Dave Pigott -
Fathi Boudra -
Michael Hudson-Doyle -
Paul Larson -
Scott Bambrough -
Vishal Bhoj -
YongQin Liu -
Zach Pfeffer