Looping in the linaro-validation mailing list. Nothing private here and would like to get input from others as well.
On Mon, Mar 5, 2012 at 11:12 AM, Andy Doan andy.doan@linaro.org wrote:
On 03/02/2012 11:50 PM, Paul Larson wrote:
On Thu, Mar 1, 2012 at 4:54 PM, Andy Doan <andy.doan@linaro.org mailto:andy.doan@linaro.org> wrote:
This email spans 3 areas and I wasn't sure who'd be interested. Feel free to add people if you'd like.
I just spent the day getting the coremark benchmark to run under lava-android-test and wanted to share my findings and methodology.
Neat!
The key obstacle is that coremark requires a registered download. ie
- you can't just throw this in the Android build and let the world
use it. This is what makes the whole process trickier than normal "androidify + lavaitize" something.
Well, hopefully not too much. A couple of observations here. First off, feel free to tell the LAVA team that you have a tarball you want to host for download into lava, inside the lava infrastructure so that it will be available to LAVA, but not outside. We have a similar situation with the binary drivers needed for panda builds. The scripts that the Android team put together were useful for getting started, but certainly not something we'd want to work with since it required interaction. What we ended up doing was building a tarball of everything it installed, and hosting it inside the lab. We could do a similar thing with coremark so that it's available for lava jobs, but not for public download.
We could merge something like this in lava-android-test, but if so, we should include instructions in case anyone outside linaro wants to use it. To use it, they would have to host coremark themselves since we can't provide it. Another option is to do it out of tree, but while lava-test supports out-of-tree tests, it so far hasn't been a priority to push this feature in to lava-android-test since the current push from Zach's team has been to include all their tests. I've expected that there will eventually be some things like this where that's not really possible though, and we have a blueprint to add out-of-tree support to lava-android-test. If this is a model you would need, just let me know and I can bump the priority on that.
My current code was done for maximum flexibility. It could be checked into lava-android-test, or we could keep it out-of-tree to avoid confusion for people who'd think they could easily run it. I'm leaning to out of tree, but don't really care.
The question I'm thinking about now is about the job submission to LAVA. I'm thinking of an approach where we create a new dispatcher command that would be semi-generic for all restricted tests and look like:
{ "command": "lava_android_test_install_**restricted", "parameters": { "tests": ["coremark"], "install_options": ["coremark_url=http://user:**pass@1.1.1 /coremark-linaro4.6"**] } },
Why would it need a _restricted version of the command? I don't think
that's necessary.
I have one security concern with this approach: Do we enforce any type of security about who can view the job definitions in LAVA? ie - if anyone could see the job definition then they'd be able to see how they could download the restricted file.
I don't think that's a problem. In fact, what I'm suggesting is to simply have it available as a download from our NAS system in the lab, that is behind the firewall and not exposed. You'll need to get us to put the file there, but then it wouldn't need to be behind yet another layer of security.
In fact, as I'm thinking about this, I don't even see a need for an out-of-tree test support in lava-android-test. The out-of-tree support is needed in lava-test because it's getting reinstalled every time. With lava-android-test, it could simply be a python module that we install one time and update as needed, then that test would be available on our server. If anyone else wanted to use it, they could, but would need to specify their own location for getting the coremark binaries.
Thanks, Paul Larson