On Mon, Mar 5, 2012 at 11:12 AM, Andy Doan <andy.doan@linaro.org> wrote:

On 03/02/2012 11:50 PM, Paul Larson wrote:

On Thu, Mar 1, 2012 at 4:54 PM, Andy Doan <andy.doan@linaro.org
<mailto:andy.doan@linaro.org>> wrote:

This email spans 3 areas and I wasn't sure who'd be interested. Feel
free to add people if you'd like.

I just spent the day getting the coremark benchmark to run under
lava-android-test and wanted to share my findings and methodology.

Neat!

The key obstacle is that coremark requires a registered download. ie
- you can't just throw this in the Android build and let the world
use it. This is what makes the whole process trickier than normal
"androidify + lavaitize" something.

Well, hopefully not too much. A couple of observations here. First off,
feel free to tell the LAVA team that you have a tarball you want to host
for download into lava, inside the lava infrastructure so that it will
be available to LAVA, but not outside. We have a similar situation with
the binary drivers needed for panda builds. The scripts that the
Android team put together were useful for getting started, but certainly
not something we'd want to work with since it required interaction.
What we ended up doing was building a tarball of everything it
installed, and hosting it inside the lab. We could do a similar thing
with coremark so that it's available for lava jobs, but not for public
download.

We could merge something like this in lava-android-test, but if so, we
should include instructions in case anyone outside linaro wants to use
it. To use it, they would have to host coremark themselves since we
can't provide it. Another option is to do it out of tree, but while
lava-test supports out-of-tree tests, it so far hasn't been a priority
to push this feature in to lava-android-test since the current push from
Zach's team has been to include all their tests. I've expected that
there will eventually be some things like this where that's not really
possible though, and we have a blueprint to add out-of-tree support to
lava-android-test. If this is a model you would need, just let me know
and I can bump the priority on that.

My current code was done for maximum flexibility. It could be checked into lava-android-test, or we could keep it out-of-tree to avoid confusion for people who'd think they could easily run it. I'm leaning to out of tree, but don't really care.

The question I'm thinking about now is about the job submission to LAVA. I'm thinking of an approach where we create a new dispatcher command that would be semi-generic for all restricted tests and look like:

{
"command": "lava_android_test_install_restricted",
"parameters": {
"tests": ["coremark"],
"install_options": ["coremark_url=http://user:pass@1.1.1/coremark-linaro4.6"]
}
},

Why would it need a _restricted version of the command? I don't think that's necessary.

I have one security concern with this approach: Do we enforce any type of security about who can view the job definitions in LAVA? ie - if anyone could see the job definition then they'd be able to see how they could download the restricted file.

I don't think that's a problem. In fact, what I'm suggesting is to simply have it available as a download from our NAS system in the lab, that is behind the firewall and not exposed. You'll need to get us to put the file there, but then it wouldn't need to be behind yet another layer of security.

In fact, as I'm thinking about this, I don't even see a need for an out-of-tree test support in lava-android-test. The out-of-tree support is needed in lava-test because it's getting reinstalled every time. With lava-android-test, it could simply be a python module that we install one time and update as needed, then that test would be available on our server. If anyone else wanted to use it, they could, but would need to specify their own location for getting the coremark binaries.

Thanks,
Paul Larson