LAVA and Android Toolchain Benchmarks
andy.doan at linaro.org
Wed Jan 18 23:39:38 UTC 2012
On 01/18/2012 11:24 AM, Zygmunt Krynicki wrote:
> On Wed, Jan 18, 2012 at 5:44 PM, Andy Doan <andy.doan at linaro.org> wrote:
>> On 01/18/2012 10:21 AM, Zygmunt Krynicki wrote:
>>> Again, could you share the code?
>> Okay, here's a drop:
>> The commit message details some of the problems the code currently has.
> Looking at it now
> Some quick comments:
> 1) We should strive to use the database to compute averages/stdev
> 2) We should materialize such changes over time to keep performance
> sensible (scanning everything is going to be too slow)
This was just a proof-of-concept, so things are sloppy. Still, I think
you are right. It will probably make building up representations like
you noted in comment 4 easier.
I haven't fully read into the dashboard_app code. But it seems to have a
concept of storing the bundle and then "analyzing it" which I think
moves it to a database representation. That type of concept might work
well for this.
I think there might even be a chance to simply make the data a "super
bundle" ie combine the bundles into one. With the "measurement" fields
just being the averages. The only other thing I'd want to add to each
measurement is the standard deviation - not sure if that can fit in the
existing data model or not (or if I'm over-thinking this)
> 3) Associating BenchmarkRun with Bundle is odd, why not with TestRun?
A BenchmarkRun contains multiple test runs. For example it may have 4
skia, 4 v8, and 4 0xbench Test Runs. We can then take the averages of
each of type to get better average results. So a BenchmarkRun can be
viewed as one job that was submitted to the validation to run against a
given android build. Then we can compare say a linaro-gcc-4.6 run with
an Android-gcc-4.4 run.
> 4) In benchmark_run.get_summary(jso) you iterate over deserialized
> bundle. You may want to iterate over the database models instead,
> computing sanitized_bundle() is pricy and it was just a hack for the
> dashboard (I should have made it private)
Sure - the original code I had written before this django app was based
on pulling streams from the validation server so it was json based.
> I'm still unsure how the computation model work ATM, I'll post again.
Sure - the code needs some cleaning up and optimizations. Thanks for
More information about the linaro-android