On Wed, Jan 18, 2012 at 5:07 PM, Andy Doan andy.doan@linaro.org wrote:
On 01/18/2012 05:25 AM, Alexander Sack wrote:
On Wed, Jan 18, 2012 at 12:16 PM, Zygmunt Krynicki zygmunt.krynicki@linaro.org wrote:
Hi, looks nice :)
On Wed, Jan 18, 2012 at 5:59 AM, Andy Doan andy.doan@linaro.org wrote:
Sorry for the wide distribution, but I wasn't sure who all would be interested.
I spent time over the last month updating the Android monthly toolchain benchmark process[1] to pull its benchmark data from LAVA tests that are stored in validation.linaro.org. Here's an example test run[2].
This month's results will be published to the wiki as I normally do. However, I spent some time last weekend looking at how to handle this on the validation server as well. I first toyed with trying to do a simple report plugin. However, it really didn't quite have everything I thought was needed.
I wound up using the "LAVA kernel CI views" project as a skeleton to create something for Android. I've got a local prototype that's starting to do just about everything I want (I'm fighting some issues the the javascript FLOT library for my charts). I'm attaching a screenshot so you can get a rough idea.
Before I really invest time, I wanted to get people's thoughts. Some big questions for me:
- Is anyone against doing this?
That's a question to TSC (regarding benchmark data)
The data policy doesn't block doing it. Worst case, it might block publishing such view unmodified to the public. But let's look at what we want to do first and then discuss the implications of the data policy.
There might be another way to look at the code I'm doing. While it has a very specific title right now "Android Toolchain Benchmark Report". Its actually quite generic and might be useful for other things. In essence, it does two things:
1) Collate measurements from multiple tests. Reduce these down to one set of data with average measurements. I also include standard deviation in there, so you can get an idea of the quality of the data.
2) Take multiple "combined results" and compare them with each other.
Excellent. I think we should keep evolving this (and keep it separate from lava-android I did). Again, could you share the code?
ZK