On 08/17/2011 04:59 PM, Michael Hope wrote:
On Wed, Aug 17, 2011 at 11:12 PM, Dave Martin dave.martin@linaro.org wrote:
On Tue, Aug 16, 2011 at 7:14 PM, Zach Pfeffer zach.pfeffer@linaro.org wrote:
Nicolas,
Thanks for the notes. As you say there are many, many things that can affect this demo. What notes like this really underscore is the importance of staying up-to-date. This demo is more about the macroscopic effects from tip support than anything else. We do have some more specific benchmark numbers at:
https://wiki.linaro.org/Platform/Android/AndroidToolchainBenchmarking
If we're confident that the benchmark produces results of a trustworthy quality, then that's fine. I don't know this benchmark in detail, so I can't really judge, other than that the results look a bit odd.
Ditto on that. Have these benchmarks been qualified? Do they represent real workloads? Where do they come from? What aspects of the system (CPU, memory, I/O, kernel, SMP) do they exercise? How sensitive are they to minor changes?
The benchmark code comes from Android: http://android.git.kernel.org/?p=toolchain/benchmark.git
I'm not an expert on benchmarking. I've just tried to focus on running these in a way that's as fair and repeatable as possible.
gnugo in particular is a problem - the results don't change across a range of toolchains which suggests it's got a silly hot loop or isn't core bound.