The parsing of test output is done by LKP, LKP save metrics to json files, our test definition decode the json file and send them to LAVA. If we want to have all the sub-metrics, I guess patch the LKP test suite and send it to upstream is the right way to go. IHMO, it can be done, but not at this stage.
The data from 'time' and 'perf' also will be saved by LKP. I think the 'avg.json' is the right file I should parse, it included metrics of the benchmark and time and perf. I added a 'LOOPS' parameter in test definition to support repeatedly run. If we run the test more then once, the data in avg.json will be the average of the runs. Here is a lava job example https://validation.linaro.org/scheduler/job/382401
Hopefully, it is what we need. Would you please check and let me know your opinion?
Thanks, Chase
On 10 June 2015 at 19:41, Alex Shi alex.shi@linaro.org wrote:
On 06/10/2015 06:53 PM, Mark Brown wrote:
On 10 June 2015 at 08:04, Alex Shi <alex.shi@linaro.org mailto:alex.shi@linaro.org> wrote:
2, performance testing often has variational results, that normally requests repeat testing and collect the average, standard deviations etc index of results. We need to collect repeatly running data and to decide how many times re-run needed.I wonder if this should be done at a level up from the test definition for a given test itself, or perhaps a library that the test definitions can use - like you say this seems like something that's going to be needed for many tests (though for some the test will already have its own implementation).
If my memory correct, the LKP should have this function in test script. We just need to tune it for each of benchmark and each different boards.
4, perf tool output is very important for kernel developer to know why we got this performance data, where need to improve, that is same important as test results. So we'd better to figure out how many perf data we can get from testing, and collect them.This might be something else that could be shared.
Yes, like other kind of profile tools etc.