On 11 June 2015 at 16:31, Alex Shi alex.shi@linaro.org wrote:
On 06/11/2015 03:26 PM, Milosz Wasilewski wrote:
On 11 June 2015 at 08:17, Alex Shi alex.shi@linaro.org wrote:
On 06/11/2015 11:55 AM, Chase Qi wrote:
The parsing of test output is done by LKP, LKP save metrics to json files, our test definition decode the json file and send them to LAVA. If we want to have all the sub-metrics, I guess patch the LKP test suite and send it to upstream is the right way to go. IHMO, it can be done, but not at this stage.
Maybe the upstream LKP don't want our specific parse for LAVA. We probably need to handle them by ourself. and if the test output can not be show out clear/appropriately, it willn't so helpful for us.
There is nothing LAVA specific there. Chase is using LKP output only and LKP doesn't save the table you presented in any way. So if we want to have the data, LKP needs to be patched.
Seems there are some misunderstanding here. I didn't mean we don't need a patch for parse. That needed. I just don't know if LKP upstream like to pick up this 'json file decode' script.
The data from 'time' and 'perf' also will be saved by LKP. I think the 'avg.json' is the right file I should parse, it included metrics of the benchmark and time and perf. I added a 'LOOPS' parameter in test definition to support repeatedly run. If we run the test more then once, the data in avg.json will be the average of the runs. Here is a lava job example https://validation.linaro.org/scheduler/job/382401
It is hard to figure out something useful from this links. https://validation.linaro.org/dashboard/streams/anonymous/chase-qi/bundles/8...
seems it doesn't work now. Could you like to resend the report when everything right.
It does work, here are detailed results: https://validation.linaro.org/dashboard/streams/anonymous/chase-qi/bundles/8...
Sorry for miss this.
For this results show, it is still organized as functional testing results. and mixed the profile data with benchmark data and even with 'split job', 'setup local dbench' etc setup step as benchmarks.
We'd better to split out our targets that just benchmark data. Also we care about the measurement value instead of 'pass' or 'fail' of benchmarks.
Alex,
In LAVA, if test failed, or result is missing, we want to know what was wrong, so we design the case to check test result of each step. Even I remove this check points, "lava-test-shell-install" and "lava-test-shell-run" are produced by LAVA for the same purpose, they exist for all test. IMO, it is a feature, not a trouble.
If we run test in the same lava-test-shell, then all test results will be saved to the same test run. I don't think we can categorize them at the moment.
The following format would be better than current. | kernel 1| kernel 2| |benchmark 1| value x | value x2| |benchmark 2| value y | value y2|
And further steps, we'd better set up an auto compare function to tracking if some measurement has regression on new kernel version. At that time, it worth to look into for details.
Alex, we're not kernel hackers and we don't know what's important and what is not.
I knew this, that is why I explain what's important or useful for kernel engineers.
I run the dbench 3 times and attached the test log from LKP, please check the json files inside and let me know what are important for you. If the numbers you are looking for not included in the json files, then we need patch LKP to pick up the numbers. If we care the numbers, I guess LKP should care too, otherwise, why we care?
Chase is asking for help identifying the important bits. Complaining that what we present is not what you want without details doesn't help :(
I am sorry, if the feature request looks just complains. I do appreciate what Riku and Chase did on this job!
Guess we should have the same goal, that is making the performance testing useful and reliable for kernel engineers in linaro. Not sth we made it in a hurry, but no one like using it, since it hard to get details and missed useful info.
I think we can get it run first, then improve it step by step. “You can't get fat from eating a single mouthful” goes a saying in China.
Thanks, Chase
milosz
Hopefully, it is what we need. Would you please check and let me know your opinion?