Hi EAS-DEV,
We are fine tuning EAS and DVFS for our product, and use WA to do auto test. In the presentation, http://connect.linaro.org.s3.amazonaws.com/hkg18/presentations/hkg18-210.pdf, it lists Test cases at Page 9 & 10. Except Vellamo, can all the following test cases be verified via WA? In addition, there are some questions to consult.
1) Temple run and Beach buggy blitz are games, how to evaluate the performance?
2) Is there any suitable benchmark for UIbench & Homescreen?
3) Currently, audio and video are real-time playback, how to check the improvement?
[cid:image001.png@01D3D58F.7F44EEE0] [cid:image002.png@01D3D58F.7F44EEE0]
BR, Erin
Hi Erin,
On Mon, Apr 16, 2018 at 06:50:12AM +0000, Erin Yang (楊雅婷) wrote:
Hi EAS-DEV,
We are fine tuning EAS and DVFS for our product, and use WA to do auto test. In the presentation, http://connect.linaro.org.s3.amazonaws.com/hkg18/presentations/hkg18-210.pdf, it lists Test cases at Page 9 & 10. Except Vellamo, can all the following test cases be verified via WA?
AFAIK, almost all testing cases can be verified with WA.
Except WA, I suggest you could take a look for LISA's wltest feature, which integrate LISA+WA, and it's friendly for results comparison within multiple runs [1].
In addition, there are some questions to consult.
Temple run and Beach buggy blitz are games, how to evaluate the performance?
AFAIK, We can use WA FPS as performance instrument for gaming [2].
Furthermore, sometimes we also can do statistics based on Janks for performance evaluation [3].
Is there any suitable benchmark for UIbench & Homescreen?
For UI benchmark, there have two programs:
- UIBench: which you could build by self in AOSP repo; - Jankbench: This is developed by Google team but I don't know if this APK has been released or not.
For Homescreen, we usually only measure the power data and for more detailed analysis, we can check the task placement on big/LITTLE CPUs.
Currently, audio and video are real-time playback, how to check the improvement?
Audio/Video are power sustainable scenarios, the system need to meet these two scenarios perofrmance requirement, e.g. audio should check overrun/underrun issue, and video can check FPS (be honest, I don't measure these metrics), we need to optimize the power data as possible, so WA can support energy instruments, in our case we are using ARM energy probe [4].
Thanks, Leo Yan
[1] https://github.com/ARM-software/lisa/blob/master/tools/wltests/README.md [2] https://pythonhosted.org/wlauto/extensions/instruments.html#fps [3] https://github.com/ARM-software/lisa/blob/master/tools/wa_user_directory/plu... [4] https://pythonhosted.org/wlauto/extensions/instruments.html#energy-probe
Hi,
I added a few details inline but Leo covered most of it.
On 16/04/18 09:34, Leo Yan wrote:
Hi Erin,
On Mon, Apr 16, 2018 at 06:50:12AM +0000, Erin Yang (楊雅婷) wrote:
Hi EAS-DEV,
We are fine tuning EAS and DVFS for our product, and use WA to do auto test. In the presentation, http://connect.linaro.org.s3.amazonaws.com/hkg18/presentations/hkg18-210.pdf, it lists Test cases at Page 9 & 10. Except Vellamo, can all the following test cases be verified via WA?
AFAIK, almost all testing cases can be verified with WA.
Except WA, I suggest you could take a look for LISA's wltest feature, which integrate LISA+WA, and it's friendly for results comparison within multiple runs [1].
To avoid any confusion, wltest uses WA3, which is the "next" branch of WA: https://github.com/ARM-software/workload-automation/tree/next
In addition, there are some questions to consult.
Temple run and Beach buggy blitz are games, how to evaluate the performance?AFAIK, We can use WA FPS as performance instrument for gaming [2].
Furthermore, sometimes we also can do statistics based on Janks for performance evaluation [3].
Is there any suitable benchmark for UIbench & Homescreen?For UI benchmark, there have two programs:
- UIBench: which you could build by self in AOSP repo;
- Jankbench: This is developed by Google team but I don't know if this APK has been released or not.
For Homescreen, we usually only measure the power data and for more detailed analysis, we can check the task placement on big/LITTLE CPUs.
Currently, audio and video are real-time playback, how to check the improvement?Audio/Video are power sustainable scenarios, the system need to meet these two scenarios perofrmance requirement, e.g. audio should check overrun/underrun issue, and video can check FPS (be honest, I don't measure these metrics), we need to optimize the power data as possible,
You can also have a look at the wltest notebook to see which statistic we are interested in:
https://github.com/ARM-software/lisa/blob/master/ipynb/wltests/sched-evaluat...
AFAIK we don't have anything for exoplayer audio that tries to validate the audio playback was ok from a user perspective. As for exoplayer video, we look at the number of dropped frames which is a good indicator of how well the video playback went.
so WA can support energy instruments, in our case we are using ARM energy probe [4].
We recommend using the Baylibre ACME cape as it's more reliable and more accurate. Just like the AEP it's supported in both LISA & WA:
https://github.com/ARM-software/lisa/wiki/Energy-Meters-Requirements#iiocapt...
Thanks, Leo Yan
[1] https://github.com/ARM-software/lisa/blob/master/tools/wltests/README.md [2] https://pythonhosted.org/wlauto/extensions/instruments.html#fps [3] https://github.com/ARM-software/lisa/blob/master/tools/wa_user_directory/plu... [4] https://pythonhosted.org/wlauto/extensions/instruments.html#energy-probe
Best regards, Valentin
-
Erin Yang (楊雅婷) -
Leo Yan -
Valentin Schneider