Comments inline...
On Fri, Sep 15, 2017 at 9:18 AM, Tom Gall tom.gall@linaro.org wrote:
Hi Todd,
I want to dive into slightly more detail here as I think this is important. Some of this might be repetitive so forgive me for that but I think we're probably due for a sync so I guess this note might help serve as that.
For VTS we aren't running the HAL tests nor the benchmarks. Is this still the correct strategy?
My preference would be to run everything in VTS unless there is a reason why a particular test can't run on Hikey (or whatever device is being used). What was the rationale for not running these tests? Capacity?
For CTS, we've been running a subset.
4.4 Ex: https://qa-reports.linaro.org/lkft/android-hikey-linaro-4.4-aosp/build/v4.4.... 4.9 Ex: https://qa-reports.linaro.org/lkft/android-hikey-linaro-4.9-aosp/build/v4.9....
I have a couple of the LMG guys working on full CTS. That's in progress and we're starting to run those test jobs, establish a baseline, fix bugs that are being found etc as Anmar noted. When we run CTS this way we're sharding it so it can span multiple boards and thus decrease the time before we have results. Here's an example run of one of the shards https://lkft.validation.linaro.org/scheduler/job/29416#bottom
All that said, as I recall in the past, there had been discussion on CTS tests to run. Looking at it from a strategic point of view, do we want to go forward with running a full CTS as part of LKFT or is the current subset what we want to use for LKFT going forward?
Thoughts?
To me the value in running CTS is just to give downstream partners confidence that there is no regression in android due to changes in the kernel -- however, I don't think CTS really exercises the kernel in a meaningful way, so a thoughtfully selected subset is fine. Again, if we have capacity, it would be nice to test as much as possible.
How was the current subset selected?
Should we meet at connect and discuss?
-Todd