Hey everyone.
In light of discussion started by bug: https://bugs.launchpad.net/linaro-android/+bug/1022537 and duplicates I think we need to reconsider non-tests build variants.
I'd like to understand how that would work:
0) Which builds need to be split to 'eng' and 'tests' builds? 1) Which images are sent to LAVA for automatic testing? 2) Which images are tested manually? (I don't suppose we'll test both) 3) How do we re-visit the split (do we? how often? on what event?)
My current feeling is that:
0: we need to split all builds and slightly re-design the a-b.l.o front page to offer twin links 1: we use the 'eng' variant for LAVA, for now 2: we use the 'eng' variant for manual tests unless everyone agrees that they can reliably test 'tests' builds 3: we look at this policy every time we {loose,gain} {h/w acceleration,board type} and do a check-list each month during monthly planning.
Feel free to post your ideas and start the discussion. Thanks! ZK
IOn 9 July 2012 09:59, Zygmunt Krynicki zygmunt.krynicki@linaro.org wrote:
Hey everyone.
In light of discussion started by bug: https://bugs.launchpad.net/linaro-android/+bug/1022537 and duplicates I think we need to reconsider non-tests build variants.
I'd like to understand how that would work:
- Which builds need to be split to 'eng' and 'tests' builds?
- Which images are sent to LAVA for automatic testing?
- Which images are tested manually? (I don't suppose we'll test both)
- How do we re-visit the split (do we? how often? on what event?)
My current feeling is that:
0: we need to split all builds and slightly re-design the a-b.l.o front page to offer twin links 1: we use the 'eng' variant for LAVA, for now 2: we use the 'eng' variant for manual tests unless everyone agrees that they can reliably test 'tests' builds 3: we look at this policy every time we {loose,gain} {h/w acceleration,board type} and do a check-list each month during monthly planning.
Feel free to post your ideas and start the discussion. Thanks!
I think the best thing, given our available resources is just to stick with the "tests" builds and to characterize the slowness reported in the bug. Non-accelerated builds should be used in a batch mode anyway since interacting with them is frustrating at best. Perhaps we just call it a day on interactively testing any non-accelerated build, relying on LAVA for this testing.
ZK
-- Zygmunt Krynicki Linaro Validation Team s/Validation/Android/
On 9 July 2012 12:54, Zach Pfeffer zach.pfeffer@linaro.org wrote:
IOn 9 July 2012 09:59, Zygmunt Krynicki zygmunt.krynicki@linaro.org wrote:
Hey everyone.
In light of discussion started by bug: https://bugs.launchpad.net/linaro-android/+bug/1022537 and duplicates I think we need to reconsider non-tests build variants.
I'd like to understand how that would work:
- Which builds need to be split to 'eng' and 'tests' builds?
- Which images are sent to LAVA for automatic testing?
- Which images are tested manually? (I don't suppose we'll test both)
- How do we re-visit the split (do we? how often? on what event?)
My current feeling is that:
0: we need to split all builds and slightly re-design the a-b.l.o front page to offer twin links 1: we use the 'eng' variant for LAVA, for now 2: we use the 'eng' variant for manual tests unless everyone agrees that they can reliably test 'tests' builds 3: we look at this policy every time we {loose,gain} {h/w acceleration,board type} and do a check-list each month during monthly planning.
Feel free to post your ideas and start the discussion. Thanks!
I think the best thing, given our available resources is just to stick with the "tests" builds and to characterize the slowness reported in the bug. Non-accelerated builds should be used in a batch mode anyway since interacting with them is frustrating at best. Perhaps we just call it a day on interactively testing any non-accelerated build, relying on LAVA for this testing.
Thanks for getting the discussion going Zyga. ;)
ZK
-- Zygmunt Krynicki Linaro Validation Team s/Validation/Android/
-- Zach Pfeffer Android Platform Team Lead, Linaro Platform Teams Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
On Mon, Jul 9, 2012 at 10:54 AM, Zach Pfeffer zach.pfeffer@linaro.org wrote:
IOn 9 July 2012 09:59, Zygmunt Krynicki zygmunt.krynicki@linaro.org wrote:
Hey everyone.
In light of discussion started by bug: https://bugs.launchpad.net/linaro-android/+bug/1022537 and duplicates I think we need to reconsider non-tests build variants.
I'd like to understand how that would work:
- Which builds need to be split to 'eng' and 'tests' builds?
- Which images are sent to LAVA for automatic testing?
- Which images are tested manually? (I don't suppose we'll test both)
- How do we re-visit the split (do we? how often? on what event?)
My current feeling is that:
0: we need to split all builds and slightly re-design the a-b.l.o front page to offer twin links 1: we use the 'eng' variant for LAVA, for now 2: we use the 'eng' variant for manual tests unless everyone agrees that they can reliably test 'tests' builds 3: we look at this policy every time we {loose,gain} {h/w acceleration,board type} and do a check-list each month during monthly planning.
Feel free to post your ideas and start the discussion. Thanks!
I think the best thing, given our available resources is just to stick with the "tests" builds and to characterize the slowness reported in
+1
the bug. Non-accelerated builds should be used in a batch mode anyway since interacting with them is frustrating at best. Perhaps we just call it a day on interactively testing any non-accelerated build, relying on LAVA for this testing.
Yes, we want to move more and more to batch mode (aka automation); I don't see this tightly connected to accell vs. non-accell builds though. That goal is valid for both.
Q: Is the "test" build for an accell platform significantly better ?
We discussed on todays call we suspected that the usability issues is at least partly caused by high first boot activity. We said we would like to check if things improve significantly on second boot before making a general decision on this.
W dniu 09.07.2012 20:06, Alexander Sack pisze:
On Mon, Jul 9, 2012 at 10:54 AM, Zach Pfeffer zach.pfeffer@linaro.org wrote:
IOn 9 July 2012 09:59, Zygmunt Krynicki zygmunt.krynicki@linaro.org wrote:
Hey everyone.
In light of discussion started by bug: https://bugs.launchpad.net/linaro-android/+bug/1022537 and duplicates I think we need to reconsider non-tests build variants.
I'd like to understand how that would work:
- Which builds need to be split to 'eng' and 'tests' builds?
- Which images are sent to LAVA for automatic testing?
- Which images are tested manually? (I don't suppose we'll test both)
- How do we re-visit the split (do we? how often? on what event?)
My current feeling is that:
0: we need to split all builds and slightly re-design the a-b.l.o front page to offer twin links 1: we use the 'eng' variant for LAVA, for now 2: we use the 'eng' variant for manual tests unless everyone agrees that they can reliably test 'tests' builds 3: we look at this policy every time we {loose,gain} {h/w acceleration,board type} and do a check-list each month during monthly planning.
Feel free to post your ideas and start the discussion. Thanks!
I think the best thing, given our available resources is just to stick with the "tests" builds and to characterize the slowness reported in
+1
the bug. Non-accelerated builds should be used in a batch mode anyway since interacting with them is frustrating at best. Perhaps we just call it a day on interactively testing any non-accelerated build, relying on LAVA for this testing.
Yes, we want to move more and more to batch mode (aka automation); I don't see this tightly connected to accell vs. non-accell builds though. That goal is valid for both.
Q: Is the "test" build for an accell platform significantly better ?
Interesting question yet I cannot answer at this time (no resources to download or build snowball image). I suspect that the issue is not related to hardware acceleration though but to the resource (memory?) consumption of the application manager that displays all the installed APKs.
Could anyone with snowball or origen please download one of the recent builds and answer Asac's question? Is the board usable on second boot?
Zach: can we somehow pre-boot a board so that the second boot is really the first boot that the user sees?
We discussed on todays call we suspected that the usability issues is at least partly caused by high first boot activity. We said we would like to check if things improve significantly on second boot before making a general decision on this.
Thanks ZK
On 9 July 2012 13:14, Zygmunt Krynicki zygmunt.krynicki@linaro.org wrote:
W dniu 09.07.2012 20:06, Alexander Sack pisze:
On Mon, Jul 9, 2012 at 10:54 AM, Zach Pfeffer zach.pfeffer@linaro.org wrote:
IOn 9 July 2012 09:59, Zygmunt Krynicki zygmunt.krynicki@linaro.org wrote:
Hey everyone.
In light of discussion started by bug: https://bugs.launchpad.net/linaro-android/+bug/1022537 and duplicates I think we need to reconsider non-tests build variants.
I'd like to understand how that would work:
- Which builds need to be split to 'eng' and 'tests' builds?
- Which images are sent to LAVA for automatic testing?
- Which images are tested manually? (I don't suppose we'll test both)
- How do we re-visit the split (do we? how often? on what event?)
My current feeling is that:
0: we need to split all builds and slightly re-design the a-b.l.o front page to offer twin links 1: we use the 'eng' variant for LAVA, for now 2: we use the 'eng' variant for manual tests unless everyone agrees that they can reliably test 'tests' builds 3: we look at this policy every time we {loose,gain} {h/w acceleration,board type} and do a check-list each month during monthly planning.
Feel free to post your ideas and start the discussion. Thanks!
I think the best thing, given our available resources is just to stick with the "tests" builds and to characterize the slowness reported in
+1
the bug. Non-accelerated builds should be used in a batch mode anyway since interacting with them is frustrating at best. Perhaps we just call it a day on interactively testing any non-accelerated build, relying on LAVA for this testing.
Yes, we want to move more and more to batch mode (aka automation); I don't see this tightly connected to accell vs. non-accell builds though. That goal is valid for both.
Q: Is the "test" build for an accell platform significantly better ?
Interesting question yet I cannot answer at this time (no resources to download or build snowball image). I suspect that the issue is not related to hardware acceleration though but to the resource (memory?) consumption of the application manager that displays all the installed APKs.
Could anyone with snowball or origen please download one of the recent builds and answer Asac's question? Is the board usable on second boot?
Zach: can we somehow pre-boot a board so that the second boot is really the first boot that the user sees?
Yeah. Paul's team is doing that as we speak to refine the bug that was filed.
We discussed on todays call we suspected that the usability issues is at least partly caused by high first boot activity. We said we would like to check if things improve significantly on second boot before making a general decision on this.
Thanks
ZK
-- Zygmunt Krynicki Linaro Validation Team s/Validation/Android/
On Mon, 2012-07-09 at 12:54 -0500, Zach Pfeffer wrote:
IOn 9 July 2012 09:59, Zygmunt Krynicki zygmunt.krynicki@linaro.org wrote:
Hey everyone.
In light of discussion started by bug: https://bugs.launchpad.net/linaro-android/+bug/1022537 and duplicates I think we need to reconsider non-tests build variants.
I'd like to understand how that would work:
- Which builds need to be split to 'eng' and 'tests' builds?
- Which images are sent to LAVA for automatic testing?
- Which images are tested manually? (I don't suppose we'll test both)
- How do we re-visit the split (do we? how often? on what event?)
My current feeling is that:
0: we need to split all builds and slightly re-design the a-b.l.o front page to offer twin links 1: we use the 'eng' variant for LAVA, for now 2: we use the 'eng' variant for manual tests unless everyone agrees that they can reliably test 'tests' builds 3: we look at this policy every time we {loose,gain} {h/w acceleration,board type} and do a check-list each month during monthly planning.
Feel free to post your ideas and start the discussion. Thanks!
I think the best thing, given our available resources is just to stick with the "tests" builds and to characterize the slowness reported in the bug. Non-accelerated builds should be used in a batch mode anyway since interacting with them is frustrating at best. Perhaps we just call it a day on interactively testing any non-accelerated build, relying on LAVA for this testing.
So we stop manually testing vexpress?
Also, if we only test 'test' builds but release 'eng' builds, then we aren't testing the thing our customers receive.
On 10 July 2012 05:08, Jon Medhurst (Tixy) tixy@linaro.org wrote:
On Mon, 2012-07-09 at 12:54 -0500, Zach Pfeffer wrote:
IOn 9 July 2012 09:59, Zygmunt Krynicki zygmunt.krynicki@linaro.org wrote:
Hey everyone.
In light of discussion started by bug: https://bugs.launchpad.net/linaro-android/+bug/1022537 and duplicates I think we need to reconsider non-tests build variants.
I'd like to understand how that would work:
- Which builds need to be split to 'eng' and 'tests' builds?
- Which images are sent to LAVA for automatic testing?
- Which images are tested manually? (I don't suppose we'll test both)
- How do we re-visit the split (do we? how often? on what event?)
My current feeling is that:
0: we need to split all builds and slightly re-design the a-b.l.o front page to offer twin links 1: we use the 'eng' variant for LAVA, for now 2: we use the 'eng' variant for manual tests unless everyone agrees that they can reliably test 'tests' builds 3: we look at this policy every time we {loose,gain} {h/w acceleration,board type} and do a check-list each month during monthly planning.
Feel free to post your ideas and start the discussion. Thanks!
I think the best thing, given our available resources is just to stick with the "tests" builds and to characterize the slowness reported in the bug. Non-accelerated builds should be used in a batch mode anyway since interacting with them is frustrating at best. Perhaps we just call it a day on interactively testing any non-accelerated build, relying on LAVA for this testing.
So we stop manually testing vexpress?
No Paul's still testing it.
Also, if we only test 'test' builds but release 'eng' builds, then we aren't testing the thing our customers receive.
No we're releasing test builds instead of eng builds. Apart from the extra APKs there shouldn't be any performance degradation.
-- Tixy
On 9 July 2012 10:54, Zach Pfeffer zach.pfeffer@linaro.org wrote:
I think the best thing, given our available resources is just to stick with the "tests" builds and to characterize the slowness reported in the bug.
Agreed overall, but I think we should have 1 full speed build (only on 1 board - maybe changing which board we support with it every couple of months so members won't get upset) -- so people can see the effect of our optimization work and so we always have something we can demo, and so we have a reasonable view on how much faster than stock AOSP we can be. That 1 build should be faster than even the optimized builds we had before, by also disabling costly debugging and profiling options in the kernel.
ttyl bero
On 14 July 2012 03:46, Bernhard Rosenkränzer bernhard.rosenkranzer@linaro.org wrote:
On 9 July 2012 10:54, Zach Pfeffer zach.pfeffer@linaro.org wrote:
I think the best thing, given our available resources is just to stick with the "tests" builds and to characterize the slowness reported in the bug.
Agreed overall, but I think we should have 1 full speed build (only on 1 board - maybe changing which board we support with it every couple of months so members won't get upset) -- so people can see the effect of our optimization work and so we always have something we can demo, and so we have a reasonable view on how much faster than stock AOSP we can be. That 1 build should be faster than even the optimized builds we had before, by also disabling costly debugging and profiling options in the kernel.
Yeah, that sounds good. I actually think we should be running on Nexus for that build, thoughts?
ttyl bero
linaro-android@lists.linaro.org
-
Alexander Sack -
Bernhard Rosenkränzer -
Jon Medhurst (Tixy) -
Zach Pfeffer -
Zygmunt Krynicki