On 12-05-22 09:10 AM, Paul Sokolovsky wrote:
Hello,
On Wed, 16 May 2012 02:27:06 +0200 Alexander Sack asac@linaro.org wrote:
On Wed, May 16, 2012 at 1:28 AM, Mathieu Poirier mathieu.poirier@linaro.org wrote:
Gents,
I've had this error twice in a row now while compiling Android on jenkins:
Build timed out (after 150 minutes). Marking the build as aborted.
The thing is, the build was still alive and in the last steps of packaging the system partition. Can we increase the limit to 200 minutes ?
Nothing against timeout bump, maybe we can try 170?
Any idea why we see this slowdown? Did you add something significant to recent snowball builds?
Well, the were no response from Android folks, so I assume no changes were done on the code side. I bumped timeout to 170mins the same day, but was shown at least 2 builds still timed out. We now have more builds to do stats, and I looked into that with more detail.
This shows it very well:
https://android-build.linaro.org/jenkins/job/doanac_tracking-panda-linaro4.6...
#9 - 2h51m, timed out #10, few days later - 2h01m, among the best of our times
I looked also in build logs for details stats to see if some specific stage (like seed update) is responsible for slowdown, and no, it's just roughly proportional slowdown for all stages (more details are at https://bugs.launchpad.net/linaro-android-infrastructure/+bug/997551). So this can be attributed only to EC2 disturbances - we had "bad week" when instances underperformed, leading to timeouts. Now, it's back to normal. We have similar "bad weeks" wrt to hanged builds, where that happens twice a day vs 1-2 a week as usual.
So, I bumped timeout to 3hrs (180mins), which should account for worst cases known so far.
Hey Paul,
Thanks for that - on my side I haven't experienced a timeout since you bumped the limit to 170mins.
Regards, Mathieu.