Hello,
On Tue, 16 Apr 2013 06:19:51 +0530 Vishal Bhoj vishal.bhoj@linaro.org wrote:
[]
This error is related to infrastructure. I am not sure if this will be resolved if the publishing is updated.
ChannelClosedException as quoted below means an EC2 instance got terminated (or otherwise "lost") behind Jenkins' back. Generally, this issue is a known non-deterministic failure issue, and bound to happen from time to time due to the nature of EC2 (complex big system, has non-zero stream of errors).
We really need to find a solution for this. We are running into this error on a regular basis nowadays.
Yes, I see that 2 builds I tried yesterday didn't succeed either, with the same error. Builds of that job look weird, because they're still in compile phase after 3.5hrs after the start - that's too long. At last 2 builds were killed at almost the same time, but it's not build timeout (set at 4:45mins, looks differently), not EC2 monitoring script (not active now, never killed running builds, only zombie instances).
I'm cc:ing Phillip just in case if he may know of anything which may kill EC2 in the "old" EC2 Linaro account in 3.5hrs?
I still think the likely cause though is master overload due to publishing issues, and would like to keep working on resolving that first. I have good results so far - "copycat" build on a sandbox finished with less than 2min: https://ec2-107-20-93-222.compute-1.amazonaws.com/jenkins/job/pfalcon_galaxy...
That's 1/2 of all work needed tho, going to deploy needed parts on production and continue with it.
The fact that it happened 3 times is worrying though. We appear to have had another zombie slave storm over weekend, and on top of that, we have https://bugs.launchpad.net/linaro-android-infrastructure/+bug/1164273 , which causes builds to prolong too much, to hit any EC2-related issues with higher probability. I'm working on that issue as top priority.
In the meantime, I restarted https://android-build.linaro.org/builds/~linaro-android/vexpress-linaro-mp/ and will watch it to make sure we have fresh succeeding build.
On 15 April 2013 10:55, Naresh Kamboju naresh.kamboju@linaro.org wrote:
Hi,
I would like to inform you that following builds failed.
https://android-build.linaro.org/builds/~linaro-android/vexpress-linaro-mp/#...
https://android-build.linaro.org/builds/~linaro-android/vexpress-linaro-mp/#...
https://android-build.linaro.org/builds/~linaro-android/vexpress-linaro-mp/#...
Build log snapshot:
Caused by: hudson.remoting.ChannelClosedException: channel is already closed Caused by: hudson.remoting.Channel$OrderlyShutdown Caused by: Command close created at
FATAL: hudson.remoting.RequestAbortedException: hudson.remoting.Channel$OrderlyShutdown
build complete log:
https://android-build.linaro.org/jenkins/job/linaro-android_vexpress-linaro-...
Best regards Naresh Kamboju
-- Best Regards, Paul
Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog