Hi all,
For some reason the jenkins instance that powers the android build system isn't auto provisioning slaves from ec2, so builds are being queued but not executed. If you notice this happen and it's blocking you, get an admin (asac, loic, patryk) to go to https://android-build.linaro.org/jenkins/computer/ and click the "provision via ec2" button, and a slave will be started and run the builds. You can click it more than once to get multiple slaves to clear the queue more quickly. The slaves will be de-provisioned after they have been idle for a little while, so we don't need to manually clean up after ourselves.
I'll try and figure out what's going on and fix it next week.
Cheers, mwh
On Fri, 11 Mar 2011 17:23:45 +1300, Michael Hudson-Doyle michael.hudson@linaro.org wrote:
Hi all,
For some reason the jenkins instance that powers the android build system isn't auto provisioning slaves from ec2, so builds are being queued but not executed. If you notice this happen and it's blocking you, get an admin (asac, loic, patryk) to go to https://android-build.linaro.org/jenkins/computer/ and click the "provision via ec2" button, and a slave will be started and run the builds. You can click it more than once to get multiple slaves to clear the queue more quickly. The slaves will be de-provisioned after they have been idle for a little while, so we don't need to manually clean up after ourselves.
I'll try and figure out what's going on and fix it next week.
I've figured it out (jenkins was counting _all_ instances running in the account it has the credentials for, not just those it launched, against its instance cap) and have worked around it for now (raised the instance cap) pending a better fix (using security groups to identify instances jenkins has launched).
Cheers, mwh
On Wed, 16 Mar 2011 11:43:57 +1300, Michael Hudson-Doyle michael.hudson@linaro.org wrote:
I've figured it out (jenkins was counting _all_ instances running in the account it has the credentials for, not just those it launched, against its instance cap) and have worked around it for now (raised the instance cap) pending a better fix (using security groups to identify instances jenkins has launched).
So it is stateless here and doesn't just remember which instances it launched?
Thanks,
James
On Wed, 16 Mar 2011 09:14:31 -0400, James Westby james.westby@canonical.com wrote:
On Wed, 16 Mar 2011 11:43:57 +1300, Michael Hudson-Doyle michael.hudson@linaro.org wrote:
I've figured it out (jenkins was counting _all_ instances running in the account it has the credentials for, not just those it launched, against its instance cap) and have worked around it for now (raised the instance cap) pending a better fix (using security groups to identify instances jenkins has launched).
So it is stateless here and doesn't just remember which instances it launched?
AIUI, it does remember, but according to the documentation, but applying the limit like this is a belt and suspenders thing in case Jenkins manages to lose this state.
Cheers, mwh
On Thu, 17 Mar 2011 10:59:53 +1300, Michael Hudson-Doyle michael.hudson@linaro.org wrote:
AIUI, it does remember, but according to the documentation, but applying the limit like this is a belt and suspenders thing in case Jenkins manages to lose this state.
Aha, thanks, that sounds somewhat sensible on its part anyway.
Are you planning to work on the more complete fix yourself, or just report the bug and move on?
Thanks,
James
On Wed, 16 Mar 2011 18:16:34 -0400, James Westby james.westby@canonical.com wrote:
On Thu, 17 Mar 2011 10:59:53 +1300, Michael Hudson-Doyle michael.hudson@linaro.org wrote:
AIUI, it does remember, but according to the documentation, but applying the limit like this is a belt and suspenders thing in case Jenkins manages to lose this state.
Aha, thanks, that sounds somewhat sensible on its part anyway.
Are you planning to work on the more complete fix yourself, or just report the bug and move on?
I did the first step yesterday: https://github.com/jenkinsci/ec2-plugin/pull/7
Whether I do the rest is up to you to some extent I guess :-) It's probably around a day's work.
Cheers, mwh
On Thu, 17 Mar 2011 16:49:12 +1300, Michael Hudson-Doyle michael.hudson@linaro.org wrote:
I did the first step yesterday: https://github.com/jenkinsci/ec2-plugin/pull/7
Great, thanks.
Whether I do the rest is up to you to some extent I guess :-) It's probably around a day's work.
I think it's worth following through.
Thanks,
James
-
James Westby -
Michael Hudson-Doyle