Hello,
Over lifetime of Linaro CI services, we've been experiencing number of episodes when service stability was influenced by Ubuntu package mirror issues, with it being used to configure each and every build slave, usually dozen times a day.
Linaro Infrastructure team and CI service stakeholders tried different approaches to solve this issue, and over time we come to conclusion that sustainable way to do it is to maintain custom Amazon Machine Images (AMI) with preinstalled packages, which will simply cut the dependency on Ubuntu mirror's 100% availability. In addition to that, they also speed up build startup time noticeably.
A special tool, linaro-ami, was developed to help maintain custom AMIs, with two main requirements being centralized configuration location for EC2 admins control, as well as being easy to use for individual custom AMI maintainers. We still may need to improve UI functionality for the latter, and would need stakeholder feedback for that, but otherwise the tool seems to work pretty well. It is available as part of linaro-aws-tools project, https://launchpad.net/linaro-aws-tools . The README is available at http://bazaar.launchpad.net/~linaro-aws-devs/linaro-aws-tools/trunk/view/hea...
ci.linaro.org and android-build.linaro.org has been already switched to new custom AMIs. Using custom AMIs for CBuild service is under consideration by Toolchain group.
So, from this point on, it's important that all stakeholders follow build slave configuration change process centered around linaro-ami tool: any package set changes should be applied to slave init script (as referenced in linaro-ami config), a new version of corresponding AMI generated, and Jenkins configuration updated with new AMI ID. (No package installation should be happen directly in Jenkins.) The Infrastructure team can help with maintaining AMIs, and is open for suggestions on improving new setup.
Thanks, Paul
Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog