On Wed, Sep 26, 2012 at 11:54 PM, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
Hello,
On Wed, 26 Sep 2012 14:00:59 +0200 Alexander Sack asac@linaro.org wrote:
On Wed, Sep 26, 2012 at 1:49 PM, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
Hello,
We keep having cases of out of disk space, latest we had is https://bugs.launchpad.net/linaro-android-infrastructure/+bug/1055546 . The reason this happens so often is that Jenkins has a broken build expiration process - it is applied to jobs which are actually built. If a job is left unattended, its builds will stay forever. This is long-standing known issue, and I manually deleted stale builds in this case, while queuing need to write a script for that, which I'm going to start with now.
To remind, we have following build retention policy:
- Release builds stay forever
- Official daily builds kept for 3 months
- Personal builds kept for 1 month
With new expiration script to be developed, many builds will be "suddenly" (but in full accordance with the policy above) gone. So, I would like to ask all Android developers to look thru their builds to check if you need anything old (and preserve it somewhere else then).
It's not clear what the new expiration script will do. Can you summarize what will change?
Nothing will change, but new functionality will be added: there will be a cronjob (in Python) which will read job configs and apply expiration settings in them unconditionally, unlike Jenkins, which applies them only in case a build is made.
Zach, Fathi, Alexander, can you please do the same for old official builds. For example, https://android-build.linaro.org/jenkins/view/Official%20builds/job/linaro-a... would be gone completely.
In addition to the above, the issue we may have that there're lot of empty personal jobs piled up (with builds removed per the policy above). They clutter view largely, for example, Bero has 10 such jobs, Zach - 8, etc. Do you still need them? Otherwise, I'd propose to add another clause to the policy and remove such empty jobs if not updated within last 3 months. Let me know what you think.
Also discussed during Leads meeting:
- make a "release build" tab; keep release builds forever confirmed
- make a "regression build" tab where all the variation builds we
don't officially release go; those will get a 1 month retention 3. later: rename the Official build tab to something better.
As far as I can tell, all new tabs would go into new CI Dashboard which is being developed by Infrastructure now. Danilo may have better insight though.
CI dashboard is far ahead from . I don't see this land in production before early next year.
While I agree we shouldn't put massive efforts into android-build before that, I feel hesitant to stop all refinements because of that.
With that a bunch of currently official builds will get a reduced retention which would give us more room to breeze.
With my maintenance-engineer-for-this-cycle hat on, I don't think any other changes besides those proposed above are needed at this time - we already have good retention policy, we just need to make sure it's actually enforced. We can queue more changes for future planning and implementation of course.
-- Best Regards, Paul
Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog