Hello,
On Wed, 26 Sep 2012 14:20:17 -0500 Zach Pfeffer zach.pfeffer@linaro.org wrote:
On 26 September 2012 06:49, Paul Sokolovsky paul.sokolovsky@linaro.orgwrote:
Hello,
We keep having cases of out of disk space, latest we had is https://bugs.launchpad.net/linaro-android-infrastructure/+bug/1055546 . The reason this happens so often is that Jenkins has a broken build expiration process - it is applied to jobs which are actually built. If a job is left unattended, its builds will stay forever. This is long-standing known issue, and I manually deleted stale builds in this case, while queuing need to write a script for that, which I'm going to start with now.
To remind, we have following build retention policy:
- Release builds stay forever
- Official daily builds kept for 3 months
- Personal builds kept for 1 month
With new expiration script to be developed, many builds will be "suddenly" (but in full accordance with the policy above) gone. So, I would like to ask all Android developers to look thru their builds to check if you need anything old (and preserve it somewhere else then).
Zach, Fathi, Alexander, can you please do the same for old official builds. For example,
https://android-build.linaro.org/jenkins/view/Official%20builds/job/linaro-a... would be gone completely.
In addition to the above, the issue we may have that there're lot of empty personal jobs piled up (with builds removed per the policy above). They clutter view largely, for example, Bero has 10 such jobs, Zach - 8, etc. Do you still need them? Otherwise, I'd propose to add another clause to the policy and remove such empty jobs if not updated within last 3 months. Let me know what you think.
What if we simplify this. We introduce a build variable that's a date or a value. Builds without such a variable disappear in a week, builds with it expire on the date show, builds marked FOREVER stick around.
I'm not sure that would be simplification: 1) it will need to be implemented; 2) people will need to know about and it not forget to use it. As I mentioned in the reply to Alexander, I don't think that we need any other actions at this time - we (Infrastructure) just need to implement existing policies thoroughly. If you think that changes above are important, please discuss them with Danilo for scheduling.
Also, what about the underling disk space issues. Android build sizes will keep going up and the number of Android builds will keep going up. Can we just get a bigger disk?
So far, large share of disk space is occupied by cruft, and size of that goes up faster than size of builds to retain. So, no matter how much disk space we get, without proper management, it will be filled in very quickly (much quicker than expected). At least that's how it was so far, so let's try to address space management issues now.