Hello,
Status update on this: Before proceeding with writing the standalone Python script described, I did another googling pass to find native Jenkins solution for a problem, even though previous my attempts didn't give much (but that might have been a year or so ago). Fortunately, this time I found some hints regarding this problem, it was suggested to use builtin Groovy interpreter which has good (complete?) native Jenkins API access.
So, such script is now cron'ized to run daily. On first day, it recovered 170Gb of free space (1/3 of entire disk space). It however implements a bit weaker expiration as our policy requires, but taking into account that questions/concerns were raised regarding proposal to remove old empty jobs, let's consider that a feature. (And I'm not proceeding with further work on this, as the main aim of maintaining enough free disk space was achieved).
More details may be available at https://bugs.launchpad.net/linaro-android-infrastructure/+bug/1055546
Please let me know if there're further questions/suggestions regarding this.
Thanks, Paul
On Thu, 27 Sep 2012 00:57:04 +0300 Paul Sokolovsky Paul.Sokolovsky@linaro.org wrote:
Hello,
On Wed, 26 Sep 2012 14:02:12 +0200 Alexander Sack asac@linaro.org wrote:
On Wed, Sep 26, 2012 at 1:49 PM, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
Hello,
We keep having cases of out of disk space, latest we had is https://bugs.launchpad.net/linaro-android-infrastructure/+bug/1055546 . The reason this happens so often is that Jenkins has a broken build expiration process - it is applied to jobs which are actually built. If a job is left unattended, its builds will stay forever. This is long-standing known issue, and I manually deleted stale builds in this case, while queuing need to write a script for that, which I'm going to start with now.
To remind, we have following build retention policy:
- Release builds stay forever
- Official daily builds kept for 3 months
- Personal builds kept for 1 month
With new expiration script to be developed, many builds will be "suddenly" (but in full accordance with the policy above) gone. So, I would like to ask all Android developers to look thru their builds to check if you need anything old (and preserve it somewhere else then).
Zach, Fathi, Alexander, can you please do the same for old official builds. For example, https://android-build.linaro.org/jenkins/view/Official%20builds/job/linaro-a... would be gone completely.
In addition to the above, the issue we may have that there're lot of empty personal jobs piled up (with builds removed per the policy above). They clutter view largely, for example, Bero has 10 such jobs, Zach - 8, etc. Do you still need them? Otherwise, I'd propose to add another clause to the policy and remove such empty jobs if not updated within last 3 months. Let me know what you think.
What harm do those empty jobs do? Could we move them to a "archived build configurations" tab? or "Build Graveyard"?
They complicate Jenkins maintenance - it's hard to see live jobs among the cruft. So, do old empty jobs do any good to be preserved instead?