Hello,
We've been having low disk space issues on android-build for some time now, and I took that as a chance to wrap up loose ends with builds expiration, etc. Well, now I'm sure that it's ok, but we still have negative delta for disk space each day.
Note that we no longer store any artifacts on android-build (*1), so disk space consumed only for build logs and other aux files. But they're actually bigger than build artifacts - we have ~100Mb build logs, and that dupped to have parsed/highlighted logs, so we have ~200Mb for each build. With almost 30 daily builds, that's 6Gb daily. Old builds expire of course, but again, so far we have negative disk space delta.
So, I'm going to dup the disk space we use, 300Gb to 600Gb. I also going to make disk layout changes we discussed previously, to make sure that disk overflow never leads to pressure on Jenkins global config files, which in turn leads to zombie builds slave storms. This is going to be done step by step, so today and tomorrow downtime should be expected (few cases of <1h tentative). I'll announce specific times on IRC. Let me know if you have any concerns.
*1 Well, some toolchain builds host artifacts still there, something to fix soon.
Thanks, Paul
Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
Btw, this is a good time to clean up older personal jobs - we have quite a lot of such, with all their builds had expired, so they just take up time in job lists.
So, if you could go to https://android-build.linaro.org/#builds=mine and deleted any stale jobs you have, that would be appreciated. I'm myself doing just that right now ;-).
On Thu, 3 May 2012 14:53:30 +0300 Paul Sokolovsky Paul.Sokolovsky@linaro.org wrote:
Hello,
We've been having low disk space issues on android-build for some time now, and I took that as a chance to wrap up loose ends with builds expiration, etc. Well, now I'm sure that it's ok, but we still have negative delta for disk space each day.
Note that we no longer store any artifacts on android-build (*1), so disk space consumed only for build logs and other aux files. But they're actually bigger than build artifacts - we have ~100Mb build logs, and that dupped to have parsed/highlighted logs, so we have ~200Mb for each build. With almost 30 daily builds, that's 6Gb daily. Old builds expire of course, but again, so far we have negative disk space delta.
So, I'm going to dup the disk space we use, 300Gb to 600Gb. I also going to make disk layout changes we discussed previously, to make sure that disk overflow never leads to pressure on Jenkins global config files, which in turn leads to zombie builds slave storms. This is going to be done step by step, so today and tomorrow downtime should be expected (few cases of <1h tentative). I'll announce specific times on IRC. Let me know if you have any concerns.
*1 Well, some toolchain builds host artifacts still there, something to fix soon.
Thanks, Paul
Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
Hello,
Ok, first stage of upgrade was performed yesterday. Looking how long it takes to even snapshot a disk (I also had to wait for fsck and resizefs later) and reading that Zach intends to limit number of daily builds, I created 500Gb partition after all.
So, now we run off of it, and 3-partition setup which will preclude overflowing of Jenkins system partition and making it go haywire. What's left is to optimize this Jenkins system partition which is so far just old 300Gb partition. So, there will be more downtime, stay tuned ;-).
On Thu, 3 May 2012 14:53:30 +0300 Paul Sokolovsky Paul.Sokolovsky@linaro.org wrote:
Hello,
We've been having low disk space issues on android-build for some time now, and I took that as a chance to wrap up loose ends with builds expiration, etc. Well, now I'm sure that it's ok, but we still have negative delta for disk space each day.
Note that we no longer store any artifacts on android-build (*1), so disk space consumed only for build logs and other aux files. But they're actually bigger than build artifacts - we have ~100Mb build logs, and that dupped to have parsed/highlighted logs, so we have ~200Mb for each build. With almost 30 daily builds, that's 6Gb daily. Old builds expire of course, but again, so far we have negative disk space delta.
So, I'm going to dup the disk space we use, 300Gb to 600Gb. I also going to make disk layout changes we discussed previously, to make sure that disk overflow never leads to pressure on Jenkins global config files, which in turn leads to zombie builds slave storms. This is going to be done step by step, so today and tomorrow downtime should be expected (few cases of <1h tentative). I'll announce specific times on IRC. Let me know if you have any concerns.
*1 Well, some toolchain builds host artifacts still there, something to fix soon.
Thanks, Paul
Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
Hello,
android-build migration to new 3-volume setup has been completed. We now should be protected from "zombie build slave storms" which we had in case of disk overflow previously. I'm currently polishing up and documenting this setup, to proceed with ci.linaro.org migration to it either.
On Fri, 4 May 2012 11:36:27 +0300 Paul Sokolovsky Paul.Sokolovsky@linaro.org wrote:
Hello,
Ok, first stage of upgrade was performed yesterday. Looking how long it takes to even snapshot a disk (I also had to wait for fsck and resizefs later) and reading that Zach intends to limit number of daily builds, I created 500Gb partition after all.
So, now we run off of it, and 3-partition setup which will preclude overflowing of Jenkins system partition and making it go haywire. What's left is to optimize this Jenkins system partition which is so far just old 300Gb partition. So, there will be more downtime, stay tuned ;-).
On Thu, 3 May 2012 14:53:30 +0300 Paul Sokolovsky Paul.Sokolovsky@linaro.org wrote:
Hello,
We've been having low disk space issues on android-build for some time now, and I took that as a chance to wrap up loose ends with builds expiration, etc. Well, now I'm sure that it's ok, but we still have negative delta for disk space each day.
Note that we no longer store any artifacts on android-build (*1), so disk space consumed only for build logs and other aux files. But they're actually bigger than build artifacts - we have ~100Mb build logs, and that dupped to have parsed/highlighted logs, so we have ~200Mb for each build. With almost 30 daily builds, that's 6Gb daily. Old builds expire of course, but again, so far we have negative disk space delta.
So, I'm going to dup the disk space we use, 300Gb to 600Gb. I also going to make disk layout changes we discussed previously, to make sure that disk overflow never leads to pressure on Jenkins global config files, which in turn leads to zombie builds slave storms. This is going to be done step by step, so today and tomorrow downtime should be expected (few cases of <1h tentative). I'll announce specific times on IRC. Let me know if you have any concerns.
*1 Well, some toolchain builds host artifacts still there, something to fix soon.
Thanks, Paul
Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
linaro-android@lists.linaro.org
-
Paul Sokolovsky