Hello,
We're finishing migration of ci.linaro.org to the latest Ubuntu LTS release (14.04). android-build.linaro.org is next in queue. I guess, it makes sense to plan migration for the same time window as ci.l.o - past-release, first week of July. But maybe Android team has special plans or suggestions, so please let us know.
This would also be good time to discuss any changes or additional requirements for the migration. Actually, just today Fathi drew my attention to the fact that there were less 5% free on a-b jobs partition. Looking forward to upcoming migration, I did thorough clean up - we now back to 11% free, and that's actually 55Gb. I also did analysis of growth rate, and my calculations give below 7Gb/year. I hope that's not surprising - most builds we produce, also expire soon, so only growing residue is actually release builds.
So, based on this analysis, we should be fine for next couple of years. You're free to disagree of course. Then I can propose 2 solutions:
1. Reconsider how 500Gb there is used. What suggests of such approach is example, ci.linaro.org, which has a lot of jobs, but of whose 300Gb, 60% are free. That's apparently because it uses much more aggressive expiration policies - for example, unneeded jobs there are actually deleted. Applying that idea to a-b, the build/CI server, do we really need to host old historical releases there, if their official place is https://releases.linaro.org anyway? I guess, that's Fathi call too.
2. If we don't want to change retention policies, and don't trust small growth rate figure either, then increasing disk space is the obvious choice. But I'd say extra 100Gb would really be enough for foreseeable future - having too big a disk means more money (include back up cost), more time when you need back up soon, more fsck time, etc., etc.
I don't think there's much more to vary with this migration between new Jenkins version (implied) and disk space, but if somebody knows other issues which would be nice to fix, let us know (but again, talk is just about Jenkins master, not slaves or something else).
Thanks, Paul
Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
Hello,
I would like to schedule migration for Friday next week, July 4. If you have any concerns, please let me know.
I also didn't receive any response regarding disk space concerns raised in the original mail (quoted below). Since then, I faced yet another with having so many jobs - it now takes quite a while to migrate them for any changes. So, my plan remains to stick with the current disk space, and then discuss and schedule cleaning of historical release builds.
If you don't agree with this tentative direction, and would rather see disk space (and maintenance effort) growing, please speak up soon.
Milo, please approve this plan and schedule otherwise if you see it ok.
Thanks, Paul
On Thu, 12 Jun 2014 22:02:56 +0300 Paul Sokolovsky Paul.Sokolovsky@linaro.org wrote:
Hello,
We're finishing migration of ci.linaro.org to the latest Ubuntu LTS release (14.04). android-build.linaro.org is next in queue. I guess, it makes sense to plan migration for the same time window as ci.l.o - past-release, first week of July. But maybe Android team has special plans or suggestions, so please let us know.
This would also be good time to discuss any changes or additional requirements for the migration. Actually, just today Fathi drew my attention to the fact that there were less 5% free on a-b jobs partition. Looking forward to upcoming migration, I did thorough clean up - we now back to 11% free, and that's actually 55Gb. I also did analysis of growth rate, and my calculations give below 7Gb/year. I hope that's not surprising - most builds we produce, also expire soon, so only growing residue is actually release builds.
So, based on this analysis, we should be fine for next couple of years. You're free to disagree of course. Then I can propose 2 solutions:
- Reconsider how 500Gb there is used. What suggests of such approach
is example, ci.linaro.org, which has a lot of jobs, but of whose 300Gb, 60% are free. That's apparently because it uses much more aggressive expiration policies - for example, unneeded jobs there are actually deleted. Applying that idea to a-b, the build/CI server, do we really need to host old historical releases there, if their official place is https://releases.linaro.org anyway? I guess, that's Fathi call too.
- If we don't want to change retention policies, and don't trust
small growth rate figure either, then increasing disk space is the obvious choice. But I'd say extra 100Gb would really be enough for foreseeable future - having too big a disk means more money (include back up cost), more time when you need back up soon, more fsck time, etc., etc.
I don't think there's much more to vary with this migration between new Jenkins version (implied) and disk space, but if somebody knows other issues which would be nice to fix, let us know (but again, talk is just about Jenkins master, not slaves or something else).
Thanks, Paul
Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
On Thu, Jun 26, 2014 at 5:36 PM, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
I would like to schedule migration for Friday next week, July 4. If you have any concerns, please let me know.
I also didn't receive any response regarding disk space concerns raised in the original mail (quoted below). Since then, I faced yet another with having so many jobs - it now takes quite a while to migrate them for any changes. So, my plan remains to stick with the current disk space, and then discuss and schedule cleaning of historical release builds.
If you don't agree with this tentative direction, and would rather see disk space (and maintenance effort) growing, please speak up soon.
Milo, please approve this plan and schedule otherwise if you see it ok.
Plan sounds good to me.
I would say that if there are no concerns (or no replies) to start cleaning up old builds, making sure they are on releases.l.o. Thanks Paul for working on this.
Ciao.
Hello,
As part of the migration, we would like to simplify email/smtp setup of android-build. Currently, it runs local smtpd to forward email. ci.linaro.org and other Jenkins instances just use google server to send email. We'd like to switch to reuse this account, which means there's one piece less software to maintain on android-build.
I hope nobody is concerned with this change in general. One issue we may expect though, is that email will come from new email address (ci_notify@linaro.org), so it may need to be approved to post to mailing list(s). To not leave too many fixes for next week, I'm going to switch android-build to new account today, to hopefully catch issues yet this week.
Let me know if you are/may be affected by these changes.
Thanks, Paul
On Thu, 26 Jun 2014 18:24:35 +0200 Milo Casagrande milo.casagrande@linaro.org wrote:
On Thu, Jun 26, 2014 at 5:36 PM, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
I would like to schedule migration for Friday next week, July 4. If you have any concerns, please let me know.
I also didn't receive any response regarding disk space concerns raised in the original mail (quoted below). Since then, I faced yet another with having so many jobs - it now takes quite a while to migrate them for any changes. So, my plan remains to stick with the current disk space, and then discuss and schedule cleaning of historical release builds.
If you don't agree with this tentative direction, and would rather see disk space (and maintenance effort) growing, please speak up soon.
Milo, please approve this plan and schedule otherwise if you see it ok.
Plan sounds good to me.
I would say that if there are no concerns (or no replies) to start cleaning up old builds, making sure they are on releases.l.o. Thanks Paul for working on this.
Ciao.
On Wed, 2 Jul 2014 00:38:16 +0300 Paul Sokolovsky Paul.Sokolovsky@linaro.org wrote:
Hello,
As part of the migration, we would like to simplify email/smtp setup of android-build. Currently, it runs local smtpd to forward email. ci.linaro.org and other Jenkins instances just use google server to send email. We'd like to switch to reuse this account, which means there's one piece less software to maintain on android-build.
I hope nobody is concerned with this change in general. One issue we may expect though, is that email will come from new email address (ci_notify@linaro.org), so it may need to be approved to post to mailing list(s). To not leave too many fixes for next week, I'm going to switch android-build to new account today, to hopefully catch issues yet this week.
And first issue already popped up - https://android-build.linaro.org/jenkins/job/linaro-android-restricted_vexpr... tried to send to linaro-android-restricted-builders@linaro.org , but that email account doesn't exist (verified by sending separate test email). Note that the job above also sends emails to linaro-android-restricted-builder-notifications@linaro.org (note "-notifications"), and that address appears to work (at least it doesn't bounce). Does anybody know what we have here? Perhaps linaro-android-restricted-builders@linaro.org just should be removed?
Let me know if you are/may be affected by these changes.
Thanks, Paul
On Thu, 26 Jun 2014 18:24:35 +0200 Milo Casagrande milo.casagrande@linaro.org wrote:
On Thu, Jun 26, 2014 at 5:36 PM, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
I would like to schedule migration for Friday next week, July 4. If you have any concerns, please let me know.
I also didn't receive any response regarding disk space concerns raised in the original mail (quoted below). Since then, I faced yet another with having so many jobs - it now takes quite a while to migrate them for any changes. So, my plan remains to stick with the current disk space, and then discuss and schedule cleaning of historical release builds.
If you don't agree with this tentative direction, and would rather see disk space (and maintenance effort) growing, please speak up soon.
Milo, please approve this plan and schedule otherwise if you see it ok.
Plan sounds good to me.
I would say that if there are no concerns (or no replies) to start cleaning up old builds, making sure they are on releases.l.o. Thanks Paul for working on this.
Ciao.
On 2 July 2014 19:29, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
On Wed, 2 Jul 2014 00:38:16 +0300 Paul Sokolovsky Paul.Sokolovsky@linaro.org wrote:
Hello,
As part of the migration, we would like to simplify email/smtp setup of android-build. Currently, it runs local smtpd to forward email. ci.linaro.org and other Jenkins instances just use google server to send email. We'd like to switch to reuse this account, which means there's one piece less software to maintain on android-build.
I hope nobody is concerned with this change in general. One issue we may expect though, is that email will come from new email address (ci_notify@linaro.org), so it may need to be approved to post to mailing list(s). To not leave too many fixes for next week, I'm going to switch android-build to new account today, to hopefully catch issues yet this week.
And first issue already popped up -
https://android-build.linaro.org/jenkins/job/linaro-android-restricted_vexpr... tried to send to linaro-android-restricted-builders@linaro.org , but that email account doesn't exist (verified by sending separate test email). Note that the job above also sends emails to linaro-android-restricted-builder-notifications@linaro.org (note "-notifications"), and that address appears to work (at least it doesn't bounce). Does anybody know what we have here? Perhaps linaro-android-restricted-builders@linaro.org just should be removed?
This can be removed. The email address were changed by ITS to match the google groups when they were doing migrations.
Let me know if you are/may be affected by these changes.
Thanks, Paul
On Thu, 26 Jun 2014 18:24:35 +0200 Milo Casagrande milo.casagrande@linaro.org wrote:
On Thu, Jun 26, 2014 at 5:36 PM, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
I would like to schedule migration for Friday next week, July 4. If you have any concerns, please let me know.
I also didn't receive any response regarding disk space concerns raised in the original mail (quoted below). Since then, I faced yet another with having so many jobs - it now takes quite a while to migrate them for any changes. So, my plan remains to stick with the current disk space, and then discuss and schedule cleaning of historical release builds.
If you don't agree with this tentative direction, and would rather see disk space (and maintenance effort) growing, please speak up soon.
Milo, please approve this plan and schedule otherwise if you see it ok.
Plan sounds good to me.
I would say that if there are no concerns (or no replies) to start cleaning up old builds, making sure they are on releases.l.o. Thanks Paul for working on this.
Ciao.
-- Best Regards, Paul
Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
linaro-android mailing list linaro-android@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-android
Hello,
Sorry there was no formal notice of android-build.linaro.org being back to production mode - I wanted to do some cosmetic changes first, I got to it only on Friday. It was also switched to LDAP/Crowd auth on Friday, which broke publishing over weekend (fixed now).
Following cosmetic changes we applied to a-b:
1. In Jenkins, more tabbed views were added. Given the big number of jobs we have there, that may help. 2. Official Linaro Jenkins theme was applied. 3. I did some progress towards resolving long load times of a-b frontend's index page. This is in review currently:
https://review.linaro.org/#/c/2483/ https://review.linaro.org/#/c/2497/
But once merged, it should improve user experience using the frontend.
So, android-build.linaro.org is back to production mode, all issues should be reported via bug reports. And given that Linaro migrates to Bugzilla at https://bugs.linaro.org/ , they should be submitted there, with product of "Linaro CI" (direct link https://bugs.linaro.org/enter_bug.cgi?product=Linaro%20CI ), component "android-build.linaro.org".
Thanks, Paul
On Wed, 2 Jul 2014 16:59:14 +0300 Paul Sokolovsky Paul.Sokolovsky@linaro.org wrote:
On Wed, 2 Jul 2014 00:38:16 +0300 Paul Sokolovsky Paul.Sokolovsky@linaro.org wrote:
Hello,
As part of the migration, we would like to simplify email/smtp setup of android-build. Currently, it runs local smtpd to forward email. ci.linaro.org and other Jenkins instances just use google server to send email. We'd like to switch to reuse this account, which means there's one piece less software to maintain on android-build.
I hope nobody is concerned with this change in general. One issue we may expect though, is that email will come from new email address (ci_notify@linaro.org), so it may need to be approved to post to mailing list(s). To not leave too many fixes for next week, I'm going to switch android-build to new account today, to hopefully catch issues yet this week.
And first issue already popped up - https://android-build.linaro.org/jenkins/job/linaro-android-restricted_vexpr... tried to send to linaro-android-restricted-builders@linaro.org , but that email account doesn't exist (verified by sending separate test email). Note that the job above also sends emails to linaro-android-restricted-builder-notifications@linaro.org (note "-notifications"), and that address appears to work (at least it doesn't bounce). Does anybody know what we have here? Perhaps linaro-android-restricted-builders@linaro.org just should be removed?
Let me know if you are/may be affected by these changes.
Thanks, Paul
On Thu, 26 Jun 2014 18:24:35 +0200 Milo Casagrande milo.casagrande@linaro.org wrote:
On Thu, Jun 26, 2014 at 5:36 PM, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
I would like to schedule migration for Friday next week, July 4. If you have any concerns, please let me know.
I also didn't receive any response regarding disk space concerns raised in the original mail (quoted below). Since then, I faced yet another with having so many jobs - it now takes quite a while to migrate them for any changes. So, my plan remains to stick with the current disk space, and then discuss and schedule cleaning of historical release builds.
If you don't agree with this tentative direction, and would rather see disk space (and maintenance effort) growing, please speak up soon.
Milo, please approve this plan and schedule otherwise if you see it ok.
Plan sounds good to me.
I would say that if there are no concerns (or no replies) to start cleaning up old builds, making sure they are on releases.l.o. Thanks Paul for working on this.
Ciao.
linaro-android@lists.linaro.org
-
Milo Casagrande -
Paul Sokolovsky -
Vishal Bhoj