Hi all,
Just a quick note to say that I'm restoring the staging database from the latest production snapshot. Staging will be down for however long this takes (maybe an hour).
Cheers, mwh
Michael Hudson-Doyle michael.hudson@canonical.com writes:
Hi all,
Just a quick note to say that I'm restoring the staging database from the latest production snapshot. Staging will be down for however long this takes (maybe an hour).
Now done. We're a little tight for disk space on staging -- I had to delete the logs and bundles to get the restore to complete. Can that be upped without trashing the instance?
Cheers, mwh
On 1 Oct 2012, at 03:06, Michael Hudson-Doyle wrote:
Michael Hudson-Doyle michael.hudson@canonical.com writes:
Hi all,
Just a quick note to say that I'm restoring the staging database from the latest production snapshot. Staging will be down for however long this takes (maybe an hour).
Now done. We're a little tight for disk space on staging -- I had to delete the logs and bundles to get the restore to complete. Can that be upped without trashing the instance?
Cheers, mwh
Hmmm. It *should* be possible. I'll let you know either way. If it is, I'll up it, if not I'll warn and we'll pick a time to do it.
Dave
On 1 Oct 2012, at 03:06, Michael Hudson-Doyle wrote:
Michael Hudson-Doyle michael.hudson@canonical.com writes:
Hi all,
Just a quick note to say that I'm restoring the staging database from the latest production snapshot. Staging will be down for however long this takes (maybe an hour).
Now done. We're a little tight for disk space on staging -- I had to delete the logs and bundles to get the restore to complete. Can that be upped without trashing the instance?
Rats. Looks like you can't do this on a live instance. Let me know when, and I'll restart it with more space. Would 100GB be enough?
Dave
Dave Pigott dave.pigott@linaro.org writes:
On 1 Oct 2012, at 03:06, Michael Hudson-Doyle wrote:
Michael Hudson-Doyle michael.hudson@canonical.com writes:
Hi all,
Just a quick note to say that I'm restoring the staging database from the latest production snapshot. Staging will be down for however long this takes (maybe an hour).
Now done. We're a little tight for disk space on staging -- I had to delete the logs and bundles to get the restore to complete. Can that be upped without trashing the instance?
Rats. Looks like you can't do this on a live instance. Let me know when,
Well, if you're working, I'm probably not :-) So whenever is fine as far as I'm concerned.
and I'll restart it with more space. Would 100GB be enough?
Given 10 Gb is tight but workable, 100 Gb almost seems excessive :) But if we have the capacity, sure, sounds good.
Cheers, mwh
On 1 Oct 2012, at 09:48, Michael Hudson-Doyle wrote:
Dave Pigott dave.pigott@linaro.org writes:
On 1 Oct 2012, at 03:06, Michael Hudson-Doyle wrote:
Michael Hudson-Doyle michael.hudson@canonical.com writes:
Hi all,
Just a quick note to say that I'm restoring the staging database from the latest production snapshot. Staging will be down for however long this takes (maybe an hour).
Now done. We're a little tight for disk space on staging -- I had to delete the logs and bundles to get the restore to complete. Can that be upped without trashing the instance?
Rats. Looks like you can't do this on a live instance. Let me know when,
Well, if you're working, I'm probably not :-) So whenever is fine as far as I'm concerned.
OK. Shut down the old ones and created a new flavor, m1.lava, which has 8GB RAM, 50GB HD, 4CPUs and 80GB ephemeral storage. Two new instances, dogfood and staging, still at the same ip addresses. If someone walks me through it or point me at a wiki it would be quite good to know how to deploy a lava server from scratch.
Dave
and I'll restart it with more space. Would 100GB be enough?
Given 10 Gb is tight but workable, 100 Gb almost seems excessive :) But if we have the capacity, sure, sounds good.
Cheers, mwh
On 10/01/2012 08:58 AM, Dave Pigott wrote:
OK. Shut down the old ones and created a new flavor, m1.lava, which has 8GB RAM, 50GB HD, 4CPUs and 80GB ephemeral storage. Two new instances, dogfood and staging, still at the same ip addresses. If someone walks me through it or point me at a wiki it would be quite good to know how to deploy a lava server from scratch.
https://wiki.linaro.org/Platform/LAVA/DevOps/ServerProvisioning
BTW - in the future, lets hold off killing an instance before giving everyone time to back things up off of it. Its just staging, but sometimes people are doing some dev work on it.
Dave Pigott dave.pigott@linaro.org writes:
On 1 Oct 2012, at 09:48, Michael Hudson-Doyle wrote:
Dave Pigott dave.pigott@linaro.org writes:
On 1 Oct 2012, at 03:06, Michael Hudson-Doyle wrote:
Michael Hudson-Doyle michael.hudson@canonical.com writes:
Hi all,
Just a quick note to say that I'm restoring the staging database from the latest production snapshot. Staging will be down for however long this takes (maybe an hour).
Now done. We're a little tight for disk space on staging -- I had to delete the logs and bundles to get the restore to complete. Can that be upped without trashing the instance?
Rats. Looks like you can't do this on a live instance. Let me know when,
Well, if you're working, I'm probably not :-) So whenever is fine as far as I'm concerned.
OK. Shut down the old ones and created a new flavor, m1.lava, which has 8GB RAM, 50GB HD, 4CPUs and 80GB ephemeral storage. Two new instances, dogfood and staging, still at the same ip addresses.
Um. That wasn't what I was expecting! I thought you were going to shut the instance down, resize the root volume and bring the same instance up again. Is that not possible?
Luckily lava-deployment-tool makes the set up pretty easy :-)
Cheers, mwh
If someone walks me through it or point me at a wiki it would be quite good to know how to deploy a lava server from scratch.
Dave
and I'll restart it with more space. Would 100GB be enough?
Given 10 Gb is tight but workable, 100 Gb almost seems excessive :) But if we have the capacity, sure, sounds good.
Cheers, mwh
Michael Hudson-Doyle michael.hudson@linaro.org writes:
Luckily lava-deployment-tool makes the set up pretty easy :-)
So ftr, here are the steps involved in setting up staging:
0) Create the instance-manager account 1) Run ldt setup, ldt install 2) Set up the devices - this was the most tedious step - this includes copying the reset-serial and pdu control scripts 3) run ldt restore-db 4) Set up the cronjobs to make the deployment-report and do code updates daily 5) Put branches in /srv/lava/branches/ 6) Set up apache to SetHandler none /deployment-report.xhtml, /staging-updates.txt through + create symlinks for same. 7) Allow instance-manger to run appropriate commands without a password
Luckily an awful lot of the staging-specific stuff had not been purged from control so I could copy this across. But it was still a pain -- maybe I'll script it up when I do dogfood!
Cheers, mwh
Michael Hudson-Doyle michael.hudson@linaro.org writes:
Michael Hudson-Doyle michael.hudson@linaro.org writes:
Luckily lava-deployment-tool makes the set up pretty easy :-)
So ftr, here are the steps involved in setting up staging:
- Create the instance-manager account
- Run ldt setup, ldt install
- Set up the devices
- this was the most tedious step
- this includes copying the reset-serial and pdu control scripts
- run ldt restore-db
- Set up the cronjobs to make the deployment-report and do code updates daily
- Put branches in /srv/lava/branches/
- Set up apache to SetHandler none /deployment-report.xhtml, /staging-updates.txt through + create symlinks for same.
- Allow instance-manger to run appropriate commands without a password
There is more, sadly:
8) realize that you should have run restore-staging-db, not ldt restore-db. 9) fix the fact that this doesn't work any more due to django being rubbish 10) create /linaro/tmp and /linaro/cache directories 11) Realize that you need to set up Apache so that the device can download the test image tarballs.
Maybe that's it?
Cheers, mwh
Luckily an awful lot of the staging-specific stuff had not been purged from control so I could copy this across. But it was still a pain -- maybe I'll script it up when I do dogfood!
Cheers, mwh
Really sorry for all the pain.
No, you can't resize when an instance is down. You have to decide what you want up front, which is a bit of a pain.
Dave
On 2 Oct 2012, at 02:26, Michael Hudson-Doyle wrote:
Michael Hudson-Doyle michael.hudson@linaro.org writes:
Michael Hudson-Doyle michael.hudson@linaro.org writes:
Luckily lava-deployment-tool makes the set up pretty easy :-)
So ftr, here are the steps involved in setting up staging:
- Create the instance-manager account
- Run ldt setup, ldt install
- Set up the devices
- this was the most tedious step
- this includes copying the reset-serial and pdu control scripts
- run ldt restore-db
- Set up the cronjobs to make the deployment-report and do code updates
daily 5) Put branches in /srv/lava/branches/ 6) Set up apache to SetHandler none /deployment-report.xhtml, /staging-updates.txt through + create symlinks for same. 7) Allow instance-manger to run appropriate commands without a password
There is more, sadly:
- realize that you should have run restore-staging-db, not ldt
restore-db. 9) fix the fact that this doesn't work any more due to django being rubbish 10) create /linaro/tmp and /linaro/cache directories 11) Realize that you need to set up Apache so that the device can download the test image tarballs.
Maybe that's it?
Cheers, mwh
Luckily an awful lot of the staging-specific stuff had not been purged from control so I could copy this across. But it was still a pain -- maybe I'll script it up when I do dogfood!
Cheers, mwh
Dave Pigott dave.pigott@linaro.org writes:
Really sorry for all the pain.
Ah well. It's working now.
No, you can't resize when an instance is down. You have to decide what you want up front, which is a bit of a pain.
Noted!
Cheers, mwh
Dave
On 2 Oct 2012, at 02:26, Michael Hudson-Doyle wrote:
Michael Hudson-Doyle michael.hudson@linaro.org writes:
Michael Hudson-Doyle michael.hudson@linaro.org writes:
Luckily lava-deployment-tool makes the set up pretty easy :-)
So ftr, here are the steps involved in setting up staging:
- Create the instance-manager account
- Run ldt setup, ldt install
- Set up the devices
- this was the most tedious step
- this includes copying the reset-serial and pdu control scripts
- run ldt restore-db
- Set up the cronjobs to make the deployment-report and do code updates
daily 5) Put branches in /srv/lava/branches/ 6) Set up apache to SetHandler none /deployment-report.xhtml, /staging-updates.txt through + create symlinks for same. 7) Allow instance-manger to run appropriate commands without a password
There is more, sadly:
- realize that you should have run restore-staging-db, not ldt
restore-db. 9) fix the fact that this doesn't work any more due to django being rubbish 10) create /linaro/tmp and /linaro/cache directories 11) Realize that you need to set up Apache so that the device can download the test image tarballs.
Maybe that's it?
Cheers, mwh
Luckily an awful lot of the staging-specific stuff had not been purged from control so I could copy this across. But it was still a pain -- maybe I'll script it up when I do dogfood!
Cheers, mwh
Michael Hudson-Doyle michael.hudson@linaro.org writes:
Michael Hudson-Doyle michael.hudson@linaro.org writes:
Michael Hudson-Doyle michael.hudson@linaro.org writes:
Luckily lava-deployment-tool makes the set up pretty easy :-)
So ftr, here are the steps involved in setting up staging:
- Create the instance-manager account
- Run ldt setup, ldt install
- Set up the devices
- this was the most tedious step
- this includes copying the reset-serial and pdu control scripts
- run ldt restore-db
- Set up the cronjobs to make the deployment-report and do code updates daily
- Put branches in /srv/lava/branches/
I forgot to set the instance up to _use_ these branches here!
- Set up apache to SetHandler none /deployment-report.xhtml, /staging-updates.txt through + create symlinks for same.
- Allow instance-manger to run appropriate commands without a password
There is more, sadly:
- realize that you should have run restore-staging-db, not ldt restore-db.
- fix the fact that this doesn't work any more due to django being rubbish
- create /linaro/tmp and /linaro/cache directories
- Realize that you need to set up Apache so that the device can download the test image tarballs.
Maybe that's it?
Cheers, mwh
Luckily an awful lot of the staging-specific stuff had not been purged from control so I could copy this across. But it was still a pain -- maybe I'll script it up when I do dogfood!
Cheers, mwh
I've partially set up dogfood today and wrote up the steps in more detail: https://wiki.linaro.org/Platform/LAVA/DevOps/SettingUpStaging
If someone wants to start automating some of this, I won't be stopping them!
Cheersm mwh
Michael Hudson-Doyle michael.hudson@linaro.org writes:
Michael Hudson-Doyle michael.hudson@linaro.org writes:
Michael Hudson-Doyle michael.hudson@linaro.org writes:
Michael Hudson-Doyle michael.hudson@linaro.org writes:
Luckily lava-deployment-tool makes the set up pretty easy :-)
So ftr, here are the steps involved in setting up staging:
- Create the instance-manager account
- Run ldt setup, ldt install
- Set up the devices
- this was the most tedious step
- this includes copying the reset-serial and pdu control scripts
- run ldt restore-db
- Set up the cronjobs to make the deployment-report and do code updates daily
- Put branches in /srv/lava/branches/
I forgot to set the instance up to _use_ these branches here!
- Set up apache to SetHandler none /deployment-report.xhtml, /staging-updates.txt through + create symlinks for same.
- Allow instance-manger to run appropriate commands without a password
There is more, sadly:
- realize that you should have run restore-staging-db, not ldt restore-db.
- fix the fact that this doesn't work any more due to django being rubbish
- create /linaro/tmp and /linaro/cache directories
- Realize that you need to set up Apache so that the device can download the test image tarballs.
Maybe that's it?
Cheers, mwh
Luckily an awful lot of the staging-specific stuff had not been purged from control so I could copy this across. But it was still a pain -- maybe I'll script it up when I do dogfood!
Cheers, mwh
Michael Hudson-Doyle escreveu:
I've partially set up dogfood today and wrote up the steps in more detail: https://wiki.linaro.org/Platform/LAVA/DevOps/SettingUpStaging
If someone wants to start automating some of this, I won't be stopping them!
I am interested in this. I've started drafting a blueprint for this:
https://blueprints.launchpad.net/lava-lab/+spec/automated-staging-deployment
On 10/07/2012 07:33 PM, Michael Hudson-Doyle wrote:
I've partially set up dogfood today and wrote up the steps in more detail: https://wiki.linaro.org/Platform/LAVA/DevOps/SettingUpStaging
Thanks. I fixed the section about the conmux hack. During the lab migration to leased line I fixed this.
The main issue was that our /etc/hosts file on control had an odd entry for 127.0.1.1. I changed our /etc/hosts to a standard looking file and updated its resolv.conf to use the gateway server we are using everywhere else. So far, so good.
Andy Doan andy.doan@linaro.org writes:
On 10/07/2012 07:33 PM, Michael Hudson-Doyle wrote:
I've partially set up dogfood today and wrote up the steps in more detail: https://wiki.linaro.org/Platform/LAVA/DevOps/SettingUpStaging
Thanks. I fixed the section about the conmux hack. During the lab migration to leased line I fixed this.
The main issue was that our /etc/hosts file on control had an odd entry for 127.0.1.1. I changed our /etc/hosts to a standard looking file and updated its resolv.conf to use the gateway server we are using everywhere else. So far, so good.
Ah OK. Thanks :)
Cheers, mwh
And when we update to Precise, resolv.conf will be generated by the net manager. Good call by asac to halt the serve upgrade.
Dave
Sent from my Aldis Lamp
On 8 Oct 2012, at 20:01, Michael Hudson-Doyle michael.hudson@linaro.org wrote:
Andy Doan andy.doan@linaro.org writes:
On 10/07/2012 07:33 PM, Michael Hudson-Doyle wrote:
I've partially set up dogfood today and wrote up the steps in more detail: https://wiki.linaro.org/Platform/LAVA/DevOps/SettingUpStaging
Thanks. I fixed the section about the conmux hack. During the lab migration to leased line I fixed this.
The main issue was that our /etc/hosts file on control had an odd entry for 127.0.1.1. I changed our /etc/hosts to a standard looking file and updated its resolv.conf to use the gateway server we are using everywhere else. So far, so good.
Ah OK. Thanks :)
Cheers, mwh
On 7 October 2012 19:33, Michael Hudson-Doyle michael.hudson@linaro.org wrote:
I've partially set up dogfood today and wrote up the steps in more detail: https://wiki.linaro.org/Platform/LAVA/DevOps/SettingUpStaging
If someone wants to start automating some of this, I won't be stopping them!
As per section 5.3 about copying scripts around:
I've updated staging slightly today. The only scripts I *think* we need are now under /usr/local/bin. This includes:
* maintenance.sh - the shell you should run when making changes * reboot-ap7952 - controls our PDU's * reset-serial, reset-serial5000, reset-serial6000 resets our various serial console ports
I'm thinking we should maintain these in BZR branch, and update:
https://wiki.linaro.org/Platform/LAVA/DevOps/ServerProvisioning
to include a way to always copy those to your /usr/local/bin
thoughts?
Andy Doan andy.doan@linaro.org writes:
On 7 October 2012 19:33, Michael Hudson-Doyle michael.hudson@linaro.org wrote:
I've partially set up dogfood today and wrote up the steps in more detail: https://wiki.linaro.org/Platform/LAVA/DevOps/SettingUpStaging
If someone wants to start automating some of this, I won't be stopping them!
As per section 5.3 about copying scripts around:
I've updated staging slightly today. The only scripts I *think* we need are now under /usr/local/bin. This includes:
- maintenance.sh - the shell you should run when making changes
- reboot-ap7952 - controls our PDU's
- reset-serial, reset-serial5000, reset-serial6000 resets our various
serial console ports
I'm thinking we should maintain these in BZR branch
+1. Can we just stick them on Launchpad (i.e. we didn't steal them from copyrighted sources we weren't supposed to etc)?
and update:
https://wiki.linaro.org/Platform/LAVA/DevOps/ServerProvisioning
to include a way to always copy those to your /usr/local/bin
That way is a debian package :-)
Cheers, mwh
On 10/09/2012 04:17 PM, Michael Hudson-Doyle wrote:
Andy Doan andy.doan@linaro.org writes:
On 7 October 2012 19:33, Michael Hudson-Doyle michael.hudson@linaro.org wrote:
I've partially set up dogfood today and wrote up the steps in more detail: https://wiki.linaro.org/Platform/LAVA/DevOps/SettingUpStaging
If someone wants to start automating some of this, I won't be stopping them!
As per section 5.3 about copying scripts around:
I've updated staging slightly today. The only scripts I *think* we need are now under /usr/local/bin. This includes:
- maintenance.sh - the shell you should run when making changes
- reboot-ap7952 - controls our PDU's
- reset-serial, reset-serial5000, reset-serial6000 resets our various
serial console ports
I'm thinking we should maintain these in BZR branch
+1. Can we just stick them on Launchpad (i.e. we didn't steal them from copyrighted sources we weren't supposed to etc)?
We should probably update the header of reboot-ap7952. It looks like it is a one-off of conmux APC script judging by the header
The other scripts were done by us.
and update:
https://wiki.linaro.org/Platform/LAVA/DevOps/ServerProvisioning
to include a way to always copy those to your /usr/local/bin
That way is a debian package :-)
Didn't know if that was over-doing it. ie, you could also just bzr-branch the usr/local/bin directory. I think Antonio's background might help us here :)
Andy Doan escreveu:
On 10/09/2012 04:17 PM, Michael Hudson-Doyle wrote:
Andy Doan andy.doan@linaro.org writes:
On 7 October 2012 19:33, Michael Hudson-Doyle michael.hudson@linaro.org wrote: and update:
https://wiki.linaro.org/Platform/LAVA/DevOps/ServerProvisioning
to include a way to always copy those to your /usr/local/bin
That way is a debian package :-)
Didn't know if that was over-doing it. ie, you could also just bzr-branch the usr/local/bin directory. I think Antonio's background might help us here :)
If those scripts are useful only for us in the LAVA lab, I think a debian package is overkill, and we could just include them in the upcoming chef recipes. If they are useful for others, then for sure I can package them; it should make be a pretty simple package.
What do you think?
On 10 Oct 2012, at 14:07, Antonio Terceiro antonio.terceiro@linaro.org wrote:
Andy Doan escreveu:
On 10/09/2012 04:17 PM, Michael Hudson-Doyle wrote:
Andy Doan andy.doan@linaro.org writes:
On 7 October 2012 19:33, Michael Hudson-Doyle michael.hudson@linaro.org wrote: and update:
https://wiki.linaro.org/Platform/LAVA/DevOps/ServerProvisioning
to include a way to always copy those to your /usr/local/bin
That way is a debian package :-)
Didn't know if that was over-doing it. ie, you could also just bzr-branch the usr/local/bin directory. I think Antonio's background might help us here :)
If those scripts are useful only for us in the LAVA lab, I think a debian package is overkill, and we could just include them in the upcoming chef recipes. If they are useful for others, then for sure I can package them; it should make be a pretty simple package.
What do you think?
I think that, going forward, at least one team in ARM will be using LAVA (one is already setting it up) and they may want to be able to do something like staging, but I would say the priority is low.
Thanks
Dave
On 10/10/2012 08:07 AM, Antonio Terceiro wrote:
Andy Doan escreveu:
On 10/09/2012 04:17 PM, Michael Hudson-Doyle wrote:
Andy Doan andy.doan@linaro.org writes:
On 7 October 2012 19:33, Michael Hudson-Doyle michael.hudson@linaro.org wrote: and update:
https://wiki.linaro.org/Platform/LAVA/DevOps/ServerProvisioning
to include a way to always copy those to your /usr/local/bin
That way is a debian package :-)
Didn't know if that was over-doing it. ie, you could also just bzr-branch the usr/local/bin directory. I think Antonio's background might help us here :)
If those scripts are useful only for us in the LAVA lab, I think a debian package is overkill, and we could just include them in the upcoming chef recipes. If they are useful for others, then for sure I can package them; it should make be a pretty simple package.
What do you think?
My guess is that there wouldn't be much interest in these scripts. They are specific to our PDU and serial console servers.
linaro-validation@lists.linaro.org
-
Andy Doan -
Antonio Terceiro -
Dave Pigott -
Michael Hudson-Doyle -
Michael Hudson-Doyle