Abner Silva abner@collabora.co.uk writes:
Hi Michael,
On Tue, 2012-06-12 at 17:07 +1200, Michael Hudson-Doyle wrote:
Those of you who have been around for a while will know that we used to package LAVA as Debian packages. This had its problems and we switched over to the world of pip and virtualenv and lava-deployment-tool.
Unfortunately, pip and virtualenv have not really given us the control we need to be confident in our deployments, and so I'm proposing to change again, this time to zc.buildout. This is less of a change than the switch away from debs -- instances of LAVA will still be a concept and the lava-deployment-tool script will still be used, at least for now.
The way buildout works is that there is a config file that specifies how -- reproducibly! -- to assemble various pieces into a deployment. I have a branch of lava-server at lp:~mwhudson/lava-server/use-buildout that contains such a config file (and various other related bits and pieces), and a branch of lava-deployment-tool that creates a buildout-based deployment.
An instance on disk in the new world is similar to but not the same as an instance is today. In particular it is not a virtual environment -- although for compatibility, it looks like one. There is still a $LAVA_INSTANCE/bin/activate script that you can source in bash to put that instance's scripts at the front of your $PATH. Inside an instance, the etc, run, tmp and var directories server the same function as they do today. There is also a new directory, code. The code directory contains a number of subdirectories, one for each revno of lava-server that has been installed into that instance, and a symlink 'current' that points to the currently active one.
There are now two merge proposals on LP:
https://code.launchpad.net/~mwhudson/lava-server/use-buildout/+merge/109768 https://code.launchpad.net/~mwhudson/lava-deployment-tool/use-buildout/+merg...
I've made an effort to update the documentation in l-d-t's README, and I've put an htmlified version of this file online at http://people.linaro.org/~mwh/ldt.html so that might be a good place to start.
As I never worked with buildout before I was wondering if there was a discussion around this topic before deciding to adopt this new deployment approach.
There was some internal discussion, but this is the first time it's been mentioned on a mailing list I guess.
I would like to read that and see what was discussed and what are the main advantages of changing how to deploy Lava once again. Is it available somewhere?
No.
Basically, as I've been the only person to do major deployments to v.l.o for several months now, I feel I'm in a pretty good position to make decisions about this! And I hate using the current scripts.
I wanted to spend a little time with buildout before proposing that we use it to make sure it was worth proposing, and probably spent a little too long: I basically implemented everything except migration from existing deployments.
In case this change really happen, I believe that would be nice to have some sort of tutorial teaching how to migrate your current setup and deploy it with buildout, without loosing your database, etc.
_Of course_ we will do this sort of thing (we need it for v.l.o!). Obviously, I don't know all the details of your deployment and so it's going to be a slightly stressful time when the change comes to be made -- I hope you can do lots of testing. I'm also happy to learn about details of your deployment and advise you on how to make the transition.
Also, I remember to have long conversations on IRC with you guys and IIRC there was an agreement that Lava would also be released as deb packages again,
I don't know which conversations you're referring to here, I don't think I was involved. Certainly, in my position as owner of the technical plan, "package server components as debs again" is not on there.
so we would have an option to use pip or debs. How is that going?
It's not.
Please, forgive me if I missed any important discussion.
Well, you weren't on the phone calls Andy and I had to talk about this. I guess it's just part of the fun of open source: I am most motivated by the problems _I_ see, and fragility of deployment was one of those. I should say that nothing I am doing will prevent you from maintaining your own version of lava-deployment-tool that uses virtualenv & pip but as we won't be using this approach ourselves this might be more work for you in the long run.
Cheers, mwh
Michael Hudson-Doyle michael.hudson@canonical.com writes:
Abner Silva abner@collabora.co.uk writes:
Hi Michael,
On Tue, 2012-06-12 at 17:07 +1200, Michael Hudson-Doyle wrote:
Those of you who have been around for a while will know that we used to package LAVA as Debian packages. This had its problems and we switched over to the world of pip and virtualenv and lava-deployment-tool.
Unfortunately, pip and virtualenv have not really given us the control we need to be confident in our deployments, and so I'm proposing to change again, this time to zc.buildout. This is less of a change than the switch away from debs -- instances of LAVA will still be a concept and the lava-deployment-tool script will still be used, at least for now.
The way buildout works is that there is a config file that specifies how -- reproducibly! -- to assemble various pieces into a deployment. I have a branch of lava-server at lp:~mwhudson/lava-server/use-buildout that contains such a config file (and various other related bits and pieces), and a branch of lava-deployment-tool that creates a buildout-based deployment.
An instance on disk in the new world is similar to but not the same as an instance is today. In particular it is not a virtual environment -- although for compatibility, it looks like one. There is still a $LAVA_INSTANCE/bin/activate script that you can source in bash to put that instance's scripts at the front of your $PATH. Inside an instance, the etc, run, tmp and var directories server the same function as they do today. There is also a new directory, code. The code directory contains a number of subdirectories, one for each revno of lava-server that has been installed into that instance, and a symlink 'current' that points to the currently active one.
There are now two merge proposals on LP:
https://code.launchpad.net/~mwhudson/lava-server/use-buildout/+merge/109768 https://code.launchpad.net/~mwhudson/lava-deployment-tool/use-buildout/+merg...
I've made an effort to update the documentation in l-d-t's README, and I've put an htmlified version of this file online at http://people.linaro.org/~mwh/ldt.html so that might be a good place to start.
As I never worked with buildout before I was wondering if there was a discussion around this topic before deciding to adopt this new deployment approach.
There was some internal discussion, but this is the first time it's been mentioned on a mailing list I guess.
I would like to read that and see what was discussed and what are the main advantages of changing how to deploy Lava once again. Is it available somewhere?
No.
Basically, as I've been the only person to do major deployments to v.l.o for several months now, I feel I'm in a pretty good position to make decisions about this! And I hate using the current scripts.
I wanted to spend a little time with buildout before proposing that we use it to make sure it was worth proposing, and probably spent a little too long: I basically implemented everything except migration from existing deployments.
In case this change really happen, I believe that would be nice to have some sort of tutorial teaching how to migrate your current setup and deploy it with buildout, without loosing your database, etc.
_Of course_ we will do this sort of thing (we need it for v.l.o!). Obviously, I don't know all the details of your deployment and so it's going to be a slightly stressful time when the change comes to be made -- I hope you can do lots of testing. I'm also happy to learn about details of your deployment and advise you on how to make the transition.
I've updated my lava-deployment-tool branch to cover this, and updated the docs a bit to describe the process:
http://people.linaro.org/~mwh/ldt.html#upgrading-from-a-pip-based-instance-t...
Cheers, mwh
Also, I remember to have long conversations on IRC with you guys and IIRC there was an agreement that Lava would also be released as deb packages again,
I don't know which conversations you're referring to here, I don't think I was involved. Certainly, in my position as owner of the technical plan, "package server components as debs again" is not on there.
so we would have an option to use pip or debs. How is that going?
It's not.
Please, forgive me if I missed any important discussion.
Well, you weren't on the phone calls Andy and I had to talk about this. I guess it's just part of the fun of open source: I am most motivated by the problems _I_ see, and fragility of deployment was one of those. I should say that nothing I am doing will prevent you from maintaining your own version of lava-deployment-tool that uses virtualenv & pip but as we won't be using this approach ourselves this might be more work for you in the long run.
Cheers, mwh
linaro-validation@lists.linaro.org
-
Michael Hudson-Doyle -
Michael Hudson-Doyle