Hello.
Over the past two weeks I've been prototyping a new tool for managing releases of Linaro validation deliverables. Some of the problems are unique (we target older releases, we are upstream) some are shared (building lots of packages together, making sure they all work, reproducing builds).
TLDR; Skip to the bottom of the message where I list the features.
I had some prior experience in this task and I wanted to see how the old solutions would map to the new environment. As we target Ubuntu Lucid aka latest LTS and most of us develop on the latest Ubuntu I wanted to ensure that all of our releases can be installed and would work correctly on Lucid. Earlier in the cycle we decided that the delivery method would be a PPA with multiple .debs, some of which may be backports required for Lucid. This is unlike using pip to pull in all the things we are interested in from various places (possibly using a requirements file).
So having those requirements my goals were to build a tool that would be useful for those two tasks: 1) Daily development helper as a sort of CI tool. 2) Monthly release helper
Both tasks have different requirements but involve similar actions. Those actions are:
*) Figuring out what packages to work on (aka project definition) *) Getting the appropriate source/revision/tag from launchpad *) Building a source packages for the target distribution *) Building a binary package in pbuilder - this is where we also run all the unit tests that tells us if something is broken.
The tricky part is where we need to build test and release *multiple* source packages from *multiple* branches and target *multiple* target distributions.
For some concrete example. LAVA today is composed of the following core packages: - lava-server - lava-dashboard - lava-scheduler - lava-dispatcher - lava-tool - lava-test (-tool) - lava-dashboard-tool - lava-scheduler-tool (upcoming)
We also have a set of support libraries: - versiontools - linaro-django-xmlrpc - linaro-django-pagination - linaro-dashboard-bundle - django-restricted-resource - django-debian - linaro-json
We also have some libraries that are required for testing: - python-django-testproject - python-django-testscenarios - python-django-testinvariants (upcoming)
We also have some backports (for both production and testing): - python-simplejson - python-django 1.2 (soon 1.3) - python-django-south - python-testtools - python-testscenarios - python-fixtures
Now that's a lot of stuff to build and release manually. Granted not everything is changing but at the very least the first group (lava-*) will see a lot of activity each month.
Now the way I see it, each month, this list needs to be released. Possibly some of them will be unchanged. In that case there is no need to actually upload new packages to a PPA. Still we need to ensure that all of them build and work on all Lucid, Maverick and so on, all up to the latest version of Ubuntu.
Since we want to build Debian packages I decided to use bzr builder to create source packages from a recipe files. Recipes are a simple (2-3 lines at most) text files that say how to assemble source branches to get Debian packaging. The key feature of builder is it's ability to create derivative packages for a particular distribution. Recipe files are also used by launchpad for building source packages. In the future this tool could actually push/pull the recipes to launchpad directly.
To build binary packages I used pbuilder. The configuration is a little more complex than simple raw pbuilder or pbuilder-dist. The configuration I made turns it into a ppa-like builder that can feed from it's own packages. Each time you build a package it will land in a small apt repository and can be used to feed another build. This is a critical feature as all of our packages depend on something not available in the main archive.
The trick is to know the right sequence of builds that would satisfy build-dependencies. I did not find any existing tool to do this and after a brief discussion with doko it seems there are no such tools available. In general the problem can be simplified to a subset of real dependencies (without conflicts, virtual packages and stuff like that) and resolved automatically. I did not want implement that as our package list can be managed manually. A special sequence file contains a list of "things" to build, in order, to satisfy dependencies.
So getting everything together, lava-ci does the following:
* FEATURES *
$ ./lava-ci project help Usage: lava-ci project [COMMAND] Summary: Manipulate and release your project
Available commands: - init - Prepare lava-ci to work with a particular project - help - This message (default)
$ ./lava-ci distro help Usage: lava-ci distro [COMMAND] Summary: Manipulate pbuilder environments
Available commands: - help - This message - show - Display list of distributions and their status (default) - enable - Enable one or more distributions - disable - Disable one or more distributions - update - Update one or more distributions
$ ./lava-ci package help Usage: lava-ci package [COMMAND] Summary: Manipulate source and binary packages
Available commands: - srcpkg - Create a source package from a recipe - help - This message - show - Display list of packages and their status (default) - wipe - Remove all source and binary packages - sequence - Build a sequence of packages following - binpkg - Create binary packages from a source package
A subset of the README file:
lava-ci - CI and release tools for the LAVA project (and others).
The purpose of this tool is to facilitate teamwork and monthly releases by automating the act of creating and testing a release from source repositories all the way to the binary packages for various platforms.
Workflow example:
*) Prepare recipe files (bzr help builder) for each maintained source package and put them in a bzr repository. Each recipe file *MUST* be named like the source package it builds.
*) Use `lava-ci project init lp:BRANCH-WITH-RECIPES`. This will create .lava-ci directory and a local checkout of you recipes in the current directory.
*) Use `lava-ci distro` to list the distributions you wish to target. For each one you are interested in do `lava-ci distro enable $distro`. For example to enable lucid and maverick you could use `lava-ci distro enable lucid maverick`.
*) Use `lava-ci package` to list available "packages" (based on recipes). You can add/edit more just by creating additional *.recipe files.
*) Use `lava-ci package src $package` to build source packages for the selected recipe. There will be one source package per distribution. You can check `lava-ci package [show]` to see what source packages are already available.
*) Use `lava-ci package bin $package` to build binary packages for the selected recipe. Again there will be multiple builds, one for each distribution. Each build will result with a number of additional debian packages being produced. You can use those packages as build-dependencies of other packages you maintain, similar to a Lauchpad PPA.
*) Write down a sequence of build operations to perform in and save it as `recipe/sequence`. This file should contain one word per line - the name of the source package to build. The order will depend on the build-dependencies among your packages. To test that building the whole sequence works do `lava-ci package wipe` followed by `lava-ci package sequence`. If the build succeeds you can tag your recipes as a working configuration and make a release.
Best regards Zygmunt Krynicki
So having those requirements my goals were to build a tool that would be useful for those two tasks:
- Daily development helper as a sort of CI tool.
- Monthly release helper
So for purpose 1, if we wanted to use the same tool to make this easy,
could we just do something like 'lava-ci project develop'? It seems to me, that on some things, this could be overkill, or not really make it that much simpler. For instance... in the case of lava-test you may want to also install python-apt, but there are other things like usbutils. In the case of the dispatcher, it can also run completely from the source branch. Except there are things like conmux that you would want to install, and configure to tell it how to get to your test machines. For situations like those, unless you have already thought of it here, I think it's no harder to just provide instructions for creating a development environment. However I'm well aware that other projects such as those dependent on the libraries you mention, and the lava-server piece can be somewhat trickier. How much of a stretch is it to extend this to cover case 1 as you mention, or do you feel for all of these it would be easier to provide a script in-tree to setup virtualenv + instructions for configuration to setup a development environment?
Thanks, Paul Larson
W dniu 15.06.2011 20:30, Paul Larson pisze:
So having those requirements my goals were to build a tool that would be useful for those two tasks:
- Daily development helper as a sort of CI tool.
- Monthly release helper
So for purpose 1, if we wanted to use the same tool to make this easy,
could we just do something like 'lava-ci project develop'?
I was writing it this morning. It actually pulls all of the branches that compose lava and gives you an ability to install them in a managed virtualenv instance.
It seems to me, that on some things, this could be overkill, or not really make it that much simpler. For instance... in the case of lava-test you may want to also install python-apt, but there are other things like usbutils. In the case of the dispatcher, it can also run completely from the source branch.
So there are three possible cases:
1) A component has no dependencies and runs from tree (awesome!) 2) A component depends on something we make 3) A component depends on something others make
Only case 1) is easy to work with. Virtually everything has some dependencies that either differ from one release to another (like python-simplejson incompatibility I ran into when testing on natty some months ago) or have more rich interface on the more recent version (and then it fails to work on lucid). For the rest we need to be able to reliably setup what is required to use and develop our code.
Except there are things like conmux that you would want to install, and configure to tell it how to get to your test machines. For situations like those, unless you have already thought of it here, I think it's no harder to just provide instructions for creating a development environment.
Instructions fail. Reproducible code works.
Reproducible code is also dog-fooding our instructions in practice.
However I'm well aware that other projects such as those dependent on the libraries you mention, and the lava-server piece can be somewhat trickier. How much of a stretch is it to extend this to cover case 1 as you mention, or do you feel for all of these it would be easier to provide a script in-tree to setup virtualenv + instructions for configuration to setup a development environment?
I think this is the core feature required for development and since so much information is already there extending it to support development is a natural consequence.
I will post updates later this evening. I added one new command tree that makes this possible.
Thanks ZK
On Wed, Jun 15, 2011 at 2:56 PM, Zygmunt Krynicki < zygmunt.krynicki@linaro.org> wrote:
I was writing it this morning. It actually pulls all of the branches that compose lava and gives you an ability to install them in a managed virtualenv instance.
Great!
It seems to me,
that on some things, this could be overkill, or not really make it that much simpler. For instance... in the case of lava-test you may want to also install python-apt, but there are other things like usbutils. In the case of the dispatcher, it can also run completely from the source branch.
So there are three possible cases:
- A component has no dependencies and runs from tree (awesome!)
- A component depends on something we make
- A component depends on something others make
But what about where other non-python packages need to be installed for it
to work properly? That was the main thing I was trying to sort out? Does the recipe have a rule to do this? In the case where post-install configuration is needed, does it point the user to instructions for how to do that?
Thanks, Paul Larson
W dniu 15.06.2011 22:49, Paul Larson pisze:
But what about where other non-python packages need to be installed for it
to work properly? That was the main thing I was trying to sort out? Does the recipe have a rule to do this?
For building this knowledge is contained in the source package. Nothing new or special here (we can build all the debs in pbuilder after all).
For development it will be provided by virtualenv (thanks to setup.py meta-data). The only thing that would not be provided are things that are not available on pypi such as usb utils for lava-test. This is a more complicated issue that I did not approach yet. We can observe the extreme when we consider that adb is something we kind-of require sometimes for some "components".
Thanks for pointing that out.
In the case where post-install configuration is needed, does it point the user to instructions for
how to
do that?
None of this is for end users. This is for developers of the lava stack and automatic testing of arbitrary collection of branches of said stack.
We _can_ and _should_ document how to use it in practice though and I plan to do that :-)
Thanks ZK
-
Paul Larson -
Zygmunt Krynicki