Hello.
Over the past two weeks I've been prototyping a new tool for managing releases of Linaro validation deliverables. Some of the problems are unique (we target older releases, we are upstream) some are shared (building lots of packages together, making sure they all work, reproducing builds).
TLDR; Skip to the bottom of the message where I list the features.
I had some prior experience in this task and I wanted to see how the old solutions would map to the new environment. As we target Ubuntu Lucid aka latest LTS and most of us develop on the latest Ubuntu I wanted to ensure that all of our releases can be installed and would work correctly on Lucid. Earlier in the cycle we decided that the delivery method would be a PPA with multiple .debs, some of which may be backports required for Lucid. This is unlike using pip to pull in all the things we are interested in from various places (possibly using a requirements file).
So having those requirements my goals were to build a tool that would be useful for those two tasks: 1) Daily development helper as a sort of CI tool. 2) Monthly release helper
Both tasks have different requirements but involve similar actions. Those actions are:
*) Figuring out what packages to work on (aka project definition) *) Getting the appropriate source/revision/tag from launchpad *) Building a source packages for the target distribution *) Building a binary package in pbuilder - this is where we also run all the unit tests that tells us if something is broken.
The tricky part is where we need to build test and release *multiple* source packages from *multiple* branches and target *multiple* target distributions.
For some concrete example. LAVA today is composed of the following core packages: - lava-server - lava-dashboard - lava-scheduler - lava-dispatcher - lava-tool - lava-test (-tool) - lava-dashboard-tool - lava-scheduler-tool (upcoming)
We also have a set of support libraries: - versiontools - linaro-django-xmlrpc - linaro-django-pagination - linaro-dashboard-bundle - django-restricted-resource - django-debian - linaro-json
We also have some libraries that are required for testing: - python-django-testproject - python-django-testscenarios - python-django-testinvariants (upcoming)
We also have some backports (for both production and testing): - python-simplejson - python-django 1.2 (soon 1.3) - python-django-south - python-testtools - python-testscenarios - python-fixtures
Now that's a lot of stuff to build and release manually. Granted not everything is changing but at the very least the first group (lava-*) will see a lot of activity each month.
Now the way I see it, each month, this list needs to be released. Possibly some of them will be unchanged. In that case there is no need to actually upload new packages to a PPA. Still we need to ensure that all of them build and work on all Lucid, Maverick and so on, all up to the latest version of Ubuntu.
Since we want to build Debian packages I decided to use bzr builder to create source packages from a recipe files. Recipes are a simple (2-3 lines at most) text files that say how to assemble source branches to get Debian packaging. The key feature of builder is it's ability to create derivative packages for a particular distribution. Recipe files are also used by launchpad for building source packages. In the future this tool could actually push/pull the recipes to launchpad directly.
To build binary packages I used pbuilder. The configuration is a little more complex than simple raw pbuilder or pbuilder-dist. The configuration I made turns it into a ppa-like builder that can feed from it's own packages. Each time you build a package it will land in a small apt repository and can be used to feed another build. This is a critical feature as all of our packages depend on something not available in the main archive.
The trick is to know the right sequence of builds that would satisfy build-dependencies. I did not find any existing tool to do this and after a brief discussion with doko it seems there are no such tools available. In general the problem can be simplified to a subset of real dependencies (without conflicts, virtual packages and stuff like that) and resolved automatically. I did not want implement that as our package list can be managed manually. A special sequence file contains a list of "things" to build, in order, to satisfy dependencies.
So getting everything together, lava-ci does the following:
* FEATURES *
$ ./lava-ci project help Usage: lava-ci project [COMMAND] Summary: Manipulate and release your project
Available commands: - init - Prepare lava-ci to work with a particular project - help - This message (default)
$ ./lava-ci distro help Usage: lava-ci distro [COMMAND] Summary: Manipulate pbuilder environments
Available commands: - help - This message - show - Display list of distributions and their status (default) - enable - Enable one or more distributions - disable - Disable one or more distributions - update - Update one or more distributions
$ ./lava-ci package help Usage: lava-ci package [COMMAND] Summary: Manipulate source and binary packages
Available commands: - srcpkg - Create a source package from a recipe - help - This message - show - Display list of packages and their status (default) - wipe - Remove all source and binary packages - sequence - Build a sequence of packages following - binpkg - Create binary packages from a source package
A subset of the README file:
lava-ci - CI and release tools for the LAVA project (and others).
The purpose of this tool is to facilitate teamwork and monthly releases by automating the act of creating and testing a release from source repositories all the way to the binary packages for various platforms.
Workflow example:
*) Prepare recipe files (bzr help builder) for each maintained source package and put them in a bzr repository. Each recipe file *MUST* be named like the source package it builds.
*) Use `lava-ci project init lp:BRANCH-WITH-RECIPES`. This will create .lava-ci directory and a local checkout of you recipes in the current directory.
*) Use `lava-ci distro` to list the distributions you wish to target. For each one you are interested in do `lava-ci distro enable $distro`. For example to enable lucid and maverick you could use `lava-ci distro enable lucid maverick`.
*) Use `lava-ci package` to list available "packages" (based on recipes). You can add/edit more just by creating additional *.recipe files.
*) Use `lava-ci package src $package` to build source packages for the selected recipe. There will be one source package per distribution. You can check `lava-ci package [show]` to see what source packages are already available.
*) Use `lava-ci package bin $package` to build binary packages for the selected recipe. Again there will be multiple builds, one for each distribution. Each build will result with a number of additional debian packages being produced. You can use those packages as build-dependencies of other packages you maintain, similar to a Lauchpad PPA.
*) Write down a sequence of build operations to perform in and save it as `recipe/sequence`. This file should contain one word per line - the name of the source package to build. The order will depend on the build-dependencies among your packages. To test that building the whole sequence works do `lava-ci package wipe` followed by `lava-ci package sequence`. If the build succeeds you can tag your recipes as a working configuration and make a release.
Best regards Zygmunt Krynicki