I thought this would be a simple question about integrating xdeb with sbuild but as it turns out there are other things here that could benefit from being discussed with a bigger audience, so I'm CCing linaro-dev.
On Wed, 2010-11-10 at 21:30 +0000, Wookey wrote:
+++ Guilherme Salgado [2010-11-10 17:49 -0200]:
Hi Wookey,
As I've mentioned on IRC, we want to have sbuild do cross builds using xdeb so that we can use it in https://wiki.linaro.org/Platform/Infrastructure/Specs/PackageDevelopmentTools.
Oh, that looks interesting. I'm very keen to have a cross-autobuilder churning through things to find out how much stuff actually cross-builds at the moment.
In fact reading that work thing you seem to be wanting to build exactly what I want to build: a system that repeatably builds packages in pristine chroots. I thought sbuild didn't normally do the pristine chroot thing? (which is why I've been using pbuilder to supply that functionality to date).
Yeah, I don't think sbuild itself provides that, so I'm using schroot's type=file chroots for that. You just point schroot to a tarball containing your chroot and it will (by default) create a new session (with a pristine chroot) every time you 'schroot -c <name>'. If you want to change the contents of the tarball you can easily do that with 'schroot -c <name>-source'. All you need for that to work is the following config:
[natty] description=Natty chroot type=file file=/var/chroots/natty.tar location=/natty
If you look at my (Still developing) cross-building spec: https://wiki.linaro.org/Platform/Foundations/Specs/CrossBuilding you'll see that I have very similar ideas, but I want to have tools to do the various aspects of this job (satisfy-cross-deps, upload things, download things, do build-ordering), rather than the unhelpfully monolithic xdeb.
It's certainly a good idea to have separate tools (or maybe even libraries?) for the various aspects, and although at first that didn't seem necessary to reach our goals in PackageDevelopmentTools, I think it's clearly becoming more important now as we find opportunities to collaborate.
Do you realise that pdebuild-cross (in maverick) already makes a chroot to do cross-building in with a single command: 'pdebuild-cross-create'?
I didn't know about that; I'm still finding my way around all these tools for cross-building debs. :)
It installs a base system plus crosstoolchains plus build-essential. (defaults to -aarmel). Currently there is just a config for maverick, but we can add some more if we want to have natty and natty+ppearses flavoured toolchains. That 'make me a cross-chroot' stuff hides the bits you detail in "Notes on using sbuild+xdeb to do cross compilation" on having to make sure source lines are put in and cross-tools installed and so on. It's all done already :-)
That sounds quite nice, and I'm all for not reinventing the wheel, so I'll have a look at pdebuild-cross.
I just ran pdebuild-cross-create and that seems to create the chroot and tar it up, so even if we go with sbuild I should be able to use pdebuild-cross to create the chroots.
That actually uses multistrap to do the hard work of setting up the chroots with the right stuff. This is useful because debootstrap can't take a base system from here and tool from there - everything has to come from one place (although we could probably fix that). multistrap has a neat cascading config scheme for making chroots so you can say 'maverick + armel-cross-build + this-toolchain'. This is very handy in the real world where the parts you need are not necessarily (yet) in the distro.
After learning and experimenting a bit, this is what I came up with. I have some specific questions further down, but any feedback you might have is appreciated.
I am nervous about sbuild+xdeb. It depends exactly what you are trying to do, but xdeb's caching and recursive build behaviour, whilst useful in some ways, also means that builds are not repeatable unless you either go to special effort to clean things our or do the builds in a disposable chroot. (pbuilder gives you the disposable thing for free, at the expense of the time it takes to unpack and install deps each time). I want a cross-builder to the same thing each time I ask it to build a package. xdeb doesn't help at all with that as currently designed. Ah, I see you want that too and propose schroot snapshots+sessions. That's cool. I've used schroot a lot for managing cross-build chroots in the past, but that was persistent ones, not snapshotted pristine ones. You'll have to show me how that works. We appear to agree completely on the principles of the thing :-)
And I hope we can agree on the tools as well. The reason why I went with sbuild is because I was told it'd be more suitable for this job than pbuilder, and one thing I liked about it is that it's got most of its functionality packaged in a library (although it's written in perl) rather than a single tool. Is pbuilder like that as well?
I still haven't worked out why everyone is so enamoured of xdeb, and keep trying to design things round it. It has some useful features,
I can't answer that, but I can speculate: - it does the job - most Linaro docs I've seen about cross building use it - we have in-house expertise on the tool - it's written in python
but it was designed as a bootstraping-from-local-sources tool, which is not the same thing at all as a tool for cross-building individual packages in a repository context. The clever bit of having heuristics to supply deps is better dealt with by actually fixing the package metadata, and for packages where that's not yet been done (arguably)
That's what MultiArch is about, right? MultiArch seems to be our long-term plan, but in the meantime we have to use something else. That's where xdeb becomes useful, AIUI.
to just brute-force install everything native _and_ cross, which works just fine and is much easier to understand. (it is slower though - I'm not sure how much in practice, or how much it really matters).
Why 'larc' BTW?
This tool is expected to grow into something that will help developers manage .deb archives as well, so it was named Linaro Archive Creator.
https://wiki.ubuntu.com/Specs/M/ARMDeveloperEnvironment has more details about what the tool should do.
We don't necessarily expect this functionality to be desirable upstream, in which case we'd need a plan to apply them at runtime so that we don't have to maintain a fork of sbuild. If you think the functionality may be wanted upstream, I'm sure we'll work together with them to figure out the best way to integrate it. Otherwise, following is one way we could approach it
sbuild's build function (lib/Sbuild/Build.pm, part of libsbuild-perl) delegates the package building to dpkg-buildpackage and does a lot of sanity checks that we might not need or may not make sense for our use cases, so we could replace the whole function with a custom implementation which delegates the (cross-)building to xdeb.
- Is that ok?
- Should xdeb be used only for cross builds?
- Is it possible to replace (monkey-patching or similar?) that function at runtime with our own implementation of it? (I realize this is more of a perl question, but I know close to nothing about perl)
I have never looked at sbuild so have no idea what the right way to do this is.
Err, I thought you would know because Steve L. told me to talk to you if I wanted to discuss implementation details. Anyway, it's good that I did because otherwise we wouldn't have found this overlap in our work.
Also, sbuild uses a (configurable) dependency resolver to figure out what build deps to install, but AIUI xdeb handles that, so I'd change sbuild to skip that step completely (by using a new dependency resolver that does nothing).
This is exactly the problem I have with trying to jam xdeb in for these jobs. It's too high-level. We should have a cross-dep-resolving tool that _just_ does that and make sbuild use it. (xapt is a simple-minded one, xdeb contains an implementation of a cleverer one but you can't use it on it's own). This would result in simple, maintainable, tools that can be used in various contexts.
xdeb seems to be factored into small functions already, so it should be easy to move that functionality into a library or a separate tool. I suspect it'd be the same for the rest of xdeb as well.
Are there any other steps currently performed by sbuild that can be avoided in our case? Or anything that I've overlooked? Or a better way of doing it?
I think there is a better way, as largely elaborated above: give sbuild a cross-dep resolver, use apt to install things of the appropriate arch, passed through dpkg-cross if no multiarch version of package available. And just issue dpkg-buildpackage -a<arch> to do cross-builds.
That sounds fine as well, and it's definitely more appealing given that we should be able to share at least parts of the new tools/libraries we develop.
I could be wrong and there is a reason why all this won't fit together, but I haven't seen one yet.
http://paste.ubuntu.com/529453/ is a diff against maverick's sbuild, along the lines of what I described above (although dirty and full of hard-coded bits). This diff allowed me to use sbuild to cross build expat for armel.
We seem to be on very much the same page, which is great. Quite a lot of what I've been putting in my spec is actually about how to implement your spec properly, it seems. Both of us seem keen on the 'use existing tools by sticking them together' where each tool does one job well. I think if we fish out the cross-dep resolver and installer logic into a separate tool then we can do all sorts of things in a satisfying way. We also need cross-dep metadata in packages for any tool to get it right. And if we had stage-build
Do you mean to fish them both out of xdeb or do you see the installer logic coming from somewhere else? I could have a look at extracting them from xdeb, just to have an idea of how much work it'd involve.
metadata in packages then we could use xdeb's build-ordering logic to bootstrap new flavours or even ports. Exactly how that works needs a bit more thought, but it'd be good to keep it in mind.
Does that all sounds reasonable to you? I'm trying to work out what the
It does!
optimal path is and where to derive the code from because there is strong pressure to do this quickly as well as properly :-)
On Thu, 11 Nov 2010 17:32:06 -0200, Guilherme Salgado salgado@canonical.com wrote:
Oh, that looks interesting. I'm very keen to have a cross-autobuilder churning through things to find out how much stuff actually cross-builds at the moment.
In fact reading that work thing you seem to be wanting to build exactly what I want to build: a system that repeatably builds packages in pristine chroots. I thought sbuild didn't normally do the pristine chroot thing? (which is why I've been using pbuilder to supply that functionality to date).
We want to enable people to testbuild packages in a chroot. That may be a building block that you can make use of to set up an autobuilder.
Yeah, I don't think sbuild itself provides that, so I'm using schroot's type=file chroots for that. You just point schroot to a tarball containing your chroot and it will (by default) create a new session (with a pristine chroot) every time you 'schroot -c <name>'. If you want to change the contents of the tarball you can easily do that with 'schroot -c <name>-source'. All you need for that to work is the following config:
[natty] description=Natty chroot type=file file=/var/chroots/natty.tar location=/natty
That's one way of using it. I believe that if you just use a plain chroot type then sbuild will attempt to remove the build-dependencies it installed at the end of its run (obviously this is less reliable).
I think there is a better way, as largely elaborated above: give sbuild a cross-dep resolver, use apt to install things of the appropriate arch, passed through dpkg-cross if no multiarch version of package available. And just issue dpkg-buildpackage -a<arch> to do cross-builds.
That sounds fine as well, and it's definitely more appealing given that we should be able to share at least parts of the new tools/libraries we develop.
sbuild already understands something about architectures, as you can e.g. do -ai386 on an amd64 system, and it will do the right thing. Perhaps it just needs this extending to handle cases where it should cross-build, and use an appropriate dep-solving algorithm and toolchain.
Also of interest is
http://lists.debian.org/debian-devel/2010/11/msg00170.html
Would that change allow us to piggyback on any other work for the cross-dep-solving, or would that basically require multi-arch to be implemented?
Thanks,
James
On Thu, Nov 11, 2010, Guilherme Salgado wrote:
In fact reading that work thing you seem to be wanting to build exactly what I want to build: a system that repeatably builds packages in pristine chroots. I thought sbuild didn't normally do the pristine chroot thing? (which is why I've been using pbuilder to supply that functionality to date).
Yeah, I don't think sbuild itself provides that, so I'm using schroot's type=file chroots for that. You just point schroot to a tarball containing your chroot and it will (by default) create a new session (with a pristine chroot) every time you 'schroot -c <name>'. If you want to change the contents of the tarball you can easily do that with 'schroot -c <name>-source'. All you need for that to work is the following config:
[natty] description=Natty chroot type=file file=/var/chroots/natty.tar location=/natty
Seems it's actually on topic, I'd love sharing my setup! :-)
To setup, I use this schroot.conf snippet: [natty] type=file description=Ubuntu natty root-users=lool file=/srv/schroot/chroot-ubuntu-natty-amd64.tar.bz2 location=/chroot-autobuild preserve-environment=true
/srv/schroot/chroot-ubuntu-natty-amd64.tar.bz2 is the chroot used on Launchpad buildds for natty/amd64. I'm attaching dl-lp-chroot which downloads such a chroot; it's a trivial wrapper around https://launchpad.net/api/devel/ubuntu/%24dist/%24arch/chroot_url URLs
To use, I then do something like: mkdir ~/xdeb-dpkg cd ~/xdeb-dpkg schroot-xdeb --apt-source --only-explicit dpkg -aarmel
(the above are just whatever xdeb args I'd like to use)
schroot-xdeb will install xdeb and a cross-toolchain; it's sadly hardcodes my local mirror, but could obviously be adapted
I should probably implement a feature to wait for confirmation before destroying the temporary chroot on failure.
PS: the above run will fail because xdeb doesn't import selinux for some reason; but it does illustrate things quite well here :-)
+++ Guilherme Salgado [2010-11-11 17:32 -0200]:
I thought this would be a simple question about integrating xdeb with sbuild but as it turns out there are other things here that could benefit from being discussed with a bigger audience, so I'm CCing linaro-dev.
Good idea. I've been planning to send a 'the state of cross building at the end of this cycle' mail very soon (once specs are done with), to say what we have so far, what works and what doesn't. (before we mess with things again :-)
On Wed, 2010-11-10 at 21:30 +0000, Wookey wrote:
+++ Guilherme Salgado [2010-11-10 17:49 -0200]:
Hi Wookey,
That sounds quite nice, and I'm all for not reinventing the wheel, so I'll have a look at pdebuild-cross.
I just ran pdebuild-cross-create and that seems to create the chroot and tar it up, so even if we go with sbuild I should be able to use pdebuild-cross to create the chroots.
OK, glad that works for you too.
I guess if we decide not to use pbuilder at all but use schroot+sbuild instead then we should call the relevant chroot-creation foo (which is really multistrap config files and a command like multistrap -f /usr/share/ubuntu/armel.conf) from somewhere else (if you look pdebuild-cross-create is very thin veneer). This is a good example of why tools to do each part well is good - because it becomes dead easy to support the use of pbuilder or sbuild, as required.
We appear to agree completely on the principles of the thing :-)
And I hope we can agree on the tools as well. The reason why I went with sbuild is because I was told it'd be more suitable for this job than pbuilder, and one thing I liked about it is that it's got most of its functionality packaged in a library (although it's written in perl) rather than a single tool. Is pbuilder like that as well?
pbuilder is all shell, but seems quite well documented and put together. I don't have strong opinions about it - we used it because it was there and it was easy to do, and I was familiar with its operation mode.
it does have minus points: One disadvantage it has is that a 'make clean' is done outside the chroot in order to generate a pristine source package, and thus you need any deps that task needs installed outside. That doesn't seem easily automable (because the package doesn't tell you which of its deps are used by make clean, although in practice a small number of packages (10?) seems to cover many cases.
The other thing I haven't yet found out how to do is to choose which tools are used inside the chroot. You can have different hook-scripts easily for different tools (e.g. building with xdeb or building with xapt+dpkg-buildpackage), but it always wants to run all of them and I don't see an easy mechanism for saying 'just run these this time' (without copying different subsets in yourself). No idea if sbuild makes this any easier or not. In actual use you don't need this sort of choice anyway if you have a tool-set that works, but it's handy for experimenting.
I still haven't worked out why everyone is so enamoured of xdeb, and keep trying to design things round it. It has some useful features,
I can't answer that, but I can speculate:
- it does the job
- most Linaro docs I've seen about cross building use it
- we have in-house expertise on the tool
- it's written in python
That last is a big minus point from my POV :-) I've had to learn python to change anything in it and I'm still rubbish at it, which doesn't help speed development. I'm much happier messing with things in perl and shell. I'm not 'ist' about these languages - it's just much easier actually get things done in ones you already know.
but it was designed as a bootstraping-from-local-sources tool, which is not the same thing at all as a tool for cross-building individual packages in a repository context. The clever bit of having heuristics to supply deps is better dealt with by actually fixing the package metadata, and for packages where that's not yet been done (arguably)
That's what MultiArch is about, right?
Not exactly, but a feature of the way it is implemented provides the metadata in dependencies that we need. And handily we can add this without actually converting the package to multiarch so we can do a whole pile of that easily.
MultiArch seems to be our long-term plan, but in the meantime we have to use something else. That's where xdeb becomes useful, AIUI.
It is, although my tests so far show that the tens-of-times-simpler xapt seems to work just as well (slightly better in fact because it doesn't barf on some packages) for cross-dependency satisfaction.
I have never looked at sbuild so have no idea what the right way to do this is.
Err, I thought you would know because Steve L. told me to talk to you if I wanted to discuss implementation details. Anyway, it's good that I did because otherwise we wouldn't have found this overlap in our work.
Indeed. And I've been meaning to look at sbuild for ages anyway as well as this job, so this is as good a time as any.
Also, sbuild uses a (configurable) dependency resolver to figure out what build deps to install, but AIUI xdeb handles that, so I'd change sbuild to skip that step completely (by using a new dependency resolver that does nothing).
This is exactly the problem I have with trying to jam xdeb in for these jobs. It's too high-level. We should have a cross-dep-resolving tool that _just_ does that and make sbuild use it. (xapt is a simple-minded one, xdeb contains an implementation of a cleverer one but you can't use it on it's own). This would result in simple, maintainable, tools that can be used in various contexts.
xdeb seems to be factored into small functions already, so it should be easy to move that functionality into a library or a separate tool. I suspect it'd be the same for the rest of xdeb as well.
I certainly hope so, and that's the general plan.
The important thing to get right is the interface points so that things fit together properly, and use the right (default) configs. I'm still working that out.
We seem to be on very much the same page, which is great. Quite a lot of what I've been putting in my spec is actually about how to implement your spec properly, it seems. Both of us seem keen on the 'use existing tools by sticking them together' where each tool does one job well. I think if we fish out the cross-dep resolver and installer logic into a separate tool then we can do all sorts of things in a satisfying way. We also need cross-dep metadata in packages for any tool to get it right. And if we had stage-build
Do you mean to fish them both out of xdeb or do you see the installer logic coming from somewhere else? I could have a look at extracting them from xdeb, just to have an idea of how much work it'd involve.
Fishing them out of xdeb seems the most obvious course so that's what's planned, (but I reserve the right to decide that it's actually easier to use other things than separate out parts of xdeb without breaking it for its existing uses). I'd be very happy to input on that from someone with better python-foo.
It does make a lot of sense to have a load of library functions and data structures that the various aspects can use.
In all of this I am also conscious that both xdeb and pdebuild-cross+xapt do already work tolerably well for some of things we want to do, and I'd actually like to get some of that stuff done (ALIP flavoured builds, basic multiarch cross-dep support) _before_ we take a big chainsaw to the tools and break everything again. Hopefully we can do both in parallel with a bit of care.
Wookey
-
Guilherme Salgado -
James Westby -
Loïc Minier -
Wookey