On Sat, 2 Apr 2011 05:47:47 +0200 Guillem Jover guillem@debian.org wrote:

Adding linaro people and debian-embedded to CC:

Hi!

On Fri, 2011-04-01 at 22:31:07 +0100, Neil Williams wrote:

...

dpkg cannot be executed inside the chroot because it has not necessarily been unpacked at this stage, it certainly has not been configured. (dpkg is running from outside the chroot).

(An Essential package does not need to be configured to be functional.)

(It's quite possibly the wrong architecture inside the chroot, so this doesn't apply.)

...

Yet dpkg-query won't work with --admindir to operate from outside:

I'm checking on this - it could be an artefact of how the chroot was initially created.

...

Anyways:

...

Multi-Arch: same packages will use /var/lib/dpkg/info/<package>:<arch>.<something>

...

For Multistrap, every file in /var/lib/dpkg/ has to be created by multistrap, based on the data obtained from dpkg -e (basically the DEBIAN/ content). This data needs only to be sufficient that dpkg can correctly configure the packages once dpkg itself is able to be executed inside the new filesystem.

It's sufficient, you just need to know the value of the Multi-Arch field.

True. I've got a change now which uses dpkg -f to check the Multi-Arch field of the actual downloaded .deb (independent of arch) and use that to detect Multi-Arch: same, then lookup Architecture in the same way, add that to the filename of the created maintainer script etc. for /path/to/chroot/var/lib/dpkg/info/*

If you are not going to install multiple instances of “Multi-Arch: same” packages, then just keep writting the files as you are currently doing, the next dpkg native run on that chroot image will automatically upgrade the db layout. Which seems to be the less onerous way to handle this. Otherwise you'll have to duplicate the db layout handling, including format file, etc, and at that point I'd say you are on your own.

Multistrap will need to be able to create chroots containing multiple instances of Multi-Arch: same packages because it needs to be able to build cross-building chroots with libc6-dev of the build and host architectures, as well as other common dependencies like libglib2.0-0 (needed as build arch by pkg-config and as host arch by everything else).

Multistrap, like debootstrap and other similar scripts, is commonly used in two roles - to create a chroot of the build architecture for building stuff in a contained environment and creating a chroot of a foreign architecture to act as a root filesystem.

As with debootstrap, I see no reason to implement two different methods to creating these chroots inside a single tool, so when creating a build arch chroot multistrap uses the same processing as when creating a foreign arch chroot. That processing has to avoid a design flaw in dpkg where the normal --unpack method would attempt to run the preinst scripts.

On Fri, 2011-04-01 at 22:43:35 +0100, Neil Williams wrote:

...

Further to this, --control-path doesn't include 'list' but $pkg:$arch.list files are created.

This was done on purpose, .list and .conffiles are internal files, the correct interfaces to access the information is dpkg-query.

Internal files maybe, but multistrap still needs to create them to handle packages of a foreign architecture.

Even once Multi-Arch is fully deployed, this will continue because the assumption with Multi-Arch is that the foreign architecture packages are being installed alongside the build architecture. When preparing a root filesystem for a foreign architecture (e.g. an embedded device), multistrap cannot put the amd64 binaries into the chroot and then add the armel, it needs to be able to unpack armel binaries into an armel chroot without running the preinst scripts or using the armel dpkg binary inside the chroot. The preinst scripts can't run on the build arch (amd64) because that would expose the chroot to the external configuration. i.e. it needs to be possible to create an armel chroot of Ubuntu natty from an i386 machine running Debian Wheezy and similarly from Ubuntu natty i386 to create a Debian Wheezy armel chroot. Currently, dpkg --unpack cannot cope with this and the existing Multi-Arch changes in dpkg do not address this problem.

This has been covered before in this bug and elsewhere without resolution, but please reconsider how dpkg - even in a Multi-Arch world - needs to support unpacking foreign architecture binaries into a foreign architecture chroot and that this mandates that dpkg *must not* attempt to execute the preinst scripts inside the foreign chroot. It can *only* do so if the packages are to be unpacked into the main, external, system.

When using --admindir and --instdir options to --unpack, dpkg *must* allow for the specified directories to actually be a foreign architecture chroot and act accordingly. Maybe this requires the addition of an --arch option to --unpack but it *does* need to be addressed. The files being created in /var/lib/dpkg/ are not architecture-dependent, it's only dpkg which needs to understand that when creating a foreign architecture chroot, the external dpkg must not execute the preinst scripts. That is all that is required.

Working out how to configure the resulting system (probably by running the preinst scripts manually with the install argument prior to running dpkg --configure -a) and whether to use an emulator or after booting, is left to the admin of the device. dpkg doesn't need to care, it simply needs to "do the right thing" when unpacking.

If this doesn't change, I will simply have to continue reimplementing dpkg behaviour in multistrap and elsewhere, causing this inconsistency to continue forever. That is not helpful. Let's solve this properly at this ideal opportunity when Multi-Arch is still in final development.

I don't need to be "on my own" on this one (and I'm not anyway because a lot of people need to have this behaviour from dpkg or from some other tool which can hack around dpkg to avoid this design flaw).

This can't be swept under the carpet again, it needs to be addressed. dpkg --unpack NEEDS to understand that when dealing with a foreign architecture chroot it CANNOT execute the preinst scripts.

I will escalate this issue further if the dpkg team claim that this functionality is not possible to implement. Now is the perfect time to fix this long standing design flaw in dpkg. On the surface, it seems a simple thing to setup: if an --arch option is used alongside --admindir and --instdir to the --unpack command, then the preinst scripts are never executed. It doesn't have to be an option called --arch (although that is probably the easiest for embedded developers to find in the dpkg man page), it could be "--no-preinst" or something.

It is just plain wrong that, even with Multi-Arch, dpkg cannot be used to bootstrap a foreign architecture into a chroot and that people like me have to invent a set of hacks to duplicate what dpkg should do itself. I've done this enough times already.