tcl8.5-dev contains /usr/lib/tcl8.5/tclConfig.sh which is a symlink pointing to /usr/share/tcltk/tcl8.5/tclConfig.sh
Currently dpkg-cross only copies symlinks across when crossing a package if the symlink points to somewhere within /lib. symlinks outside /lib are ignored because they are not arch-dependent so shouldn't need to be crossed.
The file /usr/lib/tcl8.5/tclConfig.sh is used by packages which build extensions to tcl (such as sqlite), so that files does need to be present.
The file is essentially a cache of the config options TCL was built with so that extensions can make sure they match up. However it is not arch-independent. e.g. Line 22 of that file is TCL_CC='x86_64-linux-gnu-gcc' on the x86_64 version and TCL_CC='arm-linux-gnueabi-gcc' on the armel version. Other options also differ between 32-bit and 64-bit arches for example.
So this file is arch-dependent in that it's different for each arch it is built-on but arch-independent in that it's just a shell file.
Debian policy (8.2) says: --------------- It is recommended that supporting files and run-time support programs that do not need to be invoked manually by users, but are nevertheless required for the package to function, be placed (if they are binary) in a subdirectory of /usr/lib, preferably under /usr/lib/package-name. If the program or file is architecture independent, the recommendation is for it to be placed in a subdirectory of /usr/share instead, preferably under /usr/share/package-name. Following the package-name naming convention ensures that the file names change when the shared object version changes.
Files and support programs only useful when compiling software against the library should be included in the development package for the library --------------- A maintainer reading the above could reasonably decide that /usr/share was the right place for this file, because the file itself (being just shell) is arch-independent. The problem is that the contents are arch-dependent. Policy is not at all clear on this distinction (It's been making my head hurt all day). For multiarch, or existing dpkg-cross cross-compiling, to work, arch-dependent needs to mean either form _or_ content (see below for elaboration).
The original tcl build puts both tclConfig.sh and tcl.m4 in /usr/lib but the debian packaging moves them to /usr/share/tcltk/tcl8.5 and /usr/share/aclocal/ respectively
Currently the sqlite build fails because it looks for tclConfig.sh in /usr/<triplet>/lib/tcl8.5/ and doesn't find it there, because dpkg-cross didn't copy the symlink (or original) there.
So, the questions is - which of these is the correct fix: 1) make dpkg-cross copy over symlinks even when they point into /usr/share 2) make tcl8.5 put tclConfig.sh in /usr/lib/tcl8.5 instead of /usr/share/tcltk/tcl8.5 3) make sqlite (and similar packages) look in /usr/share/tcltk/tcl8.5 instead of /usr/lib/tcl8.5 when building
I note that the supplied tcl8.5.m4 file actually lists a whole pile of possible locations for the file: for i in `ls -d ${libdir} 2>/dev/null` \ `ls -d ${exec_prefix}/lib 2>/dev/null` \ `ls -d ${prefix}/lib 2>/dev/null` \ `ls -d /usr/local/lib 2>/dev/null` \ `ls -d /usr/contrib/lib 2>/dev/null` \ `ls -d /usr/share/tcltk/tk8.5 2>/dev/null` \ `ls -d /usr/lib 2>/dev/null` \ `ls -d /usr/lib64 2>/dev/null` \
However that list doesn't include anything ending in tcl8.5 (i.e /usr/<triplet>/lib/tcl8.5 or /usr/share/tcltk/tcl8.5) and perhaps should by way of 'back-up plan', although I haven't actually looked into the detals of how that is used.
Consideration when deciding how to fix this include: Squeeze will be released in a few days with this broken so we will be stuck with the /usr/share/ file location for some time there. Any fix going into Ubuntu should make Natty easily enough.
For Multiarch tcl-dev will be a Multiarch:same package (i.e a 'library' package) (despite the name, it contains nothing but headers, libraries and the above config.sh and m4 files), so the two different tclConfig.sh files supplied by the DEB_BUILD_ARCH and DEB_HOST_ARCH packages (when cross-building) will clash and the 2nd package will not be installable. This, it seems to me, is the clinching argument that the correct fix is to change the tcl build to put these files in /usr/lib.
This also implies that policy 8.2 needs to be clarified to explain what 'arch-independent' means.
Are there other situations which expect to find the tclConfig.sh file in /usr/share? Is so those need considering.
Does anyone disgree with the above conclusions? And do people agree that policy 8.2 could be clearer on this point?
I've filed a bug containing some of the above discussion and a patch for tcl8.5. http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=611650
Wookey
(please drop cc's other than debian-policy in replies if you want to work on that)
Hi Wookey,
Wookey wrote:
Debian policy (8.2) says:
It is recommended that supporting files and run-time support programs that do not need to be invoked manually by users, but are nevertheless required for the package to function, be placed (if they are binary) in a subdirectory of /usr/lib, preferably under /usr/lib/package-name. If the program or file is architecture independent, the recommendation is for it to be placed in a subdirectory of /usr/share instead,
[...]
For multiarch, or existing dpkg-cross cross-compiling, to work, arch-dependent needs to mean either form _or_ content (see below for elaboration).
I always had thought that it's content and that the "if they are binary" is only a red herring. I agree with you that the policy ought to be clarified. Thanks for bringing this up.
Regards, Jonathan
For this specific tcl issue, there was some discussion in Debian #599206; I didn't comment back on the bug, but I liked Goswin's proposal
Hi Wookey,
On Mon, Jan 31, 2011 at 05:16:15PM +0000, Wookey wrote:
Debian policy (8.2) says:
It is recommended that supporting files and run-time support programs that do not need to be invoked manually by users, but are nevertheless required for the package to function, be placed (if they are binary) in a subdirectory of /usr/lib, preferably under /usr/lib/package-name. If the program or file is architecture independent, the recommendation is for it to be placed in a subdirectory of /usr/share instead, preferably under /usr/share/package-name. Following the package-name naming convention ensures that the file names change when the shared object version changes.
Files and support programs only useful when compiling software against the library should be included in the development package for the library
A maintainer reading the above could reasonably decide that /usr/share was the right place for this file, because the file itself (being just shell) is arch-independent. The problem is that the contents are arch-dependent. Policy is not at all clear on this distinction (It's been making my head hurt all day). For multiarch, or existing dpkg-cross cross-compiling, to work, arch-dependent needs to mean either form _or_ content (see below for elaboration).
I disagree that this would be a reasonable thing for the maintainer to do. The current policy language talks about architecture-dependence of the *file*, not of a file *format*, and there are obviously file formats that are architecture-independent but contain architecture-specific data that must therefore be part of an architecture: any package.
So I think this is a clear policy violation in the existing tcl8.5-dev package; even if 8.2 doesn't prohibit the current behavior, the FHS surely does.[1] So I think your filing of bug #611650 was the correct course of action and the maintainer appears to agree.
That said, I'm also happy to second patches to policy that clarify the wording and save maintainers from thinking it's ok to ship such files under /usr/share when it isn't.
So, the questions is - which of these is the correct fix:
- make dpkg-cross copy over symlinks even when they point into
/usr/share 2) make tcl8.5 put tclConfig.sh in /usr/lib/tcl8.5 instead of /usr/share/tcltk/tcl8.5 3) make sqlite (and similar packages) look in /usr/share/tcltk/tcl8.5 instead of /usr/lib/tcl8.5 when building
1+2: dpkg-cross should also copy over symlinks that point to /usr/share. When such symlinks exist, it's almost invariably because *something is looking for the information in that location*. So as a general policy, they should also be made available in a crossed package.
Cheers,
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On 31/01/11 20:18, Steve Langasek wrote:
On Mon, Jan 31, 2011 at 05:16:15PM +0000, Wookey wrote:
So, the questions is - which of these is the correct fix:
- make dpkg-cross copy over symlinks even when they point into
/usr/share 2) make tcl8.5 put tclConfig.sh in /usr/lib/tcl8.5 instead of /usr/share/tcltk/tcl8.5 3) make sqlite (and similar packages) look in /usr/share/tcltk/tcl8.5 instead of /usr/lib/tcl8.5 when building
1+2: dpkg-cross should also copy over symlinks that point to /usr/share.
The reason not to copy is that the target of the symlink does not exist in the -cross package and the one that exists in the native package in /usr/share contains all the wrong paths. A dangling symlink would be left if the -cross package is not installed. If the symlink is to be copied over not into /usr/share but into /usr/arm-linux-gnueabi/share then the native version of the file will still exist in /usr/share and none of the existing tools will look into /usr/arm-linux-gnueabi/share to find the correct version without a new wrapper. Nobody wins.
dpkg-cross cannot reliably identify which symlinks should be preserved and how. dpkg-cross cannot copy across all symlinks which point to /usr/share because a lot of those are unwanted.
The correct fix is 2.
The only possible thing dpkg-cross could do is reverse the symlink and move the file from /usr/share into /usr/arm-linux-gnueabi/lib/ if a symlink to /usr/lib/ exists and not create the symlink in /usr/arm-linux-gnueabi/share at all - at which point the native version of the package still contains the symlink from /usr/share to /usr/lib/ with the native paths. Is that what is actually being requested for 1.? I don't see that it's reliable - it's masking the real bug with a horrible hack, again. How does this help when the package itself is buggy by not complying with the FHS and the cross-build looks in the wrong place?
When such symlinks exist, it's almost invariably because *something is looking for the information in that location*.
When is this not actually a bug in that package?
The maintainer has agreed to fix the actual bug by implementing option 2 above, which we all seem to agree is the appropriate and optimal fix for the original issue. Under what circumstances would some other package fall into this problem *without* also being buggy in exactly the same way as tcl8.5?
Yes, let's clarify policy but I don't think it's correct to expect dpkg-cross to handle packages which are simply buggy whilst risking breaking other packages which do things properly. dpkg-cross doesn't (and shouldn't) support package-specific exceptions in how -cross packages are created.
So as a general policy, they should also be made available in a crossed package.
The /usr/share/ location *must* change in the cross package so that it can be installed alongside the native, at which point any merit of retaining /usr/share is lost in the cross package because a wrapper script will still be needed to find the modified location, handle the dangling symlink or risk getting the wrong data (because the data in this case is clearly wrong for the cross build).
Either the file is in the wrong place (as in this case) or the change of location is simply going to break without a package-specific wrapper script anyway because the locations of things in /usr/share should not (need to) change to support a cross-build.
- --
Neil Williams ============= http://www.linux.codehelp.co.uk/
+++ Steve Langasek [2011-01-31 12:18 -0800]:
So, the questions is - which of these is the correct fix:
- make dpkg-cross copy over symlinks even when they point into
/usr/share 2) make tcl8.5 put tclConfig.sh in /usr/lib/tcl8.5 instead of /usr/share/tcltk/tcl8.5 3) make sqlite (and similar packages) look in /usr/share/tcltk/tcl8.5 instead of /usr/lib/tcl8.5 when building
1+2: dpkg-cross should also copy over symlinks that point to /usr/share. When such symlinks exist, it's almost invariably because *something is looking for the information in that location*. So as a general policy, they should also be made available in a crossed package.
But if something is looking for arch-independent stuff in /lib then in general that's wrong, and I'm not aware of any examples of correctly-packaged packages that need this. Any arch-independent files will be supplied by an arch all package that the build should depend on if needed.
dpkg-cross has gone for 10 years without needing symlinks pointing into /usr/share, so I'm reluctant to add them without some evidence that it's actually correct. Its policy of only crossing arch-dependent files is the right one, I believe. (It does allow symlinks within /usr/src which presumably has/had a good reason.)
Wookey
On Tue, Feb 01, 2011, Wookey wrote:
But if something is looking for arch-independent stuff in /lib then in general that's wrong, and I'm not aware of any examples of correctly-packaged packages that need this. Any arch-independent files will be supplied by an arch all package that the build should depend on if needed.
I might be getting your point wrong, but I certainly see a lot of files in /lib itself which are arch-independent data used for early boot (before /usr is available); PNG files and text files which would be identical on all architectures.
On Tue, 1 Feb 2011 12:50:48 +0100 Loïc Minier loic.minier@linaro.org wrote:
On Tue, Feb 01, 2011, Wookey wrote:
But if something is looking for arch-independent stuff in /lib then in general that's wrong, and I'm not aware of any examples of correctly-packaged packages that need this. Any arch-independent files will be supplied by an arch all package that the build should depend on if needed.
I might be getting your point wrong, but I certainly see a lot of files in /lib itself which are arch-independent data used for early boot (before /usr is available); PNG files and text files which would be identical on all architectures.
/lib vs /usr/lib is a different argument.
I'm working on the basis that Wookey was meaning /usr/lib/ compared to /usr/share.
The point about dpkg-cross is that it doesn't blindly take everything from /usr/lib and propagate that into /usr/$triplet/lib, it picks out stuff that it knows is useful to a cross package. Symlinks are included because there's no need to copy libfoo.so.0.0.1 as libfoo.so.0 etc.
There's a complex list of regular expressions, allowing header files, static linking files, object files, .la files, anything in /usr/include/ and stuff in /usr/lib/pkg-config/ and stuff already under /usr/<host>/lib/.
This list has aggregated over time. As multiarch is as far away as ever, I will discuss pruning that list significantly once Squeeze is released, leading to a dpkg-cross 3.0.0. The final list will only include stuff which dpkg-cross can reliably identify *and* which is absolutely essential to cross-builds. /usr/share won't be included, except for pkg-config files.
+++ Loïc Minier [2011-02-01 12:50 +0100]:
On Tue, Feb 01, 2011, Wookey wrote:
But if something is looking for arch-independent stuff in /lib then in general that's wrong, and I'm not aware of any examples of correctly-packaged packages that need this. Any arch-independent files will be supplied by an arch all package that the build should depend on if needed.
I might be getting your point wrong, but I certainly see a lot of files in /lib itself which are arch-independent data used for early boot (before /usr is available); PNG files and text files which would be identical on all architectures.
Sorry, I wasn't being very clear. By 'something is looking for arch-independent stuff in /lib' I really mean in /usr/<triplet>/lib, used during cross-building, (which will be put there by dpkg-cross-ed packages) (or in /usr/lib/<archtuple> or /lib/<archtuple> put there by dpkg on multiarch systems).
Yes there are various things in /lib that are not arch-dependent. dpkg-cross does not put most(any?) of those in -cross packages. In fact this is so true that it doesn't copy /usr/lib/tcl8.5/tclConfig.sh over into the cross package either. I should have checked that. Bum.
dpkg-cross in fact only picks files out of packages by positive identification as libraries or headers. It misses out generic 'other stuff' in ((/usr)/lib, which generally works pretty well, but in this tcl case it's not suffient for tcl-depending apps to cross-build.
Bug http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=599206 discusses this issue.
dpkg-cross is intended to put files needed for cross-building into -cross packages and it's currently missing this one out. Unfortunately because it doesn't match any of the 'standard paterns for cross-useful files' this can only be fixed by adding a fairly specific regexp, which is worrying close to special-casing or deciding that in fact just fishing out specific things from /usr/lib is too conservative and we should take the view that everything in /usr/lib is potentially useful in cross-building and should be copied into -cross packages.
In multiarch world everything in (/usr)/lib is going to end up in /usr/lib/<archtuple> or /lib/<archtuple>, unless packages are re-arranged to put them elsewhere, and we expect this to work fine so perhaps dpkg-cross should start doing the same thing, and thus discuver any problems this does potentially create. Would that actually do any harm? What files which are currently missed out of -cross packages would actually cause breakage if copied over into /usr/<triplet>/lib?
I just tried a modified dpkg-cross on a pile of packages and whilst many come out the same, you do get quite a lot more files in some packages and some packages that were previously null now have stuff in them. e.g apache-modules. So there is quite a lot of bloat, but probably no breakage.
Internally we will use a dpkg-cross modified to add /usr/lib/*/tclConfig.sh to the list of things that are important for cross-building. This means we will notice any other 'awkward cases' due to missing files.
An alternative view is that anything (such as sqlite3) depending on tclConfig.sh to build tcl extensions is broken and should be changed to use some other mechanism, and until then simply cannot be cross-built using the dpkg-cross mechanism. I am not familiar enough with tcl extensions to know if this is a reasonable stance or not, but given that it works just fine, and it's not hard to deal with, and (after the fix in debian bug#611650) it will carry on 'just working' in multiarch, I'm not convinced this is a reasonable stance.
Which leaves us with deciding whether to just copy over tclConfig.sh or everying in /usr/lib/*/* in dpkg-cross?
Wookey
On Thu, 3 Feb 2011 15:39:49 +0000 Wookey wookey@wookware.org wrote:
dpkg-cross in fact only picks files out of packages by positive identification as libraries or headers. It misses out generic 'other stuff' in ((/usr)/lib, which generally works pretty well, but in this tcl case it's not suffient for tcl-depending apps to cross-build.
Bug http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=599206 discusses this issue.
... and http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=611736 is probably going to be the root bug (renamed) to handle the clarification of exactly what files we keep and how we rationalise the legacy regular expressions in dpkg-cross.
Please add further specific cases to that bug.
dpkg-cross is intended to put files needed for cross-building into -cross packages and it's currently missing this one out. Unfortunately because it doesn't match any of the 'standard paterns for cross-useful files' this can only be fixed by adding a fairly specific regexp, which is worrying close to special-casing or deciding that in fact just fishing out specific things from /usr/lib is too conservative and we should take the view that everything in /usr/lib is potentially useful in cross-building and should be copied into -cross packages.
Generally, from a cursory view of what is left out, the situation we have is this:
0: dpkg-cross supports autotools cross-building through a series of very detailed, very complex and potentially fragile regular expressions but, importantly, ALSO uses a series of very detailed, very complex and potentially equally fragile regular expressions on the CONTENTS of those files, some of which Debian is quietly trying to drop from packages because of other problems. (e.g .la files)
1: dpkg-cross has explicit support for pkg-config which appears to work perfectly well, principally because pkg-config is inherently less complex than the entirety of autoconf|automake|libtool|dpkg-shlibdeps etc.
2: dpkg-cross has support for CMake - although only as a direct result of 0: and 1:
3: dpkg-cross has no idea how to help SCons or any number of other build systems out there, but then it mostly doesn't have to because many of those simply don't compile stuff, (they create Arch:all packages) and the ones that do compile stuff aren't used by sufficient numbers of cross-building people for sufficient complaints to be seen for dpkg-cross to have had the support created.
4: Nobody gave a damn about Tcl until sqlite added bindings.
5: Nobody adds support to dpkg-cross until someone complains loudly enough *and* comes up with sane patches.
6: dpkg-cross has huge amounts of legacy code which someone added at somepoint because it was important but nobody has actually had the guts to unilaterally remove because we can't tell if the original package has since been fixed. (s/fixed/broken in a different way/).
In multiarch world everything in (/usr)/lib is going to end up in /usr/lib/<archtuple> or /lib/<archtuple>, unless packages are re-arranged to put them elsewhere, and we expect this to work fine so perhaps dpkg-cross should start doing the same thing, and thus discuver any problems this does potentially create. Would that actually do any harm? What files which are currently missed out of -cross packages would actually cause breakage if copied over into /usr/<triplet>/lib?
Let's not break dpkg-cross fundamentally for everyone though. We can choose to make a different dpkg-cross which is FAR simpler (because it blindly moves files without any kind of safeguard) but as this does not involve fixing the contents of certain files, numerous autotools packages will break. So, if people want a broken dpkg-cross for testing, let's have a dpkg-multi-cross which breaks their cross-building world (using a different Provides: and conflicting with the "standard" cross packages) and leave the existing world alone.
That version of dpkg-multi-cross would be trivial to write - unpack the .deb, unconditionally move all files in ./usr/lib to ./usr/$triplet, handle /usr/include and remove everything else. Repack the .deb as -cross. Break world.
I just tried a modified dpkg-cross on a pile of packages and whilst many come out the same, you do get quite a lot more files in some packages and some packages that were previously null now have stuff in them. e.g apache-modules. So there is quite a lot of bloat, but probably no breakage.
Retaining the changes to the contents of the files whilst simply adding lots more *stuff* to -cross packages is the least harmful option. However, because the contents haven't changed, it doesn't actually help us identify the issues that would arise with Multiarch much.
-
Jonathan Nieder -
Loïc Minier -
Neil Williams -
Steve Langasek -
Wookey