On Mon, Aug 23, 2010, Dechesne, Nicolas wrote:

...

I am still trying to setup a functional cross compilation environment for our packages. I want to be able to build without xdeb, using debuild command (this is mainly because we use other tools such as git-buildpackage).

(You might be able to arrange for git-buildpackage to call a builder which calls xdeb; but cross-building directly is fine too)

...

When I build a package which has dependency on another .so file, my ./configure script fails, pkg-config complains that it cannot find my package config file (which is available in /usr/arm-linux-gnueabi/lib/pkgconfig.

If I set PKG_CONFIG_LIBDIR to /usr/arm-linux-gnueabi/lib/pkgconfig, then my build is fine, e.g. debuild -ePKG_CONFIG_LIBDIR=/usr/arm-linux-gnueabi/lib/pkgconfig -b -aarmel -us -uc.

Is that expected? I was looking around in xdeb, and I don't find where this is being done, and I am sure it would be needed too since it ends up calling debuild...

I think we need that now too; would you mind filing a xdeb bug about that?

See Debian bug #551118 for why it's a recent in xdeb in maverick; doing it in dpkg-buildpackage was a kludge, but I don't think pkg-config has the relevant tests just yet (I don't have a triplet-pkg-config here, and I didn't see any tests for one in pkg.m4 from upstream git).

well.. this debian bug clearly explains my problem! thanks for pointing this out! so for now the only option is to set this manually? what is the long term option?

i cannot enter a bug in the xdeb launchpad project. where should I enter this bug?

xdeb should test for whether triplet-pkg-config is available and set PKG_CONFIG_LIBDIR if not.

...

                           In this process I noticed that xdeb is

also passing /etc/dpkg-cross/cross-config.armel to debuild. Should I do this to?

This is the autoconf caching mechanism; it's a very nice trick: software using autoconf can test for stuff by trying to run some code during configure; this gets disabled automatically during cross-compilation for obvious reasons; the cool trick with cache is that you can provide the cached results and configure will trust the cache blindly.

cool. what does this imply if I don't pass the cached results?

On Tue, Aug 24, 2010 at 04:22:51PM +0200, Loïc Minier wrote:

...

i cannot enter a bug in the xdeb launchpad project. where should I enter this bug?

Oh right, please use the Ubuntu source package to report bugs; we only have it in Ubuntu right now.

Shouldn't it registered as an upstream project and maintained there? (I suspect this is a Steve L. or Scott B. project)

So I gave a try to cross-compiling gstreamer0.10 and it some interesting issues.

First, I used an i386 chroot instead of an amd64 chroot since I thought that might help a bit with package names; I'm not actually sure that was needed, but it might influence results.

Second, here is how I ran xdeb: % xdeb -a armel --apt-source --only-explicit gstreamer0.10

The first thing which failed here are downloads of biarch libraries for cross import. Currently, xdeb imports binary packages source package by source package (native_import() acts on a source package), and checks for all binary packages of the source package whether they are visible on the build architecture with "apt-cache show <package name>". However a bunch of packages are architecture-specific, so this might miss packages which are only available on the host architecture, and not on the build architecture, and this might try downloading packages which are only available for the build architecture, and hence might fail. The proper fix is to have xdeb look at an APT cache for the build architecture rather than the host one, and filter based on that. This is LP #616617. However I found a workaround by not caring about biarch: - I changed is_crossable() to return False for packages with "lib32" and "lib64" - I changed native_import() to skip (don't download) packages which aren't crossable -- these weren't imported anyway!

This got me to the point where I could install build-deps, I probably fiddled with the blacklist a bit too (I listed gir1.0-glib-2.0 IIRC).

Next, the gstreamer0.10 cross-build started but promptly failed trying to link to libxml2; this was the PKG_CONFIG_LIBDIR issue you mentioned. I added: buildpackage.append('-ePKG_CONFIG_LIBDIR=/usr/%s/lib/pkgconfig' % deb_host_gnu_type)

This is LP #623478 which you filed.

Interestingly, the build broke because it couldn't find gtk-doc anymore; I didn't add /usr/share/pkgconfig, but it does show that host versus build pkg-config dependencies are a problem, and we need a $triplet-pkg-config to solve this!

So I patched the gstreamer build to turn off docs; I think gstreamer0.10 should --disable-gtk-doc when cross-compiling; in fact, this should be done in gtk-doc upstream.

This got quite far in the build: make[4]: Entering directory `/home/lool/gstreamer0.10/gstreamer0.10/gst' [...] libtool: link: arm-linux-gnueabi-ranlib .libs/libgstreamer-0.10.a libtool: link: rm -fr .libs/libgstreamer-0.10.lax libtool: link: ( cd ".libs" && rm -f "libgstreamer-0.10.la" && ln -s "../libgstreamer-0.10.la" "libgstreamer-0.10.la" ) /usr/bin/g-ir-scanner -v --namespace Gst \ --nsversion=0.10 \ -I.. \ -I.. \ -DIN_GOBJECT_INTROSPECTION=1 \ --c-include='gst/gst.h' \ --library=libgstreamer-0.10.la \ --include=GLib-2.0 \ --include=GObject-2.0 \ --include=GModule-2.0 \ --include=libxml2-2.0 \ --libtool="../libtool" \ --pkg glib-2.0 \ --pkg gobject-2.0 \ --pkg gmodule-no-export-2.0 \ --pkg gthread-2.0 \ --pkg libxml-2.0 \ --output Gst-0.10.gir \ ./gst.h ./glib-compat.h ./gstobject.h ./gstbin.h ./gstbuffer.h ./gstbufferlist.h ./gstbus.h ./gstcaps.h ./gstchildproxy.h ./gstclock.h ./gstcompat.h ./gstdebugutils.h ./gstelement.h ./gstelementfactory.h ./gsterror.h ./gstevent.h ./gstfilter.h ./gstformat.h ./gstghostpad.h ./gstindex.h ./gstindexfactory.h ./gstinfo.h ./gstinterface.h ./gstiterator.h ./gstmacros.h ./gstmessage.h ./gstminiobject.h ./gstpad.h ./gstpadtemplate.h ./gstparamspecs.h ./gstpipeline.h ./gstplugin.h ./gstpluginfeature.h ./gstpoll.h ./gstpreset.h ./gstquery.h ./gstsegment.h ./gststructure.h ./gstsystemclock.h ./gsttaglist.h ./gsttagsetter.h ./gsttask.h ./gsttaskpool.h ./gsttrace.h ./gsttypefind.h ./gsttypefindfactory.h ./gsturi.h ./gstutils.h ./gstvalue.h ./gstregistry.h ./gstparse.h ./gstxml.h \ ./gst.c ./gstobject.c ./gstbin.c ./gstbuffer.c ./gstbufferlist.c ./gstbus.c ./gstcaps.c ./gstchildproxy.c ./gstclock.c ./gstdebugutils.c ./gstelement.c ./gstelementfactory.c ./gsterror.c ./gstevent.c ./gstfilter.c ./gstformat.c ./gstghostpad.c ./gstindex.c ./gstindexfactory.c ./gstinfo.c ./gstinterface.c ./gstiterator.c ./gstmessage.c ./gstminiobject.c ./gstpad.c ./gstpadtemplate.c ./gstparamspecs.c ./gstpipeline.c ./gstplugin.c ./gstpluginfeature.c ./gstpluginloader.c ./gstpoll.c ./gstpreset.c ./gstquark.c ./gstquery.c ./gstregistry.c ./gstregistrychunks.c ./gstsegment.c ./gststructure.c ./gstsystemclock.c ./gsttaglist.c ./gsttagsetter.c ./gsttask.c ./gsttaskpool.c ./gsttrace.c ./gsttypefind.c ./gsttypefindfactory.c ./gsturi.c ./gstutils.c ./gstvalue.c ./gstparse.c ./gstregistrybinary.c ./gstxml.c /usr/lib/gcc/arm-linux-gnueabi/4.4.5/../../../../arm-linux-gnueabi/bin/ld: warning: libxml2.so.2, needed by ./.libs/libgstreamer-0.10.so, not found (try using -rpath or -rpath-link) /usr/lib/gcc/arm-linux-gnueabi/4.4.5/../../../../arm-linux-gnueabi/bin/ld: warning: libm.so.6, needed by ./.libs/libgstreamer-0.10.so, not found (try using -rpath or -rpath-link) [...] Traceback (most recent call last): File "/usr/bin/g-ir-scanner", line 38, in <module> sys.exit(scanner_main(sys.argv)) File "/usr/lib/gobject-introspection/giscanner/scannermain.py", line 330, in scanner_main glibtransformer.get_get_type_functions()) File "/usr/lib/gobject-introspection/giscanner/dumper.py", line 246, in compile_introspection_binary return dc.run() File "/usr/lib/gobject-introspection/giscanner/dumper.py", line 132, in run self._link(bin_path, o_path) File "/usr/lib/gobject-introspection/giscanner/dumper.py", line 241, in _link subprocess.check_call(args) File "/usr/lib/python2.6/subprocess.py", line 488, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['../libtool', '--mode=link', '--tag=CC', '--silent', 'arm-linux-gnueabi-gcc', '-o', '/home/lool/gstreamer0.10/gstreamer0.10/gst/tmp-introspectOGdeJg/Gst-0.10', '-g', '-O2', '-g', '-Wall', '-O2', '-Wno-error', '-L.', '-Wl,--export-dynamic', '-pthread', '-L/usr/arm-linux-gnueabi/lib', '-lgirepository-1.0', '-lgobject-2.0', '-lgmodule-2.0', '-lffi', '-lgthread-2.0', '-lrt', '-lglib-2.0', 'libgstreamer-0.10.la', '-pthread', '-Wl,--export-dynamic', '-L/usr/arm-linux-gnueabi/lib', '-lgio-2.0', '-lgirepository-1.0', '-lgobject-2.0', '-lgmodule-2.0', '-lffi', '-lgthread-2.0', '-lrt', '-lglib-2.0', '/home/lool/gstreamer0.10/gstreamer0.10/gst/tmp-introspectOGdeJg/Gst-0.10.o']' returned non-zero exit status 1 make[5]: *** [Gst-0.10.gir] Error 1 make[5]: Leaving directory `/home/lool/gstreamer0.10/gstreamer0.10/gst'

So the issue is that g-ir-scanner is broken for cross-builds.

I stopped looking into it at this point

Cheers,

On Wed, Aug 25, 2010, Wookey wrote:

I'm using amd64 - hopefully it doesn't matter.

Well I feared it might influence xdeb, since there is this bug about the need to use two caches; notably, I thought at least using a 32-bits build arch was better than a 64-bits for the visible package names. Now I don't expect it actually makes a big difference, but I wanted to mention it.

I'm a biut confused about this. I got this error in pdebuild-cross because xapt was initially missing so the cross-deps simly weren't being installed. As soon as they were installed then this part went fine. Which is what I would expect. So there must be something different about the environment in the xdeb case. I'll try and work out what is going on.

Is this on an Ubuntu host with an Ubuntu rootfs in your build environment? pbuilder doesn't cleanup the env, only debuild does. Also, dpkg-buildpackage used to set PKG_CONFIG_LIBDIR until recently, so if you're trying on an older distro (e.g. lucid) you might not see the issue we have now.

...

Interestingly, the build broke because it couldn't find gtk-doc anymore; I didn't add /usr/share/pkgconfig, but it does show that host versus build pkg-config dependencies are a problem, and we need a $triplet-pkg-config to solve this!

So I patched the gstreamer build to turn off docs; I think gstreamer0.10 should --disable-gtk-doc when cross-compiling; in fact, this should be done in gtk-doc upstream.

I left docs on. It correspondingly pulls in a lot of deps (157 cross-deps, but lots of them are null so it doesn't actually waste _too_ much time). Nothing broke.

Problems were not with the deps but with a call to pkg-config; I didn't have /usr/$triplet/share/pkgconfig in my path and I didn't cross gtk-doc to have it, but gtk-doc.pc was in /usr/share/pkgconfig, and it would probably have been picked up if I had bothered setting PKG_CONFIG_LIBDIR to pick it up. I thought the best fix here was actually to disable docs when cross-building because gtk-doc usually generates scanners and runs them, which obviously doens't work.

Yep. I think we can agree on that. It's interesting that the two build methods don't quite give the same behaviour. I'll poke it some more to see what's going on.

I think I saw your error message later down, but I looked at the first error I got.

+++ Dechesne, Nicolas [2010-08-23 23:31 +0200]:

...

Hello,

I have another question related to cross compilation...

I am still trying to setup a functional cross compilation environment for our packages. I want to be able to build without xdeb, using debuild command (this is mainly because we use other tools such as git-buildpackage).

This should work fine if you pass -aarmel

If you want to do the same thing in a controlled environment you can now also use pdebuild. I got this working last week. pdebuild-cross is just some hook scripts for pbuilder/pdebuild (and a multistrap config so it's trivial to make a cross-build chroot with base system+marcin's cross tools, but it doesn't actually matter how you get that environment created).

So that's building in a chroot rather than directly on your system, which has the advantage that your system doesn't fill up with piles of cross-packages (which tends to cause breakage eventually when the original native packages get updated), and ensures that the build is not relying on unspecified stuff which just happens to be installed. But it's slower because pbuilder unpacks chroot and installs stuff each time. If building the same thing often then updating the chroot image to have the necessary deps is a good idea.

I am building in a chroot already, on a PC running lucid, even though I am not using pdebuild-cross.

You use it exactly the same way as pbuilder by changing into the source tree and doing 'pdebuild-cross'.

Get in touch if you are interested in using it because pdebuild-cross isn't yet in maverick and you need an updated multistrap for it to be painless.

for now, we use pbuilder already on native h/w, and I am just looking for a build acceleration method, so cross compilation would be perfect. I don't think we need pdebuil-cross for now, but I will get back to you

...

When I build a package which has dependency on another .so file, my ./configure script fails, pkg-config complains that it cannot find my package config file (which is available in /usr/arm-linux-gnueabi/lib/pkgconfig.

Was that ./configure called with the right --build and --host options? It should find the pkgconfig files if so. Otherwise I'd say it was buggy.

I am using cdbs autotools classe. My debian/rules file only includes that file. So I don't control how ./configure is called, and this is why I would have expected that somehow PKG_CONFIG_LIBDIR would be set properly...

...

If I set PKG_CONFIG_LIBDIR to /usr/arm-linux-gnueabi/lib/pkgconfig, then my build is fine, e.g. debuild -ePKG_CONFIG_LIBDIR=/usr/arm-linux-gnueabi/lib/ pkgconfig -b -aarmel -us -uc.

Is that expected?

It is generally the responsibility of the build-system config tool to set PKG_CONFIG_LIBDIR correctly - i.e autofoo or cmake or whatever.

That's a problem with packages which don't use such a system, but do use pkgconfig. I haven't seen many of those but there probably are some and they need that env var setting somehow when cross-building. In fact that path ought to be /usr/arm-linux-gnueabi/lib/pkgconfig: /usr/arm-linux-gnueabi/share/pkgconfig/ as there are some packages which install arch-independent PKGCONFIG files (eg pthreads).

you're right. I know I need share/pkgconfig but not for this specific package.

I'm not sure if there are things that would break if an external build tools sets this. I can't think of any offhand, however it is currently the case that we rely on the rules files to correctly set the build environment when called in cross-build form. I'm inclined to leave that responsibility there, but I'm open to arguments that it should be done elsewhere.

...

I would appreciate any feedback on this.

What packages are you building? I'd like to be able to reproduce your issues.

well, i cannot share that right now, but these packages will be available soon on launchpad, we are still working on making them public!

Wookey