Hello Android team,

I was working on making Android toolchain buildable using Android build service, and finally I was able to do successful and reproducible builds - of AOSP pristine bare-matal toolchain so far (https://android-build.linaro.org/builds/~pfalcon/aosp-toolchain/). There were few issues which needed to be investigated and resolved, and which I would like to discuss here:

1. make -jN breakage

Android build service builds on EC2 XLARGE instances with 16 concurrent make jobs (-j16). This invariably leads to build failure sooner or later (exact location depends on other options and maybe non-deterministic at all). The failure is "error: Link tests are not allowed after GCC_NO_EXECUTABLES." which somehow sends issue-hunting the wrong trail (like sysroot issues, etc.) but after some experiments I reduced this to exactly -j >1 being used, with -j1 it went past usual failure points reliably.

2. Lack of DESTDIR support

There's standard GNU autotools variable "DESTDIR" to install package into other directory besides $prefix. It is supported for gcc, binutils, etc., but not for Android's own toolchain/build project. The usual local-use trick is just to pass another prefix just for make install target, and that's what toolchain/build's README suggests. My only concern is "cleanroomness" of the results - suppose make install suddenly wants to rebuild something (libtool used to have such habit), then new prefix may be embedded into some executable, and then hit beyond the usual usage pattern (like, with non-English locale). Still, this is more a theoretical risk, which I as a build engineer should note, but not something to much worry about.

3. libstdc++v3 build

toolchain/build's README says that the default is not to build libstdc++v3 and it must be enabled explicitly. But in the current master I found that not to be the case - it gets enabled by default. And its build requires sysroot, so I had to disable it explicitly for bare-metal build.

4. sysroot source

So, to build full-fledged toolchain, we need to supply sysroot. What should be source of it? Android build service script I started from extracts it from an official Android SDK release. Is that good enough source? I guess we'd miss some non-released optimizations that way (byteswap ARMv6 optimizations come to mind). Otherwise, what should be used instead? Obvious choice is to build Android target images, then build toolchain on that tree. But that would be too long and expensive. Should we prepare and tar sysroot and provide it as extra build artifact for Android target builds? Also, can there be any machine-specificity, or it's for sure one generic sysroot for specific Android version (with arch-specific things handled via #ifdef's)?

Of these 4, first 3 are upstream-related bugreports with known workarounds. I wanted to write them down, but I'm not sure if more can be done about them - if you think it would be useful to submit them as bugs at lp/linaro-android (or directly to upstream?), I can do it. Sysroot issue of course requires input/discussion - in mail or at some LDS session.

Thanks, Paul mailto:pmiscml@gmail.com