(sorry, first time sent from wrong email, don't know if that'll get thru)
Hello Android team,
I was working on making Android toolchain buildable using Android build service, and finally I was able to do successful and reproducible builds - of AOSP pristine bare-matal toolchain so far (https://android-build.linaro.org/builds/~pfalcon/aosp-toolchain/). There were few issues which needed to be investigated and resolved, and which I would like to discuss here:
1. make -jN breakage
Android build service builds on EC2 XLARGE instances with 16 concurrent make jobs (-j16). This invariably leads to build failure sooner or later (exact location depends on other options and maybe non-deterministic at all). The failure is "error: Link tests are not allowed after GCC_NO_EXECUTABLES." which somehow sends issue-hunting the wrong trail (like sysroot issues, etc.) but after some experiments I reduced this to exactly -j >1 being used, with -j1 it went past usual failure points reliably.
2. Lack of DESTDIR support
There's standard GNU autotools variable "DESTDIR" to install package into other directory besides $prefix. It is supported for gcc, binutils, etc., but not for Android's own toolchain/build project. The usual local-use trick is just to pass another prefix just for make install target, and that's what toolchain/build's README suggests. My only concern is "cleanroomness" of the results - suppose make install suddenly wants to rebuild something (libtool used to have such habit), then new prefix may be embedded into some executable, and then hit beyond the usual usage pattern (like, with non-English locale). Still, this is more a theoretical risk, which I as a build engineer should note, but not something to much worry about.
3. libstdc++v3 build
toolchain/build's README says that the default is not to build libstdc++v3 and it must be enabled explicitly. But in the current master I found that not to be the case - it gets enabled by default. And its build requires sysroot, so I had to disable it explicitly for bare-metal build.
4. sysroot source
So, to build full-fledged toolchain, we need to supply sysroot. What should be source of it? Android build service script I started from extracts it from an official Android SDK release. Is that good enough source? I guess we'd miss some non-released optimizations that way (byteswap ARMv6 optimizations come to mind). Otherwise, what should be used instead? Obvious choice is to build Android target images, then build toolchain on that tree. But that would be too long and expensive. Should we prepare and tar sysroot and provide it as extra build artifact for Android target builds? Also, can there be any machine-specificity, or it's for sure one generic sysroot for specific Android version (with arch-specific things handled via #ifdef's)?
Of these 4, first 3 are upstream-related bugreports with known workarounds. I wanted to write them down, but I'm not sure if more can be done about them - if you think it would be useful to submit them as bugs at lp/linaro-android (or directly to upstream?), I can do it. Sysroot issue of course requires input/discussion - in mail or at some LDS session.
Thanks, Paul mailto:pmiscml@gmail.com
Paul, thanks for getting some builds going.
On 8 May 2011 18:40, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
(sorry, first time sent from wrong email, don't know if that'll get thru)
Hello Android team,
I was working on making Android toolchain buildable using Android build service, and finally I was able to do successful and reproducible builds - of AOSP pristine bare-matal toolchain so far (https://android-build.linaro.org/builds/~pfalcon/aosp-toolchain/). There were few issues which needed to be investigated and resolved, and which I would like to discuss here:
- make -jN breakage
Android build service builds on EC2 XLARGE instances with 16 concurrent make jobs (-j16). This invariably leads to build failure sooner or later (exact location depends on other options and maybe non-deterministic at all). The failure is "error: Link tests are not allowed after GCC_NO_EXECUTABLES." which somehow sends issue-hunting the wrong trail (like sysroot issues, etc.) but after some experiments I reduced this to exactly -j >1 being used, with -j1 it went past usual failure points reliably.
- Lack of DESTDIR support
There's standard GNU autotools variable "DESTDIR" to install package into other directory besides $prefix. It is supported for gcc, binutils, etc., but not for Android's own toolchain/build project. The usual local-use trick is just to pass another prefix just for make install target, and that's what toolchain/build's README suggests. My only concern is "cleanroomness" of the results - suppose make install suddenly wants to rebuild something (libtool used to have such habit), then new prefix may be embedded into some executable, and then hit beyond the usual usage pattern (like, with non-English locale). Still, this is more a theoretical risk, which I as a build engineer should note, but not something to much worry about.
- libstdc++v3 build
toolchain/build's README says that the default is not to build libstdc++v3 and it must be enabled explicitly. But in the current master I found that not to be the case - it gets enabled by default. And its build requires sysroot, so I had to disable it explicitly for bare-metal build.
Sounds like out-of-date documentation.
- sysroot source
So, to build full-fledged toolchain, we need to supply sysroot. What should be source of it? Android build service script I started from extracts it from an official Android SDK release. Is that good enough source? I guess we'd miss some non-released optimizations that way (byteswap ARMv6 optimizations come to mind). Otherwise, what should be used instead? Obvious choice is to build Android target images, then build toolchain on that tree. But that would be too long and expensive. Should we prepare and tar sysroot and provide it as extra build artifact for Android target builds? Also, can there be any machine-specificity, or it's for sure one generic sysroot for specific Android version (with arch-specific things handled via #ifdef's)?
Of these 4, first 3 are upstream-related bugreports with known workarounds. I wanted to write them down, but I'm not sure if more can be done about them - if you think it would be useful to submit them as bugs at lp/linaro-android (or directly to upstream?), I can do it. Sysroot issue of course requires input/discussion - in mail or at some LDS session.
I think bug reports would be useful, would you file them? We'll chat at LDS.
Thanks, Paul mailto:pmiscml@gmail.com
-- Best Regards, Paul
On 9 May 2011 00:40, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
(sorry, first time sent from wrong email, don't know if that'll get thru)
Hello Android team,
hi Paul,
It is my pleasure to discuss with you.
I was working on making Android toolchain buildable using Android build service, and finally I was able to do successful and reproducible builds - of AOSP pristine bare-matal toolchain so far (https://android-build.linaro.org/builds/~pfalcon/aosp-toolchain/). There were few issues which needed to be investigated and resolved, and which I would like to discuss here:
In my humble opinion, AOSP toolchain ought to keep the same from Google binary delivery since we have no official documentation about Google's build environment and detailed instructions.
Linaro wiki contains one comprehensive page: https://wiki.linaro.org/Platform/Android/UpstreamToolchain
Although we can build on our own, it is unofficial and not fully the same as Google's.
- make -jN breakage
Android build service builds on EC2 XLARGE instances with 16 concurrent make jobs (-j16). This invariably leads to build failure sooner or later (exact location depends on other options and maybe non-deterministic at all). The failure is "error: Link tests are not allowed after GCC_NO_EXECUTABLES." which somehow sends issue-hunting the wrong trail (like sysroot issues, etc.) but after some experiments I reduced this to exactly -j >1 being used, with -j1 it went past usual failure points reliably.
Agree. Using -j1 is safe.
- Lack of DESTDIR support
There's standard GNU autotools variable "DESTDIR" to install package into other directory besides $prefix. It is supported for gcc, binutils, etc., but not for Android's own toolchain/build project. The usual local-use trick is just to pass another prefix just for make install target, and that's what toolchain/build's README suggests. My only concern is "cleanroomness" of the results - suppose make install suddenly wants to rebuild something (libtool used to have such habit), then new prefix may be embedded into some executable, and then hit beyond the usual usage pattern (like, with non-English locale). Still, this is more a theoretical risk, which I as a build engineer should note, but not something to much worry about.
If you were looking into recent change in AOSP, more target are going to be installed for unknown reasons.
I don't think it is a problem though once we have specific manners.
- libstdc++v3 build
toolchain/build's README says that the default is not to build libstdc++v3 and it must be enabled explicitly. But in the current master I found that not to be the case - it gets enabled by default. And its build requires sysroot, so I had to disable it explicitly for bare-metal build.
- sysroot source
Yes, you need to give sysroot while building toolchain.
You can get it from either NDK or built root file system.
So, to build full-fledged toolchain, we need to supply sysroot. What should be source of it?
Thanks, -jserv
Hello Jim,
On Mon, 9 May 2011 19:01:43 +0800 Jim Huang jim.huang@linaro.org wrote:
On 9 May 2011 00:40, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
(sorry, first time sent from wrong email, don't know if that'll get thru)
Hello Android team,
hi Paul,
It is my pleasure to discuss with you.
I was working on making Android toolchain buildable using Android build service, and finally I was able to do successful and reproducible builds - of AOSP pristine bare-matal toolchain so far (https://android-build.linaro.org/builds/~pfalcon/aosp-toolchain/). There were few issues which needed to be investigated and resolved, and which I would like to discuss here:
In my humble opinion, AOSP toolchain ought to keep the same from Google binary delivery since we have no official documentation about Google's build environment and detailed instructions.
I agree. But the talk is more about performing reliable builds of an Android toolchain - taken from any source, of which AOSP is one (known good apparently). It just happen to be that pristine AOSP toolchain build is already here (Linaro-optimized to follow of course).
Linaro wiki contains one comprehensive page: https://wiki.linaro.org/Platform/Android/UpstreamToolchain
Well, doing continuous build in the cloud uncovered bunch of issues not discussed there ;-).
Although we can build on our own, it is unofficial and not fully the same as Google's.
- make -jN breakage
Android build service builds on EC2 XLARGE instances with 16 concurrent make jobs (-j16). This invariably leads to build failure sooner or later (exact location depends on other options and maybe non-deterministic at all). The failure is "error: Link tests are not allowed after GCC_NO_EXECUTABLES." which somehow sends issue-hunting the wrong trail (like sysroot issues, etc.) but after some experiments I reduced this to exactly -j >1 being used, with -j1 it went past usual failure points reliably.
Agree. Using -j1 is safe.
Ok, is it well-known fact that using N>1 is unsafe? If no, should we do something about it?
- Lack of DESTDIR support
There's standard GNU autotools variable "DESTDIR" to install package into other directory besides $prefix. It is supported for gcc, binutils, etc., but not for Android's own toolchain/build project. The usual local-use trick is just to pass another prefix just for make install target, and that's what toolchain/build's README suggests. My only concern is "cleanroomness" of the results - suppose make install suddenly wants to rebuild something (libtool used to have such habit), then new prefix may be embedded into some executable, and then hit beyond the usual usage pattern (like, with non-English locale). Still, this is more a theoretical risk, which I as a build engineer should note, but not something to much worry about.
If you were looking into recent change in AOSP, more target are going to be installed for unknown reasons.
I don't think it is a problem though once we have specific manners.
- libstdc++v3 build
toolchain/build's README says that the default is not to build libstdc++v3 and it must be enabled explicitly. But in the current master I found that not to be the case - it gets enabled by default. And its build requires sysroot, so I had to disable it explicitly for bare-metal build.
- sysroot source
Yes, you need to give sysroot while building toolchain.
You can get it from either NDK or built root file system.
Are there any differences using one vs another? Are there any risks using NDK's?
There're following sessions dedicated to Android continuous builds and/or toolchains, let's discuss these questions within their scope:
https://blueprints.launchpad.net/linaro-android/+spec/linaro-android-o-conti... https://blueprints.launchpad.net/linaro-android/+spec/linaro-android-o-toolc...
So, to build full-fledged toolchain, we need to supply sysroot. What should be source of it?
Thanks, -jserv
-
Jim Huang -
Paul Sokolovsky -
Zach Pfeffer