Hello list,
I would like to make a proposal about utilizing Linaro toolchain for Android and NDK (Native Development Kit)[1].
** Motivation
There are some different perspectives between Linaro toolchain and Google Android toolchain including technical and non-technical considerations. It doesn't really work if we only replace prebuilt toolchain with Linaro toolchain because of the compatibility of Android system utilities such as ELF prelinker. Also, since Android is developed in relatively closed environment (Google style open source model), a great amount of software components are not always verified by different toolchain or build configurations. This proposal attempts to establish the compact development flow to enable Linaro optimized ARM toolchain to build Android from scratch and verify it transparently. Eventually, Android can be the reference indicator as Linaro toolchain performance and reliability.
** Brief introduction to Google Android toolchain
Inside Google, there is a dedicated compiler team working on GNU Toolchain for various purposes including server-side computing, Android, Chrome OS, etc. Google engineers submit patches to upstream for public review and maintain the toolchain for Android. Along with each Android Open Source Prokect (AOSP) release, there is a special branch in korg GIT [2] for hosting the GPL'd toolchain source code modified by Google. Usually, file "README.google" mentioned the summary, but it is not developer friendly because several changes were done within one GIT commit.
Please refer to wiki for details: https://wiki.linaro.org/Platform/Android/UpstreamToolchain
** What's wrong with Android upstream Toolchain?
In my opinion, list as following:
(1) Few information about Google improvement: Sometimes, we have to guess something from implicit GIT commitlog such as "commit gcc-4.4.3 which is used to build gcc-4.4.3 Android toolchain in master"[3]. It is hard to track and get verified carefully.
(2) Google specific improvements are absent in recent release, only enabled months later. For example, Google Compiler Team Lead, Dr. Shih-wei Liao, presented the improvements against GNU Toolchain in the middle of 2009.[4]. The report came with several impressive improvements like FDO (Feedback Directed Optimizations) and IPO (Inter-Procedure Optimizations). However, only some of them are public to AOSP and be integrated late in the middle of 2010 (Android Froyo; 2.2). Even FDO was merged in Android Froyo already, but there is few documentation and no robust method to verify by community members such as Linaro engineers.
(3) For some reasons, Google tends to deliver stable (old) toolchain plus mainline backport. It is a safe and workable approach, but sometimes developers would expect to use the latest technologies as Linaro aims to bring to the world.
(4) Few readable documentation. For example, Google already open its toolchain benchmark suite in early 2010, but there was no document specific to such important components. Furthermore, there was one file gone in public kog GIT, required by automated benchmark process. One year later, Google engineer finally put back the one to public. This implies the unusual way Google developed and delivered software.
** Linaro's Approach to enable latest technologies
Linaro android team tries to do: (1) Document Android toolchain and related utilities in korg GIT as possible as we can. (2) Early adaptation of Linaro toolchain to Android build system and verify these output systematically. (3) Backport Google changes to Linaro GCC and review in public. (4) Improve the deployment and validation flow by means of Linaro infrastructure. (5) Build and test Android system with Linaro tools. Then, figure out the regressions caused by Linaro Toolchain and/or aggressive optimizations (6) measure performance gain by Linaro tools
The detailed specification in wiki: https://wiki.linaro.org/Platform/Android/Specs/LinaroAndroidToolchain
** Implementation of Linaro toolchain for Android
We started from Android style toolchain build and move to Linaro GCC + ARM specific optimizations in mind. The initial work can be obtained by wiki: https://wiki.linaro.org/Platform/Android/Toolchain
We plan to maintain the following GIT repositories at least: * android/toolchain/build.git : Linaro-aware build system. Derived from Android toolchain build system, it can handle Linaro-GCC and Linaro snapshot/bzr. * android/toolchain/gcc-patches.git : Patchset to be applied on top of Linaro-GCC release/snapshots
The reference builder script output: $ ./linaro-build.sh --help --prefix-dir= Specify where to install (default: /tmp/android-toolchain-eabi) --gcc-src-dir= Specify where linaro gcc source is (in <toolchain>/gcc) --apply-gcc-patch=(yes|no) Apply-patch which in <toolchain>/gcc-patches directory (default: no)
Current verified combinations: * gcc-linaro: 4.5-2011.02-0 * binutils: 2.20.1 * gmp: 4.2.4 * mpfr: 2.4.1
Only gcc is replaced by gcc-linaro: 4.5-2011.02-0 and others are checked out from korg GIT.
** Summary of gcc-patches
"gcc-patches" are used as "backport" from Google changes into Linaro gcc base. Here is the summary at present:
0001-Add-linux-android.patch Add linux-android
0002-Add-support-for-Bionic-C-library.patch Add support for Bionic C library
0003-Support-compilation-for-Android-platform.patch Support compilation for Android platform
0004-Add-multilib-configuration-for-arm-linux-androideabi.patch Add multilib configuration for arm-linux-androideabi
0005-Fix-gthr-posix.h-to-support-Bionic.patch Fix gthr-posix.h to support Bionic
0006-Add-untested-support-for-Bionic-to-libstdc.patch Add [untested] support for Bionic to libstdc++
These patches are taken from Maxim Kuvyrkov of CodeSourcery in gcc-4.6 branch. Of course, we can always add changes by Google or other Android specific adaptation by this model.
** Planned improvements over Linaro toolchain for Android
(1) GCC multilib setting Default: arm, fpu and thumb. The prebuilt google toolchain use: armv5te and mandroid. We should focus on ARMv7. (2) HardFP-ABI Support for Android. (3) Patch management: Better to get the Android patches into Linaro-GCC tree eventually. (4) Build system improvement. Don't have to build gmp, mpfr everytime, and provide option to build without gdb. (5) Enable LTO (Link Time Optimization, introduced since gcc-4.5) in Android TARGET_GLOBAL_CFLAGS (6) Verify the functionality of FDO (Feedback Directed Optimization) and introduce the approaches to integrate.
** Toward Android NDK
Once Linaro toolchain for Android is ready to use, it is time to re-package Android NDK by Linaro toolchain. To do that, extra build configuration, sysroot, is required. According to Android Release Cycle & Phases[5], the repacked NDK should be verified one moth after Android public release.
Sincerely, Jim Huang (jserv)
[1] http://developer.android.com/sdk/ndk/index.html [2] http://android.git.kernel.org/ [3] http://android.git.kernel.org/?p=toolchain/gcc.git%3Ba=commit%3Bh=b094d6c4bf... [4] Smaller and Faster Android: A talk from a practitioner to fellow ones, Shih-wei Liao, Google, http://coscup.wdfiles.com/local--files/schedule-2009/AndroidOptimizationStud... [5] Android Release Cycle & Phases Draft, https://wiki.linaro.org/Platform/Android/ReleaseCycle
On Wed, Feb 23, 2011 at 8:28 AM, Jim Huang jserv@0xlab.org wrote:
Hello list,
I would like to make a proposal about utilizing Linaro toolchain for Android and NDK (Native Development Kit)[1].
** Motivation
There are some different perspectives between Linaro toolchain and Google Android toolchain including technical and non-technical considerations. It doesn't really work if we only replace prebuilt toolchain with Linaro toolchain because of the compatibility of Android system utilities such as ELF prelinker. Also, since Android is developed in relatively closed
I don't have enough background to understand this "ELF prelinker" stuff. Are you saying that because of the way how android links stuff we cannot have one code base for gcc that works for both, android and "normal libc linux"?
environment (Google style open source model), a great amount of software components are not always verified by different toolchain or build configurations. This proposal attempts to
ack. thats what we want to do. Of course, we cannot really verify what is going on behind closed walls, but we can continuously build android with our toolchain and fix issues due that in android public master and if even that doesn't work we can ensure that our android trees always work nicely with both, our gcc and android gcc.
Another thing is to make our toolchain easily consumable (like the NDK you mention at the bottom); this will increase chances that someone from google can eventually take a look at what we are doing etc. and also helps the community to use linaro toolchain to built their android distributions.
establish the compact development flow to enable Linaro optimized ARM toolchain to build Android from scratch and verify it transparently. Eventually, Android can be the reference indicator as Linaro toolchain performance and reliability.
** Brief introduction to Google Android toolchain
Inside Google, there is a dedicated compiler team working on GNU Toolchain for various purposes including server-side computing, Android, Chrome OS, etc. Google engineers submit patches to upstream for public review and maintain the toolchain for Android. Along with each Android Open Source Prokect (AOSP) release, there is a special branch in korg GIT [2] for hosting the GPL'd toolchain source code modified by Google. Usually, file "README.google" mentioned the summary, but it is not developer friendly because several changes were done within one GIT commit.
Please refer to wiki for details: https://wiki.linaro.org/Platform/Android/UpstreamToolchain
thats a good wiki page. thanks for the content. If I read the skia example correctly, we could add a test to our "normal" abrek testsuite that uses our daily android toolchain and run the skia benchmark? e.g. we could start doing this benchmarking even without having a validation solution ready for android targets?
Please let's talk to Paul how we can get the android toolchain to /opt/android as part of abrek and lets try to add this to our abrek testsuites. Until we have daily toolchain builds it would be OK to download the android toolchain tarball from a fixed place from people.linaro.org I guess.
** What's wrong with Android upstream Toolchain?
In my opinion, list as following:
(1) Few information about Google improvement: Sometimes, we have to guess something from implicit GIT commitlog such as "commit gcc-4.4.3 which is used to build gcc-4.4.3 Android toolchain in master"[3]. It is hard to track and get verified carefully.
yes, that feels like a messy situation. Do we know why they don't commit the changes as individual commits but then in next step document what they changed?
(2) Google specific improvements are absent in recent release, only enabled months later. For example, Google Compiler Team Lead, Dr. Shih-wei Liao, presented the improvements against GNU Toolchain in the middle of 2009.[4]. The report came with several impressive improvements like FDO (Feedback Directed Optimizations) and IPO (Inter-Procedure Optimizations). However, only some of them are public to AOSP and be integrated late in the middle of 2010 (Android Froyo; 2.2). Even FDO was merged in Android Froyo already, but there is few documentation and no robust method to verify by community members such as Linaro engineers.
you say that they don't publish the code for lets say the "gingerbread" toolchain in a timely fashion when they release gingerbread? Or do they ship a separate "fast" NDK/prebuilt for partners through secret channels?
(3) For some reasons, Google tends to deliver stable (old) toolchain plus mainline backport. It is a safe and workable approach, but sometimes developers would expect to use the latest technologies as Linaro aims to bring to the world.
(4) Few readable documentation. For example, Google already open its toolchain benchmark suite in early 2010, but there was no document specific to such important components. Furthermore, there was one file gone in public kog GIT, required by automated benchmark process. One year later, Google engineer finally put back the one to public. This implies the unusual way Google developed and delivered software.
Assuming good faith I would think this might just have been an oversight.
Do you know if anyone from community pointed this out to google using official android mailing lists/groups or a bug?
** Linaro's Approach to enable latest technologies
Linaro android team tries to do: (1) Document Android toolchain and related utilities in korg GIT as possible as we can.
That's good stuff and I think your wiki page is already a great contribution in that direction. What we should do though is run this through google eyes early by using official android mailing lists.
(2) Early adaptation of Linaro toolchain to Android build system and verify these output systematically.
ack. Do you know if those changes would be conflicting with what we do on "normal" linux side? e.g. do we need to maintain special android patches or can we merge those into our main trees?
(3) Backport Google changes to Linaro GCC and review in public.
This is really tricky as you said. Here again, we should propose this on android mailing lists to maybe get feedback from google team and maybe improve the way we work on that. Untangling a big patch based just on changelog feels really unefficient.
Also, we have to remember that if we pick changes out of _their_ tree, we cannot upstream those to fsf because we don't own copyright to those. Of course, for stuff they already pushed to 4.6 its not a problem to backport them from fsf trunk.
(4) Improve the deployment and validation flow by means of Linaro infrastructure.
my understanding is this:
1. we add support to build android toolchain from linaro branches to our cloud build service 2. we do this so that we either produce a full toolchain tarball that can be installed under /opt/android or a NDK tarball (or both) 3. we improve our android platform build infrastructure to allow using latest daily toolchain tarball and then we build android with a) google toolchain and b) linaro toolchain; in this way we get daily android builds for both toolchains that can go into the linaro validation farm and get the typical validation/testing and benchmarking done.
(5) Build and test Android system with Linaro tools. Then, figure out the regressions caused by Linaro Toolchain and/or aggressive optimizations
right. I think that's covered with the point above, no? The android builds done with our toolchain would also be available in public, so you can do whatever you want on top of what we already test/validate/measure automatically in the validation farm with them.
(6) measure performance gain by Linaro tools
right. for this we need to define a set of open-source benchmarks to run and ensure that those are supported in our validation framework.
The detailed specification in wiki: https://wiki.linaro.org/Platform/Android/Specs/LinaroAndroidToolchain
** Implementation of Linaro toolchain for Android
We started from Android style toolchain build and move to Linaro GCC + ARM specific optimizations in mind. The initial work can be obtained by wiki: https://wiki.linaro.org/Platform/Android/Toolchain
We plan to maintain the following GIT repositories at least: * android/toolchain/build.git : Linaro-aware build system. Derived from Android toolchain build system, it can handle Linaro-GCC and Linaro snapshot/bzr. * android/toolchain/gcc-patches.git : Patchset to be applied on top of Linaro-GCC release/snapshots
I think thats fine. however, how do we ensure that we have patches that always apply to both release/snapshots? do we maintain branches for gcc-patches.git in case you need two versions of patch X if the linaro gcc codebase diverged?
The reference builder script output: $ ./linaro-build.sh --help --prefix-dir= Specify where to install (default: /tmp/android-toolchain-eabi) --gcc-src-dir= Specify where linaro gcc source is (in <toolchain>/gcc) --apply-gcc-patch=(yes|no) Apply-patch which in <toolchain>/gcc-patches directory (default: no)
Current verified combinations: * gcc-linaro: 4.5-2011.02-0 * binutils: 2.20.1 * gmp: 4.2.4 * mpfr: 2.4.1
Only gcc is replaced by gcc-linaro: 4.5-2011.02-0 and others are checked out from korg GIT.
do we need to do something like --gcc-src-dir and -patches for binutils, gmp and mpfr as well? or would we be only interested in improving/fixing gcc for now?
Waybe we also want to support protocol schemes like git: http: and bzr+ssh:/lp: for the --gcc-src= argument. this would then automatically download/branch the source tree from the given location. What do you think?
** Summary of gcc-patches
"gcc-patches" are used as "backport" from Google changes into Linaro gcc base. Here is the summary at present:
0001-Add-linux-android.patch Add linux-android
0002-Add-support-for-Bionic-C-library.patch Add support for Bionic C library
0003-Support-compilation-for-Android-platform.patch Support compilation for Android platform
0004-Add-multilib-configuration-for-arm-linux-androideabi.patch Add multilib configuration for arm-linux-androideabi
0005-Fix-gthr-posix.h-to-support-Bionic.patch Fix gthr-posix.h to support Bionic
0006-Add-untested-support-for-Bionic-to-libstdc.patch Add [untested] support for Bionic to libstdc++
These patches are taken from Maxim Kuvyrkov of CodeSourcery in gcc-4.6 branch. Of course, we can always add changes by Google or other Android specific adaptation by this model.
Can we get a toolchain example tarball done and uploaded to people.linaro.org? I would like to verify that those work out of the box with gingerbread and if so, i would like to see those land in the main toolchain WG branch rather than adding them to our gcc-patches tree.
** Planned improvements over Linaro toolchain for Android
(1) GCC multilib setting Default: arm, fpu and thumb. The prebuilt google toolchain use: armv5te and mandroid. We should focus on ARMv7. (2) HardFP-ABI Support for Android. (3) Patch management: Better to get the Android patches into Linaro-GCC tree eventually. (4) Build system improvement. Don't have to build gmp, mpfr everytime, and provide option to build without gdb. (5) Enable LTO (Link Time Optimization, introduced since gcc-4.5) in Android TARGET_GLOBAL_CFLAGS (6) Verify the functionality of FDO (Feedback Directed Optimization) and introduce the approaches to integrate.
I really think those topics should be executed by the toolchain WG rather than in platform. I am happy that we give them guidance and support them by providing them with easy to use tools to get their job done. Also feeding them with topics is great. Please talk to Michael Hope and ask him how he wants to collect those android toolchain optimization topic ideas. Could be good input for our 11.11 requirements gathering process.
** Toward Android NDK
Once Linaro toolchain for Android is ready to use, it is time to re-package Android NDK by Linaro toolchain. To do that, extra build configuration, sysroot, is required. According to Android Release Cycle & Phases[5], the repacked NDK should be verified one moth after Android public release.
That sounds like a great idea. What's the a benefit/difference of shipping an NDK compared to just shipping a "normal" toolchain binary tarball for this purpose?
Inside Google, there is a dedicated compiler team working on GNU
Toolchain for various purposes including server-side computing, Android, Chrome OS, etc. Google engineers submit patches to upstream for public review and maintain the toolchain for Android. Along with each Android Open Source Prokect (AOSP) release, there is a special branch in korg GIT [2] for hosting the GPL'd toolchain source code modified by Google. Usually, file "README.google" mentioned the summary, but it is not developer friendly because several changes were done within one GIT commit.
Please refer to wiki for details: https://wiki.linaro.org/Platform/Android/UpstreamToolchain
thats a good wiki page. thanks for the content. If I read the skia example correctly, we could add a test to our "normal" abrek testsuite that uses our daily android toolchain and run the skia benchmark? e.g. we could start doing this benchmarking even without having a validation solution ready for android targets?
Please let's talk to Paul how we can get the android toolchain to /opt/android as part of abrek and lets try to add this to our abrek testsuites. Until we have daily toolchain builds it would be OK to download the android toolchain tarball from a fixed place from people.linaro.org I guess.
I think there's going to be more to it than that. My understand based on some quick research about skia is that it is a 2d graphics benchmark and needs to be run under android itself. Which means we first need android booting on our hardware. Am I mistaken about this?
Do we have any idea how far we are away from having linaro android images?
Thanks, Paul Larson
FWIW, skia certainly isn't android only and, at least for the purposes of getting the validation side of things up and running, could be run on a non-android build (Jammy is likely doing something like this for his work, though not oriented at abrek at the moment). Of course, I could be over-simplifying here ;-).
cheers, Jesse
On Fri, Feb 25, 2011 at 9:05 AM, Paul Larson paul.larson@linaro.org wrote:
Inside Google, there is a dedicated compiler team working on GNU
Toolchain for various purposes including server-side computing, Android, Chrome OS, etc. Google engineers submit patches to upstream for public review and maintain the toolchain for Android. Along with each Android Open Source Prokect (AOSP) release, there is a special branch in korg GIT [2] for hosting the GPL'd toolchain source code modified by Google. Usually, file "README.google" mentioned the summary, but it is not developer friendly because several changes were done within one GIT commit.
Please refer to wiki for details: https://wiki.linaro.org/Platform/Android/UpstreamToolchain
thats a good wiki page. thanks for the content. If I read the skia example correctly, we could add a test to our "normal" abrek testsuite that uses our daily android toolchain and run the skia benchmark? e.g. we could start doing this benchmarking even without having a validation solution ready for android targets?
Please let's talk to Paul how we can get the android toolchain to /opt/android as part of abrek and lets try to add this to our abrek testsuites. Until we have daily toolchain builds it would be OK to download the android toolchain tarball from a fixed place from people.linaro.org I guess.
I think there's going to be more to it than that. My understand based on some quick research about skia is that it is a 2d graphics benchmark and needs to be run under android itself. Which means we first need android booting on our hardware. Am I mistaken about this?
Do we have any idea how far we are away from having linaro android images?
Thanks, Paul Larson
linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
On 25 February 2011 22:28, Alexander Sack asac@linaro.org wrote:
On Wed, Feb 23, 2011 at 8:28 AM, Jim Huang jserv@0xlab.org wrote:
I would like to make a proposal about utilizing Linaro toolchain for Android and NDK (Native Development Kit)[1].
Added linaro-toolchai list in Cc.
** Motivation
There are some different perspectives between Linaro toolchain and Google Android toolchain including technical and non-technical considerations. It doesn't really work if we only replace prebuilt toolchain with Linaro toolchain because of the compatibility of Android system utilities such as ELF prelinker. Also, since Android is developed in relatively closed
I don't have enough background to understand this "ELF prelinker" stuff. Are you saying that because of the way how android links stuff we cannot have one code base for gcc that works for both, android and "normal libc linux"?
Take Bug #707487 for example: https://bugs.launchpad.net/binutils-linaro/+bug/707487
It is evident that Android's system utilities like soslim ("strip" implementation) and apriori ("prelink" implementation) expect the specific output of GNU Toolchain, but it sometimes varies since we would take Linaro's toolchain.
environment (Google style open source model), a great amount of software components are not always verified by different toolchain or build configurations. This proposal attempts to
ack. thats what we want to do. Of course, we cannot really verify what is going on behind closed walls, but we can continuously build android with our toolchain and fix issues due that in android public master and if even that doesn't work we can ensure that our android trees always work nicely with both, our gcc and android gcc.
Android team is known to work on this field already.
Another thing is to make our toolchain easily consumable (like the NDK you mention at the bottom); this will increase chances that someone from google can eventually take a look at what we are doing etc. and also helps the community to use linaro toolchain to built their android distributions.
Agree.
establish the compact development flow to enable Linaro optimized ARM toolchain to build Android from scratch and verify it transparently. Eventually, Android can be the reference indicator as Linaro toolchain performance and reliability.
** Brief introduction to Google Android toolchain
Inside Google, there is a dedicated compiler team working on GNU Toolchain for various purposes including server-side computing, Android, Chrome OS, etc. Google engineers submit patches to upstream for public review and maintain the toolchain for Android. Along with each Android Open Source Prokect (AOSP) release, there is a special branch in korg GIT [2] for hosting the GPL'd toolchain source code modified by Google. Usually, file "README.google" mentioned the summary, but it is not developer friendly because several changes were done within one GIT commit.
Please refer to wiki for details: https://wiki.linaro.org/Platform/Android/UpstreamToolchain
thats a good wiki page. thanks for the content. If I read the skia example correctly, we could add a test to our "normal" abrek testsuite that uses our daily android toolchain and run the skia benchmark? e.g. we could start doing this benchmarking even without having a validation solution ready for android targets?
If adb is supposed to work well on target, then you can easily use "bench.py" script mentioned in the above wiki to do several benchmarking.
Please let's talk to Paul how we can get the android toolchain to /opt/android as part of abrek and lets try to add this to our abrek testsuites. Until we have daily toolchain builds it would be OK to download the android toolchain tarball from a fixed place from people.linaro.org I guess.
Ok!
** What's wrong with Android upstream Toolchain?
In my opinion, list as following:
(1) Few information about Google improvement: Sometimes, we have to guess something from implicit GIT commitlog such as "commit gcc-4.4.3 which is used to build gcc-4.4.3 Android toolchain in master"[3]. It is hard to track and get verified carefully.
yes, that feels like a messy situation. Do we know why they don't commit the changes as individual commits but then in next step document what they changed?
I have no exact idea since I am just an observer regarding Android's GIT tree. Google engineers do send patches to FSF/GNU, but it is not always related to the GIT activities we have seen in korg.
You can search the keyword, "submit", in file gcc/gcc-4.4.3/README.google , and you will see some descriptions as following:
gcc/cp/cp-lang.c gcc/gimple.c gcc/langhooks-def.h gcc/langhooks.h gcc/langhooks.c gcc/tree-flow.h gcc/tree-ssa-dce.c gcc/testsuite/g++.dg/tree-ssa/vptr_init_dse.C gcc/testsuite/g++.dg/tree-ssa/vptr_init_dse2.C Enhancing dead code elimination to eliminate useless vptr field initialization. Owner: davidxl Status: not submitted
gcc/fold-const.c gcc/Makefile.in Fix 2045297 Owner: davidxl Status: Not submitted
The information is too few to track for us since the above "Fix 2045297" tends to indicate Google bug database number instead of FSF's.
(2) Google specific improvements are absent in recent release, only enabled months later. For example, Google Compiler Team Lead, Dr. Shih-wei Liao, presented the improvements against GNU Toolchain in the middle of 2009.[4]. The report came with several impressive improvements like FDO (Feedback Directed Optimizations) and IPO (Inter-Procedure Optimizations). However, only some of them are public to AOSP and be integrated late in the middle of 2010 (Android Froyo; 2.2). Even FDO was merged in Android Froyo already, but there is few documentation and no robust method to verify by community members such as Linaro engineers.
you say that they don't publish the code for lets say the "gingerbread" toolchain in a timely fashion when they release gingerbread? Or do they ship a separate "fast" NDK/prebuilt for partners through secret channels?
I have no idea.
(3) For some reasons, Google tends to deliver stable (old) toolchain plus mainline backport. It is a safe and workable approach, but sometimes developers would expect to use the latest technologies as Linaro aims to bring to the world.
(4) Few readable documentation. For example, Google already open its toolchain benchmark suite in early 2010, but there was no document specific to such important components. Furthermore, there was one file gone in public kog GIT, required by automated benchmark process. One year later, Google engineer finally put back the one to public. This implies the unusual way Google developed and delivered software.
Assuming good faith I would think this might just have been an oversight.
Do you know if anyone from community pointed this out to google using official android mailing lists/groups or a bug?
Google engineers sometimes pick up the issues from Google Code: http://code.google.com/p/android/issues/list And, they do discuss on mailing-list: http://developer.android.com/community/
** Linaro's Approach to enable latest technologies
Linaro android team tries to do: (1) Document Android toolchain and related utilities in korg GIT as possible as we can.
That's good stuff and I think your wiki page is already a great contribution in that direction. What we should do though is run this through google eyes early by using official android mailing lists.
Got it.
(2) Early adaptation of Linaro toolchain to Android build system and verify these output systematically.
ack. Do you know if those changes would be conflicting with what we do on "normal" linux side? e.g. do we need to maintain special android patches or can we merge those into our main trees?
In fact, GCC 4.6 already merges Android specific patches with the help of CodeSourcery. We would initially backport these patches to linaro-gcc-4.5 branch for review. Luse Cheng already did it.
However, other parts are not related to Android directly, and they might be too aggressive to generic GCC optimization, that can be the reason why Google didn't submit first.
(3) Backport Google changes to Linaro GCC and review in public.
This is really tricky as you said. Here again, we should propose this on android mailing lists to maybe get feedback from google team and maybe improve the way we work on that. Untangling a big patch based just on changelog feels really unefficient.
Ok, I got your point. However, what we need is to create workable combination of Linaro kernel + Linaro toolchain for Android integration engineers.
Alexander, I need your help to catch the attention of someone at Google.
Also, we have to remember that if we pick changes out of _their_ tree, we cannot upstream those to fsf because we don't own copyright to those. Of course, for stuff they already pushed to 4.6 its not a problem to backport them from fsf trunk.
Thanks for notice.
(4) Improve the deployment and validation flow by means of Linaro infrastructure.
my understanding is this:
1. we add support to build android toolchain from linaro branches to our cloud build service 2. we do this so that we either produce a full toolchain tarball that can be installed under /opt/android or a NDK tarball (or both)
NDK doesn't need admin permission to install.
3. we improve our android platform build infrastructure to allow using latest daily toolchain tarball and then we build android with a) google toolchain and b) linaro toolchain; in this way we get daily android builds for both toolchains that can go into the linaro validation farm and get the typical validation/testing and benchmarking done.
Yes, it would be great.
(5) Build and test Android system with Linaro tools. Then, figure out the regressions caused by Linaro Toolchain and/or aggressive optimizations
right. I think that's covered with the point above, no? The android builds done with our toolchain would also be available in public, so you can do whatever you want on top of what we already test/validate/measure automatically in the validation farm with them.
Agree.
(6) measure performance gain by Linaro tools
right. for this we need to define a set of open-source benchmarks to run and ensure that those are supported in our validation framework.
The detailed specification in wiki: https://wiki.linaro.org/Platform/Android/Specs/LinaroAndroidToolchain
** Implementation of Linaro toolchain for Android
We started from Android style toolchain build and move to Linaro GCC + ARM specific optimizations in mind. The initial work can be obtained by wiki: https://wiki.linaro.org/Platform/Android/Toolchain
We plan to maintain the following GIT repositories at least: * android/toolchain/build.git : Linaro-aware build system. Derived from Android toolchain build system, it can handle Linaro-GCC and Linaro snapshot/bzr. * android/toolchain/gcc-patches.git : Patchset to be applied on top of Linaro-GCC release/snapshots
I think thats fine. however, how do we ensure that we have patches that always apply to both release/snapshots? do we maintain branches for gcc-patches.git in case you need two versions of patch X if the linaro gcc codebase diverged?
I might need help from toolchain WG.
The reference builder script output: $ ./linaro-build.sh --help --prefix-dir= Specify where to install (default: /tmp/android-toolchain-eabi) --gcc-src-dir= Specify where linaro gcc source is (in <toolchain>/gcc) --apply-gcc-patch=(yes|no) Apply-patch which in <toolchain>/gcc-patches directory (default: no)
Current verified combinations: * gcc-linaro: 4.5-2011.02-0 * binutils: 2.20.1 * gmp: 4.2.4 * mpfr: 2.4.1
Only gcc is replaced by gcc-linaro: 4.5-2011.02-0 and others are checked out from korg GIT.
do we need to do something like --gcc-src-dir and -patches for binutils, gmp and mpfr as well? or would we be only interested in improving/fixing gcc for now?
I think focusing on linaro-gcc is pretty good. We can follow the original combination of Google.
Waybe we also want to support protocol schemes like git: http: and bzr+ssh:/lp: for the --gcc-src= argument. this would then automatically download/branch the source tree from the given location. What do you think?
Agree.
** Summary of gcc-patches
"gcc-patches" are used as "backport" from Google changes into Linaro gcc base. Here is the summary at present:
0001-Add-linux-android.patch Add linux-android
0002-Add-support-for-Bionic-C-library.patch Add support for Bionic C library
0003-Support-compilation-for-Android-platform.patch Support compilation for Android platform
0004-Add-multilib-configuration-for-arm-linux-androideabi.patch Add multilib configuration for arm-linux-androideabi
0005-Fix-gthr-posix.h-to-support-Bionic.patch Fix gthr-posix.h to support Bionic
0006-Add-untested-support-for-Bionic-to-libstdc.patch Add [untested] support for Bionic to libstdc++
These patches are taken from Maxim Kuvyrkov of CodeSourcery in gcc-4.6 branch. Of course, we can always add changes by Google or other Android specific adaptation by this model.
Can we get a toolchain example tarball done and uploaded to people.linaro.org? I would like to verify that those work out of the box with gingerbread and if so, i would like to see those land in the main toolchain WG branch rather than adding them to our gcc-patches tree.
Yes, I would like to do that later.
** Planned improvements over Linaro toolchain for Android
(1) GCC multilib setting Default: arm, fpu and thumb. The prebuilt google toolchain use: armv5te and mandroid. We should focus on ARMv7. (2) HardFP-ABI Support for Android. (3) Patch management: Better to get the Android patches into Linaro-GCC tree eventually. (4) Build system improvement. Don't have to build gmp, mpfr everytime, and provide option to build without gdb. (5) Enable LTO (Link Time Optimization, introduced since gcc-4.5) in Android TARGET_GLOBAL_CFLAGS (6) Verify the functionality of FDO (Feedback Directed Optimization) and introduce the approaches to integrate.
I really think those topics should be executed by the toolchain WG rather than in platform. I am happy that we give them guidance and support them by providing them with easy to use tools to get their job done. Also feeding them with topics is great. Please talk to Michael Hope and ask him how he wants to collect those android toolchain optimization topic ideas. Could be good input for our 11.11 requirements gathering process.
Agree.
** Toward Android NDK
Once Linaro toolchain for Android is ready to use, it is time to re-package Android NDK by Linaro toolchain. To do that, extra build configuration, sysroot, is required. According to Android Release Cycle & Phases[5], the repacked NDK should be verified one moth after Android public release.
That sounds like a great idea. What's the a benefit/difference of shipping an NDK compared to just shipping a "normal" toolchain binary tarball for this purpose?
NDK consists of some architecture specific helper scripts/headers to indicate the optimization flags and some combinations such as ARMv7 with/without NEON, etc.
If we provide NDK directly, users don't have to consider the above integration issues as far as I know.
Sincerely, Jim Huang (jserv) Android Team
On 01/03/11 07:00, Jim Huang wrote:
I think thats fine. however, how do we ensure that we have patches
that always apply to both release/snapshots? do we maintain branches for gcc-patches.git in case you need two versions of patch X if the linaro gcc codebase diverged?
I might need help from toolchain WG.
I aim to have Android support in Linaro GCC soon, but I lack the ability to test it. In fact, I have wanted to have Android support in both February's and March's release, but it isn't going to happen now.
I did build a Linaro GCC with the Android patches installed, and it did appear to build a lot of the Android tree, but the Android build is broken. There are problems with -Werror, but even when you work around those, it still won't build. :(
When somebody gives me a properly patched Android tree, that will build out-of-the-box with GCC 4.6, then I will be unblocked, and Linaro GCC will have Android support shortly afterwards. :)
Andrew
On 1 March 2011 18:07, Andrew Stubbs andrew.stubbs@linaro.org wrote:
On 01/03/11 07:00, Jim Huang wrote:
I think thats fine. however, how do we ensure that we have patches
that always apply to both release/snapshots? do we maintain branches for gcc-patches.git in case you need two versions of patch X if the linaro gcc codebase diverged?
I might need help from toolchain WG.
I aim to have Android support in Linaro GCC soon, but I lack the ability to test it. In fact, I have wanted to have Android support in both February's and March's release, but it isn't going to happen now.
hi Andrew,
Thanks for replying.
I did build a Linaro GCC with the Android patches installed, and it did appear to build a lot of the Android tree, but the Android build is broken. There are problems with -Werror, but even when you work around those, it still won't build. :(
When somebody gives me a properly patched Android tree, that will build out-of-the-box with GCC 4.6, then I will be unblocked, and Linaro GCC will have Android support shortly afterwards. :)
Jeremy and I will help you to resolve Android build issue later. At the moment, you can safely remove "-Werror" from Android build system.
Regards, -jserv
-
Alexander Sack -
Andrew Stubbs -
Jesse Barker -
Jim Huang -
Jim Huang -
Paul Larson