On 23 July 2012 03:36, Alexander Sack asac@linaro.org wrote:

On Sun, Jul 22, 2012 at 5:38 PM, Zach Pfeffer zach.pfeffer@linaro.org wrote:

...

Hey Bero/Paul

Who sets up:

http://snapshots.linaro.org/android/~linaro-android/toolchain-4.7-2012.07/1/...

and

http://android-build.linaro.org/download/linaro-android_toolchain-4.7-bzr/la...

I'm asking because I'd like to solve a problem our users are running into. The instructions listed on top of each build pages are wrong. They say things like:

$ wget --no-check-certificate http://android-build.linaro.org/download/linaro-android_toolchain-4.7-bzr/la... $ tar -jxvf android-toolchain-eabi-linaro-*

when the build is using

TOOLCHAIN_URL=http://snapshots.linaro.org/android/~linaro-android/toolchain-4.7-2012.07/1/...

I'd like to fix this so that the builds pull the right thing. I was thinking that we could commit the toolchain tarballs to git and then just pull things as part of the manifest. That would help our local users. We could also take a half step and write a script that would setup the environment and have people (and the build server) also use that.

We have the same issue with:

KERNEL_CONFIG=omap4plus_defconfig

Which should be in the build.

Anyway, I think its time to start making:

MANIFEST_REPO=git://android.git.linaro.org/platform/manifest.git MANIFEST_BRANCH=linaro_android_4.0.4 MANIFEST_FILENAME=tracking-panda.xml TARGET_PRODUCT=pandaboard TARGET_SIMULATOR=false TARGET_BUILD_VARIANT=tests SOURCE_OVERLAY="open/20120716/build-info.tar.bz2" TOOLCHAIN_URL=http://snapshots.linaro.org/android/~linaro-android/toolchain-4.7-2012.07/1/... TOOLCHAIN_TRIPLET=arm-linux-androideabi REPO_SEED_URL=http://android-build.linaro.org/seed/uniseed.tar.gz LAVA_SUBMIT=1 LAVA_SUBMIT_FATAL=0 TARGET_NO_HARDWAREGFX=1 KERNEL_CONFIG=omap4plus_defconfig LAVA_ANDROID_BINARIES=False LAVA_TEST_PLAN="busybox,0xbench,glmark2,skia,v8,mmtest,monkey,monkey_long_run" MONKEY_RUNNER_URL_1="git://android.git.linaro.org/test/linaro/android/system.git" CTS_TIMEOUT=36000

turn into:

MANIFEST_REPO=git://android.git.linaro.org/platform/manifest.git MANIFEST_BRANCH=linaro_android_4.0.4 MANIFEST_FILENAME=tracking-panda.xml

Personally, I am not against having a toolchain in git, and we probably want to use prebuilt/ directory for that. However, due to the way git stores big binaries we most likely do not want to commit daily toolchains there; just release candidates or even releases.

You know, we could check in each point on tip and then when we switch over to the release, delete the tip builds - just keeping the last one. That would save space and let us unify things.

In that case your doc problems probably wouldn't be going magically away and you would still need to come up with generic instructions on how to build with latest daily toolchain without having that committed to git.

-- Alexander Sack Technical Director, Linaro Platform Teams http://www.linaro.org | Open source software for ARM SoCs http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog

On 23 July 2012 04:42, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:

Well, that's probably a known issue, that instructions on individual jobs' pages are out-of-date/incorrect. One reason being that we have them duplicated multiple times, so it's very hard/impossible to keep them all up to date over extended period of time. One solution proposed earlier was to not have adhoc per-job instructions, but instead make those pages link to one central wiki page, with single instructions applicable to all builds, maintained and up-to-date.

+1 -- just need to make sure that wiki page is actually kept up to date, and that it does list all weird issues with certain builds.

...

I was thinking that we could commit the toolchain tarballs to git and then just pull things as part of the manifest. That would help our local users.

There're 2 problems with that. 1st: who exactly will commit it to git? Again, toolchain build is automated daily process, and automated build script cannot commit to git (unless we're talking security issues).

We can probably make that reasonably save by limiting commits by the build script to one IP address?

2nd: git (or any other source code version control system) is not designed to store large binary blobs and cannot handle them efficiently. So, toolchain tarball is currently 88Mb. In ten days, we'll have extra 880Mb in git repository.

That's a real problem. I'd propose not putting in the tarballs anyway (makes a git diff hard...). Instead, we could replace the prebuilts/* repositories in AOSP. That should make the load a lot less, given most of the time, not all subprojects change at the same time. It's still putting a lot of binaries into git though.

Obviously another alternative is to just build the toolchain as part of the normal build process, but that would make builds a lot more time intensive.

ttyl bero

Hello,

On Mon, 23 Jul 2012 12:36:35 -0700 Bernhard Rosenkränzer bernhard.rosenkranzer@linaro.org wrote:

On 23 July 2012 04:42, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:

...

Well, that's probably a known issue, that instructions on individual jobs' pages are out-of-date/incorrect. One reason being that we have them duplicated multiple times, so it's very hard/impossible to keep them all up to date over extended period of time. One solution proposed earlier was to not have adhoc per-job instructions, but instead make those pages link to one central wiki page, with single instructions applicable to all builds, maintained and up-to-date.

+1 -- just need to make sure that wiki page is actually kept up to date, and that it does list all weird issues with certain builds.

...

...

I was thinking that we could commit the toolchain tarballs to git and then just pull things as part of the manifest. That would help our local users.

There're 2 problems with that. 1st: who exactly will commit it to git? Again, toolchain build is automated daily process, and automated build script cannot commit to git (unless we're talking security issues).

We can probably make that reasonably save by limiting commits by the build script to one IP address?

Yes, sure we can limit/"fortify" access for such bot account if decided to go that way.

...

2nd: git (or any other source code version control system) is not designed to store large binary blobs and cannot handle them efficiently. So, toolchain tarball is currently 88Mb. In ten days, we'll have extra 880Mb in git repository.

That's a real problem. I'd propose not putting in the tarballs anyway (makes a git diff hard...). Instead, we could replace the prebuilts/* repositories in AOSP. That should make the load a lot less, given most of the time, not all subprojects change at the same time. It's still putting a lot of binaries into git though.

Obviously another alternative is to just build the toolchain as part of the normal build process, but that would make builds a lot more time intensive.

Yes, I wonder if putting toolchain in git will, while solving some problems, create other (delayed) ones. There're other alternatives besides those you mentioned - for example, instead of building toolchain as part of android build, there can be native android makefile to download that toolchain. Then, instead of committing entire toolchain, it will be enough to just commit changes to that Makefile to reference new toolchain...

ttyl bero

On 24 July 2012 07:20, Bernhard Rosenkränzer bernhard.rosenkranzer@linaro.org wrote:

On Jul 24, 2012 1:59 PM, "Paul Sokolovsky" paul.sokolovsky@linaro.org wrote:

...

Yes, I wonder if putting toolchain in git will, while solving some problems, create other (delayed) ones. There're other alternatives besides those you mentioned - for example, instead of building toolchain as part of android build, there can be native android makefile to download that toolchain.

Yes, but that would have pretty much the same issues as the current TOOLCHAIN_URL solution. Especially once build configs are in git, that will come down to pretty much the same thing we're doing already.

Okay, so I'd like to propose that we move forward with the git approach since it is part of the standard AOSP flow and will improve build reproducability.

Any other option gets us where we're at today:

Builds don't follow standard Android build procedures Disconnecting the toolchain from the build causes our users confusion and is always a source of problems for our users

Should we spin up a separate git to host the toolchain (and other binaries) so that we can segregate any binary blob effects?

ttyl bero

On 24 July 2012 11:15, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:

On Tue, 24 Jul 2012 09:01:36 -0500 Zach Pfeffer zach.pfeffer@linaro.org wrote:

...

On 24 July 2012 07:20, Bernhard Rosenkränzer bernhard.rosenkranzer@linaro.org wrote:

...

On Jul 24, 2012 1:59 PM, "Paul Sokolovsky" paul.sokolovsky@linaro.org wrote:

...

Yes, I wonder if putting toolchain in git will, while solving some problems, create other (delayed) ones. There're other alternatives besides those you mentioned - for example, instead of building toolchain as part of android build, there can be native android makefile to download that toolchain.

Yes, but that would have pretty much the same issues as the current TOOLCHAIN_URL solution. Especially once build configs are in git, that will come down to pretty much the same thing we're doing already.

Okay, so I'd like to propose that we move forward with the git approach since it is part of the standard AOSP flow and will improve build reproducability.

Any other option gets us where we're at today:

Builds don't follow standard Android build procedures Disconnecting the toolchain from the build causes our users confusion and is always a source of problems for our users

Should we spin up a separate git to host the toolchain (and other binaries) so that we can segregate any binary blob effects?

Yes, if you're guys sure you need it and ok with risks posed, I'd suggest that you create prebuilt-linaro as was proposed already, populate it manually, switch (some/test) Linaro builds to it from AOSP's prebuilt/ , and make sure that it overall works well for you. Then, when Danilo is back (2nd week of Aug), you can schedule automatic toolchain commit (if that's what you want) via him. If you think it's of higher priority, please talk to David.

I'm also on vacation from Thurs, Georgy remains on maintenance team to cover any immediate questions regarding Android infra.

Paul, when something like:

http://snapshots.linaro.org/android/~linaro-android/toolchain-trunk/122/andr...

is produced, is it just a matter of checking it into git instead of/or inaddition to copying it to snapshots?

Thanks, Paul

Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog

On 24 July 2012 12:06, Bernhard Rosenkränzer bernhard.rosenkranzer@linaro.org wrote:

On 24 July 2012 09:23, Zach Pfeffer zach.pfeffer@linaro.org wrote:

...

Paul, when something like:

http://snapshots.linaro.org/android/~linaro-android/toolchain-trunk/122/andr...

is produced, is it just a matter of checking it into git instead of/or inaddition to copying it to snapshots?

I think it comes down to

git clone git://android.git.linaro.org/linaro-prebuilts/gcc.git (or whatever) git checkout -b trunk origin/trunk tar xf android-toolchain-eabi-trunk-daily-linux-x86.tar.bz2 git commit -as -m "Update" git checkout -b 4.7 origin/4.7 tar xf android-toolchain-eabi-4.7-daily-linux-x86.tar.bz2 git commit -as -m "Update" git checkout -b 4.6 origin/4.6 tar xf android-toolchain-eabi-4.6-daily-linux-x86.tar.bz2 git commit -as -m "Update" git push ssh://gccbot@android.git.linaro.org:29418/linaro-prebuilts/gcc.git 4.6 4.7 trunk

That looks pretty cool Bero, thanks. Now we just need somewhere to run it.

ttyl bero