Hello,
There were increasing Android build times during last month again (going over 3.5hrs). Yesterday, I re-created the seed, but that didn't improve times match. I did more stats on builds and it turns out that we spend almost 1.5hrs in copying ever-growing overlay collection to the slaves via sluggish CopyToSlave Jenkins plugin.
That's issues known for at least half-year, and was in queue for being reworked. I didn't work on android-build closely for few months, but now that I'm back on maintenance, I'd like to start with fixing this long-overdue issue.
The idea was to pull all the needed overlay, straight to a slave. The complication here was handling of license protected files. ci.linaro.org had the same issue, and used a script to handle automatic downloads for some time. However, as of now, it was switched to other process and the script git bitrotted with recent linaro-license-protection codebase changes. I took a look at revamping it, but then though that it may be a good idea to use simpler and more explicit process, following closely click-thru usage. So, if you want to use a license-protected binaries in a build, you'd need to specify:
ACCEPT_LICENSE=<license_id>
Where <license_id> is id under which license is registered in linaro-license-protection. One good way to figure it out is to read the license page at all (well, as HTML source, the id is included in some links). Wording of variable name (as well as of commands in underlying implementation) is also explicit to point that its usage construes license acceptance.
I prototypes these changes using https://android-build.linaro.org/builds/~pfalcon/panda-jb-gcc47-tilt-trackin... which is down to 2h10m build time. I'm ready to migrate all builds tomorrow, while we're early in cycle start.
Please let me know if there're any issues with approach or proposed migration timeframe, otherwise let's make android-build rock again.
Thanks, Paul
Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
On 6 February 2013 13:33, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
Hello,
There were increasing Android build times during last month again (going over 3.5hrs). Yesterday, I re-created the seed, but that didn't improve times match. I did more stats on builds and it turns out that we spend almost 1.5hrs in copying ever-growing overlay collection to the slaves via sluggish CopyToSlave Jenkins plugin.
That's issues known for at least half-year, and was in queue for being reworked. I didn't work on android-build closely for few months, but now that I'm back on maintenance, I'd like to start with fixing this long-overdue issue.
The idea was to pull all the needed overlay, straight to a slave. The complication here was handling of license protected files. ci.linaro.org had the same issue, and used a script to handle automatic downloads for some time. However, as of now, it was switched to other process and the script git bitrotted with recent linaro-license-protection codebase changes. I took a look at revamping it, but then though that it may be a good idea to use simpler and more explicit process, following closely click-thru usage. So, if you want to use a license-protected binaries in a build, you'd need to specify:
ACCEPT_LICENSE=<license_id>
Where <license_id> is id under which license is registered in linaro-license-protection. One good way to figure it out is to read the license page at all (well, as HTML source, the id is included in some links). Wording of variable name (as well as of commands in underlying implementation) is also explicit to point that its usage construes license acceptance.
Paul, can you list the links you're looking at?
I prototypes these changes using https://android-build.linaro.org/builds/~pfalcon/panda-jb-gcc47-tilt-trackin... which is down to 2h10m build time. I'm ready to migrate all builds tomorrow, while we're early in cycle start.
Please let me know if there're any issues with approach or proposed migration timeframe, otherwise let's make android-build rock again.
I really like the fact that you've shaved an hour off the build.
One thing I'm curious about, why does this work? It seems like licensing issues and autobuild issues should be separate concerns.
Another thing. What about support for an upcoming use case where we're need multiple EULAs. Would you use a comma separated list in the define?
Thanks, Paul
Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
linaro-android mailing list linaro-android@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-android
Hello,
On Wed, 6 Feb 2013 14:30:37 -0600 Zach Pfeffer zach.pfeffer@linaro.org wrote:
[]
ACCEPT_LICENSE=<license_id>
Where <license_id> is id under which license is registered in linaro-license-protection. One good way to figure it out is to read the license page at all (well, as HTML source, the id is included in some links). Wording of variable name (as well as of commands in underlying implementation) is also explicit to point that its usage construes license acceptance.
Paul, can you list the links you're looking at?
Yes, sorry, just didn't want to overload initial mail with info. Suppose, we want to add file from this dir as SOURCE_OVERLAY: http://snapshots.linaro.org/android/binaries/origen/20121220/ . Clicking on the file in clean browser session, we get license acceptance page, and Ctrl+U'ing it, at the bottom, where HTML corresponding to buttons is, we see:
<form action="/accept-license?lic=9fa99e541b3b291f0ba460104b354d0c&url=android/binaries/origen/20121220/vendor.tar.bz2" method="post">
"9fa99e541b3b291f0ba460104b354d0c" is the hash id for this particular license, so build config needs include
SOURCE_OVERLAY=origen/20121220/vendor.tar.bz2 ACCEPT_LICENSE=9fa99e541b3b291f0ba460104b354d0c
I prototypes these changes using https://android-build.linaro.org/builds/~pfalcon/panda-jb-gcc47-tilt-trackin... which is down to 2h10m build time. I'm ready to migrate all builds tomorrow, while we're early in cycle start.
Please let me know if there're any issues with approach or proposed migration timeframe, otherwise let's make android-build rock again.
I really like the fact that you've shaved an hour off the build.
One thing I'm curious about, why does this work? It seems like licensing issues and autobuild issues should be separate concerns.
Yep, it's just switching autobuild from using sluggish layers of proxies to fast direct downloads calls for need to deal with licensing.
Another thing. What about support for an upcoming use case where we're need multiple EULAs. Would you use a comma separated list in the define?
Yes, I thought we didn't use multiply licensed overlays, but if we will, ACCEPT_LICENSE can take comma or semicolon separated list, just as SOURCE_OVERLAY does.
On Thu, 7 Feb 2013 11:15:12 +0200 Paul Sokolovsky Paul.Sokolovsky@linaro.org wrote:
[]
SOURCE_OVERLAY=origen/20121220/vendor.tar.bz2 ACCEPT_LICENSE=9fa99e541b3b291f0ba460104b354d0c
[]
One thing I'm curious about, why does this work? It seems like licensing issues and autobuild issues should be separate concerns.
Ah, after talking with Fathi, I figure where confusion may come from - any licensing stuff I talked about is related to downloading tarballs as pointed by SOURCE_OVERLAY, not to licensing of binaries as produced by the build. The latter is handled by BUILDINFO.txt as previously, and orthogonal to what was discussed in these mails. I guess, to avoid confusion, the var should be named SOURCE_OVERLAY_ACCEPT_LICENSE.
On 7 February 2013 09:10, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
On Thu, 7 Feb 2013 11:15:12 +0200 Paul Sokolovsky Paul.Sokolovsky@linaro.org wrote:
[]
SOURCE_OVERLAY=origen/20121220/vendor.tar.bz2 ACCEPT_LICENSE=9fa99e541b3b291f0ba460104b354d0c
[]
One thing I'm curious about, why does this work? It seems like licensing issues and autobuild issues should be separate concerns.
Ah, after talking with Fathi, I figure where confusion may come from - any licensing stuff I talked about is related to downloading tarballs as pointed by SOURCE_OVERLAY, not to licensing of binaries as produced by the build. The latter is handled by BUILDINFO.txt as previously, and orthogonal to what was discussed in these mails. I guess, to avoid confusion, the var should be named SOURCE_OVERLAY_ACCEPT_LICENSE.
Okay cool.
Paul, since you're currently working through this, would you mind putting together a wiki explaining everything, especially the points about licensing, etc... A couple of examples would help too and would save a few emails down the road.
-- Best Regards, Paul
Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
Hello Zach,
On Thu, 7 Feb 2013 12:33:57 -0600 Zach Pfeffer zach.pfeffer@linaro.org wrote:
[]
guess, to avoid confusion, the var should be named SOURCE_OVERLAY_ACCEPT_LICENSE.
Okay cool.
Paul, since you're currently working through this, would you mind putting together a wiki explaining everything, especially the points about licensing, etc... A couple of examples would help too and would save a few emails down the road.
Yes, I've updated https://wiki.linaro.org/Platform/Android/LinaroAndroidBuildService#Build_Con... to include info about new var.
This doesn't seem to work. The license agreement URL seem to have changed.
On 8 February 2013 22:37, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
Hello Zach,
On Thu, 7 Feb 2013 12:33:57 -0600 Zach Pfeffer zach.pfeffer@linaro.org wrote:
[]
guess, to avoid confusion, the var should be named SOURCE_OVERLAY_ACCEPT_LICENSE.
Okay cool.
Paul, since you're currently working through this, would you mind putting together a wiki explaining everything, especially the points about licensing, etc... A couple of examples would help too and would save a few emails down the road.
Yes, I've updated
https://wiki.linaro.org/Platform/Android/LinaroAndroidBuildService#Build_Con... to include info about new var.
-- Best Regards, Paul
Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
linaro-android mailing list linaro-android@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-android
linaro-android@lists.linaro.org
-
Paul Sokolovsky -
Vishal Bhoj -
Zach Pfeffer