Hello,
Alexander, I've set up builds of Android toolchain from bzr: https://android-build.linaro.org/builds/~pfalcon/linaro-toolchain-4.5-bzr/ https://android-build.linaro.org/builds/~pfalcon/linaro-toolchain-4.6-bzr/
Jim, I had to revamp bzr build in linaro-build.sh: http://git.linaro.org/gitweb?p=people/pfalcon/android/toolchain/build.git%3B... Note that it now expects bzr branch with explicit version, e.g. lp:gcc-linaro/4.5 or lp:gcc-linaro/4.6 .
Well, now the issue: the builds above are not quite ready to be daily. That's because bzr checkout is slow, and is also quite big. Normal bzr clone takes 1.1Gb and cloud build takes 1hr instead of 30min with a tarball. I tried to play with bzr checkout --lightweight, but ctrl+c'ed it when it showed that it dealt with 1.3Gb of stuff (surprisingly, after break, I saw almost empty local empty structure, so I wonder if that wasn't d/l counter, but even then it's awfully slow).
So, where do we go from here? Unless there's tip export feature directly out of remote repo for bzr (bzr export seem to requite local working copy), we'd need to play mirror, this time bzr, again %).
On Mon, 13 Jun 2011 15:41:33 +0300, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
So, where do we go from here? Unless there's tip export feature directly out of remote repo for bzr (bzr export seem to requite local working copy), we'd need to play mirror, this time bzr, again %).
You can export remotely, but it probably has the same performance as "checkout --lightweight"
The mirror service may be the best way to deal with this.
Thanks,
James
On Tue, Jun 14, 2011 at 12:41 AM, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
Hello,
Alexander, I've set up builds of Android toolchain from bzr: https://android-build.linaro.org/builds/~pfalcon/linaro-toolchain-4.5-bzr/ https://android-build.linaro.org/builds/~pfalcon/linaro-toolchain-4.6-bzr/
Jim, I had to revamp bzr build in linaro-build.sh: http://git.linaro.org/gitweb?p=people/pfalcon/android/toolchain/build.git%3B... Note that it now expects bzr branch with explicit version, e.g. lp:gcc-linaro/4.5 or lp:gcc-linaro/4.6 .
Minor nit: these are called 'series'.
Well, now the issue: the builds above are not quite ready to be daily. That's because bzr checkout is slow, and is also quite big. Normal bzr clone takes 1.1Gb and cloud build takes 1hr instead of 30min with a tarball. I tried to play with bzr checkout --lightweight, but ctrl+c'ed it when it showed that it dealt with 1.3Gb of stuff (surprisingly, after break, I saw almost empty local empty structure, so I wonder if that wasn't d/l counter, but even then it's awfully slow).
So, where do we go from here? Unless there's tip export feature directly out of remote repo for bzr (bzr export seem to requite local working copy), we'd need to play mirror, this time bzr, again %).
Make an initial shared repository, take an initial branch, tar it up, store it somewhere in the same zone, and use that to seed the checkout. That should cut the checkout to minutes and ~10 MB.
See: https://wiki.linaro.org/WorkingGroups/ToolChain/BzrTips
-- Michael
Hello Michael,
On Tue, 14 Jun 2011 08:41:21 +1200 Michael Hope michael.hope@linaro.org wrote:
[]
Note that it now expects bzr branch with explicit version, e.g. lp:gcc-linaro/4.5 or lp:gcc-linaro/4.6 .
Minor nit: these are called 'series'.
I understand that in Launchpad's terms it's series, but there's doesn't seem to be such notion on bzr level, so how they're represented in bzr - are they 2 separate repositories, or 2 branches of the same repo?
Well, now the issue: the builds above are not quite ready to be daily. That's because bzr checkout is slow, and is also quite big. Normal bzr clone takes 1.1Gb and cloud build takes 1hr instead of 30min with a tarball. I tried to play with bzr checkout --lightweight, but ctrl+c'ed it when it showed that it dealt with 1.3Gb of stuff (surprisingly, after break, I saw almost empty local empty structure, so I wonder if that wasn't d/l counter, but even then it's awfully slow).
So, where do we go from here? Unless there's tip export feature directly out of remote repo for bzr (bzr export seem to requite local working copy), we'd need to play mirror, this time bzr, again %).
Make an initial shared repository, take an initial branch, tar it up, store it somewhere in the same zone, and use that to seed the checkout. That should cut the checkout to minutes and ~10 MB.
Yes, I was thinking about some adhoc mirroring using cron, but this seems like even better idea, thanks.
See: https://wiki.linaro.org/WorkingGroups/ToolChain/BzrTips
-- Michael
On Wed, Jun 15, 2011 at 12:16 AM, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
Hello Michael,
On Tue, 14 Jun 2011 08:41:21 +1200 Michael Hope michael.hope@linaro.org wrote:
[]
Note that it now expects bzr branch with explicit version, e.g. lp:gcc-linaro/4.5 or lp:gcc-linaro/4.6 .
Minor nit: these are called 'series'.
I understand that in Launchpad's terms it's series, but there's doesn't seem to be such notion on bzr level, so how they're represented in bzr
- are they 2 separate repositories, or 2 branches of the same repo?
Interesting question. Here's my understanding: * They are both branches * The could be stored as different repos * Both are stacked on top of another branch to reduce the download and storage cost * Both are originally forked from different places of a different branch
How this ties in with the repo implementation I'm not sure.
I think the technical history of the gcc-linaro branch is: * lp:gcc is a automatic import from GCC SVN * The lp:gcc-linaro stacked branch was created from the lp:gcc/4.5 branch * The lp:gcc-linaro/4.5 was branched off lp:gcc/4.5 and stacked on top of gcc-linaro * The lp:gcc-linaro/4.6 was branched off lp:gcc/4.6 and stacked on top of the 4.5 derrived lp:gcc-linaro
-- Michael
On Mon, Jun 13, 2011 at 2:41 PM, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
So, where do we go from here? Unless there's tip export feature directly out of remote repo for bzr (bzr export seem to requite local working copy), we'd need to play mirror, this time bzr, again %).
For the reason that it is trivial to do release manifests (pinned) I would really love to see bzr feature (as well as static tarball feature) to be available in repo and then use our repo mirror to do the mirroring in the cloud for that as well. Could you check how much effort that is to add support to repo and to extend our cloud mirror to also understand bzr/tarballs?
Based on that we could decide whether we want to have a quick hack to get down the branching time quickly or if we wait till we can do this right using repo all together.
Hello Alexander,
On Wed, 15 Jun 2011 12:19:23 +0200 Alexander Sack asac@linaro.org wrote:
On Mon, Jun 13, 2011 at 2:41 PM, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
So, where do we go from here? Unless there's tip export feature directly out of remote repo for bzr (bzr export seem to requite local working copy), we'd need to play mirror, this time bzr, again %).
For the reason that it is trivial to do release manifests (pinned) I would really love to see bzr feature (as well as static tarball feature) to be available in repo and then use our repo mirror to do the mirroring in the cloud for that as well.
I'm not sure I understand exactly, do you mean repo as a (generic) repository or repo the Google's tool? I'm not sure it's really good idea (in terms of maintenance) to add bzr/tarball support to repo tool. Actually, per some recent discussion, there're good reasons to make a generic git mirror, and implement repo mirror using it.
As for release manifest, I guess I understand what you mean - we want to know from which components' versions given artifact is built, but neither I think that we can find generic solution for that (or that repo release manifest is ideal solution for this). IMHO, this should be handled on case by case basis. For example, for toolchain, it would make sense to include bzr revision from which it was built into tarball filename.
Could you check how much effort that is to add support to repo and to extend our cloud mirror to also understand bzr/tarballs?
For patching repo, we'd want to understand why we do it and for whom. Would Google accept it upstream? Somehow I doubt that, repo is a git tool which covers very specific needs of Android project. Otherwise, why would we want to maintain our own patchset on top of Google's hack-tool, which essentially just duplicates git submodules functionality?
As for our mirror service, adding such support in maintainable manner would require its complete rewrite (which will also allow to implement other functionality and improvements). Based on the currently scheduled blueprints related to cloud-buildd, closest time we can discuss and plan such rewrite in detail is at 11.08 sprint, and schedule implementation closer to the end of cycle.
Based on that we could decide whether we want to have a quick hack to get down the branching time quickly or if we wait till we can do this right using repo all together.
Thanks, Paul
On Thu, 16 Jun 2011 14:47:59 +0300, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
As for our mirror service, adding such support in maintainable manner would require its complete rewrite (which will also allow to implement other functionality and improvements).
Adding support for what would require a complete rewrite?
Thanks,
James
Hello James,
On Thu, 16 Jun 2011 16:43:57 -0400 James Westby james.westby@canonical.com wrote:
On Thu, 16 Jun 2011 14:47:59 +0300, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
As for our mirror service, adding such support in maintainable manner would require its complete rewrite (which will also allow to implement other functionality and improvements).
Adding support for what would require a complete rewrite?
Adding support for general/flexible mode of operation, and extending scope. In particular, following things were discussed/under consideration:
1. Implement generic git mirror. 2. Implement repo mirroring in terms of the above. 3. Implement more flexible mirror pre/postprocessing, to allow for improvements to areas like high-availability and updates scheduling. 4. The above would require either good rehashing of current Twisted control flow, or getting rif of Twisted. 5. Add HTTP and bzr mirroring.
Thanks,
James
-
Alexander Sack -
James Westby -
Michael Hope -
Paul Sokolovsky