Hello,
Following maintenance of android-build.linaro.org due to EC2 issues, I also carved some time to finally do performance comparison of seeded builds vs normal ones, using repo bundles (support of which was added in the spring for our tree). Well, freshly built seed for 4.3 branch is 18Gb, which is big bump from 11Gb previous initial size as I remember it.
Surprisingly, with unloaded master, it downloads in 7-8mins, just like (on average) before. But what follows is 30mins checkout. If seed is not used, the checkout (with automagic bundle download) takes ~40mins. So, as it stands, seed no longer provides any performance advantages. One way to explain it is the whole idea of seed is to pack small git objects into big archive, and that's just what bundles do eithern albeit on repository vs entire tree level.
I'd still expect seed to work somewhat faster, and have a feeling that checkout from it of 30mins is due to some non-optimal handling from newer repo versions, but I don't have anything to back that except my intuition. And this work at whole was an excursion from another things pressing in the queue, so I have nothing else to propose but:
1. To leave latest produced seed there, but disable (resource-consuming) cronjobs to update it. 2. Gradually migrate jobs on android-build off it.
Let me know if there're concerns/suggestions.
On 26 September 2013 22:47, Paul Sokolovsky paul.sokolovsky@linaro.orgwrote:
Hello,
Following maintenance of android-build.linaro.org due to EC2 issues, I also carved some time to finally do performance comparison of seeded builds vs normal ones, using repo bundles (support of which was added in the spring for our tree). Well, freshly built seed for 4.3 branch is 18Gb, which is big bump from 11Gb previous initial size as I remember it.
Surprisingly, with unloaded master, it downloads in 7-8mins, just like (on average) before. But what follows is 30mins checkout. If seed is not used, the checkout (with automagic bundle download) takes ~40mins. So, as it stands, seed no longer provides any performance advantages. One way to explain it is the whole idea of seed is to pack small git objects into big archive, and that's just what bundles do eithern albeit on repository vs entire tree level.
I'd still expect seed to work somewhat faster, and have a feeling that checkout from it of 30mins is due to some non-optimal handling from newer repo versions, but I don't have anything to back that except my intuition. And this work at whole was an excursion from another things pressing in the queue, so I have nothing else to propose but:
- To leave latest produced seed there, but disable
(resource-consuming) cronjobs to update it. 2. Gradually migrate jobs on android-build off it.
Let me know if there're concerns/suggestions.
Sounds reasonable. Lets start migratinfg the builds.
-- Best Regards, Paul
Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
Maybe it is the repo groups that make the download time to ~40 minutes. we only sync the necessary repositories(not all in the manifest) for that specified build now. but I just guess, did not do any test:)
On 27 September 2013 17:52, Vishal Bhoj vishal.bhoj@linaro.org wrote:
On 26 September 2013 22:47, Paul Sokolovsky paul.sokolovsky@linaro.orgwrote:
Hello,
Following maintenance of android-build.linaro.org due to EC2 issues, I also carved some time to finally do performance comparison of seeded builds vs normal ones, using repo bundles (support of which was added in the spring for our tree). Well, freshly built seed for 4.3 branch is 18Gb, which is big bump from 11Gb previous initial size as I remember it.
Surprisingly, with unloaded master, it downloads in 7-8mins, just like (on average) before. But what follows is 30mins checkout. If seed is not used, the checkout (with automagic bundle download) takes ~40mins. So, as it stands, seed no longer provides any performance advantages. One way to explain it is the whole idea of seed is to pack small git objects into big archive, and that's just what bundles do eithern albeit on repository vs entire tree level.
I'd still expect seed to work somewhat faster, and have a feeling that checkout from it of 30mins is due to some non-optimal handling from newer repo versions, but I don't have anything to back that except my intuition. And this work at whole was an excursion from another things pressing in the queue, so I have nothing else to propose but:
- To leave latest produced seed there, but disable
(resource-consuming) cronjobs to update it. 2. Gradually migrate jobs on android-build off it.
Let me know if there're concerns/suggestions.
Sounds reasonable. Lets start migratinfg the builds.
-- Best Regards, Paul
Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
linaro-android mailing list linaro-android@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-android
Hello YongQin Liu,
On Mon, 30 Sep 2013 00:01:09 +0800 YongQin Liu yongqin.liu@linaro.org wrote:
Maybe it is the repo groups that make the download time to ~40 minutes. we only sync the necessary repositories(not all in the manifest) for that specified build now. but I just guess, did not do any test:)
Well, 40 min is "long", it would be nice to decrease it ;-). But yes, unified manifest has to do something with that, and without groups, it would be completely unscalable. In particular, I had issues producing new seed throughout the week, and when I finally more or less (w/o private projects) synced it, it was >30Gb.
On 27 September 2013 17:52, Vishal Bhoj vishal.bhoj@linaro.org wrote:
On 26 September 2013 22:47, Paul Sokolovsky paul.sokolovsky@linaro.orgwrote:
Hello,
Following maintenance of android-build.linaro.org due to EC2 issues, I also carved some time to finally do performance comparison of seeded builds vs normal ones, using repo bundles (support of which was added in the spring for our tree). Well, freshly built seed for 4.3 branch is 18Gb, which is big bump from 11Gb previous initial size as I remember it.
Surprisingly, with unloaded master, it downloads in 7-8mins, just like (on average) before. But what follows is 30mins checkout. If seed is not used, the checkout (with automagic bundle download) takes ~40mins. So, as it stands, seed no longer provides any performance advantages. One way to explain it is the whole idea of seed is to pack small git objects into big archive, and that's just what bundles do eithern albeit on repository vs entire tree level.
I'd still expect seed to work somewhat faster, and have a feeling that checkout from it of 30mins is due to some non-optimal handling from newer repo versions, but I don't have anything to back that except my intuition. And this work at whole was an excursion from another things pressing in the queue, so I have nothing else to propose but:
- To leave latest produced seed there, but disable
(resource-consuming) cronjobs to update it. 2. Gradually migrate jobs on android-build off it.
Let me know if there're concerns/suggestions.
Sounds reasonable. Lets start migratinfg the builds.
-- Best Regards, Paul
Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog
linaro-android mailing list linaro-android@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-android
linaro-android@lists.linaro.org
-
Paul Sokolovsky -
Vishal Bhoj -
YongQin Liu