On Mon, Apr 23, 2012 at 2:15 PM, Paul Sokolovsky <paul.sokolovsky@linaro.org
wrote:
On Mon, 23 Apr 2012 12:26:19 +0200 Alexander Sack asac@linaro.org wrote:
On Mon, Apr 23, 2012 at 11:44 AM, Paul Sokolovsky < paul.sokolovsky@linaro.org> wrote:
Hello,
On Fri, we discovered that build started to take lot of time (30+ mins) on the CopyToSlave stage (one of the first on build startup) to copy EULA-protected binary blobs to a build slave. This adversely affected ability to restrict build/EULA handling changes/RCs (few timed out as was reported by Fathi).
This could be attributed only to http://snapshots.linaro.org/android/binaries/snowball/ receiving five 100MB drops over last 10 days, or 500% or so growth, which uncovered poor throughput of CopyToSlave plugin. Due to inconveniences it brings, I applied a quick dirty solution of mirroring only the latest tarball from http://snapshots.linaro.org/android/binaries/snowball/ . Let me know if you see any issues. And that solution needs to be cleaned/generalized of course.
This is about vendor.tar.bz2?
Does this prevent us from reproducing builds that use older tarballs?
Potentially, yes, realistically, noone complained yet ;-). Script doesn't delete anything, it just doesn't mirror old stuff, so unless someone deletes old files (I did), they stay there.
Can we make the mirroring smart and parse the info from the build config? e.g. mirror exactly the right tarball?
We can go deeper and deeper with applying adhoc workarounds which shouldn't be there in the first place, or can instead think out and implement access system which would allow our bot scripts to get access to things they need directly.
We didn't do that initially due to security concerns? Now with the private builds we have practically started doing that anyway, so just pulling vendor.tar.bz2 during build from slave feels like the natural way to go.