Hello,
On Fri, we discovered that build started to take lot of time (30+ mins) on the CopyToSlave stage (one of the first on build startup) to copy EULA-protected binary blobs to a build slave. This adversely affected ability to restrict build/EULA handling changes/RCs (few timed out as was reported by Fathi).
This could be attributed only to http://snapshots.linaro.org/android/binaries/snowball/ receiving five 100MB drops over last 10 days, or 500% or so growth, which uncovered poor throughput of CopyToSlave plugin. Due to inconveniences it brings, I applied a quick dirty solution of mirroring only the latest tarball from http://snapshots.linaro.org/android/binaries/snowball/ . Let me know if you see any issues. And that solution needs to be cleaned/generalized of course.
https://bugs.launchpad.net/linaro-android-infrastructure/+bug/987189
On Mon, Apr 23, 2012 at 11:44 AM, Paul Sokolovsky < paul.sokolovsky@linaro.org> wrote:
Hello,
On Fri, we discovered that build started to take lot of time (30+ mins) on the CopyToSlave stage (one of the first on build startup) to copy EULA-protected binary blobs to a build slave. This adversely affected ability to restrict build/EULA handling changes/RCs (few timed out as was reported by Fathi).
This could be attributed only to http://snapshots.linaro.org/android/binaries/snowball/ receiving five 100MB drops over last 10 days, or 500% or so growth, which uncovered poor throughput of CopyToSlave plugin. Due to inconveniences it brings, I applied a quick dirty solution of mirroring only the latest tarball from http://snapshots.linaro.org/android/binaries/snowball/ . Let me know if you see any issues. And that solution needs to be cleaned/generalized of course.
This is about vendor.tar.bz2?
Does this prevent us from reproducing builds that use older tarballs? Can we make the mirroring smart and parse the info from the build config? e.g. mirror exactly the right tarball?
On 23 April 2012 11:26, Alexander Sack asac@linaro.org wrote:
On Mon, Apr 23, 2012 at 11:44 AM, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:
Hello,
On Fri, we discovered that build started to take lot of time (30+ mins) on the CopyToSlave stage (one of the first on build startup) to copy EULA-protected binary blobs to a build slave. This adversely affected ability to restrict build/EULA handling changes/RCs (few timed out as was reported by Fathi).
This could be attributed only to http://snapshots.linaro.org/android/binaries/snowball/ receiving five 100MB drops over last 10 days, or 500% or so growth, which uncovered poor throughput of CopyToSlave plugin. Due to inconveniences it brings, I applied a quick dirty solution of mirroring only the latest tarball from http://snapshots.linaro.org/android/binaries/snowball/ . Let me know if you see any issues. And that solution needs to be cleaned/generalized of course.
This is about vendor.tar.bz2?
Does this prevent us from reproducing builds that use older tarballs? Can we make the mirroring smart and parse the info from the build config? e.g. mirror exactly the right tarball?
There is all sorts of chatter about slow copies between masters and slaves (normally the discussion is slave -> master, but I am sure the problems are the same). The workaround is to perform the copy not using the plugin, which will be a pain.
The CopyToSlave plugin looks quite simple (https://svn.jenkins-ci.org/trunk/hudson/plugins/copy-to-slave/src/main/java/...) so it isn't obvious what the problem is. Other threads have pointed to setting TCP NODELAY:
https://issues.jenkins-ci.org/browse/JENKINS-3922 - https://github.com/tivv/ssh-slaves-plugin/commit/c2fcc257161f9f01c98883b3469...
https://issues.jenkins-ci.org/browse/JENKINS-7813
Perhaps the remoting.Pipe class needs some attention? I just cloned the Jenkins repository but didn't find the class. Perhaps I am missing something. Anyway, that is what I found.
On Mon, 23 Apr 2012 12:26:19 +0200 Alexander Sack asac@linaro.org wrote:
On Mon, Apr 23, 2012 at 11:44 AM, Paul Sokolovsky < paul.sokolovsky@linaro.org> wrote:
Hello,
On Fri, we discovered that build started to take lot of time (30+ mins) on the CopyToSlave stage (one of the first on build startup) to copy EULA-protected binary blobs to a build slave. This adversely affected ability to restrict build/EULA handling changes/RCs (few timed out as was reported by Fathi).
This could be attributed only to http://snapshots.linaro.org/android/binaries/snowball/ receiving five 100MB drops over last 10 days, or 500% or so growth, which uncovered poor throughput of CopyToSlave plugin. Due to inconveniences it brings, I applied a quick dirty solution of mirroring only the latest tarball from http://snapshots.linaro.org/android/binaries/snowball/ . Let me know if you see any issues. And that solution needs to be cleaned/generalized of course.
This is about vendor.tar.bz2?
Does this prevent us from reproducing builds that use older tarballs?
Potentially, yes, realistically, noone complained yet ;-). Script doesn't delete anything, it just doesn't mirror old stuff, so unless someone deletes old files (I did), they stay there.
Can we make the mirroring smart and parse the info from the build config? e.g. mirror exactly the right tarball?
We can go deeper and deeper with applying adhoc workarounds which shouldn't be there in the first place, or can instead think out and implement access system which would allow our bot scripts to get access to things they need directly.
On Mon, 23 Apr 2012 15:15:07 +0300 Paul Sokolovsky Paul.Sokolovsky@linaro.org wrote:
On Mon, 23 Apr 2012 12:26:19 +0200 Alexander Sack asac@linaro.org wrote:
On Mon, Apr 23, 2012 at 11:44 AM, Paul Sokolovsky < paul.sokolovsky@linaro.org> wrote:
Hello,
On Fri, we discovered that build started to take lot of time (30+ mins) on the CopyToSlave stage (one of the first on build startup) to copy EULA-protected binary blobs to a build slave. This adversely affected ability to restrict build/EULA handling changes/RCs (few timed out as was reported by Fathi).
This could be attributed only to http://snapshots.linaro.org/android/binaries/snowball/ receiving five 100MB drops over last 10 days, or 500% or so growth, which uncovered poor throughput of CopyToSlave plugin. Due to inconveniences it brings, I applied a quick dirty solution of mirroring only the latest tarball from http://snapshots.linaro.org/android/binaries/snowball/ . Let me know if you see any issues. And that solution needs to be cleaned/generalized of course.
This is about vendor.tar.bz2?
Does this prevent us from reproducing builds that use older tarballs?
Potentially, yes, realistically, noone complained yet ;-).
Indeed, "yet": https://android-build.linaro.org/jenkins/job/linaro-android_snowball-ics-gcc...
20120323 is copied manually.
On Mon, Apr 23, 2012 at 2:15 PM, Paul Sokolovsky <paul.sokolovsky@linaro.org
wrote:
On Mon, 23 Apr 2012 12:26:19 +0200 Alexander Sack asac@linaro.org wrote:
On Mon, Apr 23, 2012 at 11:44 AM, Paul Sokolovsky < paul.sokolovsky@linaro.org> wrote:
Hello,
On Fri, we discovered that build started to take lot of time (30+ mins) on the CopyToSlave stage (one of the first on build startup) to copy EULA-protected binary blobs to a build slave. This adversely affected ability to restrict build/EULA handling changes/RCs (few timed out as was reported by Fathi).
This could be attributed only to http://snapshots.linaro.org/android/binaries/snowball/ receiving five 100MB drops over last 10 days, or 500% or so growth, which uncovered poor throughput of CopyToSlave plugin. Due to inconveniences it brings, I applied a quick dirty solution of mirroring only the latest tarball from http://snapshots.linaro.org/android/binaries/snowball/ . Let me know if you see any issues. And that solution needs to be cleaned/generalized of course.
This is about vendor.tar.bz2?
Does this prevent us from reproducing builds that use older tarballs?
Potentially, yes, realistically, noone complained yet ;-). Script doesn't delete anything, it just doesn't mirror old stuff, so unless someone deletes old files (I did), they stay there.
Can we make the mirroring smart and parse the info from the build config? e.g. mirror exactly the right tarball?
We can go deeper and deeper with applying adhoc workarounds which shouldn't be there in the first place, or can instead think out and implement access system which would allow our bot scripts to get access to things they need directly.
We didn't do that initially due to security concerns? Now with the private builds we have practically started doing that anyway, so just pulling vendor.tar.bz2 during build from slave feels like the natural way to go.
On Mon, 23 Apr 2012 15:08:27 +0200 Alexander Sack asac@linaro.org wrote:
On Mon, Apr 23, 2012 at 2:15 PM, Paul Sokolovsky <paul.sokolovsky@linaro.org
wrote:
On Mon, 23 Apr 2012 12:26:19 +0200 Alexander Sack asac@linaro.org wrote:
On Mon, Apr 23, 2012 at 11:44 AM, Paul Sokolovsky < paul.sokolovsky@linaro.org> wrote:
Hello,
On Fri, we discovered that build started to take lot of time (30+ mins) on the CopyToSlave stage (one of the first on build startup) to copy EULA-protected binary blobs to a build slave. This adversely affected ability to restrict build/EULA handling changes/RCs (few timed out as was reported by Fathi).
This could be attributed only to http://snapshots.linaro.org/android/binaries/snowball/ receiving five 100MB drops over last 10 days, or 500% or so growth, which uncovered poor throughput of CopyToSlave plugin. Due to inconveniences it brings, I applied a quick dirty solution of mirroring only the latest tarball from http://snapshots.linaro.org/android/binaries/snowball/ . Let me know if you see any issues. And that solution needs to be cleaned/generalized of course.
This is about vendor.tar.bz2?
Does this prevent us from reproducing builds that use older tarballs?
Potentially, yes, realistically, noone complained yet ;-). Script doesn't delete anything, it just doesn't mirror old stuff, so unless someone deletes old files (I did), they stay there.
Can we make the mirroring smart and parse the info from the build config? e.g. mirror exactly the right tarball?
We can go deeper and deeper with applying adhoc workarounds which shouldn't be there in the first place, or can instead think out and implement access system which would allow our bot scripts to get access to things they need directly.
We didn't do that initially due to security concerns?
Due to license adherence constraints more specifically, like that we don't supply open tools which allow to (silently) circumvent our own license acceptance mechanism. But later it was discussed that if we have a script which takes an obvious option, like --accept-license (and apparently prints visible message that running script with such option constitutes license acceptance), then we should be good enough. So, that's what we apparently should do closest. (And something like that is in use on ci.* already I heard.)
Now with the private builds we have practically started doing that anyway, so just pulling vendor.tar.bz2 during build from slave feels like the natural way to go.
linaro-android@lists.linaro.org
-
Alexander Sack -
James Tunnicliffe -
Paul Sokolovsky