Two jobs:
http://validation.linaro.org/lava-server/scheduler/job/23593/log_file
and
http://validation.linaro.org/lava-server/scheduler/job/23602/log_file

If you grep for root.tgz and look at the time stamps around it, the
first one took a little over 10 minutes. The second one I started maybe
an hour ago or so and it's still deploying root.tgz, so I have no idea
how long it will wind up taking. Both of these are the same image, same
job (except that I added reboots between tests for the second one to see
if that would cause any issues, but we are not yet to that point so it
shouldn't be affecting anything), and both of them ran on snowball02.
So unless I'm missing something, these "timestamp in the future"
messages are not only creating insanely large logfiles for us, but are
also drastically increasing the amount of time it takes to deploy the
root filesystem.

Additionally, I saw some things like this in the log:

tar: .: time stamp 2012-06-27 16:06:34 is 394095692.518984424 s in the
future
tar: Error is not recoverable: exiting now

0K 0%
3.46M=0.03s

Cannot write to `-' (Broken pipe).

root@master:~# [rc=2]: <LAVA_DISPATCHER>2012-06-27 09:27:39 PM WARNING:
Deploy http://192.168.1.10/images/tmp/tmpVCOkDB/root.tgz failed. 4 retry
left.
<LAVA_DISPATCHER>2012-06-27 09:27:42 PM INFO: Wait 60 second before
retry

So... I guess it's also possible the the delays are introducing additional errors and instability.
I submitted a simple fix [1] for this which I think might help. If Michael and Andy are in agreement, I'd like to see it in production quickly to see if it will make a big difference.

Thanks,
Paul Larson

[1] https://code.launchpad.net/~pwlars/lava-dispatcher/supress-timestamp-warnings/+merge/112452