Two jobs: http://validation.linaro.org/lava-server/scheduler/job/23593/log_file and http://validation.linaro.org/lava-server/scheduler/job/23602/log_file

If you grep for root.tgz and look at the time stamps around it, the first one took a little over 10 minutes. The second one I started maybe an hour ago or so and it's still deploying root.tgz, so I have no idea how long it will wind up taking. Both of these are the same image, same job (except that I added reboots between tests for the second one to see if that would cause any issues, but we are not yet to that point so it shouldn't be affecting anything), and both of them ran on snowball02. So unless I'm missing something, these "timestamp in the future" messages are not only creating insanely large logfiles for us, but are also drastically increasing the amount of time it takes to deploy the root filesystem.

Additionally, I saw some things like this in the log: tar: .: time stamp 2012-06-27 16:06:34 is 394095692.518984424 s in the future tar: Error is not recoverable: exiting now

0K 0% 3.46M=0.03s

Cannot write to `-' (Broken pipe).

root@master:~# [rc=2]: <LAVA_DISPATCHER>2012-06-27 09:27:39 PM WARNING: Deploy http://192.168.1.10/images/tmp/tmpVCOkDB/root.tgz failed. 4 retry left. <LAVA_DISPATCHER>2012-06-27 09:27:42 PM INFO: Wait 60 second before retry

So... I guess it's also possible the the delays are introducing additional errors and instability. I submitted a simple fix [1] for this which I think might help. If Michael and Andy are in agreement, I'd like to see it in production quickly to see if it will make a big difference.

Thanks, Paul Larson

[1] https://code.launchpad.net/~pwlars/lava-dispatcher/supress-timestamp-warning...