Hello,
(This mail could have epigraph "I'm starting to get CBuild" ;-) ).
I'd like to draw some attention to native CBuild builders. There appear to be quite a backlog of builds ("Pending" section at http://cbuild.validation.linaro.org/helpers/scheduler ), which appear to be caused by builders problems.
On Friday night, I noticed that tcpanda05 was hanging for around a week, and rebooted it, based on the information provided by Dave previously. Hope that's ok, as it happily cutting size of a9-daily's backlog since then (and sorry for letting know just now, again, it was off usual working hours). a9-daily's queue does decrease, but rather slowly. At the same time, that queue has tcpanda06 board commented out - it went that way with initial import "scheduler" repo, so I couldn't know exact reason. Generic one is understood - so tcpanda06 could quickly pick up jobs from other queues, but that means it spends idle half a day. So, I temporarily re-enabled it for a9-daily to process backlog, that change is clearly commented and easily bzr diff'able on toolchain64.
That ends list of issues I actively poked at, but there're more I couldn't:
1. tcserver01x5, which belongs to x86_64-heavy queue, and apparently is a VM hosted in lava lab, hangs for 16 days, with a dozen jobs in backlog. "tcserver01x5" didn't ping for me from gateway, but "tcserver01" does, so maybe it's live but has some issues.
2. a9hf-daily has single active machine, tcpanda11, so that's in permanent, and probably growing, backlog.
I understand that leveraging LAVA builds is probably the best way to resolve the issue, and I'm currently validating and tweaking those builds (but I'm not sure if build image as was provided by Michael is hardfloat or not, and how to go about adding missing one).