I see this pop up with hovering over some kernel ci jobs in build queue:
'All nodes of label "kernel_cloud" are offline'
Does this mean something is broken or will it recover on its own?
It will recover on its own. The scheduled jobs have requested a kernel cloud instance. For various reasons (e.g Instance cap reached), kernel cloud cannot be provisioned and put an instance online.