A big thunderstorm hit Austin this evening and at 9:04PM local time we lost power to several of our most important infrastructure servers in the lab. Most everything seems to have booted back up on its own. However, 2 of the 3 servers providing Ceph storage to the DeveloperCloud failed to boot back up on their own. Every VM in the DeveloperCloud is backed by Ceph so this has caused quite a bit of havoc. Additionally the main network node providing external access to the cloud failed to boot back up properly.
I currently have the Ceph cluster recovering. However, its looking like it could be a couple hours until it decides all its data is in the proper state and can be used for write access.
The network node is still giving me lots of trouble. I'll give an update once I have more information.