So, yesterday we discovered that our server room, built out quite nicely by a previous tenant, is located directly under an HVAC chiller. The landlord didn’t seem to know this, either.

This kind of equipment involves a cooling water line direct from the City main. It broke. And dumped a few thousand gallons of water onto running, servers. At least a half million dollars of fragile electronics got doused with dirty (filtered by wood, insulation, ceiling tiles) water; even more was exposed to splash or sauna-like humidity.

My team rocks, several of us worked from lunch yesterday through lunch today drying out hard drives, setting up an emergency colocation (props to Xiolink) and confiscating a conference room for a makeshift server room. It looks like we’re going to be able to limp along for a bit on equipment that has silt in crevices and copper oxide bubbles on logic boards. Hopefully the insurance companies won’t cause delay while they argue over replacement.

This site is an archive of several sources. If you'd like to keep up, follow me on Twitter.