Flooded

So, yesterday we discovered that our server room, built out quite nicely by a previous tenant, is located directly under an HVAC chiller. The landlord didn’t seem to know this, either.

This kind of equipment involves a cooling water line direct from the City main. It broke. And dumped a few thousand gallons of water onto running, servers. At least a half million dollars of fragile electronics got doused with dirty (filtered by wood, insulation, ceiling tiles) water; even more was exposed to splash or sauna-like humidity.

My team rocks, several of us worked from lunch yesterday through lunch today drying out hard drives, setting up an emergency colocation (props to Xiolink) and confiscating a conference room for a makeshift server room. It looks like we’re going to be able to limp along for a bit on equipment that has silt in crevices and copper oxide bubbles on logic boards. Hopefully the insurance companies won’t cause delay while they argue over replacement.