FloodedPosted here (week 1722).
This kind of equipment involves a cooling water line direct from the City main. It broke. And dumped a few thousand gallons of water onto running, servers. At least a half million dollars of fragile electronics got doused with dirty (filtered by wood, insulation, ceiling tiles) water; even more was exposed to splash or sauna-like humidity.
My team rocks, several of us worked from lunch yesterday through lunch today drying out hard drives, setting up an emergency colocation (props to Xiolink) and confiscating a conference room for a makeshift server room. It looks like we’re going to be able to limp along for a bit on equipment that has silt in crevices and copper oxide bubbles on logic boards. Hopefully the insurance companies won’t cause delay while they argue over replacement.