The original post: /r/datacenter by /u/Direct-Bag7295 on 2024-09-16 03:42:49.

I never thought I’d be posting something like this, but here we are. This is a real story about our experience with a datacenter in Georgia. It is true and unfortunately something we’re experiencing. Any help/guidance would be appreciated.

We signed a 12-month contract for rack space with a colocation located in the basement of an old office building (which should have been our first red flag). A sewage backup in the building caused flooding on the upper floors, and black water ended up leaking into the datacenter, flooding several racks—including ours, which damaged our equipment. Freak accident? Maybe, but who builds a datacenter with bathrooms directly above it?

On the afternoon of the flood, we lost remote connection to our systems. When we reached out to the colo owner, they mentioned they’d be happy to go reboot our servers just in case something had crashed (unlikely, but whatever). A few hours later, they went out, rebooted everything, and told us the status lights looked normal. Meanwhile, we still couldn’t access anything remotely.

Fast forward to nearly midnight when I arrived to the datacenter. When I got there, I met the owner, who seemed pretty casual about the evening. There were fans running and people swapping out a rack—it seemed rather normal. I told him we were there to check on our system, and he reassured me that everything seemed fine after the reboot. But once we got to our rack, there was a trash can on top of our unit collecting water—turns out, it was black water. Our units were covered in mineral residue, and some of the network ports and drives were no longer functioning.

I stepped outside, found the owner, and asked, “What the hell happened?” He calmly explained that the upper floors flooded from the toilets, and that it had entered the datacenter. I told him our equipment was clearly damaged, which seemed to surprise him. So, I took him back to show him. I managed to get one of our primary systems up and running long enough to do a cloud backup, and we left around 1 AM.

Over the next week, we started looking for a new datacenter (preferably closer to Florida) and tried to negotiate compensation from the current provider. We were frustrated that no one told us about the flood while we were troubleshooting, and we had to find out in person after standing in a room with hazardous waste. They informally agreed to terminate our contract and replace our damaged equipment in exchange for the old gear. We thought they were getting off easy, but we just wanted to get back to business.

After we received the compensation agreement, we let them know that their competitor (who was previously planning to move our equipment out) was aware of the incident. We didn’t want any surprises down the road, especially since the agreement had an NDA clause. Their response to this was a retraction of the compensation agreement.

Now, they’ve asked us not to return to the facility, and they still have our equipment which we were supposed to surrender as part of the deal. We can’t even ping our systems anymore, which makes us think they’ve already dismantled everything.

Has anyone else been through something like this? The equipment was less than 10K but the software and data on it is worth far more…