It's time for the monthly servicing of this server room's air conditioning, so a sysadmin pilot fish meets the AC guys when they arrive.
"The vendor had sent a team comprised of people who had never visited our office before," says fish. "I accompanied them to the server room, then headed back to my seat in the adjoining room to answer a phone call."
But fish's heart stops when he hears the AC guys screaming for help.
He rushes back to the server room, and sees in an instant what's happened. They've opened a servicing outlet without making sure the valve was closed -- and water from the high-pressure chilled water line is spraying all over the server room.
"Things got worse when we realized that they were unable to close the valve because of the water pressure," fish says. "With the help of a couple of co-workers, we started filling water into garbage bags."
But just as it looks like the situation is under control, one of the AC guys discovers that there's an electric outlet just below the line that's spraying water. And he discovers it the hard way: by getting himself electrocuted.
Power to the server room has to be cut, and all IT services go down -- a nightmare in the middle of the day at a financial services company.
"After about 30 minutes of struggling, the building maintenance guys managed to close the water line," says fish. "It took nearly five hours to restore all IT services, and by then most of the employees had gone home. Fortunately, all side panels in the server racks were closed at that time, so none of the servers, routers or switches was damaged.
"Now we have installed a new AC unit outside of the server room -- and no more chilled water lines run through the server room."
Chill out with Sharky. Send me your true tale of IT life at firstname.lastname@example.org. You'll snag a snazzy Shark shirt if I use it. Comment on today's tale at Sharky's Google+ community, and read thousands of great old tales in the Sharkives.
Get Sharky's outtakes from the IT Theater of the Absurd delivered directly to your Inbox. Subscribe now to the Daily Shark Newsletter.