At 12:30 a.m. on the Fourth of July, system administrator pilot fish gets a page from his company's outsourcer: A remote execution job is down, and the error message says it can't connect to the remote server.
So fish gets online and pings his server, but it's OK. He logs on: OK. Checks daemons: OK. "Nothing seems to be wrong," says fish. "So I restart the job and check it out."
For five minutes, fish watches as remote job runs without incident. Then he logs off and goes back to bed.
But fish's head barely hits the pillow before he gets another page. "Your job died again," outsourcer tech tells him.
Fish climbs out of bed again and repeats his troubleshooting process. Once again, everything looks OK until it mysteriously dies. Must mean it's time to call the database administrator to recycle the database, he figures.
The first database admin on the duty list doesn't answer the page or phone. So fish calls the next guy on the list and tells him the story.
"You're running jobs tonight?" replies the database admin. "It's July 4. Your boss told me no one would be on the system and I could test my script to detect when the server dies.
"I've been bringing the server down, then up again to test my scripts."
"Uh, couldn't you test this on the test server?" grumbles bleary-eyed fish.
"Oh," says database admin. "You mean I can use that server to test?"
Sharky's testing his grilling skills today, which is why this week's story from the Sharkives is a day early. But you can send me your own true tale of IT life any day of the week at firstname.lastname@example.org. You can also comment on today's tale at Sharky's Google+ community, and read thousands of great old tales in the Sharkives.
Get Sharky's outtakes from the IT Theater of the Absurd delivered directly to your Inbox. Subscribe now to the Daily Shark Newsletter.