Who me ? It’s Monday, and this week’s column has another callback to verify that those saves really worked in an unfortunately synced episode of Who, Me?
Our story comes from a reader we’ll call “John” (because that’s not his name) and is set in the glory days of Windows NT 4 and Microsoft Exchange Server 5.5.
John had earned his stripes in the technical support department of a mail-order parts company (remember those golden days before e-commerce?) and was promoted to the dizzying level of “the IT team” .
“Which included me and another guy,” he noted.
However, this meant that John had to do all sorts of exciting work, such as processing data for catalog shipments in SQL Server. When this co-worker (who was looking after the domain and dealing with Exchange’s weaknesses) left, John was given the keys to the server room. He was even able to take courses for an MCSE certification, thanks to a boss who considered himself a bit of a technical prodigy.
Time passed and nothing went wrong. The lights stayed on. Computers were buzzing. John was clearly doing a magnificent job.
Until, as he said, “the first domino fell”.
“The Exchange server,” John said, “which also served as a fax gateway (because email wasn’t really a thing yet and almost everyone received their account statement by fax – that weekly fax job was literally critical to the company’s cash flow), started blue screening.”
A look at the code on the blue screen of death (BSOD) hinted that the motherboard might be at fault.
“The boss asks me if I’m ready for the job of rebuilding Exchange,” John recalls. “I tactfully suggest we get some outside help – ‘I know a guy who’s great with Exchange!'”
OK! The plan was to do the work on Friday evening. Everything is quite simple: back up the Exchange server to tape. Save it again, just in case. Then crack on the hardware.
Everything was fine. The new motherboard came in. “We pulled out the 8-way fax board marveling at its grotesqueness and stuck it onto the new motherboard.”
At midnight, Windows was back. The exchange was back. Everything was service packed. Just the tape restore of the data…
…which worked well. some obscure
ESEUTIL commands were required and… Exchange would not start. Odd. The other backup was tried. Again, Exchange won’t start and bleats about a corrupt database.
It was now 2am, and John decided to call it quits. After all, he was staying all weekend to get things done.
It’s not hard to guess what happened next. It was found that
ESEUTIL had enough options that the potential for error was vast. “Nothing worked,” said John, “no one had a mailbox anymore.”
It was bad. But not a disaster. The backups ran every day (one full on Sunday, then daily incremental backups), so restore them, right? They have been restored…fingers have been crossed…The exchange has been restarted…
No chance. Repairs attempted using
ESEUTIL took hours, but nothing worked.
At this point, it was 4 a.m. on a Sunday morning, and the tools needed to be put down for a few hours in favor of sleep.
“Tomorrow is coming,” John said, “and this time we’re trying to bring the machine back with the old motherboard because we’ve run out of ideas. Broken screwdriver. Four hours later, same result. Back to new board mother because we found others
ESEUTIL settings. Alas. It turns out that all 36 backup tapes were corrupt.”
A weekend of sorting things had turned into just seven hours before the staff arrived at work.
As it was midnight on Sunday, despair set in.
“What’s better than no mail service at all? Empty mailboxes!”
And so there was one last push. Exchange was reinstalled and configured from scratch. The directory has been restored. Sure, everyone would come to work to find their mailbox empty, but at least it would be possible to send and receive messages from the outside world.
At 6 a.m., the duo stood outside the building, ready to deliver the bad news to staff when they arrived.
“One by one the message comes ‘your email is working but you no longer have any of your old emails’.
Responses were mixed: “The best of them was the warehouse manager: ‘No worries mate, that was pointless shit anyway.'”
Eventually, John’s boss (the one who considered himself a technical prodigy) arrived. Half an hour of questions and explanations ensued before he accepted John’s apology: “Well, OK, you’ve done everything you can. That’s OK. I have a backup of all my contacts on this subject…”
He held up his fancy new Palm PDA, the one that had given up the cable for a wireless link.
“He smugly walked into the office and prepared to retrieve his entire mailbox,” John recalled.
“Except the sync settings in the Palm software said ‘take most recent’.”
“What’s newer than Friday’s full mailbox? Today’s is empty”
And with that, the duo retreated in haste. Just like the case of “I’m going to get my coat…”
Have you ever received a sync settings slap? Or seen a Friday night just job explode into a weekend disaster? Of course you do, and you should share your story with an email to Who, Me? ®