Thursday, November 06, 2003

It's Alive

I finally got the mail server running again. Just a little more over 27 hours after taking it down. What a project that was.

I started Tuesday morning by checking backup logs. And it was good that I did. A backup had not been performed in a few weeks. I reach behind the removable drive that our data gets backed up to, and it had never been turned on (or was turned off at some point), Woops. After running a full backup, I kicked over to the Mail server itself, stopped the Information Store, and ran a copy of the MTBDATA and MDBDATA folders off to the removable drive (which took another hour).

After all that madness I had blown half the day (it was around 3PM). Down the mail server went, and out came the drives. Installing new drives was a piece of cake, and even configuring the RAID wasn't too bad. This old DPT RAID controller had a pretty good BIOS in it which allowed you to configure the RAID without a boot disk. Adding a HotSpare was a little strange. So after a few tries I got it how I wanted it, and began to "prepare the array". If you have never built a RAID-5 array before, I recommend you have something else to do. It takes hours.

Later I was able to start the Windows 2000 installation. Which naturally then wanted to to format the 'drive' as NTFS. It was right around 70 gigs worth of space. So I got to wait another hour or so for that. After some running around, dropping off the wife and kids, and returning to work, Windows 2000 was just beginning it's graphical installation mode. Hooray. Little did I know that I would be reinstalling later after have some horrible failures.

By the time it was rolling around to be midnight, I had learned some valuable new lessons.
  • Starting over may not be the best option (when you have to format a huge array of space)
  • Windows 2000 might suck a little bit, but Exchange 2003 sucks in a major way
  • When you can't open events in your event log, you may be in trouble
  • Ntdsutil.exe is a resource kit tool for repairing active directory
  • You won't need ntdsutil if you name another controller a FSMO (woops)
  • When in doubt, go to bed

I arrived home about 2:00AM after I had started installing Exchange Server 2003 (again) using /distasterrecovery mode. Once home, I kicked off my shoes, flexed my back (it hurts) and then sat back down to remotely connect and continue. Low and behold, I had gotten errors again. Although I couldn't read them. Because event log was hosed. To bed I went!!

After my 3 hour nap, I arose to drive out to Athens Ohio to teach a Linux Administration class. I was strangely awake. Must have been all the coffee I drank. About half way there, the boss calls. "It doesn't work, right?". "Nope!", I tell him. I go on to explain that I did indeed try. I REALLY tried to get it up and running. But quite literally, there are not enough hours in the day. I tell him about all the problems I was having, and how it might be wise to reapply Service Pack 4, along with other Windows Updates. He agrees to take care of that for me so I can go about my day and maybe connect later to have a look.

Work was a breeze. I had 9 students for class and they were all a lot of fun, and very willing to learn. Only problem was connecting back to the office. Nothing seemed to be working. I confirmed my suspicions by asking an employee "you guys aren't filtering outbound connections through your firewall, are you?". "Oh yeah!", he proudly replied. That answered that question. No ssh, no VNC, no nothin'!

Later he was nice enough to open up port 3389 so I was able to sneak into my home network to get some work done under remote desktop. The MAIL server was staring back at me, still a ton of errors. Yet ... I could read them all in the event log! The updates fixed that. Great.

Next I moved over the Exchange database which had been sitting on the root of the drive, and then attempted to mount the store. ERROR! What's the problem? It's looking for the database on the D: drive. There is no longer a D: drive. Hrm. Aside from that, certain elements of the mail database just are not there. I had gotten some errors running Exchange Server 2003 with the /disasterrecovery switch. So I pulled a trick. I went into command prompt and ran "subst d: c:\" which basically says, 'mount a d: drive to the root of c:'. The result is a bogus d: drive letter. That seemed to satisfy 'some' of the errors.

By this time it was 4:30 and I needed to get out of there. Before I leave, I go in and start installing Exchange again. This time, normally. So I went home, ate, played with my daughter a bit. And then sat down, and went onto the mail server. The installation has finished, without any errors. I right click on the Store, and mount it. What? It mounted? Thank God. That was an uneventful end to a VERY long day. I was so happy. 6:20PM. I could enjoy the rest of my evening and actually sleep better tonight. I call the boss who is in law class at the time and leave him a cheery voicemail message.

It was a good night. I enjoyed some private time with my wife, and slept very well. As did she. Apparently her new drug prescription has gotten her away from a drug that left her rest-less and in pain. Thanks doctor!

Now I am sitting outside the building I teach this class in. It's about 7:45 in sunny Athens Ohio. I am very early for class. My students didn't want to start class until 8:30. I am in no hurry to get inside seeing how I can only get out on port 80. Maybe I could connect back home with remote desktop, and map a secure shell to port 80. That would work. ;-)

See you all later. Thats enough blog for now.

No comments:

Post a Comment