So I'm starting this blog about my work stuff... A way to bitch about some of the hell I work at... Let's see if this works...
This week was hell... I know it's not over but my brain has shutdown for the week already. My week began on Monday... Went to work for a normal day... No biggie just a regular Monday... Mondays are never great, are they?? No this day went bad long after it was supposed to be over.
Sometime around 7 pm I got this email on my phone (I have a PDA phone, really neat but keeps me tethered to work). This email basically stated an error had occurred on my email server. So I pull out my laptop, connect into work, and check it out. Didn't look good so I figured it was best to get into work and see what the problem was... When I got there (about 8 pm) the server was limping along with errors... And low and behold my file server was blinking red with a hardware failure... What the fuck!! So I fuck with my email server for hours I had to repair a bunch of shit on the hardware side... And during that time I checked out my file server, to find out that one of the hard drives in the primary RAID array had completely failed... So the array was still functioning (thank god for RAID-5). I figured it's freakin' 11 pm, it's RAID-5, I can call in the morning... So I finally get done with the email server at 1 am, head home to wake up for 7 am... Finally got to sleep at home around 3 am.
Got back to work at 8 am. I expected a little fall out from the email server being down but I knew I had to get the hard drive ordered too... Unfortunately some users were having password problems and couldn't login to begin work... So of coarse that had to be taken care of first... Not bad just some password resets and reboots and all set. around 10 am got to my desk and sat down to call Dell. Had them on the line gave the Dell's service tag. They asked the problem, I told them, they then asked for the error code from the drive... I click to get the error code and I get nothing, I click again, nothing, click, click, click, click!!! What the fuck... I said Mr. Dell Tech.. can you call me back in 5 minutes... and I ran to the server room...
Bye bye file server!! Black screen... Yep it failed, hardcore crash!! My very first major server failure... Could have lived without ever getting one, but I was up for the challenge!! Well obviously the Dell tech sent me out a drive, we have 4 hour response warranty. So it got there quick and we began to rebuild the array on the new drive. The only problem here is that we had already found media errors on one of the other drives. Not good... Apparently when the system rebuilt the new drive using the other drive that had media errors it wasn't able to rebuild the array properly, good enough to complete the rebuild but not good enough to form a complete file system. So Windows and a lot of my data was fucked!! I had already paid for a support call with Microsoft because there was enough of a file system to almost boot Windows and we thought we might have been able to repair the system. I was wrong about that but they continued to work with me until the server was back to the way it was... So go Microsoft!! So they talked me through cleaning up Active Directory (the user database) and removing the references to the old server...
I took it from there by reinstalling the server with a fresh copy of Windows. By now it's 5 pm, I have to leave work at 5 now because it's part of me and my wife's deal for the separation plus she has a job in the evenings now too and needs me to be home exactly at 6 pm. So I had no choice I had to have the server back but I had to watch my kids so I left to go get them and bring them back to work with me... I brought one of the family teens to help watch the kids while I worked... It was kool I set them up in a cubicle near the server room with laptop and monitor and had them watch some dvd's we brought... Soooooo I got Windows installed and ready for me to begin restoring the data... But now it's 10:30 pm and my kids need to get to bed and so do I (3 hours sleep the night before). So we head home and they just pass out. I didn't last long either I think I finally wound down around 12:30 pm... But I knew I had to get in there early to get it up and going before everyone got in...
5 am the alarm is set. 6 am I'm back in the server room to begin setting the data drives back up... I begin to restore the data drives from tapes, share out all the new folders from the tapes and BAM... back and running by 10 am...
I was actually quite proud.. First failure in 3 1/2 to 4 years with these servers and we only lost it for 24 hours... I think it's a pretty good track record!!! We barely lost anything and what we lost from my understanding we were able to recreate with out too much work... So all in all I thought it was a good time... Very tired... But a good time...
So to top that day off I came home to relax after the second half of that day went really well... Got the kids straightened out and in bed... sit down about 8 pm to play on Fubar... And I end up having an awesome conversation with one of my Fubar friends until my battery is dead and I need to get to bed... It was the koolest way to wind down from a crazy couple of days. So if your reading this THANKS!!
