Daily Cup of Tech

When you have a significant system failure like we did, things have a tendency to get a bit crazy rather quickly. There are several reasons for this, some of which include:

• You are under a lot of pressure to get things done as quickly as possible
• People outside of the IT department may have little or nothing to do
• Many tasks have only one person in the company with the skills to perform the tasks at hand
• Everyone’s individual need, in their opinion, is the most important task that needs to be completed first

Because this is such a stressful time, it is important to keep a clear head and manage the situation as quickly as possible. I have put together a few key action items that you can do when this happens to you so that you can keep things on track.

Read the rest of the story…

Trackback link - http://www.dailycupoftech.com/2007/07/06/server-failure-lesson-9-manage-the-insanity/trackback/

In the past, I have run into all kinds of problems with using auto-configuration on network devices. I have been a stickler when it comes to hard coding speed and duplex settings on pretty much anything that is plugged in to my network simply because I have run into so many problems with different hardware manufacturers implementing the auto-configuration in different ways.

Plus, there are several people out there who misunderstand how speed and duplex matching works (of which I was one). Speed and duplex matching means that the speed (10/100/1000/10000 Mbps) and the duplex (half or full duplex) on both ends of the patch cable. Where many people go awry is in the implementation of auto-configuration.

I (wrongly) believed that if I did not know what the speed and duplex was at the other end of the patch cable, all I needed to do was set my system to auto-configure and it would detect what the speed and duplex on the other end was and then set my system accordingly.

Unfortunately, what actually ended up happening was the system got confused and ended up running at a very slow speed and created a large number of collisions on the network. Nothing but trouble.

So, I figured that I would avoid the whole problem by hard coding everything and being certain.

Well, this whole plan went up in smoke when I went to my new replacement server and went to manually set the network adapters to 1000/full. There was no option for this! I had never needed to do this in the past because our old switches only went to 100/full. What is going on?

The IEEE organization decided that the 1000BASE-T standard would require auto-negotiation for all of its connections. This is why it is not available for the NIC card. So I can now forget about all of those speed/duplex mismatches.

Trackback link - http://www.dailycupoftech.com/2007/07/05/server-failure-lesson-8-gig-nics-need-autoconfig/trackback/

In my mind, redundancy is one of those “Well, duh!” ideas that you just do. How great is it to have two of something when one breaks?

But, the problem with redundancy is that it can be very expensive, especially when you are talking about hardware. Redundant drives, redundant power supplies, redundant network connections…it all adds up.

I have a very interesting conversation with our company’s CEO immediately after we lost our systems. He asked me how much a new server costs. I know most of you right now are rolling your eyes because this can be the equivalent of asking how long is a piece of string. But, to avoid the obvious “it all depends” conversation I said, “About $16,000″. He took one look at me and said, “We’ve lost more than that in productivity today alone! We need to look at having a redundant system in place.”

So, this is what I came up with.

Read the rest of the story…

Trackback link - http://www.dailycupoftech.com/2007/07/04/server-failure-lesson-7-be-redundant-be-redundant/trackback/

Imagine my surprise when I went to add my new Windows 2003 server to the domain as a domain controller only to be told that the version of Active Directory that I was running in the network was the wrong type and that Active Directory needed to be upgraded to support Windows 2003.

Now, those of you who are Active Directory savvy are probably thinking to yourself right about now, “I’ll bet he has a Windows 2000 version of the Active Directory running in his environment and he needs to run ADprep to get it upgraded.”

And you would be correct except for one thing. I added two Windows 2003 servers as domain controllers to the network a couple of years ago and I upgraded Active Directory at that time! There should be no need to upgrade Active Directory again!

And believe it or not, things started to get ever stranger after that!

Read the rest of the story…

Trackback link - http://www.dailycupoftech.com/2007/07/03/server-failure-lesson-6-not-all-ad-schemas-are-the-same/trackback/

When you have a central compute go down, this can be a huge problem because the effects are so far reaching. Our server was not only a domain controller, it was also file (both user and workgroup data), print, DNS, DHCP, RADIUS, along with a few other more esoteric functions.

Each one of these functions caused us a different problem that made the end user’s life pretty miserable for a few days because practically nothing was where they were used to finding it and what they were able to get going was horrifically slow.

There were a lot of obvious problems such as files could not be found on the mapped network drive and Internet access was lost because DNS was unavailable while some users experienced complete network loss because they could not get an IP from DHCP.

But there were a lot of unexpected problems that came up as well simply because the server was so deeply embedded into each workstation’s operating system. These are some of the big “who would have thought” moments that we experienced.
Read the rest of the story…

Trackback link - http://www.dailycupoftech.com/2007/07/02/server-failure-lesson-5-server-tendrils-go-deep/trackback/