I am presently reading Malcolm Gladwell’s latest book Outliers and I find it to be interesting just like the rest of his books. While I was reading the chapter The Ethnic Theory of Plane Crashes, a footnote caught my eye. This footnote had little to do with the rest of the chapter, except that it was about a disaster. It reminded me how easy it is for our IT infrastructure to become a Three Mile Island.
Here is what Gladwell wrote:
The moment that I read this passage, I immediately thought to myself, “This is exactly what happens when IT systems fail!” Rarely is it because a server suddenly goes up in smoke (although that does happen in instances of office fires). Rather, there is usually a series of mishaps, bad decisions, and forgetfulness all mixed together with a liberal dose of Murphy’s Law.
So, what can you do to protect yourself from these small “hiccup”s before they become a Three Mile Island? Here are a few things that I can up with:
- Do not neglect your regular maintenance.
- Never assume that no news is good news.
- As boring and tedious a job as it is, go through your event logs and make sure everything is working fine.
- Assume the worst. If you did not get an error alert via e-mail, do not assume that there was no error, rather, assume your e-mail is down.
- Track small problems so that you can continue to deal with then then they are small problems.
- When big problems do occur, document the small problems that contributed to the big problem.
- When a small problem occurs, perform a worst case scenario analysis to see what could have actually happened.
- Hope for the best, but plan for the worst.
- As much as is possible and within your budget, ensure that redundant systems are in place at every step of the way. RAID5 hard drives in a system that has dual power supplies connected to two UPS systems connected to separate power companies with a complete failover system to another computer if there is a failure in one of the single points of failure (e.g. motherboard).
As a bit of motivation, here are some Murphy’s Laws that pertain to technology, courtesy of Murphy’s Law Site:
- Logic is a systematic method of coming to the wrong conclusion with confidence.
- Whenever a system becomes completely defined, some damn fool discovers something which either abolishes the system or expands it beyond recognition.
- Technology is dominated by those who manage what they do not understand.
- If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization.
- The opulence of the front office decor varies inversely with the fundamental solvency of the firm.
- The attention span of a computer is only as long as it electrical cord.
- An expert is one who knows more and more about less and less until he knows absolutely everything about nothing.
- Tell a man there are 300 billion stars in the universe and he’ll believe you. Tell him a bench has wet paint on it and he’ll have to touch to be sure. great discoveries are made by mistake.
- Always draw your curves, then plot your reading.
- Nothing ever gets built on schedule or within budget.
- All’s well that ends.
- A meeting is an event at which the minutes are kept and the hours are lost.
- The first myth of management is that it exists.
- A failure will not appear till a unit has passed final inspection.
- New systems generate new problems.
- To err is human, but to really foul things up requires a computer.
- We don’t know one millionth of one percent about anything.
- A computer makes as many mistakes in two seconds as 20 men working 20 years make.
- Nothing motivates a man more than to see his boss putting in an honest day’s work.
- Some people manage by the book, even though they don’t know who wrote the book or even what book.
- The primary function of the design engineer is to make things difficult for the fabricator and impossible for the serviceman.
- To spot the expert, pick the one who predicts the job will take the longest and cost the most.
- After all is said and done, a hell of a lot more is said than done.
- Any circuit design must contain at least one part which is obsolete, two parts which are unobtainable and three parts which are still under development.
- A complex system that works is invariably found to have evolved from a simple system that works.
- If mathematically you end up with the incorrect answer, try multiplying by the page number.
- Computers are unreliable, but humans are even more unreliable. Any system which depends on human reliability is unreliable.
- .Give all orders verbally. Never write anything down that might go into a “Pearl Harbor File.”
- Under the most rigorously controlled conditions of pressure, temperature, volume, humidity, and other variables the organism will do as it damn well pleases.
- If you can’t understand it, it is intuitively obvious.
- The more cordial the buyer’s secretary, the greater the odds that the competition already has the order.
- In designing any type of construction, no overall dimension can be totaled correctly after 4:30 p.m. on Friday. The correct total will become self-evident at 8:15 a.m. on Monday.
- Fill what’s empty. Empty what’s full. And scratch where it itches.
- All things are possible except skiing through a revolving door.
- The only perfect science is hind-sight.
- Work smarder and not harder and be careful of yor speling.
- If it’s not in the computer, it doesn’t exist.
- If an experiment works, something has gone wrong.
- When all else fails, read the instructions.
- If there is a possibility of several things going wrong the one that will cause the most damage will be the one to go wrong.
- Everything that goes up must come down.
Corollary: Not always
- Any instrument when dropped will roll into the least accessible corner.
- Any simple theory will be worded in the most complicated way.
- Build a system that even a fool can use and only a fool will want to use it.
- The degree of technical competence is inversely proportional to the level of management.
- A difficult task will be halted near completion by one tiny, previously insignificant detail.
- There is never time to do it right, but always time to do it over.
- The remaining work to finish in order to reach your goal increases as the deadline approaches.
- If there is ever the possibility of several things to go wrong, the one that will cause the most damage will be the one to go wrong.
- If something breaks, and it stops you from doing something, it will be fixed when you:
- no longer need it
- are in the middle of something else
- don’t want it to be fixed, because you really don’t want to do what you were supposed to do
- Each profession talks to itself in it’s own language, apparently there is no Rosetta Stone
- The more urgent the need for a decision to be made, less apparent become the identity of the decision maker
- It is never wise to let a piece of electronic equipment know that you are in a hurry.
- Don’t fix something that ain’t broke, ’cause you’ll break it and you still can’t fix it
- You can never tell which way the train went by looking at the track.
Chong Kwong Sheng addition:
Only by the splatter of the blood stains
The last two laws were sent by Chong Kwong Sheng
- Dobie’s Dogma:
If you are not thoroughly confused, you have not been thoroughly informed.
- A screw will never fit a nut.
- Standard parts are not.
- When working on a motor vehicle engine, any tool dropped will land directly under the center of the engine.
- Interchangeable tapes won’t.
- Never trust modern technology. Trust it only when it is old technology.
- The bolt that is in the most awkward place will always be the one with the tightest thread.
- The most ominous phrase in science: “Uh-oh . . .”
- The 2nd worst thing you can hear the tech say is “Oops!” The worst thing you can hear the tech say is “oh s**t!”
- Any example of hardware/software can be made fool-proof. It cannot, however, be made damn-fool-proof.
- The Rossemblat Graphic Insult Theory:
When any technological change is made, we have a graphic insult curve. No mater how high the insult curve climb, the important thing is how long it goes.
- Bahaman’s Law:
for any given software, the moment you read software reviews and manage to master it, a new version of that software appears.
The new version always manages to change the one feature you need most.
- In today’s fast-moving tech environment, it is a requirement that we forget more than we learn.
- It is simple to make something complex, and complex to make it simple.
- Measurements will be quoted in the least practical unit; velocity, for example, will be measured in ‘furlongs-per-fortnight’.
- In electronics repair the part with the highest failure rate will always be located in the least accessible area of the equipment.
- Multi-million pound technology is worthless in the hands of morons.
- The rule of Protection:
If you install a 50¢ fuse to protect a 100$ component, the 100$ component will blow to protect the 50¢ fuse.
- Karl Imhoff was a German engineer who developed sewage treatment systems in the early 1900’s. His biggest contribution was the Imhoff Tank, which allows sewage to settle. The Imhoff Law relates to bosses everywhere. The law goes as follows:
The largest chunks always rise to the top.
- High tech man-year = 730 people trying to finish a project before lunch.
- An expert will always state the obvious.
- The boss is always right.
Corollary: If the boss is wrong, refer back to the rule.
- On a cruise ship, the one, most important part you don’t have in stock always breaks on a Friday evening, just when you left harbor and the next time you will be in harbor is a Sunday or Christmas eve.
- The chance a copy machine will brake down is proportional to the importance of the material that needs to be copied and inversely proportional to the amount of time till the material will be needed.
- Maintenance department neglect customer’s complains till it starts installations in customer’s new projects.
- Murphy’s Law on HVAC systems:
An HVAC (Heating Ventilating and Air Conditioning) engineering firm, will invariably lease office space in a building with a lousy HVAC system.
All the engineers can do is shiver or sweat and moan about it, and say how they would fix it if the building owner actually gave a damn.
- The probability any machine breaks down increases with the importance of expected visit.
- if it works in theory, it won’t work in practice.
if it works in practice it won’t work in theory.
- Research Law:
No matter how clever and complete your research is, there is always someone who knows more.
- Somers’ Law of Repair:
No part ever fails where you can reach it, or where there is enough light to see how to replace it.
- Any tool dropped will fall where it can cause the most damage.
- Any wire cut to length will be too short.
- Equivalent replacement parts aren’t.
- When you finally update to a new technology, is when everyone stop supporting it.
- Interchangeable parts aren’t
- The proposed size of any project is inversely proportional to the size the project will eventually become.
Corollary: Any project that can consume more resources before reaching it’s final state will do so.
This will happen faster than you think.
Also, the investors will not be happy.
Sent by Jon Proesel
- The less intelligent the idea, and the person stating it, the more likely it will be funded.
- A man with one watch is certain about time. A man with two watches isn’t.
- The more knowledge you gained, the less certain you are of it.
- If you think you understand science (or computers or women), you’re clearly not an expert
- Technicians are the only ones that don’t trust technology
The last four laws were sent by Jan Wenall
- All impossible failures, will happen at the test site.
Corollary: All impossible failures will happen on the clients desktop
- The more you want to contact someone over an instant messenger is inversely proportional to the chances that they will be on-line.
- The more important your email is, the worse your email client will screw it up.
- The degree to which a device will function is directly proportional to the number of times it has been bashed and inversely to its cost.
- A device having an indestructible component or is user serviceable is deemed unsafe until it’s replaced by an expensive, unobtainable, inefficient component which needs constant servicing.
- Assaf’s Laws of Replacement Parts
- A failed 25¢ part cannot be replaced by a new 25¢ part, but by a sub-assembly whose cost is equal to or greater than that of the device in need of the part
- The cost and availability of a replacement part are in inverse proportion to the cost of the whole system: a $1500 device will fail because of the burnout of a 25¢ capacitor. But the 25¢ capacitor is either
- no longer manufactured
- manufactured only by a company in Outer Mongolia with an 18-month backlog
- available only as part of a $1450 sub-assembly
All things mechanical/electrical will catastrophically fail after the guarantee has expired, unless an extended guarantee has been purchased.
Sent by Blair Murray
- The Harvard Principle:
Under the most rigorously controlled conditions of temperature, humidity, pressure, etc., the organism will do as it damn well pleases.
- First Law of Linear Equations:
Given any system n linear equations, there will be n+1 unknowns/li>
- The disappearance of a nagging error in a system is explicable only in terms of insignificant contribution of the source to that system
- The repairman will have never seen a model quite like yours before
- Law of Repairmen:
The repairman fixes your machine to break down the next day and charges for a new machine.
- While technology progresses at the speed of light it’s implementation is filtered through the speed of bureaucracy
- In theory there is no difference between theory and practice, but in practice there is.
- Stationary engineering law:
never underestimate incompetency
Got a horror story about how one problem just added to another? Have your own version of a Murphy’s Law? Let everyone know in the comments!