Volume
1, Issue 2, February 1997
Copyright (C) 1997 by Steve Litt
Volume
1, Issue 2, February 1997
Welcome to the second issue of Troubleshooting Professional Magazine. This issue explores the role of diagnostic tools of various kinds, from artificial intelligence help-desk software to network diagnostics to computerized service manuals. Why they work and why they fail. And how to maximize the chances of their succeeding in your organization.
I have a love/hate relationship with diagnostic tools. On one hand, nobody had better try to take away my voltmeter, my diagnostic software, or my help files (a form of "smart manual"). A recent multi-day computer upgrade made me realize just how helpless I am without great diagnostic tools. On the other hand, I've seen diagnostic tools fail miserably as a substitute for hiring and properly training the right people.And I've seen blind belief in diagnostic tools turn simple problems into show-stopping disasters. How can something be so good and so bad? I think the answer is how you catagorize diagnostic tools, expert systems and smart manuals. Are they tools, or are they solutions?
In this issue you'll see articles This Ain't the Summer of Love and A Tale of Two Industries, discussing just how necessary good diagnostic tools and smart manuals are today. I Have Just the Solution For You! discusses the differences between the tool perspective and the solution perspective. Two Mistakes, Two Lessons reveals the outcome in two (disguised) case histories where the wrong perspective was used. Finally, Another Mistake, Another Lesson is a case history where a Troubleshooter learned the hard way how necessary diagnostic tools are today.
Thanks for reading the second issue of Troubleshooting Professional. If you're a Troubleshooter, this is your magazine. Enjoy!.
1967. The summer of love. Be-ins, happenings, Hippies in Golden Gate Park. Flowers in your hair. Sergeant Pepper's Lonely Hearts Club Band. And Chrysler Corporation manufactured my car.
Yes, I drive a 1967 Dodge Coronet. It's thirty years old. A classic. 318, automatic, AC, heater, AM radio and the shiny white original paint job seen only on California cars. I can't go anywhere without someone asking about it. But that's not why I like it. I like my car because I understand everything in it. And if I need more info, I read the Chilton Manual I bought for twelve dollars at Pep Boys.
Computer controlled smog, speed sensitive steering, individual climate control, air bags, multi-driver programmable seat adjustments. Hundreds of high-tech features. Don't even go there unless you're factory-trained. And unless you're using the factory diagnostic tools and manuals. For systems as complex as modern cars, the Universal Troubleshooting Process must be augmented with proper diagnostic tools and smart service manuals.
Cars are now equipped with diagnostic trouble code generators. These codes guide the technician in narrowing the scope of the problem. Repair facilities can now obtain (expensive) sophisticated computers to likewise guide the technician. And modern automotive service manuals contain not only diagrams and descriptions, but also diagnostic testing sequences for common symptoms and syndromes. This allows the technician to follow an established path, rather than forging his own. Of course, when needed, he or she can leave that path and form his or her own diagnostic procedure.
The point is this. Modern cars have exceeded the human ability to keep the entire Mental Model in mind, and have exceeded the marketplaces ability to pay for repairs using pure Universal Troubleshooting Process. Today's automotive technician adds diagnostic tools and smart manuals to his or her arsenal to stay competitive. The shade-tree mechanic will soon take his place in history with flower-haired hippies.
Two industries, two responses to a single problem. The industries: automobiles and computers. The problem: complexity.
In this author's opinion, today's average web-connected, Win95 enabled power-user Pentium computer with 10 to 50 applications is as complex as a 1997 luxury car. To get an idea, I exported my Win95 registry file and removed all blank lines. The resulting text file was over 32,000 lines long. Some configuration options were one line long, some spanned several lines and included multiple (some over 30) configuration parameters. The bottom line is this -- it's reasonable to assume that the registry has between 10,000 and 100,000 "adjustments", any one of which can produce strange symptoms or stop the computer if wrong. And of course, these "adjustments" can be wrong in combination. And all of this interacts with the hardware...
As detailed in the previous article, the car companies are providing their highly trained and paid technicians with high quality diagnostic tools and smart manuals, the assumption being that repairs will be done at dealerships. They are pushing Troubleshooting UP toward highly skilled company technicians. The computer industry takes the opposite approach. Their phone support is often manned by lightly trained and paid personnel, and whenever possible they "trickle down" troubleshooting to the end-user.
Each approach has its advantages and disadvantages. When you get your car fixed, you rest assured that an entire car division's resources are in the hands of the technician. But if you're a civilian, no matter how knowledgeable, it's no longer possible to adequately repair your own car -- you don't have the tools. Contrast this to computer repair, where the end user can effect his or her own repairs (and in fact is encouraged to do so), but can feel like he or she is "on his own".
The hardware and software companies' "trickle down troubleshooting" has created a thriving industry for troubleshooters and toolmakers. New diagnostic tools are created daily. Some are shrink-wrapped, some shareware or mail order. Some diagnostic tools look inside hard-to-reach subsystems. An example is the CommInfo tools, which isolate hardware from the operating system. Others act more like intelligent manuals, evaluating the system and telling the technician the purpose and options of the configuration. Still others, like Quarterdeck's WinProbe 95, do some of both.
Two industries, two approaches, and two commonalities. The Commonalities?
1. Troubleshooters in both industries *must* use good Troubleshooting Process. That's always been true. It always will be. And with 1972 cars and 1992 computer systems, it was enough.
2. With today's complexity, Troubleshooters need more. They must be armed with good diagnostic tools to "see inside" hard to reach (physically or mentally) components, and intelligent manuals to give the Troubleshooter a good Mental Model of the system.
Anyone remotely connected with the computer industry hears it every day -- solutions. Packaged solutions, shrink-wrapped solutions, software solutions, solution providers. What is a solution? Websters Unabridged Dictionary's first definition is:
1. the act of solving a problem, question, etc; (emphasis is mine)
Now I'd like to quote the first few sentences of the fourth chapter of the book "Breakthrough Thinking" by Gerald Nadler and Shozo Hibino, ISBN 1-55958-004-6:
SOLUTIONS CAN'T BE MASS PRODUCED! They can't be bought and sold. A piece of diagnostic software, or test equipment, or automated service manual is not a solution. It's a tool used to find the solution. It's vital Management understand this distinction, yet often they don't. Why not?
Imagine yourself a director of a large corporation. Stockholders buy quick and sell quick, so they want quick results. Gotta get next quarters numbers up! Two salesmen show up at your door. One offers you the *tools* to let your people start to solve a problem. The other offers you a *solution* to the problem. Which sounds more tempting?
A "solution" sounds like a done deal. You can go on to other things. It can replace under-trained workers, or bad process, or whatever ails your organization. It's a magic bullet. Contrast that with the notion of a tool. You and your employees must take responsibility for the use of the tool. Employees must be trained in the use of the tool, how it fits in with their other tools, the system to be fixed, and the Troubleshooting Process. It just doesn't sound like the kind of thing that can boost next quarters numbers.
Too often decision makers go for the "solution". And boy are they sorry. The article following this one, titled "Two Mistakes, Two Lessons" examines the consequences of treating these tools as solutions.
As Columbo would say, just one more thing: The quote from Nadler and Hibino says people blow their chance at an optimal solution out of a desire not to "reinvent the wheel". Does that mean we should reinvent the wheel every time? Hardly. We can use the same set of TOOLS (diagnostic tools, smart manuals, expert systems and Universal Troubleshooting Process), over and over again, to craft individual SOLUTIONS to individual problems. Only the unique parts of each solution are "re-invented".
These two stories are based on actual events, but all names and a few peripheral facts have been changed to avoid recognition.
Help desk and service crew backlog at O'Connor Industries averaged three days, and users were fuming. Repairs were done wrong, with frequent callbacks. I met with the Director of Technical Services, Peter Pinder. Peter was well dressed, well spoken, and intelligent.
I explained that by giving his technicians training in the Troubleshooting Process, he'd enable those technicians to cut the backlog from three days to three hours and eliminate 90% of the callbacks. Many problems would be solved on the initial phone call. Peter listened politely.
When I was done, Peter Pinder leaned back in his chair, steepled his fingers, and pontificated. He'd already solved the problem. His solution -- Help Desk Ninja from Beyond Help Software. Help Desk Ninja was an "artificial intelligence expert system" help desk software which would "do most of the diagnosis for the techs". His technicians would become clerk/data entry people, which, he implied, was all they were capable of anyway. His techs didn't need Troubleshooting Process Training -- the software would do it all.
Peter was fired two years later. Any gains from Help Desk Ninja's "artificial intelligence expert system" were more than offset by its quirkiness. The techs still worked the same way, so the backlog remained the same. Help Desk Ninja was scrapped by Peter's successor.
It was Monday morning at 9:30pm when I got an urgent call from Ned Werkadam, network administrator for Bantlan's Billing and Collections. Ned's a good friend of mine, and he knows what I can do with the Universal Troubleshooting Process. His multi-server LAN was down, bills weren't going out, and he couldn't find the problem. Could I come out right away and take a look at it?
Ned knows me. He knows I'm a great Troubleshooter. And he know that I'm no network expert. I consider it an honor that he asked me in on the case. I drove right over.
When I arrived onsite Ned briefed me. Producing a network diagram, he showed the location and results of each test he'd done so far. Looking at his work, it was apparent he'd already isolated it to a single M454 box. But he didn't realize it. I asked him to find a spare M454 to swap, verify that it fixed the problem, then troubleshoot the broken box to the board level while the network continued on its way.
Ned wasn't enthusiastic. Swapping the box would take 4 hours. After I showed him, using Troubleshooting Divide and Conquer logic, how his own work clearly indicated the fault was in the M454 box, he agreed. Then Charles Cheney walked in.
Charles Cheney was a CNE with Network Ninjas and Gurus, Inc., whom Ned had called right after calling me. Charles was knowledgeable and brash -- a take-charge kind of guy. And he took charge.
"No, don't swap out that box, the problem's not in that box. My diagnostic software says it's not in that box". Turning to me, Charles said "look guy, no offense, but I'm the network expert here".
Ned turned from Charles, to me, and back again. After a minute's deliberation, Ned said hesitantly "OK, we won't swap out that box just now. Thanks Steve". There was no more I could do for Ned, so I left.
The next month, I heard it through the grapevine that Ned's LAN was down an entire week. Curiosity got the better of me, so I dropped in on him.
I closed his office door and spoke first. "I hear it took
a week to solve that problem. Are you OK?"
"I had some trouble, but I'm still employed", was the reply.
I couldn't resist. "So what was the problem"?
Ned strained to keep a straight face. "It was a circuit board".
I went in for the kill. "And which box was that circuit board in".
Ned's sheepish grin spread from ear to ear. "You know damn well which
box it was in."
We laughed, and the conversation turned to other things.
October 19, 1996, I bragged to my friends that I would "spend the whole weekend troubleshooting my new 166 Pentium", then swaggered into the computer swap meet, bought a motherboard, memory, and everything else, and went home to put it together. I won't terrorize you with the details, but it took me more than a week, six trips to various vendors, and about 50 Win95 re-installs to get it working the way it should. I wasn't bragging or swaggering anymore. My confidence in myself and in the Universal Troubleshooting Process was shaken.
After the repair was complete, during the Take Pride troubleshooting step, I thought about what went wrong. Why did the method I had successfully used for fifteen years suddenly come close to failure?
And I came to this conclusion: 1992 is gone. George Bush isn't president, and computers aren't simple enough to go it alone with Universal Troubleshooting Process. You need diagnostic tools to see inside IRQs, ports and devices, and a smart manual to help you with the mother of all adjustment arrays, The Registry.
The next day I purchased Quarterdeck's Winprobe for the registry, and CTS's PortInfo and IRQinfo to see ports and IRQs. I went on the lookout for other diagnostics. I began planning a Diagnostic Tools Section for Troubleshooters.Com (you can see it from the home page now).
Three days later I had an "external modem won't work" symptom on another computer. 30 seconds with CTS's PortInfo guided me in the direction of a bad jumper setting...
Here's where letters to the editor will go. This is the second issue, and there have been no letters to the editor :-(. Please write.
All letters become the property of the publisher (Steve Litt), and may be edited for clarity or brevity. We reserve the right to not publish letters we deem in bad taste (bad language, obscenity, hate, lewd, violence, etc.).
These first few issues, we'd greatly appreciate feedback type letters. Especially valuable are balanced letters, mixing suggested alternatives with criticism, or mixing areas of improvement with praise. Flames for flames sake or sycophantic praise aren't as valuable.
Submit letters to the editor to Steve Litt's email address, and be sure the subject reads "Letter to the Editor". We regret that we cannot return your letter, so please make a copy of it for future reference.
We anticipate five to eight articles per issue, with issues coming out monthly. The next few issues we'll be looking for articles on how to bring the Troubleshooting Process to all areas of business and society. This can be done as an essay (like the articles above), with humor, or with a case study. A Troubleshooting poem would be nice. We are *not* looking for ultra-technical or system-specific articles in the next few issues. Submissions should be between 250 and 2000 words long.
All submissions become the property of the publisher (Steve Litt). We do not currently pay for articles. Troubleshooters.Com reserves the right to edit any submission for clarity or brevity. Any published article will include a two sentence description of the author, a hypertext link to his or her email, and a phone number if desired. Upon request, we will include a hypertext link, at the end of the magazine issue, to the author's website, providing that website meets the Troubleshooters.Com criteria for links and that the author's website first links to Troubleshooters.Com.
Submissions should be emailed to Steve Litt's email address, with subject line Article Submission. The first paragraph of your message should read as follows:
After that paragraph, write the title, text of the article, and a two sentence description of the author.
http://www.troubleshooters.com is author
Steve Litt's website.
http://comminfo.com: Computer Telecommunication
Systems, Inc makes the CommInfo Tools.
http://www.quarterdeck.com: Quarterdeck
makes WinProbe 95.