Troubleshooters.Com Presents

Troubleshooting Professional Magazine

Volume 5 Issue 12, December 2001
Distinctions and Commonalities
Copyright (C) 2001 by Steve Litt. All rights reserved. Materials from guest authors copyrighted by them and licensed for perpetual use to Troubleshooting Professional Magazine. All rights reserved to the copyright holder, except for items specifically marked otherwise (certain free software source code, GNU/GPL, etc.). All material herein provided "As-Is". User assumes all risk and responsibility for any outcome.

[ Troubleshooters.Com | Back Issues ]

Life is "trying things to see if they work" -- Ray Bradbury


Editor's Desk

By Steve Litt
The idea for this month's theme surfaced as an idea for a great "Life After Windows" column on learning through reverse engineering. The example was going to be my identification of a LaTeX command via exploiting the differences between LyX files whose only difference was the Table of Contents vertical spacing.

That led to the thought that distinctions are a vital tactic of  reverse engineering. About a minute later it occurred to me that distinctions and commonalities were mentioned voluminously throughout my new troubleshooting book, which then lead to the realization of the huge role consciousness of distinctions and commonalities play in our lives.

So in the end, the article on reverse engineering was promoted to the Linux Log column, with the rest of the magazine devoted to distinctions and commonalities.

This issue of Troubleshooting Professional Magazine is devoted to the principles and use of distinctions and commonalities in Troubleshooting and other activities. So kick back and relax as you read this issue. And remember, if you're a Troubleshooter, this is your magazine. Enjoy!

Steve Litt is the main author of Samba Unleashed. He can be reached at Steve Litt's email address.

Vote for Your Favorite Troubleshooting Professional

By Steve Litt
Now is the time to vote for your favorite Troubleshooting Professional issue of 2001 (by month) and/or your favorite article of 2001 (by month and title).  You can also vote for your all time favorite issue (by year and month) and/or all time favorite article (by year, month and title).

Submit your entry to Steve Litt's email address. The results will be announced in our fifth anniversary issue, which comes out in January.

Steve Litt has been editor of Troubleshooting Professional Magazine for five years. He can be reached at Steve Litt's email address.


By Steve Litt
My introduction to the concept of distinctions occurred in Anthony Robbins' 1991 best seller, "Awaken the Giant Within". The concept went right over my head. Robbins spoke of "key distinctions" and "unique distinctions". He used phrases like "I can't overemphasize the power and value of gaining even one, single distinction...". Having never encountered the word "distinction" in a book or conversation, I called Robbins' company to ask the meaning of the word. The dictionary had been no help, having basically said "that which indicates individuality".

The people at Robbins' company discussed it amongst themselves and with me for a couple minutes, and we all agreed that basically "distinction" means "difference". So true and yet so oversimplified. Anthony Robbins was years ahead of his time, and it would be almost 10 years before I understood not only the word, but exactly why he used that word, and the fact that this one word provided an insight into the workings of Anthony Robbins' mind. But from the time of my 1991 reading of Robbins' book until my writing of the December 2000 Troubleshooting Professional Magazine, I thought of "difference" and "distinction" as synonyms, and wondered why Robbins had made such a big deal of distinctions.

While writing the December 2000 TPM I finally came to understand the difference (actually the distinction) between "difference" and "distinction". You won't find it in any dictionary, but it's there, and the savvy Troubleshooter or problem solver better understand it. It's an issue of connotation, not definition. Use of the word "difference" usually focuses attention on the two entities being compared. Use of the word "distinction" focuses on the difference itself. In other words:
  4 > 3
  4 > 3

"Distinction" is more of an entity in and of itself, which is what we want for Troubleshooting and problem solving. In those types of mental activity, the key to mastery is understanding the distinction, not just the two items being contrasted.

Every recognized problem carries an immediate distinction -- the discrepancy between what is and what should be. From there it's usually possible to identify some other entity, be it a company, person, server, or other entity, that does not exhibit the unwanted behavior. The question then becomes, what are the other distinctions between the functional and dysfunctional entities, and of those many distinctions, which are actually causes of the behavioral distinction. Those causational distinctions are what Anthony Robbins would call "key distinctions". The undesired behavior is what the Universal Troubleshooting Process would call "the symptom".

The tactics of distinctions might be something like this:

  1. Seek distinctions
  2. Recognize the distinction
  3. Evaluate the distinction
  4. Use the distinction
Easier said than done, but let me take a shot at it:

Seek distinctions

Actively be on the lookout for any and all distinctions. Make it a policy to look for distinctions in anything that happens. On occasions when you're unusually proficient, look for and remember distinctions. When you're unusually bungling, don't beat yourself up -- look for and remember distinctions. When you see someone succeeding where you fail, seek distinctions. When you see someone fail where you succeed, once again seek distinctions. Failure is often a pointer to success strategies, even for the unconsciously competant.

When you see clustered groups of similarities (commonalities), look for a common trait that's distinct from those not in the group.

Understand the power of distinctions, and whenever you observe anything, take a moment to rearrange the observation into distinctions.

Recognize the distinction

Much easier said than done. One way to recognize the distinction is by observation. Observation is not a simple matter -- my new Troubleshooting book devotes an entire chapter to the subject of high performance observation. Another route to distinction recognition is the writing of others. Certainly, by reading this issue of TPM, you will now and forever understand the distinction between "difference" and "distinction".

We technological Troubleshooters use a sort of scientific method to discover distinctions. Starting with a hypothesis, we use a measurement, strongarm, or combination of the two, in order to attempt to toggle the symptom. If we succeed, we establish a causational distinction between the system's functional and dysfunctional state. If we fail to toggle the symptom, we acquire the distinction that the section we tested is not causational. Why is that a distinction? Because it's a difference between that section and the section that DOES contain the root cause.

The methods in the preceding paragraph require plenty of observation and creative thinking. Most of all, they require the conscious understanding of just how important distinctions are.

Evaluate the distinction

Anthony Robbins recommends finding a "model" -- a person who has achieved what you want to achieve, and then copying him in order to achieve what he achieved. But obviously you cannot copy everything about the person. If he's a billionaire and lives in a $20 million house, you cannot copy that. If he's a native of Hungary, and you're a native of the US, you can't change your native country. So the task at hand is to identify which distinctions are causative of the distinction you'd like to eliminate (the results you'd like to emulate), and then model those key distinctions.

In the case of technological Troubleshooting, you find a funny entry in the configuration file, so you try by various means to determine whether this funny entry (a distinction) is causative of the symptom. Note that in this case distinction evaluation tends to merge with distinction use.

Use the distinction

Once you've found a distinction you know to be causative, the next step is to use that distinction by repairing it or strongarming it as to be identical with that component of the functional system. In Anthony Robbins' philosophy that would mean "modelling" the person with the distinct behavior, or forcing your mind into the distinct thought pattern.

In the case of a Troubleshooter, it would mean replacing the part causing the distinction.

Distinctions Rule Technological Troubleshooting

The ONLY way to narrow the root cause scope is to find distinctions. What is different about the working system than a nonworking system? What is different about the subsystems that can toggle the symptom, compared to those that cannot. In other words, which subsystems can toggle the symptom, or not?.

The more distinctions you recognize, the quicker your Troubleshooting work will go.

Distinctions in Step 3

Step 3 of the Universal Troubleshooting Process is "Get a complete and accurate symptom description". This is by far the greatest opportunity to uncover distinctions. When did it start happening, and when did it not? Where did it happen, and where did it not? Who did it happen to, and to whom did it not? What work was done around the time of first observance, and what was not? How do you make it happen, and how do you not?

You'll notice that often the "is not" sounds unnecessary because the "is not" represents the universe of possible things, but nevertheless it forms a distinction. However, in many technological troubleshooting situations it's customary not to bother with the "is not" set because it's so universal.

The better your acquired symptom description, the faster your Troubleshooting, because the more inclusive your symptom description, the more distinctions you start with.

Distinctions in Step 4

Step 4 is "Reproduce the Symptom". Here's where you test the user's symptom reproduction sequence, and find any differences between what the customer or user said and what actually reproduces the symptom. Sometimes those differences can give insight (read that as distinctions) about the environment in which the user was operating.

Distinctions in Step 6

Step 6 is "Narrow it Down". Step 6 is a continuous search for distinctions that serve to narrow the root cause scope. Because you're seeking one component out of thousands or tens or even hundreds of thousands, the operant distinction is "where could the root cause be, and where is it definitely not". The tactic is to evaluate the distinction of whether a component can toggle the distinction, or not. This implements the process of elimination, approaching binary search to the extent possible.

Distinctions in Human Knowledge

A surprising amount of human knowledge is stored as, or at least can be organized as, distinctions. Which neighborhoods are safe to walk through, and which aren't? Which stores are economical, and which are not? What patterns of thought induce happiness, and which do not? Which Troubleshooting tactics yield quick solutions, and which do not? Which authors impart useful information, and which do not? What is useful, and what is not?


Distinctions are a powerful discovery tool, useful in Troubleshooting and in other facets of life. A process to take advantage of distinctions might look something like this:
  1. Seek distinctions
  2. Recognize the distinction
  3. Evaluate the distinction
  4. Use the distinction
In the Universal Troubleshooting Process, distinctions are especially important in steps 3, 4 and 6.

In solving life's problems and taking advantage of life's opportunity, one extremely important distinction is "what's important, and what's not".

Steve Litt's newest book is a complete 300+ page treatise titled "Troubleshooting Techniques of the Successful Technologist".  Steve can be reached at Steve Litt's email address.

Breakthrough Distinctions

By Steve Litt
Beyond all of life's mundane distinctions lie those distinctions that point the way to breakthroughs. I call these "breakthrough distinctions". I believe they're what  Anthony Robbins meant by "key distinctions".

A breakthrough distinction enables you to make a fundimental and positive change in your performance. It's a single fact which, when you act upon it, completely changes your performance or your life.

Before discussing this further, let me give some personal examples...

Personal Examples of Breakthrough Distinctions

If I had any breakthrough distinctions during childhood I don't remember them. That's just not the way I was trained to think. That being said, I was a firm believer in outlines since middle school (junior high school, grade 6-8, etc). Although not conscious of distinctions, or even ways of thinking or performing, I instinctively knew that outlines were useful design and task tools, whereas linear narratives and "winging it" were not useful design and task tools.

The first breakthrough distinction I remember was discovering that the best way of troubleshooting audio equipment was by iteratively dividing the remaining root cause scope in half -- binary search. Not sequential search, and not directly seeking the root cause, but ruling out huge sections. That single distinction immediately catapulted me from the company's bottom 10% to their top 20%, and as time went on opened an ever increasing variety of doors to opportunity.

About a year later, while fixing a television for which I had neither service manual nor training, I discovered that no matter what the equipment under repair, my same binary search technique was the optimal troubleshooting methodology. There wasn't a different methodology for each type of system, but instead was one methodology for all systems. After switching from audio repair to computer programming 4 years later, this distinction enabled me to instantly become the company's best troubleshooter and debugger.

During a recession-induced bout of unemployment a few years later, I pondered the distinctions between my unemployed self, and halfwit aquaintances who were employed and in fact paid more than I ever had been. One distinction was obvious -- the halfwits were glibly adept at describing their activities using the latest buzzwords, and I was not. After 3 weeks of practicing buzzwords, I used that distinction to land my highest paying contract to date. As years passed that distinction resulted in a prosperous career, and eventually my book "A Job Seeker's Guide: Untold Truths of the Job Market". When united with another distinction, it would produce my book "Rapid Learning: Secret Weapon of the Successful Technologist".

Not all my breakthrough distinctions were home grown. Some came from others. During the early 1990's I was a vocal participant on the fringe of the Quality Movement. Through my friends who were Quality professionals I learned that a great way to facilitate human performance is through following a process -- not using tools or learning tons of facts. I changed my Troubleshooting course from a tools-centered to a process-centered curriculum, after which it was much more readily assimilated by attendees, and much more sought after. The value of process just might be the most important distinction I've learned to date.

In writing the December 1997 issue of Troubleshooting Professional I pondered the fact that besides being able to "buzzword" my way into jobs and contracts, I was able to learn the unknown technology in days, so that my performance exceeded even my overblown glib buzzwording descriptions of my skills. How did I do that? The distinction was that the learning was terminology-first, not technology-first. When writing "Rapid Learning: Secret Weapon of the Successful Technologist" in 1999, the terminology-first learning was incorporated into the Rapid Learning Process (link to flow chart shown in URL's section).

From the late 1980's I had believed that the Universal Troubleshooting Process could be used for all problems -- not just technological ones. But my success with problems in fuzzily defined systems was spotty. What accounted for the difference? The distinction became clear while writing the December 2000 Troubleshooting Professional. In well defined systems the solved state degenerates into "the as designed state and behavior", whereas in fuzzily defined systems the solved state must be fully analyzed. A second distinction appeared while beginning my new Troubleshooting book. That second distinction is that the Universal Troubleshooting Process utilizes and depends on abundant, cheap, easy, fast and safe diagnostic tests. It succeeds when such tests are abundant, and fails when such tests are scarce.

As you can see, over the last 20 years I've exploited breakthrough distinctions to develop a high performance troubleshooting process, a high performance job-finding method, a high performance learning process, and to determine where to and where not to use the Universal Troubleshooting Process. Now that I'm conscious of distinction exploitation and experienced in its use, I expect much more knowledge to come my way. But it took me 20 long, hard, trial and error filled years. There's got to be a better way.

Using Distinctions as a Tool for Personal Improvement

If somebody had shown me this issue of Troubleshooting Professional in the mid 1970's, the 20 years would have been cut to 5, after which I'd have developed many more life-changing processes and methodologies. My 20 years were wasted, but you can benefit. If there's one breakthrough distinction I can give you, it's that the most efficient way to improve your life is by recognizing, evaluating and exploiting breakthrough distinctions.

Be VERY conscious of distinctions. Be on the lookout for them. When you see one, give it lots of deep thought. Ask yourself how the distinction fits in with your other knowledge. Ask yourself how you can use it.

If something goes exceptionally well, ask yourself what was different about this time from the other times. Look at your behavior, at circumstances, at the behavior of others, at technology. Same thing if something goes exceptionally badly. Both triumphs and flameouts should yield breakthrough distinctions.

When someone else succeeds where you fail, don't skulk -- look for distinctions. Remember my distinction between my unemployed self and my fast talking employed acquaintences? I had plenty of reasons to skulk, but instead I exploited the differences into a hypothetical distinction, proved the distinction a couple weeks later, and used it ever since.

Exploit the differences. If you're not succeeding at something, find people who are. What are the differences? Some are relevant and some aren't. If you have enough successful people to model, the factors that matter will be common to most, while the irrelevant factors will cancel out.

Once you've found a distinction, test it. Find a non-destructive test. For instance, if it's a new way to do job interviews, do it at an interview that doesn't matter. If it doesn't cause damage, move up to interviews that matter a little more, until you've proved the distinction and start living it.

Remember, for a better life, think distinctions.

Steve Litt is the author of Rapid Learning: Secret Weapon of the Successful Technologist. He can be reached at Steve Litt's email address.


By Steve Litt
In 1981 my shop was inundated with blown BJC 221 receivers (name changed to protect the innocent). Some had blown fuses, some blown output transistors, some blown bridge rectifiers, and some with blown power transformers. Worse yet, I'd repair them and the same units would blow again. For a while I used higher rated power transistors, but the same units would come back again and again.

Finally recognizing this as a serious problem, I lined up four blown BJC 221's on my bench, and looked from one to the next. A few minute's observation revealed that every one of them had at least one power transistor that had physically pulled away from the heat sink. From there it was easy to observe that every one had pulled away due to the melting of the plastic mounting washer.

The failure mode was now obvious. Normal heating caused the inferior mounting washer to deform a little, allowing the transistor to pull slightly away from the heat sink, reducing cooling for the transistor. The now hotter transistor further melted the washer, allowing the transistor to further pull away. This vicious circle continued until the transistor got so hot as to short circuit, at which time it drew catastrophic current. Some were lucky enough to only blow a fuse. Some blew the bridge rectifier before the fuse did its job. On a few unlucky units, the short blew the power transformer. What was clear was that the root cause was the inferior plastic mounting washers.

I ordered some high temperature mounting washers. For every BJC 221 in the shop, I replaced every mounting washer with a high temperature washer, and then recorded the serial number of the unit. They never came back. What makes this interesting is that when I phoned in the fix to the BJC company, BJC said their engineers in Japan had determined the solution to be higher power transistors, which I had already proven didn't fix the problem. Unbelievably, the washers cost about 4 cents per receiver, whereas the higher quality transistors cost about $4.00 per receiver.

The preceding is a true story of a textbook case of finding and exploiting commonalities. The lone solder jockey employing the commonality technique found the solution that an army of engineers missed. It bears repeating -- the power is in the process.

How do commonalities gain their diagnostic power?

Interestingly enough, commonalities by themselves are powerless. If every receiver in the world had power transistors separated from the heat sink, noticing the separation would have been useless.

What the commonality did was point out a distinction.  The distinction was that blown BJC 221's have pulled away transistors, whereas other receivers do not. Armed with that distinction, it was childs play to drill down to melted hardware, and then solve the problem.

When seeking distinctions, always look for commonalities. Even when not seeking distinctions, be on the lookout for commonalities.

Here are other commonalities I've found:

On a final note, be sure to look at the January 2001 Troubleshooting Professional Magazine, themed "Exploiting Perceived Similarities". It's definitely about commonalities, but back in those days I didn't understand that commonalities point to distinctions. Live and learn :-)
Steve Litt's newest book is a complete 300+ page treatise titled "Troubleshooting Techniques of the Successful Technologist".  Steve can be reached at Steve Litt's email address.

Wasted Youth: Don't Make the Same Mistakes I Made

By Steve Litt
I learned absolutely nothing of life in my teens and twenties. I learned to ride a bicycle. I learned to roller skate very fast. I learned some things I'm not going to talk about and wouldn't recommend. But until the age of 29 I was unable to solve any problem, be it mechanical, technological, or personal. As I blundered through life, sometimes it was fun, sometimes it was ugly, but it was never focused.

All because nobody told me what you're reading in this issue of Troubleshooting Professional Magazine. If somebody had told me how to use distinctions and commonalities when I was 14, it would have trimmed 15 years of blundering off my life. Within 5 years of being told about distinctions and commonalities, I would have invented processes to accomplish what I needed. With no more effort than I spent on the dead-end jobs of my 20's, I could have been wealthy.

I'm not complaining, mind you. The process of discovery has been thrilling, and I wouldn't trade it for all the money in the world. But how much more could I have discovered if I didn't first have to discover how to discover?

Don't make the mistakes I did. Instead, follow the trail I cut. Seek, recognize, evaluate and use distinctions and commonalities.

The question you're probably asking yourself right now is "will this work?". After all, you can look in any newspaper, on any television station, any Internet spam, and someone claims to hold the key to life-changing information. And most of it is, to put it nicely, fully digested and expelled horse food. Why should my claim about distinctions and commonalities be any different?

For starters, I'm not charging $99.00 or $999.00 for this information. I'm giving it to you free, with no further books, classes, courses or seminars to buy. Second, using distinctions and commonalities is trivially easy, and you can start using them with little committment. Just start being aware of differences and similarities you observe, and if you find them, try to find distinctions that caused the observed differences and similarities. Finally, you need a way of evaluating all those enlightenment seminars and business opportunity workshops. You can use distinctions and commonalities to evaluate them. Whatever area you'd like to succeed in, research those who have succeeded and those who have failed. Find commonalities in all the failures, commonalities in all the successes, and distinctions between the two groups. If the results indicate a factor which you believe the seminar will genuinely illuminate, the seminar is worth looking into. Otherwise, the seminar is a time wasting sales pitch.

So follow the path I cut through the jungle of human thought. Use my 20 year conclusions as your starting point. Imagine how far you'll go.

Steve Litt's newest book is a complete 300+ page treatise titled "Troubleshooting Techniques of the Successful Technologist".  Steve can be reached at Steve Litt's email address.

Life After Windows: 8 Month Progress Report

By Steve Litt
Life After Windows is a regular Troubleshooting Professional column, by Steve Litt, bringing you observations and tips subsequent to Troubleshooters.Com's Windows to Linux conversion.

Status on April 1st

The April 2001 Troubleshooting Professional detailed Troubleshooters.Com's conversion from the Windows desktop to the Linux desktop. It contained a description of what was left to do:
Work Yet to be Done

When I waved a fond good-bye to Bill Gates in early March, Bill and I didn't completely part company. I still use the Windows box to run my CD writer hardware, my Zip drive, and my scanner. The CD writer and Zip drive will probably be easy to move over to the Linux box, but the scanner stays where it is -- it doesn't support Linux. 

More importantly, I still haven't found a suitable replacement for MS Word in writing documents more than 30 pages -- especially books. Nor have I found a suitable replacement for Windows Draw, for non-diagram type drawing. Diagrams are handled by the superior Dia GPL package. 

All of this is OK. Linux is progressing rapidly, so I'd expect that sooner or later there will be a Linux tool superior to MS Word for writing books. And with the advent of XML and the Scalable Vector Graphics (SVG) language, I highly suspect that a replacement for Windows Draw is coming down the pike.  But I'll probably use Windows Draw for years to come on old drawings, because Windows Draw stopped all development years ago, and they don't export to a usable vector format. And because Windows Draw, good as it was, was just an also-ran in the graphics world, I've found no Linux packages that could import .drw files. 

So my Windows box sits in the corner, an appliance to run legacy software. In my opinion, that's the ideal use of Windows. 

CD writer

The CD writer was transferred to the Linux machine about a month later. It was a little difficult, so once I understood the process, I placed instructions for CDRW integration on the net (see URL's section). Once the CDRW was on my Linux box, I was able to say goodbye to that lame Adaptec Easy CD Creator software, and create a script using tar, cdrecord and mkisofs., so that now backups are as simple as issuing the command
allback 011201
in order to back up to a file called d011201.tgz on the CDRW. My allback command also creates a checksum file on the CDRW, so in 10 years I can test the CD for corruption. Finally, I test the backup against the original data with the following command, which must be done from the root directory:
tar dzf /mnt/cdrom/d011201.tgz
If the command proceeds silently, then there are no differences detected between the backup and the original file system. Backups are so much easier these days.

My old CDRW was an ancient Acer 2x. I recently bought a 16/12/40x and installed it on my experimental machine. It worked perfectly. Soon it will go in my main machine, after which backups will be much faster.

Zip drive

I have little doubt I could have gotten my zip drive to work on my Linux box. Most of my buddies have done so. But of my 4 IDE ports, only 2 are compatible with CD's and Zip Drives. On those two ports I have the boot drive, the CDRW, the CD. I didn't want to put the Zip drive on the same IDE chain as my root drive hard disk, so I left it on the Windows machine. In this day of cheap CDRW media and nice apps like xcdroast and gtoaster, a Zip drive isn't as essential as it once was.


My scanner broke, and I haven't yet bought a replacement. I'll probably get a SCSI scanner for maximum Linux compatibility.

Windows Draw

There's never been, and probably never will be, a vector graphics app as good as Micrografx Windows Draw. Its versatility and ease of use were outstanding. Its collection of vector clip art was unmatched.

But Windows Draw has problems. It's a proprietary orphan. Micrografx sold Windows Draw to some unknown company that basically sank it to the bottom of the ocean. So it's unsupported, illegal to copy, and there's no source code, even though nobody derives profit out of the software. These days I use Windows Draw only for files already done in Draw.

For other work I use a combination of Dia, Gimp and Kontour (was Killustrator). This is not as good as Windows Draw, but I get by.

MS Word

I thought it would be impossible to replace MS Word for writing books. Word had great styles, its keystroke shortcuts were productive, and its outline view enabled you to switch between design and authoring mode. In April 2001 I thought nothing in Linux had Word's ease of use, outlining capability, and ability to handle large documents.

A few months later, I'm happy to say I'm putting the finishing touches on a book written entirely on a Linux box with Open Source software. I created a set of Vim configurations and scripts called VimOutliner, and used VimOutliner to create a 970 headline outline for the book. VimOutliner is faster and more productive than the outlining facilities in MS Word, Wordperfect, or Grandview.

I then wrote a simple script to transfer that outline into an empty LyX file, so that the LyX file contained the entire book structure using the proper headings. From then on writing the book was a simple typing task. For graphics I used Gimp, Dia and Kontour.

LyX has tons of wonderful features, but the feature most valuable to me is its easy to parse text native format. Time and again I wrote scripts to manipulate my book using Perl and Vim. LyX didn't have a word counter (unless you spellchecked), so I wrote a wordcounting shellscript. When I couldn't figure out the LyX user interface for making tables, I made them with Vim.

LyX has an incredibly steep and long learning curve. Luckily, there's tons of LyX and LaTeX documentation on my Mandrake distribution. There's an excellent LyX website and mailing list. And because of LyX's easy to read native format, I found ways to deduce style to format conversion using reverse engineering (which is, and always will be, completely legal on Open Source).

So how did my book turn out?

Great! Beautifully typeset by LyX/LaTeX, with just the right margins and binding margin. Styles are consistent throughout the entire book. It looks professional. For the first time, I've gotten an accurate table of contents without the need to "fine tune" the page numbers. And this is my first book with an index -- I just wasn't up to the task back in the days when I had to write books in MS Word or WordPerfect.

Fax Software

I couldn't get fax software to work reliably in Windows, and I can't get it to work at all in Linux. The good news is that Linux improves at an accellerated rate, so it's probable within a year faxing from Linux will be trivial.

Life is Great

I rarely turn on my Windows box these days. Its primary usage is sending out my troubleshooting course, which is in MS Powerpoint format because all my customers have Powerpoint. Interestingly enough, I've switched the instructor notes over to LyX, which I then export to PDF format.

Most of my activities are more productive under Linux than Windows. The one exception is certain types of graphic tasks, where I sorely miss Micrografx Windows Draw (but don't miss it enough to put any more data into that format).

Steve Litt is the author of the course on the Universal Troubleshooting Process.  He can be reached at Steve Litt's email address.

Linux Log: Reverse Engineering for Fun and Profit

By Steve Litt


Most proprietary licenses prohibit reverse engineering. Modern laws such as UCITA tend to give such prohibitions teeth. Given the current big-business political climate of the United States, one could end up in serious legal trouble performing reverse engineering on proprietary applications. I'm not telling you what to do -- that's your decision.

The purpose of this article is simply to show you how to use distinctions and commonalities to reverse engineer the software of your choice, in order to learn more about that software or its native data format.

Naturally, it's perfectly legal to reverse engineer Open Source software.

Discovering LyX

LaTeX is a complex and competant markup language for typesetting. It typesets a document as an entire document, not as a series of pages, paragraphs, lines or letters. It has provisions for headings, table of contents, index, bibliography, chapters, headers, footers, and much, much more.

LyX is a front end for LaTeX. By using LyX, the author avoids the brain-stealing task of inserting markup. The LyX author types his material at top speed, and then prints out a beautifully typeset book.

But life with LyX isn't perfect. Adding new styles (which are called "Environments" in LyX), or changing the formatting of existing styles, is extremely difficult. The first problem is that LyX native format and LaTeX tags look so similar that it's hard to know when you're using LyX and when you're using LaTeX.  This is combined with the fact that there's relatively little documentation on LyX, and what there is is either descriptions of trivial tasks, or written for the person with over a year's LaTeX experience. So the relative newcomer who wants his book formatted in ways other than the LyX default faces some issues.

One day I needed to format my table of contents with a spacing factor of 0.8 instead of the default 1.0. LyX has a document-global spacing factor adjustment, which is accessible through the menu system. So I made two identical documents via file copy, and then made the spacing in one of them 0.8. Next I exported each file to a LaTeX file, ran diff on the LaTeX files, and here's the result:
[slitt@mydesk slitt]$ diff junk1.tex junk2.tex
> \usepackage{setspace}
> \setstretch{0.8}
[slitt@mydesk slitt]$

So I added \usepackage{setspace} to the document preamble, and added \setstretch{0.8} to a local part of the table of contents, and bang -- I had the table of contents, and only the table of contents, spaced at a 0.8 factor.

Look what happened here. The LaTeX codes for 0.8 spacing were a black box, but 0.8 spacing can be incorporated at the LyX level. Or not.

So I created a LyX document with 0.8 spacing, and exported that document to LaTeX. At this point, the contains the proper LaTeX codes. But the codes are intermixed with literally hundreds of others. Which codes are necessary for 0.8 spacing, and which are used for different things? Here's where distinctions take over.

The next step is to create a LyX document identical to the one with 0.8 spacing, except that the new document has default spacing. After exporting to LaTeX, I now have LaTeX files for 0.8 and for default. Whatever is different about the two files defines the codes that can toggle between 0.8 and default spacing. So I ran a diff command and came up with exactly two codes. The distinction between LaTeX producing default and 0.8 spacing is those two lines.

Since then, I've used the same methodlogy several times to deduce specific LaTeX codes needed for my LyX layout file.

Distinction-based Reverse Engineering Process

As discussed in the preceding example, you can reverse engineer using distinctions. The idea is to toggle a single factor in the computer program, and isolate the differences made in the native file layout. By doing so you can determine the function of every byte of the native format file, and thus interoperate with it (or possibly migrate away from it). Distinction based reverse engineering is best done as a process: When checking for differences, use the diff command on ascii file formats such as XML, HTML, LaTeX, etc. For determining differences in binary files you might need to write your own program, because on binaries diff simply reports whether the files are the same or different. You can use the cmp program to find the first differing byte, and then navigate to that byte in an editor.

Check for compression

Many file formats initially appearing binary are really compressed text -- usually XML. So try using the zless command on the file. Try other decompressions too. Naturally, if you find it decompressible, reverse engineer with the decompressed format.

Consistency testing

You need to know whether distinctions discovered in the native format file are meaningful, or whether they're random distinctions over time. For instance, using Microsoft Word with the fast save option, you can save the same document four times and get four different files, because you're saving a differential on top of an original. Another way that document saves could be inconsistent is if a timestamp is saved somewhere within the file. In such a situation, the difference would be very localized, and the bytes defining the time could probably be deduced, after which those bytes could be ignored.

So before reverse engineering, take your intended document and save it several times, and verify that all the saves produce identical files. Once you're saving identical files you can go ahead to single character substitution.

Single character substitution

NOTE: You can usually skip this step if the native file format is a text file (LaTeX, HTML, XML, etc).

Native file formats are easiest to reverse engineer if characters of actual content have a one to one relationship with bytes in the file. You can determine that by creating a document with a single letter (start with A), and then change it to B, and C, and D..., saving each file separately and then diffing them. One of three things will happen:

  1. A single byte of the file is impacted, and that byte is identical to the letter being changed in the document
  2. A single byte of the file is impacted, and that byte is NOT identical to the letter being changed in the document
  3. More than one byte is impacted
If it's #3, reverse engineering this file is beyond the scope of this article. Otherwise, you have a pretty good idea of the relationship. Repeat the preceding test with longer files. Also, try changing the character to a punctuation character. Many file formats handle punctuation differently.

Once you're confidently familiar with the relationship between document content characters and bytes in the native file format, it's time to actually add a character.

Single character additions and deletions

NOTE: You can usually skip this step if the native file format is a text file (LaTeX, HTML, XML, etc).

Add a character after the one you changed in the preceding test. If all is well, this adds one byte to the file length. If all is well, a new byte is added right after the one examined in the preceding test. Maybe there are more bytes added -- one representing the character and one or more representing properties of the character. Get familiar with the native file changes that come with additions and deletions of characters in the document.

Property toggles

Change a word to bold, and check the differences. Once you think you have a handle on how bolding is accomplished, make a copy of the file and try to bold a different word from within your text editor, and then look at that new file in the application. If the word is properly bolded, you have a good handle on how things get bolded.

Go through the same procedure for any other properties of interest.

Reverse Engineering Open Source

Why in the world would one reverse engineer Open Source? After all, you already have the source code -- so reverse engineering is unnecessary.

Or is it? Remember my experience with LyX? Sure, I could have gone through the source of latex2e and surmised how to get 0.8 spacing. But it would have been brutal, and latex2e would have known nothing about the all important setspace package. My distinction based reverse engineering explorations provided an answer within 5 minutes.

Open Source is ideal for reverse engineering because most native format files are either XML or some other markup. Most of the ones that appear binary are really just compressed XML or markup. This means evaluating distinctions is a simple diff command. Because Open Source has no incentive to obfuscate the native format, its data files are typically very readable, understandable, parsable, and reverse engineerable. Often the fastest way to understanding an Open Source package is through reverse engineering.

An even greater reason for reverse engineering an Open Source package is to add a feature without messing with source code. I needed to convert my VimOutliner generated tab indented outline to a LyX heading structure. LyX provided no way to do this (why should it -- VimOutliner didn't exist when LyX was created). No problem. I reverse engineered the LyX native format enough to make a script to convert VimOutliner outlines to LyX code, and used Vim to paste in the converted code. Bingo -- the outline became the book's heading structure. Since converting to Linux on the desktop, time and time again I've reverse engineered the native format of a data file, and then tweaked it with a Perl program or a Vim script.

For Fun and Profit

The fun part is obvious -- anything having to do with computers is fun, and if it's Open Source, it's scandalously fun. But how about profit? My 0.8 spacing and some other reverse engineered tweaks enabled me to trim 6 pages off the book, thereby saving on paper, handling, and shipping. As a matter of fact, my entire book is a monument to distinction based reverse engineering of Open Source.
Steve Litt is president of Linux Enthusiasts and Professionals of Central Florida (LEAP-CF). He can be reached at Steve Litt's email address.

Letters to the Editor

All letters become the property of the publisher (Steve Litt), and may be edited for clarity or brevity. We especially welcome additions, clarifications, corrections or flames from vendors whose products have been reviewed in this magazine. We reserve the right to not publish letters we deem in bad taste (bad language, obscenity, hate, lewd, violence, etc.).
Submit letters to the editor to Steve Litt's email address, and be sure the subject reads "Letter to the Editor". We regret that we cannot return your letter, so please make a copy of it for future reference.

How to Submit an Article

We anticipate two to five articles per issue, with issues coming out monthly. We look for articles that pertain to the Troubleshooting Process, or articles on tools, equipment or systems with a Troubleshooting slant. This can be done as an essay, with humor, with a case study, or some other literary device. A Troubleshooting poem would be nice. Submissions may mention a specific product, but must be useful without the purchase of that product. Content must greatly overpower advertising. Submissions should be between 250 and 2000 words long.

Any article submitted to Troubleshooting Professional Magazine must be licensed with the Open Publication License, which you can view at At your option you may elect the option to prohibit substantive modifications. However, in order to publish your article in Troubleshooting Professional Magazine, you must decline the option to prohibit commercial use, because Troubleshooting Professional Magazine is a commercial publication.

Obviously, you must be the copyright holder and must be legally able to so license the article. We do not currently pay for articles.

Troubleshooters.Com reserves the right to edit any submission for clarity or brevity, within the scope of the Open Publication License. If you elect to prohibit substantive modifications, we may elect to place editors notes outside of your material, or reject the submission, or send it back for modification. Any published article will include a two sentence description of the author, a hypertext link to his or her email, and a phone number if desired. Upon request, we will include a hypertext link, at the end of the magazine issue, to the author's website, providing that website meets the Troubleshooters.Com criteria for links and that the author's website first links to Troubleshooters.Com. Authors: please understand we can't place hyperlinks inside articles. If we did, only the first article would be read, and we can't place every article first.

Submissions should be emailed to Steve Litt's email address, with subject line Article Submission. The first paragraph of your message should read as follows (unless other arrangements are previously made in writing):

Copyright (c) 2001 by <your name>. This material may be distributed only subject to the terms and conditions set forth in the Open Publication License, version  Draft v1.0, 8 June 1999 (Available at (wordwrapped for readability at The latest version is presently available at

Open Publication License Option A [ is | is not] elected, so this document [may | may not] be modified. Option B is not elected, so this material may be published for commercial purposes.

After that paragraph, write the title, text of the article, and a two sentence description of the author.

Why not Draft v1.0, 8 June 1999 OR LATER

The Open Publication License recommends using the word "or later" to describe the version of the license. That is unacceptable for Troubleshooting Professional Magazine because we do not know the provisions of that newer version, so it makes no sense to commit to it. We all hope later versions will be better, but there's always a chance that leadership will change. We cannot take the chance that the disclaimer of warranty will be dropped in a later version.


All trademarks are the property of their respective owners. Troubleshooters.Com(R) is a registered trademark of Steve Litt.

URLs Mentioned in this Issue