Introduction

This page is a simple introduction to the problems with systemd, intended for the person wondering what all the systemd fuss is about. As always, there are two sides of the story, and the typical starting point for pro-systemd arguments is Lennart Poettering's The Biggest Myths blog entry. Several counter arguments to Poettering's "Biggest Myths" are put forth by Jude Nelson's Systemd: The Biggest Falacies.

This page differs from those because it's simpler and works from the analogy of a hypothetical new car electrical system to systemd. Of course, analogies are never perfect, so once you've digested the matter on this page, you can move on to more detailed documentation.

The New Electrical System

Note:

This electrical system is used as a broad analogy, not as a one to one feature analogy with systemd. Like all analogies, it's imperfect. But I think it gets the broader ideas across.

Imagine that, a few years from now, a company (call them Blue Spark) comes out with a new type of electrical system for cars. Let's call the new electrical system System M, where M stands for "Modern".

Instead of having your basic 12 to 14 volt wiring throughout the car, this new electrical system sports an alternator outputting a 40kHz square wave alternating between plus and minus 60 volts, with its center tap going to ground, so that the maximum voltage between ground and either power cable (yes, there are two of them now), is 30 volts on every half cycle. 60 volts switched at a 40kHz rate is just right for switching regulators, switched mode power supplies, and inverters. These two 60 volt, 40kHz cables are accompanied by a data channel cable to tell each consumer of electric power how to optimally use that power.

What this means is that electrical cables no longer need to carry the huge amperages of 12 volt electrical systems. It also means that each consumer of the alternator's output can, with its own switching regulator, efficiently tailor the perfect voltage for its application, or even change its voltage and timing, in real time, for best performance, by reading "messages" addressed to it in the data channel cable. Spark plug voltage and timing becomes much simpler. High power and low power audio can both be accommodated, without the insane currents implicit in twelve volt sound systems. With various components able to send and receive messages on the data channel cable, the duties of the central computer are lessened. Today's birdsnest of sensing wires are replaced by a single electrical cable with special data channel cable.

This is a System

System M is a coordinated and matched system, not a bunch of parts. Each part is built specifically to work with all the others, following the specification of the electrical system.

At 40kHz, the electrical skin effect means that only the outer 0.33 mm (millimeters) of any wire actually conducts electricity, so for efficiency, the electrical system's cables consist of thirty of #22 insulated wires, braided in a strategic way, accompanied by a special #20 stranded data channel cable. The special cables to the starter motor have 240 such #22 wires.

To accommodate various splices, the car is built with several strategically situated special Tee couplers to take power and signal off the main cable. This is important, because you can't Tee into this new style cable with a wire clamp, the way you could on the old 12 volt systems. If you need to add a new power consuming device, you join it into one of those Tee connectors with a special Y connector that replaces the consumed Tee. In fact, a trip to the auto store showcases all sorts of connectors, adapters, and extensions for the new electrical system.

With the 40kHz square wave, Radio Frequency Interference (RFI) needs to be addressed, so the specification calls for various, strategically placed capacitors, inductors, and ferrite beads throughout the electrical system. Also, body parts for cars with the new electrical system must be designed to shield the outside world from RFI, and must be installed in a manner that preserves their RFI shielding, complete with a highly specified grounding system that includes 60 wire cables and special, weld-on Tees to connect that cable. It's important that when something is added or changed, no ground loops or RFI be created. States that have smog testing now also test for RFI.

Everything works together to produce optimal results. Everything is designed to the new electrical system. Design becomes much easier, and the differences between various car brands become much less pronounced. This is the electrical system for the modern car.

This Brave New World

Cars are a very old technology. The Ford Model T was introduced in 1908. The System M electrical system brings cars into the modern era. Even though a few old-time mechanics don't like it, the rest of us know that this is the electrical system for the modern car. Everything works together for a flawless, unified result. It's like a symphony orchestra.

This modern, unified system provides and requires modern, unified diagnostic tools and tactics. In the old days, when the car didn't start, you could measure the voltage across and current through the starter motor, so that if the voltage is OK and the motor doesn't turn, it's a bad starter motor. The new way is to plug your special new Blue Spark brand System M test hardware into an available Tee connector, view messages to and from the starter motor, as well as several other messages, go through a table of variables, and determine whether or not the starter is bad. Naturally, this takes a modern technician. Shadetree mechanics can no longer do the job, but that's a good thing because Blue Spark educates and certifies System M qualified auto technicians to work on your car.

Bill Zolche

Bill Zolche is a Sysadmin with a wife (Karen) and two kids. One of the many ways the family has put their budget squarely in the black is that they buy older, cheaper used cars, and Bill does most of the service on the cars. In 2010 he bought his 2007 Jeep Patriot for simplicity: Standard transmission, two wheel drive, manual roll up windows and manual door locks, and a tiny 2.0 liter engine leaving enough engine compartment room to hold a high school dance. At 227,000 miles, it's still going strong, although Bill's replaced the clutch twice, four sets of motor mounts, a set of shocks, and seven sets of brake pads. Bill has the knowledge and the tools to do almost all the work himself.

Until recently, Karen drove a 1998 Buick LeSabre. It's much more complicated than his Jeep, but he's done almost all the maintenance himself. At 185,000 miles, it needed its second replacement transmission (Bill can't replace or rebuild automatic transmissions), so they decided to invest in a newer car. They bought a used Danville Grayhound (from Danville Motors, Inc.). The Danville Grayhound has the new System M electrical system. Six months later the brakes suddenly failed, but luckily Karen had the presence of mind to throw it into low gear and turn off the ignition.

Bill has fixed plenty of failed brakes in his more than twenty years of maintaining his own cars. Usually catastrophic brake failures are due to a failed power booster or master cylinder or a brake fluid leak, but this car contains nothing hydraulic. The brakes are electrical brakes. All Bill could ascertain was that all four brakes were acting the same. He couldn't go farther. So he took it to the closest place, Sammy's Neighborhood Garage.

Bill and Karen got the Danville Grayhound back kinda-sorta fixed, but with dangerously low brake power. So they took it to Adenson's Automotive Repair, which was a much longer tow, but it's a small shop with an outstanding reputation. The service manager at Adenson's took a quick look, said that whoever worked on the car last had made several kludges to the electrical system that could cause any number of side effects. The service manager said they don't work on System M cars because System M complexity would require them to hire System M certified techs, whose pay is astronomical. The service manager suggested taking it to a dealer. He also said the dealer would need to back out the "repair" from Sammy's, before fixing the actual problem.

So Bill took the Danville Grayhound to the service department at Wexler's Danville. The car was repaired perfectly, at a cost of $1200.00, $500 of which was to undo Sammys' "repair" and some damage that it did, and $700 for new windshield wiper motors. The wiper motors were outputting bogus signals addressed to the electric thermostat, causing the thermostat to output signals with the address code of the brake pedals, and those signals were interpreted by the brakes as brake pedal not pressed, so the brakes didn't engage.

Three years later, Danville Motors Inc. issued a Technical Service Bulletin (TSB, basically a recall), specifying free replacement of that model's windshield wiper motors and powered thermostat, because they were known to cause brake failure. That did Bill no good: He had sold the car, at a considerable loss, right after the repair was complete. Bill and Karen buy only non-System M cars now.

Adenson's Automotive Repair

Vince Adenson opened Adenson's Automotive Repair thirty years ago, out of a closed-down gas station with a single lift. In the intervening years Adenson's had grown to a six lift facility with more business than it could handle. Word was out that Adenson's was a little more expensive, but they always fixed the problem quickly and on the first try. Vince Adenson gave all his technicians special training on the mindset and process of troubleshooting, so that they all prioritized narrowing down over fixing, and process over memorization. Vince had hired a lot of high school kids, training them to become full time mechanics, doing the work right.

However, the past five years, more and more car companies had switched to the System M electrical system, and those cars were now out of warranty. Vince didn't want to send his technicians to expensive Blue Spark certification training, because of the cost and the fact that a lot of Blue Spark certified technicians couldn't troubleshoot their way out of a paper bag. So Vince was accepting less and less cars for repair. The business was still very profitable, but the writing was on the wall: As the remaining fleet of non-System M cars got older and older, cost of repair would exceed cost of replacement, and he'd be scraping the bottom of the barrel.

Vince sold the business for a reasonable amount: Enough to buy two fast food franchises. He hired several of his former mechanics as managers and workers, and treated them well. Meanwhile, the new owners of Adenson Automotive Repair spent a fortune on System M certified technologists, most of whom could name every component in the car but couldn't find the root cause. Adenson's Automotive Repair, under new management, went broke.

Blue Spark

Life has been good for Blue Spark management and stockholders. Revenue from their Education and Certification department has quadrupled as mechanics and their employers saw the futility in troubleshooting System M equipped cars without Blue Spark training. As an added benefit, during training, Blue Spark scouted and recruited the best and brightest, for the Blue Spark Consulting and Service department.

Large repair facilities, especially dealerships, have realized that all Blue Spark System M certified technicians were not created equal, and to get a competitive advantage, many of these large repair facilities now have a Blue Spark consulting and service contract, to bring in genuine Blue Spark consultants to handle the toughest repairs.

Blue Spark's marketing consists of encouraging car manufacturers and aftermarket manufacturers to make System M compliant components and subsystems, including giving them design help. An example of such help was a recent sound system that reads error codes on the data channel cable, and speaks those errors on the sound system. This sound system also allows drivers to speak the desired speed into the sound system, after which the desired speed is sent as a message to the engine. Car companies and dealerships like these features because they sound good in commercials and sales pitches, so even more car companies adopted System M. A few engineers pointed out that these features would have been just as easy to implement on the old 12 volt system, but they were dismissed as "graybeards" and "Model T lovers".

Most people love System M. Blue Spark loves it for obvious reasons. Dealerships love it because it removed small garages as competition for their service departments. Car companies and salesmen love it because of all those features they can sell. Used car salesmen love it because prices for pre-System M cars increase every year, although they worry about the abysmally low resale prices of used System M cars. Car manufacturers are selling more new cars than ever, because as repair prices and complexity rise and repair reliability falls, more people opt to buy rather than repair.

System M certified auto technicians love System M. Relieved of competition from less educated technicians, their pay has gone up by forty percent. Even incompetents who formerly couldn't keep a job are highly sought after due to their certifications.

A few people don't like System M. Many small repair facilities have been forced out of business for lack of specialized training and diagnostic equipment. Experienced automotive technicians who couldn't afford System M training are having to leave the business. And a few consumers with older System M equipped cars have noticed that auto repair prices have doubled in the past five years, but those who see a correlation between those prices and System M are dismissed as Model T Lovers.

Parallels

Both System M and systemd replace something simple with something complex. In each case, the new system has complex input and output specifications, and in each case, the system requires a substantial portion of the the machine's (the car's or OS's) components to meet those complex specifications. In each case, a fault can be caused by a component seemingly unrelated to the location of the fault, often through a chain of causes and effects spanning the entire machine. Both systems make the overall machine much more difficult to repair. Both are promoted by vendors who profit by teaching and consulting, thus standing to gain from a more complex system. With both systems, people who bring up the points contained in this paragraph are summarily dismissed as people stuck in the past.

The Moral of the Story

Hopefully the preceding analogy made this one thing clear: The more you tie lots of things together, the harder it becomes to do repairs and maintenance. The following graphic shows why:

Troubleshooting is the repeated process of elimination. Ideally, you rule out half the remaining root cause scope with each diagnostic test or measurement. When each component, subsystem or system has one or two simple inputs and outputs, it's easy to determine which side of the point being tested contains the root cause. But when each component, subsystem or system has input from numerous other components, subsystems and systems, and sends output to numerous other components, subsystems and systems, the associated indirections and feedback loops make splitting the remaining root cause scope more difficult, possibly by an order of magnitude or two.

The systemd init system is "only" an init system in the same way as the analogous System M electrical system is "only" an electrical system. They both require most systems and subsystems to conform to their complex specifications, unless you want to do kludges like Sammy's Neighborhood Garage. Both make troubleshooting much more difficult. And the interaction entanglement of both makes more likely problems like "defective windshield wiper motors cause brake failure".

[ Training | Troubleshooters.Com | Email Steve Litt ]