Troubleshooters.Com® and Linux Library Present:

An Open Letter to Linas Vepstas

Copyright © 2016 by Steve Litt

CONTENTS:

Update 2/2/2023

On 2/2/2023 Linas Vepstas updated his rant against Linux based on his Ubuntu and Debian experiences, presumably because of the Freedesktop/Redhat/Poettering "inspired" /usr merge. So he fights his Linux distribution by trying to undo the /usr merge, and of course this led to more difficulties. Once again, if Linas wants to use Ubuntu and Debian, he needs to accept the carved in stone way those distros do things. Otherwise, he should simply change distributions. Because he seems like a "close to the metal" type guy, I'd recommend Void Linux.

Tl;dr

Linas Vepstas wrote what might be called an essay or might be called a rant, titled Open Letter to Debian and Ubuntu Developers. His first paragraph looks like this:

There used to be a time when Linux was a joy to use. Now, its a head-ache inducing slog through the bowels of the operating system. You have to be a brain surgeon and a rocket scientist. You have to work over-time. You have to have the patience of a saint and the perseverance of a boxer. I am so tired of this. I'm exhausted.

The entire essay continues on a similar note. Although the title implies this is a rant about Ubuntu and Debian, he seems to paint the entirety of Linux Land with the same broad brush. And that would be factually wrong.

"Factually wrong" doesn't mean he hasn't pointed out some serious problems. He has. I and many other Linux users see the same problems he identifies. What's "factually wrong" is that these problems are built into the combination of kernel, system software, and applications generally called either "Linux" or "GNU/Linux". And his implication that there's no reasonable way for a user to avoid these problems is also factually wrong.

The bottom line of my objection to his essay is this: Nobody should use software they don't like, especially if there's a reasonable alternative. And by extension, why is Linas still using Debian and Ubuntu and systemd and Firefox and Chrome and Gnome? There are reasonable alternatives to every single one of them.

But don't blame Linas: A lot of us are guilty of the same thing. The final section of this document digs into that fact.

My Letter to Linas

Hi Linas,

In your essay, titled Open Letter to Debian and Ubuntu Developers, you bemoan inability to boot, and all sorts of other bugs. In the essay you say:

"What's at the root cause of this? Sure, its some combination of programmer hubris, lack of system test, inexperienced and callous coders."

All causes you named are certainly causes, but you missed a huge cause, possibly the root cause. Possibly the root cause because it enables the causes you mention: Users accepting the bugginess that correlates to complex, monolithically entangled software whose software construction can be held in place only by continuous effort from multiple programmers, some of whose "day job" is maintaining that overly complex software. Gnome, KDE, systemd, Firefox and probably Chromium/Chrome, huge and complex initramfs early boots that do a lot more than just mount the root partition. Situations where initramfs, init, and desktop environments depend heavily on each other. Layer upon layer of abstraction whose intent is to make it easier for the people dealing with the next-outer layer, but the sum of those layers make troubleshooting as indeterminate as the ordering of daemons spawned by systemd.

Despite the title of your essay, as it goes on and on, it starts to look less like a repudiation of Debian and Ubuntu, and more like a repudiation of the world of Linux, and this is unfair. And given your painting of all of Linux Land with this broad brush, it begins to look to me like you really are continually using monolithic spaghetti monuments like Debian, Ubuntu, systemd, KDE, Gnome, and Firefox. If you've found an alternative, why not replace the rant with an essay praising the alternatives? For the reasons given in the preceding two sentences, the rest of my open letter to you assumes that you really are using software like Debian, Ubuntu, systemd, KDE, Gnome, and Firefox

Linas, why are you still using eye-candy filled, Bob/clippy laden distros? Lots of distros let you get right in and edit config files so that your computer performs to your specifications, not to Clippy's best guess as to your desires. Why do you use Gnome? This layer on top of X has a thick interface to the init system: What's that all about? And more importantly, given its gratuitous communications inside and outside of itself, why would you expect it to perform consistently or even properly?

This is not to say there aren't multitudes completely happy with Debian or Ubuntu, with systemd and Gnome and Firefox. Those programs have lots of developers watching over their every change, so they fail only on corner cases or with overuse. Many folks are happy as clams using them, and have no problems. But you're not one of those people.

These programs don't work for you. Either you exceeded their duty cycles, or you configured them in ways they weren't intended to be configured, or you expect too much from them, or you're simply having bad luck with them. Whichever, doesn't matter. The essential point is this: Why are you still using them? There are excellent alternatives. I use a set of those alternatives.

I prioritize working simplicity over "pretty" or "we do it all for you", and my selection of software reflects that prioritization. My Void Linux box inits with the Runit init system: An init system so simple that anyone understanding the Unix filesystem and elementary shellscripting understands Runit, and can effortlessly make their own run scripts (the equivalent of sysvinit "init scripts" and systemd "unit files"). It's trivial to make your own daemons to work with Runit. It really is that simple. Life is good, my computer boots every time, there's no problem I can't diagnose.

Rather than participating in the iffy and fuzzily defined sharing of nonvolatile UEFI variables by motherboard and OS (with the risk of bricking the mobo if things go wrong), I MBR boot a small SSD root partition to which GPT partitions from huge spinning disks are mounted. Works beautifully, just like it has for decades. The OS interfaces with the Mobo settings through the thinnest of interfaces, if at all. I get the best of both worlds: the simple module encapsulation of MBR boot, and the huge and limitless in number partitions of GPT formatted spinning disks. Life is good, my computers boot, and I don't worry about bricking my mobos.

My "desktop environment" is Openbox assisted by Suckless Tools' dmenu program selector, and a few well-placed hotkeys. I don't use Firefox because it slows the computer to a crawl: The Chromium browser provided by the Void Linux package manager seems to work just fine, but when it doesn't, Luakit steps up to the rescue.

People mock my "ugly" and "1995" user interface. They tell me that mere mortals cannot operate my computer: Mere mortals must be led by the nose through every eventuality because they're just too stupid (usually phrased as "you're not a programmer") to edit a config file. This "users are too stupid" argument is nonsense made up by big corporate software to justify continuous revenue streams based on features that get in our way. To see a smoking gun, go to This interview with Red Hat CTO, Brian Stevens, search the first occurrence of the word complexity, and read that paragraph and the one below it. That's the explanation for the explosive complexification of some realms of Linux in the past two decades. Those two paragraphs also provide a possible explanation when you ask why systemd came uncomfortably close to conquering the world.

Anyway, my "ugly" and "1995" user interface might not be operable by the mere mortals who have drunk the "stupid" flavored drink of Jobs, Gates, and Brian Stevens, but they're perfectly operable by a person with a little motivation and half a brain. And they always boot. And seldom crash or even get the slows. And they're so modular I can incorporate almost any behavior I want into my computer. You could do this too, Linas.

About the part of your essay saying the following:

"What's at the root cause of this? Sure, its some combination of programmer hubris, lack of system test, inexperienced and callous coders."

If you want another cause, and very likely the root cause, look in the mirror, because you're part of it. When Ubuntu and Debian gave you garbage, you kept using it. When Firefox kept crashing, you kept using it. You continue to use Gnome. You made it painless for these software projects to keep serving you complex, crashy crap.

If you and all the rest who have griped about slow performance, instability and difficulty of repair had actually migrated to the alternatives, the overcomplexified software projects would have lost relevance, lost developers, and either died or delivered a simple, modular program. Migrating away from crashy entangled monoliths would have been a service to the entire Free Software community, and if you'd written about it, many others would have followed in your footsteps. And there's another service you could have performed, and are perhaps uniquely positioned to perform...

Your essay could have explained, in language someone who has drunk the "I'm too stupid to edit config files" flavored drink could understand, why even though they can't see it, internal software complexity is bad for them and will eventually cause them heartache.

In the realm of politics, one often hears "If you're not part of the solution, you're part of the problem." I think this is true in the realm of Free Software too.

What This Means To All Of Us

In my Letter to Linas I outlined an essay he could have written that would have been both more accurate and more constructive. The section you're reading right now is a first attempt at such an essay, for the population at large.

Linas was right about one thing: There's some truly opaque, crashy and inefficient Free Software out there. He pretty much got it right with his examples.

But there's also some excellent Free Software out there. If you're using Ubuntu, Debian, Redhat, Centos, OpenSuSE, Mint, or any other Linux distribution with layer upon layer of abstraction to "do it all for you", and it doesn't work for you or you don't understand it to the extent you desire or it doesn't boot or it takes 10 minutes to shut down or you can't find a systematic, intelligent way of troubleshooting it, try a less ornamental and "helpful" distribution.

Devuan comes to mind first and foremost: It still does a lot for you, but it uses sysvinit instead of systemd, and the community is very friendly toward setups with lightweight applications. The Manjaro-OpenRC distro (Manjaro with an OpenRC init) is another user friendly distro without the complexification bestowed by systemd.

If you want to exercise ultimate control over your computer and are willing to put in the learning and work that entails, try Void Linux. It's the most logical, conceptually understandable Linux distro I've encountered. But maybe you want even more control. Try Gentoo or Funtoo. Both init with the technically acclaimed OpenRC, and both enable you to compile your programs exactly how you want them. On the other hand, perhaps all you really want is Debian before it went systemd. Try PC-BSD. It's user friendly, conceptually simple, and solid. Speaking of solid and BSD, if you want a strictly metal server (no Virtual Machines), OpenBSD is about the most solid system I've seen. If you want a distro that does nothing for you, forces you to do everything, including calculating software dependencies, so that your computer is exactly how you specified, even if you broke some rules, Slackware's your distro. Perhaps you have nothing against systemd and want a distro that, other than systemd, is conceptually simple. That would be Arch. The point is, if an overblown and overdecorated distro is giving you trouble, use one of the conceptually simple distros.

Then there's the "Desktop Environment", whatever that might mean. If you use Unity, KDE or Gnome and strange things are happening, that's perfectly understandable. These three desktop environments are a lot like a game of Jenga. They might work well as shipped from the factory, but over a period of config changes and upgrades, their complexity makes them ever more rickety, or introduces hard to diagnose bugs. If such things are happening to you with one of these full features behamoths, switch to a simpler user interface such as Xfce, LXDE, or even Openbox, with or without a panel added to the system. To compensate for some of the drag and drop workflow shortcuts you lost in the switch, install Suckless Tools' Dmenu, and life will take on unbelievable productivity.

Don't settle for unstable, indeterminate, or difficult to repair software. Vote with your feet. If you don't like your software, move to something simpler and more stable. Not only will you make your life easier, but your rejection of bloatware will be one more drop in the ocean of pushback against bloat. If enough people follow your lead, we'll all get better software.

Of course, prioritizing simplicity and stability will doubtlessly earn you scorn from the fanboiz. The scorn follows a process, something like the following, when you tell them "I don't use program Z":

They're fanboiz. It's what they do, and it has nothing to do with you. Ignore them and keep using the software of your choice.

One piece of advice, though: Don't criticize them for the tech they choose either. If they're in love with their Ubuntu system with KDE, systemd, Pulseaudio, Firefox with all the plugins, and Gnome-Files, well, that's what works for them. The day you tell them to abandon the software they love, let's say Ubuntu/systemd/Unity, for Funtoo/OpenRC/Openbox/dmenu, you've become the fanboy.

The Case For Simplicity and Modularity

People who have neither repaired things professionally nor designed technological systems professionally can be forgiven for not knowing the cost of complexity, or what modularity really means. If you never peal off the case or troubleshoot the software, it looks like a black box regardless of what's inside.

Modularity is the single best predictor of repairability. Modularity requires more than composition by modules. It requires that modules communicate strictly on a need to know basis, and that they communicate via a simple interface. Typically, a module is built from several sub-modules that talk to each other, but the sub modules talk to modules outside their own only through their parent module. By limiting talk to a "need to know" basis, thereby limiting the lines of communication, when it's time to troubleshoot, the system makes it easy for you to trace cause and effect. The following is a diagram of a very modular system:

Diagram of a modular system

The preceding diagram has 16 sub-modules inside 4 main modules. The upper left main module is obviously the master module: The only one connecting to all the other main modules. This configuration has very few data flow possibilities, making it trivial to troubleshoot. Any module or sub-module can be swapped out trivially, and any interaction line can be measured, strongarmed to a value or rerouted, knowing exactly what it should affect and what it shouldn't. The preceding diagram is more modular than most systems, but many systems endeavor to go in the direction of the preceding module.

The following diagram shows a system with 16 modules, but with promiscuous intermodular communication:

Diagram of a non-modular system

Same 16 modules, but what a difference! If you disconnect one interaction line, what will be the effect on the entire system? The change keeps rebounding along interaction lines from module to module. It's like troubleshooting in an echo chamber or a house of mirrors. Nothing but feedback loops. No wonder a small config change or a minor software update can screw it up in a way difficult to diagnose by any logical process, at which time you're forced to ask on StackExchange or UbuntuForums and cross your fingers.

You might encounter a bloat fanboy making the point that the modules don't talk to each other: Each one has exactly one interface with a central bus. That explanation looks great on paper, but it's silly: The central bus is just a traffic circle allowing any module to talk to any of the others.

As you would expect, fanboiz have talking points excusing, or even promoting such promiscuous interaction:

  1. It's modular. It's composed of modules, isn't it?
  2. We offer our own set of tools for troubleshooting our software.
  3. End users shouldn't troubleshoot software problems.

My replies follow, respectively:

  1. How stupid do you think I am? Anyone who has spent 6 months repairing machines, hardware or software, knows that the kind of modularity promoting repairability precludes promiscuous interactions. Save this soundbyte for the newbs.
  2. That's nice, at least until there's a problem your tools can't solve, and then the Troubleshooter is out of luck, because overly complex systems require overly complex tools, so the Troubleshooter can't make her own tools the way she could on simple systems. Oh, and every user must learn a set of tools for this one software system. What if every software system required their own unique toolset?
  3. Please have your credit card ready.

Modularity means dividing the whole into parts, and those parts into sub-parts, on and on until every part of the system is simple. It also means that the individual parts talk only to other parts that they need to talk to, rather than promiscuously talking to a slew of parts.

Resilience

This document came about as a reply to somebody who griped about Ubuntu, griped about Debian, griped about systemd, griped about Firefox and Chrome and Gnome, but nowhere in his gripe did he say he moved away from these technologies. This is quite common behavior. More than a year after the end of the systemd wars on the Debian-User mailing list, people are still griping about systemd. If they don't like systemd, why are they using Debian, whose very politics demanded systemd?

I think the answer is lack of resilience. Ask a systemd-griper who is still using systemd why he doesn't move to Devuan, Manjaro-OpenRC, Void Linux, OpenBSD, PC-BSD, Slackware, PCLinuxOS, Gentoo, or Funtoo, and the answer is usually like this:

Sounds like this guy enjoys griping a lot more than he dislikes systemd. Either that, or he lacks resilience.

Debian 7 is old, and it's not coming back. The closest you're going to get to "Debian 8 without systemd" is Devuan. Otherwise, you must be resilient enough to work with unfamiliar situations. Maybe a chroot install, maybe a rolling release, maybe a change to Linux's cousin, BSD, maybe do installation and updating by compilation, maybe use a distro with few users and therefore less tech support opportunity. In computer usage as well as most other human endeavors, resilience is predictive of happiness and success.

About Devuan being Beta: Lots of people are using Devuan for production systems, and have been since Devuan Alpha 2. And Devuan has a big, friendly, helpful and knowledgeable mailing list to help you.

Same principles for "desktop environments". The only way Unity, Gnome and KDE do their spectacular mouse-foo and beautifully transparency-shadowed looks is with zillions of lines of code, and we all know what risks you run with so much code. So if Unity, Gnome or KDE start treating you wrong, don't go looking for "Gnome without problems", instead have the resiliency to work with a simpler product like Xfce, or better still LXDE, which absolutely never fails or acts in an unexpected manner.

Whether you work on BSD, Linux, Mac or Windows, without sufficient resiliency you'll soon be a disgruntled user. On the other hand, with abounding resiliency, you'll be happy anywhere you go.

As a Matter of Fact, You ARE Smart Enough

"I can't configure my computer without GUI assistants. I'm not a programmer!"

Steve Jobs and Bill Gates did their job well: They brainwashed a big part of the population into thinking only Geek geniuses can edit config files and run commands. Jobs' and Gates' brainwashing is all the more impressive given that the Apple II and MS-Dos ran from the command line, with millions of secretaries, business owners, lawyers, accounting people, and students getting lots of work done on that command line. When they came out with Mac and Windows, respectively, they brought out a propaganda (scuse me, advertising) campaign to convince us we were too stupid to work with the command line, and a lot of us bought in.

And now a lot of people believe they're too stupid to edit a few config files or run a few commands. Sadly, instead of recognizing this as a mental block bought and paid for by some megacorporations, they believe it to be some sort of law of the universe, like gravitation. "Objects accelerate toward earth at 9.8 meters/second2 and I'm too stupid to configure my computer without GUI!"

When you meet someone with this mental block, explain that non-GUI configuration is a process:

  1. Web search to make sure this software can be configured at the config file level. Most simple software can.
  2. Web search to locate the config files for this software.
  3. Make a backup of every one of the config files for this software.
  4. Understand that if some config files are under your home directory and others are under /etc, you probably want to change the ones under your home directory.
  5. Web search and experiment to find how to change the config files.
  6. Restart the software to see the result.
  7. Continue changing config files and restarting the software until you achieve what you wanted.
  8. Write down what you did, so next time it will take one minute instead of an hour.

The last item on the preceding list sounds pretty daunting: An hour to make one config change? Don't worry, that doesn't last long. As you build up your repertoire of config tweaks, most will be less than a minute. Such a tweak on simple software (Openbox, for instance) could end up taking less time than searching the equivalent GUI config change on complexified software like Gnome. And sometimes, you get the best of both worlds. With Openbox, for instance, it has a GUI configurator called obconf capable of doing some, but not all, configuration. You can use obconf to change something, then do a directory listing to find out what file just changed, search in that file for the change you made, and understand how the config file works. It's doable, and gets much easier over time.

Bottom Line

Bottom line: Don't use software you hate. Find alternatives, and if those alternatives seem inconvenient, have the resiliency to get past the inconvenience. Remember that the vast majority of high school graduates are smart enough to administer their computers without GUI translators.

If bloatware is doing right by you, enjoy it. If it becomes a problem, find an alternative, and help others follow in your footsteps. When you meet that guy, who prefers griping about his software to finding alternatives, tell him this: "If you're not part of the solution, you're part of the problem!

[ Training | Troubleshooters.Com | Email Steve Litt ]