I always find myself sitting down to read Out of the Tar Pit[0] at least a couple times per year. It has been—and continues to be—one of the most seminal texts in my career. I still remember the first time I read the following passage on complexity, and how it just turned on all the mental light bulbs:
>> Essential Complexity is inherent in, and the essence of, the problem (as seen by the users).
>> Accidental Complexity is all the rest — complexity with which the development team would not have to deal in the ideal world (e.g. complexity arising from performance issues and from suboptimal language and infrastructure).
>> Note that the definition of essential is deliberately more strict than common usage. Specifically when we use the term essential we will mean strictly essential to the users’ problem (as opposed to — perhaps — essential to some specific, implemented, system, or even — essential to software in general).
The best skill I've learned, and continued to practice and improve, is the ability to strip how we talk about problems we want to solve with software down to what's truly essential to the problem. Making a habit of doing so helps clarify the contours of the problem itself, and improves discussions around solving because the boundaries of what's truly essential become clear—and then everyone involved knows that every choice we make from that point is additional, accidental complexity we are adding to the problem ourselves.
Far too often I have seen even greenfield software quickly ratchet up the overall complexity because the people making choices don't take the time to really isolate the problem from the software—but instead frame the problem within the context of languages, frameworks, architecture, infrastructure, and so on, and then just start slinging code at the problem.
If you haven't yet read Into the Tar Pit, it truly changed the way I look at and think about software and problem complexity. You may find value in it, as well.
This rings so true. I noticed a consistent level-up in my abilities once I started to seek the essence of the problem. I ask myself: “I start with this information. The desired output is X. What is the essence of the data transformation that takes me from the input to X?”
When I boil down the task to its nature as a data transformation, the solution flows from my understanding of the problem, and I’ve found that my choice of tools flows transitively from there pretty easily. The problem is “isolated” from the software as you said which makes it so much easier to reason about things.
I sadly have not gotten much traction when I try and advocate for this mindset in our industry.
As an aside: It reminds me of a funny point from secondary education. Did you take AP tests in high school? If you did, you might remember as I do a consistent refrain that teachers used to beat into students preparing for the tests: “Answer the question” Over and over we heard this ad nauseam until it became second nature, whether for AP English or AP Physics - and it was good advice! Because the number one mistake students make on those exams is not actually answering the question asked, which even when couched in the most wonderful prose, results in a failing mark.
I think software engineering is often pretty similar. Even the best, most sophisticated tools will not produce a working solution if you don’t understand the problem.
> I sadly have not gotten much traction when I try and advocate for this mindset in our industry.
Yeah, I know what you mean. It's become a bit of a primary signal for how I evaluate a company's engineering culture. I've been lucky to work with some fantastic people who really get it, and I've also struggled and suffered through working with those who do not.
> Even the best, most sophisticated tools will not produce a working solution if you don’t understand the problem.
I'm sure we've all seen some awful monstrosities—or created them ourselves—that we could call a technically working solution to a given problem ... but it doesn't mean anyone wants to work on it. Keeping complexity at bay requires finding the simplest solutions that isolate essential and accidental complexity. Simplicity is hard, and it requires doing this well, constantly. It is [ahem] essential to spend the time required to isolate the problem and articulate it clearly. If you can't isolate and articulate the problem without referencing your tech stack and tooling, or your explanation gets all muddy and convoluted, you haven't actually identified the essential complexity of a problem. You're still stuck in accidental complexity territory. And that's a horrible place to be designing and architecting your software from.
It's also critical to note that over the lifetime of any piece of software, as new things come up—new bugs, new features, etc—you have to keep re-engaging the same process, and evaluating/reflecting on how new things fit (or don't!) within your existing architecture/design. Failing to do so is what drives toward infinite complexity and endless "tech debt" in poorly designed software. Well-designed software isolates and encapsulates all the accidental complexity into its own space(s), leaving open avenues to adjust and expand the software. Well-designed interfaces allow you to refactor, reshape, and grow the internals of a problem domain in isolation from its callers. This requires discipline from a software team—and its leadership—to take the time necessary to adjust and refactor as priors change. Such work should always be moving the needle toward greater team velocity and individual productivity.
> Did you take AP tests in high school?
Yep, sure did! I definitely remember what you're describing here.
> If you can't isolate and articulate the problem without referencing your tech stack and tooling, or your explanation gets all muddy and convoluted, you haven't actually identified the essential complexity of a problem
100%. I don't have much to add but I've really enjoyed our discussion.
>> Essential Complexity is inherent in, and the essence of, the problem (as seen by the users).
>> Accidental Complexity is all the rest — complexity with which the development team would not have to deal in the ideal world (e.g. complexity arising from performance issues and from suboptimal language and infrastructure).
>> Note that the definition of essential is deliberately more strict than common usage. Specifically when we use the term essential we will mean strictly essential to the users’ problem (as opposed to — perhaps — essential to some specific, implemented, system, or even — essential to software in general).
The best skill I've learned, and continued to practice and improve, is the ability to strip how we talk about problems we want to solve with software down to what's truly essential to the problem. Making a habit of doing so helps clarify the contours of the problem itself, and improves discussions around solving because the boundaries of what's truly essential become clear—and then everyone involved knows that every choice we make from that point is additional, accidental complexity we are adding to the problem ourselves.
Far too often I have seen even greenfield software quickly ratchet up the overall complexity because the people making choices don't take the time to really isolate the problem from the software—but instead frame the problem within the context of languages, frameworks, architecture, infrastructure, and so on, and then just start slinging code at the problem.
If you haven't yet read Into the Tar Pit, it truly changed the way I look at and think about software and problem complexity. You may find value in it, as well.
[0]: https://curtclifton.net/papers/MoseleyMarks06a.pdf