Accidental Complexity

July 05, 2021

Most modern software applications have no shortage of complexity. As components are split from each other to take advantage of distributed architectures, we find that the scope of uncertainty and possibility increases by orders of magnitude.

This is exacerbated by the need to drive distributed architectures in order to achieve extreme scale to meet the demands of a vague storied requirement: “information at our fingertips in the blink of an eye”.

In order to support the Mercurian (Fred that is.) demands of “we want it all, we want it now”, the specialization of characteristics necessitated by architectural components becomes more heterogeneous. Evolutionarily speaking, the distribution of components only grows through the adolescence of technological innovation, and with it grows the complexity.

There are two types of complexity:

  • Necessary Complexity
  • Accidental Complexity

Necessary Complexity

Necessary complexity is the minimal complexity required to solve the problem that has been presented. For instance, if I’m going to sort elements within a collection, I have to evaluate every element in the collection. In terms of time complexity, I can never have a solution that is “less” than O(N).

This particular example is fairly black and white. However, practically speaking, there are many problem domains that aren’t as easy to evaluate. There are circumstances like the Traveling Salesman, or the Three Generals problem where either the inputs are too great to calculate, or there are Byzantine circumstances, et al. In addition to the problem itself, perhaps there are requirements concerning how the problem is solved. For instance, there may be a need to have sufficient parallelism, such that algorithms that might provide optimal performance in a single-threaded solution wouldn’t be as viable.

Let’s try to clarify the definition of necessary complexity in light of the weeds I’ve just mentioned.

Implementation details are rarely specified by architects unless they are a critical aspect of the architecture. If I’m designing an eCommerce site, the implementation details of searching through content is more than likely going to be left to the development teams. However, If I’m designing a search engine, I’m probably going to be more involved in the search algorithms and implementation. This means that complexity is driven by practical necessity. We don’t need to labor over comparing every possible algorithm if we know that certain industry standards satisfy both the reasonable requirements of non-critical characteristics, as well as meet the demands dictated by the critical ones.

It is extremely rare that we will need to challenge the performance or attributes of existing algorithms. For the most part, software algorithms have reached a stage of maturity, such that optimization creeps forward at an incredibly slow pace. For now, most performance gains come from the physical components driven by software instructions, as opposed to the algorithms themselves.

Despite what might be seen as a limitiation, there are advantages to this maturity. As algorithms become more stable, they become communication devices. These algorithms, data structures and patterns are easier to study and understand. This allows the architecture of software to be disseminated in an effective way within and across engineering organizations. This approaches a collective comprehensiveness to understanding, which baselines the simplicity of the architectural solution.

All else being equal, necessary complexity is the minimum amount of complexity required to solve the problem based on the available technology and resources at the time the problem needs to be addressed, while allowing the overall design and architecture to be understood by those who will implement it.

Accidental Complexity

Accidental complexity is noise. It is every aspect of the solution that makes it harder to understand, implement, delivery or otherwise coalesce with the original intent.

Ideally, every dollar spent and minute dedicated to the end goal would be constrained to solving the problem at hand. Unfortunately, this is impossible.

In order to validate that the solution is going to address the problem, we have to create tests. We have unit tests to ensure that the code actually works. We have acceptance tests that ensure that we are bound to acceptance criteria.

Over the years, new paradigms have emerged to simplify testing. Behavior Driven Development (BDD) simplifies the syntax of acceptance tests so that the tests are constructed in language semantics similar to business requirements. Monitor Driven Development (MDD) provides a mechanism to continuously test a solution to ensure that it holistically and continuously fulfills the desired end goal. This provides temporal and aggregate dimensions to validation.

Beyond testing, there are administrative and support factors such as monitoring, logging, admin access, et al. There are the stages and tools involved with release, delivery and/or deployment of the software. There are even the evolutionary fitness functions used to ensure that development adheres to architectural constraints and guidelines.

Entire frameworks have been written to support the simplified paradigms. Much of the DevOps cultural phenomenon is focused on providing tools and augmentations that help reduce the noise generated by accidental complexity.

One might argue that accidental complexity, in totality, is unavoidable. “What does it matter where the logic exists to support my solution?”

It matters in terms of abstraction. Referring back to the nature of growing complexity as systems become more distributed, we have mechanisms to abate the complexity in the design of the software itself. API-driven development ensures that each module, service, or bounded context is encapsulated in a manner that external consumers interact only with the nouns and verbs of the API. They only need to be concerned with the “what”. The “how” is the responsibility of the curators of the service abstracted by the API.

APIs provide a considerable amount of simplicity by hiding the unruly details. As a consumer of an API, this allows me to budget my focus in greater degree to the problem I’m trying to solve. I’ll spend less time context-switching to alien component implemention, which means I will spend less overall time delivering my piece of the overall solution. This improves my own productivity, decreases the time to deliver, inversely increasing the velocity of the release cycle. Coincident to the economy of time is the associative decrease in cost.

Abstraction, as a function of work and day-to-day operations, consolidates the cost of delivering product to the hands of customers.

Most of this is pretty intuitive. While there are thousands of pages in the form of books, articles and blogs written in support of these ideas, one can come to the same conclusion with a cursory understanding of software development life cycles, business and money management.

Unfortunately, it is far more common to see companies negatively impacted by accidental complexity than to see them flourish with lean processes. In my own experience, failure to right the ship comes from a healthy amount of ignorance and resistance to change. This is often driven by momentum.

Stop me if you've heard this before: 

    "We don't have the time to fix it."

Or this: 

    "We've sunk a lot of money and time into this solution."

and so on...

Addressing the Accident

If you expect there to be a one-size-fits-all solution, or a “quicker picker upper”, then you haven’t spent very long in the software game. It just doesn’t work that way. There is no golden hammer.

However, in lieu of a skeleton key solution, a procedural framework exists. First, we have to consider what causes accidental complexity.

In many cases, I’ve found that companies opt to build their own test frameworks or tooling. This isn’t a problem unless the company skips the evaluation stage. If my business is eCommerce, at first glance, it doesn’t make much sense for me to build my own release and deployment pipeline.

Typically, it makes sense to create a formal evaluation of the existing solutions, and score or weight those comparisons against the business requirements.

In most cases, off-the-shelf software meets fundamental requirements. This is a considerable savings in terms of time to market, supportability and cost (especially if you choose open source solutions.)

There is no golden hammer.

OTS solutions are great, but like anything else, they have limitations. Truly open source solutions tend to have limited, community-driven support, and there is often a hard ceiling concerning the supported scale of the solution. This is often overlooked in evaluations. We must always look ahead. If the business expects or targets a given scale, then this should always be a factor in our evaluations. At some point, we may have to change solutions, provide integration or customization code, or provide a do-it-yourself (DIY) solution.

So-called “Enterprise” OTS solutions tend to be pay-to-play versions of open source software. Subscribing to these services often includes an extended feature set not available to open source/community versions, as well as support contracts. The cost of these solutions is usually the primary focus during evaluations, but I recommend looking deeper. Get on the phone and talk to someone. Watch a demo of the extended features. Research the support experience.

I’ve dealt with vendors whose enterprise solutions were absolutely phenomenal and worth every penny. At the same time, I’ve dealt with vendors whose extended feature set could easily be provided by locally developed integrations, and whose support experience was inconceivably poor.

Sometimes there aren’t going to be available tools, the existing tools aren’t going to meet your needs, or your requirements will conflict with what is out there. (You also might be in direct competition with the tools!)

In these cases, building your own, or some hybrid solution of build and buy is required.

This is ok. Flexible architectures are a necessity. Technology changes at an alarming rate. Brittle designs that are intended to “stand the test of time” do so more often than not at great expense to the developers and the users. The best architectures are those that can evolve in a manner that is as painless and transparent as possible to the end users, while being cost effective and uneventful for the developers and architects who deliver it.

Evolutionary architectures allow accidental complexity to be addressed in a temporally flexible fashion. What is good to day, might not be tomorrow. If we are continuously testing and measuring the system, we will begin to see the stress points long before strain grows to failure. This allows us to navigate the complexity of our solution intelligently with thoughtful intent, minimizing the accidental nature of the complexity of our system.

Before I sail off into the wild blue yonder, I want to emphasize the term “accidental complexity”. Specifically, I want to focus on the word “accidental”. While ignorance and change resistance are problems to be solved in any organization, they aren’t malicious problems. There are many reasons that organizations fall into these patterns, most of which are entirely valid. As of this writing, software is still created by people. We are fallable, funny creatures. If we attempt to solve accidental complexity by treating it as willful misconduct or intended slight, we’re more likely to exacerbate the problem.

Any attempt to rectify challenges in our operational models must be done with compassion and a mindset of inclusion and collaboration. It was “just an accident”.

Just like spilled milk, we’ll clean it up and pour a new glass.


Profile picture

Written by Ed Mangini
A Technology Blog about useful stuff.