Insights from a perfectionist about Over-Engineering

Foreword

Intro

We as software engineers are always trying to do our best when it comes to being innovative, improving our systems to work better and faster, perhaps with a better design, or a more comprehensive codebase. We all have some preference when it comes to doing our bests which we try to achieve at all times.

The main drive of this motivation is our necessity as "digital craftspeople" to express ourselves through quality work, along with the personal realization we feel by doing a great job, with great quality, that challenges us and takes ourselves further. It's motivating, isn't it? Assuming risks and getting out of the comfort zone is incredibly funny, our brain's reward system goes crazy with unpredictability.

To help achieve that challenge, innovation, and quality in Software Engineering we usually think that we need the best tools available, so we'll have fewer things to worry about, and can concentrate our efforts in the process of creating great products. On top of that, having the best tools could improve our quality of life (allowing us not to work under pressure, avoids overwork, and also helps us to sleep better at night). Furthermore, "the right set of tools" could even enhance our productivity through self-satisfaction with work, everyone has their preoccupations and is willing to create something to be proud of.

Many times during the design and development of products we take unmeasured solutions for a simple problem. After all, we want to have not just the right set of tools, but the best right? How can we be ground-breaking, innovative, disruptive, and pick-your-buzzword-poison otherwise? Well, as Nathan Marz (creator of Apache Storm) puts better in his suffering-oriented programming:

… don't build technology unless you feel the pain of not having it. It applies to the big, architectural decisions as well as the smaller everyday programming decisions. Suffering-oriented programming greatly reduces risk by ensuring that you're always working on something important, and it ensures that you are well-versed in a problem space before attempting a large investment.

This method describes a good way to think about LEAN and evolving products through an MVP concept, helping to keep track of what really matters when it comes to a good balance between Product and Engineering efforts.

As we're daily overfed with information, it's easy to make mistakes trying to choose the right set of tools to work with. From picking Frameworks to Operating Systems, and even the cloud provider to host our systems and products. It's OK to make mistakes, we all have a great first impression about all choices we could have done, if you read AWS or GCP documentation you'll be impressed with their magical solutions to your problems, where you can just throw everything in (including your credit card), and everything will be fine, right? The magic cloud will solve all of your problems. Yeah, maybe.

What is the problem I am trying to solve here?

One good example of the current hype, when it comes to applications is Docker containers and Kubernetes. Kubernetes is the open-source version of Google's Borg, a great Linux containers orchestration tool developed to orchestrate applications on Google's data center.

Kubernetes is great, but the hype goes too far sometimes with companies running even Production transactional databases on it, as well as entire monoliths and Stateful services. At this point, we have to look back and ask ourselves: "What problem I'm trying to solve here?". Because, if you take a second look, these decisions are kind of a "Hydra" solution, "for every head chopped off, the Hydra would regrow two heads", or even better: these solutions are creating more problems, by trying to solve problems that may not even exist.

Yeah, Google orchestrated MySQL instance deployment using Borg. The first version (POC) was released in 2008 and finished by 2009 at that time the revenue of the Ad service was estimated at USD ~22.9 Bi. Ask yourself, do your database serves a USD 22.9 BILLION service? Do you really need orchestration there? Chances are, and let's face it, You Are Not Google. This is an extreme example but it serves to illustrate the main concept of suffering-oriented programming:

... don't build technology unless you feel the pain of not having it.

A nice quote from "You Are Not Google" to sink in:

Don’t even start considering solutions until you understand the problem. Your goal should be to “solve” the problem mostly within the problem domain, not the solution domain.

Otherwise, in case we insist on the inappropriate (not necessarily wrong) solution, we're going to spend some extra time dealing with the consequences (i.e. chopping additional Hydra heads). Worth noting that dealing with the consequences is not something bad, as long as you have the resources (time and money) to invest into learning and rework, investing some resources into inappropriate software solutions could even be seen as a way of training with higher outcomes (learnings) than conferences, courses, and books. There is a lot of lessons and knowledge to be extracted from these experiments.

Learning from our experiences is the only path to success, and failures teach best. Failures were also the motivation for writing this article to keep reminding myself (:

As Software Engineers the problem space analysis oftentimes fail due to an underrated aspect, mostly unnoticed: on the other side of the line is a user of this software.

And guess what?

He doesn't care if you're running Elixir inside a container on Kubernetes, using Container OS or Core OS, which you provisioned with your bare hands, and have polished bit by bit to be XYZ ms faster than the Vanilla version. As long as you respond to their requests, and don't break things.

Innovation has nothing to do with the fact that you want to use cutting-edge technology, and it's not about how fast you spend money on those solutions either. It's about delivering value to your customers, and enrich their experience from the interactions with your product.

If you're going through some orchestration problems, having 10+ micro-services with some asynchronous task-based workers (e.g. Python's Celery). Then, maybe it's time to use Kubernetes. But, as an engineer you should know that the best path is to put some solutions on the table, run some benchmarks and compare them, so you'll have data to help in your decision, and choose what's the right solution for your problem, at the right time. We just have to keep asking ourselves: "What is the problem I'm trying to solve here?".

Conclusion

There's a quote from a great investor called Benjamin Graham that says:

If you are looking for investments*, choose them the way you would buy groceries, not the way you would buy perfume. -- Graham, The Intelligent Investor (1973)

We should carefully look to where we're going with our choices. So we don't overspend and keep things going for more time, thus we go further.

I learned from my own experience that over-engineered decisions end up bringing more pain than solving problems, and it currently happens through early improvements on the system, timing really matters. Many times we try to solve all problems at once (even those we don't have), and it brings more problems, like high costs of maintenance and infrastructure, or under-utilization of the resources.

Sooner or later, the Over-Engineering bill will come as Hydra heads keep growing up accumulating technical debt. Be mindful when analyzing the problem space, pick the right tool for the job that eases your real pain (not the imaginary one).