[{"data":1,"prerenderedAt":241},["ShallowReactive",2],{"\u002Fen\u002Fpost\u002F2018\u002F10\u002Finsights-from-a-perfectionist-about-over-engineering":3},{"id":4,"title":5,"author":6,"body":7,"createdAt":229,"description":230,"extension":231,"meta":232,"navigation":233,"path":234,"seo":235,"slug":6,"stem":236,"tags":237,"__hash__":240},"posts\u002Fposts\u002F2018\u002F10\u002Finsights-from-a-perfectionist-about-over-engineering.md","Insights from a perfectionist about Over-Engineering",null,{"type":8,"value":9,"toc":220},"minimark",[10,15,23,27,30,33,36,52,62,75,87,91,94,100,132,144,147,156,159,162,165,169,175,178,191,195,198,203,206,214,217],[11,12,14],"h2",{"id":13},"foreword","Foreword",[16,17,19],"callout",{"type":18},"note",[20,21,22],"p",{},"This article is an open letter for me to keep reminding myself about what\nto prioritize when developing software, I am as much of a sinner in this aspect\nas the next person.",[11,24,26],{"id":25},"intro","Intro",[20,28,29],{},"We as software engineers are always trying to do our best when it comes to being\ninnovative, improving our systems to work better and faster, perhaps with a\nbetter design, or a more comprehensive codebase. We all have some preference\nwhen it comes to doing our bests which we try to achieve at all times.",[20,31,32],{},"The main drive of this motivation is our necessity as \"digital craftspeople\" to\nexpress ourselves through quality work, along with the personal realization we\nfeel by doing a great job, with great quality, that challenges us and takes\nourselves further. It's motivating, isn't it? Assuming risks and getting out of\nthe comfort zone is incredibly funny, our brain's reward system goes crazy with\nunpredictability.",[20,34,35],{},"To help achieve that challenge, innovation, and quality in Software Engineering\nwe usually think that we need the best tools available, so we'll have fewer\nthings to worry about, and can concentrate our efforts in the process of\ncreating great products. On top of that, having the best tools could improve our\nquality of life (allowing us not to work under pressure, avoids overwork, and\nalso helps us to sleep better at night). Furthermore, \"the right set of tools\"\ncould even enhance our productivity through self-satisfaction with work,\neveryone has their preoccupations and is willing to create something to be proud\nof.",[20,37,38,39,43,44,51],{},"Many times during the design and development of products we take unmeasured\nsolutions for a simple problem. After all, we want to have not just the right\nset of tools, but the ",[40,41,42],"strong",{},"best"," right? How can we be ground-breaking, innovative,\ndisruptive, and pick-your-buzzword-poison otherwise? Well, as Nathan Marz\n(creator of Apache Storm) puts better in his ",[45,46,50],"a",{"href":47,"rel":48},"http:\u002F\u002Fnathanmarz.com\u002Fblog\u002Fsuffering-oriented-programming.html",[49],"nofollow","suffering-oriented programming",":",[53,54,55],"blockquote",{},[20,56,57,61],{},[58,59,60],"span",{},"…"," don't build technology unless you feel the pain of not having it. It applies\nto the big, architectural decisions as well as the smaller everyday programming\ndecisions. Suffering-oriented programming greatly reduces risk by ensuring that\nyou're always working on something important, and it ensures that you are\nwell-versed in a problem space before attempting a large investment.",[20,63,64,65,70,71,74],{},"This method describes a good way to think about LEAN and evolving products\nthrough ",[45,66,69],{"href":67,"rel":68},"https:\u002F\u002Fdzone.com\u002Farticles\u002Fwhat-is-minimum-viable-product-and-how-to-build-it",[49],"an MVP concept",", helping to keep track of what ",[40,72,73],{},"really"," matters when it\ncomes to a good balance between Product and Engineering efforts.",[20,76,77,78,81,82,86],{},"As we're daily overfed with information, it's easy to make mistakes trying to\nchoose the right set of tools to work with. From picking Frameworks to Operating\nSystems, and even the cloud provider to host our systems and products. It's OK\nto make mistakes, we all have a great first impression about all choices we\ncould have done, if you read AWS or GCP documentation you'll be impressed with\ntheir magical solutions to your problems, where you can just throw everything in\n(including your credit card), and everything will be fine, right? The magic\ncloud will solve ",[40,79,80],{},"all of your"," problems. Yeah, ",[83,84,85],"em",{},"maybe",".",[11,88,90],{"id":89},"what-is-the-problem-i-am-trying-to-solvehere","What is the problem I am trying to solve here?",[20,92,93],{},"One good example of the current hype, when it comes to applications is Docker\ncontainers and Kubernetes. Kubernetes is the open-source version of Google's\nBorg, a great Linux containers orchestration tool developed to orchestrate\napplications on Google's data center.",[20,95,96,97,86],{},"Kubernetes is great, but the hype goes too far sometimes with companies running\neven Production transactional databases on it, as well as entire monoliths and\nStateful services. At this point, we have to look back and ask ourselves: \"What\nproblem I'm trying to solve here?\". Because, if you take a second look, these\ndecisions are kind of a \"Hydra\" solution, \"for every head chopped off, the Hydra\nwould regrow two heads\", or even better: these solutions are creating more\nproblems, by trying to solve problems ",[40,98,99],{},"that may not even exist",[20,101,102,103,108,109,114,115,118,119,122,123,128,129,51],{},"Yeah, Google orchestrated MySQL instance deployment using Borg. The first\n",[45,104,107],{"href":105,"rel":106},"https:\u002F\u002Fsre.google\u002Fsre-book\u002Fautomation-at-google\u002F",[49],"version (POC) was released in 2008 and finished by 2009"," at that time the revenue\nof the Ad service was ",[45,110,113],{"href":111,"rel":112},"https:\u002F\u002Fwww.statista.com\u002Fstatistics\u002F266249\u002Fadvertising-revenue-of-google\u002F",[49],"estimated at USD ~22.9 Bi",". Ask yourself, do your database\nserves a ",[40,116,117],{},"USD 22.9 BILLION service","? Do you ",[83,120,121],{},"really need"," orchestration there?\nChances are, and let's face it, ",[45,124,127],{"href":125,"rel":126},"https:\u002F\u002Fblog.bradfieldcs.com\u002Fyou-are-not-google-84912cf44afb",[49],"You Are Not Google",". This is an extreme example\nbut it serves to illustrate the main concept of ",[83,130,131],{},"suffering-oriented\nprogramming",[53,133,134],{},[20,135,136,139,140,143],{},[58,137,138],{},"..."," don't build technology unless you ",[40,141,142],{},"feel the pain"," of not having it.",[20,145,146],{},"A nice quote from \"You Are Not Google\" to sink in:",[53,148,149],{},[20,150,151,152,155],{},"Don’t even start considering solutions until you ",[40,153,154],{},"understand"," the problem. Your\ngoal should be to “solve” the problem mostly within the problem domain, not the\nsolution domain.",[20,157,158],{},"Otherwise, in case we insist on the inappropriate (not necessarily wrong)\nsolution, we're going to spend some extra time dealing with the consequences\n(i.e. chopping additional Hydra heads). Worth noting that dealing with the\nconsequences is not something bad, as long as you have the resources (time and\nmoney) to invest into learning and rework, investing some resources into\ninappropriate software solutions could even be seen as a way of training with\nhigher outcomes (learnings) than conferences, courses, and books. There is a lot\nof lessons and knowledge to be extracted from these experiments.",[20,160,161],{},"Learning from our experiences is the only path to success, and failures teach\nbest. Failures were also the motivation for writing this article to keep\nreminding myself (:",[20,163,164],{},"As Software Engineers the problem space analysis oftentimes fail due to an\nunderrated aspect, mostly unnoticed: on the other side of the line is a user of\nthis software.",[11,166,168],{"id":167},"and-guesswhat","And guess what?",[20,170,171,172,86],{},"He doesn't care if you're running Elixir inside a container on Kubernetes, using\nContainer OS or Core OS, which you provisioned with your bare hands, and have\npolished bit by bit to be XYZ ms faster than the Vanilla version. As long as you\nrespond to their requests, and ",[40,173,174],{},"don't break things",[20,176,177],{},"Innovation has nothing to do with the fact that you want to use cutting-edge\ntechnology, and it's not about how fast you spend money on those solutions\neither. It's about delivering value to your customers, and enrich their\nexperience from the interactions with your product.",[20,179,180,181,183,184,187,188,86],{},"If you're going through some orchestration problems, having 10+ micro-services\nwith some asynchronous task-based workers (e.g. Python's Celery). Then, ",[83,182,85],{},"\nit's time to use Kubernetes. But, as an engineer you should know that the best\npath is to put some solutions on the table, run some benchmarks and compare\nthem, so you'll have data to help in your decision, and choose what's the right\nsolution for your problem, ",[40,185,186],{},"at the right time",". We just have to keep asking\nourselves: ",[83,189,190],{},"\"What is the problem I'm trying to solve here?\"",[11,192,194],{"id":193},"conclusion","Conclusion",[20,196,197],{},"There's a quote from a great investor called Benjamin Graham that says:",[53,199,200],{},[20,201,202],{},"If you are looking for investments*, choose them the way you would buy\ngroceries, not the way you would buy perfume.\n-- Graham, The Intelligent Investor (1973)",[20,204,205],{},"We should carefully look to where we're going with our choices. So we don't\noverspend and keep things going for more time, thus we go further.",[16,207,208],{"type":18},[20,209,210,211,213],{},"The original text is: \"If you are shopping for common stocks ",[58,212,138],{},"\". But, as\na Software Engineer, I just switched the syntax so we could adapt it to more\nuse-cases (:",[20,215,216],{},"I learned from my own experience that over-engineered decisions end up bringing\nmore pain than solving problems, and it currently happens through early\nimprovements on the system, timing really matters. Many times we try to solve\nall problems at once (even those we don't have), and it brings more problems,\nlike high costs of maintenance and infrastructure, or under-utilization of the\nresources.",[20,218,219],{},"Sooner or later, the Over-Engineering bill will come as Hydra heads keep growing\nup accumulating technical debt. Be mindful when analyzing the problem space,\npick the right tool for the job that eases your real pain (not the imaginary\none).",{"title":221,"searchDepth":222,"depth":222,"links":223},"",2,[224,225,226,227,228],{"id":13,"depth":222,"text":14},{"id":25,"depth":222,"text":26},{"id":89,"depth":222,"text":90},{"id":167,"depth":222,"text":168},{"id":193,"depth":222,"text":194},"2018-10-28T00:00:00+02:00","Software engineers are always trying to do their best when it comes to being innovative and improving their systems. This article helps to put that willingness into perspective and drive it in the right direction.","md",{},true,"\u002Fposts\u002F2018\u002F10\u002Finsights-from-a-perfectionist-about-over-engineering",{"title":5,"description":230},"posts\u002F2018\u002F10\u002Finsights-from-a-perfectionist-about-over-engineering",[238,239],"product engineering","development","yTzIm_ItOjQ74WTmohaj0WtSJXvhgVpjsO_xMEaYALg",1778441744238]