[{"data":1,"prerenderedAt":457},["ShallowReactive",2],{"\u002Fen\u002Fpost\u002F2021\u002F08\u002Freal-life-terraform-refactoring-guide":3},{"id":4,"title":5,"author":6,"body":7,"createdAt":444,"description":445,"extension":446,"meta":447,"navigation":448,"path":449,"seo":450,"slug":451,"stem":452,"tags":453,"__hash__":456},"posts\u002Fposts\u002F2021\u002F08\u002Freal-life-terraform-refactoring-guide.md","Real-life Terraform Refactoring Guide",null,{"type":8,"value":9,"toc":433},"minimark",[10,15,33,36,45,51,54,58,80,89,101,104,123,127,130,140,147,150,161,168,177,183,191,194,206,211,218,226,230,233,239,243,246,252,256,266,272,276,283,293,299,311,314,331,337,341,353,370],[11,12,14],"h2",{"id":13},"intro","Intro",[16,17,18,19,26,27,32],"p",{},"As reality hits, the unavoidable fact of dealing with a hard-to-manage Terraform\n",[20,21,25],"a",{"href":22,"rel":23},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FBig%5Fball%5Fof%5Fmud",[24],"nofollow","Big ball of mud"," code base comes in. There is no way around natural growth and\nevolution of code bases and the design flaws that come with it. Our Agile\nmindset is to ",[20,28,31],{"href":29,"rel":30},"https:\u002F\u002Fwww.brainyquote.com\u002Fquotes\u002Fmark%5Fzuckerberg%5F453439",[24],"\"move fast and break things\"",", implement something as simple as\npossible and let the design decisions for the next iterations (if any).",[16,34,35],{},"Refactoring Terraform code is actually as natural as developing it, time and\ntime again you will be faced with a situation where a better structure or\norganization can be achieved, maybe you want to upgrade from a home-made module\nto an open-source\u002Fcommunity alternative, maybe you just want to segregate your\nresources into different states to speed-up development. Regardless of the goal,\nonce you get into it, you will realize that Terraform code refactoring is\nactually a basic missing step on the development process that no one told you\nbefore.",[16,37,38,39,44],{},"As the ",[20,40,43],{"href":41,"rel":42},"http:\u002F\u002Fnathanmarz.com\u002Fblog\u002Fsuffering-oriented-programming.html",[24],"Suffering-Oriented Programming"," mantra dictates:",[46,47,48],"blockquote",{},[16,49,50],{},"\"First make it possible. Then make it beautiful. Then make it fast.\"",[16,52,53],{},"So, time to make the Terraform code beautiful!",[11,55,57],{"id":56},"how-to-break-a-big-ball-of-mud-strangle-it","How to break a big ball of mud? STRANGLE IT",[16,59,60,64,65,68,69,74,75,79],{},[61,62,63],"code",{},"\u003Cjoke>"," Martin Fowler has already written everything there is to write about\n(early 2000s) DevOps, Agile, and Software Development. Therefore, we could\nreference Martin Fowler for virtually anything Software related ",[61,66,67],{},"\u003C\u002Fjoke>",", but\nreally, the ",[20,70,73],{"href":71,"rel":72},"https:\u002F\u002Fmartinfowler.com\u002Fbooks\u002Frefactoring.html",[24],"Refactoring book"," is ",[76,77,78],"strong",{},"THE"," reference on this subject.",[16,81,82,83,88],{},"Martin Fowler shared the ",[20,84,87],{"href":85,"rel":86},"https:\u002F\u002Fmartinfowler.com\u002Fbliki\u002FStranglerFigApplication.html",[24],"Stangler (Fig) Pattern",", which describes a strategy to\nrefactor a legacy code base by re-implementing the same features (sometimes even\nthe bugs) on another application.",[46,90,91,98],{},[16,92,93,97],{},[94,95,96],"span",{},"..."," the huge strangler figs. They seed in the upper branches of a tree and\ngradually work their way down the tree until they root in the soil. Over many\nyears they grow into fantastic and beautiful shapes, meanwhile strangling and\nkilling the tree that was their host.",[16,99,100],{},"This metaphor struck me as a way of describing a way of doing a rewrite of an\nimportant system.",[16,102,103],{},"In this document we are going to follow the same idea:",[105,106,107,117,120],"ol",{},[108,109,110,111,116],"li",{},"implement the same feature on a different ",[20,112,115],{"href":113,"rel":114},"https:\u002F\u002Fwww.terraform-best-practices.com\u002Fkey-concepts#composition",[24],"Terraform composition",";",[108,118,119],{},"migrate the Terraform state;",[108,121,122],{},"delete (kill) the previous implementation.",[11,124,126],{"id":125},"the-mono-repository-monorepo-approach-to-legacy","The mono-repository (monorepo) approach to Legacy",[16,128,129],{},"Let's suppose that your Terraform code base is versioned in a single repository\n(a.k.a. monorepo), following the random structure displayed below (just to help\nillustrate)",[131,132,137],"pre",{"className":133,"code":135,"language":136},[134],"language-text",".\n├── modules\u002F    # Definition of TF modules used by underlying compositions\n├── global\u002F     # Resources that aren't restricted to one environment\n│   ├── aws\u002F\n├── production\u002F # Production environment resources\n│   └── aws\u002F\n└── staging\u002F    # Staging environment resources\n    └── aws\u002F\n","text",[61,138,135],{"__ignoreMap":139},"",[16,141,142,143,146],{},"On this example each directory corresponds to a Terraform state. In order to\napply changes you have to walk to a path and execute ",[61,144,145],{},"terraform",".",[16,148,149],{},"The structure on this example repository was created a few hypothetical years\nago when the number of existing microservices and resources (DB, message queues,\netc) was significantly smaller. At the time, it was feasible to keep Terraform\ndefinitions together because it was easier to maintain, Cloud resources were\nmanaged with one-shot!",[16,151,152,153,156,157,160],{},"As the time went by, the number of Products and the team grew, and engineers\nstarted facing concurrency issues: Terraform lock executions on a shared storage\nwhen someone else is running ",[61,154,155],{},"terraform apply"," as well as a general slowness on\n",[76,158,159],{},"every execution"," since the number of data sources to sync is frightening.",[16,162,163,164,146],{},"A mono-repository approach is not necessarily bad, versioning is actually\nsimpler when performed in one single repository. Ideally, there won't be many\nchanges on the scale of GiB meaning that it is safe to proceed on this one ",[165,166,167],"em",{},"as\nlong as the Terraform remote states are divided",[169,170,172,173,176],"h3",{"id":171},"splitting-the-modules-sub-path-to-its-own-repository","Splitting the ",[61,174,175],{},"modules"," sub-path to its own repository",[16,178,179,180,182],{},"One thing to mention though is the ",[61,181,175],{}," sub-path, this one could be stored\nin a different git repository to leverage its own versioning. Since Terraform\nmodules and its implementations don't always evolve in the same pace, keeping\ntwo distinct version trees is beneficial. Additionally, a separated repository\nfor Terraform modules allows the specification of \"pinned versions\", e.g.:",[131,184,189],{"className":185,"code":187,"language":188,"meta":139},[186],"language-hcl","module \"aws_main_vpc\" {\n  source = \"git::https:\u002F\u002Fgithub.com\u002Fterraform-aws-modules\u002Fterraform-aws-vpc.git?ref=2ca733d\"\n  # Note the ref=${GIT_REVISION_DIGEST}\n}\n","hcl",[61,190,187],{"__ignoreMap":139},[16,192,193],{},"That reference for a module's version should always be specified, regardless if\nit comes from an internal\u002Fprivate repository or public. When you specify the\nversion, you are ensuring reproducibility.",[16,195,196,197,199,200,205],{},"Therefore, let's move the ",[61,198,175],{}," sub-path to another git repository,\nfollowing instructions from ",[20,201,204],{"href":202,"rel":203},"https:\u002F\u002Fstackoverflow.com\u002Fquestions\u002F359424\u002Fdetach-move-subdirectory-into-separate-git-repository\u002F17864475#17864475",[24],"this StackOverflow answer"," so that the git commit\nhistory is preserved:",[207,208,210],"h4",{"id":209},"_0","0.",[16,212,213,214,217],{},"Walk to the monorepo path and create a branch from the commits at\n",[61,215,216],{},"monorepo\u002Fmodules"," path",[131,219,224],{"className":220,"code":222,"language":223,"meta":139},[221],"language-bash","MAIN_BIGGER_REPO=\u002Fpath\u002Fto\u002Fthe\u002Fmonorepo\ncd \"${MAIN_BIGGER_REPO}\"\ngit subtree split -P modules -b refact-modules\n","bash",[61,225,222],{"__ignoreMap":139},[207,227,229],{"id":228},"_1","1.",[16,231,232],{},"Create the new repository",[131,234,237],{"className":235,"code":236,"language":223,"meta":139},[221],"mkdir \u002Fpath\u002Fto\u002Fthe\u002Fterraform-modules && cd $_\ngit init\ngit pull \"${MAIN_BIGGER_REPO}\" refact-modules\n",[61,238,236],{"__ignoreMap":139},[207,240,242],{"id":241},"_2","2.",[16,244,245],{},"Link the new repository to your remote Git (server)",[131,247,250],{"className":248,"code":249,"language":223,"meta":139},[221],"git remote add origin \u003Cgit@git.com:user\u002Fterraform-modules.git>\ngit push -u origin master\n",[61,251,249],{"__ignoreMap":139},[207,253,255],{"id":254},"_3","3.",[16,257,258,261,262,265],{},[94,259,260],{},"OPTIONAL"," Cleanup inside ",[61,263,264],{},"$MAIN_BIGGER_REPO",", if desired",[131,267,270],{"className":268,"code":269,"language":223,"meta":139},[221],"cd ${MAIN_BIGGER_REPO}\ngit rm -rf modules\ngit filter-branch --prune-empty \\\n    --tree-filter \"rm -rf modules\" -f HEAD\n",[61,271,269],{"__ignoreMap":139},[169,273,275],{"id":274},"lets-start-strangling-the-repository","Let's start strangling the repository",[16,277,278,279,282],{},"Now that a substantial piece of code was moved somewhere else, it is time to\nput the ",[20,280,87],{"href":85,"rel":281},[24]," in practice.",[16,284,285,286,289,290,292],{},"Move all the existing content as-is to the ",[61,287,288],{},"legacy"," sub-path, keeping the same\nrepository and change history (commits). It also allows applying the ",[61,291,288],{},"\ncode as it used to be from one of those paths.",[131,294,297],{"className":295,"code":296,"language":136},[134],".\n└── legacy\n    ├── global\n    │   └── aws\n    ├── production\n    │   └── aws\n    └── staging\n        └── aws\n",[61,298,296],{"__ignoreMap":139},[16,300,301,302,307,308,310],{},"Once the content is moved to legacy, the idea is to follow the ",[20,303,306],{"href":304,"rel":305},"https:\u002F\u002Fwww.oreilly.com\u002Flibrary\u002Fview\u002F97-things-every\u002F9780596809515\u002Fch08.html",[24],"Boy Scout rule","\nin order to strangle the ",[61,309,288],{}," content little by little (unless you are\nreally committed to migrating it all at once, which is going to be exhaustive).",[16,312,313],{},"The Boy Scout rule goes like:",[105,315,316,323,326],{},[108,317,318,319,116],{},"every time a task that involves deprecated code appears, we implement it on\n",[20,320,322],{"href":321},"..\u002Fstdout\u002Fblog\u002F2021\u002F03\u002Fterraform-best-practices.org","the new structure",[108,324,325],{},"import the Terraform state to keep the Cloud resources that a given code\nrepresents\u002Fdescribes;",[108,327,328,329,146],{},"remove the state and the code from ",[61,330,288],{},[16,332,333,334,336],{},"Until there is nothing left inside ",[61,335,288],{}," (or there are only unused\nresources\u002Fleft-behinds that could be destroyed\u002Fgarbage collected either way).",[207,338,340],{"id":339},"import-state-remove-state-and-code-from-what-where","Import state? Remove state and code from what? Where?",[16,342,343,344,347,348,146],{},"That will depend on the kind of resource we are migrating from the remote state,\non the bottom of each ",[61,345,346],{},"resource"," on Terraform's provider documentation you can\nfind a reference command to import existing resources into your Terraform code\nspecification. e.g.: ",[20,349,352],{"href":350,"rel":351},"https:\u002F\u002Fregistry.terraform.io\u002Fproviders\u002Fhashicorp\u002Faws\u002Flatest\u002Fdocs\u002Fresources\u002Fdb%5Finstance#import",[24],"AWS RDS DB instance",[16,354,355,356,359,360,365,366,369],{},"Suppose we want to replace the code of the AWS RDS Aurora defined in\n",[61,357,358],{},"production\u002Faws"," and then re-implement the same using ",[20,361,364],{"href":362,"rel":363},"https:\u002F\u002Fgithub.com\u002Fterraform-aws-modules\u002Fterraform-aws-rds-aurora",[24],"the community module",".\nAfter creating the corresponding sub-path to the monorepo according to your\npreference, provisioning the bucket and initializing the Terraform ",[61,367,368],{},"backend",":",[105,371,372,386,409],{},[108,373,374,375,379,380],{},"implement the definition of the community module\n",[20,376,378],{"href":362,"rel":377},[24],"github.com\u002Fterraform-aws-modules\u002Fterraform-aws-rds-aurora"," with the closest\nparameters from the existing one; e.g.:",[131,381,384],{"className":382,"code":383,"language":188,"meta":139},[186],"module \"aws_aurora_main_cluster\" {\n  source  = \"terraform-aws-modules\u002Frds-aurora\u002Faws\"\n  version = \"~> 5.2\"\n\n  # ...\n}\n",[61,385,383],{"__ignoreMap":139},[108,387,388,389,395,398,399,402,403],{},"import the Terraform states from the previous (existing) cluster",[131,390,393],{"className":391,"code":392,"language":223,"meta":139},[221],"terraform import 'aws_aurora_main_cluster.aws_rds_cluster.this[0]' main-database-name\nterraform import 'aws_aurora_main_cluster.aws_rds_cluster_instance.this[0]' main-database-instance-name-01\nterraform import 'aws_aurora_main_cluster.aws_rds_cluster_instance.this[1]' main-database-instance-name-02\n\n# ...\n",[61,394,392],{"__ignoreMap":139},[396,397],"br",{},"then if you haven't yet and would like to \"match reality\" between the\nexisting and the specified resource, run ",[61,400,401],{},"terraform plan"," a few times and\nadjust the parameters until Terraform reports:",[131,404,407],{"className":405,"code":406,"language":136,"meta":139},[134],"No changes. Your infrastructure matches the configuration.\n",[61,408,406],{"__ignoreMap":139},[108,410,411,412,414,415,421,423,424,426,427],{},"last but not least, remove the corresponding resources from the ",[61,413,288],{},"\nTerraform state so that it doesn't try to keep track of the changes and also\ndon't try to destroy once the resource definition is no longer in that code\nbase:",[131,416,419],{"className":417,"code":418,"language":223,"meta":139},[221],"# Hypothetical name of the resource inside production\u002Faws\u002Fmain.tf\nterraform state rm aws_rds_cluster.default \\\n    'aws_rds_cluster_instance.default[0]' 'aws_rds_cluster_instance.default[1]'\n\n# ...\n",[61,420,418],{"__ignoreMap":139},[396,422],{},"once that is performed, feel free to remove the corresponding resource's\ndefinition from the ",[61,425,288],{}," code.",[131,428,431],{"className":429,"code":430,"language":188,"meta":139},[186],"resource \"aws_rds_cluster\" \"default\" {\n  # ...\n}\n\nresource \"aws_rds_cluster_instance\" \"default\" {\n  count = var.number_of_database_instances\n\n  # ...\n}\n",[61,432,430],{"__ignoreMap":139},{"title":139,"searchDepth":434,"depth":434,"links":435},2,[436,437,438],{"id":13,"depth":434,"text":14},{"id":56,"depth":434,"text":57},{"id":125,"depth":434,"text":126,"children":439},[440,443],{"id":171,"depth":441,"text":442},3,"Splitting the modules sub-path to its own repository",{"id":274,"depth":441,"text":275},"2021-08-11T00:00:00+02:00","Want to know how to better organize existing Terraform code? If you grasp these ideas, it could even serve for not-yet Infrastructure as Code resources. Jump in and take a look.","md",{},true,"\u002Fposts\u002F2021\u002F08\u002Freal-life-terraform-refactoring-guide",{"title":5,"description":445},"real-life-terraform-refactoring-guide","posts\u002F2021\u002F08\u002Freal-life-terraform-refactoring-guide",[454,455,145],"infrastructure-as-code","cloud","jPrVDAjqTerXgzb6UVNqa_9QMs0RvtE05oSR-ywTlgQ",1778441743697]