[{"data":1,"prerenderedAt":312},["ShallowReactive",2],{"\u002Fen\u002Fpost\u002F2021\u002F02\u002Fjenkins-rebuilding-it-phase-by-phase":3},{"id":4,"title":5,"author":6,"body":7,"createdAt":297,"description":298,"extension":299,"meta":300,"navigation":301,"path":302,"seo":303,"slug":304,"stem":305,"tags":306,"__hash__":311},"posts\u002Fposts\u002F2021\u002F02\u002Fjenkins-rebuilding-it-phase-by-phase.md","The Freeletics CI\u002FCD rebuild, phase by phase",null,{"type":8,"value":9,"toc":288},"minimark",[10,15,29,32,36,41,44,60,68,74,78,81,89,92,105,108,112,120,126,140,150,160,164,171,195,199,202,210,213,217,230,233,274,281,285],[11,12,14],"h1",{"id":13},"picking-up-where-we-left-off","Picking up where we left off",[16,17,18,23,24,28],"p",{},[19,20,22],"a",{"href":21},"\u002Fposts\u002F2021\u002F01\u002Fjenkins-five-years-of-cicd-debt","Part 1"," covered what we\ninherited and why we chose to rebuild Jenkins rather than replace it.\n",[19,25,27],{"href":26},"\u002Fposts\u002F2021\u002F01\u002Fjenkins-boring-security-by-design","Part 2"," covered the\nsecurity foundation: authorization that stays accurate without manual upkeep,\nand secrets decoupled from the configuration release cycle. This post is about\nthe build system itself (the piece that made Dependabot Monday mornings a\nnon-event).",[16,30,31],{},"One constraint shaped everything: we couldn't afford a hard cutover. Freeletics'\nback-end and web deployment pipelines were running continuously. The redesign\nhad to happen alongside normal operations, phase by phase, with the old system\nstill running until each piece was ready to replace.",[11,33,35],{"id":34},"phase-2-kill-the-job-builder-introduce-kaniko","Phase 2: Kill the Job Builder, introduce Kaniko",[37,38,40],"h2",{"id":39},"first-things-first-reproducibility","First things first: reproducibility",[16,42,43],{},"The hardest requirement to state and the easiest to overlook: a Jenkins\ninstallation that can be deleted and recreated from scratch without losing\nanything. No manual configuration living only in someone's memory, no jobs\nthat were \"created by hand one afternoon\" and never written down.",[16,45,46,47,53,54,59],{},"To get there, JJB had to go. Its replacement was a combination of\n",[19,48,52],{"href":49,"rel":50},"https:\u002F\u002Fgithub.com\u002Fjenkinsci\u002Fconfiguration-as-code-plugin",[51],"nofollow","Jenkins Configuration as Code (JCasC)","\nfor the Jenkins system-level settings and\n",[19,55,58],{"href":56,"rel":57},"https:\u002F\u002Fgithub.com\u002Fjenkinsci\u002Fjob-dsl-plugin",[51],"Job DSL"," for job definitions,\nboth managed through Terraform. Every Jenkins job became a pull request. Every\nsystem configuration change was reviewable, auditable, and rollback-able.",[16,61,62,63,67],{},"The practical impact was felt immediately. Previously, understanding what jobs\nJenkins was running required either opening the Jenkins UI or reading the\nbundled JJB YAML inside the Helm Chart. Now, the full picture lived in the\nTerraform repository alongside everything else. New team members could onboard\nto Jenkins by reading code, not by poking around a UI. This is\n",[19,64,66],{"href":65},"\u002Fposts\u002F2019\u002F01\u002Fdevops-benefits#configuration-andor-infrastructure-as-code","Infrastructure as Code"," applied to CI\nconfiguration:",[69,70,71],"blockquote",{},[16,72,73],{},"\"Easy to recreate the infrastructure, if it is necessary to move everything\nto another place, this can happen with a few manual interactions; allows for\na code review of infrastructure and configurations, which consequently brings\na culture of collaboration in the development, sharing of knowledge.\"",[37,75,77],{"id":76},"docker-in-docker-out-kaniko-in","Docker-in-Docker out, Kaniko in",[16,79,80],{},"Docker-in-Docker (DinD) was how Jenkins built container images: a Docker daemon\nrunning inside a container, building images for other containers. It works, but\nit requires running privileged Kubernetes pods (a security concern on shared\ninfrastructure) and carries well-documented instability issues under high build\nconcurrency (the same concurrency problem that made Dependabot mornings painful).",[16,82,83,88],{},[19,84,87],{"href":85,"rel":86},"https:\u002F\u002Fgithub.com\u002FGoogleContainerTools\u002Fkaniko",[51],"Kaniko"," builds images from a\nDockerfile without needing a Docker daemon at all. It runs as a regular\nKubernetes pod with no elevated privileges, uses the container registry\ndirectly for cache, and handles concurrent builds gracefully because each build\nis fully isolated.",[16,90,91],{},"Migration covered every server-side application:",[93,94,95,99,102],"ul",{},[96,97,98],"li",{},"All Rails back-end services;",[96,100,101],{},"Web applications;",[96,103,104],{},"Internal tools (AI and analytics).",[16,106,107],{},"Each application got its own Kaniko cache repository in ECR. Layer caching\nacross builds meant the build times stayed competitive with the DinD baseline\nwhile removing the concurrency issues that caused the Monday morning hangs.",[37,109,111],{"id":110},"the-shared-library-was-doing-too-much","The Shared Library was doing too much",[16,113,114,115,119],{},"The Jenkins Groovy Shared Library had a ",[116,117,118],"code",{},"run()"," method that orchestrated the\nentire pipeline from a single entry point: git checkout, build, tag, push to\nECR, Slack notification. One call to rule them all. The appeal was obvious:\none-liner Jenkinsfiles across every repository.",[16,121,122,123,125],{},"The problem showed up the moment someone needed to deviate from the happy path.\nWant to add a custom build arg? Override the image tag format? Skip the Slack\nnotification for a specific branch? None of that was possible without touching\nthe Shared Library and potentially breaking every other pipeline that depended\non the same ",[116,124,118],{}," method.",[16,127,128,129,132,133,136,137,139],{},"The refactoring introduced a ",[116,130,131],{},"KanikoBuilder"," class that pipelines compose\nexplicitly from their ",[116,134,135],{},"Jenkinsfile",". The Shared Library provides the building\nblocks; the per-repository ",[116,138,135],{}," decides how to assemble them:",[141,142,148],"pre",{"className":143,"code":145,"language":146,"meta":147},[144],"language-groovy","@Library('jenkins_shared_library@master') _\n\nimport com.example.KanikoBuilder\n\ndef kanikoBuilder = new KanikoBuilder(repositoryName: \"foobar\",\n                                      steps: this)\n\npodTemplate(yaml: kanikoBuilder.getPodTemplate()) {\n    node(POD_LABEL) {\n        stage('Git - Fetch code') {\n            env.GIT_COMMIT = gitCheckout(branchName: env.BRANCH_NAME)\n        }\n\n        stage(\"Build - Container Image\") {\n            kanikoBuilder.setImageTag(env.GIT_COMMIT)\n            kanikoBuilder.build()\n        }\n    }\n}\n","groovy","",[116,149,145],{"__ignoreMap":147},[16,151,152,153,156,157,159],{},"The ",[116,154,155],{},"steps: this"," argument passes the pipeline context into ",[116,158,131],{},",\nletting the library add stages at runtime. Per-repository Jenkinsfiles became\nshort and readable. Cross-cutting changes (new tag format, new registry, Slack\nintegration update) could now be made once in the library and propagated across\nall pipelines on the next library release.",[37,161,163],{"id":162},"image-multi-tagging","Image multi-tagging",[16,165,166,167,170],{},"The QA stack resolved images by tag convention, and it needed multiple tags\napplied in a single build pass. Kaniko accepts N ",[116,168,169],{},"--destination"," arguments, so\nthe library was extended to receive a tag list and generate the flag string:",[93,172,173,179,189],{},[96,174,175,178],{},[116,176,177],{},"qa-\u003CSHA1>"," applied always;",[96,180,181,184,185,188],{},[116,182,183],{},"qa-latest-master"," applied when the branch is ",[116,186,187],{},"master",";",[96,190,191,194],{},[116,192,193],{},"qa-\u003Cbranch-name>"," applied optionally, declared in the Jenkinsfile.",[37,196,198],{"id":197},"human-readable-credential-ids","Human-readable credential IDs",[16,200,201],{},"A minor change with outsized quality-of-life impact: all Jenkins credential IDs\nwere UUIDs. Debugging a failed credential lookup meant grep-ing through the\nHelm values to cross-reference the UUID against a label. After the migration:",[141,203,208],{"className":204,"code":206,"language":207},[205],"language-text","slack_token\naws_credentials\nnpm_token\ngithub_api_token\n","text",[116,209,206],{"__ignoreMap":147},[16,211,212],{},"All Jenkinsfiles across repositories were updated to reference the new IDs.\nTime spent debugging credential issues dropped to near zero.",[11,214,216],{"id":215},"the-result","The result",[16,218,219,220,224,225,229],{},"By Feb\u002F2021, the Jenkins installation was unrecognizable from what it had been\nsix months earlier. The Monday morning Dependabot problem was gone. The\n",[19,221,223],{"href":222},"\u002Fposts\u002F2019\u002F01\u002Fdevops-genesis#mura-unevenness","Mura"," was eliminated at the\nsource rather than managed around. The ",[19,226,228],{"href":227},"\u002Fposts\u002F2019\u002F01\u002Fdevops-benefits#conclusion","boring\nplatform"," goal (\"no unexpected\nbehavior that makes your heart pump faster, no surprises, it just works\") had a\nnew data point: 50 concurrent Kaniko builds triggered without a single hang.\nBuild concurrency that used to cause cascading failures now just worked, because\nKaniko pods are isolated by design and don't share a Docker daemon to fight\nover.",[16,231,232],{},"A summary of what changed:",[93,234,235,242,248,254,264],{},[96,236,237,241],{},[238,239,240],"strong",{},"Kubernetes-native workers:"," ephemeral pods spawned per job, sized to the\nworkload type, torn down when the job finishes;",[96,243,244,247],{},[238,245,246],{},"Fully reproducible:"," delete the Helm release, recreate it from Terraform\nand JCasC, get the same Jenkins back (zero manual state);",[96,249,250,253],{},[238,251,252],{},"Single Groovy Shared Library:"," one codebase for back-end, web, coach,\nand tracking pipelines (previously each maintained their own ad-hoc\nimplementations with duplicated logic);",[96,255,256,259,260,263],{},[238,257,258],{},"GitHub OAuth with team-based RBAC:"," access control that matches the\nexisting GitHub team structure (one place to manage, one place to audit;\nfull decision in ",[19,261,27],{"href":262},"\u002Fposts\u002F2021\u002F01\u002Fjenkins-boring-security-by-design#authorization-as-an-engineering-problem",");",[96,265,266,269,270,273],{},[238,267,268],{},"Decoupled secrets:"," Jenkins configuration releases and secrets changes\nhave independent cycles; both are fully auditable (see\n",[19,271,27],{"href":272},"\u002Fposts\u002F2021\u002F01\u002Fjenkins-boring-security-by-design#splitting-the-concerns",").",[275,276,278],"callout",{"type":277},"note",[16,279,280],{},"CircleCI stayed in place for iOS and Android builds. The redesign scope\nwas server-side applications only (closing the mobile\u002Fserver-side gap was\nexplicitly deferred).",[37,282,284],{"id":283},"whats-next","What's next",[16,286,287],{},"ChatOps for deployment triggering was designed but didn't ship in this phase:\na Hubot script accepting deployment commands from a Slack channel and\ntranslating them to Jenkins pipeline triggers, mirroring the mobile deployment\nworkflow already in use on CircleCI. The interface design was drafted\n(including ACL for restricting deployment permissions to a named group), but\nthe implementation was deprioritized. The deployment flow still goes through\nthe Jenkins UI.",{"title":147,"searchDepth":289,"depth":289,"links":290},2,[291,292,293,294,295,296],{"id":39,"depth":289,"text":40},{"id":76,"depth":289,"text":77},{"id":110,"depth":289,"text":111},{"id":162,"depth":289,"text":163},{"id":197,"depth":289,"text":198},{"id":283,"depth":289,"text":284},"2021-02-01T00:00:00+01:00","The build system: Kaniko replacing Docker-in-Docker, a Groovy Shared Library redesign that cut per-repository boilerplate to near zero, and the change that made Dependabot Monday mornings a non-event. What we shipped and what it changed.","md",{},true,"\u002Fposts\u002F2021\u002F02\u002Fjenkins-rebuilding-it-phase-by-phase",{"title":5,"description":298},"jenkins-rebuilding-it-phase-by-phase","posts\u002F2021\u002F02\u002Fjenkins-rebuilding-it-phase-by-phase",[307,308,309,310],"ci-cd","jenkins","kubernetes","infrastructure-as-code","n5nfhedVJ-EYbEBmFtVLCzTselrsryfyUXKnh8PMjeg",1778441744104]