[{"data":1,"prerenderedAt":200},["ShallowReactive",2],{"\u002Fen\u002Fprojects\u002F2021\u002Ffreeletics-jenkins-redesign":3},{"id":4,"title":5,"body":6,"createdAt":175,"description":176,"extension":177,"meta":178,"navigation":189,"path":190,"seo":191,"slug":192,"stem":193,"tags":194,"website":198,"__hash__":199},"projects\u002Fprojects\u002F2021\u002Ffreeletics-jenkins-redesign.md","Freeletics: Jenkins CI\u002FCD Redesign",{"type":7,"value":8,"toc":166},"minimark",[9,14,19,23,27,30,53,57,61,64,70,89,95,101,105,135,139,142],[10,11,13],"h1",{"id":12},"introduction","Introduction",[15,16,18],"h2",{"id":17},"summary","Summary",[20,21,22],"p",{},"Freeletics ran three fragmented CI\u002FCD systems in parallel: Jenkins for back-end\nand web, CircleCI for mobile, and Travis for tests. Jenkins itself was at least\nfive years out-of-date, built on a customized Jenkins Job Builder (JJB) fork\nthat didn't support Jenkins Pipelines, deployed through a Helm Chart that\nembedded secrets directly in its values file (coupling every configuration\nchange to a secrets release).",[15,24,26],{"id":25},"problem","Problem",[20,28,29],{},"Three specific failure modes drove the redesign:",[31,32,33,41,47],"ul",{},[34,35,36,40],"li",{},[37,38,39],"strong",{},"Morning build storms:"," Dependabot merged PRs in batches at the start of\nthe day, triggering simultaneous Docker image builds that overwhelmed\nJenkins master-to-slave HTTP communication and caused widespread job hangs;",[34,42,43,46],{},[37,44,45],{},"Modernization blocked:"," the outdated JJB fork rejected Pipeline definitions\nat deploy time, making it impossible to adopt any Jenkins feature released in\nthe past two years;",[34,48,49,52],{},[37,50,51],{},"Untestable, untouchable Helm Chart:"," JJB YAML was rendered inside Go\ntemplates and executed during chart install. Any change carried the risk of a\nbroken Jenkins release with no rollback path that didn't also revert secrets.",[10,54,56],{"id":55},"solution","Solution",[15,58,60],{"id":59},"technical-implementation","Technical Implementation",[20,62,63],{},"Executed in four sequential phases:",[20,65,66,69],{},[37,67,68],{},"Phase 1 - Tool evaluation (Feb-Mar\u002F2020):"," benchmarked Docker image build\ntimes across CircleCI, GitLab CI (shared and self-hosted runners with Kaniko),\nand Jenkins. Jenkins produced the fastest server-side builds due to lower\nlatency and full control over runner hardware sizing. Decision: invest in\nJenkins, redesign from scratch.",[20,71,72,75,76,80,81,84,85,88],{},[37,73,74],{},"Phase 2 - Pipeline modernization (Aug\u002F2020):"," replaced JJB with Jenkins\nConfiguration as Code (JCasC) and Job DSL templates managed through Terraform,\nmaking every job definition a reviewable pull request. Migrated all Docker image\nbuilds from Docker-in-Docker to Kaniko (running as unprivileged ephemeral\nKubernetes pods). Redesigned the Jenkins Groovy Shared Library around a\ncomposable ",[77,78,79],"code",{},"KanikoBuilder"," class, reducing per-repository Jenkinsfiles to\ndeclarative build specifications. Introduced image multi-tagging (",[77,82,83],{},"qa-\u003CSHA1>",",\n",[77,86,87],{},"qa-latest-master",") to support the QA stack's tag-based image resolution.",[20,90,91,94],{},[37,92,93],{},"Phase 3 - Authorization (Sep\u002F2020):"," implemented GitHub OAuth, mapping\nJenkins RBAC roles directly to GitHub team membership. Replaced open admin\naccess (any G-Suite account) with a reviewable, auditable access model using\nthe same workflow as the rest of the infrastructure.",[20,96,97,100],{},[37,98,99],{},"Phase 4 - Secrets decoupling (Sep\u002F2020):"," separated secrets management from\nthe Helm Chart release cycle. Static credentials (AWS IAM keys, API tokens)\nare Sops-encrypted in the repository and synced to Jenkins Credentials Store\nthrough JCasC. Runtime secrets (Kubeconfigs, Kubernetes credentials) are stored\nin AWS Secrets Manager and read on-the-fly by pipelines via the credentials\nprovider plugin. Jenkins Helm releases became configuration-only operations.",[15,102,104],{"id":103},"impact-and-results","Impact and results",[31,106,107,113,123,129],{},[34,108,109,112],{},[37,110,111],{},"Fully reproducible deployments:"," the Jenkins Helm release can be deleted\nand recreated from Terraform + JCasC with complete fidelity (no manual state,\nno out-of-band configuration);",[34,114,115,118,119,122],{},[37,116,117],{},"Build time advantage preserved:"," migrating from Docker-in-Docker to Kaniko\nmaintained Jenkins' benchmark advantage over alternatives (~2:00 for\n",[77,120,121],{},"fl-backend-rails"," vs ~10:01 on CircleCI, as of the Feb\u002F2020 evaluation);",[34,124,125,128],{},[37,126,127],{},"Unified pipelines:"," a single Groovy Shared Library now covers back-end,\nweb, coach, and tracking applications (previously each had independent\nad-hoc Jenkinsfile implementations with duplicated logic);",[34,130,131,134],{},[37,132,133],{},"Auditable secrets:"," Sops-encrypted catalog in version control provides full\nchange history for credentials, replacing opaque values embedded in a Helm\nrelease.",[15,136,138],{"id":137},"write-up","Write-up",[20,140,141],{},"The full story is documented in a three-part series:",[31,143,144,152,159],{},[34,145,146,151],{},[147,148,150],"a",{"href":149},"\u002Fposts\u002F2021\u002F01\u002Fjenkins-five-years-of-cicd-debt","Part 1: Freeletics CI\u002FCD: five years of debt (and why we kept Jenkins)","\n-- what we inherited, the benchmark data behind the decision to invest, and the\ndesign goals that shaped the rebuild.",[34,153,154,158],{},[147,155,157],{"href":156},"\u002Fposts\u002F2021\u002F01\u002Fjenkins-boring-security-by-design","Part 2: Boring security on Freeletics Jenkins, by design","\n-- authorization that doesn't require a spreadsheet, and secrets decoupled from\nthe configuration release cycle.",[34,160,161,165],{},[147,162,164],{"href":163},"\u002Fposts\u002F2021\u002F02\u002Fjenkins-rebuilding-it-phase-by-phase","Part 3: The Freeletics CI\u002FCD rebuild, phase by phase","\n-- the build system itself: Kaniko migration, Groovy Shared Library redesign,\nand the change that made Dependabot Monday mornings a non-event.",{"title":167,"searchDepth":168,"depth":168,"links":169},"",2,[170,171,172,173,174],{"id":17,"depth":168,"text":18},{"id":25,"depth":168,"text":26},{"id":59,"depth":168,"text":60},{"id":103,"depth":168,"text":104},{"id":137,"depth":168,"text":138},"2021-02-01T00:00:00","End-to-end redesign of a 5-year-old Jenkins CI\u002FCD platform: replaced Jenkins Job Builder with Pipelines as Code managed through Terraform, migrated all Docker builds to Kaniko on Kubernetes, decoupled secrets management from the Helm Chart, and unified back-end and web CI\u002FCD through a Groovy Shared Library.","md",{"duration":179,"tools":181},{"from":180,"to":175},"2020-08-01T00:00:00",[182,183,184,185,186,187,188],"jenkins","kubernetes","kaniko","terraform","groovy","aws secrets manager","helm",true,"\u002Fprojects\u002F2021\u002Ffreeletics-jenkins-redesign",{"title":5,"description":176},"freeletics-jenkins-cicd-redesign","projects\u002F2021\u002Ffreeletics-jenkins-redesign",[195,196,197],"ci-cd","platform-engineering","infrastructure-as-code",null,"asnvLELu8UgsGnhCEGdHWKoLa26quHFLWuAgs_N6Mic",1778441743770]