[{"data":1,"prerenderedAt":189},["ShallowReactive",2],{"\u002Fen\u002Fpost\u002F2021\u002F01\u002Fjenkins-boring-security-by-design":3},{"id":4,"title":5,"author":6,"body":7,"createdAt":174,"description":175,"extension":176,"meta":177,"navigation":178,"path":179,"seo":180,"slug":181,"stem":182,"tags":183,"__hash__":188},"posts\u002Fposts\u002F2021\u002F01\u002Fjenkins-boring-security-by-design.md","Boring security on Freeletics Jenkins, by design",null,{"type":8,"value":9,"toc":170},"minimark",[10,15,19,22,25,33,37,45,48,55,61,64,67,75,79,82,89,92,95,103,107,110,113,126,132,139,143,146,149,152,160,163],[11,12,14],"h1",{"id":13},"the-access-control-nobody-chose","The access control nobody chose",[16,17,18],"p",{},"At some point during Jenkins' original setup at Freeletics, the Google OAuth\nplugin was configured in development mode. That mode grants every account in\nthe G-Suite domain admin access to Jenkins. It was left in production. By the\ntime I ran an access review in 2020, every Freeletics employee (regardless of\nrole) could open Jenkins, modify jobs, trigger deployments to any environment,\nand change system configuration.",[16,20,21],{},"Nobody intended this. It was a default.",[16,23,24],{},"That distinction matters: security debt rarely accumulates through deliberate\nchoices. It accumulates through configuration that worked well enough to ship\nand was never revisited. Each individual default is harmless; the pattern is\nnot. A CI system that any employee can reconfigure is an audit finding waiting\nto happen (and, more practically, a deployment pipeline that can be triggered\nor altered by someone who has no business doing so).",[16,26,27,28,32],{},"The fix wasn't just \"restrict access.\" That framing leads to the wrong design.\nThe real question was: ",[29,30,31],"em",{},"how do you build access control that stays accurate over\ntime without requiring manual maintenance?"," Access lists that require human\nupkeep drift. They drift because people change roles, people leave, and nobody\nhas time to be the steward of a permissions spreadsheet.",[11,34,36],{"id":35},"authorization-as-an-engineering-problem","Authorization as an engineering problem",[16,38,39,44],{},[40,41,43],"a",{"href":42},"\u002Fposts\u002F2021\u002F01\u002Fjenkins-five-years-of-cicd-debt#the-evaluation-stay-or-leave","When we decided to rebuild Jenkins",",\nthe design goals included making every aspect of the system reproducible from\ncode. Authorization was no exception.",[16,46,47],{},"Two options were evaluated.",[16,49,50,54],{},[51,52,53],"strong",{},"Google OAuth with groups"," would restrict access to specific G-Suite groups\nrather than the entire domain. The off-boarding story is clean: deactivate a\nG-Suite account and that person loses Jenkins access immediately. The problem is\nthe authorization model itself. G-Suite groups are designed for e-mail\ndistribution, not access control. A Jenkins permission group built on top of\nG-Suite groups would live outside any existing engineering workflow: group\nmembership changes happen in the G-Suite admin console, not through pull\nrequests; there's no audit trail a developer can query; and keeping groups\nsynchronized with actual team structure requires manual intervention every time\nsomeone changes teams.",[16,56,57,60],{},[51,58,59],{},"GitHub OAuth"," maps Jenkins authorization directly to GitHub team membership.\nAt Freeletics, GitHub teams already reflected engineering structure (back-end,\nweb, ops, coach) and were already managed through pull requests. Using them as\nthe Jenkins authorization source meant zero new model to maintain: one source of\ntruth, one place to make changes, one place to audit them.",[16,62,63],{},"The off-boarding trade-off is real: deactivating a G-Suite account doesn't\nautomatically remove someone from a GitHub team, so there's a separate removal\nstep required. That gap was consciously accepted.",[16,65,66],{},"Q: Is a manual off-boarding step a deal-breaker?\nA: Only if the alternative is more reliable in practice. A model requiring\nconstant maintenance to stay accurate has a short mean-time-to-drift; a model\nrequiring one deliberate step at a specific moment has a narrow, bounded failure\nmode. GitHub OAuth's gap is known, documented, and owned. Google group drift is\nopen-ended and invisible until an audit surfaces it.",[16,68,69,70,74],{},"The outcome: GitHub OAuth with team-based RBAC. Access requests happen through\npull requests on the GitHub teams configuration. New engineers, role changes,\ncontractor access: all of it reviewable, auditable, and executed through a\nworkflow that already exists. Any access review is a ",[71,72,73],"code",{},"git log"," away from being\ndone.",[11,76,78],{"id":77},"the-helm-chart-had-a-dirty-secret","The Helm Chart had a dirty secret",[16,80,81],{},"The secrets problem was a separation of concerns violation wearing a Helm chart\ncostume.",[16,83,84,85,88],{},"The Jenkins Helm Chart stored secrets encrypted via ",[71,86,87],{},"helm-secrets"," inside the\nHelm values file. Encrypted at rest, available at deploy time (it worked). The\nproblem surfaced the moment you needed to change anything else.",[16,90,91],{},"Every Helm release (every plugin update, every executor count adjustment, every\nconfiguration change) carried the full secrets bundle along for the ride.\nRolling back a Helm release because a plugin update broke something also rolled\nback the secrets version. Two completely independent concerns (configuration and\nsecrets) shared one release train, which meant neither could change independently.",[16,93,94],{},"Q: What's the actual cost of that coupling?\nA: Imagine a plugin update that breaks the build pipeline on a Monday morning.\nYou roll back the Helm release. The rollback also silently reverts a credential\nrotation that happened the previous week. You now have live pipelines running\nagainst rotated credentials that your Jenkins hasn't been told about. The blast\nradius of a configuration change includes your secrets state, and there's no way\nto disentangle the two without a full re-deploy.",[16,96,97,98,102],{},"This is ",[40,99,101],{"href":100},"\u002Fposts\u002F2019\u002F01\u002Fdevops-genesis#muda-waste","Muda"," at the infrastructure level:\nwork performed to fix a configuration change bleeds into your secrets state. Every\ndeployment carries overhead that has nothing to do with what you're actually\ntrying to change. That overhead is invisible until something breaks.",[11,104,106],{"id":105},"splitting-the-concerns","Splitting the concerns",[16,108,109],{},"The redesign cut the coupling entirely. Secrets and configuration got independent\nrelease cycles and separate ownership.",[16,111,112],{},"Secrets fell into two categories, and the distinction drove the design:",[16,114,115,118,119,125],{},[51,116,117],{},"Runtime secrets"," (Kubeconfigs, Kubernetes service account credentials,\nanything that rotates regularly or whose value pipelines read at execution time)\nwere moved to AWS Secrets Manager. The\n",[40,120,124],{"href":121,"rel":122},"https:\u002F\u002Fgithub.com\u002Fjenkinsci\u002Faws-secrets-manager-credentials-provider-plugin",[123],"nofollow","AWS Secrets Manager Credentials Provider plugin","\nextends the Jenkins credentials API to read from Secrets Manager on the fly.\nRotation happens in Secrets Manager; pipelines pick up the new value on the next\nrun with no Helm release required.",[16,127,128,131],{},[51,129,130],{},"Static credentials"," (AWS IAM keys, API tokens, service credentials that\nchange infrequently) stayed Sops-encrypted in the infrastructure repository\nand were synced to the Jenkins Credentials Store through JCasC on each Helm\nrelease. The repository is the audit log: every change goes through a pull\nrequest, every reviewer can see exactly what changed, and git history is the\nversioning.",[133,134,136],"callout",{"type":135},"note",[16,137,138],{},"AWS SSM Parameter Store was ruled out early. The Jenkins plugin for SSM\ndoesn't obfuscate secret values from build log output, meaning credentials can\nappear in plaintext in job logs. HashiCorp Vault covers all three requirements\nand more, but running and operating a Vault cluster is itself a non-trivial\nplatform engineering task. The operational overhead outweighs the marginal\nbenefit over AWS Secrets Manager at this team size and scope.",[11,140,142],{"id":141},"boring-security-by-design","Boring security, by design",[16,144,145],{},"The through-line across both changes is the same: the right security posture is\nthe one that stays correct without requiring ongoing effort to maintain.",[16,147,148],{},"Authorization that maps to existing GitHub teams doesn't drift because it\ninherits all the maintenance work that already happens around GitHub teams. When\na new engineer joins, they get added to a GitHub team (a step that was already\nhappening) and Jenkins access follows. When someone changes teams, their Jenkins\npermissions change with it. The security model is a side effect of normal\nengineering operations, not a parallel task stacked on top of them.",[16,150,151],{},"Secrets with independent release cycles don't get silently rolled back as\ncollateral damage from configuration changes. Rotation happens in one place and\npropagates without a deployment. The audit trail lives in the repository where\nanyone can query it.",[16,153,154,155,159],{},"Neither of these is \"hardening\" in the traditional sense. They're design choices\nthat make the system easier to operate correctly than incorrectly (which is\nwhat ",[40,156,158],{"href":157},"\u002Fposts\u002F2019\u002F01\u002Fdevops-benefits#conclusion","a boring platform"," actually means). Not\nzero incidents, but no surprises: the kind of security that doesn't require\nemergency firefighting because the defaults are already aligned with the intent.",[161,162],"hr",{},[16,164,165,166],{},"With authorization and secrets on solid footing, the next phase was the build\nsystem itself: replacing Docker-in-Docker with Kaniko, redesigning the Groovy\nShared Library, and making the Dependabot Monday mornings a non-event.\n",[40,167,169],{"href":168},"\u002Fposts\u002F2021\u002F02\u002Fjenkins-rebuilding-it-phase-by-phase","Part 3: the execution.",{"title":171,"searchDepth":172,"depth":172,"links":173},"",2,[],"2021-01-24T00:00:00+01:00","Every G-Suite account had Jenkins admin access. Nobody chose this; it was the default, and it was never revisited. How authorization and secrets management were rebuilt to make the right thing the path of least resistance.","md",{},true,"\u002Fposts\u002F2021\u002F01\u002Fjenkins-boring-security-by-design",{"title":5,"description":175},"jenkins-boring-security-by-design","posts\u002F2021\u002F01\u002Fjenkins-boring-security-by-design",[184,185,186,187],"ci-cd","jenkins","security","platform-engineering","2DB0dXVXmHsa1V9kIcvNyq4p-N0cM0tOGZLklH4hAjk",1778441744118]