projects/
user photo

Freeletics: Jenkins CI/CD Redesign

Published on
ci-cd
platform-engineering
infrastructure-as-code

Introduction

Summary

Freeletics ran three fragmented CI/CD systems in parallel: Jenkins for back-end and web, CircleCI for mobile, and Travis for tests. Jenkins itself was at least five years out-of-date, built on a customized Jenkins Job Builder (JJB) fork that didn't support Jenkins Pipelines, deployed through a Helm Chart that embedded secrets directly in its values file (coupling every configuration change to a secrets release).

Problem

Three specific failure modes drove the redesign:

  • Morning build storms: Dependabot merged PRs in batches at the start of the day, triggering simultaneous Docker image builds that overwhelmed Jenkins master-to-slave HTTP communication and caused widespread job hangs;
  • Modernization blocked: the outdated JJB fork rejected Pipeline definitions at deploy time, making it impossible to adopt any Jenkins feature released in the past two years;
  • Untestable, untouchable Helm Chart: JJB YAML was rendered inside Go templates and executed during chart install. Any change carried the risk of a broken Jenkins release with no rollback path that didn't also revert secrets.

Solution

Technical Implementation

Executed in four sequential phases:

Phase 1 - Tool evaluation (Feb-Mar/2020): benchmarked Docker image build times across CircleCI, GitLab CI (shared and self-hosted runners with Kaniko), and Jenkins. Jenkins produced the fastest server-side builds due to lower latency and full control over runner hardware sizing. Decision: invest in Jenkins, redesign from scratch.

Phase 2 - Pipeline modernization (Aug/2020): replaced JJB with Jenkins Configuration as Code (JCasC) and Job DSL templates managed through Terraform, making every job definition a reviewable pull request. Migrated all Docker image builds from Docker-in-Docker to Kaniko (running as unprivileged ephemeral Kubernetes pods). Redesigned the Jenkins Groovy Shared Library around a composable KanikoBuilder class, reducing per-repository Jenkinsfiles to declarative build specifications. Introduced image multi-tagging (qa-<SHA1>, qa-latest-master) to support the QA stack's tag-based image resolution.

Phase 3 - Authorization (Sep/2020): implemented GitHub OAuth, mapping Jenkins RBAC roles directly to GitHub team membership. Replaced open admin access (any G-Suite account) with a reviewable, auditable access model using the same workflow as the rest of the infrastructure.

Phase 4 - Secrets decoupling (Sep/2020): separated secrets management from the Helm Chart release cycle. Static credentials (AWS IAM keys, API tokens) are Sops-encrypted in the repository and synced to Jenkins Credentials Store through JCasC. Runtime secrets (Kubeconfigs, Kubernetes credentials) are stored in AWS Secrets Manager and read on-the-fly by pipelines via the credentials provider plugin. Jenkins Helm releases became configuration-only operations.

Impact and results

  • Fully reproducible deployments: the Jenkins Helm release can be deleted and recreated from Terraform + JCasC with complete fidelity (no manual state, no out-of-band configuration);
  • Build time advantage preserved: migrating from Docker-in-Docker to Kaniko maintained Jenkins' benchmark advantage over alternatives (~2:00 for fl-backend-rails vs ~10:01 on CircleCI, as of the Feb/2020 evaluation);
  • Unified pipelines: a single Groovy Shared Library now covers back-end, web, coach, and tracking applications (previously each had independent ad-hoc Jenkinsfile implementations with duplicated logic);
  • Auditable secrets: Sops-encrypted catalog in version control provides full change history for credentials, replacing opaque values embedded in a Helm release.

Write-up

The full story is documented in a three-part series: