← docs
docs · v0.2

Release Engineering — The Pipeline as the Single Source of Truth

v0.2 · last updated 2026-07draft

DRAFT — pending Harsh's verification and employer sign-off.

This page documents the release-engineering platform Harsh built and operated at Auction.com across roughly seven years (2019→2026): the Jenkins pipeline as the single driver of release state, the two-layer library-and-DSL design that made it reproducible, and the trunk-based daily-release cadence the platform eventually enabled. It is drawn from two bodies of work — the release orchestration that tied pipeline execution to an issue tracker's release lifecycle, and the longer arc of the shared library and job-DSL platform that carried it — with the hard-won friction kept as seasoning.

1. The pipeline as the single source of release truth

There were two sources of truth for "what is being released": the CI/CD pipeline (Jenkins) and the release object in an external issue tracker, whose stages ran Planned → Deploying → Deployed-to-CI → Deployed-to-QA → Released. Humans kept them in sync by hand — error-prone, and release status was never trustworthy at a glance.

The redesign made one authority the writer and the other the mirror: pipeline execution drives release state. A set of stage-driven Jenkinsfiles — one per environment and flow — covers CI, QA, PROD, PROD-DR, on-demand release, mobile release, cleanup, and release-request flows: roughly nine distinct pipelines. A release-steps module in the shared library talks to the tracker's release API (GraphQL) to resolve a pipeline, list releases and their stages, fetch a build, create or advance a release, mark it released, attach issues, and publish release notes.

Pipeline stages map 1:1 to release stages. As a deploy pipeline clears a stage, it advances the corresponding release stage — a successful CI deploy moves the release to "Deployed to CI," and so on up to "Released." Two release modes coexist: a scheduled Standard release and an On-Demand release kicked off by a submit-release-request job and dispatcher.

flowchart LR
  subgraph pipeline["Jenkins pipeline stage · writer"]
    checkout["checkout / build"]
    deployCI["deploy -> CI"]
    deployQA["deploy -> QA"]
    deployPROD["deploy -> PROD"]
  end

  subgraph tracker["tracker release stage · mirror"]
    planned[Planned]
    deployedCI[Deployed-to-CI]
    deployedQA[Deployed-to-QA]
    released[Released]
  end

  checkout --> planned
  deployCI --> deployedCI
  deployQA --> deployedQA
  deployPROD --> released

The rejected alternative — tracker drives pipeline — was considered and turned down: the build is the ground truth of what actually deployed, so it should be the writer. The tracker only reflects it.

2. Lifecycle sync: best-effort, never fails a deploy

Release bookkeeping is important, but it must never block shipping. Every tracker call is wrapped in try/catch, logs a warning, and continues: deploy reliability outranks sync fidelity. A sync failure is a paper cut, not a blocked release. Hard-failing the deploy on a sync error was explicitly rejected as a design.

Three real bugs shaped the integration, and each one left a constraint behind it:

Code-level toggles act as operational escape hatches: skip publishing release notes, or skip creating new placeholder releases, for edge cases.

Reusable pattern: syncing a pipeline to an external stateful system needs idempotency + locking + defensive reads — and every external call wrapped so a sync failure never fails the deploy. The external system is a mirror, not a gate.

3. The two-layer platform: how (library) + what (DSL)

Underneath the release orchestration sits a two-layer design that carried the whole org's CI for seven years:

Reusable pattern: put the "how" (steps) in a shared library and the "what jobs exist" (topology) in DSL. Two axes, two repos, both as code.

4. Seven years visible in the history

The platform's shape changed repeatedly as the org, the cloud, and the tooling changed, and each change is legible in the commit history:

5. Decisions and their tradeoffs

Each major choice was a tradeoff made under real constraints:

6. Friction worth keeping (war stories, as seasoning)

The platform's design was shaped as much by what broke as by what was planned. A few of the harder lessons, kept short:

7. The cadence this platform enabled

The cumulative effect of the platform above is the release cadence it made cheap: a move from bi-weekly to daily releases with a <1% rollback rate, on an in-house Jenkins + GitHub Actions toolchain (replacing a six-figure-per-year SaaS suite) running 10K+ monthly pipeline executions. GitOps delivery via Argo CD and the trunk-based workflows sit on top of this pipeline layer; this page is the part of the stack that made "a release is a build, and the build is the truth" operationally real.

8. Honest gaps

Impact numbers are not in the source material: how many services the shared library serves, builds per day, mean pipeline duration over time, and any before/after "releases mislabeled per month" or "manual minutes saved" figure. A headline version of this story ("maintaining CI for a large org for seven years") would need those pulled from Jenkins. They are not in the material this page was distilled from, and nothing here invents them.

Sources

This page was distilled from Harsh's commit history and session logs across the Jenkins shared library, the job-DSL repo, and the release-orchestration work — cross-checked against the in-repo PRD and resume. No internal hostnames, credential identifiers, ticket IDs, or proprietary source code appear; the pipelines and library are described, not copied. Every architectural claim above traces to that material.