Release engineering from scratch: build the artifact once, prove where it came from, and version the whole mesh without breaking it

Your CI is green. The tag is cut. Then someone asks the question that should be trivial: “Is the binary running in production the exact thing we tested in staging — and can you prove where it was built?” If the honest answer is “we rebuilt it for prod, so… probably,” you don't have a release process. You have a habit.

Release engineering is the discipline that turns “probably” into “here's the commit, here's the signed provenance, here's the audit trail.” This guide builds that system from an empty repo, in the order you'd actually build it: branching first (it decides everything downstream), then promoting one immutable artifact through environments, then signing it and proving its origin, and finally the part everyone underestimates — versioning a set of components that depend on each other without shipping a diamond-shaped disaster.

What you're actually building — one artifact, fully traceable

Google's SRE book frames release engineering around four principles, and they make a good spine for the whole build: a self-service model (teams run their own releases through tooling, not tickets), high velocity (“frequent releases result in fewer changes between versions,” which makes every release smaller and safer to debug), hermetic builds (“builds are insensitive to the libraries and other software installed on the build machine”), and enforcement of policies and procedures (access control over who can approve, build, and deploy).

Everything below serves one measurable goal: you can point at any running artifact and produce the commit it came from, prove it was built by your pipeline and not tampered with, and rebuild it byte-for-byte if you have to. Traceability, reproducibility, audit. Hold onto that — it's the test every design decision has to pass.

Prerequisites (Table 1):

Read the four phases in order. Each one assumes the previous is in place. You can't promote an artifact you rebuild; you can't sign provenance for a build you can't trace to a branch.

Phase 1 — Pick the branching model that lets you ship, not the one with the nicest diagram

The branching model is the foundation because it sets your integration frequency, and integration frequency sets everything else — how big your merges are, how late conflicts surface, and whether “release” is a routine event or a quarterly ordeal.

The default that works — trunk-based development

Use trunk-based development. Paul Hammant's definition is the one to internalize: developers “collaborate on code in a single branch called ‘trunk’ and resist any pressure to create other long-lived development branches.” Small teams commit straight to trunk several times a day. Scaled teams use short-lived feature branches — hours to a day or two — purely for review and a CI check, then integrate.

The mechanism that makes this safe is feature flags. You merge incomplete work to trunk behind a flag that's off in production, so trunk stays releasable while the feature is still being built. That's what lets you integrate continuously without shipping half-done features — and it decouples “merged” from “released,” which you'll want in every later phase.

When you need to stabilize a specific release, you cut a release branch from trunk just-in-time, harden it, ship, and delete it. Fixes still land on trunk first, then get cherry-picked onto the release branch — never the other way around. Google runs exactly this pattern: they “branch from the mainline at a specific revision and never merge the branch back,” and cherry-pick bug fixes in, so “the exact contents of each release remain known and controlled.”

Cut a hardening branch from trunk, then cherry-pick a fix that already landed on trunk:

# tag the exact revision you're releasing from
git switch main && git pull
git switch -c release/1.4.x        # short-lived, hardened, deleted after

# a fix lands on trunk FIRST (upstream-first), then comes down:
git switch main && git cherry-pick --help  # (fix merged to main via PR)
git switch release/1.4.x
git cherry-pick <sha-of-fix-on-main>      # release contents stay known
git tag -s v1.4.3 -m "v1.4.3"            # signed tag; see Phase 3

Why git-flow is the wrong default now

Git-flow (master, develop, feature, release, hotfix) was a genuinely good model for its era. The decisive evidence against it for modern delivery isn't a hot take — it's the author's own note of reflection, added to the original 2010 post in 2020: “If your team is doing continuous delivery of software, I would suggest to adopt a much simpler workflow (like GitHub flow) instead of trying to shoehorn git-flow into your team.” He's explicit that git-flow was designed for versioned software with multiple production versions in the wild, not for continuously-delivered web apps.

The structural problem: long-lived develop and release branches mean code isn't truly integrated for days, which is the exact opposite of continuous integration. Fowler's framing is that low integration frequency produces “larger, riskier merges that surface conflicts late” and breeds integration fear. DORA's research consistently ties trunk-based development to elite delivery performance, and Atlassian now labels git-flow a legacy workflow.

The lightweight middle — GitHub flow and GitLab flow

GitHub flow is trunk-based development with a pull-request gate: branch off main, commit, open a PR, review, merge to the always-deployable default, delete the branch. It's the model Driessen himself points people to. GitLab flow layers environment branches (e.g. production, pre-production) and release branches on top, with an upstream-first policy — fix on main, then cherry-pick down. (That environment-branch detail is from GitLab's own docs, which I could only reach single-source at write time — treat the specifics as GitLab-flavored, not gospel.)

Comparison (Table 2) — pick by how you release:

You should now have: a trunk that's always releasable, short-lived branches, feature flags for incomplete work, and a just-in-time release-branch recipe. That's the substrate the artifact rides on.

Rollback note. Branching is cheap to change — the risk isn't the model, it's mid-flight half-adoption. Switch the whole team at once; a repo running git-flow and trunk-based development simultaneously gets the downsides of both.

Phase 2 — Build the artifact once, then promote the same bytes through every environment

Here's the single most important rule in the whole guide, and the one most often broken: build once, deploy many. You build one immutable, versioned artifact at merge time, push it to a registry, and then promote that exact artifact — same digest — through dev, staging, and production. You never rebuild it per environment.

Why rebuilding per environment quietly destroys traceability

If you rebuild for staging and again for prod, you now have three artifacts built at different times, possibly against different transitive dependency versions, on different runners. The thing you tested is not the thing you shipped — and you can't prove otherwise. Google's hermetic build principle exists precisely so a build is reproducible and self-contained; promotion is how you turn that reproducibility into an operational guarantee. The practitioner consensus is blunt about the payoff: consistency across environments, faster deploys, and a clean audit trail, because there are simply fewer moving builds to go wrong.

Where the environment differences go — runtime config, not rebuilds

The obvious objection: environments are different — different database URLs, feature toggles, secrets. Correct, and that's exactly why config must be injected at deploy/run time, not baked in at build time. Environment variables, mounted config, a config service — the artifact reads them at startup. Fowler's related warning: the environment branch (encoding per-env differences as source commits) is an anti-pattern for the same reason, because it makes the code differ across environments instead of the config.

Promotion in practice — immutable digest, moved not remade

Promotion is a metadata operation: you take the artifact already in the registry and mark it approved for the next environment. In an OCI registry you promote by digest (@sha256:…), not by a tag, because tags are mutable and a digest is the artifact's identity. A GitOps flow expresses each environment as its own config in git and promotes by updating the pinned digest via a pull request — which doubles as your audit record.

Build once, then promote the same digest; never re-docker build for prod:

# BUILD ONCE at merge — capture the immutable digest
docker build -t registry.example.com/app:1.4.3 .
docker push  registry.example.com/app:1.4.3
DIGEST=$(docker inspect --format='{{index .RepoDigests 0}}' \
         registry.example.com/app:1.4.3)   # registry.example.com/app@sha256:abc...

# PROMOTE the same bytes — no rebuild. Just re-tag the digest per env.
crane copy "$DIGEST" registry.example.com/app:staging
crane copy "$DIGEST" registry.example.com/app:prod     # identical artifact

# Config is injected at DEPLOY time, not build time:
#   kubectl set env / configMap / env vars — the image never changes.

RC → GA — the promotion path as a lifecycle

The immutable artifact is your release candidate (RC) the moment it's built. Promotion is the RC earning trust: it passes dev, gets promoted to staging, passes there, and only then is promoted to production and blessed as GA. This maps cleanly onto Fowler's maturity branch idea — the “head” at each maturity level (QA, staging, prod) is just a pointer to the latest artifact that reached that stage. If your release cadence is fixed rather than continuous, wrap this in a release train: releases leave on a schedule, and a feature either makes the train or catches the next one. Progressive delivery (canary, blue-green) then rides on top — same artifact, gradually widened exposure.

If you just need a working mental model, you can stop here — trunk plus build-once-promote-many is a real, defensible release process. The rest is for people who need to prove integrity and coordinate many components: signing, provenance, and the versioning mesh.

Phase 3 — Prove where the artifact came from: signing and provenance

You have one traceable artifact per release. Now make it tamper-evident and provable — because a green pipeline and a valid signature are not the same as knowing the artifact is what you think it is.

Why this is non-negotiable — SolarWinds and xz

Two incidents rewrote how the industry thinks about this. In SolarWinds (2020), attackers compromised the build system and injected the SUNBURST backdoor into Orion updates that were then signed with SolarWinds' legitimate code-signing certificate. Roughly 18,000 organizations received the tampered update. The lesson is uncomfortable: a signature only proves who published the artifact, not what happened inside the build. A valid signature on a compromised build is worthless.

In xz utils (CVE-2024-3094, 2024), a malicious actor spent years social-engineering their way into maintainership of a core open-source library and slipped a backdoor into the release tarballs — code that wasn't in the clean git tree. The defense against both isn't more trust; it's build provenance: a signed, machine-verifiable record of exactly how and from what the artifact was built.

Signing, the easy part — Cosign keyless

Sigstore's Cosign makes signing genuinely a two-line job, and the keyless flow removes the worst part of signing — managing long-lived private keys. Keyless signing uses your CI's OIDC identity to get a short-lived certificate from Fulcio, signs, and records the signature in the Rekor transparency log. No key to leak.

Keyless sign in CI, then verify by identity (not by a stored public key):

# SIGN (keyless) — OIDC identity from the CI job; short-lived Fulcio cert,
# signature logged in Rekor. No private key to manage.
cosign sign registry.example.com/app@sha256:abc...

# VERIFY — you assert WHO signed it and WHICH issuer, not "trust this key"
cosign verify registry.example.com/app@sha256:abc... \
  --certificate-identity=https://github.com/acme/app/.github/workflows/release.yml@refs/tags/v1.4.3 \
  --certificate-oidc-issuer=https://token.actions.githubusercontent.com

# Key-based / KMS is still supported if you need it:
cosign sign --key awskms://<arn> $IMAGE

Notice what verification checks: not “is this signed by a key I trust,” but “was this signed by this workflow, on this tag, via this OIDC issuer.” That identity binding is the point.

Provenance, the important part — SLSA levels

SLSA (Supply-chain Levels for Software Artifacts) turns “prove the build” into a graded target. Provenance, in SLSA's words, is documentation of “how the artifact was built, including the build platform, build process, and top-level inputs.” The build track has four levels:

SLSA v1.0 build track (Table 3):

The SolarWinds attack is exactly the L3 threat — tampering inside the build. That's why “just sign it” (roughly L2) isn't enough for a serious threat model; you want provenance from a hardened builder.

What provenance actually looks like

SLSA provenance rides inside an in-toto attestation — in-toto being the CNCF-graduated “framework to secure the integrity of software supply chains” that makes transparent “what steps were performed, by whom and in what order.” The statement names the artifact by digest and carries a predicate describing the build:

{
  "_type": "https://in-toto.io/Statement/v1",
  "subject": [
    { "name": "app", "digest": { "sha256": "abc..." } }
  ],
  "predicateType": "https://slsa.dev/provenance/v1",
  "predicate": {
    "buildDefinition": {
      "buildType": "https://example.com/buildtype/v1",
      "externalParameters": {
        "repository": "https://github.com/acme/app",
        "ref": "refs/tags/v1.4.3"
      },
      "resolvedDependencies": [
        { "uri": "git+https://github.com/acme/app@refs/tags/v1.4.3",
          "digest": { "gitCommit": "7fd1a60b..." } }
      ]
    },
    "runDetails": {
      "builder": { "id": "https://github.com/actions" },
      "metadata": { "invocationId": "build-12345" }
    }
  }
}

Read it top-down and you have your traceability guarantee in machine-verifiable form: this digest was built from this repo at this ref and commit by this builder. That's the SolarWinds question, answered by construction.

Wiring it into a real pipeline — npm and GoReleaser

You rarely hand-write provenance. Ecosystem tooling generates and signs it for you, provided the build runs on a supported cloud CI.

• npm: npm publish --provenance generates a provenance attestation linking the package to its source and build, signs it via Sigstore, and logs it publicly. It only works from a supported cloud CI (GitHub Actions, GitLab) — you can't fake it locally. Consumers verify with npm audit signatures. One honest caveat straight from npm's docs: “Established provenance does not guarantee the package has no malicious code.” Provenance is traceability, not safety — don't oversell it.
• GoReleaser: signs archives, checksums, and Docker manifests with Cosign (v3 combines cert + signature into a single .sigstore.json bundle) and generates SLSA provenance via slsa-github-generator. Consumers check it with slsa-verifier, which verifies the cryptographic signature and that the source repo matches. The workflow needs id-token: write to sign and contents: write to attach the assets.

Rollback note. Signing/provenance is additive — turning it on can't break existing consumers, but turning on enforcement (rejecting unsigned artifacts at admission) can. Roll out verification in warn-only mode first, then enforce.

Phase 4 — Version a mesh of interdependent components without breaking their consumers

This is the phase people underestimate, and it's where release engineering gets genuinely hard. One artifact is easy. A set of services and libraries that depend on each other, released on different cadences, consumed by teams you don't control — that's the real job.

Start with SemVer, but know what it can't promise

SemVer 2.0.0 is the shared vocabulary: MAJOR for incompatible API changes, MINOR for backward-compatible additions, PATCH for backward-compatible fixes; 0.y.z means “anything may change.” Use it. But understand its ceiling, because your whole coordination strategy depends on how much you trust it.

Google's Software Engineering at Google puts it sharply: “SemVer is a lossy-compression shorthand estimate for ‘How risky does a human think this change is?’” It's an estimate, not a guarantee, and it fails in two directions. It overconstrains: change an internal API nobody uses and you still must bump MAJOR, forcing needless incompatibility. And it underconstrains, via Hyrum's Law — “with a sufficient number of users, every observable behavior of your system will be depended upon by someone” — so a “safe” PATCH breaks people anyway. Plan for SemVer to be wrong sometimes; don't build a system that assumes it's a contract.

The hard shape — the diamond-dependency problem

A diamond dependency conflict is when two libraries in your tree depend on a common library at incompatible versions, and no single version of that common library satisfies everyone — so no working set of versions exists. The classic shape: A depends on B and C; B needs D:1, C needs D:2. Pick either D and something breaks.

What makes it genuinely hard is where the fix has to come from. The root (A) can't fix it — it doesn't control D's version, its libraries do. The leaf (D) can't fix it — it can't retroactively be two versions. The fix must come from an intervening library (B or C) re-releasing against a compatible D. That's why diamond conflicts are slow to resolve: they require coordinated action from parties who each think it's someone else's problem. Microservices and monorepos amplify it — more edges, more diamonds.

Strategy A — the monorepo: turn a dependency problem into a source-control problem

Google's stated preference is telling: “prefer source control problems over dependency-management problems.” Inside a monorepo you have full visibility, so you can coordinate versions atomically and even go “live at head” — no internal versions at all, providers test against every consumer before committing. That eliminates the diamond entirely for internal deps, at the cost of a big testing/CI burden.

For libraries you publish out of the monorepo, Changesets is the pragmatic tool. A contributor runs npx changeset, declares a bump type per package and a changelog note, and commits that intent. At release, changeset version consumes all changesets, bumps each package to the right SemVer, writes changelogs, and — crucially — bumps the dependents of changed packages automatically. Then changeset publish pushes anything ahead of the registry.

The important design choice is how coupled your versions are:

• Independent (default) — each package versions on its own. Least coupling; best when packages evolve at different rates.
• linked — a group shares a version number, but only the packages that actually changed get bumped and published.
• fixed — a group shares a version and all of them bump and publish together, even unchanged ones. This is lockstep. Lerna's fixed mode does the same.

There's also updateInternalDependencies (patch vs minor), which controls whether bumping a depended-on package widens the depender's range — a direct lever on how much your consumers can deduplicate.

Choosing coupling deliberately in .changeset/config.json:

{
  "$schema": "https://unpkg.com/@changesets/config/schema.json",
  "changelog": "@changesets/cli/changelog",
  "commit": false,
  "access": "public",

  // fixed: this whole group ALWAYS bumps + publishes together (lockstep)
  "fixed": [["@acme/design-system-*"]],

  // linked: share a version NUMBER, but only changed packages publish
  "linked": [["@acme/api-client", "@acme/api-types"]],

  // controls whether an internal-dep bump widens the consumer's range
  "updateInternalDependencies": "patch"
}

Strategy B — multi-repo: pin, range, and release in trains

When components live in separate repos with separate release cadences, you can't atomically version them. Instead:

• Pin, then widen deliberately. Consumers pin to a known-good version and widen to a compatibility range only when they've verified it. This is Minimum Version Selection's insight, which Google endorses: selecting the minimum satisfying versions yields “high-fidelity builds… as close as possible to the ones the author developed against” — the versions the author actually tested, not the newest ones a SAT solver could find.
• Release in trains, not big bangs. When several interdependent components must move together, use a scheduled release train — components hit a known compatibility baseline on a cadence — rather than a “big-bang” flag day where you cut every component at once and pray. Big-bang coordinated releases fail because a single component's regression blocks the entire set.
• Automate the version derivation. Both semantic-release and release-please derive versions from Conventional Commits — fix: → PATCH, feat: → MINOR, feat!/BREAKING CHANGE: → MAJOR — so the version isn't a human guess. release-please's model is especially nice for coordination: it maintains an always-up-to-date Release PR per repo that you merge when ready, so the release is reviewable before it happens.

Commit type drives the bump; the tool decides the number:

fix(api): handle empty payload        # -> PATCH  (semantic-release / release-please)
feat(api): add cursor pagination      # -> MINOR
feat(api)!: drop v1 auth header       # -> MAJOR  ("!" or a BREAKING CHANGE: footer)

# release-please then keeps a "Release PR" open; merging it tags the release.
# semantic-release runs on CI after tests pass and publishes automatically.

Choose by topology (Table 4):

You should now have: a versioning strategy matched to your topology — atomic coordination where you have visibility, disciplined pinning and trains where you don't, and automated, commit-derived version numbers so releases don't hinge on someone remembering to bump.

Hardening for production and scale

Once the four phases are in place, the extensions that separate a demo from a real system:

• Hermetic, reproducible builds. Pin toolchains and dependencies so a build is “insensitive to the libraries and other software installed on the build machine.” Reproducibility is what lets provenance be verified, not just asserted.
• Enforce provenance at admission. Move from signing to requiring valid provenance before an artifact can be promoted to prod — but roll it out warn-only first (see the Phase 3 rollback note).
• Policy enforcement + self-service. Access control over who can approve, build, and deploy (Google's fourth principle), delivered through tooling so teams stay autonomous rather than filing tickets.
• Progressive delivery on the promoted artifact. Canary and blue-green ride on top of build-once-deploy-many — same digest, gradually widened exposure — so a bad release is caught before full rollout.

Common pitfalls — the anti-patterns that quietly undo all of this

• Rebuilding per environment. The cardinal sin. The thing you tested is no longer the thing you ship, and traceability is gone. Build once, promote the digest.
• Mutable latest tags. A tag that moves is not an identity. Promote and deploy by immutable digest; reserve human-readable tags for convenience, never for what's actually running.
• Trusting a signature as if it were provenance. SolarWinds was validly signed. A signature proves who published; provenance proves how it was built. You need both.
• Unsigned artifacts in the promotion path. If any hop accepts unsigned or unverifiable artifacts, the chain is only as strong as that hop.
• Lockstep-versioning everything. Fixed/lockstep versioning (bumping unchanged packages) is right only when the group is genuinely one product. Applied broadly, it forces meaningless releases and hides which component actually changed.
• Big-bang coordinated releases. Cutting every interdependent component at once means one regression blocks the whole set. Use release trains and pinning so components move on a known baseline, not a flag day.
• Long-lived branches under a CD banner. Running git-flow's develop/release branches while claiming continuous delivery gives you late, fearful merges — the exact thing CD is supposed to remove.
• Encoding environments as branches. The environment-branch anti-pattern makes code differ per environment; put the differences in runtime config instead.

What to do on Monday

Starting from nothing? Get on a trunk with short-lived branches and feature flags first — it unblocks everything else. Then make your CI build one immutable artifact and promote the digest instead of rebuilding; that alone buys you real traceability. Add Cosign keyless signing (two lines) and turn on your ecosystem's provenance (npm publish --provenance, or GoReleaser + slsa-github-generator) in warn-only mode. Only once that's solid, invest in the versioning mesh — Changesets if you're a monorepo, Conventional-Commit automation and release trains if you're multi-repo. Do them in that order; each phase assumes the last.

The whole system is in service of one sentence you should be able to say without flinching: “Here's the commit this artifact came from, here's the signed proof of how it was built, and here's exactly how its version relates to everything that depends on it.”