Your 2 a.m. kubectl fix reverted in ten seconds — GitOps drift control that survives an incident

It's 2:00 AM. Production is down. Checkout pods are crash-looping under a traffic spike and the on-call engineer does the obvious thing: kubectl scale deploy/checkout --replicas=10 to ride out the surge. For a few seconds the extra pods come up and error rates fall. Then, without anyone touching anything, the deployment drops back to replicas: 3 and the system falls over again.

Nobody undid the fix. Argo CD did. Its application controller noticed that the live cluster no longer matched Git, flagged the app OutOfSync, and — because selfHeal: true is set — re-applied the manifest from main five seconds later. The engineer's emergency patch was, from GitOps's point of view, drift. And drift gets reconciled away.

This is the central tension of running GitOps in production: the same self-healing that keeps 400 microservices from rotting into snowflake state is the thing that fights your responder at the worst possible moment. The answer is not to turn self-heal off. It's to design a continuous-delivery plane where absolute infra enforcement and developer autonomy don't collide — where the reconciled surface is small, the fast-changing config lives outside it, and there's a documented break-glass lever for the nights the paved road isn't enough.

Why the 2 a.m. patch vanished — the core tradeoff

Argo CD's job is to make one statement true forever: live cluster state equals the desired state declared in Git. To do that its application-controller runs a reconciliation loop. On a schedule (the app refresh, ~3 minutes by default) and on events, it renders the manifests from your Git source, diffs them against what's actually running, and computes a sync status. When the two agree, the app is Synced. When they don't — a field differs, a resource exists that Git doesn't declare, or one Git declares is missing — the app is OutOfSync. That gap is drift.

Drift has two honest sources, and it's worth separating them because the "fix" is different:

Here is the subtlety most teams miss: auto-sync and self-heal are two different triggers. Plain automated reacts to Git moving — a new commit. It does not, on its own, react to the cluster moving. Per the Argo CD docs, "changes made to the live cluster will not trigger automated sync" unless selfHeal is explicitly enabled. Self-heal is the opt-in that says "also revert cluster-side drift." When it's on, and the live state diverges, the controller re-attempts the sync after the self-heal timeout — 5 seconds by default, tunable via the --self-heal-timeout-seconds flag on the application controller. That five seconds is the exact window between the on-call engineer's kubectl scale and its silent erasure.

So the tradeoff is real, and both ends are bad:

The instinct is to dial self-heal down to escape the friction on the right. That's a mistake — it just walks you back toward state decay on the left. The way out isn't a dial between two failure modes. It's changing the shape of the problem: reconcile hard, but reconcile less surface, and give humans a real door instead of a wall.

Harden the sync policy instead of loosening it

Counterintuitively, the safe production posture is more automation, not less. A half-enforced GitOps setup — auto-sync on, self-heal off, prune off — is the worst of both worlds: it applies Git forward but tolerates cluster drift indefinitely, so you get all the rigidity of GitOps with none of the guarantees. Turn it all the way on, then carve out the exceptions deliberately.

The full syncPolicy

Here is a production Application for a multi-tenant EKS platform. Every field earns its place:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: checkout
  namespace: argocd
spec:
  project: payments
  source:
    repoURL: https://github.com/acme/platform-manifests.git
    targetRevision: main
    path: apps/checkout/overlays/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: checkout
  syncPolicy:
    automated:
      prune: true          # delete resources removed from Git (default: false)
      selfHeal: true       # revert live cluster drift back to Git
      allowEmpty: false    # refuse to sync an app down to zero resources
    syncOptions:
      - CreateNamespace=true
      - PruneLast=true               # prune only after everything else is healthy
      - ApplyOutOfSyncOnly=true      # apply only what actually changed
      - RespectIgnoreDifferences=true # honor ignoreDifferences during sync, not just diff
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

The two load-bearing flags are the dangerous-sounding ones. prune: true is off by default in Argo CD as a safety mechanism — without it, deleting a manifest from Git leaves the resource orphaned in the cluster forever, which is its own flavor of drift. selfHeal: true is the enforcement teeth from Figure 1. Together they make the statement "the cluster equals Git" actually true in both directions: nothing extra survives, nothing manual sticks.

The syncOptions are the sharp-edge guards. PruneLast=true defers deletions until every other resource is healthy, so a bad sync doesn't delete a running resource before its replacement is up. ApplyOutOfSyncOnly=true stops Argo re-applying hundreds of unchanged objects on every loop. And RespectIgnoreDifferences=true is the one people forget — it makes the ignoreDifferences rules below apply during sync, not merely in the diff view. Without it, self-heal will happily re-apply a field you told the UI to ignore.

Stop fighting other controllers: ignoreDifferences

Not all drift is a human mistake. An HPA scaling spec.replicas from 3 to 12 under load is drift by Argo's definition — live state no longer matches the replicas: 3 in Git — and blind self-heal will fight the autoscaler, yanking replicas back down mid-spike. Same story for values injected by mutating webhooks (sidecar containers, default resource limits, service-mesh annotations). These fields are meant to be owned by something other than Git.

The tool for this is ignoreDifferences. The canonical HPA case tells Argo to leave the replica count alone:

spec:
  ignoreDifferences:
    # HPA owns the replica count; don't let self-heal revert it
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas
      managedFieldsManagers:
        - kube-controller-manager
    # A mutating webhook injects a sidecar; ignore that container
    - group: apps
      kind: Deployment
      jqPathExpressions:
        - '.spec.template.spec.containers[] | select(.name == "istio-proxy")'

Two ways to point at a field. jsonPointers targets an exact path (/spec/replicas). jqPathExpressions selects list items by content — useful when a webhook appends a container you can't address by index. The managedFieldsManagers refinement is the precise one: it says "ignore this field only when kube-controller-manager owns it in metadata.managedFields," so a human hand-editing replicas is still caught as drift while the HPA is not. Pair every ignoreDifferences entry with RespectIgnoreDifferences=true in syncOptions or self-heal will ignore your ignore rule.

Make Git the only write path: RBAC

Every technique so far assumes drift is rare enough to reason about. That assumption only holds if humans can't casually change the cluster out-of-band. If half your engineers have cluster-admin and a habit of kubectl apply-ing hotfixes, self-heal will be in a permanent low-grade war and you'll never trust it.

So the reconciliation strategy is incomplete without an access strategy. Two layers:

• Kubernetes RBAC on the cluster: humans get read/exec/logs for debugging, but write verbs (create, patch, delete) on managed workloads are reserved for the Argo CD service account. The path to change a Deployment is a pull request, not a terminal.
• Argo CD RBAC on the control plane: the sync action lets a user reconcile an app to its declared Git state; the far more dangerous override action lets a user push arbitrary local manifests in place of the configured source, bypassing Git entirely. Grant override to almost no one, scope sync per AppProject, and you've closed the out-of-band door on both ends.

The payoff is conceptual, not just security hygiene: when Git is the only routine write path, drift stops being background noise. Any OutOfSync that isn't a pending commit is now genuinely anomalous — a controller doing its job, or a break-glass action someone should be paged about. That's the state you want self-heal operating in.

The developer escape hatch — autonomy without drift

A hardened paved road that has no off-ramp doesn't survive contact with reality. Developers need to rotate a secret, flip a feature flag, and dial a timeout without waiting on a five-minute reconcile — and responders need to override the machine at 2 a.m. The trick is to serve those needs without letting them become untracked cluster edits. You do that by moving the fast-changing state off the reconciled surface entirely, and by giving incidents a real, documented lever.

Decouple secrets: External Secrets Operator

Secrets are the classic reason people reach for kubectl edit: the value changes often, it must not live in Git, and rotating it shouldn't require a deploy. External Secrets Operator (ESO) solves all three by making the live Secret a projection of AWS Secrets Manager rather than a Git-managed object. A SecretStore declares the source and auth; an ExternalSecret declares what to pull and how often:

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: aws-secretsmanager
  namespace: checkout
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      # EKS Pod Identity / IRSA supplies the IAM role; no static keys in-cluster
      role: arn:aws:iam::123456789012:role/checkout-eso
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: checkout-db
  namespace: checkout
spec:
  refreshInterval: 1h          # ESO re-pulls and rotates the live Secret
  secretStoreRef:
    name: aws-secretsmanager
    kind: SecretStore
  target:
    name: checkout-db          # the native Secret ESO creates/owns
    creationPolicy: Owner
  data:
    - secretKey: password
      remoteRef:
        key: prod/checkout/db
        property: password

What's in Git is the intent (which secret to fetch), never the value. Argo reconciles the ExternalSecret and SecretStore objects; ESO owns the resulting Secret, which Argo isn't managing and therefore never flags as drift. Rotating the database password becomes an update in Secrets Manager that ESO picks up within the refreshInterval — no commit, no sync, no kubectl edit, no drift. The fast-changing thing simply isn't on the reconciled surface.

Decouple toggles: runtime parameters and feature flags

The same principle generalizes past secrets. Anything that changes on a human timescale — a feature flag, a rollout percentage, a per-tenant limit — should be read at runtime by the app, not baked into a manifest that Argo reconciles. Put those values behind a feature-flag service or a runtime-config store the application reads on the fly. Flipping a flag is then a control-plane action that takes effect in seconds and never touches a Kubernetes object, so it can't drift and can't be reverted by self-heal. Git still owns the shape of the deployment; the flag store owns the knobs. Two surfaces, no overlap.

This is the resolution to the whole tradeoff. Most of what tempts an engineer toward a live cluster edit — secrets, flags, tunables — can be designed off the reconciled surface. Do that, and self-heal almost never has anything legitimate to fight.

The one thing you can't design away: break-glass

Almost. There will still be the genuine 2 a.m. incident where the fix is a structural change Argo would revert — scaling a Deployment, deleting a wedged pod-disruption budget, patching a resource in a way Git doesn't yet reflect. For that you need an explicit, rehearsed break-glass procedure, not an improvised scramble to figure out how to make Argo stop.

Break-glass = suspend reconciliation for the single affected app, act, then restore GitOps — and backfill Git so the emergency change becomes permanent the right way:

# 1. Break glass: suspend auto-sync + self-heal for THIS app only
argocd app set checkout --sync-policy none

# 2. Do the emergency fix; self-heal will NOT revert it now
kubectl -n checkout scale deploy/checkout --replicas=10

# 3. After the incident, make the change permanent in Git
#    (open a PR that sets replicas: 10), then restore GitOps:
argocd app set checkout --sync-policy automated --self-heal --auto-prune

# Equivalent declarative form if you drive Argo via manifests:
kubectl -n argocd patch application checkout --type merge \
  -p '{"spec":{"syncPolicy":{"automated":null}}}'

Three properties make this safe. It's scoped — --sync-policy none disables reconciliation for one Application, not the fleet, so you're not ripping self-heal out of 400 services to save one. It's reversible — one command restores the exact policy. And it's auditable — it runs through Argo CD's API under RBAC, so who broke glass and when is recorded, unlike a silent kubectl patch. Restrict this action to the on-call role, put the two commands in the runbook, and rehearse them in game days so nobody is reading docs during a real outage.

Pitfalls that turn the paved road back into a minefield

• Self-heal on, RespectIgnoreDifferences off. Your ignoreDifferences rule silences the diff view, but self-heal still re-applies the field on sync and fights the HPA anyway. The rule only takes effect during sync when RespectIgnoreDifferences=true is in syncOptions. This is the single most common "why is Argo still reverting my replicas" bug.
• Over-broad ignoreDifferences. A group: '*' / kind: '*' catch-all to quiet noise hides real drift everywhere. Scope each exception to a specific kind and path.
• Prune off "for safety." Leaving prune: false feels cautious, but it lets deleted-from-Git resources linger as orphans — drift in the other direction. Use prune: true with PruneLast=true so deletions happen last, after replacements are healthy.
• Break-glass without step 3. Suspending an app and never backfilling Git leaves permanent hidden drift; the next re-sync silently reverts the emergency fix. The incident isn't closed until Git matches reality and sync is re-enabled.
• Fleet-wide disable under pressure. Disabling auto-sync on a parent/app-of-apps to stop one service freezes reconciliation for everything it manages. Scope break-glass to the single leaf Application.
• Wide-open cluster RBAC. If engineers can kubectl apply hotfixes freely, self-heal lives in a constant war and you'll never trust it. Reserve write verbs on managed workloads for Argo's service account; hand humans read/exec/logs.
• Assuming the automated schema is stable. Newer Argo CD nests automated.enabled: true; older versions treat a present automated: {} as enabled. Setting it to null (not {enabled: false}) is the reliable way to disable via patch. Check your version.