Argo CD's control plane, taken apart
A design-level tour of how Argo CD’s control plane is built — the components, the reconcile loop, and no database of its own.
TL;DR
- One idea, factored into small services. Desired state lives in git, live state lives in your clusters, and the application-controller closes the gap between them. Every other component exists to serve that one loop.
- There is no database. The only durable state is git plus each cluster's own etcd — where the
Applicationresources and the live objects live. Redis is a throwaway cache you can delete at any time. - The loop runs two ways. A git webhook refreshes the one changed app instantly; a periodic timer re-reconciles everything on a schedule as the safety net. Same reconcile, two triggers.
- It scales by adding replicas, not by rearchitecting. The API server is stateless, the repo-server is horizontally scalable, and the application-controller shards its managed clusters across a StatefulSet.
- The pieces degrade independently. Lose Redis and you get a slow cache rebuild, not an outage. Lose the repo-server and syncs stall but nothing is destroyed. Any one component's blast radius is small by design.
- v3.x added a genuinely new component. The commit-server, behind the Source Hydrator, is the first piece with write access to git — split out so that write credential lives in exactly one place.
At a glancethe whole control plane in one diagram
Before we take the components apart one by one, here they all are at once — every service in the control plane, plus the two things it treats as the source of truth: Git (desired state) and your managed clusters (live state). The numbered path traces a single change from the front door to the cluster; the sections that follow walk the same path one box at a time.
| API server | Deployment | The front door — Web UI, CLI, and CI/CD reach everything through it (gRPC/REST, auth, RBAC). |
| Dex | Deployment | Bundled SSO federation that authorises those callers (skip it if you wire an OIDC provider directly). |
| repo-server | Deployment | Clones Git and renders the target manifests (Helm / Kustomize / Jsonnet / plugins). |
| application-controller | StatefulSet | The engine — pulls the rendered manifests, diffs desired against live, and syncs. |
| Redis | Deployment | Disposable cache for rendered / live / diff results; lose it and it rebuilds. |
| Managed clusters | external | Where the diff is applied — the live state lives in each cluster’s etcd, not in Argo CD. |
| ApplicationSet controller | Deployment | Templates one Application per target and hands them to the controller. |
| notifications controller | Deployment | Turns state changes into messages (Slack, webhooks, …). |
| commit-server | Deployment | New in v3.x — renders hydrated manifests back into Git (the only component with git write access). |
API serverthe stateless gateway every client goes through
The argocd-server is the gRPC/REST gateway — what the Web UI, the argocd CLI, and your CI/CD systems talk to, and the only component most humans ever touch directly.
Its responsibilities are broad but shallow: application management and status reporting, invoking operations (sync, rollback, user-defined actions), managing repository and cluster credentials (as Kubernetes secrets), authentication (delegated to external identity providers), RBAC, and forwarding git webhook events.
Dexoptional SSO federation
The argocd-dex-server is a bundled OIDC provider that federates your identity system into Argo CD — because not every provider speaks OIDC cleanly, and not every OIDC provider exposes what Argo CD needs (like group membership).
The decision:
- Use direct OIDC (the
oidcConfigsetting, no Dex) when you already run an OIDC-compliant provider — Okta, Auth0, Keycloak, Google, Microsoft Entra. It's simpler: one fewer component, and Argo CD talks to your IdP directly. - Use Dex when your provider doesn't do OIDC (SAML, LDAP), or when you want a connector feature Dex provides — the classic one being mapping GitHub organizations and teams into OIDC group claims for RBAC. Dex can also fetch groups the IdP won't put in the ID token (Google is the usual example).
Repo-serverturns Git into rendered manifests
The argocd-repo-server answers one question: given this repo, revision, and path, what are the Kubernetes manifests? It caches the rendered output in Redis keyed by repo, revision, and path, so repeated reconciliations of an unchanged source cost nothing (--repo-cache-expiration, 24h default).
| 1 | repo-server clones the repo at the requested rev / path |
| 2 | renders the manifests with the matching tool (Helm / Kustomize / Jsonnet / plugin) |
| 3 | caches the rendered output in Redis, keyed repo/rev/path (24h) |
| 4 | serves the manifests to the app-controller over gRPC |
The same app three ways — what you write, what lives in Git, and what the repo-server hands the controller:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: web
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/acme/deploy.git
targetRevision: main
path: apps/web # repo-server clones + renders this
destination:
server: https://kubernetes.default.svc
namespace: web
syncPolicy:
automated: # auto-sync + self-heal
selfHeal: true
prune: true# apps/web/ in the git repo -- a Helm chart (config kept DRY)
apps/web/
Chart.yaml
values.yaml # replicaCount: 3, image.tag: v1.4.2
templates/
deployment.yaml
service.yaml# what repo-server returns over gRPC, cached in Redis
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
spec:
replicas: 3
template:
spec:
containers:
- name: web
image: ghcr.io/acme/web:v1.4.2Application-controllerthe reconciliation engine
The argocd-application-controller is where reconciliation actually happens — a Kubernetes controller that, for every Application, runs the loop the figure below traces.
An Application is not a workload
One term to pin down before we go on: an Application. It sounds like your running app, but it isn’t one. An Application is an Argo CD custom resource — a small record that says “keep this git path synced to this cluster and namespace.” It runs no containers. When Argo syncs it, it applies ordinary Kubernetes objects — a Deployment, a ReplicaSet, some Pods — and those run your code.
The two even live in different places. The Application objects are Kubernetes resources in the argocd namespace, generated from git and reconciled by the application-controller. The Deployment → ReplicaSet → Pods they produce live in the target cluster, run by Kubernetes’ own controllers. Argo just keeps that second set matching git — and reports Synced (matches git?) and Healthy (Pods up?).
Application is a control-plane pointer, not a workload. Config lives in the GitOps repo; Argo’s ApplicationSet turns it into Application objects in the argocd namespace; each Application applies its manifests to the target cluster’s Kubernetes API, and Kubernetes’ own controllers turn them into the real Deployment / ReplicaSet / Pods. Argo never runs a container itself.Two triggers, one loop
The application-controller runs the same reconcile on two triggers. An event — a Git webhook, or an edit to an Application spec — refreshes just the affected app, so a push lands in seconds. A periodic timer (roughly every three minutes by default) re-syncs every app regardless, catching drift the events missed. The figure below contrasts the two.
A change, end to end
When the webhook fires, the change threads through the whole control plane: the API server records it, the repo-server renders the manifests, and the application-controller diffs desired against live and syncs the delta into the cluster. The numbered path traces one commit from Git to running pods, alongside the ApplicationSet flow that generated the Application in the first place.
| Reconcile loop (A) — keeps each Application in sync with Git | |
| A1 | a git push fires a webhook to the API server |
| A2 | the API server marks the Application for refresh in etcd, which invalidates the repo-server’s cached manifests |
| A3 | the application-controller picks up the refreshed Application from etcd |
| A4 | the controller requests a render from the repo-server (repo, path, revision) |
| A5 | the repo-server clones / fetches Git and renders the manifests |
| A6 | the repo-server returns the manifests to the controller and caches them in Redis |
| A7 | the controller reads the cluster’s live state via the watch on the Kubernetes API |
| A8 | the controller computes the three-way diff (desired vs live vs last-applied) |
| A9 | the controller applies the manifests to the cluster’s Kubernetes API (sync) |
| A10 | Kubernetes creates Deployment → ReplicaSet → Pods |
| A11 | the controller writes the Application status back to etcd |
| ApplicationSet flow (B) — creates the Applications the A-loop then reconciles | |
| B1 | the ApplicationSet controller reads its ApplicationSet CR from etcd (it is a Kubernetes object too) |
| B2 | it runs its generators — the git generator reads Git via the repo-server (current Argo CD versions) and the cluster generator reads the cluster list |
| B3 | it produces one parameter set per match |
| B4 | it renders the template and creates one Application per set into etcd, each owned by the ApplicationSet — those Applications are what the A-loop reconciles |
Application in sync, while the ApplicationSet controller’s B1–B4 flow reads its ApplicationSet CR and creates those Applications. The Application CRs live in the cluster’s etcd; Redis is a separate cache for the repo-server’s rendered manifests. Git reaches Argo only through the API server’s webhook, and Argo reads Git through the repo-server.The periodic sweep
The same machinery also runs on a clock. Instead of one webhook refreshing one app, the timer sweeps every managed Application in turn — re-rendering and re-reconciling each so nothing drifts silently. It is the bulk safety net to the webhook’s targeted, on-demand refresh.
Periodic reconcile (A) — the application-controller sweeps every Application on its own timer | |
| A1 | the application-controller’s per-app reconcile timer fires (~3 min) for this Application |
| A2 | it reads the Application from etcd via its informer cache |
| A3 | it asks the repo-server to render (repo / path / revision) |
| A4 | the repo-server clones / renders — a cache hit in Redis if the commit SHA is unchanged |
| A5 | the repo-server returns the manifests and caches them in Redis |
| A6 | the controller reads the cluster’s live state via the watch |
| A7 | it computes the three-way diff (desired vs live vs last-applied) |
| A8 | no-op if in sync — only if it drifted does it apply to the cluster’s Kubernetes API |
| A9 | Kubernetes creates Deployment → ReplicaSet → Pods (only if applied) |
| A10 | the controller writes the Application status back to etcd |
| ApplicationSet flow (B) — creates the Applications the A-loop then reconciles | |
| B1 | the ApplicationSet controller reads its ApplicationSet CR from etcd (it is a Kubernetes object too) |
| B2 | it runs its generators — the git generator reads Git via the repo-server (current Argo CD versions) and the cluster generator reads the cluster list |
| B3 | it produces one parameter set per match |
| B4 | it renders the template and creates one Application per set into etcd, each owned by the ApplicationSet — those Applications are what the A-loop reconciles |
Application on its own schedule — it renders through the repo-server (a Redis cache hit when the commit SHA is unchanged), computes the three-way diff, and re-applies only what drifted (a no-op when in sync). The ApplicationSet flow B1–B4 still creates the Applications the timer then reconciles; the Application CRs live in etcd and Redis stays a separate manifest cache.Redisthe disposable cache
Redis holds the caches: rendered manifests, observed live state, and diff results. The docs are blunt — it is "only used as a disposable cache and can be safely rebuilt without service disruption."
If Redis vanishes, nothing permanent breaks: the controller and repo-server rebuild their caches from git and the live clusters. You get a brief load spike on git and the Kube API while caches warm, and slower UI/reconciles for a few minutes. No Application is lost, no sync undone, no desired state forgotten — none of it ever lived in Redis.
ApplicationSet controllerone template, many Applications
A raw Application describes one deployment to one place — it doesn't scale to "this app on all 40 clusters." The ApplicationSet controller (its own controller, a separate project until it merged into core Argo CD in v2.3 — initially beta) owns the ApplicationSet CRD and generates, updates, and deletes Application resources from a single template. v2.3+
The power is in the generators — they produce the parameter sets substituted into the template, one Application per set (Matrix / Merge compose two generators, e.g. every app from Git × every cluster from Cluster). The ApplicationSet controller doesn't reconcile against clusters itself.
| 1 | a generator produces a set of parameters (List / Cluster / Git / PR / SCM / Matrix) |
| 2 | the ApplicationSet controller renders the shared template into one Application per set |
| 3 | the application-controller then syncs each generated Application like any hand-written app |
Zoom in on where those Applications come from — the same web fleet, seen from the ApplicationSet controller’s side:
| 1 | reads the ApplicationSet manifest (template + generator) and the cluster list from Git |
| 2 | runs its generator → one parameter set per matching cluster (prod-us, prod-eu) |
| 3 | renders the template once per parameter set |
| 4 | creates one Application object per set (web-prod-us, web-prod-eu), each owned by the ApplicationSet |
| 5 | hands off → the application-controller then reconciles each Application (the reconcile loop) |
ApplicationSet — a template plus a generator — together with the cluster list into many Application objects, one per matching cluster. It does not sync anything; the application-controller does.One ApplicationSet fans a single template out into one Application per cluster:
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: web-fleet
namespace: argocd
spec:
generators:
- clusters: {} # one param set per registered cluster
template:
metadata:
name: 'web-{{name}}' # {{name}} / {{server}} filled per cluster
spec:
project: default
source:
repoURL: https://github.com/acme/deploy.git
targetRevision: main
path: apps/web
destination:
server: '{{server}}'
namespace: web# the controller renders one Application per cluster the generator saw,
# then syncs each exactly like a hand-written app
web-eu-prod -> https://eu-prod.example.com
web-us-prod -> https://us-prod.example.com
web-ap-staging -> https://ap-staging.example.com| 1 | you apply the Root Application once to etcd (kubectl / CI / Terraform) — the only manual step; GitOps cannot create its own entry point |
| 2 | the application-controller syncs the Root Application (rendered from Git via the repo-server) and applies its manifests — which are ApplicationSet CRs — into etcd |
| 3 | the ApplicationSet controller reads each ApplicationSet CR, runs its generators, and creates one child Application per match into etcd |
| 4 | the application-controller reconciles each child Application and deploys its workloads (Deployment → ReplicaSet → Pods) to the target cluster |
Notifications controllerturning state changes into messages
The notifications controller (originally the separate argocd-notifications project, now folded into core) watches Application state and emits notifications. Its model is two-part:
- Triggers define when. A trigger is a named condition — a predicate expression (evaluated by the
antonmedv/exprengine) — that returns true when a notification should fire, plus a reference to the templates to use. TheoncePerfield deduplicates, so you get one message per observed git revision rather than one per reconcile. - Templates define what. A template (Go
html/template, configured in theargocd-notifications-cmConfigMap) renders the message body. Templates are reusable and referenced by multiple triggers.
The rendered message goes out through a configured service — Slack, a generic webhook, email. Like Dex, it sits off the critical path: if it's down, deployments still reconcile; you just aren't told about them.
Commit-serverwrites hydrated manifests back to Git (v3.x)
The newest and most architecturally interesting addition. Be precise about its status: the Source Hydrator is an alpha feature, introduced in the v3.x line and actively evolving through v3.1 and v3.3. v3.x, alpha, re-verify
The problem it solves. Helm and Kustomize keep config DRY but hide the actual manifests that hit the cluster — when something breaks at 2am you have to reproduce the render locally to see it. The Source Hydrator implements the "rendered manifest pattern": it renders the templates and commits the plain manifests back to a git branch before syncing, so git shows exactly what will be applied.
The flow uses three fields on spec.sourceHydrator:
drySource— where the templates live (repoURL, path, targetRevision). The "dry" (DRY) source.syncSource— the branch where hydrated manifests are pushed and then synced from.hydrateTo— an optional staging branch, so hydrated output can go through review/approval before it lands on the sync branch.
Output must be deterministic — the same dry commit must always hydrate to the same manifests, so no unpinned dependencies or non-deterministic template functions. (And don't pair it with secret-injecting tools like Helm+SOPS, or you'd commit secrets to git; use a secrets operator on the destination instead.)
Why the commit-server is its own component
Pushing to git needs write credentials, and every other component that touches git — the repo-server — only ever reads. Rather than hand write access to the read component, v3.x introduced a dedicated argocd-commit-server whose entire job is git writes for hydrated manifests.
It exposes a single gRPC call, CommitManifests, carrying the target repo and branch, the dry commit's provenance (full SHA256, author, message, time), and the rendered manifests. The controller builds the request; the commit-server performs the push.
The rationale is blast radius. Git write credentials now live in one small, single-purpose service instead of being smeared across the component that also clones arbitrary repos and runs templating tools. If something gets push access to your source of truth, you want it to be the least-exposed component you have — the whole reason the commit-server is a separate box.
| 1 | repo-server reads the DRY templates from drySource |
| 2 | renders them and hands the plain manifests to the commit-server |
| 3 | commit-server commits the hydrated YAML back to git — CommitManifests gRPC, the only git-write creds |
| 4 | app-controller syncs the hydrated syncSource branch |
| 5 | and applies the plain manifests to the cluster |
The Source Hydrator in YAML — the spec, the DRY template on main, and the plain manifest the commit-server writes to the sync branch:
spec:
sourceHydrator:
drySource: # templates live here (DRY)
repoURL: https://github.com/acme/deploy.git
path: apps/web
targetRevision: main
syncSource: # plain YAML is committed + synced here
targetBranch: env/prod
path: apps/web
hydrateTo: # optional staging branch for review
targetBranch: env/prod-next# apps/web/values.yaml on main -- the actual manifests are hidden
# behind Helm templating until they are rendered
replicaCount: 3
image:
repository: ghcr.io/acme/web
tag: v1.4.2# apps/web/manifests.yaml on env/prod -- plain YAML the
# commit-server wrote; git now shows exactly what hits the cluster
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
spec:
replicas: 3
template:
spec:
containers:
- {name: web, image: ghcr.io/acme/web:v1.4.2}The state modelthree stores of truth, one loop
Argo CD is a GitOps controller, and it keeps its truth in exactly three stores. Git holds the desired state — the manifests, or the templates that render into them. The managed cluster’s own etcd holds the live state: the running Kubernetes objects and the Application custom resources both live there. And Redis is a disposable cache of rendered manifests — throwaway, safe to delete, never a source of truth. Git and the cluster’s etcd are external to Argo CD; only Redis is its own.
One reconcile loop runs across all three. The application-controller (1) reads the desired manifests from Git — rendered by repo-server, (2) reads the live state back from the cluster’s etcd, (3) diffs desired against live, (4) syncs the delta into the cluster, and (5) caches the rendered manifests in Redis so the next pass stays cheap. That’s the whole storage story: no relational database, no external queue, no object store. Wipe every Argo CD pod and its Redis, reinstall, and it rebuilds its entire world from Git and the clusters — which is what makes everything else as disposable as it is. Every component below either feeds that loop, runs it, or reacts to its output.
| 1 | application-controller reads the desired manifests from Git — rendered by repo-server |
| 2 | reads the live state back from the managed cluster’s etcd |
| 3 | diffs desired against live inside application-controller |
| 4 | syncs the delta into the cluster, applying desired over live |
| 5 | caches the rendered manifests in Redis to keep the next pass cheap |
Application CRs, Redis is a throwaway cache — and one controller loop closes the gap. Nothing durable lives inside Argo CD.Putting it togetherone change through the whole loop
| 1 | a developer pushes a commit — a webhook (or 3-min poll) notifies the API server |
| 2 | the API server marks the Application for refresh in etcd, which invalidates the repo-server’s cached manifests |
| 3 | the application-controller requests a render and the repo-server clones Git and renders the desired manifests over gRPC |
| 4 | the controller diffs desired against the live watch cache (three-way) via the Kubernetes API |
| 5 | it syncs — applying the manifests to the cluster, which creates Deployment → ReplicaSet → Pods |
| 6 | Redis caches the rendered manifests throughout — a cache, never a source of truth |
| 7 | notifications fire — a trigger sends a message |
Application CRs live in the cluster’s etcd; Redis is only a cache for the repo-server’s rendered manifests. Nowhere in the path is there a write to an Argo CD datastore, because there isn’t one.Nowhere in that path is there a write to an Argo CD datastore, because there is no such datastore. The desired state was already in git; the live state ends up in the cluster's etcd; Redis just made it faster.
In productionthe HA topology worth running
Argo CD ships HA manifests for production that encode the scaling story component by component:
| Component | Shape | HA move |
|---|---|---|
| API server | Stateless Deployment | 3+ replicas; set ARGOCD_API_SERVER_REPLICAS to match |
| Repo-server | Deployment | Scale replicas for render throughput |
| Application-controller | StatefulSet | Add replicas as the managed fleet grows |
| Redis | Cache | HA mode: Redis + Sentinel (3 instances) behind HAProxy |
| Dex | In-memory | Single replica (multi-replica is inconsistent); off the sync path |
| ApplicationSet / Notifications | Controllers | Run them; both are off the critical sync path |
| Commit-server v3.x | gRPC service | Only if you use the Source Hydrator; holds git write creds |
Two practical constraints in the HA install: it wants at least three nodes (pod anti-affinity rules spread the replicas), and IPv6-only clusters aren't supported.
Watch-outsthe pitfalls to plan for
- A cold Redis is a thundering herd. Losing Redis is safe but not free — a large fleet warming its cache can spike load on git and the Kube API. HA Redis avoids the event entirely.
- The repo-server serializes mutating renders. One clone per instance plus single-concurrency mutation means monorepos with heavy templating bottleneck here first. Scale repo-server replicas.
- The Source Hydrator is alpha and must be deterministic. Non-deterministic renders will commit noise to git on every pass, and secret-injecting tools will leak secrets into git. Don't adopt it casually. re-verify status
GroundingSources
- Argo CD docs — Architectural Overview: https://argo-cd.readthedocs.io/en/stable/operator-manual/architecture/
- Argo CD docs — Component Architecture (developer guide): https://argo-cd.readthedocs.io/en/stable/developer-guide/architecture/components/
- Argo CD docs — High Availability: https://argo-cd.readthedocs.io/en/stable/operator-manual/high_availability/
- Argo CD docs — Source Hydrator (v3.0): https://argo-cd.readthedocs.io/en/release-3.0/user-guide/source-hydrator/
- Argo CD docs — Commit Server proposal: https://argo-cd.readthedocs.io/en/latest/proposals/manifest-hydrator/commit-server/
- Argo CD docs — User Management / SSO (Dex vs OIDC): https://argo-cd.readthedocs.io/en/stable/operator-manual/user-management/
- Argo CD docs — ApplicationSet controller & Generators: https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/
- Argo CD docs — Notifications (triggers & templates): https://argo-cd.readthedocs.io/en/stable/operator-manual/notifications/
- DeepWiki (argoproj/argo-cd) — State Comparison and Diff Engine: https://deepwiki.com/argoproj/argo-cd/3.3-state-comparison-and-diff-engine
- CNCF — Argo CD v3 announcement (Apr 2025): https://www.cncf.io/announcements/2025/04/01/cloud-native-computing-foundation-announces-argo-cd-v3-update-to-enhance-scalability-and-security-for-kubernetes/
- InfoQ — Argo CD v3.1 (OCI support, UI): https://www.infoq.com/news/2025/08/argocd-oci-support-new-ui/