The Idea: Git as the Source of Truth#
The contract is simple: whatever is in git is what the cluster should run. Not what you applied last week, not what a CI
script pushed yesterday, not what someone edited live to debug a crash. Git. If you want to change something on the
cluster, you change it in git and let ArgoCD carry it through. The flip side: if you need to test a quick fix directly
on the cluster, you have to suspend auto-sync first — otherwise ArgoCD will revert your change on the next
reconciliation cycle. You can do this from the ArgoCD UI (App Details → Summary → Disable Auto-Sync) or via CLI
(argocd app set <app> --sync-policy none). Re-enable when done. Treat it as a temporary escape hatch, not a workflow.
This is different from treating git as a record of what happened. Here git is the input, not the output.
What We’re Deploying#
A minimal nginx app managed via Kustomize. nginx is a good choice here — it’s small, starts instantly, and its welcome page confirms the deployment is reachable. The structure looks like this:
config/sample-app/
base/
deployment.yaml # nginx:1.27, replicas: 2
service.yaml # ClusterIP on port 80
kustomization.yaml
overlays/
local/
kustomization.yaml # references ../../base
apps/sample-app/
application.yaml # ArgoCD Application CRDArgoCD will watch config/sample-app/overlays/local. When you push a change to that path, ArgoCD picks it up and syncs.
You might wonder why not just put everything in one flat folder. For a single environment that works fine. The
base/overlays split pays off the moment you have more than one environment — local, staging, production. The base
holds everything that’s the same across all of them: the Deployment definition, the Service, the container image. Each
overlay then patches only what differs — production might run 5 replicas instead of 2, staging might point to a
different image tag, local might skip resource limits entirely. You make the change once in base and all environments
pick it up. Environment-specific tweaks stay isolated in their overlay and don’t bleed into each other.
For this series we only have a local overlay, so the structure might feel like extra ceremony. It’s there so the
pattern is already in place when a second environment shows up — adding overlays/staging/ later is a one-liner, no
restructuring required.
Before applying anything, validate that Kustomize renders correctly:
kubectl kustomize config/sample-app/overlays/localYou should see the fully rendered Deployment and Service YAML printed to stdout — two documents separated by ---.
If you see an error like no such file or directory or accumulating resources, the path in kustomization.yaml
is wrong. If the output looks like valid Kubernetes YAML with kind: Deployment and kind: Service, you’re good.
Writing the Application Manifest#
An ArgoCD Application is a Kubernetes CRD — a custom resource that lives in the cluster itself. This is a key design
choice: the configuration that tells ArgoCD what to deploy is itself stored as a Kubernetes object, not in a config
file on your laptop or a database somewhere. You can kubectl apply it, version it in git, and manage it with the same
tools you use for everything else.
Each Application answers three questions: where is the source (git repo + path), where is the destination (cluster +
namespace), and how should syncing work (manual or automated). That’s the whole contract between ArgoCD and your repo.
Here’s what one looks like:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: sample-app
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/ravikrs/blog-argocd
targetRevision: HEAD
path: config/sample-app/overlays/local
destination:
server: https://kubernetes.default.svc
namespace: sample-app
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=trueWalking through the key fields:
spec.source.repoURL— the git repo ArgoCD clones. If you’re using your own repo, update this URL before applying.spec.source.targetRevision: HEAD— always track the latest commit on the default branch. You can pin to a branch, tag, or specific commit SHA instead.spec.source.path— the directory inside the repo that ArgoCD renders. Everything outside this path is ignored.spec.destination.server: https://kubernetes.default.svc— deploy into the same cluster ArgoCD is running in. This is the in-cluster address; for external clusters you’d use the cluster’s API server URL.spec.destination.namespace— the namespace to deploy resources into.spec.syncPolicy.syncOptions: CreateNamespace=true— create the destination namespace automatically if it doesn’t exist. Without this, the sync fails ifsample-appnamespace isn’t already there.project: default— ArgoCD Projects control which repos and clusters an Application can use.defaultallows everything. Post 21 covers Projects and RBAC.
One thing to sort out first: if your repo is private, ArgoCD needs credentials to clone it. If your repo is
public (as blog-argocd is), skip this section entirely — no credentials needed.
For private repos, you can’t commit those credentials into the repo you’re trying to protect (that’s the chicken-and-egg problem). There are a few ways to solve this:
- Manual Kubernetes Secret — create the secret once by hand and git-ignore the file. Simple, no extra tooling. Fine for local development. This is what we’re doing here.
- Sealed Secrets — encrypt the secret and commit the encrypted form to git. Decryption happens in-cluster.
- External Secrets Operator (ESO) — pull secrets from an external store (Vault, AWS Secrets Manager, etc.) at runtime. The secret never touches git at all. This is the production-grade approach and we will cover it later in this series.
Creating a repo-scoped PAT on GitHub
Use a fine-grained PAT rather than a classic token — it limits the blast radius to a single repo:
- Go to GitHub → Settings → Developer settings → Personal access tokens → Fine-grained tokens
- Click Generate new token
- Set Resource owner to your account and Repository access to the specific repo
- Under Permissions, set Contents to
Read-only— that’s all ArgoCD needs to clone - Generate and copy the token immediately (GitHub won’t show it again)
The secret template lives at secret-templates/repo-secret.yaml in the companion repo. Copy it into the secret/
folder (which is git-ignored) so you can fill in real values without risking an accidental commit:
mkdir -p secret && cp secret-templates/repo-secret.yaml secret/repo-secret.yamlReplace the placeholders with your actual values:
sed -i '' \
-e 's|<YOUR_GH_USERNAME>|your-github-username|g' \
-e 's|<YOUR_GH_REPO>|blog-argocd|g' \
-e 's|<YOUR_GITHUB_PAT>|ghp_xxxxxxxxxxxx|g' \
secret/repo-secret.yamlOr open the file and edit it directly — either way, secret/ is in .gitignore so it will never be committed.
Apply it:
kubectl apply -f secret/repo-secret.yamlArgoCD auto-detects any Secret in the argocd namespace labelled
argocd.argoproj.io/secret-type: repository. After you apply it, verify ArgoCD sees the repo:
CLI:
argocd repo listUI: Settings (gear icon) → Repositories — the repo should appear with a green connected status. A red status means the credentials are wrong or the repo URL doesn’t match.
Sync Policies: Manual vs Automated#
Before applying the Application, it’s worth understanding what syncPolicy does — because as soon as you apply the
manifest, ArgoCD starts acting on it.
Without a syncPolicy, ArgoCD detects drift but waits for you to trigger a sync manually. That’s useful when you want
to review changes before they hit the cluster.
With automated, ArgoCD polls git every 3 minutes and syncs any detected diff automatically. Two sub-settings matter:
selfHeal: true — if someone runs kubectl edit directly on the cluster, ArgoCD reverts the change on the next
reconciliation cycle. The cluster always converges toward git.
prune: true — if you delete a manifest from git, ArgoCD deletes the resource from the cluster. Without this, removed
files leave orphaned resources indefinitely.
Both default to false. For a learning setup, turning them both on is fine. For production, you might want
selfHeal: true but leave prune: false until you’re confident about what you’re removing.
Applying the Application and Watching the Sync#
Apply the Application manifest:
kubectl apply -f apps/sample-app/application.yamlCheck status:
CLI:
argocd app get sample-appUI: click on sample-app in the main dashboard. You’ll see a graph of all the resources ArgoCD deployed —
Deployment, Service, Pods — each with their own sync and health indicators.
Both show Sync Status and Health Status. Initially it might show OutOfSync while ArgoCD fetches the repo. After a
few seconds it should move to Synced and Healthy.
ArgoCD polls git every 3 minutes by default. That’s fine for production but slow when you’re actively iterating. Force an immediate sync via CLI or UI:
argocd app sync sample-appOr in the ArgoCD dashboard: open the app and click Sync → Synchronize. Both trigger an immediate fetch, diff, and
apply. The poll interval is configurable (--app-resync on argocd-application-controller) but the default is fine
for most setups.
Verify the pods came up:
CLI:
kubectl get pods -n sample-appUI: in the app detail view, the Deployment node expands to show individual ReplicaSets and Pods. Each pod shows its
phase (Running, Pending, Error) with a colour indicator. Click any pod to see its logs directly in the browser
without needing kubectl logs.
Making a Change and Watching It Deploy#
Edit the replica count in config/sample-app/base/deployment.yaml:
spec:
replicas: 3Commit and push:
git add config/sample-app/base/deployment.yaml
git commit -m "scale sample-app to 3 replicas"
git pushArgoCD will detect the change within 3 minutes. Before syncing, it’s worth previewing exactly what will change:
CLI:
argocd app diff sample-appThis shows a kubectl-style diff — lines prefixed with - are what’s currently on the cluster, + is what git says it
should be. Good habit before any sync, especially in production.
UI: open the app and click App Diff — the same diff rendered visually, resource by resource.
Once you’ve confirmed the diff looks right, sync:
CLI:
argocd app sync sample-appUI: open sample-app in the dashboard and click Sync → Synchronize.
Watch the pods scale up:
CLI:
kubectl get pods -n sample-appUI: the app detail view updates in real time — watch the Deployment node show 3/3 replicas as the new pod comes
up alongside the existing two.
That loop — push to git, ArgoCD detects the diff, syncs the cluster — is the entire GitOps cycle.
One thing worth knowing: ArgoCD watches the entire repo for HEAD changes, but only renders and diffs the path specified
in the Application. A push to docs/ or README.md triggers a reconciliation check but produces no cluster operations
if the manifests at the watched path didn’t change. It’s a lightweight cycle, not a full sync.
Pruning: What Happens When You Delete a Resource from Git#
With prune: true, deleting a manifest from git deletes the corresponding resource from the cluster on the next sync.
Remove a file, push, sync — the Deployment (or Service, or Ingress) disappears.
Without prune: true, the resource stays in the cluster forever. This can be confusing: you delete a file from git,
push, ArgoCD shows the app as OutOfSync, but nothing changes on the cluster until you manually delete the resource.
You can spot this state in both places:
CLI: argocd app get sample-app — shows OutOfSync under Sync Status, and lists the orphaned resource as an
extra item in the diff.
UI: the app card on the dashboard turns yellow with OutOfSync. In the app detail view, the orphaned resource
appears with an orange extra badge — it exists in the cluster but not in git.
For cleanup-heavy workflows (experimenting with different resources), prune is very useful. For stability-first setups where accidental deletion is a bigger risk than orphaned resources, leave it off and manage deletions explicitly.
Key Concepts Recap#
Desired state: what git says the cluster should look like. The rendered manifests at spec.source.path.
Actual state: what’s actually running in the cluster right now.
Sync status: whether desired state and actual state match. Synced means they match. OutOfSync means there’s a
diff — either git changed and the cluster hasn’t caught up, or someone edited the cluster directly.
Health status: whether the deployed resources are actually healthy. A Deployment can be Synced (matches git) but
Degraded (pods crashing). ArgoCD tracks both independently. The four health states you’ll see:
Healthy— all resources are running and passing their health checksProgressing— resources are being rolled out; pods are starting or terminating. Normal during a deploy.Degraded— something is wrong: pods crashing, image pull failing, readiness probe not passingMissing— the resource exists in git but not in the cluster at all
These concepts come up in every ArgoCD interaction. When something looks wrong, the first question is always: what does git say, what is the cluster running, and what’s unhealthy?
Common Gotchas#
App stuck in ComparisonError
ArgoCD couldn’t render the manifests. Usually a Kustomize build failure.
CLI:
argocd app get sample-app # shows the error message under ConditionsUI: the app card shows ComparisonError in red. Click the app → the error appears at the top of the detail view.
Fix: run kubectl kustomize config/sample-app/overlays/local locally to reproduce the error. Usually a missing file
or wrong path in kustomization.yaml.
App is Synced but Degraded
The manifests matched git, but the pods are unhealthy. This is a runtime problem, not a sync problem.
CLI:
kubectl get pods -n sample-app
kubectl describe pod <pod-name> -n sample-app # check Events section
kubectl logs <pod-name> -n sample-appUI: click the app → click the unhealthy Pod node → Logs tab shows the container logs inline.
Common causes: wrong image name, image pull policy issues, crashing container, failed readiness probe.
Sync keeps failing with namespace not found
The destination namespace doesn’t exist and CreateNamespace=true is missing from syncOptions. Add it to the
Application manifest and re-apply.
argocd app get shows Unknown health
ArgoCD couldn’t determine health — usually because the argocd-repo-server is still starting up or the resource type
doesn’t have a built-in health check. Wait a moment and re-check; it typically resolves on the next reconciliation.
Cleanup#
When you’re done experimenting, delete the Application:
CLI:
argocd app delete sample-appUI: open the app → App Details → Delete.
By default this is a cascade delete — ArgoCD deletes the Application CRD and all the cluster resources it manages (the Deployment, Service, namespace). If you want to remove ArgoCD’s management but leave the resources running, use:
argocd app delete sample-app --cascade falseThis removes the Application object but leaves sample-app namespace and its pods untouched.
Further Reading#
Kustomize
- Kustomize docs — official guide covering bases,
overlays, and patches. If
kustomization.yamlis unfamiliar, start here. - Kustomize GitHub — examples and the full reference for fields like
commonLabels,namePrefix, andpatchesStrategicMerge.
ArgoCD Application CRD
- ArgoCD Application spec reference — the
full list of fields for
spec.source,spec.destination, andspec.syncPolicy. Useful when the manifest isn’t behaving as expected. - Automated sync and self-heal — details on polling interval, prune behaviour, and when to prefer manual sync over automated.
Private repos
- ArgoCD private repo credentials — covers
SSH keys, HTTPS tokens, and the
argocd.argoproj.io/secret-type: repositorylabel convention used in this post.
Sync and health
- ArgoCD resource health — explains how ArgoCD
determines
Healthy,Degraded,Progressing, andMissingfor different resource types. - Sync phases and waves — not needed yet, but worth bookmarking for when you need ordered deployments (e.g. CRDs before controllers).