Argo CD Fixed My Drift, Then Deployed My Bad Release

This project started with a simple goal: run Kubernetes without keeping an EKS cluster online every day.

In I Wanted Kubernetes Without an Always-On EKS Bill, I built an always on k3s lab on my home server and proved that I could deploy, update, and roll back an application.

The rollback worked, but it exposed the next problem. Kubernetes restored Version 2 while the saved YAML still declared Version 3. I corrected the file manually, but the recovery depended on repairing the running cluster and its saved instructions separately.

In The Rollback Worked. My Next Deploy Could Break It Again, I designed a safer path. The automated build process would test and publish an exact image, then stop at a Git pull request. Git would record the reviewed version. Argo CD, running inside Kubernetes, would make the cluster follow that record.

Now I needed to prove that the design worked outside a diagram.

I followed one release from source code to running Pods. Then I tested two opposite failures:

• The cluster was wrong while Git was correct.

• Git contained a bad setting while the cluster followed it correctly.

Those experiments showed both the value and the limit of GitOps. Automation can make the cluster match Git, but it cannot decide whether the human-approved version in Git is a good one.

CI Built the Release but Did Not Deploy It

The GitHub Actions workflow my continuous integration, or CI, worker ran the application tests and checked the Kubernetes package before building anything. Its job was to prove and publish a release, not to change the cluster.

After validation, Buildx created a Linux AMD64 image with the full source commit baked into /version:

docker buildx build \
  --platform linux/amd64 \
  --build-arg "APP_VERSION=$GITHUB_SHA" \
  --tag "$image_name:$GITHUB_SHA" \
  --provenance=mode=max \
  --sbom=true \
  --push \
  application

After publishing the image, CI read its registry digest. A digest is the image's content fingerprint: if the image changes, the digest changes. CI updated one field in gitops/environments/homelab/values.yaml. For the final measured release, the automated pull request contained this one-line change:

-  digest: sha256:170bcc91b9c4bfcbdc194c004cb38d6c89094e9b49a1a128ed3a0507976fef97
+  digest: sha256:f8303e07e737f40fa5479b73a148261cf4221f4d4b52ebf095b74d2cb4ef5be5

The workflow opened the pull request and stopped. A human reviewed and merged the digest. That merge not a deployment command in CI was the release instruction.

Argo CD Made the Cluster Follow Git

Argo CD was the worker inside Kubernetes. It repeatedly compared the version recorded in Git with the version running in the cluster and worked to make them agree. This comparison-and-correction loop is called reconciliation.

I installed a pinned, resource-limited Argo CD Core deployment with no permanent API server or public UI. Its Services remained internal ClusterIP Services.

The Application reads both the Helm chart and homelab values from the public repository. Automated sync, pruning, and self-healing are enabled:

syncPolicy:
  automated:
    enabled: true
    prune: true
    selfHeal: true

At the healthy baseline, Argo CD reported Synced, Healthy, and operation Succeeded. Two application Pods were ready with zero restarts.

I retained the complete artifact trace:

application commit
    -> registry digest
    -> Git desired state
    -> Deployment image
    -> both Pod runtime image IDs
    -> /version response

The Git declaration, Deployment, and both Pod image IDs agreed on the same registry digest. /version returned the source commit baked into that image.

The four Argo CD components used a combined 19 millicores of CPU and 85 MiB of memory in the retained steady-state sample. That measurement describes this home lab; it is not production sizing guidance.

First, I Changed the Cluster Behind Git’s Back

Git declared two replicas. I deliberately changed the live Deployment to one:

kubectl -n delivery-api scale deployment/delivery-api --replicas=1

The first poll already showed one replica and Argo CD OutOfSync. Git remained unchanged with replicaCount: 2.

Forty-two seconds after the mutation, the Deployment was back to two replicas and the Application was Synced again.

This proved self-healing for that specific field. Git remained correct, so Argo CD could restore the live cluster from the durable declaration.

The next failure reversed that relationship.

Then I Put a Bad Setting in Git

I merged a reviewed configuration change from:

readinessMode: ready

to:

readinessMode: fail

The change was valid YAML. It passed the Helm schema and lint checks. At runtime, the new Pod stayed alive but /ready returned HTTP 503.

Forty-nine seconds after the merge, the evidence showed:

Argo CD sync:    Synced
Argo CD health:  Progressing
New Pod:         Running, not Ready
Old Pods:        Running, Ready

Synced meant the cluster matched Git. Progressing meant Kubernetes could not complete the rollout.

GitOps had not malfunctioned. It had correctly delivered a harmful declaration.

The Deployment used maxUnavailable: 0, so the two old ready replicas remained while the replacement failed readiness. One sampled request through the private Service returned {"status":"ready"}. That proves availability at that moment, not zero downtime across the experiment.

Recovery Had to Correct Git

Patching only the live Deployment would leave Git declaring the bad readiness mode. Argo CD could then reapply it.

I reverted the bad merge in Git:

git revert --mainline 1 69ac4fe6de392aed4affc573ea20a2bb3f4ee59f
git push

Argo CD pulled the revert and restored readinessMode: ready. Forty-two seconds after the recovery commit, the Application was Synced and Healthy with two updated, ready replicas.

Argo CD did not choose the known-good version. I chose it by reverting Git; Argo CD reconciled that decision.

I Timed a Fresh Release End to End

The first image promotion happened before Argo CD was installed, so I did not use it for a continuous promotion-to-healthy claim. I ran another application release with Argo CD already active and started the monitor before merging the digest pull request.

The monitor sampled every two seconds, so the healthy timestamp has up to two seconds of sampling uncertainty.

During the rolling update, the Deployment already declared the new digest while a request still reached an old ready Pod and returned the previous commit. A few seconds later, both updated replicas were ready and /version returned:

{"service":"delivery-api","version":"84db89f7f968c5860b07a7efd6a42e21895d9e5c"}

That was expected. A rolling update changes replicas gradually, the Deployment declaration and every serving Pod do not switch at the same instant.

What the Implementation Proved

Application CI published the artifact and proposed the environment change without receiving cluster authority. Git retained the reviewed digest. Argo CD pulled and reconciled it. Kubernetes reported rollout health.

The failure tests also exposed the limits:

• Self-healing corrected live drift only because Git remained correct.

• Argo CD faithfully applied the bad readiness declaration from Git.

• Recovery required a human decision recorded as a Git revert.

• One successful request did not prove zero request loss.

• A single-node k3s cluster could not demonstrate high availability.

GitOps did not remove operational judgment. It moved release and recovery decisions into a place where they were reviewable, durable, and reconcilable.

Conclusion

The first article created an affordable, always-on Kubernetes lab. The second used its rollback problem to design one reviewable release path. This implementation proved how that path behaved when either the cluster or Git was wrong.

The common lesson across the three articles is that a working cluster is only one part of a delivery platform. The approved release, permission to deploy, health signals, and recovery decision also need clear owners.

The code, chart, GitOps configuration, runbooks, and sanitized evidence are in the public pull-based-kubernetes-delivery repository.