36. GitOps with Argo CD

36.1 Principles (OpenGitOps)

1. Declarative: the whole system is described as data.
2. Versioned and immutable: Git is the source of truth with full history.
3. Pulled automatically: an in-cluster agent pulls; CI never holds cluster credentials.
4. Continuously reconciled: drift is detected and corrected.

Practical wins: every change is a reviewed PR, every rollback is git revert, an audit trail comes for free, and disaster recovery is "point a new cluster at the repo".

36.2 Install and bootstrap

helm repo add argo https://argoproj.github.io/argo-helm
helm upgrade --install argocd argo/argo-cd -n argocd --create-namespace \
  --set configs.params."server\.insecure"=true --wait
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d; echo
kubectl -n argocd port-forward svc/argocd-server 8088:80     # http://localhost:8088

36.3 App of apps: the platform layer

platform/
├── root.yaml                    # the only thing applied by hand
├── apps/
│   ├── cert-manager.yaml        # Argo CD Application -> Helm chart + values
│   ├── envoy-gateway.yaml
│   ├── cnpg.yaml
│   ├── kube-prometheus-stack.yaml
│   ├── loki.yaml  tempo.yaml  otel-agent.yaml  otel-gateway.yaml
│   ├── keda.yaml  argo-rollouts.yaml  kyverno.yaml  external-secrets.yaml  velero.yaml
│   └── orbit.yaml               # ApplicationSet: one Application per environment
└── <component>/values.yaml

# platform/apps/kube-prometheus-stack.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: kube-prometheus-stack
  namespace: argocd
  annotations: { argocd.argoproj.io/sync-wave: "-5" }     # platform before apps
  finalizers: [resources-finalizer.argocd.argoproj.io]
spec:
  project: platform
  sources:
    - repoURL: https://prometheus-community.github.io/helm-charts
      chart: kube-prometheus-stack
      targetRevision: "77.*"          # pin the current major; Renovate bumps it via PR
      helm: { valueFiles: [$values/platform/kube-prometheus-stack/values.yaml] }
    - repoURL: https://github.com/acme/orbit
      targetRevision: main
      ref: values
  destination: { server: https://kubernetes.default.svc, namespace: monitoring }
  syncPolicy:
    automated: { prune: true, selfHeal: true }
    syncOptions: [CreateNamespace=true, ServerSideApply=true]   # SSA: large CRDs exceed the client-side annotation limit

# platform/apps/orbit.yaml: one Application per environment overlay
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata: { name: orbit, namespace: argocd }
spec:
  goTemplate: true
  generators:
    - list:
        elements:
          - { env: dev,     autoSync: true }
          - { env: staging, autoSync: true }
          - { env: prod,    autoSync: true }   # still gated: changes reach prod only via a reviewed PR
  template:
    metadata: { name: "orbit-{{.env}}" }
    spec:
      project: orbit
      source:
        repoURL: https://github.com/acme/orbit
        targetRevision: main
        path: "deploy/overlays/{{.env}}"
      destination: { server: https://kubernetes.default.svc, namespace: "orbit-{{.env}}" }
      syncPolicy:
        automated: { prune: true, selfHeal: true }
        syncOptions: [CreateNamespace=true]
        retry: { limit: 5, backoff: { duration: 10s, factor: 2, maxDuration: 3m } }

kubectl apply -n argocd -f platform/root.yaml     # root Application pointing at platform/apps/
argocd app list
argocd app diff orbit-prod
argocd app history orbit-prod

(For real multi-cluster setups, use a cluster generator instead of a list, so each environment's cluster is registered in Argo CD and its Applications target it.)

36.4 Sync waves and hooks: ordering

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"       # lower waves first: CRDs/namespaces < DB < migrations < apps
    argocd.argoproj.io/hook: PreSync          # run before the main sync (migrations, §38)
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation

36.5 Promotion and rollback

• Promotion = a PR that copies the digest from the staging overlay to the prod overlay. Automate opening the PR; keep merging human (or gated on the staging SLO). Tools like Kargo or Argo CD Image Updater automate multi-stage promotion.
• Rollback = git revert the promotion commit. Argo CD syncs the previous digest. With canaries (§37), a bad release usually aborts itself before anyone has to.
• Drift: selfHeal reverts manual kubectl edits. For a break-glass manual change, disable auto-sync on that Application first, and record why.

War story: with prune: true, removing a file from Git deletes the resource. Someone refactored Kustomize paths, a PVC-owning StatefulSet briefly vanished from the rendered output, and Argo CD pruned it. Protect stateful resources with argocd.argoproj.io/sync-options: Prune=false,Delete=false, use reclaimPolicy: Retain on the StorageClass, and review Argo CD diffs in PRs (the argocd app diff output, or a rendered-manifests diff check in CI).

⚠️ Caveat (verify): the kube-prometheus-stack major in targetRevision is a placeholder for whatever is current. Look it up with helm search repo prometheus-community/kube-prometheus-stack. Check the ApplicationSet template fields against your Argo CD version.