26. Packaging: Kustomize and Helm

Rule of thumb used by many platform teams: Helm for software you consume (Prometheus, cert-manager, CNPG), Kustomize for software you own (plain YAML you can read, patched per environment). Both are first-class in Argo CD.

26.1 Kustomize: base + overlays

deploy/
├── base/
│   ├── kustomization.yaml
│   ├── orders/{deployment.yaml,service.yaml,pdb.yaml,serviceaccount.yaml,config.env}
│   ├── catalog/...
│   ├── worker/...
│   ├── redis/statefulset.yaml
│   ├── postgres/cluster.yaml
│   └── routes/httproute.yaml
└── overlays/
    ├── dev/kustomization.yaml
    └── prod/{kustomization.yaml,patches/*.yaml}

# deploy/base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: orbit
labels:
  - pairs: { app.kubernetes.io/part-of: orbit }
    includeSelectors: false
resources:
  - orders/deployment.yaml
  - orders/service.yaml
  - orders/pdb.yaml
  - orders/serviceaccount.yaml
  - catalog/
  - worker/
  - redis/statefulset.yaml
  - postgres/cluster.yaml
  - routes/httproute.yaml
configMapGenerator:
  - name: orders-config
    envs: [orders/config.env]      # name gets a content hash -> config change = rolling update
images:
  - { name: orbit-orders,  newName: ghcr.io/acme/orbit-orders }
  - { name: orbit-catalog, newName: ghcr.io/acme/orbit-catalog }
  - { name: orbit-worker,  newName: ghcr.io/acme/orbit-worker }

# deploy/overlays/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: orbit-prod
resources: [../../base]
images:   # CI bumps these (§35): digest-pinned
  - { name: orbit-orders,  newName: ghcr.io/acme/orbit-orders,  digest: sha256:4f1c... }
  - { name: orbit-catalog, newName: ghcr.io/acme/orbit-catalog, digest: sha256:9ab0... }
  - { name: orbit-worker,  newName: ghcr.io/acme/orbit-worker,  digest: sha256:c77e... }
patches:
  - path: patches/orders-resources.yaml
  - target: { kind: Cluster, name: orbit-db }
    patch: |-
      - op: replace
        path: /spec/storage/size
        value: 100Gi

kustomize build deploy/overlays/prod | less               # render
kubectl diff -k deploy/overlays/prod                       # compare with the cluster
kustomize build deploy/overlays/prod | kubeconform -strict -summary -ignore-missing-schemas

26.2 Helm: authoring a chart

helm create orbit-service     # scaffold; then delete what you don't need

orbit-service/
├── Chart.yaml
├── values.yaml
├── values.schema.json        # validates values: catches typos like "replica: 3"
└── templates/
    ├── _helpers.tpl
    ├── deployment.yaml
    ├── service.yaml
    ├── pdb.yaml
    └── hpa.yaml

# templates/deployment.yaml (excerpt)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "orbit-service.fullname" . }}
  labels: {{- include "orbit-service.labels" . | nindent 4 }}
spec:
  {{- if not .Values.autoscaling.enabled }}
  replicas: {{ .Values.replicaCount }}
  {{- end }}
  selector:
    matchLabels: {{- include "orbit-service.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      annotations:
        # restart pods when the rendered config changes
        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
      labels: {{- include "orbit-service.selectorLabels" . | nindent 8 }}
    spec:
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}{{ if .Values.image.digest }}@{{ .Values.image.digest }}{{ else }}:{{ .Values.image.tag | default .Chart.AppVersion }}{{ end }}"
          resources: {{- toYaml .Values.resources | nindent 12 }}

helm lint ./orbit-service
helm template orders ./orbit-service -f values-prod.yaml | kubeconform -strict
helm upgrade --install orders ./orbit-service -n orbit -f values-prod.yaml --atomic --timeout 5m
helm history orders -n orbit
helm rollback orders 3 -n orbit
helm plugin install https://github.com/databus23/helm-diff && helm diff upgrade orders ./orbit-service -f values-prod.yaml

--atomic rolls back automatically if the upgrade doesn't become healthy within --timeout.

Best practice: never helm install third-party charts with default values in prod. Pin chart versions, commit a values file, render it (helm template) in CI, and review the diff. A chart's minor version bump can change CRDs, RBAC or defaults.

26.3 Validate manifests before they reach a cluster