37. Progressive delivery: canary with automatic rollback

Rolling updates are gated only on readiness, which says the pod works, not that the new code is correct. A canary sends a small share of real traffic to the new version, measures it against the SLO, and proceeds or aborts automatically.

37.1 Install Argo Rollouts with the Gateway API plugin

helm upgrade --install argo-rollouts argo/argo-rollouts -n argo-rollouts --create-namespace \
  --set dashboard.enabled=true

# ConfigMap argo-rollouts-config in argo-rollouts: enable the Gateway API traffic router plugin
apiVersion: v1
kind: ConfigMap
metadata: { name: argo-rollouts-config, namespace: argo-rollouts }
data:
  trafficRouterPlugins: |
    - name: "argoproj-labs/gatewayAPI"
      location: "https://github.com/argoproj-labs/rollouts-plugin-trafficrouter-gatewayapi/releases/download/<latest-version>/gatewayapi-plugin-linux-amd64"   # pin a release

(The controller also needs RBAC to patch HTTPRoutes; the plugin's README has the ClusterRole.)

37.2 Convert orders to a Rollout

Replace the Deployment with a Rollout. The pod template is identical, so the golden manifest carries over unchanged. Add a second Service for the canary and reference both from the HTTPRoute (orders weight 100, orders-canary weight 0).

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata: { name: orders, namespace: orbit }
spec:
  revisionHistoryLimit: 5
  selector: { matchLabels: { app.kubernetes.io/name: orders } }
  template:
    # ... exactly the pod template from §24.3, plus:
    #   env: APP_VERSION from fieldRef metadata.labels['app.kubernetes.io/version']
  strategy:
    canary:
      stableService: orders
      canaryService: orders-canary
      trafficRouting:
        plugins:
          argoproj-labs/gatewayAPI:
            httpRoute: orbit-api
            namespace: orbit
      analysis:                                   # background analysis during the whole rollout
        templates: [{ templateName: orders-slo }]
        startingStep: 1
        args:
          - name: canary-version
            valueFrom: { podTemplateHashValue: Latest }
      steps:
        - setWeight: 5
        - pause: { duration: 2m }
        - setWeight: 25
        - pause: { duration: 5m }
        - setWeight: 50
        - pause: { duration: 5m }
        # 100% after the last step

37.3 The analysis: abort if the canary breaks the SLO

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata: { name: orders-slo, namespace: orbit }
spec:
  args: [{ name: canary-version }]
  metrics:
    - name: error-ratio
      interval: 1m
      failureLimit: 1
      successCondition: "len(result) == 0 || isNaN(result[0]) || result[0] < 0.01"   # no data yet = not a failure
      provider:
        prometheus:
          address: http://kps-prometheus.monitoring.svc:9090
          query: |
            sum(rate(http_server_request_duration_seconds_count{job="orbit/orders",k8s_pod_name=~"orders-{{args.canary-version}}-.*",http_response_status_code=~"5.."}[2m]))
            /
            sum(rate(http_server_request_duration_seconds_count{job="orbit/orders",k8s_pod_name=~"orders-{{args.canary-version}}-.*"}[2m]))
    - name: p95-latency
      interval: 1m
      failureLimit: 1
      successCondition: "len(result) == 0 || isNaN(result[0]) || result[0] < 0.3"
      provider:
        prometheus:
          address: http://kps-prometheus.monitoring.svc:9090
          query: |
            histogram_quantile(0.95, sum by (le) (rate(http_server_request_duration_seconds_bucket{job="orbit/orders",k8s_pod_name=~"orders-{{args.canary-version}}-.*"}[2m])))

(Canary pods are named orders-<pod-template-hash>-xxxxx, and k8s.pod.name was promoted to a label in §30.1. That's how the query isolates the canary from the stable version.)

kubectl argo rollouts get rollout orders -n orbit --watch
kubectl argo rollouts promote orders -n orbit        # skip the current pause
kubectl argo rollouts abort orders -n orbit          # manual abort -> traffic back to stable

Try it: release a version of orders that returns 500 for 5% of requests while k6 runs. The analysis fails at the 5% or 25% step, traffic snaps back to stable, and at most a few percent of a few minutes of traffic saw the bug. Compare that with a rolling update, which would have reached 100% in under a minute.

⚠️ Caveat (verify): (1) Fill in the Gateway API plugin version and add the ClusterRole from the plugin README; without it the controller can't patch HTTPRoutes. On arm64, use the -linux-arm64 binary. (2) The analysis queries depend on k8s.pod.name being promoted to the k8s_pod_name label (§30.1 caveat) and on canary pod names containing the pod-template hash. Check the actual names with kubectl get pods. (3) isNaN / len(result) come from the expr language Argo Rollouts uses; test the template with a dry run on a non-critical service first.

37.4 Other strategies