42. Going to the cloud: Orbit on EKS

Everything so far runs on kind. In the cloud, only the bottom layer changes: cluster, network, load balancers, storage classes, identity. The GitOps repo, add-ons and app manifests stay the same, apart from an overlay.

42.1 Infrastructure as code with Terraform

infra/
├── modules/                   # (or use the community modules directly)
├── envs/
│   ├── staging/main.tf
│   └── prod/main.tf           # separate state + separate AWS account per env
└── bootstrap/                 # state bucket, lock table, OIDC role for CI

# infra/envs/prod/main.tf  (sketch: check each module's docs for current input names)
terraform {
  required_version = ">= 1.10"
  backend "s3" {
    bucket       = "acme-tfstate-prod"
    key          = "orbit/eks.tfstate"
    region       = "us-east-1"
    use_lockfile = true
  }
  required_providers { aws = { source = "hashicorp/aws", version = "~> 6.0" } }
}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 6.0"
  name    = "orbit-prod"
  cidr    = "10.40.0.0/16"
  azs             = ["us-east-1a", "us-east-1b", "us-east-1c"]
  private_subnets = ["10.40.0.0/19", "10.40.32.0/19", "10.40.64.0/19"]
  public_subnets  = ["10.40.96.0/22", "10.40.100.0/22", "10.40.104.0/22"]
  enable_nat_gateway     = true
  one_nat_gateway_per_az = true                    # a single NAT is a single point of failure
  private_subnet_tags = { "karpenter.sh/discovery" = "orbit-prod", "kubernetes.io/role/internal-elb" = 1 }
  public_subnet_tags  = { "kubernetes.io/role/elb" = 1 }
}

module "eks" {
  source             = "terraform-aws-modules/eks/aws"
  version            = "~> 21.0"
  name               = "orbit-prod"
  kubernetes_version = "1.34"
  vpc_id             = module.vpc.vpc_id
  subnet_ids         = module.vpc.private_subnets
  endpoint_public_access = false                   # API reachable via VPN/bastion only
  enable_cluster_creator_admin_permissions = true

  addons = {
    coredns                = {}
    kube-proxy             = {}
    vpc-cni                = { before_compute = true }
    eks-pod-identity-agent = { before_compute = true }
    aws-ebs-csi-driver     = {}
  }

  # small, stable on-demand group for system add-ons and Karpenter itself; Karpenter does the rest
  eks_managed_node_groups = {
    system = {
      instance_types = ["m7g.large"]
      ami_type       = "AL2023_ARM_64_STANDARD"
      min_size = 3, max_size = 4, desired_size = 3
      labels = { "node-role" = "system" }
      taints = { system = { key = "CriticalAddonsOnly", value = "true", effect = "NO_SCHEDULE" } }
    }
  }
}

cd infra/envs/prod && terraform init && terraform plan -out tf.plan && terraform apply tf.plan
aws eks update-kubeconfig --name orbit-prod --region us-east-1

⚠️ Caveat (verify): the Terraform module majors (vpc ~> 6, eks ~> 21, AWS provider ~> 6) and v21 input names (name, kubernetes_version, addons, endpoint_public_access) were written from memory of the v21 changes. Run terraform init && terraform validate and compare with each module's README and UPGRADE notes. Check that kubernetes_version is a version EKS currently offers (aws eks describe-cluster-versions). The S3 backend's use_lockfile needs Terraform 1.10+.

42.2 What changes in the GitOps repo for EKS

42.3 Workload identity: no static cloud credentials, ever

# IAM role the orders ServiceAccount assumes, scoped to one bucket
resource "aws_iam_role" "orders" {
  name               = "orbit-prod-orders"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{ Effect = "Allow", Principal = { Service = "pods.eks.amazonaws.com" },
                   Action = ["sts:AssumeRole", "sts:TagSession"] }]
  })
}
resource "aws_iam_role_policy" "orders_s3" {
  role   = aws_iam_role.orders.id
  policy = jsonencode({ Version = "2012-10-17", Statement = [{ Effect = "Allow",
    Action = ["s3:PutObject"], Resource = "arn:aws:s3:::acme-orbit-invoices/*" }] })
}
resource "aws_eks_pod_identity_association" "orders" {
  cluster_name    = module.eks.cluster_name
  namespace       = "orbit-prod"
  service_account = "orders"
  role_arn        = aws_iam_role.orders.arn
}

The AWS SDK in the pod picks up short-lived, auto-rotated credentials. No access keys in Secrets, nothing to leak. The equivalents are GKE Workload Identity Federation and AKS Workload Identity.

42.4 Cluster upgrades without drama

Kubernetes ships 3 minors a year, and managed providers support each for roughly 14 months (then extended support costs extra). Upgrade at least twice a year:

1. Read the changelog's deprecations. Run pluto detect-all-in-cluster / kubent for removed APIs.
2. Upgrade add-ons to versions that support the target minor (CNI, CSI, CoreDNS, Karpenter, ingress/gateway, operators).
3. Upgrade the control plane one minor at a time (1.33 → 1.34).
4. Roll the nodes: Karpenter drift replaces them automatically and respects PDBs. This is where bad PDBs and single replicas bite (§22.3).
5. For high-risk jumps, use blue/green clusters: build a new cluster at the new version, sync it from Git, shift traffic via DNS/LB weights, then delete the old one. GitOps is what makes this cheap.

⚠️ Caveat (verify): provider support windows and pricing for extended support change; check the EKS/GKE/AKS version calendars.

42.5 GKE and AKS equivalents