Phase 3 — Production Readiness (Days 61–90)

Week 9: Backup & Stateful Data Protection

Goal: Know what to back up, how, and how to verify restores.

Topics:

• What needs backing up: etcd (cluster state), PVCs (application data), ConfigMaps/Secrets
• etcd backup and restore: etcdctl snapshot save/restore
• Velero: cluster backup/restore, scheduled backups, namespace migration
• Volume snapshots: CSI VolumeSnapshot API
• Database-specific backup: pg_dump, MySQL dump, mongodump + automation scripts
• Backup retention policies, encryption at rest, off-site storage (S3, GCS)
• Backup verification: automated restore tests
• RPO (Recovery Point Objective) and RTO (Recovery Time Objective) definitions

Hands-on Labs:

• Take an etcd snapshot on a kind cluster; simulate cluster loss; restore from snapshot
• Install Velero with MinIO (local S3); schedule hourly namespace backups
• Backup Postgres PVC with Velero; delete namespace; restore and verify data
• Write a cron-based pg_dump script that uploads to S3 with a 7-day retention policy

Week 10: Disaster Recovery & High Availability

Goal: Design and validate systems that survive zone failures, node failures, and full-region outages.

Topics:

• HA cluster topology: multi-master control plane, etcd quorum (3 or 5 nodes)
• Pod Disruption Budgets (PDB)
• Node affinity, pod anti-affinity, topology spread constraints
• Cluster Autoscaler and Karpenter (AWS)
• Multi-region/multi-cluster strategies: active-active vs active-passive
• Global load balancing: AWS Route 53 / GCP Cloud DNS with health checks
• Cross-region data replication: RDS Multi-AZ, Cloud SQL HA, Redis Sentinel
• Disaster recovery runbook: RTO/RPO targets, communication plan, blameless postmortem template
• Chaos engineering introduction: LitmusChaos, Chaos Monkey

Hands-on Labs:

• Configure PDB for the sample app (maxUnavailable: 1)
• Set pod anti-affinity so replicas spread across nodes
• Simulate a node failure (kubectl delete node); verify zero-downtime
• Run a LitmusChaos pod-delete experiment; observe Grafana during chaos
• Write a DR runbook: "Region us-east-1 is unavailable — steps to fail over to us-west-2"

Week 11: Cloud Platforms — AWS, GCP, Azure

Goal: Navigate and operate managed services across the three major clouds.

Topics (parallel track — pick one cloud deep, survey the other two):

AWS (primary):

• EKS: cluster creation, node groups, Fargate profiles, IAM Roles for Service Accounts (IRSA)
• ECR, ECS (awareness), Lambda (awareness)
• VPC, subnets, security groups, NACLs, VPC peering
• IAM: roles, policies, least-privilege, instance profiles
• RDS, ElastiCache, S3, CloudFront
• CloudWatch: Logs, Metrics, Alarms, Container Insights
• Load balancers: ALB, NLB, AWS Load Balancer Controller
• Terraform / AWS CDK for infra-as-code

GCP (secondary):

• GKE Autopilot vs Standard; Workload Identity
• Cloud Run, Cloud Functions (awareness)
• VPC, Cloud NAT, Shared VPC
• Cloud Monitoring, Cloud Logging, Error Reporting
• Artifact Registry, Cloud Build

Azure (tertiary):

• AKS, ACR, Azure DevOps Pipelines
• Azure Monitor, Log Analytics Workspace
• Managed identities, RBAC

Hands-on Labs:

• Stand up an EKS cluster with Terraform (or eksctl); deploy sample app
• Configure IRSA so pods can read from S3 without storing credentials
• Set up CloudWatch Container Insights on EKS
• Deploy the same app to GKE; compare CLI/console experience with AWS
• Use Terraform to provision VPC + EKS + RDS as code; terraform plan/apply/destroy

Resources:

• Terraform: Up & Running by Yevgeniy Brikman
• AWS Well-Architected Framework (free whitepaper)
• GCP Architecture Center (cloud.google.com/architecture)

Week 12: Integration Week — Full-Stack Ops System

Goal: Wire everything learned into one coherent, production-grade ops system.

Deliverables:

• Sample microservice app running on EKS (or GKE) via ArgoCD GitOps
• Full Prometheus + Grafana + Loki observability stack
• GitHub Actions CI pipeline: lint → test → build → push → trigger ArgoCD sync
• Canary rollout via Argo Rollouts with automatic rollback on error rate spike
• Velero scheduled backups to S3 with verified restore
• DR runbook documented and tested (simulate region failure)
• All infra defined in Terraform; terraform apply from scratch in < 30 min
• On-call runbook: links to dashboards, rollback steps, escalation path