46. Distributed Job Scheduler

Business context. An internal platform (like cron, Airflow, or a batch scheduler) that many teams use for billing runs, reports, ML training, and cleanups. Outcomes: jobs run on time, exactly when they should, at most once where that matters, with operational visibility. A missed billing run is revenue lost or delayed.

Requirements. Schedule one-off and recurring jobs (cron expressions); dependencies (DAGs); retries with backoff; priorities and quotas per team; timeouts; 10M job executions per day; no single point of failure; a job should not run twice concurrently unless that's allowed.

Design.

API/UI ─▶ Job definitions DB (strongly consistent)
Scheduler (leader-elected or sharded by job_id hash; each shard owned via lease):
   scans "next_run_at <= now + lookahead" using an index on next_run_at
   ─▶ creates execution record (job_id, scheduled_time) UNIQUE  ← dedupe across scheduler failover
   ─▶ enqueue to priority queues (per team quota)
Workers ─▶ claim execution (lease with heartbeat + fencing token) ─▶ run ─▶ report status
   lease expires (worker died) ─▶ execution returns to queue (at-least-once) ─▶ jobs must be idempotent,
   or use fencing tokens when writing to external systems
DAG engine: when an execution succeeds, evaluate dependents ─▶ enqueue ready nodes
Observability: per-job SLA alerts ("billing job not done by 06:00"), history, logs

Deep dives.

• Exactly-once scheduling: the unique (job_id, scheduled_time) execution record means a scheduler failover can't create a duplicate run.
• At-least-once execution: workers can die mid-job. Leases and heartbeats detect this, and jobs must be idempotent (or use checkpoints and fencing).
• Thundering herd at midnight: everyone schedules jobs at 0 0 * * *. Spread jobs that tolerate it with jitter, and enforce team quotas and priorities.
• Time zones and DST: store schedules with an explicit time zone; define what happens on the skipped and repeated hours.

Strategic lens. Buy or adopt before building (Temporal, Airflow, Argo, cloud schedulers). Build only if there are scale, multi-tenancy, or integration needs those can't meet. The platform's value is in ownership, alerting, and quotas, not the scheduling loop itself.

🏗️ Flow: firing a cron job safely across failover.

 Scheduler shard S (owns job_ids by hash, holds lease L with fencing token 57)
   12:00:00  due: job "nightly-billing" scheduled_time=2026-10-07T00:00Z
   1. INSERT executions(job_id, scheduled_time, token=57) UNIQUE(job_id, scheduled_time)
      ├─ conflict → already launched by previous owner → skip
   2. enqueue execution → queue(priority=high, team=billing)
   3. worker claims (lease 60 s, heartbeat every 20 s) → runs → writes results with execution_id
   4. success → mark done → trigger dependents (DAG) ; failure → retry policy (backoff, max 3)
   S crashes after step 1, before 2 → new owner (token 58) scans "launched but not enqueued"
      → enqueues (idempotent by execution id)

🌍 How it's built in production.

• Google's distributed cron (SRE book chapter "Distributed Periodic Scheduling with Cron"): the cron service is Paxos-replicated, and the leader records its intent to launch a job before launching it, plus the completion afterwards. A new leader can then tell whether a launch may have happened and avoid double launches. Jobs are encouraged to be idempotent, because the gap between "intended" and "done" can't be closed completely.
• Meta "Async" (2023, "Asynchronous computing at Meta") and FOQS: a platform for deferred and async work at massive scale, with priorities, quotas, and capacity-aware scheduling, so that "run later" jobs fill off-peak capacity.
• Airflow (from Airbnb), Temporal (from Uber's Cadence), and Argo Workflows (Kubernetes-native): the usual buy/adopt options.

⚠️ Corner cases.

• Missed runs after downtime: the scheduler was down from 00:00 to 03:00. Run every missed instance (catch-up), only the latest, or none? Make it a per-job policy (catchup=true/false).
• Overlapping runs: a job takes 70 minutes on a 60-minute schedule. Choose skip, queue, or allow concurrency per job, and enforce it with a per-job lock plus a fencing token.
• DST: a job at 02:30 local time doesn't exist on spring-forward day and happens twice on fall-back day. Define the behavior, or schedule critical jobs in UTC.
• Upstream data not ready: a job runs on time on incomplete input. Use data-availability sensors (wait for the partition or a success marker) instead of time-based assumptions.
• Backfills that compete with production: a 90-day backfill starves the daily SLA jobs. Use separate queues and quotas for backfills.
• Zombie workers: a worker whose lease expired keeps running and writes results after a retry has started. Fencing tokens on output writes, or idempotent overwrite by execution ID.