62. Edge functions: JavaScript next to your users

Edge functions run your JavaScript in data centers close to users: Cloudflare Workers, Vercel Functions on the edge runtime, Deno Deploy, Netlify Edge Functions, Fastly Compute. Most of them run code in V8 isolates, the same sandboxing unit as a browser tab (§18), not in containers. One process hosts thousands of isolates, an isolate starts in milliseconds, and memory is measured in megabytes. The trade-offs: a CPU time budget per request, ~128 MB of memory, and Node built-ins only through compatibility layers. The programming model is the browser's: Request, Response, fetch, crypto.subtle and Web Streams (§54).

The most common edge job in AI apps is an LLM gateway: authenticate the user, enforce limits, attach the secret API key, and stream the response back. This one was tested in Node 22 against a fake OpenAI-compatible upstream (a self-hosted vLLM, Ollama or llama.cpp server looks the same):

// worker.js — Workers module syntax; only web-standard APIs, so it ports to
// Deno Deploy, Vercel, Netlify and Bun unchanged.
// env: UPSTREAM_URL, UPSTREAM_KEY, MODEL, JWT_SECRET, APP_ORIGIN, RATE_LIMITER (optional)

const enc = new TextEncoder();
const b64url = (s) => Uint8Array.from(atob(s.replace(/-/g, "+").replace(/_/g, "/")), (c) => c.charCodeAt(0));

async function verifyJWT(token, secret) {
  const [header, payload, signature] = token.split(".");
  if (!signature) return null;
  const { alg } = JSON.parse(new TextDecoder().decode(b64url(header)));
  if (alg !== "HS256") return null;                     // never let the token pick its own algorithm
  const key = await crypto.subtle.importKey("raw", enc.encode(secret), { name: "HMAC", hash: "SHA-256" }, false, ["verify"]);
  const valid = await crypto.subtle.verify("HMAC", key, b64url(signature), enc.encode(`${header}.${payload}`));
  if (!valid) return null;
  const claims = JSON.parse(new TextDecoder().decode(b64url(payload)));
  return claims.exp * 1000 > Date.now() ? claims : null;
}

// Best-effort per-isolate limiter; production uses the platform's limiter binding
// or a Durable Object, because isolates are many and short-lived.
const hits = new Map();
async function allow(env, key) {
  if (env.RATE_LIMITER) return (await env.RATE_LIMITER.limit({ key })).success;
  const now = Date.now();
  const h = hits.get(key);
  if (!h || h.reset < now) return hits.set(key, { n: 1, reset: now + 60_000 }), true;
  return ++h.n <= 20;
}

const json = (status, body, headers = {}) =>
  new Response(JSON.stringify(body), { status, headers: { "Content-Type": "application/json", ...headers } });

export default {
  async fetch(request, env, ctx) {
    const cors = {
      "Access-Control-Allow-Origin": env.APP_ORIGIN,    // one origin, never "*" (§28)
      "Access-Control-Allow-Headers": "Authorization, Content-Type",
      "Access-Control-Allow-Methods": "POST",
      Vary: "Origin",
    };
    if (request.method === "OPTIONS") return new Response(null, { status: 204, headers: cors });
    if (request.method !== "POST" || new URL(request.url).pathname !== "/chat") return json(404, { error: "Not found" }, cors);

    const token = request.headers.get("Authorization")?.replace(/^Bearer /, "") ?? "";
    const user = await verifyJWT(token, env.JWT_SECRET);
    if (!user) return json(401, { error: "Unauthorized" }, cors);
    if (!(await allow(env, user.sub))) return json(429, { error: "Too many requests" }, { ...cors, "Retry-After": "60" });

    const { messages } = await request.json().catch(() => ({}));
    if (!Array.isArray(messages) || messages.length > 40) return json(400, { error: "Bad messages" }, cors);
    const clean = messages
      .filter((m) => m.role === "user" || m.role === "assistant")      // the client can't smuggle in "system"
      .map((m) => ({ role: m.role, content: String(m.content).slice(0, 4000) }));

    // The server, not the client, picks the model and the token budget.
    const upstream = await fetch(`${env.UPSTREAM_URL}/v1/chat/completions`, {
      method: "POST",
      headers: { "Content-Type": "application/json", Authorization: `Bearer ${env.UPSTREAM_KEY}` },
      body: JSON.stringify({
        model: env.MODEL, stream: true, max_tokens: 800,
        messages: [{ role: "system", content: "You are the Support Desk assistant." }, ...clean],
      }),
      signal: request.signal,                           // user hits Stop -> the upstream generation is cancelled
    });
    if (!upstream.ok || !upstream.body) return json(502, { error: "Model unavailable" }, cors);

    // Stream straight through, metering bytes on the way.
    let bytes = 0;
    const meter = new TransformStream({
      transform(chunk, controller) { bytes += chunk.byteLength; controller.enqueue(chunk); },
      flush() { ctx.waitUntil(logUsage(env, { user: user.sub, bytes })); },   // after the response, off the hot path
    });
    return new Response(upstream.body.pipeThrough(meter), {
      headers: { ...cors, "Content-Type": "text/event-stream", "Cache-Control": "no-store" },
    });
  },
};

What the test run checked:

Edge design notes:

• The response streams through without buffering. Returning upstream.body.pipeThrough(...) passes chunks on as they arrive, with backpressure (§54). Never await upstream.text() in a proxy, because the user would see nothing until the whole answer was done.
• ctx.waitUntil keeps the isolate alive after the response for logging and metering, without delaying the user.
• Data gravity. An edge function 10 ms from the user that queries a database 150 ms away is slower than a server next to the database. Use the edge for auth, routing, caching, rate limits, A/B assignment and personalization from cookies. Put heavy data work next to the data.
• Frameworks. Hono (§73) is the de facto router for this world. Its middleware for JWT, CORS and rate limits replaces most of the hand-written code above.