58. WebLLM: running a real LLM entirely in the browser

WebLLM (@mlc-ai/web-llm) runs quantized open-weight LLMs (Llama, Qwen, Phi, Gemma, SmolLM and more, 160+ prebuilt builds at the time of writing) on WebGPU, behind an OpenAI-compatible API. The MLC compiler (Apache TVM) turns each model into WebGPU kernels ahead of time. Weights are 4-bit quantized (q4f16_1 means 4-bit weights with fp16 activations) and cached by the browser after the first download.

Numbers from the run for this guide, on an Apple-silicon laptop:

Bigger models are smarter and much heavier. A 1B model is roughly 0.9 GB, and a 7–8B model is 4–5 GB and needs a capable GPU. Pick the smallest model that does the job: extraction, classification, routing and short rewrites work well at 0.5–3B parameters.

The code: engine in a worker, UI on the main thread

// llm-worker.js — the engine (and its GPU work) lives off the main thread
import { WebWorkerMLCEngineHandler } from "@mlc-ai/web-llm";
const handler = new WebWorkerMLCEngineHandler();
self.onmessage = (msg) => handler.onmessage(msg);

// main.js
import { CreateWebWorkerMLCEngine, prebuiltAppConfig } from "@mlc-ai/web-llm";

const MODEL = "Qwen2.5-0.5B-Instruct-q4f16_1-MLC";
if (!navigator.gpu) throw new Error("No WebGPU: fall back to the edge model (§62)");

await navigator.storage.persist?.();      // ask the browser not to evict 266 MB of weights
const engine = await CreateWebWorkerMLCEngine(
  new Worker(new URL("./llm-worker.js", import.meta.url), { type: "module" }),
  MODEL,
  { initProgressCallback: ({ progress, text }) => showProgress(progress, text) },
);

// Streaming, exactly like the OpenAI SDK:
const chunks = await engine.chat.completions.create({
  messages: [
    { role: "system", content: "Answer in one short sentence." },
    { role: "user", content: "What does the event loop do in JavaScript?" },
  ],
  stream: true,
  stream_options: { include_usage: true },
  temperature: 0,
});
for await (const chunk of chunks) {
  appendToBubble(chunk.choices[0]?.delta?.content ?? "");
  if (chunk.usage) console.log(chunk.usage.extra);   // tokens/s, time to first token…
}
// stop button: engine.interruptGenerate();   new conversation: engine.resetChat();

Structured output: JSON mode with a schema

Small models are unreliable when asked to "reply in JSON". WebLLM can constrain decoding to a JSON Schema, so the output is always parseable and always has the shape you asked for:

const res = await engine.chat.completions.create({
  messages: [{ role: "user", content: "Extract the order id and intent from: 'please refund my order A-1001, it arrived broken'" }],
  response_format: {
    type: "json_object",
    schema: JSON.stringify({
      type: "object",
      properties: { orderId: { type: "string" }, intent: { type: "string", enum: ["refund", "track", "other"] } },
      required: ["orderId", "intent"],
    }),
  },
  temperature: 0,
});
JSON.parse(res.choices[0].message.content);   // { orderId: "A-1001", intent: "refund" }  (real output)

Constrained JSON turns a 0.5B model into a dependable intent router: it classifies the request on the device, privately, and only the actions that need a server go over the network.

Production checklist for in-browser LLMs

• Capability check first. No navigator.gpu, too little memory, or a mobile device on battery: use a remote model with the same interface (see the provider pattern below and §62).
• Make the download honest. Show the size before downloading, show progress, and offer "use the cloud model instead". Cache Storage holds the weights. Check navigator.storage.estimate() against the quota.
• Handle GPU loss. Drivers reset and laptops switch GPUs. device.lost (or an engine error) should trigger a reload, not a blank page.
• Treat output as untrusted (§66). An LLM in the browser is still an LLM: prompt injection, invented facts and unsafe HTML all apply.

One interface for every model keeps the UI ignorant of where the model runs:

// provider.js — the UI only ever calls provider.stream(messages, { signal })
export function webllmProvider(engine) {
  return {
    async *stream(messages, { signal } = {}) {
      signal?.addEventListener("abort", () => engine.interruptGenerate(), { once: true });
      const chunks = await engine.chat.completions.create({ messages, stream: true });
      for await (const c of chunks) yield c.choices[0]?.delta?.content ?? "";
    },
  };
}

export function edgeProvider(url, getToken) {           // §62's edge function
  return {
    async *stream(messages, { signal } = {}) {
      const res = await fetch(url, {
        method: "POST", signal,
        headers: { Authorization: `Bearer ${await getToken()}`, "Content-Type": "application/json" },
        body: JSON.stringify({ messages }),
      });
      if (!res.ok) throw new Error(`model error ${res.status}`);
      for await (const event of readSSE(res)) {           // §59
        if (event === "[DONE]") return;
        yield JSON.parse(event).choices[0]?.delta?.content ?? "";
      }
    },
  };
}

export const provider = navigator.gpu ? webllmProvider(await loadEngine()) : edgeProvider("/chat", getToken);