59. Built-in AI and streaming conversational UIs

The browser's own models

Chrome ships a family of built-in AI APIs backed by an on-device model (Gemini Nano), downloaded once by the browser and shared by every site. Your page downloads no weights at all:

Summarizer, Translator and Language Detector: stable in desktop Chrome since 138.
Prompt API (LanguageModel): stable for Chrome extensions. On regular web pages, Chrome has run it as an origin trial. Check its status for your Chrome version before relying on it.
Writer, Rewriter and Proofreader: origin trials.

Requirements are steep: a desktop OS, several GB of free disk and a capable GPU. Treat these APIs as a progressive enhancement:

async function summarize(text) {
  if (!("Summarizer" in self)) return null;                  // not supported: hide the button
  const availability = await Summarizer.availability();      // "unavailable" | "downloadable" | "downloading" | "available"
  if (availability === "unavailable") return null;
  const summarizer = await Summarizer.create({
    type: "key-points", format: "markdown", length: "short",
    monitor(m) { m.addEventListener("downloadprogress", (e) => showProgress(e.loaded)); },
  });
  let out = "";
  for await (const chunk of summarizer.summarizeStreaming(text)) out += chunk;
  summarizer.destroy();
  return out;
}

// Prompt API, where available (extensions, or a page enrolled in the origin trial):
const session = await LanguageModel.create({
  initialPrompts: [{ role: "system", content: "You classify support tickets." }],
});
const label = await session.prompt(ticketText, {
  responseConstraint: { type: "string", enum: ["billing", "shipping", "account", "other"] },
});

The APIs are standardized in the W3C WebML community group, so other engines may implement the same shapes. Microsoft Edge ships an implementation backed by its own on-device model.

Streaming chat UI: the details that make it feel fast

Whether tokens come from WebLLM, a built-in model or a server, the UI problems are the same. Each pattern below is in the Support Desk project (§63) and covered by its Playwright suite (§77).

1. Read Server-Sent Events over fetch, not EventSource. EventSource can only send GET requests without custom headers, and chat needs POST with a body and a CSRF header. Parsing SSE yourself takes about 20 lines:

// api.js — yields one parsed event per `data:` frame
export async function* readEvents(res) {
  const reader = res.body.pipeThrough(new TextDecoderStream()).getReader();
  let buf = "";
  for (;;) {
    const { value, done } = await reader.read();
    if (done) return;
    buf += value;
    let end;
    while ((end = buf.indexOf("\n\n")) !== -1) {      // a blank line ends one event
      const frame = buf.slice(0, end);
      buf = buf.slice(end + 2);
      const data = frame.split("\n").filter((l) => l.startsWith("data:"))
                        .map((l) => l.slice(5).trimStart()).join("\n");
      if (data) yield JSON.parse(data);
    }
  }
}

2. Batch DOM writes to one per frame. A fast model can deliver 100+ deltas per second. Appending each one forces style and layout work per token. Buffer the deltas and flush once per requestAnimationFrame (§15), which is at most one write per frame:

let pending = "", scheduled = false;
const flush = () => { bubble.append(pending); pending = ""; scheduled = false; };
for await (const event of readEvents(res)) {
  if (event.type === "text-delta") {
    pending += event.delta;
    if (!scheduled) { scheduled = true; requestAnimationFrame(flush); }
  }
}
if (pending) flush();

3. Stop means abort. Pass an AbortController's signal to fetch. Aborting closes the connection, the server sees close and stops generating (Support Desk's server cancels its agent run), and the reader throws an AbortError that the UI renders as "(stopped)".

4. Accessibility. Put the transcript in a role="log" container. It is an implicit aria-live="polite", so screen readers announce new messages without interrupting. Set aria-busy="true" while streaming. Support Desk's first version put role="log" on an <ol>, which removes the list semantics and leaves the <li> children orphaned. axe-core caught it in the test suite (§77), and it was fixed by making the log a <div>.

5. Render model output as text, or sanitize it. Model output is untrusted input (§66). The safe default is textContent. For Markdown, render to HTML and then sanitize it with the browser's Sanitizer API, falling back to DOMPurify:

function renderMarkdown(el, markdown) {
  const html = markdownToHtml(markdown);            // e.g. marked / micromark
  if ("setHTML" in Element.prototype) el.setHTML(html);   // Firefox 148+, Chrome 146+: strips scripts & handlers
  else el.innerHTML = DOMPurify.sanitize(html, { RETURN_TRUSTED_TYPE: true });
}

6. Scroll politely. Auto-scroll only if the user is already at the bottom (el.scrollHeight - el.scrollTop - el.clientHeight < 40). Someone scrolling back to reread an answer should not be yanked down by each new token.