54. Iterators, generators and streams: data that arrives over time

Real applications deal in sequences that are big, slow or endless: paginated APIs, log tails, LLM token streams, file uploads. JavaScript gives you one model for all of them. A pull-based iterator hands out the next value on request. A stream adds asynchrony and backpressure on top.

Iterators and generators

The protocol is small. An iterable has a [Symbol.iterator]() method that returns an iterator. An iterator has a next() method that returns { value, done }. Generators write that state machine for you, pausing at every yield:

function countdown(n) {                 // the protocol, by hand
  return {
    [Symbol.iterator]() { return this; },
    next: () => (n > 0 ? { value: n--, done: false } : { value: undefined, done: true }),
  };
}
[...countdown(3)];                      // [3, 2, 1]

function* fibonacci() {                 // the same idea, as a generator
  let [a, b] = [0, 1];
  for (;;) { yield a; [a, b] = [b, a + b]; }   // infinite, and that's fine: it's lazy
}

yield works in both directions. next(value) resumes the generator, and the paused yield expression evaluates to value. Redux-Saga built a whole effects system on this, and it is how async/await was first implemented on top of generators (§14):

function* conversation() {
  const name = yield "What's your name?";
  const lang = yield `Hi ${name}! Favourite language?`;
  return `${name} likes ${lang}`;
}
const chat = conversation();
chat.next().value;               // "What's your name?"
chat.next("Ada").value;          // "Hi Ada! Favourite language?"
chat.next("JavaScript");         // { value: "Ada likes JavaScript", done: true }

Iterator helpers (ES2025): lazy pipelines without arrays

Array methods are eager. arr.map(f).filter(g).slice(0, 5) creates two full intermediate arrays. Iterator helpers put map, filter, take, drop, flatMap, reduce, some, every, find, forEach and toArray on every iterator, and they evaluate lazily, one value at a time. That makes infinite sequences usable:

fibonacci()
  .filter((n) => n % 2 === 0)
  .map((n) => n.toLocaleString("en-US"))
  .take(6)
  .toArray();          // ["0", "2", "8", "34", "144", "610"]: only 15 fibs ever computed

// Map/Set iterators get the helpers too; Iterator.from() wraps any iterable.
new Set(["js", "browser", "v8", "css"]).values().filter((t) => t.length <= 2).toArray(); // ["js", "v8"]

Async iteration: pagination that reads like a loop

An async generator can await and yield. Consumers drive it with for await...of. The caller never sees the pagination, and nothing is fetched until the caller asks for it:

async function* allUsers() {
  let page = 0;
  while (page !== null) {
    const res = await fetch(`/api/users?page=${page}`).then((r) => r.json());
    yield* res.users;              // hand out this page's users one at a time
    page = res.next;               // null on the last page
  }
}

for await (const user of allUsers()) {
  if (user.id === "user4") break;  // break calls the generator's return(): no more fetches
  render(user);
}

const everyone = await Array.fromAsync(allUsers());   // ES2026: collect it all

Web Streams: chunks, transforms and backpressure

ReadableStream, WritableStream and TransformStream are the platform's streaming primitives. They are the body of every fetch response, they back CompressionStream and TextDecoderStream, and the same API runs in Node.js, Deno, Bun and every edge runtime (§62). A TransformStream sits in the middle of a pipe, and pipeThrough connects them.

The most useful transform you will write parses a byte stream into messages. The network splits data wherever it likes, so a JSON object can arrive in three pieces. Buffer until you have a full line:

// Newline-delimited JSON, parsed as it streams in: the shape of LLM APIs,
// log tails and progress feeds.
function ndjson() {
  let buffer = "";
  return new TransformStream({
    transform(chunk, controller) {
      buffer += chunk;
      const lines = buffer.split("\n");
      buffer = lines.pop();                  // keep the incomplete last line
      for (const line of lines) if (line.trim()) controller.enqueue(JSON.parse(line));
    },
    flush(controller) {                      // the stream ended: parse what's left
      if (buffer.trim()) controller.enqueue(JSON.parse(buffer));
    },
  });
}

const res = await fetch("/api/generate", { method: "POST", body });
const events = res.body.pipeThrough(new TextDecoderStream()).pipeThrough(ndjson());
for await (const e of events) if (e.token) output.append(e.token);

In the test run, the input arrived in 7-byte chunks ({"token":"Hel"}\n{"tok…) and the output was still exactly Hello, world.

Backpressure is what makes streams more than callbacks. Every stream has a queue with a high-water mark. A pull-based source is asked for more data only while the queue is below it, so a slow consumer automatically slows the producer down:

let produced = 0;
const stream = new ReadableStream(
  { pull(controller) { controller.enqueue(++produced); if (produced === 100) controller.close(); } },
  new CountQueuingStrategy({ highWaterMark: 3 }),
);
const reader = stream.getReader();
// …50ms later, nobody has read anything:
produced;            // 3, not 100: the source stopped when the queue was full
await reader.read();
produced;            // 4: one slot freed, one more pulled

The same mechanism makes fetch upload 5 GB files without holding them in memory, and lets a Service Worker stream a response while it is still being generated.

Support note. for await (const chunk of readableStream) works in Chrome 124+, Firefox and Node, but check Safari before relying on it. stream.getReader() with a read() loop works everywhere, which is why §59's chat client uses it.

🎯 Challenge. Write async function* lines(response) that yields one line of text at a time from any fetch response. Then use it with iterator-style code: count the lines of a 100 MB log without holding more than one chunk in memory.