Ask a model for a structured result and the simplest thing to do is wait. The request goes out, ten or twenty seconds pass, a complete JSON blob comes back, you parse it, you move on. It works, and for a background job it is exactly right. For anything a human is watching, it feels broken. The screen sits empty while the model thinks, and empty screens read as failure.

Streaming fixes the feel. Instead of waiting for the whole response, you consume the model's output token by token and surface it as it arrives, so the user sees the answer building rather than a spinner. The catch is that streaming structured output is a genuinely harder problem than streaming plain text, because a half-finished JSON object is not valid JSON, and your parser knows it.

What streaming actually buys you

The win is perceived latency, not total latency. The model takes just as long to finish, but the user stops waiting on the first token instead of the last one. For a long structured result, that is the difference between an interface that feels dead and one that feels alive.

There is a second, quieter benefit for agents specifically: you can act on parts of the result before the whole thing lands. If an agent streams a plan as a list of steps, you can start rendering, validating, or even dispatching the early steps while the later ones are still being written. Whether that is safe depends entirely on the work, which is the pitfall we will come back to.

Why partial JSON is the hard part

Plain text streams trivially: every chunk is just more text to append. Structured output does not, because the format has rules that a prefix violates. Halfway through generation you might be holding this:

{"summary": "the auth module handles token refr

That is not parseable. The string is unterminated, the object is unclosed, and a standard JSON.parse will throw. You cannot simply run the normal parser on each chunk and hope. You need a parser that tolerates an incomplete document and tells you what it can about the part that has arrived.

Approach 1: a tolerant partial parser

The most general approach is an incremental parser that accepts a prefix of a JSON document and returns the best complete interpretation of what it has seen so far. Conceptually it closes the open structures for you: the unterminated string becomes a complete string with the text so far, the open object becomes a valid object with the keys completed to that point.

You do not usually write this yourself. Several small libraries do exactly this (partial or streaming JSON parsers), and the pattern is the same across them: feed the accumulated buffer on each update, get back a valid object representing the current state.

let buffer = "";
for await (const chunk of modelStream) {
  buffer += chunk;
  const partial = parsePartialJson(buffer); // tolerant of an incomplete tail
  if (partial) render(partial);             // partial is a valid object so far
}
const final = JSON.parse(buffer);           // strict parse once the stream ends

The important habit is the last line. Stream for the feel, but do a strict parse and full validation once the stream closes. The partial views are for display and early reaction; the authoritative object is the completed one.

Approach 2: yield fields as they complete

A lighter approach, when you control the schema, is to stream at the granularity of top-level fields rather than characters. Ask the model to emit the object in a known key order, and surface each value as its key finishes. The user sees summary fill in, then risks, then steps, each appearing whole. It is less general than a tolerant parser, but it sidesteps most of the half-token ugliness because you only ever show a field once it is complete.

Approach 3: schema-guided parsing

If you already know the shape you expect, the parser can lean on it. Knowing that steps is an array of objects with a title and a detail lets the parser make sensible decisions about a partial tail, and lets you type the partial result instead of passing around loose dictionaries. Several structured-output libraries combine a schema with streaming for this reason: the schema is both the validation contract at the end and a guide for interpreting the middle.

The pitfalls that bite

  • Acting on data that later changes. A field visible mid-stream is not final until the stream ends. If you dispatch a side effect based on a partial value, you can act on something the model revises a token later. Read partial state for display; gate irreversible actions on the completed, validated object.
  • Validating only at the end. Streaming can tempt you to skip validation because the data looked fine as it flowed by. Do not. The strict parse and schema check at close is non-negotiable, because a stream that looked healthy can still finish malformed.
  • Retries mid-stream. If the connection drops halfway, you have a partial buffer and a decision: restart from scratch or attempt to resume. Most designs restart, because resuming a token stream cleanly is more trouble than it saves. Make that choice on purpose.
  • UI flicker. Re-rendering the whole partial object on every token can thrash the interface. Debounce updates, or render at field granularity, so the screen builds smoothly instead of strobing.

When not to stream

Streaming is a UX optimization, and it has a cost in complexity. If the output is short, if no human is waiting, or if you cannot safely use the result until it is complete anyway, skip it. A background classification step gains nothing from streaming and pays for the extra parsing machinery. Stream where a person is watching a long result build. Everywhere else, wait for the whole thing and keep the code simple. This is the same budgeting instinct that shows up when you treat the context window as an architecture constraint: spend complexity where it earns its keep.

The honest limitation

Streaming makes a slow response feel fast. It does not make it correct, and it does not make it faster. The model still takes as long, the answer is still only trustworthy once validated, and the partial views you show are a courtesy to the user, not a source of truth your system should rely on. Done carelessly, streaming trades a spinner for a subtly wrong early render, which is a worse failure than honest waiting. The discipline is to stream the appearance and gate the substance.

It also interacts with portability: not every provider streams structured output the same way, so a design that leans hard on one provider's streaming behavior is a design you will rework when you design for model swaps.

At Loopsfinity we stream where a person is watching a long result and fall back to a plain wait where nothing is, because the point is a responsive interface, not streaming for its own sake. The mechanics of how we parse and validate are ordinary; the judgment is knowing which outputs deserve the treatment. That judgment is part of the wider picture in AI agent architecture.