Look closely at what an AI feature actually does over a day and you find two very different kinds of work mixed together. Some of it a person is waiting on right now: the reply in the chat, the code the developer asked for, the answer on the screen. The rest is work that only you care about the timing of: nightly summaries, re-analyzing a repository after a merge, classifying a backlog, grading last week's outputs. The second kind does not need to be fast. It just needs to be done by some deadline that is hours, not milliseconds, away.

Most teams pay the same price for both. Every call goes through the same real-time API at the same real-time rate, including the half that no human is waiting on. That is the easiest money you are leaving on the table, and picking it up is what the batch API is for. Cutting your batch API LLM cost is not a clever trick. It is just refusing to pay a rush fee on work that was never in a rush.

The lever: roughly half off, for patience

The major model providers offer a batch mode: you submit a set of requests, they process them within a turnaround window (commonly up to 24 hours), and in exchange you pay about 50 percent less per token on both input and output. Treat that number as a live figure to confirm against current provider pricing, because rates move, but the shape has held: you trade immediacy for a large, standing discount.

That is a different lever from the other two in this cluster. Prompt caching cuts the cost of repeated context, and it stacks on top of batching. Model routing sends easy work to a cheaper model. Batching is orthogonal to both: it discounts the work regardless of which model runs it or what the prompt looks like, purely because you were willing to wait. You can apply all three to the same job. The point of prompt caching is to stop paying for the same tokens twice; the point of model routing is to stop overpaying for easy tokens; the point of batching is to stop paying rush rates on patient tokens.

What actually belongs in the batch lane

The test is simple: is a human blocked on this result? If nobody is watching the spinner, the work is a batch candidate. In a real coding-agent system, that turns out to be a lot of the total volume:

  • Analysis over a codebase. Scanning a repository, building or refreshing an understanding of it, mapping dependencies. Nobody is staring at the screen while it runs.
  • Offline evaluation. Grading yesterday's agent runs, scoring outputs against a rubric, regression checks. These feed dashboards, not a live request.
  • Bulk classification and extraction. Labeling a backlog of issues, tagging documents, extracting fields from a pile of inputs.
  • Re-processing after a change. Recomputing something because a file merged or a config changed, where "within the hour" is more than good enough.

The hot lane, by contrast, is anything a person is actively waiting on: the interactive coding turn, the answer in the UI, the review that a developer is standing by for. That work stays on the real-time path, because there latency is the product and a discount that costs you hours is not a discount worth taking.

Splitting the pipeline into two lanes

The change is mostly organizational, not technical. You are drawing a line through your workload and routing each job to the side it belongs on.

1. Tag every job kind by latency need. Go through the async jobs your system runs and mark each one "interactive" or "deferrable." Be honest. A lot of things feel urgent that are not; if the result lands in a report or a queue rather than in front of a waiting person, it is deferrable. 2. Give the deferrable work its own path. Collect those jobs and submit them through the provider's batch endpoint instead of the real-time one. In practice this means an async submit-and-poll flow: you hand over a set of requests, you get an id, you collect results when the window closes. 3. Size the window to the work, not the wish. If a summary needs to exist by morning, a several-hour turnaround is fine. Set expectations downstream so nothing in the product assumes an instant answer from a job you deliberately made patient. 4. Keep a fallback for the occasional rush. Sometimes a normally-deferrable job genuinely becomes urgent. Allow a per-job override that promotes it to the hot lane, and treat that as the exception you pay full price for on purpose.

Done well, the split is invisible to users and shows up only on the bill. The interactive experience is unchanged, and a large slice of your token spend quietly drops by about half.

Where it fits with everything else

Batching is one move in a larger cost discipline, and it pairs naturally with knowing your ceiling. If you are also capping runaway agent spend with budgets and iteration limits, the batch lane is where a lot of the non-urgent spend lives, so it is a good place to apply looser but real caps. The full picture of where the money goes, and which lever to reach for first, is laid out in the economics of AI coding agents.

The honest limitation

Batching buys you a discount by spending your patience, and patience is not always yours to spend. The turnaround window is a real constraint: if a job's result is needed in seconds, the batch lane is simply the wrong tool, and forcing it there to save money is how you ship a slow product. There is also operational overhead in running an async submit-and-collect flow that a single synchronous call does not have, so very low volumes may not be worth the plumbing. And a batch job that fails partway still has to be reconciled, which is its own small engineering cost.

So treat batching as a lever you pull where it fits, not a default you force everywhere. The saving is real and large, but only on work that was genuinely willing to wait.

This is a distinction we lean on inside Loopsfinity: the work a person is waiting on runs on the fast path, and the analysis that is not blocking anyone runs where it is cheaper. How we draw and operate that line is our own, but the principle is available to any team willing to ask, job by job, whether anyone is actually waiting.