The first time you get an LLM to return clean JSON that matches your schema, it feels like the hard part is over. The model stops rambling. Every field is present, every enum is legal, the parser never throws. You wire it into a pipeline and move on.

Then, a week later, something downstream breaks in a way that makes no sense, and you trace it back to a field that was perfectly formatted and completely false. The agent reported that a function lived in auth/session.ts. The JSON was valid. The file was real. The function was not in it.

Schema validation tells you the answer has the right shape. It tells you nothing about whether the answer is true. For a chatbot, that gap is a nuisance. For an agent that acts on its own output, it is the whole ballgame.

Two different failures, two different checks

It helps to separate two things a model can get wrong, because they need different defenses.

The first is format: missing fields, illegal enums, malformed nesting. Structured output and schema validation handle this well now. This problem is largely solved, and you should lean on it.

The second is substance: the shape is legal but the content is wrong. A referenced file that does not exist. A dependency claimed between two modules that never import each other. A cost estimate with no basis. No schema can catch these, because nothing about them is malformed. They are lies told in valid grammar.

The mistake is assuming the first check covers the second. It does not, and the more confident your model is, the more dangerous the gap becomes.

Cheap checks first, expensive checks second

The good news is that a lot of substance errors are catchable by boring, deterministic code, and you should exhaust those before you reach for anything cleverer.

If the model names a file, check that the file exists. If it names a symbol, grep for it. If it claims module A depends on module B, look at the imports. These are not AI problems; they are fs.existsSync and a parser. They are fast, free, and deterministic, and they catch a surprising share of confident hallucinations before those errors ever propagate.

Only once the cheap checks pass is it worth spending a model call to judge the things code cannot: is this plan coherent, does this summary actually reflect the diff, is this acceptance criterion testable. Using an LLM as a judge works, but it is the expensive layer, so put it last and keep each judgment narrow. One call that asks "is this one specific claim supported by this one specific piece of evidence" is far more reliable than one call asking "is all of this good." Put together, those two layers, cheap deterministic checks and narrow judges, are what a real eval harness for a coding agent is built from.

Not everything that fails is a failure

There is a subtler trap. When you start grading agent output, you will find cases the agent could not have gotten right: the input was underspecified, a dependency was missing, the task was genuinely blocked. If you score those as failures, you punish the agent for being honest about uncertainty, and you teach your metrics to lie.

So distinguish wrong from blocked. An agent that says "I cannot answer this without X" is behaving correctly. Fold that into your evaluation as its own outcome, not as a failure, or your quality numbers will quietly reward guessing over honesty. That is the opposite of what you want from anything that ships code.

The honest limitation

Evaluation does not make an agent correct. It makes an agent's errors visible and cheap before they reach production, which is a different and more achievable goal. You will still get things wrong. The point is to be wrong at the eval layer, where a failed check costs a retry, instead of at the merge layer, where it costs an incident.

This is the discipline underneath making an AI coding agent reliable on a platform like Loopsfinity, where agents propose plans and changes against a real codebase: every step that an agent produces is checked, cheaply where possible and with judgment where necessary, and a human still approves what matters before it ships. Not because the model is untrustworthy, but because "valid JSON" and "true" are not the same claim, and the distance between them is exactly where the expensive bugs live.