The demo worked. That tells you almost nothing.

A change goes in. Someone pulls up the query they have been staring at all week, the one the change was built to fix, and now the result is better. The room nods. Ship it. I have been in that room, on both sides of the demo, and the nod is the moment everyone in it agrees to trust a sample of size one that the presenter chose.

That is what a demo is. Not a lie, exactly. A demo is a true statement about a single input you hand-picked because it works. The query was selected after the change, by the person who wants the change to land. It carries no information about the rest. And at the point where the system actually does something to a real user, only one question matters: was it right at scale, on inputs you didn’t choose? That is a measurement question, and the most common failure I see is teams answering it with a better model when what they needed was a way to know.

Producing the action is not the same as knowing it was right

The shape of these systems is intent in, action out. A user types a few words, or clicks, or says nothing and just shows up, and somewhere in the middle you infer what they meant and then you do something about it. Most of the engineering attention goes to that middle: personalization, ranking, intent inference, the ontology that puts the person and the options in the same terms. That work is real, and all of it goes into producing the action. None of it can tell you whether the action was right.

Evaluation is the layer that can tell you. Monitoring already answers the operational questions: did the model run, did the latency hold. Evals answer the other one: did what you did match what they meant. That makes evaluation a peer of ranking and personalization and intent inference, not a QA step that happens after them. You would never call ranking a checkbox you tick before launch. Evals get treated that way constantly, and that’s the mistake.

Trust is a measurement problem before it is a model problem. You do not earn trust by making the model bigger. You earn it by being able to state, with evidence, that the action was right on inputs you did not curate. If you cannot measure that, every improvement is a demo: it worked on the inputs someone picked.

The metric was the eval, and the eval was wrong

Here is the standard version of the failure, the one whole teams live through without noticing.

A team builds a ranker and optimizes it for the click. The reasoning is clean. People click what they want. More clicks means more relevance. The metric goes up, and so does every surface number around it: engagement, click-through, session depth. The team has, by every measure it watches, made the system better.

But the ranker has learned what people tap, not what they came for. Those are different objectives that happen to agree most of the time, which is exactly what makes the gap so hard to see. It optimizes for the bait-shaped result and demotes the result that would have ended the session because the person got what they needed and left. The team is getting better at the wrong objective. Nothing warns them, because the numbers are all going up.

The lesson is not “pick a better metric,” though that helps. The metric was the eval: the click-through number is what the team used to decide whether a change helped. And a bad eval is worse than no eval at all, because it scores the regression as a win. That’s the proxy-metric trap: the evaluation itself is broken, so it keeps approving the change that is doing the damage.

A demo answers “can it”; a system answers “is it right”

Once you have been burned this way you start to hear the difference between two questions that sound identical.

A demo answers: can it do this? You point it at one input and it produces a good output and the answer is yes, it can. A system answers a harder question: is it right, at scale, on inputs I didn’t choose, and do I know when it isn’t? The last clause is the one everyone skips. It is not enough to be right on average. You have to know when you are wrong, because a regression you don’t catch ships like any other change.

The bridge between the two is a harness: a fixed set of inputs with known-right answers, run on every change. It reports two numbers: what got better and what got worse. The improvement number always gets built, because the person who made the fix checks the case they were fixing. The regression number rarely does. The query that used to work and now does not is not in the demo, and there is nothing else in the process that would catch it. The harness is what catches it.

Golden datasets are the boring asset that is the moat

The center of the harness is a golden dataset: a curated, versioned, argued-over set of examples where the right answer is settled by human judgment. Someone looked at this input and decided, on the record, what the correct action is, and that decision is now the standard the system is measured against.

A customer of ours at Velou runs a big marketplace for secondhand furniture, where every piece is one of a kind. The founder told us the machine identifies the wood better than appraisers can, and he was thoroughly impressed. I was too, for a different reason: that claim can be tested. Collect pieces where the wood is known for certain and you have a golden dataset. Against ours, the machine gets the wood right 99.8 percent of the time. Wood type is one attribute in that catalog’s ontology, and every attribute in it can be graded the same way. Most claims about these systems cannot be tested like that.

Everyone underinvests in this. I understand why. It is slow. It doesn’t ship. No demo has ever been given of a golden dataset. Building one means a few people arguing, example by example, about what the right answer should be, and writing the answers down. It looks like overhead. So it gets skipped, and changes go out on trust. The demo looked good, and that was the check.

The reason to build it anyway is that it outlives the model. You will swap the model five times: a new base, a new provider, a new fine-tune. Each swap is a leap of faith unless you have something stable to measure against. The golden dataset is what stays still. It turns every model swap from “trust us, it feels better” into a number that moved a known direction on known inputs. That is most of the moat, and the dataset only gets more valuable as the models underneath it churn.

Offline metrics lie in one specific way

You will run most of your evals offline, against the dataset, before anything touches a user. This is correct. Offline is fast and cheap, and you can run it on every change. But offline metrics lie, and they lie in one specific, knowable way: they measure the world as your dataset froze it, a photograph of weather that kept moving.

So an offline win can be an online loss, and the mechanism is always the same: the metric drifted from the intent somewhere between the freeze and the deploy. The distribution moved. It is the click-through trap in a new place: a metric that used to track what you actually wanted and no longer does.

Pick offline metrics that survive contact with production: ones whose relationship to the real outcome is stable, not ones that happen to be easy to compute. And treat every offline win as a hypothesis, not a result. The offline number says “this change probably helped.” The online number gets to say “this change helped.” Until the second one agrees with the first, you have a candidate. Most regressions ship exactly this way: an offline win reported as a fact.

A judge is just another model

When the right answer is too nuanced to score with a rule, the move now is to grade the model with a model: an LLM as judge. It works, and it is the only approach that scales. But a judge is just another model, and its wrong grades look exactly like its right ones.

The discipline is to treat the judge as a system that itself needs evaluating. It gets its own golden dataset: examples where humans have decided what the correct grade is. You measure the judge’s agreement rate with those humans before you trust a single score it produces. If a judge only agrees with your human graders seventy percent of the time, you can’t trust it: you’re scoring the system with something that’s wrong three times in ten. And agreement is not a one-time check. The judge drifts as the inputs drift, as the model under it changes, as the prompt picks up edits. So you re-anchor it against fresh human labels on a schedule.

Skip this and the number is still a guess, one that looks rigorous because a model produced it. That is the proxy-metric trap again, one level down, where it’s harder to catch.

The hardest eval is whether it should have acted at all

This is where most teams have not had to go yet.

When the system suggests, surfaces a result, offers a ranking, proposes an option, accuracy is enough. Was the suggestion good? Score it and move on. The user is the backstop; they take it or they don’t. But the moment the system acts instead of suggests, the eval surface changes shape entirely. Now the wrong action has a cost the user did not consent to. So you stop only measuring whether the output was good and you start measuring abstention: did it hold back when it should have. You measure calibration: did its confidence track its correctness. You measure what has no equivalent in a suggestion system: knowing when not to act.

The hardest eval is whether the system should have acted at all, which is a different question from whether the answer was good. It is hard because the system is just as confident when it is wrong. The only difference is whether it knew the boundary. Most teams have not built this eval, because most systems still suggest. The harness has to be there before the first action a system takes on its own (booking the family’s flights, sending the reply, moving the money).

The demo is not the system

Everyone ships the model. Everyone narrates the demo. The cherry-picked query, the nod, the numbers all going up. That part is easy and it is everywhere, and none of it answers the question at the point of action. Was it right on the inputs you didn’t pick.

That answer has to be earned. It takes a golden dataset, so there is a fixed standard to be right against; a regression number, so the case that got worse is counted; a judge checked against human graders; and offline metrics treated as hypotheses until production agrees. That list is the work, and none of it makes a good demo.

Every action these systems take starts from a guess about what someone meant. The eval is how you know the guess was right. The demo is not the system.

Read next → Every Life Decision Is a Ranking Problem