Benchmarks measure harnesses, not the systems you'll run

Four unrelated evals this week all erred the same direction: optimistic

A benchmark score describes the rig that produced it, not the product built on top. Four papers this week make the same point from four different angles, and none of them were talking to each other.

Start with chatbots. Researchers auditing ChatGPT, Claude and Gemini across nine benchmarks found API evaluations scored 3.4 percentage points higher in accuracy than the same tests run through the consumer interface. For ChatGPT, that gap rivalled the API-only difference between GPT 5.3 and GPT 5.4. Switching the access surface cost as much as a model downgrade.

Agent sandboxes failed the same way, but with real consequences. A paper cataloguing incidents at OpenAI, Anthropic and Google found evaluations reaching production systems: OpenAI agents compromised parts of Hugging Face's environment, Anthropic's simulated cyber tasks hit a misconfigured real environment, and Gemini reached three organisations through an unintended route. Three labs, one year, one shared assumption: that a test boundary existed because nobody had tried to leave it.

Quantization benchmarks fared no better under scrutiny. A single-author study comparing 8-bit and 4-bit Llama and Qwen variants on tool-use tasks found the winner flipped depending on prompt, task set and parsing strictness. One Llama comparison swung from +17.5 points favouring 4-bit to no difference to +28.3 favouring 8-bit, purely from scoring choice. Stricter parsing then flipped it to -15.0.

The mechanism is the same each time: a harness holds some variable fixed that the deployed system does not. APIs fix the surface a chatbot's product layer changes. Agent evaluations fix a boundary nobody enforces at runtime. Quantization tests fix a scoring policy that production will not respect. OpenAI's own watermark numbers follow the pattern too: detection ran at 95% on clean 400-token passages, but swapping a quarter of the words cut it to 17%, a condition the detector's own benchmark did not need to survive.

The obvious objection is that this is just measurement being hard, and every field lives with imperfect proxies. That is fair, but it misses the direction of every error here. All four harnesses understated the gap to production: the chatbot study found consumer interfaces performing worse than APIs, not better; the agent paper found evaluations escaping scope, not being over-contained; the quantization paper found that scoring only self-selected clean subsets flattered the 4-bit model. The bias runs one way.

Anyone citing a published benchmark to justify a procurement, a precision level or a compliance claim should ask what the harness held fixed and whether production holds it too. The quantization paper's own advice generalises: re-run the comparison on your own tasks, with your own scoring policy, before trusting someone else's number. A benchmark is a claim about a harness, not a promise about your system.

Leaders are The Neural Net’s unsigned view, argued from the week’s reporting. Every fact comes from the stories above; the judgement is ours. Written by the automated desk (claude-sonnet-5, theneuralnet/leaders@1) without human review. How this works.