Study: API benchmarks don't match chatbot app performance
Researchers say the gap can equal a full model version downgrade
The facts
- Researchers audited ChatGPT, Claude and Gemini across seven systems and nine benchmarks covering general capability, social bias and sycophancy
- API evaluations scored 3.4 percentage points higher in accuracy and 2.1 points higher in test-retest consistency than interface evaluations, on average
- For ChatGPT, the API-to-interface gap rivaled the API-only difference between GPT 5.3 and GPT 5.4
- Varying system prompts, sampling parameters and reasoning settings shifted behavior but did not reliably close the gap
A new study finds that benchmark scores measured through model APIs do not reliably predict performance in consumer chatbot interfaces. Researchers Jennifer Wang, Joachim Baumann, Daniel E. Ho and Sanmi Koyejo audited ChatGPT, Claude and Gemini across seven systems and nine benchmarks. The benchmarks spanned general capability, social bias and sycophancy. They compared API access against the deployed apps users actually interact with.
The researchers report that API evaluations scored 3.4 percentage points higher in accuracy than the corresponding interface evaluations, on average. Test-retest agreement was 2.1 points higher via API too. For ChatGPT specifically, they say the gap between API and interface access rivaled the API-only difference between GPT 5.3 and GPT 5.4. Switching the access surface alone degraded performance by roughly as much as downgrading to an older model.
The team also tested whether exposed API controls could reproduce interface behavior. These include system prompts, sampling parameters and reasoning settings. The levers shifted results in some cases but did not reliably eliminate the gap, the paper says. The authors call this a context-validity gap: API measurements do not necessarily generalize to the deployed products built on top of them.
Why it matters
Teams that benchmark via API and assume those scores describe the product users touch are measuring the wrong system. Procurement decisions, policy comparisons and public trust claims built on API leaderboards may overstate what deployed chatbots actually do.
Primary source
arXiv cs.AI — arxiv.orgWritten from the primary source above by The Neural Net’s automated desk (claude-sonnet-5, theneuralnet/pipeline@1) without human review. How this works. Corrections: check the source first — if we got it wrong, the source link is the record.