The Leaderboard Was Never the Benchmark

I still read the leaderboards. When a new model drops I check where it lands, same as everyone. And the rankings are not worthless: they measure something real, human preference on a fixed set of prompts, graded the same way for every model. But I have watched the ranking disagree with my own repo enough times to stop treating the crown as an answer. The model that topped the board last month is not the one I kept open all week.

The mistake was never in the tests. It is in what we ask them to be. A benchmark is an exam. And the moment a score starts deciding which model gets used, that exam becomes a target, and targets get gamed. What you actually needed was a test the model could not study for. That was never going to be the leaderboard.

An exam is only honest until someone studies for it

Start with what a benchmark actually is. IBM breaks it into three steps: set up a dataset, run every model on identical inputs, score the results onto a leaderboard. Two public designs dominate. Standardized suites run models through a fixed battery of graded tests, and Artificial Analysis's methodology is the rigorous version, formal metrics and explicit formulas. Crowd-vote arenas work differently: two anonymous models answer your prompt, you vote, and the votes aggregate into a chess-style Elo rating, which is how LMArena ranks. Both are legitimate designs. Both measure the same kind of thing: performance on set questions, under set conditions, on test day.

They are accurate at that. But accurate-on-test-day is not the same as useful-on-your-work, and the gap has structure. There is saturation: when the top models all cluster near the ceiling, the test can no longer tell a great model from a merely good one, and BIG-Bench saturated less than a year after release. There is contamination, "the unintended overlap between training and test data," where the model may have trained on the answer key, found to be widespread in one analysis of 31 models on math reasoning. IBM says the limit out loud: benchmarks "can't predict how well a model will operate in the real world."

A target, and targets get gamed

Goodhart's Law is the whole story in one line: when a measure becomes a target, it ceases to be a good measure. Every public ranking inherits it. The moment labs optimize for a score, the score drifts away from the capability it was supposed to proxy.

And this has been measured, not just theorized. A 2025 study, The Leaderboard Illusion, found that access to arena data was worth a relative performance gain of up to 112% on the arena's own distribution: climbing the board by learning the test, not by getting better. The same paper counted 27 private model variants one provider tested before a major release, with only the best score published. The clean illustration was the Llama-4 episode in April 2025, when the version that topped the leaderboard was not the model shipped to the public. For a fair account, pushback from arena defenders included, Simon Willison's breakdown is the one to read. His aside is the tell: crowd votes reward "bulleted lists and answers of a very specific length." Not better answers. Longer, tidier ones.

So to be fair to them: leaderboards are not worthless. They measure preference on invented prompts, which is a real thing to measure. It is just also why your own experience, on your actual repo, so often disagrees with the ranking.

The one test you can't study for is the one that didn't exist yet

Every exam I have described is administered. It is given before deployment, in clean conditions, on fixed inputs the lab controls. Real work is the opposite of all three. It is observed after deployment, on messy tasks a person actually needed done. And that difference hides the property that matters most: real work is the ultimate held-out set. It did not exist until the user created it, so no model could have trained on it. You can contaminate a benchmark. You cannot contaminate tomorrow's ticket.

This is not a fringe position. Serious evaluation is already straining toward realism. The first of Cameron Wolfe's design principles for a useful benchmark is "realistic," and he points to CursorBench, which sources its data from real coding-agent sessions. The frontier of eval is spending real effort trying to become real work. Real work was sitting there the whole time.

Said as a single contrast: crowd-vote leaderboards measure preference on invented prompts, standardized suites measure performance on fixed tests, and the work measures whether the job got done.

The signals that survive contact with real work

Once you observe work instead of administering a test, you get different signals. Five of them, and each answers one blunt question: did the model actually do the job?

Completion. Did the task reach a finish, or stall out and get abandoned? That is an outcome, not a proxy. An exam scores the answer; the work scores whether the job got done.

Switch-away. Did the person quietly swap models mid-task? This is the one I trust most. A vote in a sandbox is free. A switch costs you time, which makes it a revealed preference under real stakes. It is the honest vote.

Tool reliability. Did it call its tools and integrations correctly, run after run? A model can write beautiful prose and still botch every tool call.

Infra recovery. Did it survive rate limits, timeouts, and transient errors and still finish? Lab suites run in clean rooms. Your Tuesday does not.

Blind quality. How good is the output when the grader cannot see which model produced it? That is the only version that controls for the brand halo and the self-preference bias that open grading quietly smuggles in.

The switch-away signal has backing beyond my own instinct. In April 2025, granting the arena's overfitting problem, Andrej Karpathy proposed usage-based rankings, the kind OpenRouter publishes, as a candidate top-tier eval precisely because they are "very difficult to game": a user switching models is "directly voting for some combo of capability+cost." That is the vote that counts. Not a thumbs-up in a sandbox, but a person moving their real work onto the model that does it.

The honest catch: this measure can be gamed too

Here is where I have to turn the argument on myself, because the alternative is to do the exact thing the leaderboards do, project confidence right up until the thing gets gamed. Goodhart does not spare me. A real-work signal is still a measure, and any measure can mislead. There is sample skew, since some models simply get more usage. There is task-mix, since my work is not your work. And the moment any of these five becomes a public number to optimize toward, it comes under the same pressure that hollowed out the leaderboard.

So the honest version ships with guardrails, not a trophy. Separate interactive work from automation and relays, because a cron job that runs for days never emits a "done" the way a person at a keyboard does, and counting it would corrupt the completion signal. Grade blind. Report direction, not a decimal. Measure in aggregate and anonymized, patterns across many people, no individual's raw prompts read or published. And do not publish a per-model target for anyone to optimize against, at least not yet. The tell that a real-work eval is honest is that it is still hedging in the exact spot where a leaderboard would already be bragging.

Judge the model on the work

There is no universal best model, and there was never going to be. The one that drafts your email is wrong for your debugging. The one that wins the arena may lose on your repo. The only question that survives all of it is the unglamorous one: for the work you actually needed done, which model finished it, and which one did you not switch away from.

Which is the same place I keep landing. Once anyone can generate anything, the output stops being the scarce thing. A leaderboard score is an output: cheap, plentiful, and now demonstrably gameable. Whether the work got done is an outcome, and the outcome is the only part that ever compounded.

The leaderboard was never the benchmark. The work you needed done always was.