Why last year's benchmark figure cannot be compared with this year's

The model releases of the past year got the attention. Something else happened alongside them, with considerably less: almost every organisation that measures models seriously published a correction to its own instrument. One withdrew its measure entirely.

That is the news that matters to anyone comparing two vendors, because a figure from a measurement whose ruler has moved is not comparable with a figure from the same measurement last year.

METR changed its method, and the numbers with it

METR measures how long a task a model can complete. The measure is called time horizon, and it is defined as the task length, counted in how long a skilled human needs, at which the model succeeds half the time.

In January 2026 METR published version 1.1 of the method and expanded the task suite. The same models, unchanged, got new values. GPT-4 from March 2023 fell by 35 per cent. GPT-4 from November 2023 fell by 57 per cent. GPT-5 rose by 55 per cent. o3 rose by 29 per cent.

Nobody retrained anything. The measure changed.

That makes the single most important practical consequence easy to state: a figure from the old method version and a figure from the new one must never appear in the same comparison. The figures in the chart below all come from method version 1.1.

The curve rises, and that is a real development. But it should be read with three caveats that METR themselves write out.

The measure is not how long a model can work on its own. It is, in their words, the amount of serial human labour it can replace with a fifty per cent success rate. The error bars have historically been about a factor of two in each direction and get worse for the most recent models as the suite saturates. And measurements above sixteen hours are, according to METR themselves, unreliable with the current task suite.

The comparison between 50 and 80 per cent cannot be made

A tempting reading is that the horizon at fifty per cent success rises steeply while the horizon at eighty per cent stands still, meaning that models are getting better at succeeding sometimes but not at succeeding consistently. That reading is common and it does not survive the source. METR's original work from March 2025, the study that reports horizons at both fifty and eighty per cent success, puts the doubling time at eighty per cent at 213 days and at fifty per cent at 212, the same within the margin of error. And in a note on the measure's limitations they write that the horizons at twenty, fifty and eighty per cent are not independent estimates of one another, because they come from the same two-parameter curve. Improved performance on short tasks can, in that model, lower the eighty per cent horizon.

Setting the two horizons against each other compares a measure with itself.

A chart of the two would therefore have illustrated a property of the curve fitting rather than a property of the models.

The difference in level is well established, and it is measured within a single method version. That same study puts Claude Sonnet 3.7 at around 15 minutes at eighty per cent success, against 59 minutes at fifty. Demanding consistency cuts the task to roughly a quarter, and that is the figure that matters to anyone putting something into production. The 60 minutes shown for the same model in the chart above is a fifty per cent figure from version 1.1 and does not belong beside the fifteen. The same observation measured on agents, meaning the difference between succeeding once and succeeding four times in a row, is covered in when an agent has earned its place.

OpenAI stopped using SWE-bench Verified

In February 2026 OpenAI published Why we no longer evaluate SWE-bench Verified. The reasoning is unusually direct: improvements on the measure no longer reflect meaningful improvements in real software development, but increasingly reflect how much the model was exposed to the benchmark at training time.

That is how it is usually quoted, and it is half the reason. The other half has numbers, and they are worse.

At least 59.4 per cent of the audited tasks therefore have tests that reject solutions which work. A model that saw the problem during training passes anyway, because it knows what the test demands although the task never said so. The benchmark rewards memory of this particular set, and the two problems are in practice the same problem.

Both faults favour the model that has already seen the task, which is why they compound.

OpenAI has not stopped measuring coding ability. They recommend a successor. But between the two measures there is a gap nobody can convert across.

GPQA is saturated, FrontierMath Tier 4 is not

Epoch AI wrote in May 2026 about what they call the classic reasoning benchmarks. Of GPQA, a set of graduate-level science questions, they note that it had remarkable staying power but is by now clearly saturated.

At the same time there is real progress where the tasks are still hard. On FrontierMath Tier 4, the most difficult level, Epoch recorded a new high in January 2026: 31 per cent against a previous best of 19. That is a large move on a set of problems few models have solved at all.

The record does carry a caveat that Epoch write out themselves. The run was done manually in the ChatGPT web interface, after their own scaffold timed out against the API. The two figures therefore do not come from the same setup. That does not make 31 per cent untrue, but it makes the comparison with 19 weaker than it looks.

Hallucination and long inputs

Vectara's hallucination leaderboard, updated in May 2026, measures one bounded thing: how often a model invents when asked to summarise a given document and explicitly instructed not to use its own knowledge. The lowest measured rate is 1.8 per cent.

Two things are worth noting. The figure belongs to a model of 32 billion parameters, not to any of the large labs' flagships. And the measure can be improved by answering more briefly or by declining more often, which is why the table reports answer rate and summary length alongside the rate itself.

Chroma's report on context rot tested 18 models and found that they do not use their context uniformly, but that performance grows increasingly unreliable as input length grows, even on simple tasks. The report is from July 2025 and tested the generation current at the time, so it says something about that generation and not automatically about today. That nobody has published a comparable counter-measurement since is itself an observation.

Three questions for a vendor showing a benchmark result

None of the measurements above is bad. On the contrary, they are good precisely because they correct themselves in public, and anyone who does that deserves to be quoted carefully.

The consequence for an investment decision is concrete all the same. When a vendor shows a benchmark result, three questions decide whether the number means anything:

1. Which version of the measure? METR's method change moved unchanged models between minus 57 and plus 55 per cent. A figure without a version cannot be compared with anything.
2. Under what conditions was it run? Manually in a web interface is not the same as through an API in a standardised setup, and Epoch's own record shows the difference can arise for purely practical reasons.
3. How wide is the interval? A factor of two in each direction is common in this literature. A figure without an uncertainty interval is a headline, not a measurement.

The fourth question is harder and is rarely asked: does this measure something that resembles what we are going to do? SWE-bench Verified measured something that looked like software work and that in more than half of cases penalised working code. It took four years, and an audit by the organisation leaning on it hardest, for anyone to write that down.