When an AI agent is ready for production, and when a rule is enough
A benchmark saying an AI agent can do the task answers the wrong question. The question that decides the cost is how often it does, and there is a measure for that.
Key insights
- The usual measure, pass^1, answers whether an agent solved the task. Pass^k answers whether it solved the same task several times in a row, and the second measure is what decides the cost.
- The EU AI Act requires accuracy and robustness of high-risk systems in Article 15, against risks the legislator itself calls unpredictable model behaviour. Pass^k puts a number on that property.
- The scaffold around the model affects outcome and cost at least as much as the choice of model. In independent GAIA runs the priciest configuration cost over three times the best and scored lower.
- A correctly written rule does the same thing every time. An agent should therefore have to earn its place, and mainly does so when inputs vary in form and not only in content.
An AI agent that solves the task in the demonstration has shown one thing: that the task can be solved. It has not shown that it solves the task next time, or the time after that. The difference between those two statements decides what the automation costs, and it rarely appears in the material on the table when the decision is made.
The measure that separates a result from a sample
The researchers behind the τ-bench benchmark proposed a measure in 2024 that they called pass^k. The usual measure, pass^1, answers whether the agent solved the task. Pass^k answers whether it solved the same task k times in a row. The difference sounds academic and is not. It is the difference between a demonstration and a process.
Consistency is not a trait an agent happens to have but something built into the scaffold around it, and it can be specified.
The study tested the strongest function calling model of the day, gpt-4o. In the retail domain it solved around 61 per cent of the tasks on a single run. When the same task had to succeed eight times in a row, the share fell to around 25 per cent. The model is old and the numbers should be read as exactly that, a measurement from 2024. What has not aged is the shape of the result. The share falls to below half when the same task has to succeed eight times running, and that follows from how agents work rather than from which model happened to be tested.
Same model and same domain in both bars. The study gives the figures as approximate.
Source: τ-bench, arXiv 2406.12045, measured on gpt-4o in τ-retail, 2024
Bjarnason, Silva and Monperrus find the same drop in coding agents
τ-bench is one study, one year and one domain. A second research group has since measured the same thing on a different kind of agent. In the preprint On Randomness in Agentic Evals, posted to arXiv on 6 February 2026 and with no known review, Bjarnason, Silva and Monperrus collected 60,000 agentic trajectories: 500 tasks from SWE-Bench-Verified, ten independent runs per task and configuration, three models and two scaffolds. The paper does not state when the measurements were taken. What can be established is that version one went up on 6 February 2026 and that one of the models is a December 2025 snapshot.
The two measures look alike and point in opposite directions. Pass@k means at least one of k attempts succeeds, an optimistic measure. Pass^k means all k attempts succeed, a pessimistic one. The τ-bench figures above are pass^k, which is what makes them comparable with the new ones.
For DeepSWE-preview on the r2e-gym benchmark, 34.4 per cent of tasks were solved on the first attempt. With five attempts the configuration reached 52.9 per cent, 18.5 percentage points more where a single success is enough. Only 15.5 per cent of tasks were solved on all five attempts, less than half the first-attempt rate. The same group also measured the spread between runs on SWE-Bench-Verified: a single-run pass@1 estimate varies by 2.2 to 6.0 percentage points depending on which run is picked, with standard deviations above 1.5 percentage points even at temperature 0.
Two research groups, two domains, different models and two years apart produce the same shape. That the spread survives at temperature 0 says it does not come from random sampling alone, but also from how the agent and the scaffold around it work. The drop between one attempt and several in a row is therefore something to engineer against, and the requirement follows from it: measure the same case across repeated runs, and buy the configuration that holds.
Knowing this changes the question being asked. Not whether the agent can handle your process, but how often it handles it and what happens on the occasions it does not. The second question can be costed. The first cannot.
What the rules already ask for
The EU AI Act sets requirements on accuracy, robustness and cybersecurity for systems classified as high risk. Those requirements sit in Article 15. When the union legislator amended parts of the regulation in July 2026, it also set out which risks the requirements exist for: bias, unpredictable model behaviour, poor robustness or accuracy, vulnerability to third party attack, and lack of transparency in the AI system.
Unpredictable model behaviour is the legislator's own phrase. Pass^k is one of very few measures that puts a number on that exact property. A system that succeeds two times out of three is not unreliable in some vague sense. It is unreliable in a way that can be written into technical documentation and tracked over time.
Which systems are covered, and from when, changed over the summer. The new deadlines are covered in our review of the high-risk requirements.
Every rerun has a price
If an agent has to be rerun in order to succeed, the rerun is not free. It costs tokens, it costs waiting time, and where somebody has to check the output it costs working time.
The research group behind the Holistic Agent Leaderboard at Princeton runs agent configurations itself rather than reproducing suppliers' own figures, and reports both accuracy and cost. On the GAIA benchmark their generalist agent with Claude Sonnet 4.5 reached just under 75 per cent accuracy at a reported run cost of around 178 dollars. A configuration built on Claude Opus 4.1 in high mode cost around 562 dollars and reached just over 68 per cent. Three times the cost, lower accuracy.
The cost covers the whole benchmark run. Accuracy is stated in the label.
Source: Holistic Agent Leaderboard, Princeton, runs from August and September 2025
The figures are from 2025 and will go out of date. The relationship between them is what is worth carrying away. The scaffold around the model affects both outcome and cost at least as much as the choice of model does, and that is the part suppliers rarely describe. The same pattern as in AI costs: from licence to consumption, one level further in.
When a rule is enough
A rule has one property no language model has. It does the same thing every time. Expressed in this article's measure, a correctly written rule holds just as reliably however many times in a row it runs.
That does not make rules better. It makes them cheaper to trust, and it means an agent should have to earn its place. Three questions usually settle it.
- Can the task be written as a decision tree? If somebody in the business can put the rules on a single page, it is a rule and not a judgement, and then it should be built as a rule.
- Do the inputs vary in form or only in content? Free text, images and documents that look different every time are what the agent is good at. Structured fields are handled more cheaply by a rule.
- What does an error cost? An error caught immediately and corrected cheaply tolerates low consistency. An error that carries on into a customer contract or an administrative decision does not, and there the requirement is pass^k rather than pass^1.
What to ask of a supplier
Three things, and all three can be written into a contract.
- Ask for consistency, not accuracy. Ask how often the same task is solved several times in a row, and on your own kind of case rather than on a public benchmark.
- Ask for cost per resolved case, not per run. A price per run hides the reruns, and it is the reruns that separate a budget from a surprise.
- Ask what sits around the model. Tools, memory, error handling and the number of permitted retries all shape the outcome, and two suppliers using the same model can deliver entirely different consistency.
The yardstick deserves scrutiny in its turn
There is reason to be careful with the benchmarks themselves as well. A research group at Berkeley has published a systematic audit of agent benchmarks showing that several of them can be scored highly by exploiting how they are constructed, without the tasks actually being solved. The authors conclude that benchmarks need to be secure by design, in the way systems need to be.
That is uncomfortable for anyone who cites benchmarks. The conclusion is not that the numbers are worthless, but that a result from a public benchmark never replaces a measurement on your own cases. That measurement is a contained exercise run on your own cases, and it is the only one that answers the question you actually have.
Common questions
Pass^k measures how often an AI agent solves the same task several times in a row. The measure was proposed in 2024 by the researchers behind the τ-bench benchmark. The usual measure, pass^1, only says the agent solved the task once. Pass^8 says it solved it eight times running. For an organisation about to let an agent handle a recurring step, the second number is the relevant one, because a process runs many times rather than once.
No. A successful demonstration shows the task can be solved, not that it is solved every time. In the original τ-bench study the strongest function calling model of the day solved around 61 per cent of the tasks on a single run in the retail domain, but only around 25 per cent when the same task had to succeed eight times in a row. Models have improved since, but the gap between one run and several in a row is a property of how agents work and does not close on its own.
A rerun costs tokens, waiting time and, in many cases, the working time of whoever checks the output. That cost is invisible in a price per run, which is the reason to ask for a price per resolved case instead. In independent runs of the GAIA benchmark published by the Holistic Agent Leaderboard at Princeton, the priciest of three configurations cost around 562 dollars and scored lower than a configuration costing around 178 dollars.
For systems classified as high risk, the EU AI Act sets requirements on accuracy, robustness and cybersecurity in Article 15. When the union legislator amended parts of the regulation in July 2026, it set out the risks those requirements exist for, and unpredictable model behaviour along with poor robustness or accuracy are named explicitly. The regulation prescribes no particular metric, but consistency over time is a property the rules care about.
A rule is better when the task can be written down as a decision tree and the inputs are structured. A rule does the same thing every time, which in this article's measure means it holds however many times in a row it runs. An AI agent adds value mainly when inputs vary in form, meaning free text, images or documents that look different from one instance to the next, and not only in content.
Three questions go a long way, and all three can be written into a contract. Ask how often the same task is solved several times in a row on your own kind of case, rather than asking for accuracy on a public benchmark. Ask for cost per resolved case rather than per run, because a price per run hides the reruns. Ask what sits around the model in terms of tools, memory, error handling and permitted retries, because two suppliers using the same model can deliver very different consistency.
With reservations. A research group at Berkeley has published a systematic audit showing that several leading agent benchmarks can be scored highly by exploiting how they are built, without the tasks actually being solved. The conclusion is not that benchmarks are worthless, but that a number from a public benchmark never replaces a measurement on your own cases.
By measuring how often it solves your own typical cases several times in a row, and by setting that requirement against what an error costs. An error caught immediately and corrected cheaply tolerates lower consistency. An error that carries on into a customer contract or an administrative decision does not. The measurement runs on your own cases and answers a question no public benchmark can.
If this lands on your desk, we should talk.
Ampliro Insights
New analysis, roughly weekly.
We write when the rules change and when something turns out to work in practice. One piece at a time, no sequences, and you can leave from any issue.
We store your address to send Ampliro Insights, and for nothing else. More in the privacy policy.