Who decides when your AI agent has to be rebuilt
OpenAI promises at least six months' notice before a generally available model is retired, Anthropic at least sixty days for publicly released ones, and Google no fixed period at all. An AI agent in production inherits that calendar, and the calendar sets the cost.
Key insights
- Providers publish when models are shut down. OpenAI promises at least six months' notice for generally available models, Anthropic at least sixty days for publicly released ones, and Google states no period at all.
- Researchers at Carnegie Mellon and the Hong Kong University of Science and Technology compared ten pairs of model updates and found that 58.8 per cent of prompt and model combinations lost accuracy.
- In 87.9 per cent of the cases where overall accuracy improved across an update, at least one previously correct answer still regressed. A rising average is therefore not proof.
- 63.8 per cent of the regressions happened when the model was at its most confident. Most of the errors therefore arrive without the model signalling any uncertainty at all.
- An agent resting on the shape of the answer breaks visibly at a model change. One resting on the model's judgement merely gets worse, and that is only noticed by someone measuring.
An AI agent that has been built, tested and put into production is not finished. The model underneath it has an end date, and someone else sets that date. Providers publish their deprecations openly, but they promise very different amounts of notice, and the spread runs from two weeks to six months.
The question is not whether the agent will be rebuilt. It is how often, how much, and what that costs.
The shutdown dates are published, and the notice periods differ sharply
OpenAI writes its policy out in its deprecations documentation: at least six months' notice before a generally available model is retired, at least three months for specialised variants, and for preview models as little as two weeks. The promise carries a caveat of its own: it holds unless safety or compliance concerns require a faster timeline, in which case OpenAI undertakes to give as much notice as it reasonably can. On 11 June 2026 GPT-5 and the o3 models were deprecated, with shutdown on 11 December the same year.
Anthropic promises at least sixty days for publicly released models. Claude Opus 4.1 was deprecated on 5 June 2026 and retired on 5 August, two months later. The dates on that page apply only to Anthropic's own platforms, meaning the Claude API, Claude Platform on AWS and Microsoft Foundry. Partner-operated platforms, Amazon Bedrock and Google Cloud, set their own retirement schedules, so a model's lifecycle status and dates can differ. Which route you bought through therefore shapes your calendar.
Google states no period at all. The documentation says the shutdown dates in its table indicate the earliest possible dates on which a model might be retired, and that the exact date is communicated to users in advance. How far in advance is not stated.
| Provider | Notice promised before shutdown |
|---|---|
| OpenAI | At least six months for generally available models, at least three for specialised variants, as little as two weeks for preview models. Holds unless safety or compliance requires a faster timeline |
| Anthropic | At least sixty days for publicly released models |
| No stated period. The shutdown dates in the table are the earliest possible ones, and the exact date is communicated in advance |
Three providers, three different promises. Choosing a model means choosing a calendar as well.
The published dates are the visible half. The other half is change that arrives under an unchanged model name, and that is what the two measurements below are about.
An update that raises the average can still turn a right answer wrong
Wanqin Ma at the Hong Kong University of Science and Technology, with Chenyang Yang and Christian Kästner at Carnegie Mellon, measured what happens to a finished prompt when the model underneath it is updated. (Why) Is My Prompt Getting Worse? was posted in November 2023 and presented at the IEEE and ACM conference CAIN 2024. The population is five models from OpenAI's GPT-3.5 family, released between March 2022 and September 2023, tested on two datasets for detecting toxic content: 1,000 comments from Civil Comments and 174 discussions from GitHub. Ten pairs of model updates were compared.
The models are old and the figures should be read as a measurement from that time. What makes the study worth reporting is the shape of the result rather than the figures themselves.
Of the prompt and model combinations, 58.8 per cent lost accuracy across an update, and of those, 70.2 per cent lost more than five percentage points. Counted on individual answers, 10.9 per cent went from right to wrong.
The figure that carries the most weight is a different one. In 87.9 per cent of the cases where overall accuracy improved across an update, at least one previously correct answer still regressed.
An average that rises can hide the fact that the one answer your process was built on is now wrong.
And 63.8 per cent of the regressions happened when the model was at its most confident, measured as entropy zero. The authors note that the models are on average more uncertain about the answers that flip. Most of the regressions arrive without that signal all the same, which is why a model update has to be tested against known correct answers rather than against a sense that it looks fine.
What breaks visibly in an AI agent is the format, not the judgement
A more recent measurement points to where the fractures sit. At NTNU in Trondheim, Mohd Sameen Chishti, Damilare Peter Oyinloye and Jingyue Li tested seven Claude models against 25 prompts in three risk categories: authentication functions, data validation and structured output. Each prompt was run three to five times. Test Before You Deploy was posted on 30 April 2026.
The material is small and the testing was done through a web interface rather than against the programming interface, which has to be stated before anything is built on the figures. The direction is worth noting all the same. Tasks requiring structured output drifted most, while functions for database queries and authentication remained stable. Two models changed which exception type they threw, and a third began returning errors prematurely. The authors also note that a database function which passed every test one day failed a test they call safe encoding the next, under the same model name. They read that as a silent infrastructure change, while writing at the same time that their version tracking remained approximate and relied on visible model names and timestamps.
That points to a distinction worth having when an agent is being budgeted, and it concerns how the failure shows itself. An agent resting on the model's judgement loses accuracy instead of stopping. That it loses any at all is what the Carnegie Mellon material shows: toxicity detection is a judgement task, and that is where the 58.8 per cent was measured. An agent resting on the shape of the answer, on it arriving in a particular structure with a particular field name or a particular exception, stops working outright instead. The first is only visible in a measurement. The second is visible immediately, which makes it cheaper to detect and more expensive to design away.
It is the same dividing line that settles when a rule is enough and when it is not, which is the question in When is an AI agent ready for production.
The objection that carries weight
There is a counterweight. The measurement carrying the most weight in this text was made on models from the GPT-3.5 family between 2022 and 2023, a generation of which four of the five tested models have already been shut down, while the fifth, gpt-3.5-turbo-instruct, shuts down on 28 September 2026. That the shape of the phenomenon is unchanged in today's models is an assumption and not a result.
Nor is staying on one model version an option. The shutdown dates sit in the providers' own tables, so declining to update is not declining the work. Waiting also means making a larger jump at once, because several updates then reach the agent together rather than one at a time.
An evaluation suite is built on your own cases, and those do not change when the model does.
And the most recent material has to be handled carefully. The NTNU study is exploratory, rests on 25 prompts and was run through a web interface. It is enough to point out where to look. It is not enough to calculate how often this happens.
The two questions that decide how often you do this again
Two organisations running the same agent on the same model can face very different work, and the difference does not lie in the model.
Does the agent rest on the model's judgement or on the shape of the answer? Does it read a document and assess what it is about, or does it require the answer to come back with fixed fields in a fixed order? Format-bound agents break visibly at a change and need rebuilding. Judgement-bound ones do not break, they get worse, and that is noticed only by someone measuring. Both cost, but they cost different things. The answer can be found without waiting for the next update: swap the model in a test environment and see both what stops working and what merely gets slightly worse.
Can someone at your organisation run the evaluation without whoever built the agent? That is the question deciding whether the deprecation dates are a planning item or a crisis. If the suite exists, is documented and can be run internally, a model change is bounded work with a known outcome. If it is missing, every change is an assumption, and then the handover is the work rather than the model.
Common questions
The running cost is made up of three things: re-running the evaluations when the model underneath the agent is updated or replaced, fixing what those evaluations find, and keeping the integrations alive. How large it becomes depends on how tightly the agent is bound to one model's way of answering. An agent that needs a particular field name or a particular exception in order to work requires effort at every update touching that shape.
The provider sets the pace, not you. OpenAI deprecated GPT-5 and the o3 models on 11 June 2026 with shutdown on 11 December the same year. Anthropic deprecated Claude Opus 4.1 on 5 June 2026 and retired it on 5 August. An organisation running an agent on a named model version should therefore check the provider's deprecation table before the budget is set, because the cadence sits there rather than in its own plan.
Agent drift is the gap between how an agent behaved when it was built and how it behaves now, without anyone changing its code. It has two sources. The model underneath is updated or replaced, which shifts both judgement and output shape, and the integrations the agent reaches into move on their own schedule. Neither announces itself, which is why drift is found by measurement rather than by observation.
It differs sharply. OpenAI promises at least six months for generally available models, at least three months for specialised variants and as little as two weeks for preview models, unless safety or compliance concerns require a faster timeline, in which case it undertakes to give as much notice as it reasonably can. Anthropic promises at least sixty days for publicly released models, and the dates on that page apply only to its own platforms, with Amazon Bedrock and Google Cloud setting their own schedules. Google states no period at all and writes that the shutdown dates in its table are the earliest possible ones.
Yes, and it can do so even when the update is an improvement overall. Researchers at Carnegie Mellon and the Hong Kong University of Science and Technology compared ten pairs of model updates and found that 58.8 per cent of prompt and model combinations lost accuracy, and that in 87.9 per cent of the cases where overall accuracy rose across an update there was still at least one answer that went from right to wrong. The measurement was made on models from the GPT-3.5 family and the figures should be read as a measurement from that time.
Agent evals are a set of your own cases with known correct answers, run against the agent before and after every change. They are needed because a model update cannot otherwise be judged: the difference does not show in a spot check and it rarely shows in the provider's own figures. The suite also keeps its value when the model changes, because it is built on your cases rather than on the model.
Only until the shutdown date. Staying on one model version defers the work without removing it, and it moves the moment to a day the provider has chosen. Waiting also means a larger jump at once, because several updates then reach the agent together rather than one at a time.
Three things, all of which can be written into a contract. That the evaluation suite is built on your own cases and handed over with the agent. That the model version the agent was built against is written down, along with what has to be done at a change. And that someone at your organisation can run the evaluation without the vendor in the room, because an agent only its builder can keep honest is a dependency rather than an asset.
If this lands on your desk, we should talk.
Ampliro Insights
New analysis, roughly weekly.
We write when the rules change and when something turns out to work in practice. One piece at a time, no sequences, and you can leave from any issue.
We store your address to send Ampliro Insights, and for nothing else. More in the privacy policy.