Tech Evolution #8 — on the gap between the number the industry publishes and the number your business actually runs on.
The most quoted number in agentic AI is a 50% number, and almost nobody says the 50% part out loud.
Where we actually are
METR's task-completion time horizon is the cleanest capability metric the field has produced. Its definition is precise, and the precision is the point: it is "the task duration (measured by human expert completion time) at which an AI agent is predicted to succeed with a given level of reliability." Two levels get published — 50% and 80%.
What that means in practice is easier to see in METR's own worked example. A GPT-5 agent measured at a time horizon of roughly 2 hours 17 minutes does not succeed on 50% of every 2-hour task. On tasks a human expert takes 90 minutes to 3 hours to do, it "succeeds 100% of the time for around one-third of the tasks, fails 100% of the time for around one-third of the tasks, and sometimes succeeds and sometimes fails on the remaining third."
That is three different populations wearing one average. A third you can automate, a third you cannot, and a third that will look automated right up until the day it isn't. The headline number tells you nothing about which bucket the task in front of you belongs to.
The shape of the decay is now reasonably well described. Toby Ord showed that agent success rates on METR's suite fit an extremely simple model: a constant rate of failing during each minute a human would take to do the task. Success declines exponentially with task length, and each agent can be characterised by its own half-life. His reading of why is the useful part — long tasks "involve increasingly large sets of subtasks where failing any one fails the task."
A March 2026 preprint pushed on this directly. Khanal, Tao and Zhou ran 10 models across 23,392 episodes on a 396-task benchmark, splitting duration into four buckets and three domains, and their framing is the one I think matters: existing benchmarks measure capability — whether a model succeeds on a single attempt — while production deployments require reliability — consistent success across repeated attempts. They report that the two diverge systematically as tasks get longer, with "multi-rank inversions at long horizons." The model you would pick from the leaderboard is not necessarily the model you would pick to run unattended.
Three of their findings cut against intuition hard enough to be worth stating plainly. Reliability decay is domain-stratified: their software-engineering graceful-degradation score falls from 0.90 to 0.44 as tasks lengthen, while document processing stays nearly flat (0.74 to 0.71). Frontier models have the highest meltdown rates — up to 19% — "because they attempt ambitious multi-step strategies that sometimes spiral." And memory scaffolds "universally hurt long-horizon performance across all 10 models," which is to say the standard fix made every single model worse.
Now the part that reframes everything above. METR's Thomas Kwa, in a note clarifying the limitations of his own team's metric, writes that "a 50% time horizon of X hours does not mean we can delegate tasks under X hours to AIs," because "some (reliability-critical and poorly verifiable) tasks require 98%+ success probabilities to be worth automating." He then explains why the 98% number is not on the chart: time horizons at 99%+ reliability "cannot be fit at all without much larger and higher-quality benchmarks" — you would need roughly 300 highly diverse tasks in each time bucket, with under 1% label noise.
Not every deployment needs 98%. Plenty of agent work is cheap to check and cheap to redo, and for that class the 50% number is a perfectly serviceable guide. But for the reliability-critical, poorly-verifiable end of the work — the part that actually justifies taking a human out of the loop rather than moving them further back in it — the public measurement apparatus stops exactly where the decision begins. We have good numbers at 50%, decent numbers at 80%, and no number at all at the threshold that decides the thing.
The trajectory
The growth is real and it is fast. METR's original suite showed the 50% horizon doubling about every 196 days across 2019–2025. Their Time Horizon 1.1 update in January 2026 expanded the suite from 170 to 228 tasks and re-fit the trend: 130.8 days for models since 2023, and 88.6 days for models since 2024. Extrapolating the long-run line, Kwa's team put a one-month (167 working hours) horizon at around 2030.
Here is what that extrapolation does not license.
The error bars are enormous. In the current suite the frontier point estimate is 320 minutes with a confidence interval running from 170 to 729 minutes — better than the previous version, and still a factor of two in each direction. The ceiling is close: METR added a note in May 2026 that "measurements above 16 hrs are unreliable with our current task suite." Only 5 of the 31 long tasks (8h+) have measured human baselines; the rest are estimates. Time horizons vary across domains by orders of magnitude — the same methodology finds visual computer-use tasks 40 to 100 times lower. And the tasks are self-contained and well-specified by design, which METR is explicit about: they are closer to what a new hire with no context could do than to what an experienced professional does inside a codebase they know.
The single most important line for anyone doing capacity planning is Kwa's: "Doubling the time horizon does not double the degree of automation. Even if the AI requires half as many human interventions, it will probably fail in more complex ways requiring more human labor per intervention." Fewer failures, more expensive each. That is not a curve that flattens the supervision budget on schedule.
And here is the evidence against my own thesis, because an argument without one is advocacy. METR's own FAQ notes that "the trends for 50%- and 80%-time horizons are very similar," and that they therefore expect higher reliability levels to follow a similar trend. If that holds, reliability is not structurally stuck — it is offset from capability by a fixed distance that calendar time keeps closing, and the thing I am calling a gap is a lag of months, not a wall. The reliability-science paper is also one preprint from one group on one benchmark, not a replicated result; its authors say as much by calling it a framework rather than a finding. And the best-known evidence that AI assistance can cost time — METR's randomized trial where 16 experienced developers took 19% longer on 246 real issues while believing AI had made them 20% faster — used early-2025 tooling on mature repositories those developers already knew, and METR now banners that page with "these results are out of date," having published newer data in February 2026.
My claim survives all of that, but only in its narrow form: not "reliability is hopeless," but "there is no published measurement at the reliability level where deployment decisions are made, and the distance between 80% and 98% is where the entire operational cost lives."
What if…
Speculation from here. Flagged, not asserted. Each of these may simply not happen.
What if the field stops reporting success rates and starts reporting verification cost? The honest unit for a business is not "how often does it succeed" but "how many minutes of human review does one agent-hour create." That number is measurable today — every team running agents already produces it accidentally. For it to become standard, two things would have to be true: someone would have to make it cheap to instrument, and buyers would have to ask for it in procurement instead of asking for a benchmark score. The signal to watch is the first serious vendor that publishes a review-cost curve instead of a leaderboard position.
What if reliability turns out to be an architecture property rather than a model property? The memory-scaffold result points that way — adding the component designed to extend the horizon degraded every model tested. If that replicates, the gains at 98% would come from failure boundaries, checkpointing and narrowing the class of input an agent is trusted with, not from the next model. What would have to be true: the finding survives other suites, and someone demonstrates a scaffold that measurably raises the high-reliability horizon without raising capability. If it goes that way, the advantage shifts from whoever has the best model to whoever designs the best containment — which is unusually good news for small teams.
What if the domain stratification hardens into an economic split? Software engineering degraded steeply in that study; document processing barely moved. Push that out and you get two different markets — genuine automation in flat-degradation domains, and permanent human supervision everywhere else, priced as supervision rather than as headcount replacement. What would have to be true: the stratification is a property of the work, not an artifact of how those particular tasks were scored.
What I'm doing about it
I run a fleet of about thirty scheduled agents on my own operation — research, publishing, security triage, and the job pipeline. It has been running for months, which is long enough to have opinions grounded in something other than a demo.
Almost none of my real failures have been capability failures. They have been a route that worked twice and failed on the third run. A scheduler that silently dropped roughly 38 hours of runs. A file-upload path that reported success while corrupting the payload — it transcribed about 24,600 characters of a 31,400-character file and returned no error at all.
So the operating rule in my fleet is not trust, it is proof. Before any irreversible action, the agent has to produce artifact-level evidence — byte count, MIME type, item count, and a screenshot of the resulting state — and the tool's own return value is explicitly not admissible. "The tool said ok" is a claim. The artifact is the evidence. That distinction is the whole difference between an agent that works at 80% and one you can leave alone, and it costs one extra read per action.
The second habit is older and comes from a quantitative research harness I built and eventually shut down: hypotheses get written into a ledger with a kill threshold set before the test runs. Twelve hypotheses tested, twelve killed with evidence. That is not a market opinion and it is certainly not advice — it is the same engineering discipline pointed at a different target. Decide in advance what result would make you stop, then honour it when the result arrives.
Both habits exist for the same reason. A 50% time horizon is a statement about a benchmark. A production system is a claim about the ninety-ninth run. The work between those two things — verification, boundaries, proof — does not disappear because the model got better at the fiftieth. It just gets easier to forget you were supposed to be doing it.
Next week: what "human in the loop" actually costs once you measure it in minutes instead of asserting it in principle. Research home: blackicelabs.ca