METR Experiment Recalculated: On Complex Tasks and Reviews, Developers Using AI Lose 24% of Their Hours

In 2025, the independent research center METR measured the work of 16 experienced open-source developers across 246 real tasks drawn from their own repositories: with AI assistants, a typical ticket took 19% longer to close than doing it manually. Independent researcher Solomon Shalom Lijo, in a paper titled "Counting the Bandages: On measuring the time that AI saves" (published on arXiv on October 23, 2026), recalculated the team's total time expenditure using METR's open logs. Accounting for prolonged slowdowns on complex tasks, developers using AI actually lost 24% of their working time (or 34% without adjusting for the initial size of the tasks), while fixing issues flagged during code review took an average of 14.

9 minutes per ticket versus 9.7 minutes without the AI. The author then examined OpenAI's GDPval benchmark and, across 540 Claude Code sessions, demonstrated why stopping the clock at draft submission is meaningless: minutes saved during generation translate into missed bugs and hours of rework.

How the same logs produced three different numbers: 19%, 24%, and 34%

Lijo reproduced the calculations using METR's public repository and showed that the same 246 completed tasks yield three different figures depending on how team time is measured:

• A 19% increase comes from the geometric mean of the logarithms — the figure METR itself used in its headline. It reflects the typical shift for a single individual task.
• A 24% increase emerges when you divide all actual hours spent with AI by the hours spent without it. With AI, the spread in completion times is noticeably wider: when a developer gets stuck in failed prompting iterations on a large ticket, these multi-hour tails eat into the total time budget more heavily than the logarithmic average suggests.
• A 34% increase shows up in raw hours, without correcting for the initial sampling skew. Random assignment happened to put larger tickets into the AI bucket: before starting work, developers themselves estimated these tasks at an average of 116 minutes, versus 106 minutes in the control group.
• Fixes after code review were measured on 230 of the 246 tickets: reworking code flagged by a reviewer took 14.9 minutes with AI versus 9.7 minutes manually. Even so, Makarov's statistical bounds on the same data leave a wide corridor — AI genuinely sped things up somewhere in the range of 2.6% to 82.1% of individual tasks. A single average number obscures which tickets the tool actually saves time on and which ones it drags down.

Why GPT-5's hundredfold speedup shrinks to 1.1–3.6x

A century ago, engineer Frank Gilbreth tried shaving with two razors at once: he saved 44 seconds in front of the mirror but spent two minutes bandaging the cuts afterward. Modern AI benchmarks make the same mistake when they exclude result verification from the measurement.

A striking example is OpenAI's applied benchmark GDPval. In its announcement, the company claimed flagship models solve tasks for industry professionals roughly 100 times faster and cheaper: GPT-5 generates an answer in 4.5 minutes versus 404 minutes of manual expert labor. But the GDPval paper itself builds in a workflow with mandatory acceptance review of every answer:

• reading and verifying a single draft by a live specialist takes 109 minutes (27% of the total manual work time);
• by Amdahl's law, even with an error-free model, the speedup of the entire process is capped at `404 / (4.5 + 109) = 3.56x`, or at most 3.6 times;
• GPT-5's answers only outperformed the expert ones in 39% of tasks, so once you factor in reworking weak drafts, the GDPval paper's own authors estimate the real speedup at just 1.12–1.39 times.

The same pattern shows up in a field experiment with business consultants (Dell'Acqua et al., 2026) on tasks outside the model's reliable zone. With GPT-4, participants completed solutions in 26.2 minutes instead of 37.7 minutes for the control group — but the share of correct answers dropped from 84.5% to 60.0%. Using the break-even defect threshold formula Lijo introduces, the 11.5 minutes saved only pay off if catching and fixing an error that reached the client costs less than 47 minutes of consultant work.

A 540-session

Claude Code experiment: how inflated trust shuts off review

To measure in practice how a draft author's reputation affects the depth of review, Lijo ran a series of controlled tests on AI agents. The basis was quarterly 10-Q financial reports filed by U.S. companies with the Securities and Exchange Commission (SEC) through June 30, 2026. The reports were published after July 1, 2026 — past the models' training cutoff — so the AI couldn't have memorized the numbers from its weights.

The drafting agent (`claude-haiku-5-5`) compiled a summary of 9 metrics (revenue, net income, and operating margin for three companies), deliberately seeding one error — a line from a different document, an outdated quarter, or swapped figures — in 30% of cases. The document was then checked by a reviewing agent (`claude-sonnet-5-5`, running in an isolated Claude Code session) equipped with a tool for line-by-line verification against the SEC source. Under the prompt's rules, verifying one number cost 3 minutes (27 minutes for all 9 lines), while an error that slipped through to the client carried a 10-hour (600-minute) penalty. Full cross-checking paid off once the defect rate exceeded 4.5%.

The experiment revealed three specific effects:

• An "all or nothing" mode. When the reviewer was honestly told the defect rate was high (10% to 30%), it cross-checked all 9 lines and caught 100% of errors in all 40 paired tests. But there was no gradual, selective checking — around the 4.5% threshold, the model switched sharply between full verification and zero queries to the source.
• Blindness from an understated defect estimate. On the same drafts with a real 30% error rate, the reviewer was told in the prompt that the author's error rate was ten times lower, at 3%. In 29 of 40 sessions, `Sonnet 5.5` never even opened the source report, checking an average of just 2 of the 9 numbers. This saved 21 minutes on verification, but a fabricated figure made it into the client-facing report in 22.5% of summaries instead of zero — a net loss of 135 minutes per document.
• Caveats in the prompt proved useless. In a separate branch, the "3% defect rate" figure was paired with an explicit instruction to treat that estimate as unverified. In 26 of 40 runs, the model acknowledged in its reasoning that the figure couldn't be trusted — and still submitted the summary without a single cross-check. The numerical anchor outweighed the textual warning.

How to measure AI's benefit to a team without fooling yourself

Instead of a single stopwatch that stops "at first commit," the author proposes tracking three metrics when rolling out AI tools (code and raw logs are open on GitHub in the `tsb-bench` repository and on Zenodo):

1. How many total hours the team spends on a task from start to finish, including code review and rework of returned pull requests. 2. How many extra errors slip into the final output — based on blind acceptance testing in groups with and without AI. 3. At what error cost the speedup stops paying off, and when manual oversight can be safely turned off: to remove mandatory review with 95% confidence at an acceptable defect rate of 0.85%, the tool must first complete 352 verified tasks in a row without a single failure.