What the AI ROI research actually measured: METR, MIT's 95%, the Copilot trials and more

METR, MIT's 95%, the UK and Australian Copilot trials, Forrester and eight more AI studies: base, method, who paid and what not to conclude, as of Nov 12, 2026.

The most quoted AI productivity figures measure different things, and most of them measure what people said. Three of the studies here are randomized trials. One found experienced developers 19% slower with AI while they believed they were 20% faster. None measured an organization's cash return from its own books. The one ROI figure among them, Forrester's 116% for Microsoft 365 Copilot, is modeled for a composite company in a study Microsoft commissioned.

This page sets out each study's base, method, exact claim and who paid for it, and what not to conclude from it. Every figure links to the authors' or publisher's own page, read on November 12, 2026, except MIT's, which MIT no longer hosts and which links to an archived copy. Several widely quoted figures have since been revised by their own authors; the changed figures are listed below.

Which AI productivity studies are randomized?

Three: METR's developer trials, Cui, Demirer and colleagues' Copilot field experiments at Microsoft, Accenture and a Fortune 100 company, and Dell'Acqua and colleagues' experiment with BCG consultants. Each randomly decided who or which task got AI, and measured the work.

Two more have a small randomized part. Choi and Xie ran a framed experiment with 99 accountants on one task, and the UK Department for Business and Trade timed 11 people on four tasks, with and without Copilot. Brynjolfsson, Li and Raymond used a staggered rollout, which is close to an experiment but not randomized. Humlum and Vestergaard linked surveys to Denmark's national earnings and employment registers, and Stanford RegLab ran a preregistered benchmark test. The rest are surveys, interviews, self-reported time or a vendor's model.

Method Studies
Randomized trial of real or designed work METR 2025 and 2026, Cui et al., Dell'Acqua et al.
Staggered rollout or administrative data Brynjolfsson et al., Humlum and Vestergaard
Preregistered benchmark test Stanford RegLab
Mixed, mostly self-reported UK cross-government trial, UK DBT, Australian Treasury, Choi and Xie
Survey or interviews MIT NANDA, DORA, Thomson Reuters
Vendor-commissioned model Forrester for Microsoft

What did the software developer studies find?

They disagree, and the disagreement is mostly about what was counted. METR timed experienced developers on their own projects and found AI slowed them in early 2025. Cui and colleagues counted completed pull requests across three companies and found 26% more. DORA asked developers, and more than 80% said AI made them more productive.

Study Who ran it, who paid Base and method What it found What not to conclude
METR, early-2025 developer study (July 10, 2025; paper) METR, a nonprofit "funded by donations"; developers paid $150 an hour 16 experienced open-source developers, 246 issues in their own repositories, early 2025. Randomized by task With AI allowed, developers "take 19% longer to complete issues." They expected a 24% speedup and afterward believed AI had sped them up by 20% That AI slows developers in general. METR now marks the page "out of date," and its own table says the study is not evidence that AI fails to speed up most developers
METR, uplift update (February 24, 2026) METR 57 developers, 800+ tasks, from August 2025. Same randomization; times self-recorded Developers from the first study: tasks took an estimated 18% less time with AI (range: 38% less to 9% more). New developers: 4% less (15% less to 9% more) That AI now makes developers 18% faster. Both ranges include no effect, and METR calls the data "an unreliable signal," partly because 30% to 50% of developers said they held back some tasks they didn't want to do without AI
Cui, Demirer, Jaffe, Musolff, Peng and Salz (February 2025 draft; Management Science, November 2026) MIT, Princeton and Wharton economists, with two Microsoft Research co-authors; two authors funded by the MIT GenAI Initiative; the companies ran the experiments 4,867 developers at Microsoft, Accenture and a Fortune 100 electronics maker, given GitHub Copilot at random, for 2 to 8 months "a 26.08% increase (standard error: 10.3%) in completed tasks"; less experienced developers adopted more and gained more That Copilot adds 26% to output everywhere. The authors call each experiment "noisy," results vary by company, and a completed pull request is a count, not value
DORA, State of AI-assisted Software Development (September 2025) Google Cloud and IT Revolution, with sponsors including Swarmia, Thoughtworks, Datadog and Jellyfish 4,867 survey respondents, June 13 to July 21, 2025, plus 78 interviews 90% use AI at work and more than 80% "believe" it raised their productivity; 30% report little or no trust in AI-generated code; AI adoption goes with higher throughput and more delivery instability That AI causes faster delivery. The report says it speaks "in terms of comparisons," not effects, and the productivity figure is respondents' belief

The Cui figure and DORA's survey happen to share a base of 4,867; they are different groups. Two larger 2026 studies of coding tools, of 655 firms and of 500,000 open-source developers, are covered in More AI code isn't more product.

What did the studies of support agents, consultants, accountants and other workers find?

The studies of one job each found gains on the work they measured, largest for less experienced people, and none measured what the gain was worth to the firm. The one study that followed earnings and hours across a whole country's workers found no effect.

Study Who ran it, who paid Base and method What it found What not to conclude
Brynjolfsson, Li and Raymond, "Generative AI at Work" (Quarterly Journal of Economics, May 2025; authors' version) Stanford and MIT economists; funded by the Stanford Digital Economy Lab; data from the AI vendor and one client 5,172 customer-support agents at a Fortune 500 software company, mostly in the Philippines, 2020–21. Staggered rollout, difference-in-differences AI assistance raised "issues resolved per hour, by 15% on average," about 30% for less experienced and lower-skilled agents, with "small declines in quality" for the most skilled That it applies to other jobs. The authors say it covers "a particular AI tool, used in a single firm, within a single occupation," and they had no data on wages
Dell'Acqua et al., "Navigating the Jagged Technological Frontier" (2023 working paper; Organization Science, March 2026) Harvard, Wharton and MIT researchers with three BCG Henderson Institute co-authors; BCG ran the experiments; the research was funded in part by Harvard Business School 758 BCG consultants, spring 2023, GPT-4. Preregistered randomized experiment on designed tasks, not client work Inside AI's capabilities, 12.2% more tasks and 25.1% faster with better quality. On a task outside them, consultants with AI were less likely to be right: 60% to 70% correct, against 84.5% without That AI makes consultants 40% better. The published abstract drops that figure, and the tasks were timed exercises, not billed engagements
Choi and Xie, "Human + AI in Accounting" (Journal of Accounting Research, 2026; working paper) Stanford GSB and MIT Sloan; field data from an unnamed AI accounting software vendor, which the authors say had no role in the analysis 277 accountants surveyed; 79 small and mid-sized firms' platform data, over 200,000 transactions; a framed experiment with 99 accountants AI use goes with more clients per accountant (18% per standard deviation of use, in the October 2025 revision), 12% more general-ledger detail and a monthly close 7.5 days faster That AI caused these gains. The authors say most results "should be interpreted as descriptive associations rather than causal effects," and the gains are gross of the platform's cost
Humlum and Vestergaard (NBER working paper, revised March 2026; first circulated as "Large Language Models, Small Labor Market Effects") University of Chicago and University of Copenhagen economists; funded by two University of Chicago research centers; no vendor funding stated Workers in 11 occupations exposed to AI chatbots in Denmark, surveyed in 2023 and 2024 (about 25,000 workers at 7,000 workplaces in 2024), linked to national earnings and employment registers. Difference-in-differences "precise null effects on earnings and recorded hours at both the worker and workplace levels, ruling out effects larger than 2% two years after the launch of ChatGPT"; adopters report saving about 3% of their hours That AI changes nothing at work. The time savings are self-reported, the register data runs to the end of 2024, it covers Denmark only, and NBER working papers have not been peer-reviewed

What did the government Copilot trials find?

Staff said Microsoft 365 Copilot saved them time, and none of the three trials found that the time became more output. All three were mostly self-reported, and none had a control group for the whole trial.

Study Who ran it, who paid Base and method What it found What not to conclude
UK cross-government experiment (June 2, 2025) Government Digital Service, across 12 organizations 20,000 employees, September 30 to December 31, 2024; 7,115 survey responses, usage data for 14,500 "Trial participants saved an average of 26 minutes a day when using M365 Copilot" That anyone timed 26 minutes. Participants picked a range from "none or less than 5 minutes" to "more than an hour," and the report took each range's midpoint. It "was not possible to identify how time saved was spent"
UK Department for Business and Trade (August 28, 2025) DBT's own evaluation team 1,000 licenses, October to December 2024, about 70% volunteers and 30% picked at random; 300 diaries, 19 interviews, 11 people timed on four tasks Self-reported savings on most tasks, largest on written documents; scheduling and image generation took longer. Timed, Copilot users were slower at an Excel analysis and much faster at summarizing reports. It "did not find evidence that time savings have led to improved productivity" That Copilot lowers productivity. The body says productivity "was not a key aim of the evaluation," and the timed tasks had 11 people
Australian Treasury trial (February 2025) An internal evaluation by the Australian government's evaluation unit, within Treasury 218 volunteer staff, May 20 to August 23, 2024; surveys of 61 to 153, a manager survey of 49, focus groups 49% of respondents said Copilot sped up their tasks; 80% of managers saw no change in their team's timeliness; 59% said it helped with little or none of their weekly work That Copilot pays for itself in 13 minutes a week. That figure is a break-even estimate: the time an APS6 employee would need to move to higher-value work to cover the license. The report says it "did not explicitly measure time saved"

The Australian government's separate whole-of-government trial, run by the Digital Transformation Agency in 2024, is left out because its pages couldn't be read. Measuring Microsoft 365 Copilot ROI covers how Microsoft's own dashboard values Copilot time.

What do MIT's 95% and the vendor studies actually show?

MIT's figure comes from interviews and a conference survey, and Forrester's and Thomson Reuters' come from companies that sell to the people they describe. Each is a statement about what people reported or what a model assumes, not a measured return.

Study Who ran it, who paid Base and method What it found What not to conclude
MIT NANDA, "The GenAI Divide: State of AI in Business 2025" (July 2025; MIT's link, archived August 18, 2025) Project NANDA at the MIT Media Lab; no funder stated Labeled "Preliminary Findings": a review of 300+ public AI initiatives, interviews at 52 organizations and 153 survey responses from senior leaders at four conferences, January to June 2025 "95% of organizations are getting zero return," and most pilots have "no measurable P&L impact" That 95% of AI projects fail. The report calls its figures "directionally accurate based on individual interviews rather than official company reporting," gives two definitions of success, and MIT no longer hosts it
Forrester, Total Economic Impact of Microsoft 365 Copilot (March 2025) Vendor-commissioned: it "is commissioned by Microsoft and delivered by Forrester Consulting" 16 decision-makers at 12 organizations interviewed and 367 users surveyed, combined into a modeled composite of 25,000 employees and $6.25 billion revenue, over three years An ROI of 116%, net present value of $19.7 million, payback in 10 months That you will see 116%. Forrester "makes no assumptions as to the potential ROI that other organizations will receive." Its 2024 enterprise study projected 112% to 457%
Thomson Reuters, Future of Professionals 2025 (June 2025) Thomson Reuters, which sells AI tools to the professionals it surveys 2,275 responses from legal, tax, accounting, audit, risk and trade professionals, February and March 2025 Legal professionals "expect to free up nearly 240 hours per year," which it says "unlocks an average annual value of $19,000 per professional" That anyone saved 240 hours. It is a prediction, the report doesn't say how the $19,000 was worked out, and the 2026 edition (1,816 respondents) doesn't repeat it

In Stanford RegLab's test, the legal research tools from LexisNexis and Thomson Reuters gave a hallucinated answer to between 17% and 33% of questions. That is an error rate on a hard benchmark, not a measure of time or value.

Study Who ran it, who paid Base and method What it found What not to conclude
Magesh et al., "Hallucination-Free?" (Journal of Empirical Legal Studies, 2025; preprint) Stanford RegLab and Yale researchers; no vendor funding stated 202 preregistered legal questions, March to May 2024, answers scored by hand Lexis+ AI and Ask Practical Law AI hallucinated on 17% of answers, Westlaw AI-Assisted Research on 33%, and GPT-4 on 43% That today's tools err this often in practice. The authors say the test "only captures a point in time," the questions were designed to be hard, and the rate isn't an estimate of real-world use

Which widely quoted AI figures have changed?

Six of these studies have since been revised, retitled or replaced by their own authors, so a figure quoted from an earlier version may no longer appear in the current one.

Study Figure still quoted What the current version says
Brynjolfsson, Li and Raymond 14% on average, 34% for novices, 5,179 agents (NBER, 2023) 15% on average, about 30% for less experienced and lower-skilled agents, 5,172 agents (Quarterly Journal of Economics, 2025)
Humlum and Vestergaard, "Large Language Models, Small Labor Market Effects" Effects larger than 1% ruled out; time savings of 2.8% (May 2025) Retitled "Still Waters, Rapid Currents" (NBER, revised March 2026): "precise null effects on earnings and recorded hours," ruling out effects larger than 2%; adopters report saving about 3% of their hours
Dell'Acqua et al. 19 percentage points less likely to be correct; more than 40% higher quality (2023 working paper) "19% less likely to produce correct solutions" and "significantly improved quality" (Organization Science, 2026)
Choi and Xie 55% more weekly client support (May 2025) 18% more per standard deviation of AI use (October 2025 revision)
METR 19% slower METR marks the 2025 result "out of date" and its 2026 update has wide ranges on either side of zero
Thomson Reuters 240 hours and $19,000 a year From the 2025 edition; the 2026 edition doesn't include them

What does this research mean for measuring your own AI ROI?

It means three things for your own numbers: don't rely on what people say, don't count freed hours as cash, and measure against your own baseline rather than a study's average.

Self-reports run ahead of measurement. METR's developers believed AI made them 20% faster while the clock showed them 19% slower. About half of Australian Treasury's trial staff said Copilot sped up their tasks, while 80% of their managers saw no change in their team's timeliness. METR's own 2026 survey of technical workers found a median self-reported speedup of 3x, and METR warns that "survey results are not necessarily grounded in reality." A survey tells you whether people like a tool. It doesn't tell you how much time it saves.

Freed hours aren't cash. In Denmark, two years after ChatGPT's launch, chatbots had no measurable effect on workers' earnings or recorded hours, ruling out anything larger than 2%, while users reported saving about 3% of their time. The UK's DBT found no evidence that time saved led to improved productivity. Brynjolfsson and colleagues work out that their support firm could handle the same issues with 12% fewer worker-hours, but that becomes cash only if staffing actually changes. Freed capacity is hours freed, priced at what they cost. It becomes cash when a cost actually falls or the hours are billed.

Your own baseline beats a benchmark. The same tool helped on some tasks and hurt on others: DBT's Copilot users were faster at summarizing and slower at Excel analysis, and BCG's consultants were more often wrong on the task outside AI's reach. Cui and colleagues' effect varied by company. Forrester's composite isn't your company. A baseline is the figure before the AI that later readings are compared against, and only yours tells you what the AI changed for you.

For what finance leaders say about measuring AI ROI, from Gartner, Deloitte, EY and others, see What CFO surveys say about measuring AI ROI.

Where does Roiva fit?

Roiva measures the return on an organization's own AI initiatives, and it is built around the three points above: time the work, keep hours apart from cash, and compare against your own baseline.

  • Timed, not self-reported. A time study in Roiva times one task with the AI and without it, assigns each participant's days without it at random, and publishes the means as the formula's baseline and reading. Roiva supplies no minutes of its own.
  • Hours beside cash, never added. Freed capacity is reported beside cash value and never added to it. ROI is cash only, and a value figure counts once it is approved, by a person or by an auto-approve setting the account turns on.
  • Rules before results. A measurement charter records the counting rules finance signs before any results exist.

Roiva doesn't benchmark you against these studies, and a time study of a few people has the same limits of sample size the DBT and METR authors describe.

How was this page put together?

Roiva wrote it. AI agents read each study from the authors' or publisher's own page on November 12, 2026 and quoted from it, and the main figures were then checked against saved copies of those pages. Each figure links to its source. Where a study has been revised, the current version is quoted and the older figure is listed under the changed figures.

Some sources couldn't be read in full. MIT no longer hosts the NANDA report, so it is read from the Internet Archive's copy of MIT's own link. The journal versions of the Cui and Dell'Acqua papers were read only as abstracts, so their other figures come from the authors' working papers, and the Brynjolfsson figures come from the authors' version, whose abstract matches the journal's. The Australian Digital Transformation Agency's trial is left out because its pages couldn't be read.

Roiva sells software that measures the return on AI initiatives, so research showing that self-reported AI gains are unreliable suits us. That is a reason to check this page, and the links are there so you can. Corrections are welcome through the contact page.

See it against your own numbers

The free AI ROI Exposure Assessment runs Roiva's published math on your inputs — five minutes, no account, no card.

Get the next one

New field notes by email as they're published. Or subscribe to the RSS feed.