The pitch for AI coding tools is simple: everyone ships faster. The research is more interesting than the pitch. One controlled trial found a 55% speedup. Another, on senior engineers working in their own codebases, found a 19% slowdown. Both are real, and the gap between them is the part worth understanding before you bet a roadmap on a tool.
Key Takeaways
- In a randomized trial, developers using GitHub Copilot finished a task 55% faster (CI 21–89%), with the biggest gains for less-experienced devs (Peng et al., GitHub RCT).
- In a 2025 METR trial of experienced developers working in their own repos, the tools produced a 19% slowdown, even though the devs believed they were 20% faster (METR).
- The difference is task context: greenfield and well-defined vs large, high-context, established code.
- AI raises the value of senior judgment; it doesn't replace it.
The 55% Study: Where AI Clearly Wins
The optimistic number comes from a real randomized controlled trial. Developers were asked to build an HTTP server in JavaScript, and the group with Copilot finished 55% faster than the control, with a confidence interval of 21 to 89% (Peng et al.). The benefit was largest for less-experienced developers. That fits intuition: on a well-scoped, common task, an AI that has seen ten thousand HTTP servers is a genuine accelerator, and it helps a junior more because it fills gaps they don't have yet.
The 19% Study: Where AI Quietly Costs
Now the number nobody puts on a slide. In 2025, METR ran a randomized trial with 16 experienced open-source developers, each averaging five years and around 1,500 commits in their own large repositories. They worked on real tasks from their own projects: bug fixes, refactors, features. Before starting, they expected AI to speed them up about 24%. Afterward, they felt it had sped them up about 20%. The measured result was a 19% slowdown (METR).
Read that again, because the self-perception gap is the scary part. These weren't skeptics fumbling a new tool. They were experienced engineers who came away convinced the AI helped, while the stopwatch said it hurt. In a big, familiar codebase, the time spent prompting, reading, and correcting AI output exceeded the time it saved. This isn't a lone result either: Google's DORA 2024 research found AI adoption associated with small decreases in delivery throughput and stability, not the gains teams expected (DORA 2024).
| Context | AI effect | Why |
|---|---|---|
| Greenfield, well-defined task | Big speedup | Common patterns, low context needed |
| Junior developer | Larger gain | Fills knowledge gaps |
| Large, familiar codebase | Slowdown | Prompt + review + fix > time saved |
| Senior in their own repo | Feels faster, measures slower | Perception diverges from reality |
What This Means for Staffing
The takeaway isn't "AI tools are good" or "bad." It's that they shift where value lives. They compress the easy, common work, the exact work a junior used to spend days on, which means the scarce skill is increasingly judgment: knowing what to build, catching the subtly-wrong suggestion, and owning an unfamiliar codebase. That's the skill a model doesn't provide, and it's what we test for in how to verify a senior engineer. It's also why we think AI raises, not lowers, the premium on senior engineers, a point we made in AI-proficient engineers and why they matter. See available engineers.
The Honest Counterpoint
The METR study is one trial with 16 developers, and the tools improve every quarter, so don't read a single slowdown number as a permanent verdict. AI coding assistants are genuinely useful, especially for boilerplate, unfamiliar languages, and first drafts. The point isn't to ban them. It's to be honest that they're not a uniform speed button, that the people who benefit most are often the ones with the least context, and that a senior engineer's felt productivity with AI can diverge from their real productivity. Measure outcomes, not vibes.
Frequently Asked Questions
Do AI coding assistants make developers faster?
Sometimes dramatically, sometimes not. A controlled trial found a 55% speedup on a well-defined task, while a 2025 METR trial found experienced developers 19% slower in their own large repositories. Context decides.
Why would AI slow down a senior engineer?
In a large, familiar codebase the engineer already knows what to do. The time spent prompting the AI, reading its output, and fixing the subtly-wrong parts can exceed the time it saves.
Should we adopt AI coding tools?
Yes, but measure real outcomes rather than trusting how fast people feel. Expect big wins on boilerplate and junior tasks, smaller or negative effects on deep work in established systems.
Do AI tools replace senior engineers?
No. They compress routine work and raise the premium on judgment, reviewing AI output, and owning unfamiliar code, which are senior skills a model doesn't supply.
The Bottom Line
AI coding assistants are real and useful, but "everyone ships faster" is marketing, not data. The gains concentrate on well-defined tasks and less-experienced developers; on deep work in established codebases, even senior engineers can lose time while feeling faster. Adopt the tools, measure the outcomes, and keep investing in the judgment they can't supply.
Roberto Espinoza is CEO of Ruzora, which helps US startups hire pre-vetted senior LATAM engineers in 72 hours. See available engineers.
