How much does AI impact development speed?
A randomized trial at Google estimates a 21% speed boost from AI coding tools.
Welcome to the latest issue of Engineering Enablement, a weekly newsletter sharing research and perspectives on developer productivity.
This week I read How much does AI impact development speed?, a randomized controlled trial from Google that explores how much AI coding tools can speed up task completion. The researchers recruited nearly 100 full-time engineers at Google and tested three production AI features on an enterprise-grade task in the company’s internal development environment.
My summary of the paper
The study set out to answer a simple but important question: How much does AI actually speed up development in a real-world enterprise setting?
To find out, the researchers ran a randomized controlled trial with 96 full-time Google engineers. Developers were randomly assigned to complete a complex task with or without three AI-enhanced features: AI Code Completion, Smart Paste, and Natural Language to Code. Developers used their real IDE (a customized version of VS Code), real internal tools, and a realistic task: editing 10 files and 474 lines of code to build a logging service on Google’s internal infrastructure.
Here’s what the researchers found:
AI sped up task completion by ~21%
On average, developers with access to AI completed the task in 96 minutes, compared to 114 minutes without AI. After controlling for other factors like experience and daily coding time, the researchers estimated a 21% improvement in speed (although their confidence interval was large). This result mirrors a prior enterprise study that found a ~26% increase in task completion with Copilot.
Developers who code more daily saw greater gains
The largest speed gains from AI were observed among developers who code more hours per day. While not statistically significant, the interaction effect was large, suggesting that those coding daily may benefit more from AI tools. The authors hypothesize that this may be because AI tools still make mistakes, and developers have to carefully check and fix the code the AI suggests. So developers who spend more time coding in their daily work may be better at spotting issues and making quick edits—and in turn, may be more likely to benefit from using AI.
Seniority and prior AI usage didn’t change outcomes
The researchers wanted to explore whether other factors—in particular, seniority and prior AI experience—influence how much developers gain from AI tools in terms of speed improvements. They used Google’s career ladder levels to investigate the relationship between productivity gains and seniority. They also asked developers to self-report on their experience with AI usage prior to the study.
Contrary to some earlier studies, the researchers found that neither seniority nor frequency of AI tool use significantly impacted task speed. More senior developers were faster overall, but they didn’t disproportionately benefit from AI. Similarly, developers who had frequently used AI tools in the past didn’t complete tasks significantly faster than those with less prior experience.
Final thoughts
There’s still much to learn about how AI impacts different aspects of developer productivity. This study contributes to that conversation by showing that AI tools may reduce task completion time by around 21% in enterprise settings. But it’s still unclear from the research the true impact AI is having and what factors—like experience, tool usage, or coding habits—meaningfully drive gains.
The authors emphasize that the impact of AI will likely change over time as the tools evolve, workflows adapt, and developers become more skilled. And while speed is important, it’s only one part of the picture. Future research is needed to understand how AI affects code quality, team dynamics, and long-term developer growth.
Who’s hiring right now
This week’s featured DevProd & Platform job openings. See more open roles here.
Dropbox is hiring multiple Infrastructure and AI Dev Tools roles | US (Remote)
Atlassian Williams Racing is hiring a Software Engineer - Engineering Acceleration | Grove, UK
ScalePad is hiring a Head of AI Engineering & Enablement | Canada (Remote or in-office)
Lyft is hiring a Senior Software Engineer - Developer Experience | Toronto, ON
That’s it for this week. Thanks for reading.
-Abi