Five studies changing how I think about AI in software engineering

AI compressed the upstream work. What does that mean for everything downstream?

Welcome to the latest issue of Engineering Enablement, a weekly newsletter sharing research and perspectives on developer productivity.

🗓 Join me on July 23 for a readout of the upcoming State of AI Impact in Engineering: Q2 Report. We’ll discuss new findings from DX’s data on AI tool usage, spend, and impact across 500+ organizations. Register here.

---

Every once in a while, several independent papers arrive at roughly the same time and collectively tell a bigger story than any one of them does alone. This week, I’m sharing five recent papers that have significantly influenced how I’m thinking about AI and software engineering.

Each paper tackles a different question. Some measure the productivity impact of AI coding assistants. Others examine how those gains propagate through the software delivery process, explore what developers actually want from future AI systems, or reconsider the kinds of debt we should be paying attention to in an AI-assisted world.

Despite coming from different research groups and using very different methodologies, they all seem to be converging on the same underlying story.

AI is compressing the upstream work of software engineering. The more I sat with these papers, the less I found myself asking, “Is AI making developers faster?” and the more I found myself asking, “What happens after the code is written?” Are we actually shipping more value? Where do the new bottlenecks emerge? And what are the costs if understanding can’t keep pace with generation?

After reading these five papers, I came away with one overarching conclusion: we’re generating code faster than we’re generating the systems needed to safely understand, verify, and deliver it.

A quick note on disclosure: three of these papers come from people I know and work with extensively. None of the papers are mine.

Here they are, in the order I’d recommend reading them.

1. GitHub Copilot and Developer Productivity

Paper: Heilman, A., Kyllo, A., Murphy-Hill, E. GitHub Copilot and Developer Productivity: An Observational Dose-Response Analysis.

The first paper I want to highlight tackles the familiar question of whether GitHub Copilot makes developers more productive, but it does so with one of the more clever research designs I’ve seen.

Rather than simply comparing Copilot users to non-users (which are getting harder and harder to find), the authors control for Active Coding Time (i.e., how much time developers spend actively engaging with development tools) and examine how productivity changes within the same engineer over 43 weeks across a population of 16,223 developers.

The payoff of this design is that it compares engineers to themselves rather than to one another. Using that approach, the authors found that weeks with the highest Copilot usage were associated with ~40% more completed PRs per hour of coding time than weeks with no usage.

The relationship showed a clear dose-response pattern (a way to do a causal analysis, once everyone is already using the tools). More Copilot engagement was associated with more PR throughput, although the gains appeared to level off at very high usage.

The authors ran seven robustness and falsification tests to rule out alternative explanations (team-level effects, generic AI engagement, PR slicing, shifts toward easier work). The positive association remained remarkably consistent.

Interestingly, the gains were not concentrated in tiny PRs. The strongest effects were observed for larger PRs (7+ files), arguing against the idea that developers are simply breaking work into smaller units.

It’s a thoughtful analysis and shows that we’re not just coding more, we’re increasing coding efficiency as well. These findings anchor many of the studies that follow in this roundup.

2. Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools

Paper: Demirer, M., Musolff, L., Yang, L. Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools.

The next paper I’m highlighting was published by the National Bureau of Economic Research. It analyzes AI adoption across 100,000+ GitHub developers and asks a more nuanced question than Heilman’s: when AI makes individual coding steps faster, how much of that gain actually survives all the way to shipped software?

The authors examine how AI productivity gains propagate through a hierarchy of software development: lines of code → files → commits → pull requests → projects/repos → releases.

They found that AI is clearly increasing coding activity, and the gains grow with each generation of tools. They estimate roughly +40% more commits from autocomplete, growing to +140% from interactive coding agents, and finally +180% from autonomous agents.

However, those gains fall off significantly as work moves through the software delivery process. The largest effects are seen in code generation, but smaller effects appear in repos touched, small still in releases shipped, and ultimately software consumed by users. Even with very large increases in coding activity, the effect on shipped software is much smaller, topping out at roughly +30% more releases. This is illustrated in figure 2 below.

One of the findings I found most interesting is that they estimate a low elasticity of substitution (~0.25) between AI-generated output and human effort. That’s an economics concept that measures how replaceable human work is with AI output. As a methodology nerd, and someone with an economics degree, I found this particularly clever — they infer this elasticity from how AI productivity gains attenuate across the delivery process. Their estimate suggests AI and human work are still largely complements rather than substitutes, with substantial human effort still required to review, integrate, validate, and ship software.

One open question is whether the observed fall-off through the delivery process is some fundamental limit of software engineering, or simply the fact that organizations have not yet adapted their processes to an agentic world.

If Heilman tells you Copilot is making engineers measurably faster, this paper asks the harder question: faster at what, exactly?

3. The Impact of AI Coding Assistants on Software Engineering

Paper: Vella, A., Blincoe, K. The Impact of AI Coding Assistants on Software Engineering: A Longitudinal Study.

The next study I want to highlight is unique because it isn’t just a snapshot in time, it’s a six-month longitudinal study of 95 professional software engineers. It also calls into question a relationship that we’ve long believed to be a bedrock of developer experience.

The study was done using two questionnaires six months apart, mixed-methods, with reflexive thematic analysis on the open-ended responses.

Vella found that productivity perceptions were stable and strongly positive over time. 84% of study participants reported improvement at both time points. Consistent with the first two studies in this round-up, the story of accelerated throughput is real and persistent.

The really striking finding is what the authors call the productivity-experience paradox. Among the matched cohort, the share of engineers reporting worse DevEx on at least one dimension nearly doubled in just six months, from 14% to 27%. Flow state was the most vulnerable; cognitive load eroded modestly; feedback loops actually improved.

More importantly: while the cross-sectional correlations between DevEx and productivity were strong, the change scores didn’t correlate. Productivity and developer experience appear to be decoupling over time in AI-assisted workflows. For those of us who’ve spent years working with the SPACE and DevEx frameworks, that’s worth sitting with.

While this study didn’t have a particularly large population, the findings were significant and rigorously validated, proving that a study doesn’t have to be massive if the strength of results is strong enough. This longitudinal design is rare and valuable, and the productivity-experience decoupling is the kind of finding worth replicating in larger populations.

4. To Copilot and Beyond: 22 AI Systems Developers Want Built

Paper: Choudhuri, R., Badea, C., Bird, C., Butler, J., DeLine, R., Houck, B. AI Where It Matters: Where, Why, and How Developers Want AI Support in Daily Work.

Last year I published a paper called AI Where It Matters, and my co-authors ended up writing a 2nd paper based on the original survey responses (860 Microsoft developers across roles, domains, and geographies). The paper outlines a roadmap of 22 AI tools that developers want beyond just code generation, centered around a concept they call “bounded delegation.” A lot of this echoes what the rest of this round-up is circling:

• The “right-shift” burden. Because AI is speeding up code generation, it’s creating a massive bottleneck downstream. Devs are getting flooded with more code to review, more production incidents to debug, and documentation that falls behind faster than ever.

• The move to verification. Developers don’t want more code-generation assistants; they want AI embedded into verification tasks — tools that automatically assemble log/trace “case files” for on-call incidents, PR reviewers that catch complex business logic flaws before human review, change-aware test generation that knows which assertions actually matter.

• “Bounded delegation.” There is a strict boundary around where developers want AI to stop. Developers want AI to absorb the tedious “assembly work” surrounding their craft (updating docs, writing edge-case unit tests), but never the core logic, architecture, or critical decision-making. Notably, developers drew this line even for tasks they acknowledged AI could plausibly handle — suggesting it’s not just about capability gaps and won’t move just because models improve.

• Four non-negotiable guardrails. For future AI tools to be adopted, developers say they must enforce explicit authority scoping (no auto-approvals), clear data provenance, explicit uncertainty signaling (the AI must admit when it doesn’t know something), and least-privilege security access.

You can check out both papers and an interactive website here: aka.ms/ai-where-it-matters

5. From Technical Debt to Cognitive and Intent Debt

Paper: Storey, M. From Technical Debt to Cognitive and Intent Debt: Rethinking software health in the age of AI

I’ve saved this for last because I think this is the most important paper I’ve read in a long time. Margaret-Anne Storey makes a generational argument: the metaphor we’ve used for decades to think about software health—technical debt—is no longer sufficient. AI is reducing technical debt (through refactoring, test generation, automated review) while quietly accelerating the accumulation of two other forms of debt that matter more in this era.

Technical debt lives in code. It accumulates when implementation decisions compromise future changeability. AI is genuinely helping here.

Cognitive debt lives in people. It accumulates when a team’s shared understanding of a system erodes faster than it’s replenished. When AI generates the code, developers may accept it without building the same mental model they would have built by writing it themselves. Multiply that across a team and over time, and you get “an accumulation of not knowing.”

Intent debt lives in artifacts. It accumulates when the goals, constraints, and rationale that guide a system—the things both humans and AI agents need to work safely—are unclear, unwritten, or forgotten. As more development is AI-assisted, intent debt becomes a first-order constraint on what AI can actually do for you.

The three debts interact and compound. Intent debt causes cognitive debt; cognitive debt causes technical debt; technical debt amplifies cognitive debt. Managing software system health requires attention to all three layers, not just the one easiest to measure.

The four practical implications Storey draws are worth reading in full, but the headline is: treat understanding as a deliverable. Just as working code is a product of software development, shared understanding should be treated as a first-class deliverable, not something that happens as a side effect of writing code.

Final thoughts

Read together, these five papers say something stronger than any of them say individually. AI is genuinely making code generation faster, and the per-engineer efficiency gains are real (Heilman). But those gains don’t survive the trip to shipped software at anywhere near the same magnitude (Demirer). The bottleneck has moved downstream to review, integration, verification, and understanding. Developers feel it, they’re explicitly asking for tools to address those bottlenecks while refusing to delegate the parts of the job they consider craft (Choudhuri). The lived experience of working this way is more uneven than the productivity numbers suggest, with flow and cognitive load eroding even as throughput holds (Vella). And the deepest cost may be one we don’t yet measure: the slow erosion of shared understanding, which is what makes any system safe to change (Storey).

The bottleneck has moved. Our tools, metrics, and team designs haven’t moved with it yet. That’s where the next several years of work in our field are going to happen.

---

That’s it for this week. And make sure to sign up for my upcoming live research readout covering new findings on AI’s impact, where we’ll discuss data from both DX and the broader industry.

-Brian