Subscribe now

Over the past 16 months, DX has been running a longitudinal study on AI’s impact on engineering velocity across a sample of more than 400 engineering organizations. We found that as AI tool usage increased by an average of 65%, median PR throughput increased by just under 8%. Most organizations are landing in the 5–15% range—a meaningful gain, but far below the 3x or 10x expectations many leaders are being held to.

That gap raises some questions. Why aren’t the gains higher? Where is reclaimed time actually going? And how should leaders be thinking about where to invest next?

This is what and I discussed to kick off the mainstage program at the inaugural DX Annual, which brought together roughly 500 senior engineering and developer productivity leaders for a day focused on navigating the AI era. Brian is a well-known developer productivity researcher, and someone we’ve referenced often in this newsletter—from past interviews with him (here and here) to summaries of papers he has co-authored (like this one and this one). In this conversation, Brian asked me about the study methodology and what sparked this research, as well as what’s limiting gains, how to measure AI’s impact, and how leaders can unlock greater gains.

Today we’re releasing the full report. You can download it here: AI and Engineering Velocity: A Longitudinal Analysis.

What follows is a lightly edited transcript of my conversation with Brian at DX Annual.

Brian: To kick us off, Abi—you’re sitting on one of the largest datasets on how engineers work in the real world. This new report digs into what’s happening as AI adoption scales across the industry. Before we get into the findings, what motivated this research?

Abi: Thanks, Brian. A little background on this study: Nearly every conversation I’ve had with folks over the past few months has started the same way. My CEO, our executives, are expecting astronomical gains because there’s so much hype about AI in the media, but that’s not what we’re seeing on the ground. How do we close that gap or set realistic expectations? How do we even know what to aim for?

I’d bet everyone reading this has experienced that to some extent. So at DX, we started asking ourselves the same question. What can we see in the data about what companies are actually seeing? That’s what we’ve been digging into. This is just the beginning; there’s a lot more to unpack and a lot of nuance, but at least we now have a first look at what’s happened over the past 16 months as AI adoption has matured across companies.

Brian: I’d love to nerd out on research methodology for a minute. Walk us through how DX actually investigated this. How did you design the study and decide which companies to include?

Abi: Our customer network at this point is about 500 companies. For this study, we took a sample of those. We looked for companies that had reached a point of maturity in their developers’ adoption of AI: we defined that as over 75% monthly active usage of AI coding tools. In almost all cases, there’s been a significant rise in adoption over the past 12 months. We also narrowed the scope to companies with over 100 engineers and excluded companies that had gone through major liquidity events, M&A, IPOs, or regulatory changes that could have had an impact.

In terms of methodology, it’s really hard to single out causality. What is AI and what are confounding factors? One approach I see a lot is cross-sectional analysis: comparing developers who use AI to developers who don’t. There are flaws in that approach because oftentimes the developers using AI were already the ones coding the most. They always look better.

What we did instead is look at longitudinal data at the organizational level. As AI adoption has matured, what has been the impact on organizational velocity and throughput?

Brian: What were your hypotheses, and what did you actually find?

Abi: As a developer myself, I had some gut-feel hypotheses, but honestly, from a study standpoint, we really didn’t know. We were pushing the team, asking “what’s the answer?” We wanted to know just as much as you probably do.

What we ultimately found is that most organizations fall in the 10-15% range in terms of PR throughput increase. The actual median was around 8%, the mean was around 11%. So a real gain, but well below what most leaders expect.

It’s much more modest than many CEOs, who are talking to their CEO friends about the 10x gains they’re supposedly delivering, would expect. In our engagements with companies and customers, it wasn’t completely surprising. There are a lot of factors underlying that, which we’ll get into.

Brian: One of the central themes of the SPACE framework is that developer productivity is nuanced and about much more than counting activities. So given that, why did you choose PR throughput as your primary measure?

Abi: Measuring velocity or productivity is inherently challenging. We did look across a number of different metrics, including the full spectrum of SPACE and the DX Core 4. But for what we wanted to focus on and publish, we felt PR throughput was the most relevant and practical right now, mostly because that’s what we see most organizations talking about and focusing on. We wanted to meet the world where it’s at.

We also explored our proprietary metric TrueThroughput, which uses AI to weight each PR by relative size and complexity so it takes some of the noise out. We liked that signal, but again, for relevance and practicality, PR throughput is a more accessible metric. We have more data on other metrics and how those have been affected, and we’ll eventually publish those as well.

Brian: I suspect that, for many, a 5%, 7%, 10% increase in PR throughput matches what they’ve been feeling. But as you mentioned, when I talk to business leaders, they’re often expecting 20, 30, 40, 50% increases in productivity. So why do you think the gains are lower than what people outside the industry might expect?

Abi: Our team conducted follow-up interviews to really dig into that question. There were a number of factors. These are just the top themes, not the full list.

The top one probably wouldn’t surprise most of us: coding is not the primary bottleneck for engineers. Brian, at Microsoft, you published a study on where engineering time is spent, and only about 14% of developer time is spent coding. It’s a very small part of where our time and money goes. So if AI is only optimizing that slice, it partially explains why the enterprise velocity gains are limited.

We also heard how AI tools and new automation introduce new bottlenecks. Things like review burden, technical debt, cognitive debt—which is a newer concept we can talk about. We also heard about challenges with adoption—a lot of social friction and cultural clashes that are inhibiting full adoption. There were a few more, but those were the major themes.

Brian: You don’t have to go far out in the distribution before you see some organizations with substantially higher gains. Any time the median and mean are that different, it implies outliers. So what set apart some of the companies with disproportionately high gains from the more typical experience?

Abi: We don’t have a great answer to this yet. Our first focus was on why the gains aren’t higher. The next question is: for those where gains are higher, why?

There are two parts to how we want to approach that. One is that we’re working to tease out whether the companies at the top of the range are seeing real, material gains or superficial, inflated numbers. We want to peel back the onion.

Two, assuming the gains are real, we want to understand what strategies and factors are driving those results. I think a lot of it boils down to what I’ve seen anecdotally: an all-in culture, a fully bought-in organization with centralized rollout and championing of AI tools. It’s more of a cultural shift toward infusing AI into the entire SDLC, not just coding, with an aligned push to drive adoption.

Brian: That aligns with some recent research I published showing that even something like leadership advocacy significantly increased the gains people saw from AI. You said something earlier that I think is really important and often misunderstood: software engineering is so much more than coding. Coding may be central to our identity as developers, but it’s only about 14% of our day.

Brian: One of the things I’ve been seeing in my research is that, yes, we’re producing 5, 7, 10, 15% more pull requests, but we’re doing it substantially more efficiently. For hands-on keyboard time spent coding, we’re producing about 40% more PRs per hour. That hints that we are reclaiming some coding time. Do you have any idea where that time is getting reinvested?

Abi: This has been a similarly perplexing question. For the past six months, I’ve heard customers asking: we’re seeing meaningful self-reported time savings from developers, but we’re not seeing that show up in our output and throughput. Where’s the disconnect? Where’s that time going?

We don’t know yet—this requires more research. But one preliminary hypothesis is that if developers only spend 14% of their time coding, then it makes sense that the time they’re recouping is ratably distributed across their activities. Only 14% of the time savings go back into coding, which would explain why we don’t see one-to-one output gains mapping to time savings.

Another hypothesis ties back to why the gains aren’t higher: some of the time savings come with side effects that require more time, like overseeing the work, QAing it, reviewing it. There’s also the lag factor: what do developers do while they’re waiting for agents to produce the code?

So those are some of the hypotheses, but we don’t have a clear answer yet.

Brian: There’s a reason we don’t just measure lines of code as a good measure of productivity. Writing more code isn’t always the right answer. As we increase velocity, what are some of the potential unwanted side effects you’ve been seeing?

Abi: Quality and cost are the two I hear customers talking about constantly. Cost is something we’re all trying to wrangle or at least get visibility into. Quality is something we all understand is an underlying risk, but I think it’s still early days. We did see Amazon go very public with this, but there’s some delay between the technical debt we may be creating and the consequences that follow.

The biggest thing I’ve been talking to customers about is a cultural risk I’ve been calling false velocity. I see this in so many organizations right now. And to some extent, even developer productivity leaders can fall into this trap, being so focused on showing off how much faster and more prolific we are with AI, without asking whether it’s leading to meaningful improvement.

You probably have engineers in your organization showing off really impressive things they can do with Claude or whatever tool. But are we asking: is your team’s product velocity actually increasing? Leaders are talking about how many more PRs and lines of code they’re generating, but is their roadmap actually accelerating? Is the quality of their products and code sustainable? There’s a real risk of being so focused on demonstrating what we can do that we’re not paying attention to whether we’re actually getting better—whether we’re materially improving our businesses.

Brian: So what’s your recommendation to engineering leaders who want to apply AI beyond just code generation? Where should they be looking?

That’s the biggest question I’m hearing from leaders right now. A lot of folks are at a point where they’ve rolled out one or more popular coding tools. Adoption is in a fairly satisfactory place, and they’re asking: what next? Especially if you’re sitting at that 10% number and asking how to get further.

The things I’m hearing about and seeing are, first, looking left and right of code. We’ve accelerated coding to some extent, but if that’s only 14% of where our time and money goes, what about the other 86%? How do we accelerate and optimize that?

Second, beyond thinking about the SDLC, there’s the question of how we improve the developer experience more deeply. Things like deep work and documentation—the friction points developers experience. How can we leverage AI to materially improve those?

And the third theme is this idea—there are many names for it—autonomous engineering, async engineering, background agents. I think today most of us are focused on leveraging AI to accelerate human work. Humans are still in the cockpit, steering and monitoring the work of the agents. The question is: how do we complement that acceleration with augmentation? Meaning more autonomous agents that are truly augmenting your human workforce and working in parallel, not just underneath your developers. Those are the three big themes.

Brian: You and I are both huge fans of Dr. Margaret-Anne Storey’s work. She’s been talking a lot about cognitive debt and the human cost of these AI transformations. What have you been looking at in that area?

Abi: I’m a big fan of that research. I think it falls under what I was talking about earlier in terms of risks and the time delay; we’re not always thinking about what the consequences might be 12 months from now if we adopt certain ways of working.

The loss of human understanding of the systems we’re building is a really interesting risk. There are different takes on how material it is. Some people argue it doesn’t matter if humans don’t deeply understand the systems, because they can use AI to quickly regain that understanding when needed. Others argue: no, if you need to call the mechanic, they should have mastery over the machine. I don’t think we know yet, but it’s a valid question.

Brian: We’ve hinted at it throughout this conversation—are we actually delivering innovation faster, or are we just moving bottlenecks around? My first reaction to an unanswered question is always: how do we measure it? Any thoughts on how leaders should be thinking about measurement as AI transforms how we work?

Abi: Our approach to measurement—some things stay the same and some things need to evolve. How we think about the overall software organization, things like quality and velocity, I think those stay the same. And especially if we’re trying to understand how things are changing with AI, you need consistent measures pre, post, and during this transformation.

But there are also new tools, new ways of working, new workflows being born every week. Measuring these new ways of working requires new approaches.

I’ll touch on two things. One is that I think increasingly there’s a need to separate how you’re measuring and framing AI’s impact. You should be thinking about acceleration—how much faster are our humans—as one bucket. And augmentation—how much more capacity are we generating with agents—as a second bucket. Thinking of those as two separate components in your overall formula is useful.

The question of how to measure agents is especially interesting. We’ve been exploring this idea of agent hourly rate: the human-equivalent hours an agent delivers, divided by its cost. That gives you a return-on-investment figure you can compare to human hourly rate, which is an interesting comparison point.

Something really new—not even in our minds last September—is this idea of agent experience. In the same way DX was founded on the idea of measuring developer effectiveness by going to developers and getting signal from them, we’ve applied a similar approach to agents. How do you measure AI agent effectiveness? You go to the agents and get feedback from them on where their bottlenecks and constraints are. We’ve just rolled that out. We’re surveying agents, which is pretty wild.

Brian: That is wild. To wrap up our conversation, give me one quick finding. What’s one thing you’ve learned about making agents more effective?

Abi: You have to ask them, just like we ask our developers. It’s really early days, but we’ve initially begun measuring four factors. For example, we ask the agents: how were the requirements you were given? How easily were you able to understand the codebase you were working in? How well were you steered by the human you were pairing with? We get quantitative metrics from that as well as qualitative feedback—the agent explains where it had challenges or bottlenecks. That’s the idea.

Brian: I love it—the same principles of measurement apply. That is unfortunately all the time we have, but I think we covered a lot of ground.

Abi: A special thank you to Brian, and everyone who attended DX Annual. Stay tuned for the session recordings, coming soon.

This week’s featured DevProd job openings. See more open roles here.

• Cashea is hiring an Infrastructure & Developer Productivity Platform Engineering Manager | Remote

• BNY is hiring an SVP, Application Development Manager | Pittsburgh, PA

• Figma is hiring a Staff Software Engineer, Developer Experience | Remote; US

• Leidos is hiring a Platform Engineer | Remote; US

• Weave is hiring a Senior Platform Engineer, Data Infrastructure | Remote; US

That’s it for this week. Thanks for reading.

Share

Subscribe now

Last week we held our first-ever DX Annual event in San Francisco, bringing together nearly 500 engineering and platform leaders for a full day focused on developer productivity and AI. Thank you to everyone who joined us. Session recordings will be available in the coming weeks.

Abi: This week we have a guest post from Dr. Margaret-Anne Storey, a Professor of Computer Science and a Canada Research Chair in Human and Social Aspects of Software Engineering. Margaret-Anne is one of the most widely published researchers on developer productivity, having co-authored the SPACE and DevEx frameworks amongst many other works.

Margaret-Anne: Earlier this year I published two posts exploring how generative and agentic AI may be quietly shifting where the most significant risks in software development lie, away from technical debt and code quality, and toward something harder to see and measure: the erosion of shared understanding across teams. This is what I refer to as cognitive debt. The response to these posts surprised me, as practitioners confirmed that cognitive debt was a significant challenge they were facing. They also proposed concrete suggestions for recognizing and mitigating cognitive debt. I’m combining both posts below, with light edits, as they tell a connected story.

For readers who want to go deeper, two papers extend these ideas. In Theory of Troubleshooting, co-authored with Arty Starr, we ground the cognitive debt concerns in cognitive science, showing how making sense of unexpected system behavior places considerable demands on working memory and attention, and how prolonged troubleshooting leads to cognitive fatigue with real implications for developer well-being. In From Technical Debt to Cognitive and Intent Debt I propose a Triple Debt Model that adds a third dimension to this framework: intent debt, the erosion of externalized rationale that both developers and AI agents need to work with to safely maintain and evolve a codebase.

How generative and agentic AI shift concern from technical debt to cognitive debt

The term technical debt is often used to refer to the accumulation of design or implementation choices that later make the software harder and more costly to understand, modify, or extend over time. Technical debt nicely captures that “human understanding” also matters, but the words “technical debt” conjure up the notion that the accrued debt is a property of the code and effort needs to be spent on removing that debt from code.

Cognitive debt, a term gaining traction recently, instead communicates the notion that the debt compounded from going fast lives in the brains of the developers and affects their lived experiences and abilities to “go fast” or to make changes. Even if AI agents produce code that could be easy to understand, the humans involved may have simply lost the plot and may not understand what the program is supposed to do, how their intentions were implemented, or how to possibly change it. Where cognitive load is what developers experience in the moment, cognitive debt is a project-level property, capturing how a team loses understanding over time.

Cognitive debt is likely a much bigger threat than technical debt, as AI and agents are adopted. Peter Naur reminded us some decades ago that a program is more than its source code. Rather a program is a theory that lives in the minds of the developer(s) capturing what the program does, how developer intentions are implemented, and how the program can be changed over time. Usually this theory is not just in the minds of one developer but fragments of this theory are distributed across the minds of many, if not thousands, of other developers.

I saw this dynamic play out vividly in an entrepreneurship course I taught recently. Student teams were building software products over the semester, moving quickly to ship features and meet milestones. But by weeks 7 or 8, one team hit a wall. They could no longer make even simple changes without breaking something unexpected. When I met with them, the team initially blamed technical debt: messy code, poor architecture, hurried implementations. But as we dug deeper, the real problem emerged: no one on the team could explain why certain design decisions had been made or how different parts of the system were supposed to work together. The code might have been messy, but the bigger issue was that the theory of the system, their shared understanding, had fragmented or disappeared entirely. They had accumulated cognitive debt across their team faster than technical debt, and it paralyzed them.

This dynamic echoes a classic lesson from Fred Brooks’ Mythical Man-Month. Adding more agents to a project may add more coordination overhead, invisible decisions, and thus cognitive load. Of course, agents can also be used to manage cognitive load by summarizing what changes have been made and how, but the core constraints of human memory and working capacity will be stretched with the push for speed at all costs. The reluctance to slow down and to do the work that Kent Beck calls “make the hard change easy” is what will lead to cognitive debt and cognitive load in the future.

In a breakout session at a recent Future of Software Engineering Retreat (arranged by Martin Fowler and Thoughtworks) we discussed how developers need to slow down and use practices such as pair programming, refactoring, and test-driven development to address technical debt AND cognitive debt. By slowing down and following these practices, cognitive debt can also be reduced and shared understanding across developers and teams rebuilt.

But what can teams do concretely as AI and agents become more prevalent? First, they may need to recognize that velocity without understanding is not sustainable. Teams should establish cognitive debt mitigation strategies. For example, they may wish to require that at least one human on the team fully understands each AI-generated change before it ships, document not just what changed but why, and create regular checkpoints where the team rebuilds shared understanding through code reviews, retrospectives, or knowledge-sharing sessions.

Second, we need better ways to detect cognitive debt before it becomes crippling. Warning signs include: team members hesitating to make changes for fear of unintended consequences, increased reliance on “tribal knowledge” held by just one or two people, or a growing sense that the system is becoming a black box. These may be signals that the shared theory is eroding.

Finally, this phenomenon demands serious research attention. How do we measure cognitive debt? What practices are most effective at preventing or reducing it in AI-augmented development environments? How does cognitive debt scale across distributed teams or open-source projects where the “theory” must be reconstructed by newcomers? As generative and agentic AI reshape how software is built, understanding and managing cognitive debt may be one of the most important challenges our field faces.

I explored these questions further in a recent keynote at the ICSE Technical Debt Conference and Panel. Cognitive debt tends not to announce itself through failing builds or subtle bugs after deployment, but rather shows up through a silent loss of shared theory. As generative and agentic AI accelerate development, protecting that shared theory of what the software does and how it can change may matter more for long-term software health than any single metric of speed or output.

Discussion: What I’m hearing about cognitive debt (so far)

After publishing the original post (above) on cognitive debt, it sparked thoughtful discussion across different communities. I’ve synthesized what I’m hearing below, and am connecting it to other reflections I’ve been reading.

There’s a growing concern about shared understanding

Several practitioners, including Simon Willison and others on a Hacker News discussion of a Martin Fowler article, describe experiencing cognitive debt directly. They talk about getting lost in their own projects and finding it harder to confidently add new features. They can move faster, but they lose the deeper sensemaking that connects decisions to intent, and intent to code.

This is not just about code quality. It is about whether individual developers and product teams can maintain a coherent mental model of what the system is doing and why.

Across these discussions, one theme is consistent: velocity can outpace understanding.

Cognitive debt hurts developers, not just the software

Technical debt lives in the code. Cognitive debt lives in people.

When shared understanding erodes, the pain shows up in:

• Loss of confidence when making changes

• Heavier review burden

• Debugging friction

• Slower onboarding

• Stress and fatigue

The software may be “working”, but the theory of the system becomes harder to access and keep track of. The cost is not only structural. It is experiential.

Siddhant Khare has written about AI fatigue. Steve Yegge reflects on burnout emerging from AI-accelerated development. Annie Vella eloquently writes about the emotional and cognitive experience of uncertainty when systems become harder to reason about. These perspectives reinforce that this is not just an engineering discipline issue, but one that affects how developers feel and function.

Cognitive debt, like technical debt, must be repaid

Martin Fowler notes that, like technical debt, cognitive debt must eventually be repaid. I agree.

But rebuilding lost knowledge requires restoring the distributed theory of the system. That includes capturing intent, the rationale behind decisions, key constraints, and how the architecture supports change. That theory is not stored in code alone. It is distributed across people, documentation, tests, conversations, tooling, and increasingly, AI agents.

Repayment means maintaining all of these, not just refactoring code or updating architecture documents. Under pressure to move quickly, whether in startups racing to learn or in large organizations pushing AI adoption, that repayment can feel expensive and easy to defer.

“This is just engineering”, but incentives are changing

Several commenters, including Michael Würsch, argue that cognitive debt reflects a failure of good engineering discipline. Clear specifications, rigorous reviews, extensive testing, and explicit architecture documentation should prevent knowledge loss.

In principle, I agree. But in practice, the incentives are shifting. AI lowers the cost of producing structure. It becomes easier for structure to evolve faster than shared understanding can stabilize. Even disciplined teams must consciously throttle or shape their practices to keep understanding aligned with change.

Specifications and documents are not sufficient if they are not living artifacts that teams actively engage with.

Emerging mitigation strategies

Encouragingly, many readers shared how they are mitigating cognitive debt.

They describe:

• More rigorous review practices

• Writing tests that capture intent

• Updating design documents continuously

• Treating prototypes as disposable

Some also describe using AI to reduce the cost of these practices, and even to support cognitive tracking, dependency management, and explanation.

Used deliberately, AI may help make cognitive work more visible rather than obscuring it.

The open question: How will high-performing teams adapt?

High-performing teams have always managed technical debt intentionally. As AI is adopted by startups and large companies, the question becomes how teams will manage cognitive debt.

How will they shape socio-technical practices and tools to externalize intent and sustain shared understanding? How will they use Generative and Agentic AI not only to accelerate code production, but to maintain their collective theory?

As AI reduces technical friction, shared understanding may become the bottleneck on performance.

I am continuing to watch how this evolves. If you are seeing mitigation practices that work in real teams, I would love to learn from them. As mentioned above, check out the article that goes deeper into how to recognize and mitigate cognitive debt and also proposes using the concept of intent debt to capture when decisions and the why behind a system are not captured for future humans and agents to refer to.

This week’s featured DevProd job openings. See more open roles here.

• Cashea is hiring an Infrastructure & Developer Productivity Platform Engineering Manager | Remote

• BNY is hiring an SVP, Application Development Manager | Pittsburgh, PA

• Figma is hiring a Staff Software Engineer, Developer Experience | Remote; US

• Leidos is hiring a Platform Engineer | Remote; US

• Reddit is hiring a Staff Software Engineer, Developer Experience | Remote; US

• Weave is hiring a Senior Platform Engineer, Data Infrastructure | Remote; US

That’s it for this week. Thanks for reading.

Share

Subscribe now

📅 Live research readout: Sign up here to join Abi for a readout and interactive discussion about the impact of AI on engineering velocity.

Since releasing our Q4 2025 AI Impact Report, reasoning models have improved, agentic workflows have gained traction, and teams are finding new ways to integrate AI into how they build.

Each quarter, we analyze the prior quarter’s data to benchmark AI adoption rates, productivity impact, and how teams are using these tools. For this edition, we expanded our dataset by over 40% more data points to give you a more representative picture.

We’ve compiled this analysis into our AI-Assisted Engineering: Q1 2026 Impact Report, which we are releasing today.

The full report covers industry adoption rates (now at 93%), tooling benchmarks, and the growing risks of “Shadow AI.” You can read the full report here. Below are two previews of interesting shifts we’ve observed since last quarter.

Engineering managers are shipping 4x more code than six months ago

Perhaps the most dramatic structural shift we observed is the changing nature of the engineering manager role.

In Q4, we noted that engineering managers who used AI daily were shipping twice as many pull requests over the period of time the sample was taken. That figure has doubled again: engineering managers are now shipping 4x as much code.

Why this matters: The definition of a developer is expanding. As agentic tools and AI assistants drastically lower the barrier to context-switching and boilerplate generation, managers are finding it far easier to get back into the codebase without sacrificing their leadership responsibilities. Coupled with an industry-wide trend of flattening organizational structures, AI is enabling the true return of the player-coach, allowing EMs to stay technically sharp and contribute directly to the product.

The question then is: if EMs can meaningfully contribute code again, should org structures evolve to reflect that? From what we’re hearing, it’s top of mind for many leaders, but most are still in “watch and learn” mode. No one we’ve spoken to has made structural changes based on this shift yet, but we’ll continue to monitor this trend.

Junior engineers now save more time with AI than Staff+ engineers

In our Q4 2025 report, Staff+ engineers who used AI daily were regaining more time than any other tenure band. Just three months later, the data tells a different story.

Junior engineers who use AI daily are now saving 4.9 hours per week, edging out daily Staff+ users who are saving 4.8 hours.

While time savings alone doesn’t tell a full story—in fact, this shift should come with heightened attention to quality, maintainability, and security—it is interesting to see this change in work impact among junior engineers.

Why this matters: Staff+ engineers, who naturally spend more time mentoring and reviewing rather than writing net-new code, are still seeing benefits, but junior engineers have officially taken the lead in total time saved. We provide a look at AI’s impact on quality in the full report; we’ll continue to pay close attention to quality measures amongst junior developer cohorts in the future.

What else is in the report?

The full Q1 2026 report provides comprehensive data and visual benchmarks on:

• Rust-specific improvements: How improvements in model reasoning and agentic loops have caused Rust to leapfrog other modern languages in AI-driven time savings.

• Quality volatility: Why some companies are seeing change failure rates swing by 2% (a 50% increase in defects), and why automated testing is no longer optional.

• The shadow AI risk: How developers are bypassing enterprise guardrails, and why acceptable use policies are critical for ensuring safe use of AI

• Small vs. large enterprises: Why companies with fewer than 200 developers are outpacing large enterprises in efficiency gains.

Final thoughts

As always, a reminder: there is no “average” experience with AI impact. Looking at industry trends and averages like those in this report can help contextualize your performance and do some pattern-matching, but I want to caution against using these averages as a guide to what your performance should look like. The truth is that AI impact is very asymmetrical and looks very different for each organization. Some are doing very well when it comes to throughput, but quality is suffering; some have done a great job of using AI for migrations, but are struggling to incorporate AI in testing and release processes. Averages abstract away the nuances of each company’s transformation journey. The best way to use this information is to enhance—not replace—your own metrics and AI strategy.

This week’s featured DevProd job openings. See more open roles here.

• Cashea is hiring an Infrastructure & Developer Productivity Platform Engineering Manager | Remote

• BNY is hiring an SVP, Application Development Manager | Pittsburgh, PA

• Figma is hiring a Staff Software Engineer, Developer Experience | Remote; US

• Leidos is hiring a Platform Engineer | Remote; US

• Reddit is hiring a Staff Software Engineer, Developer Experience | Remote; US

• Weave is hiring a Senior Platform Engineer, Data Infrastructure | Remote; US

That’s it for this week. Thanks for reading.

Share

In this episode, I’m joined by Eleanor Millman, Senior Staff Product Manager, and Mina Tawadrous, Associate Director of Platform Engineering at SiriusXM, to discuss how platform teams can scale prioritization without relying on revenue.

We talk through how SiriusXM moved beyond RICE to build a custom framework for internal platforms, using weighted factors like developer speed, reliability, cost, and trust to guide decisions across teams.

We also explore their concept of “assumptions as code,” in which teams store and reuse assumptions in a central repository to reduce misalignment and improve decision-making, with AI helping to surface and validate those assumptions.

We close with how this system is shaping SiriusXM’s 2026 prioritization approach and what it signals about a broader shift toward builder-driven product development.

Some takeaways:

Prioritization breaks without a shared system

• Prioritization does not scale naturally across teams. What works for one team breaks down at the org level with multiple stakeholders and competing requests

• Platform teams lack a clear revenue signal. Unlike product teams, they must prioritize based on indirect impact

• A shared framework aligns decisions. Without it, prioritization defaults to local optimization and noise

RICE is a starting point, not a solution

• Standard frameworks miss key dimensions for platform teams. Urgency and indirect impact are not captured well

• “Impact” needs to be decomposed. SiriusXM broke it into developer speed, reliability, cost, security, and more

• The framework must evolve over time. Iteration was critical to making it useful in practice

Weighting forces real tradeoffs

• You cannot prioritize everything at once. Increasing one dimension (like cost) necessarily deprioritizes others

• Assigning weights makes decisions explicit. Leaders must commit to what matters this quarter

• The output drives alignment across teams. A single prioritized list reduces cross-team conflicts

Data and conversation work together

• The framework creates a place to attach data. Metrics like reliability scores inform prioritization decisions

• Disagreements surface quickly. Teams can see where assumptions or inputs differ

• Conversations, not just scores, drive alignment. The value comes from debating inputs, not just ranking outputs

Assumptions are the real bottleneck

• Most disagreements come from hidden assumptions. Teams often believe they are aligned when they are not

• Assumptions can be conflicting, invisible, or stale. All three create friction in decision-making

• Making assumptions explicit improves clarity. It becomes easier to validate or challenge them

Storing assumptions as code scales learning

• Assumptions are stored in a central repository. User research and data become reusable across teams

• This reduces duplicated effort. Teams don’t need to rediscover the same insights repeatedly

• It creates a shared source of truth. Assumptions become visible, versioned, and easier to update

In this episode, we cover:

(00:00) Intro

(01:17) Mina’s role and path into platform engineering

(02:03) Eleanor’s background and shift into product

(03:15) Scaling prioritization across platform engineering teams

(05:41) Aligning platform priorities with stakeholders

(09:08) Evolving RICE into a platform-specific prioritization framework

(11:33) Iterating on the prioritization framework over time

(16:57) How the framework, data, and conversations drive alignment

(19:06) Storing assumptions as code in a central repository

(26:47) Resolving assumption conflicts with user interviews

(30:47) How stored assumptions integrate with AI workflows

(35:30) Standard mode and different user personas

(37:20) The industry shift towards builders

(41:04) The challenges of platform engineering

(43:36) How SiriusXM is prioritizing in 2026

Referenced:

• Measuring AI code assistants and agents

• SiriusXM

• VMware

• How SiriusXM revamped their platform and developer experience

• RICE Scoring Model | Prioritization Method Overview

• The evaporating cloud: A tool for resolving workplace conflict

Subscribe now

📅 Live research readout: Sign up here to join Abi for a readout and interactive discussion about an upcoming report that looks at the impact of AI on engineering velocity.

Onboarding new hires has always been an expensive and time-consuming process, and an area where AI has the opportunity to have a meaningful impact. In Q4 2025, when we looked at Time to 10th PR (a measure we use to track ramp-up time), we saw AI already having a dramatic effect. In some companies, Time to 10th PR was cut in half: from 91 days with no AI usage to 49 days with daily AI use.

We revisit this data quarterly. In our most recent analysis, we’re seeing that trend continue: onboarding time is faster today than it was in Q4 of last year.

For this latest cut, we analyzed data from a random sample of 400 companies during the period from October 2025 to February of 2026. We measured the average number of days between a developer’s start date and their 10th merged PR. We specifically looked at an aggregate of engineers who showed daily use of AI and compared how ramp-up time changed from October 2025 to February 2026.

The dataset includes large, global organizations with 500+ developers, spanning both tech and non‑tech companies.

As of April 2026, the current average Time to 10th PR across our dataset is 33 days, down from 39 days in Q4 2025. That represents a further ~15% decrease quarter-over-quarter, and more than a 50% reduction since Q1 2024, before AI usage was widespread. The change this quarter isn’t as dramatic as the 50% reduction we saw last year, but it is still meaningful. This trend begs the question: will onboarding times continue to fall, or are we approaching a natural floor where many of the remaining steps are not easily compressed by AI? As more agentic solutions are devised for onboarding processes, we may see this figure continue to decline.

Part of the improvement we’re seeing since last quarter comes from the fact that AI is no longer an optional, individual choice in many organizations. Developers are guided to use AI to ask questions about the codebase, to explain architectural decisions, and to propose draft changes that senior engineers can review. The result is that the “figuring things out” phase is shorter.

At the same time, Time to 10th PR is a narrow measure. It tells us how quickly someone reaches a specific milestone, but it does not, on its own, tell us about the quality of those changes, the amount of rework they generate, or the depth of understanding new hires have of the systems they’re touching. That will be something we explore more in future analyses.

Stay tuned for the full report coming soon.

This week’s featured DevProd job openings. See more open roles here.

• Awardco is hiring an Engineering Manager, Cloud Platform | San Francisco, CA

• BNY is hiring an SVP, Application Development Manager | Pittsburgh, PA

• Figma is hiring a Staff Software Engineer, Developer Experience | Remote; US

• Leidos is hiring a Platform Engineer | Remote; US

• Reddit is hiring a Staff Software Engineer, Developer Experience | Remote; US

• Weave is hiring a Senior Platform Engineer, Data Infrastructure | Remote; US

That’s it for this week. Thanks for reading.

Share

In this special episode of Engineering Enablement, I welcome back Jesse Adametz, this time as host.

In our conversation, we explore how AI is showing up across the SDLC, not just in code generation, and how it is shifting bottlenecks across the development process. We unpack what “AI readiness” actually means in practice, and why it often comes down to developer experience fundamentals like documentation, environments, and feedback loops.

We also discuss why enablement matters more than tool choice, how teams are thinking about measuring ROI, and what changes as background agents become more common. Finally, we explore how the role of the engineer may evolve, what questions teams are still trying to answer, and the challenges of non-engineers contributing to codebases.

Some takeaways:

AI is expanding beyond coding into the full SDLC

• The focus has shifted from code generation to the entire software lifecycle. Teams are applying AI to planning, prototyping, review, and documentation—not just writing code.

AI readiness is a developer experience problem

• The biggest blockers to AI adoption are long-standing DX gaps. Missing documentation, inconsistent environments, weak CI, and unclear system boundaries all limit effectiveness.

• Tool choice is not the primary driver of success. Models and tools are evolving too quickly for this to be a durable advantage.

• Some organizations are formalizing AI enablement as a function. Dedicated teams are emerging to drive adoption and share practices.

Measuring AI ROI is messy and still evolving

• Correlation vs causation makes attribution difficult. High AI usage often correlates with already high-performing engineers.

• Longitudinal analysis is more reliable than snapshots. Tracking changes over time gives better insight into impact.

• Token spend introduces real cost considerations. AI creates a direct, variable cost that organizations must evaluate.

AI impact falls into two buckets: amplification and augmentation

• Amplification improves human productivity. This includes higher throughput, time savings, and better developer experience.

• Augmentation extends capacity beyond humans. Agents begin to act as additional “headcount,” completing work independently.

• These require different measurement approaches. Amplification focuses on human output, while augmentation focuses on agent output relative to cost.

Background agents shift how work gets done and where bottlenecks appear

• Agents enable work to happen outside the human loop. Tasks can be completed asynchronously and proactively.

• This changes the developer role. Engineers move toward reviewing, guiding, and orchestrating agent output.

• Human workflows can become the bottleneck. If agents produce work faster than humans can process it, the constraint shifts.

• This reframes productivity. The question becomes where human involvement adds the most value.

Specs and documentation are becoming critical infrastructure

• AI makes documentation a core dependency. It directly impacts the quality of outputs.

• Poor documentation leads to poor results. Agents can duplicate systems or make incorrect assumptions without context.

• Documentation is shifting from optional to essential. It is now foundational for both human and AI productivity.

In this episode, we cover:

(00:00) Intro

(02:12) Where AI is showing up across the SDLC

(05:53) AI readiness and its link to developer experience

(08:23) Why enablement, education, and experimentation matter more than tool choice

(13:05) The case for a dedicated enablement team

(14:50) Measuring AI ROI: challenges and tradeoffs

(19:46) Background agents and token spend

(24:12) Measuring agent output with PR throughput

(26:58) How the engineer role might change

(31:01) Specs and documentation in the age of AI

(33:11) Non-engineers writing code

(35:30) What’s changing in the SDLC and open questions

Referenced:

• Measuring AI code assistants and agents

• Lessons from Twilio’s multi-year platform consolidation

• The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win

• How Claude remembers your project - Claude Code Docs

• specIsJustCode : r/ProgrammerHumor

Subscribe now

📅 Live research readout: Sign up here to join Abi for a readout and interactive discussion about an upcoming report that looks at the impact of AI on engineering velocity.

Each quarter, DX publishes data on how AI is being used at 500+ organizations and the impact it’s having. One metric we’ve been following is the percentage of code that’s written by AI—today’s newsletter shares a preview of what we’re seeing.

Currently, we measure the “percentage of AI-generated code” by asking developers directly how much of their merged code they believe is written by AI. This measure is captured quarterly by DX. In Q1 2026, developers at more than 500 organizations reported their average percentage of AI-authored code, along with how frequently they use AI tools (daily, weekly, or monthly). We aggregated these self-reported percentages to calculate the mean share of merged AI-generated code for each usage segment and overall.

We ran the previous Q4 analysis in the same way. In the future, we will collect this data and compare against a new system-based measure that automatically tracks the percentage of AI-generated code.

For this quarter, here’s a preview of what we’re seeing. Stay tuned for the full report later this month.

AI-generated code reaching production sees a slight increase

Since last quarter, the average share of merged code authored by AI has moved from 22% to 27.4%. While that’s directionally up, it’s not as meaningful of a change as we had expected.

The stability around 30% could be read two ways. On one hand, it may simply mean that current models and tooling are producing roughly the same proportion of merge-ready code as they were last quarter. However, there’s reason to believe something else is going on. The second half of 2025—particularly November and December—is widely regarded as a turning point for AI-assisted development. Models like Opus 4.5 represented a significant leap in capability, and some have gone so far as to say that anything before November 2025 shouldn’t even be used as a baseline, given how much changed in a short window.

If that’s true, then the more likely explanation for the stability around 30% is that most teams haven’t yet fully adapted to take advantage of those improvements. New models don’t automatically translate to more merged AI code; developers still need to update their workflows, build trust in the output, and find the right use cases for more capable tools.

There’s some evidence to support this interpretation. The daily users segment saw the largest increase in AI-generated code, moving from 24.1% to 30.8%. Meanwhile, weekly and monthly users, who are less likely to have adjusted their habits, saw smaller increases.

Final thoughts

This analysis focuses on how much merged code is AI-generated, but it doesn’t answer questions like whether a higher percentage of AI-generated code is associated with changes in quality. It also begs the question of whether there are certain types of organizations, or groups of developers, that are merging more vs. less AI-generated code.

We’ll explore these questions, and others like them, in the full Q1 AI Impact Report.

Additionally, in future reports we’ll compare this self-reported measure against a telemetry-based metric that tracks AI-generated code directly from version control systems. This will not only provide another lens on AI’s impact, but also give us a more granular view of how patterns are changing over time and where those changes are concentrated.

Stay tuned.

This week’s featured DevProd job openings. See more open roles here.

• Amplitude is hiring an Infrastructure Engineer | Lindon, UT

• Awardco is hiring an Engineering Manager, Cloud Platform | San Francisco, CA

• BNY is hiring an SVP, Application Development Manager | Pittsburgh, PA

• Figma is hiring a Staff Software Engineer, Developer Experience | Remote; US

• Leidos is hiring a Platform Engineer | Remote; US

• Lob is hiring a Staff Platform Engineer | Remote; US

• Weave is hiring a Senior Platform Engineer, Data Infrastructure | Remote; US

That’s it for this week. Thanks for reading.

Share

Subscribe now

Last week I hosted a live AMA about measuring the impact of AI and where organizations are with their rollout today. (Special thanks to Jesse Adametz for hosting the discussion.) One of the questions that came up had to do with measuring ROI: specifically, how should we think about correlation vs. causation? Leaders want to be careful about whether they should attribute changes in metrics to AI tools.

I answered the question live—you can download and watch the full recording here—or read this week’s newsletter for my take.

Many organizations make this mistake: They compare developers who use AI heavily to those who use it less, notice that the heavy users have higher throughput, and then conclude that AI must be the cause. The problem with this kind of analysis—“do people who use AI more have higher code throughput?”—is that in many cases, the developers who use AI more are the ones who were already coding more in the first place.

That’s why most of the time, longitudinal analysis (looking at how things change over time) is more informative. But it’s also harder to do. You need clean data over a long enough period, and you still have to account for confounding factors. For example, one big confound right now is the heightened pressure in many companies to increase throughput. Leaders are simultaneously pushing for more output and more AI usage. Because of this, some of the increase in throughput is likely due to this pressure itself—a classic case of Goodhart’s law, where once a metric becomes a target, it stops being a good measure.

When thinking about the ROI of AI more broadly, it’s useful to break it into two buckets:

• Amplification: How much more productive are humans thanks to AI? Here we look at:

  • Throughput (are engineers shipping more by using AI?)

  • Time saved (how much time do developers feel they’re saving in specific workflows?)

  • Developer experience scores (are AI tools improving the overall developer experience?) We can then convert improvements in developer experience into time savings, for example using something like a developer experience index.

• Augmentation: To what extent are you actually extending your engineering capacity by using agents, as if they were additional headcount? One unit we like to use here is human-equivalent hours: how much work are agents delivering, how much would that have taken a human, and then you divide that by how much it cost. Dividing the human-equivalent hours by the cost gives you an agent hourly rate. If that effective rate is low, it indicates a high-ROI place to invest.

This amplification/augmentation framing is also helpful when talking to executives. You can say: to some extent we’re amplifying our humans, and to some extent we’re augmenting our workforce with agents.

Another common and related question that tends to come up in this discussion: if we agree that lines of code is a poor metric, what should we use instead to measure AI’s impact?

LOC is a noisy metric for a simple reason: A low-effort change can involve many lines of code, and a high-effort change can involve very few lines of code. That was already true before AI, and it becomes even worse with AI-generated code, which tends to produce more lines than a human might. This makes LOC even more inflated and noisy.

For these reasons—both pre- and post-AI—we’ve preferred metrics like PR throughput. It’s still imperfect, but it gives a more normalized view of “change throughput”: how many atomic changes are we pushing through the system? That makes it a less noisy high-level signal than raw LOC.

At DX, we’ve also developed a metric called TrueThroughput, which is a weighted version of PR throughput. It incorporates lines of code as one of several inputs to weight PRs, but LOC is not the sole or primary metric. Even so, all of these metrics are imperfect; you’re really choosing between different degrees of imperfection. In that space, LOC is significantly less useful than PR throughput as a primary signal. This lines up with broader industry experience as well. Many large tech companies that have studied this problem in depth have also converged on some form of change throughput as their preferred signal for tracking impact, rather than relying on raw lines of code.

To summarize my perspective on the correlation vs causation question: a simpler way to frame AI’s ROI is to look at how much it amplifies your existing developers, and how much it augments your organization with agent-driven capacity.

Download and rewatch the full AMA discussion here. My response to this specific question starts at 16:50.

This week’s featured DevProd job openings. See more open roles here.

• Amplitude is hiring an Engineering Manager, Cloud Platform | San Francisco, CA

• BNY is hiring an SVP, Application Development Manager | Pittsburgh, PA

• Figma is hiring a Staff Software Engineer, Developer Experience | Remote; US

• Lob is hiring a Staff Platform Engineer | Remote; US

• Mastercard is hiring a Vice President, Software Engineering | O’Fallon, MO; Boston, MA

• Plaid is hiring a Software Engineer - Platform | New York, NY

• Vercel is hiring a DX Engineer | Hybrid (Austin, New York City, San Francisco)

That’s it for this week. Thanks for reading.

Share

Subscribe now

Last week, we shared data showing that the current productivity gains from AI in engineering are more modest than many headlines imply. To better understand this result, we’ve been interviewing developers across a range of companies to hear what they’re experiencing, and to dig into the reasons behind these more modest gains.

This newsletter summarizes the key themes we’re hearing in those conversations so far, along with direct anonymous quotes from the interviews we performed. This work is ongoing, so if you have something to add, please share your perspective in the comments or on LinkedIn. We’d love to hear from you.

Why are the real‑world gains from AI more modest than many people expect?

Developers we spoke with pointed to several reasons. The most common: coding is a relatively small share of how engineers spend their time, so even meaningful acceleration there has limited impact on overall throughput. Beyond that: new bottlenecks are being created, and organizational and social friction hampers AI’s potential impact.

Here’s what we heard.

1. Coding isn’t the main bottleneck

The most common theme we heard: AI does accelerate developers’ ability to write code, but coding only represents a percentage of the work engineers actually do. Even if AI cut coding time in half, it would only affect a fraction of the total time spent delivering a feature.

• A lot of the slowdown isn’t from our tools, but from either other teams, services or processes.

• Many senior devs say: “writing code is the easy part”... The bottleneck is the human side: alignment, planning, scoping, code reviews, etc.

• AI can help developers write code fast, but writing faster code is not always more productivity. Most developers code only a small proportion of their time. There might be an expectation that this percentage is higher from those who write the headlines, but the reality is that most of the time developers spend is on navigating process and quality, change management, regulation, etc.

• The easy tasks are a little easier. The tedious tasks are a little less annoying. Larger/more complicated tasks — I may be able to shave off hours to a day… but that’s not happening every day, and that doesn’t necessarily mean I ship 3x more PRs. Just means a 4d task can take 3.

2. Partial SDLC automation creates new bottlenecks, so AI gains cap out

AI has sped up code generation, but that has created or exposed bottlenecks downstream, particularly in code review and validation. More code is being produced, but the processes for checking that code haven’t scaled with it, and developers report that the time saved writing is often consumed by the extra scrutiny that AI-generated code requires.

• AI can very significantly speed up initial engineering time, but often that saved time is spent on extended reviews, fact checking or issue remediation, resulting in net-zero productivity gain.

• The opportunities for LLMs to save large amounts of time on new projects is offset by time spent, sometimes ineffectively, attempting to use them on existing projects whose complexity and intricacies lead the LLM to make changes that are at minimum difficult to review even for experienced owners of the code.

• People do not have all parts of SDLC AI-enabled, wherever it does not exist becomes a bottleneck (planning, code review, verification). When people are trying to remove those bottlenecks, there are often lack of guardrails that creates misdirection and churned motions.

3. Social friction is a barrier to adoption

Polarization between pro- and anti-AI engineers, ambiguity about when and how AI should be used, and the absence of peer champions all slow the rate at which teams develop effective workflows with these tools.

• There’s misalignment amongst engineers – the camps are pretty polarized in terms of pro-AI and anti-AI. When anti-AI engineers have status and loud voices it is challenging for everyone to know how to interact with ai tooling.

• AI tooling isn’t discussed enough. Engineers are often unsure whether it’s high status or low status to be talking about using AI, so they just don’t. And people are reluctant to state their true views. If people say what they might really believe, they might seem crazy. In many organizations it is not socially acceptable to suggest that we re-evaluate changes to the code review process for example.

• Having at least one other teammate who is bought in to using AI tools is essential, being an isolated solo-adopter does not allow you to materialize the gains in a meaningful way. Software development is a team sport.

4. Tooling and skill gaps are limiting gains

These two factors are hard to separate: immature tools make the learning curve steeper, and developers who are early in that curve get less out of the tools. Both are limiting gains, though developers who shared this view generally see it as a temporary problem.

On the skill side, using AI effectively is its own discipline. On the tooling side, AI assistants don’t always slot cleanly into existing developer workflows. The integrations that would make them feel native to how teams already work aren’t there yet, and getting agents to operate autonomously on real infrastructure remains an unsolved problem for many.

On skill gaps:

• Someone just starting out with these tools is not as effective as someone who has incorporated them into their workflows for much longer. I have 1000+ hours of purely agentic coding experience and I have so much to learn. Many beginners may have just a few hours or tens of hours. Learning to agentically code is a skill.

• Most of the task isn’t the actual coding, it’s articulating a fuzzy problem in a clear, easily understandable way.

• This is more short term, but the pace of change in the tooling (e.g., the emergence of skills) means I’m spending a fair amount of time getting to know how to use the tools appropriately; which usually means going down various rabbit holes to discover how NOT to use them. That creates some (temporary?) friction that means I might fall back to my own skills when there’s production pressures.

Note: Leaders can distribute DX’s prompting guide and advanced prompting guide internally with developers as a way to help them build a foundational practice with AI.

On tool maturity:

• The tools aren’t mature enough to fit into existing systems cleanly, and most developers haven’t figured out how to use them well yet. Not because they’re bad engineers — the workflow is just genuinely new and nobody’s handed them a playbook.

• The difference between frontier models and models even a few months back can be significant.

• Most developers are used to working inside well-defined tools — your IDE, your terminal, your CI pipeline. Everything has a clear place and a clear trigger. AI assistants don’t work that way… So you end up with higher adoption among developers who naturally gravitate toward that kind of open-ended tooling, and much lower adoption among developers who just want something that fits cleanly into what they’re already doing.

5. Most AI tools lack important context

AI tools perform well on problems that are self-contained and well-documented. But most real engineering work isn’t like that. The context that matters—why a system was designed a certain way, what the implicit rules are, what business constraints apply—typically isn’t written down. It lives in people’s heads. Until that knowledge is made explicit and accessible, AI will keep hitting a ceiling on the kinds of problems it can reliably help with.

• The bigger issue I keep running into is that most codebases and systems aren’t set up for AI to actually help. Not an architecture problem — more that the knowledge of how things work lives in people’s heads. Why this service behaves this way, what the implicit contract between these two systems is, why that design decision got made three years ago — none of that is written down anywhere. An AI assistant can’t reason over a Slack thread from an archived channel or the mental model of the engineer who built it. Until that context gets surfaced into something concrete, you’re always going to hit a ceiling on what the tooling can do.

• AI doesn’t have the context I do on making those decisions, as LLMs are built of massive stats, not direct knowledge of the niche ends of what I’m accomplishing.

• I would (anecdotally) put forth that a lot of my work with AI is trying to get it setup to do the work with proper guardrails. It’s not safe to let run loose on our code-bases or production facing infrastructure as its not deterministic.

• AI models are generally good at understanding what we are trying to achieve, especially when they have access to the workspace or codebase. They can often provide useful solutions for specific use cases, particularly when the problem is related to foundational aspects of a technology or programming language. However, what they often lack is a deeper understanding of the business context and the various factors that influence business logic. That broader context is important to fully understand the bigger picture of a project. In some cases, this limitation can be mitigated by carefully crafting prompts, but when the business logic becomes complex or the prompts become too large for relatively simple tasks, people may step back and revert to more traditional approaches.

Other observations about the current state of AI tools

While speaking with developers, we heard a few other interesting themes about what they’re currently experiencing.

1. Documentation robustness is improving, but its quality is yet to be determined.

• The biggest quality gain I’m seeing is in improved documentation. Developers hate writing docs, so it will always be the last thing they tackle, or the first to get cut... if they tackle it at all. At least for me, I found our documentation getting much more robust thanks to AI being able to take that task off our plate. And it’s not just great for the developers to have access to these docs. Giving the AI agents access to it provides a quicker context boost to know how to tackle the next project.

• Using it to write code documentation leads to terribly incorrect information, an active detriment to future work, creating more work. This is really painful. It’s just not its forte.

2. AI is especially powerful in greenfield projects.

• Could you whip up an app from scratch 3x faster with AI? I would believe that. And if I had to just guess, that’s where I think a lot of the hype comes from. Someone says “look I created this thing from scratch in 1 day!” But, I think it is known that AI isn’t quite as proficient when working on a mature, large codebase...with legacy code and lots of nuance. Which is where most ICs are living in day to day, I think.

• While multi-tasking on my main efforts, I was able to build an entire new microservice from scratch to tests an idea and provide a proof of concept of a new paradigm (this has massive comprehension debt, I haven’t even looked at the code, but the service does what I expect it to).

• It has helped a lot with doing things like fast PoCs to check if something will work.

3. There’s concern that developers are offloading critical thinking.

• I would say I’m losing muscle memory on how to do things (e.g. terminal commands like checking git history for when a file changed, checking what version of a dependency is installed, etc.). I’ve grown dependent on the AI to remember the right commands for me.

• I think if someone blindly uses AI for everything, he/she slowly forgets day to day things.

• AI helps speed up some tasks but may result in folks offloading their thinking. Particularly amongst junior developers, the ‘improved initial speed’ can come at the expense of skill development. Overall speed improvement is negligible.

• I have also found myself having to explain to more junior engineers why what AI said about a problem isn’t the correct answer for us in a particular situation. AI empowers more junior engineers to work longer without asking engineering questions of the more senior folks. That is a sharp double-edged sword that I am still learning how to work with when mentoring people.

• [For me, AI’s] handiness is augmented by me knowing when it’s wrong due to my engineering experience.

Final thoughts

Two takeaways stand out to me from these conversations. First, the gains from AI code assistants are structurally limited by how much of the job is actually coding. Microsoft’s research puts coding at around 16% of a developer’s time. Even dramatic acceleration to code generation can only move the overall needle so much.

Second, it’s the human aspects of the software development lifecycle that are the biggest bottlenecks. Handoffs between teams, the skill required to effectively prompt and manage agents, and the review burden that AI-generated code creates—these are where time is going, and where the next wave of productivity gains may come from.

For leaders, this raises a couple of important questions. Which parts of the SDLC beyond coding are candidates for AI assistance? And are the human conditions for capturing AI’s upside in place?

The 10% figure is not the ceiling. But closing the gap will likely require focusing on the parts of the job that AI hasn’t touched yet.

If you have something to add, please share in the comments or on LinkedIn.

This week’s featured DevProd job openings. See more open roles here.

• Amplitude is hiring an Engineering Manager, Cloud Platform | San Francisco, CA

• BNY is hiring an SVP, Application Development Manager | Pittsburgh, PA

• Figma is hiring a Staff Software Engineer, Developer Experience | Remote; US

• Lob is hiring a Staff Platform Engineer | Remote; US

• Plaid is hiring a Software Engineer - Platform | New York, NY

• Vercel is hiring a DX Engineer | Hybrid (Austin, New York City, San Francisco)

• Zillow is hiring a Senior Product Manager, Developer Experience | Remote; US

That’s it for this week. Thanks for reading.

Share

Subscribe now

🗓 Join Abi on March 19th for a live Q&A session. He will address some of the more pressing questions we’ve received around measuring AI impact, the impact of tool choice, and more. Register here.

Edit: We've updated the data in this post to reflect the latest results from our longitudinal study.

Social media and vendor marketing have set high expectations for AI, suggesting as much as 2-3x productivity gains. But from the data we’re seeing, the reality on the ground is far more modest.

At DX, we’re currently conducting a longitudinal study to measure the long-term impact of AI adoption on key engineering productivity metrics. As part of this study, we analyzed data from a random sample out of 400 companies between November 2024 through February 2026 to track whether teams are shipping more pull requests as AI adoption increases.

We found that, during this time, AI usage increased significantly—by an average 65%. However, PR throughput only increased by 7.76%.

Note: This figure is particularly robust because we’ve filtered out potential gamification effects by excluding teams that set PR throughput targets for individual engineers, which could drive metric inflation rather than genuine output.

What this means for leaders

A ~10% gain is consistent with what we’re hearing from engineering leaders more broadly: most organizations are landing in the 5–15% range. It is a real improvement, but it’s a long way from the 2–3x gains many executives and boards have come to expect. AI is moving the needle, but leaders may need to reset expectations internally.

Why gains aren’t higher

To understand what’s driving this, we spoke with developers across several of these organizations. The explanation we heard most consistently: writing code was never the bottleneck.

As one senior developer put it: “The easy tasks are a little easier. The tedious tasks are a little less annoying. A four-day task might take three. But that doesn’t mean I’m shipping 3x more PRs.”

AI may be accelerating the coding portion of the job. But coding represents a relatively small slice of how engineers actually spend their time. Planning, alignment, scoping, code review, and handoffs—the human parts of the SDLC—remain largely untouched.

What’s next

We’re continuing to investigate the long-term effects of AI in engineering teams. The full study will explore why some teams are capturing more of the upside than others, and what leaders can do to close that gap. More to come.

This week’s featured DevProd job openings. See more open roles here.

• Amplitude is hiring an Engineering Manager, Cloud Platform | San Francisco, CA

• Figma is hiring a Staff Software Engineer, Developer Experience | Remote; US

• Plaid is hiring a Software Engineer - Platform | New York, NY

• UserTesting is hiring a Software Engineer, Developer Experience (Platform) | Spain

• Vercel is hiring a DX Engineer | Hybrid (Austin, New York City, San Francisco)

That’s it for this week. Thanks for reading.

Share

Subscribe now

🗓 Next month, you can join me for a live Q&A session. I’ll address some of the more pressing questions we’ve received around measuring AI impact, the impact of tool choice, and more. Register here.

This week on the Engineering Enablement podcast, we were joined by Uma Namasivayam, Senior Director of Engineering Productivity at Dropbox. Uma’s team owns engineering productivity across Dropbox’s roughly 1,000 engineers: everything from CI/CD systems and telemetry infrastructure to the company’s AI tooling rollout.

During the conversation, Uma shared specifics on how Dropbox drove AI adoption from one-third of engineers to three-quarters, why they chose to deploy multiple AI coding tools rather than starting by standardizing on one, and how they ended up building their own internal AI platform.

Below is a lightly edited excerpt from this conversation. You can listen to the full episode here.

A lot of engineering orgs treat developer productivity as an engineering problem. You have a different take on that.

Uma: I think of productivity at Dropbox—or anywhere, really—as a sociotechnical problem. There absolutely has to be a strong investment in the technology itself: improving the reliability of your systems, improving speed, and reducing friction in your tooling. But then there’s the element of collaboration, of culture, of working with leadership, with developers themselves, and with the people team.

A concrete example: one of the dimensions of developer productivity is deep work—can developers actually code uninterrupted? That’s not necessarily an engineering problem. Working with our chief people officer, we had to literally think about how to restructure meeting times, how to carve out focus blocks for employees. That required a completely different set of partners than fixing a slow CI pipeline. That’s why having a common language across all of those groups, and bringing them together around that language, was so important. You have to attack productivity from multiple different angles.

How did the rollout of AI coding tools intersect with the DevEx work that was already underway?

Uma: I think of them as two parallel work streams that have a lot of overlap in the middle. DevEx is about incremental, systematic friction reduction. It requires aligning with leadership, defining the problems clearly, and making steady progress. AI is about speed. It’s about getting the best tools into developers’ hands quickly, experimenting fast, and staying on the cutting edge.

Before AI can really deliver on its promise, the foundational systems—build and test, telemetry, production observability—have to be in a strong place. Developers need to trust that when they push code through an AI-assisted workflow, the quality guardrails are actually there. Without that trust, you can’t go really hard on AI.

How did you actually get adoption moving? What drove the initial uptake?

Uma: Early on, roughly one-third of our engineers were using AI tools organically, so people were just finding tools they liked on their own. That’s a decent starting point, but it’s not a strategy. The real inflection point came when our exec team made AI a clear company priority. Within about three months of that top-down signal, we got to around three-quarters of engineers using tools on a weekly basis.

With that said, top-down mandates only take you so far. After that initial push, we had to go deeper, starting by looking at what was actually blocking adoption in different parts of the engineering population, and addressing those pockets specifically. That’s where a product mindset comes in: understanding your customers’ actual pain points rather than assuming one solution works for everyone.

You offer developers multiple AI coding tools rather than standardizing on one. What’s the thinking there?

Uma: It comes back to treating this like a product problem. Different teams have genuinely different needs. Our mobile developers, for example, couldn’t use the tools that work well for other parts of the codebase. We had to find something specific to their use case. So we deliberately chose not to be a single-tool shop.

We also learned quickly that some tools that work at a smaller scale fall apart at Dropbox’s scale. We were piloting an AI code review tool and it just didn’t hold up. That pushed us toward building some of this in-house. We now have an internal platform that handles the backend complexity specific to Dropbox’s monorepo, and that other teams can build on. The build-versus-buy decision turns out to be really critical when you’re operating at this scale and at this speed.

One practical thing that also helped: we worked with our procurement and legal and security teams to dramatically reduce the time it takes to evaluate and approve new AI tools, getting that review process down to around three days. When the market is moving this fast, your ability to experiment is only as good as how quickly you can get tools in front of developers.

Once you’ve expanded beyond code completion, what does AI usage actually look like across the SDLC?

Uma: Code completion was the obvious starting point. Once we felt we’d gotten what we could from that, we started looking at the rest of the development lifecycle: code review, testing, and debugging. Every stage of the SDLC is on the table.

What we discovered is that a lot of the commercially available tools for these adjacent use cases didn’t hold up at our scale or for our specific codebase. That’s what led us to build in-house. One of our developers took it upon himself to figure out what kind of platform we could build, using Claude and Claude Code as the foundation, that would work within Dropbox’s environment and that others could build on top of. That platform now handles the backend complexity: deployment, monorepo scale, and testing guardrails. If a team wants to build an AI-assisted code review product, they start from that platform rather than from scratch. It’s one of the things I’m most proud of from the past year.

What’s the hardest unsolved problem you’re carrying into 2026?

Uma: Connecting developer productivity improvements to actual business outcomes. We can show that DXI improved, that AI adoption is up, and that developers are saving hours. But the arc from “developers are more productive” to “we shipped more value to customers faster”—that instrumentation isn’t there yet, and I don’t think anyone in the industry has fully cracked it.

What we’re seeing is that the capacity unlocked by AI is naturally flowing toward migrations and tech debt reduction. That’s actually pretty cool—give engineers more capacity, and they automatically invest it in the right things. But as a leader, I need to be able to answer the CFO and the exec team: where is this capacity going, and how does it connect to revenue? We’re working toward that. If anyone listening has cracked that code, I’d genuinely love to talk.

You can listen to the full conversation with Uma on the Engineering Enablement podcast.

This week’s featured DevProd job openings. See more open roles here.

• American Express is hiring a Sr. Manager, Digital Product Management - DevProd | Hybrid - London UK

• CoreWeave is hiring a Sr. Software Engineer - Developer Experience | Livingston NJ; New York, NY

• DoorDash is hiring an Engineering Manager - Developer Experience | San Francisco, CA; Sunnyvale, CA; Seattle, WA; Los Angeles, CA; New York, New York

• Figma is hiring a Staff Software Engineer, Developer Experience | Remote; US

• Plaid is hiring a Software Engineer - Platform | New York, NY

• UserTesting is hiring a Software Engineer, Developer Experience (Platform) | Spain

That’s it for this week. Thanks for reading.

Share

Subscribe now

🗓 Next month, you can join us for a live Q&A session with DX CEO Abi Noda. He’ll address recent questions around measuring AI impact, the impact of tool choice, and more. Register to join here.

In just the past year, AI has set into motion several major shifts in how engineering organizations are structured, including how teams are composed, where responsibilities sit, and what skills matter. These changes are still unfolding, but patterns are emerging.

To help name some of those shifts, I interviewed two analysts who spend their time talking with engineering leaders across industries: Ruchi Goyal, Global AI Practice Leader at Accenture, and Dr. Kate Holterhoff, Senior Industry Analyst at RedMonk. Below I’ve summarized the observations they shared—with the hope that naming these patterns is a useful reference point for leaders to recognize and navigate the same changes in their own organizations.

Hiring priorities are emphasizing AI fluency and code quality

The software industry is still working through a hiring slump, but the vacuum is being filled by specific AI-centric roles.

• The “AI Engineer” trend: We are seeing a surge in “AI Engineer” titles on LinkedIn. These are roles that are less about training foundational models and more about building the glue between LLMs and production code. (See example roles on Indeed here).

• Quality in focus: At the same time, a new skill is becoming central to engineering hiring: the ability to distinguish good code from mediocre AI-generated code. As Ryan Dahl and others have noted, AI removes the barrier to writing code, but it introduces a new one: dealing with slop.

• Because a focus on quality has become more important, some organizations are seeing friction emerge in the code review process.

Fragmented AI experiments are consolidating into centralized platforms

Early AI adoption in most organizations was scattered, with individual teams running their own experiments with different tools and no shared standards. That’s changing. Enterprises are moving toward centralized models such as a formal AI Center of Excellence or a hub-and-spoke structure that sets organization-wide direction. Goyal notes that “a lot of clients are looking into merging Developer Productivity and Internal Tools teams into one.” This can enable a number of new strategies for the organization:

• Consolidation: Organizations are folding AI initiatives into their internal developer platforms, creating governed paths for how AI gets used across the organization. Without that structure, teams may end up making independent decisions about tools and access, which can be hard to unwind later.

• Orchestration: Managing individual agents is becoming less of the challenge, and coordinating multiple agents working together is where the complexity now lives. Organizations that have already worked through DevOps maturity will recognize the pattern: the evolution tends to move from automation, to orchestration, to choreography. AI adoption is following a similar arc.

• AI is spreading across engineering organizations to the point where most developers are expected to use it as a matter of course. As Holterhoff put it, “AI is becoming water.” As that happens, Platform teams are becoming increasingly responsible for the governance and prompt hardening that makes this safe.

A new operational layer is emerging: LLMOps

As AI becomes part of the engineering infrastructure, someone has to own the operational layer that keeps it running reliably. In many organizations, that responsibility is coalescing into what’s being called LLMOps. Many Platform and DevProd teams are absorbing this responsibility.

• Prompt engineering to guideposts: Ad hoc prompting works for individuals but doesn’t scale across a team or organization. Turning prompts into reusable templates means you’re building institutional knowledge rather than leaving every developer to figure it out independently, and it creates a consistent quality baseline across the SDLC.

• The SWAT Team approach: In many orgs, a specialized team of engineers handles model integration and AI infra, often leveraging the heavy lifting provided by big cloud providers. Organizations that stand up a dedicated group for this work move faster and make fewer costly mistakes, while the rest of the engineering org can focus on building.

• Deployment convergence: We are seeing tighter integration between development environments and production. (For example, Expo’s integration with Replit allows developers to deploy directly to production, bypassing traditional friction points.) This creates real efficiency gains, but it also means the guardrails that used to exist between those environments need to be rethought, which is something Platform teams are becoming increasingly responsible for.

The human factor still matters the most

Despite the automation, the primary challenges are still cultural. As RedMonk has famously noted, “The developers are everything.” AI tools are being designed to support practitioners, not replace them.

• Murky ROI: Individual developers see the value of AI tools clearly enough that many are willing to pay for them personally. But at the organizational level, the picture is murkier. Leaders are measuring the impact of AI while reading headlines claiming 3x productivity gains… and most aren’t seeing numbers that match.

• Security as a skill: A major skill gap isn’t learning to prompt; it’s learning how to use AI securely. Training now focuses heavily on privacy, vetting specific LLMs, and ensuring that agentic workflows don’t bypass critical security guardrails.

• Software engineers as self-optimizers: Developers are naturally inclined to try new tools. That’s generally a good thing, but it means organizations need vetted, safe environments for that experimentation to happen in. The risk is that developers could adopt AI in ways the organization can’t see or govern.

These changes are moving quickly, and most organizations are still early in figuring out what they mean in practice. There’s no single right answer yet, which is part of what makes it valuable to hear from analysts with visibility across many organizations. A special thanks to Ruchi Goyal and Dr. Kate Holterhoff for sharing their perspective for this newsletter.

This week’s featured DevProd job openings. See more open roles here.

• American Express is hiring a Sr. Manager, Digital Product Management - DevProd | Hybrid - London UK

• CoreWeave is hiring a Sr. Software Engineer - Developer Experience | Livingston NJ; New York, NY

• DoorDash is hiring an Engineering Manager - Developer Experience | San Francisco, CA; Sunnyvale, CA; Seattle, WA; Los Angeles, CA; New York, New York

• Figma is hiring a Staff Software Engineer, Developer Experience | Remote; US

• Plaid is hiring a Software Engineer - Platform | New York, NY

• UserTesting is hiring a Software Engineer, Developer Experience (Platform) | Spain

That’s it for this week. Thanks for reading.

Share

Developer productivity is often treated as a tooling problem or a sentiment problem. In reality, it’s neither. It’s a socio-technical systems problem that spans engineering foundations, leadership alignment, organizational design, and culture.

In this episode, I’m joined by Uma Namasivayam, Senior Director, Engineering Productivity at Dropbox, to explore how Dropbox approaches developer experience at scale. We talk about why productivity needs to be framed as a business problem, how executive alignment creates the conditions for meaningful change, and what it takes to treat developer experience as a real product with developers as customers.

We also dig into Dropbox’s approach to AI adoption. Uma shares why strong foundations, such as build, test, and observability, are prerequisites for AI to actually accelerate work, how Dropbox encourages daily AI use without mandating a single tool, and where build-versus-buy decisions break down at scale.

We close with an honest look at what remains unsolved: how to connect gains in developer productivity and AI-driven capacity to real business outcomes, and where engineering leaders should focus next in 2026.

Some takeaways:

Developer productivity is a socio-technical problem

• Productivity cannot be solved through tooling alone; it spans engineering systems, leadership behavior, organizational structure, and people practices.

• Problems like build and test are engineering problems, while problems like focus time and interruptions are people problems, and both matter equally.

• Treating productivity as a system forces tradeoffs to be explicit, rather than hidden inside isolated tooling initiatives.

Executive alignment matters more than any single metric

• Top-down sponsorship creates permission to act, especially when productivity work cuts across org boundaries.

• A shared framework creates alignment, not answers; its value is giving leaders and engineers a common language.

• System metrics matter more than single metrics, because productivity improvements rarely move one dimension in isolation.

• Distributed accountability makes productivity a company problem, not a developer experience team problem.

Developer experience works best when treated as a product discipline

• Developers are customers, and their experience must be understood through both qualitative feedback and quantitative signals.

• Good system metrics do not guarantee good developer experience, which is why sentiment and perception matter.

• DX surveys surface where systems break differently for different teams, such as desktop, mobile, and web developers.

• Continuous feedback loops are essential, combining surveys, direct conversations, and usage data.

• Internal communication is part of the product, reinforcing to developers that their feedback leads to real change.

Prioritization requires structure, not intuition

• Finite capacity makes prioritization unavoidable, even in large, well-resourced engineering orgs.

• Segmenting developer populations clarifies tradeoffs, since different teams experience different bottlenecks.

• DX survey data provides a defensible starting point, but prioritization still requires judgment.

• Leadership-level stack ranking helps resolve conflicts, especially when multiple teams compete for attention.

• Frameworks make hard decisions easier to explain, even when they do not make them easy.

AI and developer experience must advance in parallel

• AI accelerates work, while developer experience reduces friction, and both are required for sustained gains.

• Foundational systems act as plumbing, enabling trust in speed, quality, and safety.

• Without strong CI, testing, and observability, faster code creation increases risk instead of value.

• Trust in guardrails enables confidence in AI-assisted development, especially at scale.

AI adoption succeeds through choice, not mandates

• Early organic adoption revealed real developer needs, rather than forcing a single tool.

• Different teams require different AI tools, particularly for mobile, desktop, and large-repo workflows.

• Supporting multiple tools increased adoption, rather than reducing it.

• Daily use depends on fitting AI into existing workflows, not adding extra steps.

• Habits matter more than access, which is why SDLC-level integration is critical.

Build vs. buy decisions change at scale

• Many AI tools fail when tested at large-company scale, despite working well in smaller contexts.

• Cost and performance become gating factors, not feature completeness.

• Internal platforms can abstract complexity, enabling teams to build AI workflows safely and consistently.

• Shared internal platforms unlock reuse, allowing teams to innovate without rebuilding infrastructure.

• Speed of iteration remains the primary differentiator, even when building in-house.

In this episode, we cover:

(00:00) Intro

(00:45) Dropbox’s engineering org

(01:59) Why developer productivity is a business problem

(04:08) The role of executive sponsorship in developer productivity

(06:02) How DX’s Core Four framework created a shared language

(08:13) Treating developer experience as a product

(11:30) How Dropbox prioritizes developer experience work

(14:20) The challenge of tying developer experience to business outcomes

(16:38) How AI and developer experience intersect at Dropbox

(18:35) The prerequisites for AI adoption to accelerate work

(20:26) How Dropbox encourages daily AI use

(23:12) AI use beyond code completion

(25:00) Managing AI tool demand at scale

(27:56) Early results from Dropbox’s AI efforts

(30:05) Progress on developer experience at Dropbox

(32:55) Advice for organizations investing in developer experience

(34:25) Capacity tradeoffs for developer experience

(35:59) The unanswered questions around AI and capacity in 2026

Referenced:

• DX Core 4 Productivity Framework

• Dropbox.com

Subscribe now

In 2025, we saw engineering leaders focus on rolling out AI coding assistants at scale across their organizations. As these tools became more widely used, it became clear that outcomes depended less on having access to AI and more on how teams were educated and enabled. In response, DX published the Guide to AI Assisted Engineering, outlining best practices and high-value prompting use cases to help engineering teams use AI effectively in their day-to-day work.

Now, as organizations move beyond pilots, the focus has shifted from adoption and enablement to operational improvements and more complex use cases. Successfully applying AI in these contexts requires more structured prompting practices than those used during early experimentation. To support that next step, we’ve created our first supplement to the original guide: Advanced Prompting Guide for AI Engineering.

This new guide follows the same format as the original, with clear Do and Don’t scenarios, full prompt examples, and code output examples. It is vendor-agnostic, with an emphasis on prompting structure, constraints, and context so the techniques can be applied across tools and architectures.

Inside, you’ll find prompt and code examples that focus on:

• Complexity management - For systems with cascading rules or conflicting requirements, the guide demonstrates graph-based prompting to reveal hidden dependencies, prioritize rules, and deal with changing state

• Governance and quality - Workflows that execute controlled validation loops, which result in higher accuracy, and can deal with more edge cases

• Risk mitigation - Dual-implementation strategies can yield more bulletproof outcomes, especially when dealing with critical transactions that require 100% accuracy

• Operational efficiency - Techniques like diff-only refactoring can reduce invasive changes to large, complex code repositories, as well as reduce tokens

These use cases are drawn from interviews, educational talks, and community interaction, and can be deployed across multiple scenarios. Whether you’re using coding assistants, building prompts for agents, or writing specs for spec-driven-development, you’ll find applicable methods in the guide.

One other important update: When the original guide was published, it was written primarily for developers. But 2025 was a pivotal year for elevating traditional non-builders. As highlighted in our Q4 AI Impact Report, engineering leaders are shipping more code, and designers and PMs have the ability to create deeper designs and prototypes. We still encourage engineering leaders to distribute this guide to their engineering teams, but this guide need not be exclusive to engineers. Whether engineer, designer, PM, or leader, if you are working on complex problems, this guide can provide useful perspectives.

Download the guide

This week’s featured DevProd job openings. See more open roles here.

• American Express is hiring a Sr. Manager, Digital Product Management - DevProd | Hybrid - London UK

• CoreWeave is hiring a Sr. Software Engineer - Developer Experience | Livingston NJ; New York, NY

• DoorDash is hiring an Engineering Manager - Developer Experience | San Francisco, CA; Sunnyvale, CA; Seattle, WA; Los Angeles, CA; New York, New York

• Experian is hiring a Software Engineering Manager - Security Platform | Remote

• Gusto is hiring a Sr. Platform Engineer | Denver, CO; San Francisco, CA; Atlanta, GA; Austin, TX; Chicago, IL; Miami, FL; Seattle, WA

• Notion is hiring a Senior Platform Engineer | Dubai, United Arab Emirates

• Plaid is hiring a Software Engineer - Platform | New York, NY

That’s it for this week. Thanks for reading.

Share

Subscribe now

🗓️ We recently announced DX Annual, our flagship conference for developer productivity leaders navigating the AI era. Go here to learn about the event and request an invite to attend.

One of the insights from the AI impact report released last quarter was that newer AI-native tools appeared to be outperforming others (see page 14). Even in side-by-side deployments, AI-native tools like agentic IDEs were associated with higher throughput compared to older or less specialized solutions.

Given how quickly the space is evolving, we decided to revisit this data to see whether those patterns still hold. In this latest analysis, we looked at PR throughput and median adoption rate across a slightly larger sample of companies (219, versus the 170 we examined last quarter). Here’s an updated look at what we’re seeing:

Comparing our latest Q1 2026 data to our Q4 2025 report we observe a performance jump consistently across all tools. This indicates that trends from last quarter’s data are continuing, and that stronger performance is being realized as tool maturity and adoption practice improves.

It’s notable that the throughput rankings themselves have not changed over the last three months, with newer, agentic tools still ranking highest in terms of velocity impact. Regardless of overall rank, we observed velocity increases across every tool, with a few major shifts since Q4 2025:

• Cursor has seen the largest increase in PR throughput. In Q4 2025, daily users merged a median of 2.8 PRs. Today, that number has jumped to 4.1, representing a 46% increase in throughput for frequent users.

• Claude leads for weekly and monthly users, with PR throughput exceeding 4.0. This is a significant move from the 2.6 (weekly) and 2.2 (monthly) reported just one quarter ago.

• GitHub Copilot daily users have also accelerated, moving from 2.5 PRs/week in Q4 to 3.61 today.

• Tabnine continues to show the lowest throughput at 1.83 for daily users. As noted in the previous report, this is likely because Tabnine is common in large enterprises where PR throughput is lower overall due to organizational complexity.

The last quarter of 2025 has been a landmark period for AI, with broader industry education and best practice converging with improved models, like Opus 4.5 and GPT-5.2, and better tool workflows such as those found in Cursor’s Agent Mode and Claude Code’s spec-driven focus.

Despite rapid advancement and innovation, many unsolved problems remain. Limitations such as memory and context constraints provide technical hurdles, while lack of organizational alignment on policies and practices continue to drive cultural challenges. With all this opportunity for improvement, I expect to see deliberate advancements continue, and productivity gains rise even further in the coming months.

While we were analyzing PR throughput data associated with different AI tools, we also looked at which tools are becoming daily drivers. (Note: adoption metrics don’t directly reveal productivity impact, as is mentioned in the DX AI Measurement Framework. But adoption metrics can give us a signal into which tools are being used for daily tasks versus those used only for specialized, less regular tasks.)

• GitHub Copilot remains the leader in overall stickiness, with the highest daily adoption rate at 9.76%. Its integration into the existing workflow may be helping it become a constant companion for developers. Additionally, for many businesses, Copilot is easy to procure since an existing purchasing agreement with Microsoft is likely to already be in place.

• Cursor is rapidly becoming a primary workspace, boasting an impressive 31.56% adoption rate among weekly users.

• Windsurf and Tabnine show a longer-tail adoption pattern. Windsurf, in particular, has the highest monthly adoption at 35.87%, suggesting it is being used as a powerful specialized tool for specific complex tasks rather than a total IDE replacement for most.

Summary

The data provides evidence for the belief that organizations will realize greater productivity gains as developers become more familiar with AI tools. Further, the significant jump we see in throughput shows how quickly the space is advancing. It’ll be fascinating to watch and see whether we see even higher throughput numbers next quarter, or gains will start to level over time.

I’ll conclude with a few reminders on how to use this data:

1. Correlate usage with throughput: Don’t just measure seat count. Ensure that your highest-cost tools are being used by your Daily or Weekly cohorts where the throughput gains are most visible.

2. Avoid vendor lock-in: Because the performance of these tools is evolving so quickly, we recommend a multi-vendor approach. Different tools excel at different frequencies (e.g., Copilot for daily assistance, Claude for weekly deep-dives). See our webinar on data-driven evaluations of AI tools for more information.

3. Measure to confirm ROI: Use these benchmarks to establish a baseline for your own organization. While AI metrics tell you what is happening, your core metrics, like PR throughput, confirm whether the investment is actually working.

This week’s featured DevProd job openings. See more open roles here.

• American Express is hiring a Sr. Manager, Digital Product Management - DevProd | Hybrid - London UK

• CoreWeave is hiring a Sr. Software Engineer - Developer Experience | Livingston NJ; New York, NY

• DoorDash is hiring an Engineering Manager - Developer Experience | San Francisco, CA; Sunnyvale, CA; Seattle, WA; Los Angeles, CA; New York, New York

• Experian is hiring a Software Engineering Manager - Security Platform | Remote

• Gusto is hiring a Sr. Platform Engineer | Denver, CO; San Francisco, CA; Atlanta, GA; Austin, TX; Chicago, IL; Miami, FL; Seattle, WA

• Notion is hiring a Senior Platform Engineer | Dubai, United Arab Emirates

• Plaid is hiring a Software Engineer - Platform | New York, NY

That’s it for this week. Thanks for reading.

Share

Subscribe now

🗓️ In case you missed it: we recently announced DX Annual, our flagship conference for developer productivity leaders navigating the AI era. Go here to learn more and request an invite to attend.

Business leaders understand the ability of centralized developer productivity and internal platform teams to promote better throughput and faster time to market for software features. But, how large should these teams be? What’s the best proportion of developer productivity specialists to other engineers within the company?

To find an answer, our team analyzed data from 39 companies with varying engineering team sizes across a number of industry verticals and looked for trends in organizational composition for teams explicitly dedicated to developer productivity and experience. We found that engineering leaders typically dedicate 2% to 6% of total headcount to centralized developer productivity. However, this ratio is not linear: as organizations scale beyond 1,000 engineers, the percentage of dedicated headcount steadily decreases due to tooling leverage and automation.

I’m excited to continue following these trends as AI efforts become more centralized and embedded in developer productivity teams, as we may see it lead to further hiring and expansion of responsibilities.

Defining the developer productivity umbrella

To ensure these benchmarks weren’t artificially inflated, we used a narrow definition of what counts as a team focused strictly on developer productivity. We only looked at teams that are unambiguously dedicated to internal platforms, developer productivity, and developer experience, and didn’t include teams focused on SRE or infrastructure, for example.

Specifically, we included the following teams in our analysis:

• Developer Experience and Productivity: Teams explicitly named “Developer Experience,” “DevEx,” “Developer Productivity,” “Developer Enablement,” or “Engineering Enablement”

• Engineering Effectiveness: Teams focused on “Engineering Productivity,” “Engineering Effectiveness,” or “Engineering Excellence”

• Internal Developer Platform Engineers: Groups building “Internal Developer Platforms”, “Internal Developer Portals,” “Developer Frameworks,” “Documentation Systems,” “Service Catalogs” or “Developer Tools”

• Build & Release Infrastructure: “CI Infrastructure,” “CI Platform,” “Build Infrastructure,” “Build Systems,” “Core Automation Platforms” focused on CI/CD, and “Release Engineering” teams

• Developer Education and Support: “Developer Education,” “Engineering Support Organization,” and developer onboarding teams

• Test Infrastructure: Teams dedicated to “Test Infrastructure” and “Test Frameworks” (not QA or product testing)

We excluded several categories that often get conflated with developer tooling, such as:

• General Cloud Infrastructure: Cloud operations, data center management, storage infrastructure, and network infrastructure teams

• Business Platform Teams: “Data Platform,” “ML Platform,” “Analytics Platform,” “Payment Platform,” “Content Platform,” and other product or customer-facing platforms

• Site Reliability Engineering (SRE): While SRE teams are critical, they focus on production reliability rather than developer tooling (except when explicitly named as developer-facing)

• Security Infrastructure: Security operations, compliance, and infrastructure security teams

• Product Infrastructure: Teams building infrastructure for customer-facing products rather than internal tools

• Lab Infrastructure: Physical and virtual infrastructure for product testing and validation

Most companies dedicate 4.7% of engineering headcount to developer productivity

In looking at our first sample of teams that have less than 1,000 engineers, we found that most teams dedicate between 2-6% of their overall engineering headcount to centralized developer productivity functions, with an average of 4.7% across the board. Some range past 8% on the high end, and others range below 2%.

For the average team then, per engineer, that means roughly one productivity engineer per seventeen developers at the high range, or one for every fifty engineers at the lower range. The distribution is shown in the chart below.

These numbers are not surprising, given the variance between company cultures, repository/build architectures, and general attitudes towards productivity. DORA’s State of DevOps reports suggest 10-20% investment between centralized teams and embedded productivity work – our study focuses on centralized, named roles. In general, we’ll find higher investment amongst engineering teams with more complicated deployments (microservices, polyglot stacks, monorepo), and lower investment amongst teams that focus mostly on CI/CD with more monolithic architectures.

When we looked at headcount ratios, we also analyzed team structure: even if two companies have 20 productivity engineers, they may organize them differently. Most organizations distribute their developer productivity team members across 2 to 6 distinct teams. However, some companies take a specialist model, spinning up as many as 15 specialized teams to tackle narrow problems like test optimization or internal portals, while others prefer a single, centralized developer experience or developer productivity unit that handles all enablement tasks under one roof.

Benchmarks by company size and industry vertical

While the 2–6% range is a benchmark, the data shows that investment doesn’t scale linearly. As engineering organizations grow beyond 1,000 people, the percentage of dedicated DevProd headcount starts to drop. This is likely due to diminishing returns in investment, as teams grow and scale, single developer productivity resources can take advantage of more automation, knowledge sharing, tooling, and teamwide collaboration to reduce the need for headcount.

We also saw clear differences in how different industries value DevProd investment:

• Technology (4.89%): These companies lead the pack, averaging about one productivity specialist for every 20 engineers. Because software is their primary profit center, shipping features faster directly impacts their bottom line.

• Fintech and Financial Services (4.36%): These organizations follow closely behind as they continue to modernize their infrastructure and prioritize engineering speed.

• Retail (3.8%) and Large Enterprise (3.32%): These sectors typically show lower ratios, though the latter (large enterprises) may have more to do with the diminishing returns noted in the trends of organizations with 1000+ engineers.

How to use these numbers

These numbers shouldn’t be used as a strict quota, but as a guide for your organizational strategy. The “right” team size is one that balances these industry averages with the specific bottlenecks your developers are facing.

If you are building a new “center of excellence,” use these ratios to set your initial headcount. To ensure you’re getting a real return on that investment, combine these benchmarks with a measurement framework like the Core 4 to track whether your new team is actually removing friction and improving the developer’s day.

This week’s featured DevProd job openings. See more open roles here.

• American Express is hiring a Sr. Manager, Digital Product Management - DevProd | Hybrid - London UK

• Capital One is hiring a Product Manager - Developer Experience | Plano TX; McLean VA; Richmond VA

• Gusto is hiring a Sr. Platform Engineer | Denver, CO; San Francisco, CA; Atlanta, GA; Austin, TX; Chicago, IL; Miami, FL; Seattle, WA

• Plaid is hiring a Software Engineer - Platform | New York, NY

• Reddit: Staff Software Engineer - Developer Experience | Remote - United States

That’s it for this week. Thanks for reading.

Share

As we close out 2025, I wanted to step back and take stock of what we have actually learned about AI adoption in engineering organizations. Not just where usage has increased, but where impact is real, where it is overstated, and what questions remain unanswered.

In this special year-end episode, I’m joined by Brian Houck from Microsoft, Collin Green and Ciera Jaspan from Google, and Eirini Kalliamvakou from GitHub. Together, we unpack the research each of them worked on this year and explore how leading organizations are thinking about AI measurement, developer experience, and long-term productivity. We talk candidly about why measuring AI’s impact is so difficult, why familiar metrics like lines of code keep resurfacing despite their flaws, and how multidimensional approaches like SPACE and DORA offer a more realistic lens.

We also look ahead to 2026. We discuss how AI is beginning to reshape the identity of the developer, how junior engineers’ skill sets may evolve, where agentic workflows are gaining traction, and why some of the most widely shared AI studies were misunderstood. This episode is an honest conversation about moving past hype and toward a more grounded, evidence-based approach to AI adoption in engineering teams.

Some takeaways:

Measuring AI impact requires multiple lenses

• There is no single metric that can capture AI’s impact. Developer productivity and experience are inherently multidimensional, requiring trade-offs to be evaluated across speed, quality, collaboration, and meaning.

• Frameworks like SPACE and DORA help avoid metric tunnel vision. They encourage teams to examine complementary signals rather than optimizing one dimension at the expense of others.

• Measurement must reflect systems, not tools. AI does not operate in isolation; its impact depends on organizational context, workflows, and existing engineering practices.

Why familiar metrics keep failing us

• Lines of code remains a deeply misleading metric. AI tends to generate verbose code, making raw output a poor proxy for productivity, quality, or long-term maintainability.

• More code does not equal better outcomes. Excess code can increase maintenance burden, technical debt, and cognitive load over time.

• Easy-to-measure metrics are often the most dangerous. Their simplicity makes them attractive during periods of uncertainty, even when they obscure what is actually changing.

The limits of tracking AI-generated code

• Measuring the percentage of AI-generated code oversimplifies reality. AI may write, delete, refactor, or reorganize code in ways that raw percentages fail to capture.
  AI-generated code does not inherently signal higher risk. In some contexts, AI output may be more consistent or higher quality than human-written code.

• These metrics are better used as supporting signals, not goals. They can inform budgeting, experimentation, or adoption patterns but should not drive performance targets.

How AI is reshaping the role of the developer

• Developers are shifting from implementers to orchestrators. Advanced AI users spend more time framing problems, setting context, and validating outcomes than writing raw code.

• AI fluency is becoming a core skill. Knowing how to guide, correct, and collaborate with agents is increasingly important.

• Adoption follows a progression. Developers tend to move from skepticism to exploration, collaboration, and eventually strategic use as expectations recalibrate.

What this means for junior engineers

• Skill development may accelerate rather than disappear. Junior engineers may practice delegation, planning, and system-level thinking earlier by working with AI agents.

• Technical fundamentals still matter. Understanding architecture, requirements, and failure modes remains essential for supervising AI-generated work.

• Interpersonal skills risk being deprioritized. Managing agents is not the same as managing people, raising concerns about how collaboration skills develop over time.

AI is not just a productivity tool

• Creativity and innovation benefit from friction. Research suggests that exposing decision points and seams can create space for new ideas rather than faster repetition.

• Automating everything is not always desirable. Removing all toil may reduce opportunities for learning, insight, and creative problem-solving.

• AI should augment thinking, not replace it. Tools that surface trade-offs and choices can support better outcomes than those that simply eliminate effort.

High-leverage AI use cases focus on toil

• Developers spend only about 14% of their time writing code. Optimizing coding alone rarely leads to large productivity gains.

• The biggest opportunities lie in removing friction. Documentation, compliance tasks, incident response, flaky tests, and knowledge discovery consistently rank as top pain points.

• AI excels at work developers dislike but must still do. Automating dull, repetitive tasks can improve satisfaction and free time for meaningful work.

Why leadership and change management matter

• AI adoption is a human problem before it is a technical one. Organizations that understand developer pain points deploy AI more effectively.
  Agentic workflows amplify organizational differences. Teams with strong experimentation cultures and feedback loops move faster and with less friction.

• Culture determines outcomes. How leaders communicate expectations, normalize experimentation, and support learning shapes whether AI adoption succeeds or stalls.

Looking ahead to 2026

• Task parallelization is an emerging frontier. Developers are beginning to use agents to explore multiple solution paths simultaneously.

• Collaboration with agents will redefine productivity. Teams, not just individuals, will increasingly work alongside AI systems.

• Research must evolve with the work itself. New workflows will require new metrics, new telemetry, and new ways of understanding impact.

Lessons from the METR paper

• Context matters more than headlines suggest. Results showing slower performance often reflected expert developers working in familiar codebases.

• AI may help most where familiarity is lowest. New domains, unfamiliar systems, and onboarding scenarios show different outcomes.

• Media oversimplification distorts understanding. Nuance is critical when interpreting AI research, especially as studies move into real-world environments.

In this episode, we cover:

(00:00) Intro

(02:35) Introducing the panel and the focus of the discussion

(04:43) Why measuring AI’s impact is such a hard problem

(05:30) How Microsoft approaches AI impact measurement

(06:40) How Google thinks about measuring AI impact

(07:28) GitHub’s perspective on measurement and insights from the DORA report

(10:35) Why lines of code is a misleading metric

(14:27) The limitations of measuring the percentage of code generated by AI

(18:24) GitHub’s research on how AI is shaping the identity of the developer

(21:39) How AI may change junior engineers’ skill sets

(24:42) Google’s research on using AI and creativity

(26:24) High-leverage AI use cases that improve developer experience

(32:38) Open research questions for AI and developer productivity in 2026

(35:33) How leading organizations approach change and agentic workflows

(38:02) Why the METR paper resonated and how it was misunderstood

Referenced:

• Measuring AI code assistants and agents

• Kiro

• Claude Code - AI coding agent for terminal & IDE

• SPACE framework: a quick primer

• DORA | State of AI-assisted Software Development 2025

• Martin Fowler - by Gergely Orosz - The Pragmatic Engineer

• Seamful AI for Creative Software Engineering: Use in Software Development Workflows | IEEE Journals & Magazine | IEEE Xplore

• AI Where It Matters: Where, Why, and How Developers Want AI Support in Daily Work - Microsoft Research

• Unpacking METR’s findings: Does AI slow developers down?

• DX Annual 2026

Subscribe now

🗓️ We recently announced DX Annual, our flagship conference for developer productivity leaders navigating the AI era. Go here to learn about the event and request an invite to attend.

This week, I’m summarizing “The GenAI Divide: State of AI in Business in 2025,” a report that MIT’s Project NANDA (Networked Agents and Decentralized AI) produced earlier this year. The report examines why, despite massive investment in AI, most organizations are seeing little to no business impact. Drawing on interviews, surveys, and analysis of hundreds of AI initiatives, the authors argue that enterprise AI outcomes have split into two camps: a small minority extracting real value, and the rest stuck in pilots. They call this gap the GenAI Divide.

For DevProd and Platform leaders, this report provides a useful lens for understanding why AI initiatives stall inside engineering organizations. It also counters the “AI will replace engineers” narrative.

My summary of the paper

Headlines often suggest that engineers will soon be replaced by AI, or that the software industry is on the brink of collapse. While AI tools are improving quickly and demos are impressive, Project NANDA set out to study how organizations are actually using these systems in practice. As the authors put it, the GenAI Divide reflects “high adoption but low transformation,” with 95% of organizations seeing zero measurable return.

The researchers conducted 53 structured interviews and collected survey responses from 153 senior leaders. The observation window covered roughly six months following initial AI pilots, and the authors are careful to describe the findings as a directionally accurate snapshot rather than a definitive market analysis.

Here are the key findings from the report.

The pilot-to-production gap

The most visible expression of the AI Divide is the steep drop-off from experimentation to real deployment. While many organizations evaluate AI tools, very few succeed in moving them into production. Roughly 60% evaluated task-specific or enterprise AI systems, about 20% reached pilot stage, and just 5% deployed systems that delivered sustained productivity or P&L impact.

Large enterprises struggled most with this transition. They led in pilot volume and investment, but had the lowest rates of successful scale-up. Mid-market organizations moved faster and more decisively, often reaching full implementation within 90 days. Across successful cases, the difference was ultimately about focus:

• Successful teams focused on narrow, workflow-specific use cases with clearly defined operational outcomes, rather than broad or generalized AI deployments.

• Implementation ownership sat with domain leaders and frontline managers, not centralized AI labs or exploratory teams, enabling faster decision-making and clearer accountability.

• AI systems were embedded directly into existing tools, processes, and data flows, reducing context switching and allowing them to operate within day-to-day work rather than alongside it.

Buy beats build

Organizations that crossed the AI Divide approached implementation differently. Rather than building AI systems entirely in-house, they were far more likely to partner with external vendors. Externally sourced tools reached deployment at roughly twice the rate of internal builds, which frequently failed due to brittleness and poor workflow fit.

Additionally, successful buyers treated AI vendors less like SaaS providers and more like service partners. They demanded deep customization, evaluated tools based on measurable outcomes, and expected systems to integrate into existing workflows with minimal disruption.

Individual productivity gains are not the same as organizational transformation

The report makes it clear that while many teams report that AI helps engineers move faster on specific tasks, those gains rarely translate into systemic impact unless teams design the workflows around them. Two critical reasons highlight this point:

1. 70% of users in the study prefer AI for quick tasks like generating simple unit tests or performing basic refactoring, but for complex projects involving multi-week work and stakeholder management, 90% of users still prefer human colleagues.

2. 66% of executives noted that they want systems in their core workflows that learn from feedback and have more robust memory, while many agents repeat mistakes and fail at complex tasks repeatedly.

As one CIO interviewed noted: “Once we’ve invested time in training a system to understand our workflows, the switching costs become prohibitive.” For now, AI will remain a powerful feature of the engineer’s overall toolkit, but not a replacement for the engineer.

Ultimately, the dividing line between human and agent isn’t just “intelligence,” it’s context. Engineers don’t just generate code; they maintain a mental model of the codebase, the business requirements, and the team’s historical preferences, culture, and style. Current AI solutions lack the contextual adaptability and true persistent memory necessary to perform complex human tasks.

Humans remain in the loop

Across the successful deployments examined, a consistent pattern emerged: AI systems were designed to augment human judgment, not replace it. Engineers remained responsible for architecture, correctness, and evolution, while AI handled acceleration, suggestion, and pattern recognition within defined constraints.

Analyzing the findings shows that the future isn’t one in which engineers disappear, but rather one in which their roles shift. There’s less time spent on toilsome rote work and more on the difficult problems of system design, quality, integration, and decision-making.

Final thoughts

The most revealing takeaway from this report is where AI adoption breaks down. The sharp drop from pilots to production suggests that real constraints only surface once AI is embedded in long-lived, complex workflows. That same pattern makes claims about AI replacing engineers feel premature.

If AI were truly ready to replace engineers, we would already be seeing faster delivery at scale, shrinking teams, and systems that reliably design, build, and integrate software with minimal human oversight. Instead, the report shows that most AI efforts fail at the point where real-world context, dependencies, and coordination matter most.

This represents more responsibility for engineers, not less. Software engineering has never been just about producing code. The companies getting real value from AI aren’t the ones trying to eliminate engineers. They’re the ones who understand AI’s capabilities to significantly increase throughput and innovative capacity. These teams invest in adoption, integration, and measurement.

This is the final issue of the Engineering Enablement newsletter in 2025. Enjoy the holidays, and we’ll see you in January.

Subscribe now

Subscribe now

🗓️We recently announced DX Annual, our flagship conference for developer productivity leaders navigating the AI era. Go here to learn about the event and request an invite to attend.

In 2025, AI-assisted engineering moved from an experiment to a core business expectation.

At the start of the year, adoption rates varied widely across organizations, and a path toward widespread use was just beginning to come into focus. Now, at the end of 2025, that picture looks very different: roughly 90% of developers across the industry are using AI tools at least once a month to get work done, with more than 40% relying on them every day.

As adoption has increased, so have our questions about its impact. Looking back at 2025, we’ve learned a lot as an industry about how AI is changing the way software gets made. To close out the year, I hosted a research roundtable with prominent voices in the AI and developer productivity research space. I invited them to reflect on what we’ve learned so far, and to share the questions they’re carrying into 2026.

A few clear themes emerged from that conversation. Below, I expand on those themes and add my own perspective on what they mean for the year ahead.

Watch the full discussion with Brian Houck (Microsoft), Ciera Jaspan (Google), Collin Green (Google), and Eirini Kalliamvakou (GitHub) here.

Lines of code is still a bad metric. Don’t let uncertainty and change make you reach for it

Building on themes from the last decade of developer productivity research, 2025’s research into measuring AI impact landed in a familiar place: there’s no single number that tells you whether AI is actually making a difference. Across the industry, organizations are using a broad range of metrics to measure impact (like the AI Measurement Framework, or check out this research piece on how Google, Microsoft, GitHub, and others are actually measuring productivity).

But even if there are a lot of good patterns out there, there’s also one bad pattern that this group of researchers called out: using lines of code (LOC) as a measurement for AI impact. This raw output metric is easy to measure, and in the absence of a clear alternative (especially in times of change), it can be tempting to reach for numbers that seem predictable and objective. Collectively, this group warned not to mistake output for impact. AI lends itself to writing a lot of lines of code, but that doesn’t necessarily mean a positive impact on your teams, organizations, or business.

Talking to AI and not talking to your colleagues might not be great for teams longer-term

Brian Houck, Sr. Principal Applied Scientist at Microsoft and co-author of the SPACE Framework of Developer Productivity, shared insights from his paper The SPACE of AI: Real-World Lessons on AI’s Impact on Developers, which shows how the impact of AI tools can vary widely across five key dimensions of developer productivity: Satisfaction, Performance, Activity, Collaboration, and Efficiency. While 90% of developers report that AI makes them more productive, fewer than half agree that it makes them more collaborative and communicative with their teammates. As teams use AI for longer periods of time, it will be interesting to see how this pattern plays out when it comes to knowledge sharing or even the long-term maintenance of codebases.

AI changes what skills developers need, but also how they perceive themselves

Eirini Kalliamvakou, Research Advisor at GitHub, shared some details from her recent research The new identity of a developer: What changes and what doesn’t in the AI era. As developers become more fluent with AI, their identity is shifting from traditional “code producer” toward a role focused on directing, delegating, and validating AI-assisted workflows, with creative judgment and strategic orchestration becoming central to their craft.

Interestingly, many of today’s heavy AI users started out as skeptics. Hands-on experience with the tools often changed both their sentiment and their expectations.

This identity shift has real implications for organizations, particularly around career progression, hiring, and upskilling. Both companies and developers need to place more value on AI fluency, systems thinking, and judgment, rather than raw coding output alone. And keeping pace with the ecosystem will require broader AI enablement, not just tool-specific training programs.

Is AI the death of the junior developer, or an accelerant to help them level up faster?

Lots of folks, from new grads to seasoned developers, are concerned about how AI will impact the talent pipeline. The usual narrative is that AI puts junior developers at risk of extinction, because why hire a junior when a senior developer can just delegate tasks to an AI agent instead? This could be a short-sighted optimization that leaves us with no developing talent a few years from now.

Ciera Jaspan from Google offered a compelling alternative perspective: what if the skills that devs need to level up in seniority—strong problem-solving skills, managing work, delegating work, and clearly defining outcomes—are now learned earlier by junior engineers because of the ways they need to interact with agents? Previously, these skills would be delayed because a tech lead or senior team member would take on the brunt of the project and professional management for these juniors. But when juniors are in the role of team lead for a handful of agents, do they actually level up faster because they get more practice solving problems end-to-end, even if their time spent actually typing code is reduced? Of course, in order for this to happen, companies still need to hire juniors, which isn’t always the case anymore. Find out how you might observe this pattern within your own teams.

Collin Green, another Google researcher, connected this back to earlier concerns about communication. Faster leveling through AI interaction doesn’t automatically translate to stronger collaboration skills. If juniors primarily work with AI rather than people, what does that mean for their professional development? And if seniors spend less time mentoring, what downstream effects might that have as well?

Is automation and code generation the right focus for the future of AI tools?

Collin advised on an AI paper that had a slightly different focus: creativity in software engineering. If creativity is the goal, and not productivity or automation, tools might better help us reimagine how work gets done and lead to more impactful outcomes, rather than just getting to the same outcomes faster. A shift in focus from creativity to productivity changes how tools are built and how we use them. The current emphasis on automation covers only a small surface area, with many developers coding only about 1 day a week on average. Everything else—scoping, experimenting, and validating ideas—exists in a very creative space, which AI is well-suited to help with, but many tools started with an emphasis on productivity and efficiency.

At the same time, there are so many tasks classified as “toil” that would serve developers well to be automated. These types of tasks make developers look at their to-do list and say, “ugh, today is not going to be great.” But Collin shared, we’ve had really solid research and technology for automating tasks for the last 40+ years. We should be automating things that are “dull, dirty, and dangerous.” But the most consistent problems that add friction or toil to developers’ days—like tech debt, lack of documentation, compliance tasks, even expense reports—still haven’t been solved with automation. Can AI change that, or are they just too complex to be automated? Is AI the right tool to automate it anyway?

Headlines will oversimplify AI research to the point of being incorrect, even if the research itself is full of nuance

Earlier this year, METR published a study that showed how, in some contexts, developers actually slow down when using AI, even if their own perception is that they were moving faster. This discrepancy between self-perception and actual result definitely made a big splash in our industry, and not a week goes by that I don’t hear someone cite the study as evidence that we’re all careening off an AI-generated cliff.

Importantly, the study itself had a lot more nuance and depth than the one-line headlines captured. And since one-line headlines are mostly what gets shared on LinkedIn and other platforms, it didn’t take long for the (well-done) METR study to be oversimplified and distilled down to the simple point that AI makes developers slower, which wasn’t really the point of the paper at all.

But the headline that AI isn’t actually as helpful as promised definitely resonated with people, and the study was widely shared. Many people felt drawn to the results that more accurately reflected their own personal experience trying to get started with a tool that was unreliable. For others, it was a good antidote to all the hype. One thing that the METR study did show was that AI research was now taking place in situ, meaning with real developers solving real problems, and specifically their problems.

An important thing to remember as 2026 will surely bring even more headlines that are greatly oversimplified summaries: stay curious about what’s behind the number. For the METR study, which was “AI makes devs 19% slower.” But not all devs, and not all context. Those questions are missing from the headlines, so you need to ask them yourself.

Looking to 2026

Closing out 2025, I can confidently say we haven’t seen the full spectrum of AI impact yet. We’ve made a lot of progress on understanding the impact of AI on teams, but we still need to keep digging to assess whether AI is achieving what we want it to, and to fully understand the impact on not just the whole software delivery lifecycle, but all levels of organizations.

One thing is clear to me though: the companies who will see the biggest wins with AI in 2026 are the ones who deeply understand their existing bottlenecks. The real acceleration from AI doesn’t come from using the newest models and testing every new tool; it comes from pointing AI at the real problems that slow developers down.

This week’s featured DevProd job openings. See more open roles here.

• American Express is hiring a Sr. Manager, Digital Product Management - DevProd | Hybrid - London UK

• Capital One is hiring a Product Manager - Developer Experience | Plano TX; McLean VA; Richmond VA

• Gusto is hiring a Sr. Platform Engineer | Denver, CO; San Francisco, CA; Atlanta, GA; Austin, TX; Chicago, IL; Miami, FL; Seattle, WA

• Plaid is hiring a Software Engineer - Platform | New York, NY

• Reddit: Staff Software Engineer - Developer Experience | Remote - United States

• Whatnot is hiring a Software Engineer - Platform | San Francisco, Los Angeles, Seattle, NYC

That’s it for this week. Thanks for reading.

Share

AI engineering tools are evolving fast. Every month brings new coding assistants, debugging agents, and automation capabilities. I want to help engineering leaders take advantage of that innovation while avoiding costly experiments that distract from real product work.

In this episode, Abi Noda and I share a practical, data-driven approach to evaluating AI tools. I walk through how to shortlist tools by use case, design structured trials that reflect real work, select representative participants, and measure impact using baselines and proven frameworks. My goal is to give you a way to test and adopt AI tools with confidence and a clear return on investment.

Some takeaways:

Data-driven evaluations are essential

• Structured, measurable trials prevent bias. Without them, decisions are driven by novelty hype or a few loud voices.

• Define a clear business outcome first (reduce toil, improve delivery speed, or raise code quality).

• Evaluations must inform real decisions, not just check a procurement box.

Choose the right set of tools to evaluate

• Group tools by use case and interaction mode (chat, agentic IDEs, code review assistants, etc.) to ensure fair comparisons.

• Match shortlist size to org capacity to support multiple cohorts and reliable results.

• Multi-vendor strategies reduce lock-in in a rapidly shifting market.

Re-evaluations are essential, not optional

• Incumbent tools must be retested as capabilities evolve and new challengers emerge.

• Triggers for re-evaluation include major feature launches, organic developer adoption of a new tool, and upcoming renewal cycles.

• Every challenger tool evaluation requires a baseline of the incumbent, so you can compare like-for-like.

• A cadence of every 8–14 months ensures decisions reflect the current reality, not the past purchase.

Design trials around research questions

• Start with a hypothesis. It keeps experiments aligned to actual goals.

• Developer sentiment is necessary but insufficient without measurable outcomes.

• Success criteria must be defined in advance to avoid subjective decision-making.

Select representative participants

• Diverse cohorts reveal real impact across languages, teams, and seniority levels.

• Include skeptical and late adopters to uncover onboarding and enablement needs.

• Volunteer-only trials distort results and won’t scale to full org rollout.

Run evaluations long enough to capture true behavior

• Eight to twelve weeks is the minimum to get past the novelty phase and into sustained usage.

• Align evaluation windows to procurement cycles so insights guide buying decisions.

• Short trials lead to false signals and either inflate enthusiasm or create false negativity.

Use self-reported time savings carefully

• Self-reporting is a strong early indicator of perceived usefulness.

• Humans misremember time, often benchmarking against recent AI use.

• Treat CSAT and time savings as directional, not the final truth.

• Objective metrics validate real ROI, including throughput, quality, and innovation time.

Expect variation rather than a single winner

• Different tools shine in different contexts, so multiple standards are often the best path.

• Continuous re-evaluation is required as capabilities evolve every quarter.

• The right goal isn’t the “best tool”, but the best tool for each problem space.

In this episode, we cover:

(00:00) Intro: Running a data-driven evaluation of AI tools

(02:36) Challenges in evaluating AI tools

(06:11) How often to reevaluate AI tools

(07:02) Incumbent tools vs challenger tools

(07:40) Why organizations need disciplined evaluations before rolling out tools

(09:28) How to size your tool shortlist based on developer population

(12:44) Why tools must be grouped by use case and interaction mode

(13:30) How to structure trials around a clear research question

(16:45) Best practices for selecting trial participants

(19:22) Why support and enablement are essential for success

(21:10) How to choose the right duration for evaluations

(22:52) How to measure impact using baselines and the AI Measurement Framework

(25:28) Key considerations for an AI tool evaluation

(28:52) Q&A: How reliable is self-reported time savings from AI tools?

(32:22) Q&A: Why not adopt multiple tools instead of choosing just one?

(33:27) Q&A: Tool performance differences and avoiding vendor lock-in

Where to find Laura Tacho:

• LinkedIn: https://www.linkedin.com/in/lauratacho/

• X: https://x.com/rhein_wein

• Website: https://lauratacho.com/

• Laura’s course (Measuring Engineering Performance and AI Impact): https://lauratacho.com/developer-productivity-metrics-course

Where to find Abi Noda:

• LinkedIn: https://www.linkedin.com/in/abinoda

• Substack: ​​https://substack.com/@abinoda

Referenced:

• Measuring AI code assistants and agents

• QCon conferences

• DX Core 4 engineering metrics

• DORA’s 2025 research on the impact of AI

• Unpacking METR’s findings: Does AI slow developers down?

• METR’s study on how AI affects developer productivity

• Claude Code

• Cursor

• Windsurf

• Do newer AI-native IDEs outperform other AI coding assistants?