Subscribe now

🗓 Join Justin Reock and me on June 18th for a research briefing on measuring AI agents, revisiting the Core 4, and more. Register here.

Last month, I had the privilege of attending the inaugural Developer Experience Research Forum at UC Irvine. It brought together researchers and practitioners from across academia and industry for a day of talks, conversations, and the kind of honest debate that only happens when the right people are in the same room.

The day included a panel that I won’t ever forget. For the first time since the SPACE framework was published in February 2021, all six of its authors were together in person: Nicole Forsgren, Margaret-Anne (Peggy) Storey, Chandra Maddila, Thomas Zimmermann, Jenna Butler, and me. I want to start by saying thank you to each of them. Collaborating on SPACE has been one of the most meaningful experiences of my career, and getting to sit alongside these colleagues five years later and take stock of what the framework has become was genuinely moving. I’m grateful to Tom, Iftekhar Ahmed, and the UCI team for making it happen, and to André van der Hoek for his amazing job moderating.

Unfortunately I don’t have a recording to make available to those who were unable to attend, but here is my attempt to capture the highlights of that conversation. The panel was wide-ranging, driven largely by questions from the audience, and covered a lot of ground. I’ll do my best to do it justice.

How SPACE came to be

For those less familiar with the backstory, SPACE did not emerge from a formal research program or a planned initiative. The idea started with Nicole. DORA, which she co-created, provided a measurement framework for software delivery, but there was a need for something that addressed developer productivity more broadly. So she reached out to a handful of colleagues (which thankfully included me), arrived with a few dimensions sketched out in her head, and the rest took shape over a series of Teams calls during what was still largely a remote-work world.

What’s remarkable is that several of us had never met in person before that project. We built the framework together, at a distance, and then watched it travel far beyond anything we had imagined. As I said on the panel:

“We could have never imagined that it was going to sort of grow into the thing it became. I don’t think you should ever hope to do something like that, because you’ll never be able to quite capture it. It’s like... right place, right time, things came together.”

The framework itself is straightforward in concept: five dimensions to consider when thinking about developer productivity. Satisfaction and wellbeing. Performance. Activity. Communication and collaboration. Efficiency and flow. The core argument is that productivity cannot be reduced to a single metric, and that a meaningful measurement approach should draw from at least three dimensions and include at least one perceptual measure. Metrics chosen well will often create productive tension with each other, and that tension is a feature, not a flaw.

Chandra reflected on the process of building it, including a detail I had honestly forgotten: the framework's original working name was not SPACE at all. It was FACTS. Trust was in there from the beginning, under a different label. Looking back on the framework more than five years later, Peggy said:

“I think we did a really good job. I think that the five dimensions have really held up really well. But they’re big — each of those dimensions are such huge concepts. And maybe what we need to do now is look at each of these in turn.”

Activity metrics: newly controversial, newly important

No dimension generated more discussion on the panel than Activity, and that’s not an accident. Activity metrics are the ones most organizations default to because they’re the easiest to instrument. They’re also the ones most prone to misuse.

What made the conversation interesting is that the panel did not argue for abandoning Activity measurement. The argument was more nuanced than that. Jenna put it directly:

"I actually think this is one of the areas where SPACE is newly important again, because you may have seen headlines about what percent of codebases are AI generated at this point. And I'm like, we're back there. We wrote about this a decade ago... some of those activity metrics like lines of code and PRs are newly resurfacing, and people are forgetting that we knew that this wasn't the greatest plan in isolation."

Chandra added that the scale has changed in a way that makes the problem even more acute. A single developer working with a swarm of agents can now generate an extraordinary volume of pull requests. The count alone tells you almost nothing about the quality, the impact, or the experience of the work.

The more useful question is not whether to measure activity, but which activity metrics are worth measuring and what you plan to do with them. I offered an example from my own work: Time-To-First-PR, meaning how long it takes a new hire to check in their first piece of code.

“Obviously easy to game, right? Have a new hire check in a trivial first PR... Turns out when you try to game it, when you explicitly try to have a trivial first check-in, it still leads to positive long-term outcomes. Why? Because that first code check-in has nothing to do with the code. It’s about learning your environment, setting up your system.”

I believe that good metric design involves choosing metrics where gaming them still gets you the outcome you actually want.

The politics of productivity measurement

One of the most candid moments of the panel came in response to an audience question about whether productivity measurement is inherently neutral or whether it inevitably becomes a political tool. The honest answer is that it is both, and you have to design for that reality.

Tom made the point that having five dimensions rather than one makes it structurally harder to play politics with the data. When you look at multiple dimensions simultaneously, you’ll often find they point in different directions, and that tension forces more careful thinking.

Jenna was direct about something that deserves to be said plainly. There is an elephant in the room across the industry right now about how many developers organizations need, and productivity metrics are being watched closely in that context.

"We tend to decouple from products and we're very... hoard-y with our data. We will give them trends. We'll let them know this is what's happening on a broad scale, or doing this had this impact. But we are not allowing individual managers, directors to look at people's information. We protect that because in theory, happy workers are productive workers. People who are terrified are not."

… and Nicole added additional framing that I found useful:

"Some data is better than no data... I know that for many of us here, we really do our best to measure in a way that is very neutral. But I know I'll have execs and other business divisions come to me and they'll say, 'Well, I need this [metric] to go up.' And I was like, 'Amazing. That's not on me. That's on you. I can give you the information and you can figure out if it goes up [or] down and why.'"

One practical safeguard worth noting is that some organizations deliberately bucket metrics together, so that no one can drive up a single number without being held accountable for the others in the cluster. It makes the kind of narrow gaming that distorts incentives structurally harder to do.

The C in SPACE: the most underinvested dimension

If Activity is the dimension that gets the most attention, Communication and Collaboration may be the one that gets the least. That gap is growing more consequential, and as Peggy put it, needs more focus than ever:

“Development is a team sport. And with AI, I don’t think there’s anyone in this room that doesn’t think that collaboration and communication hasn’t changed... If anybody here is thinking of using SPACE, make C one of the first things you look at.”

What makes this particularly important right now is that AI has changed collaboration patterns in ways we are only beginning to understand. Developers report asking questions of their AI tools that they used to ask colleagues. The texture of team communication is shifting. And yet the measurement infrastructure for tracking that dimension barely exists in most organizations. I shared a finding from my SPACE of AI paper that felt particularly relevant:

“C is the only dimension of SPACE that the majority of developers do not believe that AI has improved. Every other dimension showed improvement. Not C.”

That's a significant signal, and I think it points toward where the field needs to invest its energy next.

What would you add to SPACE today?

The question the audience asked that will stay with me longest was a simple one: if you were writing SPACE now, what would you add?

The five dimensions have held up well. The panel was in agreement on that. But five years of AI acceleration, remote and hybrid work, and a rapidly shifting sense of what software development even means has surfaced things the original framework didn’t fully anticipate. While we might not add new dimensions, if we were updating it today, we would add focus for:

• Trust: This was the most consistent answer across the panel, acting as a foundational bedrock for satisfaction and performance.

• Cognitive and Intent Debt: Based on Peggy’s recent work, are we losing overall understanding of codebases as AI writes more of them?

• Deskilling: The worry that relying heavily on automated tools will cause core engineering capabilities to atrophy over time.

• AI Addiction: Within the wellbeing dimension, tracking addiction-like behaviors with generative AI tools.

Ultimately, as Chandra mentioned, the strength of SPACE is that organizations can dial up or down different dimensions to meet rising needs. You don’t need to change the structure; you just need to rebalance the rubric.

"Things like well-being are very, very, very important. So I think reducing focus a little bit on the activity side... that doesn't fundamentally change what SPACE is. You can just use SPACE but rebalance the rubric."

We acknowledge that the framework was never designed to be a perfect model. But, as Peggy put it, that doesn’t mean it isn’t valuable.

"Some models are wrong, some are useful. It was supposed to change the conversation. It was supposed to make people think about the different aspects of productivity. And I think it did that."

That feels right to me. The framework was designed to change the conversation. I think it did. The work now is to keep refining what we measure within that space, with the same care we brought to defining it in the first place.

Final thoughts

It was a remarkable day for me. I’m grateful to UC Irvine for hosting it, to my co-authors for showing up, and to everyone in that room for asking the hard questions. If the industry continues to ask them with this much rigor and honesty, I think the next five years will be even more interesting than the last.

This week’s featured DevProd job openings. See more open roles here.

• Ashby is hiring an Staff Platform Engineer | Remote

• BambooHR is hiring a VP of Engineering | Utah (Hybrid)

• Cashea is hiring an Infrastructure & Developer Productivity Platform Engineering Manager | Remote

• Figma is hiring a Staff Software Engineer, Developer Experience | Remote; US

• Morgan Stanely is hiring an AI Platform Engineer - Vice President | New York

That’s it for this week. Thanks for reading.

Share

Abi Noda is joined live at DX Annual by three engineering leaders shaping AI adoption at scale: Tim Bozarth, Corporate Vice President in Microsoft’s CoreAI division; Nancy Wang, CTO of 1Password; and Taroon Mandhana, CTO of AI and Teamwork at Atlassian. Together, they discuss how AI is changing engineering organizations, from team structures and planning cycles to hiring, governance, and measurement.

The panel explores how the profile of a great engineer is evolving, why smaller cross-functional teams are becoming more effective, and what happens when product managers, designers, and customer support teams start contributing code. They also share why they are encouraging AI adoption through enablement, training, and local champions rather than mandates, and how AI is shifting more of the software development lifecycle toward planning and validation.

Finally, they discuss where human judgment remains essential, how to measure adoption and manage token usage, and how to connect AI investments to business outcomes while preserving room for experimentation and learning.

Some takeaways:

Rethink team structures for faster learning

• Smaller teams are becoming more effective for zero-to-one work. AI reduces the cost of implementation, making alignment and rapid iteration the primary bottlenecks when searching for product-market fit.

• Planning cycles are getting shorter. Instead of locking in 12- to 18-month roadmaps, teams are shifting toward quarterly planning to adapt more quickly to changing technology and market conditions.

• Large-scale org changes can wait. Several panelists emphasized that experimentation and learning should come before major structural redesigns.

The best engineers think like makers

• A maker’s mindset matters more than mastery of a specific tool. The most effective engineers stay focused on outcomes and use whatever tools help them build valuable products.

• Product and engineering skills are converging. Strong engineers increasingly combine technical depth with product judgment, customer empathy, and design sensibility.

• Agency is becoming a defining trait. Engineers who can work across functions, navigate ambiguity, and drive decisions are gaining leverage as AI handles more of the implementation work.

Expect more people to participate in software creation

• Prototypes are replacing long-form requirements documents. Interactive demos often lead to faster and more productive conversations than detailed specifications.

• Product managers, designers, and customer support teams are starting to write code. AI is lowering the barrier for non-engineers to contribute directly to the software development lifecycle.

• Quality systems become more important as more people contribute code. Robust tests, review processes, and deployment safeguards are essential when more people can generate production code.

Drive adoption through enablement, not mandates

• Outcomes matter more than activity. The goal is not maximizing AI usage for its own sake, but improving speed, ease, and product quality.

• Local champions accelerate adoption. Teams learn fastest when respected peers demonstrate practical ways to use AI in real workflows.

• Activity metrics are diagnostic, not the objective. Low usage can signal where teams need more training, support, or better tools.

The AI-native SDLC shifts work toward planning and validation

• Plan and validate are becoming the highest-leverage activities. As AI accelerates code generation, more human effort shifts toward defining what to build and evaluating whether it meets expectations.

• Operations and incident response are ripe for automation. Engineering teams are beginning to use AI to triage alerts, investigate incidents, write postmortems, and reduce the time spent on routine operational work.

• Human judgment remains essential. Leaders were unanimous that critical decisions around quality, security, and accountability still require people in the loop.

Measure outcomes, costs, and learning

• Token usage is becoming the new cloud bill. Engineering leaders are applying FinOps-style discipline to monitor and forecast AI spending.

• North Star metrics provide a clearer signal. Examples include idea-to-value, innovation time, and product quality.

• Experimentation deserves budget. Some AI usage will not generate immediate ROI, but it can create learning that compounds into long-term competitive advantage.

In this episode, we cover:

(00:00) Intro

(01:08) Introducing the panelists

(02:16) AI’s impact on engineering team structures and planning cycles

(05:00) How the role of the engineer is changing and what makes a great engineer

(10:11) The opportunities and challenges of non-engineers writing code

(15:26) Encouraging AI adoption without mandating it

(21:25) What an AI-native SDLC looks like and why human judgment still matters

(30:56) Measuring AI adoption, token usage, and ROI

(37:06) How to tie AI investments to business outcomes

Where to find Nancy Wang:

• LinkedIn: https://www.linkedin.com/in/wangnancy

Where to find Taroon Mandhana:

• LinkedIn: https://www.linkedin.com/in/taroonm

Where to find Tim Bozarth:

• LinkedIn: https://www.linkedin.com/in/tbozarth

Where to find Abi Noda:

• LinkedIn: https://www.linkedin.com/in/abinoda

Referenced:

• DX Core 4 Productivity Framework

• Microsoft

• 1Password

• Atlassian

• Jira

• Confluence

• Loom

• Rovo

• Amazon Operating Cadence - Working Backwards

Jennifer St Pierre is Senior Vice President of Developer Experience and Transformation at Dell Technologies, where she leads the strategy for how Dell’s Infrastructure Solutions Group builds, operates, and evolves software.

In this session from DX Annual, Jen argues that the biggest challenge in adopting agentic AI is not the technology itself, but the people transition behind it. Drawing on lessons from earlier shifts like Agile, DevOps, and cloud adoption, she explains why organizations that treat AI as a simple tooling rollout may get compliance, but not commitment.

Jen outlines five leadership imperatives for navigating the transition: building a shared understanding of why change is happening, defining a clear future state, clarifying how roles will evolve, creating psychological safety for experimentation, and aligning metrics and organizational structures with new ways of working. Throughout the talk, she emphasizes that while AI may generate code, humans remain responsible for direction, judgment, and meaning.

Some takeaways:

Treat AI adoption as a people transformation

• Technology transitions are really people transitions. The hardest part of adopting agentic AI is not the tooling itself, but helping engineers understand how their roles, workflows, and career paths will evolve.

• Compliance is not the same as commitment. Organizations that treat AI as a tooling rollout may achieve adoption metrics, but they will struggle to build the trust and engagement needed for lasting change.

• Every major platform shift follows a familiar pattern. New technologies create excitement, skepticism, fear, and eventually productivity gains that become the new normal.

Build a shared understanding of why AI adoption matters

• Start with an honest explanation of why the change is happening. If developers do not understand the business rationale, they are likely to assume the initiative is primarily about cost reduction.

• Shared understanding does not require universal agreement. It means everyone is working from the same candid view of market pressures, strategic goals, and organizational intent.

• Framing shapes emotional response. Positioning AI as a way to help engineers focus on more strategic work creates a very different reaction than simply saying it will increase productivity.

Define a clear future state

• A vague vision creates fear. When people cannot picture what their work will look like in 12 to 18 months, they tend to imagine replacement, stagnation, or obsolescence.

• Role clarity is essential. Teams need to understand what skills will matter, how performance will be measured, and which responsibilities will increase or diminish.

• Specificity beats slogans. Concrete expectations about how AI will be used help people see where they fit in the new model.

Create psychological safety for experimentation

• Teams need permission to make mistakes. AI adoption requires experimentation, and experimentation inevitably involves missteps and imperfect results.

• Psychological safety helps teams surface problems earlier. When engineers feel safe speaking up, leaders get better information and can address issues before they escalate.

• Silence is expensive. Leaders who discourage candor risk making decisions based on filtered or incomplete information.

Align metrics and organizational structures

• Old metrics can reinforce old behaviors. Measuring lines of code or heroic firefighting may encourage exactly the habits AI should help organizations move beyond.

• Metrics and structure must evolve together. Governance, incentives, funding, and performance systems need to support the behaviors leaders want to see.

• Transformation should survive without constant reminders. If the desired behaviors disappear as soon as leaders stop talking about them, the change has not yet become embedded.

Lead the transformation intentionally

• Leaders must model the change themselves. Using AI tools, sharing lessons learned, and being transparent about failures builds credibility.

• Career paths must be made explicit. Engineers want to know how they can continue to grow and whether deep technical expertise will remain valuable.

• AI may generate code, but humans generate direction. Judgment, context, and meaning remain the most valuable contributions people bring to software development.

In this episode, we cover:

(00:00) Intro

(00:13) Why every major technology shift is ultimately a people transition

(05:00) AI-generated code and the evolving role of software engineers

(07:43) The importance of developing a shared understanding

(12:00) Defining a clear future state and how engineering roles will evolve

(19:12) How psychological safety enables experimentation and honest feedback

(22:41) Why metrics and organizational structure must evolve for the age of AI

(25:40) Why leaders must drive AI transformation intentionally

Where to find Jennifer St Pierre:

• LinkedIn: https://www.linkedin.com/in/jennifer-st-pierre-4935a81

Referenced:

• Measuring developer productivity with the DX Core 4

• Understand team effectiveness

Jason Valentino is Head of Software Engineering Strategy at BNY, where he oversees developer tooling, DevEx, platform workflows, and software delivery governance across more than 8,000 engineers.

In this session from DX Annual, Jason shares how BNY moved beyond AI coding assistants to rethink the entire software delivery lifecycle. He explains how his team identified bottlenecks across the SDLC, prioritized automation opportunities, and applied AI to planning, peer review, testing, change management, and compliance workflows.

Jason also discusses what it takes to scale AI inside a highly regulated enterprise, including rewriting policies, partnering closely with risk and audit teams, and building a culture that encourages experimentation and rapid sharing of ideas.

Some takeaways:

Start with the 3X stress test

• Ask what breaks if engineering throughput triples. Jason’s team began by assuming AI would dramatically increase the volume of pull requests, code reviews, and releases, then identified which systems and processes would become bottlenecks.

• Map every step of the SDLC. BNY listed each task across planning, coding, testing, peer review, and release governance to understand which steps were manual, partially automated, or already well-instrumented.

• Use developer sentiment to prioritize investments. By combining workflow analysis with DX survey data, the team focused on the areas causing the most friction rather than chasing the latest AI use case.

Apply AI in three distinct ways

• Use IDE and CLI tools to amplify individual developers. Tools like Claude Code, Windsurf, and Codex help engineers move faster while still working within established guardrails.

• Deploy autonomous agents for repetitive work. BNY’s “digital workers” handle tasks like access requests, backlog grooming, and other low-value activities that engineers would rather avoid.

• Embed AI directly into workflows. The biggest gains come when AI is triggered automatically as part of code review, change management, and testing rather than relying on developers to invoke tools manually.

Use small automations to compound over time

• Automate the tedious parts of planning. BNY added AI capabilities to Jira to draft stories and epics, lint requirements, and assign confidence scores.

• Turn one automation into the next. Once a high-quality story exists, it becomes the foundation for generating test cases and other downstream artifacts.

• Look for highly manual actions. Jason recommends watching how teams actually work and identifying repetitive tasks that are prime candidates for automation.

Rebuild governance for an AI-assisted world

• Rewrite policies and controls. Existing language around code review, approvals, and software delivery often assumes humans perform every step and must be updated to reflect AI-assisted workflows.

• Bring risk and audit teams in early. Rather than presenting finished solutions for approval, BNY collaborates with governance partners while designing new approaches.

• Codify deterministic rules. AI can handle routine work automatically, while larger or riskier changes are routed to humans for additional oversight.

Treat duplication as a feature, not a bug

• Expect multiple teams to solve the same problem. In a large organization, some overlap is inevitable when thousands of people are experimenting with AI.

• Use show-and-tell to surface innovation. BNY hosts weekly sessions where teams demonstrate what they’ve built and share lessons learned.

• Consolidate the best ideas. Once similar solutions emerge, platform leaders can combine the strongest features into shared capabilities.

Create a culture that rewards experimentation

• Start saying yes. Jason’s advice to engineering leaders is to lower barriers and put promising ideas in front of users quickly.

• Treat internal tools like products. Successful experiments are documented, shared, and iterated on rather than left as one-off hacks.

• Make engineering fun again. For Jason, one of the biggest wins of the past year has been seeing teams energized by the opportunity to solve meaningful problems with AI.

In this episode, we cover:

(00:00) Intro

(01:20) Early results from AI coding tools at BNY

(04:08) The 3X stress test: What breaks if engineering throughput triples?

(06:56) Three ways to apply AI across the SDLC: IDE and CLI tools

(08:07) Using autonomous AI agents for repetitive engineering tasks

(09:16) Embedding AI directly into SDLC workflows

(12:27) Why leaders should encourage experimentation and “start saying yes”

(15:00) Q&A: How platform and productivity teams are evolving to support AI

(16:33) Q&A: Rewriting policies and controls for AI-assisted software delivery

(17:52) Q&A: How AI is affecting software quality and test ownership

(19:00) Q&A: What Jason is most proud of: Practical examples of AI across the SDLC

(20:30) Q&A: How BNY handles duplicated work across AI initiatives

(22:30) Q&A: How BNY uses AI to support regulatory and compliance work

(23:30) Q&A: Automating code reviews and change tickets

(25:55) Q&A: How increased AI-driven throughput is affecting on-call and reliability

(27:11) Q&A: How BNY works with risk and audit partners to move quickly with AI

(29:01) Q&A: How BNY scales successful AI use cases across the organization

(30:42) Q&A: What Jason is most proud of after BNY’s busiest year with AI

Where to find Jason Valentino:

• LinkedIn: https://www.linkedin.com/in/jasonvalentino

Referenced:

• AI-assisted engineering: Q4 impact report

• Measuring AI code assistants and agents

• Measuring developer productivity with the DX Core 4

• Windsurf

• Claude Code by Anthropic | AI Coding Agent, Terminal, IDE

• Codex | AI Coding Agent

Recorded live at DX Annual, Abi Noda, co-founder and CEO of DX, joins Brian Houck of Microsoft to share an early look at DX’s new research on AI’s impact on engineering velocity.

Drawing on data from a sample of DX customers, they discuss what companies are actually seeing as AI adoption matures. Most organizations in the study saw pull request throughput increase by 10 to 15 percent—far more modest than the 10x gains often promised in industry headlines.

They explore why coding remains only a small part of developer work, where time saved by AI may be going, and the unintended consequences of moving faster, from shifting bottlenecks to “false velocity.” Abi also shares how engineering leaders are applying AI beyond coding and how DX is evolving its measurement framework to account for both human and agent productivity.

Some takeaways:

Most organizations are seeing modest gains from AI

• PR throughput is increasing by about 10 to 15 percent. Across DX’s sample, most organizations saw measurable improvements, but the gains were far smaller than the 10x productivity increases often cited in industry headlines.

• The median improvement was closer to 8 percent. While some organizations saw larger gains, the typical impact was more incremental than transformational.

• Even modest gains can be meaningful at scale. A 10 percent increase in throughput can represent a significant improvement when applied across hundreds or thousands of engineers.

Coding is only one part of the productivity equation

• Developers spend only about 14 percent of their time writing code. If AI primarily accelerates coding, its impact on overall engineering velocity will naturally be constrained.

• The biggest bottlenecks often lie elsewhere. Planning, reviews, testing, documentation, and coordination still consume the majority of engineering time.

• Time savings do not map neatly to output gains. Organizations can see meaningful reductions in coding effort without a proportional increase in pull request volume.

Why productivity gains are lower than many leaders expected

• Coding is not the primary bottleneck. Improving a small slice of the development process only moves the overall system so far.

• Automation creates new bottlenecks. Faster code generation can increase pressure on reviews, QA, and technical oversight.

• Social friction slows adoption. Skepticism, inconsistent usage, and unrealistic expectations can limit the benefits of AI tools.

• Tool and skill gaps compound over time. Engineers need both the right tools and the knowledge to use them effectively.

• AI tools still lack context. Limited understanding of business logic and codebase nuances can reduce output quality.

Beware of false velocity

• More code does not necessarily mean more business value. Teams can increase pull request counts without meaningfully accelerating roadmap delivery.

• Quality and cost remain critical concerns. Organizations are closely monitoring technical debt, token spend, and long-term maintainability.

• Faster output can create delayed consequences. The full impact of AI-generated code may not become apparent until months later.

The biggest opportunities lie beyond coding

• The remaining 86 percent of engineering work is the next frontier. Leaders are applying AI to planning, documentation, incident response, and other parts of the SDLC.

• Autonomous agents can augment human capacity. Instead of simply speeding up developers, organizations are exploring how agents can work in parallel.

• Developer experience still matters. Improving focus time, documentation, and workflow friction can amplify the benefits of AI.

Measurement frameworks are evolving

• Some metrics remain constant. Velocity, quality, and developer experience are still essential signals.

• Acceleration and augmentation should be measured separately. Leaders need to distinguish between human productivity gains and work performed autonomously by agents.

• Agent experience is an emerging concept. DX is beginning to survey AI agents directly to understand their constraints, bottlenecks, and effectiveness.

Cognitive debt is a new concern

• AI can reduce understanding while increasing output. Developers may ship code more quickly while building a weaker mental model of the systems they maintain.

• Short-term efficiency can create long-term costs. Reduced comprehension may make future debugging and maintenance more difficult.

• The long-term effects are still uncertain. Engineering leaders are only beginning to understand the human consequences of AI-assisted development.

In this episode, we cover:

(00:00) Intro

(00:53) What motivated DX’s research into AI’s impact on engineering velocity

(02:36) How DX designed the study and selected companies

(04:54) What DX’s data reveals about AI’s impact on engineering throughput

(06:31) Why PR throughput was the most practical metric to publish

(08:21) Why AI productivity gains are lower than many leaders expected

(10:24) How an all-in culture can amplify AI productivity gains

(12:35) Why it’s hard to track where AI-generated time savings are going

(15:04) Unintended consequences of AI-driven productivity gains

(17:12) Why leaders should look beyond coding to the rest of the SDLC

(19:43) Cognitive debt and the human costs of AI-assisted development

(21:33) How DX’s AI measurement framework is evolving

(24:42) How to make agents more effective

Where to find Brian Houck:

• LinkedIn: https://www.linkedin.com/in/brianhouck/

Where to find Abi Noda:

• LinkedIn: https://www.linkedin.com/in/abinoda

Referenced:

• DX Core 4 Productivity Framework

• DORA, SPACE, and DevEx: Which framework should you use?

• Time Warp: The Gap Between Developers’ Ideal vs Actual Workweeks in an AI-Driven Era - Microsoft • Research

• How Generative and Agentic AI Shift Concern from Technical Debt to Cognitive Debt

• Measuring AI code assistants and agents

Subscribe now

This week I’m sharing a new paper I co-authored with Jenna Butler, Margaret-Anne Storey, Travis Lowdermilk, Steven Clarke, and Emerson Murphy-Hill: 8 Myths on Software Engineering and GenAI. Drawing on recent large-scale studies, developer interviews, and field observations, the paper unpacks eight of the most persistent misconceptions about AI in software engineering.

Engineering leaders and teams can use this paper to set realistic internal expectations and benchmark their own AI ROI against real-world data.

My summary of the paper

The eight myths fall into three groups: how developers actually spend their time, how to measure AI’s impact, and how AI gets adopted in real organizations. The through-line is that the gap between AI’s promise and its measured impact has less to do with the models and more to do with the surrounding system of work.

What I hope makes this paper worth reading isn’t that any single myth is new; most have been circulating in the developer productivity research community for the last couple of years. What’s new is putting them side by side and showing how they reinforce each other.

Time, bottlenecks, and lines of code (Myths 1–3)

A 2025 study that I co-authored found that developers spend only 14% of their time writing code, consistent with prior research showing coding hovers between 11% and 18% of a typical day. The rest is design, meetings, review, coordination and administrative tasks.

That makes AI-as-code-generator a smaller lever than the headlines suggest, which aligns with DX’s findings that the typical organization is seeing a 7.8% increase in code throughput. Accelerating one phase can shift load into the next, and 8 Myths found that for one internal AI coding agent, only about half of the PRs were ultimately accepted, with 15% abandoned and 15% stuck waiting on a human reviewer.

The paper posits that measuring impact by lines of code, including AI-generated lines, repeats a mistake the field has known about for a decade. As Bill Gates put it: “Measuring software productivity by lines of code is like measuring progress on an airplane by how much it weighs.” Volume metrics incentivize the wrong behaviors and inflate the very review and quality burden that’s already the bottleneck.

To bridge the gap between bad metrics and true impact, many teams look to PR Throughput as a more complete measure of “work”. Tracking completed PRs is a significant step up from lines-of-code, and is a metric I have long supported, when used appropriately. However, even throughput requires caution in an AI era, and is important to use in a basket of metrics, as it is in the DX AI Measurement Framework.

Effects vary more than the headlines suggest (Myths 4–5)

The research on AI development tools is genuinely mixed. Some studies show large gains, others show neutral effects, and one study of experienced open-source developers found AI tools even increased implementation time by 18% on average.

The variance isn’t random. Familiar tasks benefit more than unfamiliar ones. Developer experience, motivation, and problem-solving style all shape outcomes. Even the prompt matters: one study found semantically equivalent rewrites produced different code in 46% of cases and changed correctness in 28%.

This is further proof why the “10x developer” narrative doesn’t hold up. Productivity gains measured on isolated, toy-sized tasks rarely survive contact with real codebases and real teammates, and prior research suggests much of the variance between developers is a property of the task, not the person.

Adoption is a systems problem, not an individual one (Myths 6–8)

Despite the headlines, only 10% of developers in one Microsoft survey expressed concern that AI tools might take their jobs. Most describe AI as expanding their creative capacity. More time on architecture, mentorship, and brainstorming, less on lookups.

But adoption itself is harder than the market assumes. The paper showed that 80% of developers use AI tools, but only 29% trust their accuracy. Recent research also identifies a “competence penalty”—developers, particularly women and older engineers, receive harsher evaluations for AI-assisted work even when the output is identical.

And almost all of the existing research still studies a single developer paired with a single tool, placing the burden of productivity on the individual. Historically, real productivity gains haven’t come from individuals optimizing their own work, they’ve come from systematic changes at the organizational level. AI may be the first technology where organizations have spent millions on licenses without a clear plan for how to extract value from them.

Final thoughts

The through-line across these eight myths is that AI’s impact in software engineering is shaped more by the system around the developer than by the developer themselves. Coding is a small share of the work. Lines of code is a poor measure. Effects vary by task, person, and context. And adoption depends on trust and organizational support far more than on tool capability.

For engineering leaders, that points to a different set of questions than the ones the market tends to ask. Not “how much code did AI write?” but “where in our system is AI actually relieving friction, and where is it just shifting pressure downstream?” Not “how do we get developers to use it?” but “what would it take for our developers to trust it?”

Treating AI adoption as an engineering systems problem, not a productivity hack, is what separates the organizations seeing real value from those still chasing the myths.

This week’s featured DevProd job openings. See more open roles here.

• Ashby is hiring an Staff Platform Engineer | Remote

• BambooHR is hiring a VP of Engineering | Utah (Hybrid)

• Cashea is hiring an Infrastructure & Developer Productivity Platform Engineering Manager | Remote

• Figma is hiring a Staff Software Engineer, Developer Experience | Remote; US

• Morgan Stanely is hiring an AI Platform Engineer - Vice President | New York

That’s it for this week. Thanks for reading.

Share

Subscribe now

In “The SPACE Of AI”, we found that AI efficiency is a skill: as developers spend more time with these tools, their ability to extract value compounds. This is supported by another recent study which found that developers with higher usage were more likely to report AI making them more productive.

To explore this further, we looked at the trajectory of time savings from AI coding assistants: how quickly developers reach their peak gains, and whether those gains hold over time.

To explore this, we looked at a sample of DX data from over 500 companies over the last year (May 2025-April 2026). Our analysis focused on self-reported time savings, where developers estimated the number of hours per week they saved through the use of AI coding assistants. By tracking individual migration patterns between time-savings bands (categorized as low: <4 hrs/week and high: 6+ hrs/week), we could see how quickly gains are achieved and whether they are sustained.

What we’re seeing: Peak times savings are temporary

When we followed individual developers over four quarters to see how their time savings evolved, a few things stood out.

• Low initial savings often lead to higher future gains. Nearly one in three developers (31.4%) who started in the lowest time-savings band climbed to the highest band during the study period, indicating that early AI gains are a starting point rather than a fixed ceiling. In “The AI Native Developer,” we found that AI fluency increases with use, which raises an open question: how does this ramp-up in time savings map to the stages of AI fluency we identified?

• Time savings from AI ramp up quickly with continued use. Among developers who reached the highest time-savings band, roughly 7 in 10 (69.7%) got there in less than two quarters. This rapid progression is likely driven by a combination of factors, ranging from how quickly a developer adapts their workflow to how well their specific tasks align with AI capabilities.

• Peak time-savings are temporary, and may fade. Of those developers who reached peak time savings, two-thirds (66.1%) reported lower time savings in the quarters that followed.

  • Gains achieved in one quarter are not sustained. Among developers who hit the highest time-savings band in one quarter or less, just over half (50.5%) did not report that level of savings in future quarters. While we don’t know the exact cause of these fluctuations, the speed of the initial spike may reflect a burst of enthusiasm or a concentration of easy-win tasks but it does not translate into a durable shift in productivity.

  • A longer ramp-up does not show staying power. For developers who took two quarters to reach peak savings, the drop-off is even steeper. 79% did not report high time savings in future quarters. This suggests that even a more gradual adoption path does not insulate developers from the plateau, and that time using the tool alone isn’t enough to sustain peak productivity gains.

What might be causing the plateau?

Before exploring potential causes, an important caveat to mention is that our study covers a limited number of quarters, so the patterns we’re seeing here are early observations rather than settled conclusions. As we gather more data, we’ll be able to test these patterns over longer time horizons and revise the picture accordingly.

With that in mind, we want to offer a few possible explanations for the plateau. We can’t yet say which of these are doing the most work, or whether there are other factors we haven’t yet considered, but each is consistent with what we’re seeing in the data and worth investigating further.

System-level constraints

Individual efficiency gains create secondary challenges at the system level. While task-level coding is accelerating, the time saved is frequently redistributed into areas that are currently under-measured, such as increased experimentation, deeper architectural exploration, and quality improvements. Our prior research found that the median engineering organization sees a 7.8% increase in PR throughput from AI. Real, but more modest than headline claims would suggest, and consistent with the idea that the surrounding system is absorbing much of the individual-level efficiency. We may be observing a shift where the bottleneck moves from code production to system coordination. In some cases, engineering teams appear to be shipping faster than the surrounding product management and verification processes can support.

The task ceiling

The plateau could also be tied to a task ceiling. Early gains often come from automating high-volume, low-complexity work. Once these are optimized, developers may struggle to apply AI to more complex areas like architectural design or legacy refactoring. We need more research into specific use cases to understand if the plateau is universal or if developers who move AI “upstream” into design or “downstream” into debugging sustain higher gains.

Shifting baseline of productivity

Even when developers reach high time savings quickly, sustaining that perceived impact is difficult. One possible explanation is a new normal effect: once a developer integrates AI into their workflow, the resulting efficiency gains become the baseline. When surveyed in subsequent quarters, developers are no longer comparing their performance to a pre-AI workflow, but to their newly optimized standard.

Why this matters for engineering leaders

Individual efficiency gains are fragile. While AI coding assistants can deliver meaningful gains for a significant share of adopters, the large share of developers falling back from their peak savings suggests that time with the tool alone isn’t enough to sustain those gains. As developers produce code faster, they often hit ceilings, like slower code reviews and architectural bottlenecks, that neutralize individual gains. This suggests that the plateau in time-savings may not be a failure of the tool or the user, but a sign that the bottleneck has shifted from individual code production to team-level coordination. It highlights a need for deeper research into whether certain use cases, like complex debugging or requirements drafting yield more durable gains than the high-volume, low-complexity tasks that typically drive initial adoption spikes.

This week’s featured DevProd job openings. See more open roles here.

• Ashby is hiring an Staff Platform Engineer | Remote

• BambooHR is hiring a VP of Engineering | Utah (Hybrid)

• Cashea is hiring an Infrastructure & Developer Productivity Platform Engineering Manager | Remote

• Figma is hiring a Staff Software Engineer, Developer Experience | Remote; US

• Leidos is hiring a Platform Engineer | Remote; US

• Morgan Stanely is hiring an AI Platform Engineer - Vice President | New York

• Weave is hiring a Senior Platform Engineer, Data Infrastructure | Remote; US

That’s it for this week. Thanks for reading.

Share

Subscribe now

In case you missed it, we just released the DX Annual recording library. You can watch the recordings by clicking through the Sessions page here. We’ll release recordings on the podcast soon as well, stay tuned.

There are several assumptions about AI circulating in executive teams and boardrooms that are shaping strategy today. For example, some leaders believe AI will ultimately mean fewer developers, or at least fewer junior engineers. Some believe AI adoption must be mandated from the top in order to take hold.

These beliefs are rarely black and white. So at DX Annual, we convened a panel of senior engineering and research leaders to debate these questions in the open. Our aim was to surface the nuance behind these assumptions, give our audience exposure to contrasting viewpoints, and equip them to articulate their own position with greater clarity and confidence.

The panel:

• Rafe Colburn, Chief Product and Technology Officer at Etsy, where he’s led engineering for 14 years through multiple waves of technology change

• Jesse Adametz, Sr. Director of Engineering, Platform Engineering at Twilio, overseeing developer tooling and productivity for thousands of engineers

• Eirini Kalliamvakou, Research Advisor at GitHub, whose work on developer productivity measurement has shaped how the industry thinks about engineering effectiveness

• Collin Green, Senior Staff UX Researcher at Google, where he studies developer workflows and the factors that drive or block productivity

• Brian Houck, coauthor of the SPACE framework (and now Distinguished Researcher at DX), whose research on AI’s impact on engineering spans many thousands of developers and managers

Each panelist was read a series of statements and asked for their opinion. Below is a recap of each statement and the panelists’ reactions. You can watch the full session here.

Will AI mean fewer engineers?

Statement: An AI-first SDLC means fewer engineers.

Rafe: Thumbs down. A job is a bundle of tasks, and AI is a clean substitute for some of them. People might not be doing those things anymore. But the demand for software isn’t going away, it’s going up. The price of building software is going to go down. So I don’t think fewer engineers is a likely scenario in the near future. I bet against it.

Brian: I agree, it’s not about fewer engineers, it’s about doing more. But one nuance: in the future, what do we even define as a software engineer? The bundle of tasks we think of today that make up a software engineer may change. Writing code is a core part of the identity. The people who do that specific thing may go down, but the total number of makers and builders won’t.

Collin: You’ll probably be able to do the same amount of stuff with fewer people, but that doesn’t mean demand will go down. Small companies that want to accomplish something will be able to get over the barrier to entry faster. It may actually increase demand for engineering skills.

Is AI accelerating technical debt?

Statement: AI is currently creating technical debt faster than it is helping us refactor it.

Eirini: Because you said “currently,” yes. Things with code generation are moving at a pace where the rest of the system hasn’t evolved enough to match it. We’re also seeing the anticipation of technical debt throttling how much developers use agents. They’re trying to balance the risk of something with the velocity of something, and it’s mental math that has to happen in the moment. There are ways to manage against the accumulation, like event-driven or scheduled agents that do maintenance and hygiene for codebases. It’s not a perfect solution, but it’s working for now. The accumulation of technical debt is certainly a big issue.

Jesse: It’s too definitive of a statement. We look at our engineers in quartiles, and our highest-performing engineers are also our highest-performing AI engineers. AI is an amplifier. If you’ve got high-performing engineers, they weren’t putting garbage into the system before and they’re still not. It’s garbage in, garbage out.

Rafe: I’ve worked at Etsy for 14 years. Proportionally, are we creating more tech debt now than five or ten years ago? No. The old code is still there. We’re creating a greater volume of code, but I’m not sure a greater proportion of it is tech debt. Automated code reviews are helping, and you can ask a coding agent to look at patterns in the code. The greater volume probably brings more tech debt in absolute terms, but I don’t think it’s an accelerator for tech debt at a greater rate than it’s an accelerator for anything else.

Brian: Disagree with the others. Organizations are optimizing for PR velocity, not cleanliness. And cognitive debt, understanding our systems less as AI writes more of the code, is a form of technical debt that’s growing. In the short term, tech debt is going up relative to PR velocity.

Collin: Technical debt is a business decision, not strictly an engineering one. If businesses have the same tolerance for risk and the same market pressures, they’ll incur the same total amount. But viewing agentic development as low-cost increases the risk of imprudent decisions.

Eirini: The choice of what even are you building becomes so important.

Will most code be AI-generated within five years?

Statement: In five years, more than 50% of code in most organizations will be written by AI.

Collin: If we’re talking about new code, yes, maybe even sooner than five years. But it’s worth asking whether that’s code that would have been written by humans otherwise, or just a larger total volume. Also worth considering how much of it will be rapidly rewritten.

Jesse: I interpreted the statement differently. If the claim is 50% of all code in a codebase, that’s wild. Too many companies have too much legacy software, and there’s no advantage to rewriting all of it.

Rafe: Agreed it has to be new code for the statement to hold.

Is the future of engineering about managing agents?

Statement: The future of software engineering is more about managing agents rather than about writing code.

Brian: In an idealized world, yes. But humans aren’t great at multitasking. Microsoft’s internal research shows that even experienced developers under time pressure revert to sequential, single-threaded agentic workflows. People with prior management experience (delegation, context-switching) will be more successful, but it’s a different skillset that we’re not hiring for yet.

Rafe: Thumbs up, loosely. Even using Claude Code today, you’re already managing multiple agents, it’s just abstracted away. The hobby-project vision of orchestrating 15 agents simultaneously isn’t the future. AI will help manage that complexity. But will a large chunk of your work be mediated by at least one agent? Yes. Agent orchestration is not a full-time human job.

Jesse: As a director, my brain naturally goes to managing agents. But the identity of a software engineer matters. Not everyone wants to be a manager, and not everyone is good at it. A blanket statement that everyone will manage agents is too much.

Eirini: I interpreted “managing” as the real engineering work of setting agents up for success: defining intent, setting constraints and guardrails, providing context, and then verifying output. That’s real engineering at a different level of abstraction. If that’s what the statement means, it’s a strong thumbs up.

Collin: Micromanaging agents is a failure mode. That leads to task-switching bottlenecks.

Do leaders need to mandate AI adoption?

Statement: Leaders need to mandate AI usage to make sure adoption is moving along.

Jesse: Not what we’ve seen. Twilio did light enablement (install defaults, basic guidance), and adoption took off on its own. No top-down mandate needed.

Rafe: People outside of engineering orgs often push for mandates because they’re afraid of falling behind. But mandates lead to shallow adoption and “tokenmaxxing.” AI adoption is inevitable, so why mandate something inevitable? If someone had said a year or two ago that every single talk at the DX conference would be about AI, people would have said no way—but is there anything else to talk about? Telling software engineers AI is going to be part of their job just feels like you’re fighting last year’s battle this year. Leaders should focus on removing friction and making it easy to create value, not pushing people.

Brian: I’m running a study of ~600 engineers and managers. The biggest disagreement: the majority of engineering managers think AI usage is a reasonable individual performance metric. Engineers disagree. That’s a myth to dispel. Activity metrics are useful for understanding patterns, but should not be used as direct performance measures.

Jesse: If organizations aren’t crystal clear about how AI will be measured, people will fill the gap with their biggest anxieties. Twilio isn’t putting AI usage in career frameworks anytime soon, but the reality is you’re compared to your peers.

Rafe: If you went to a job interview today and said “I don’t use AI coding tools,” it would be like saying you’re a programmer who refuses to use a text editor. It would just sound strange. Adoption is inevitable without a mandate.

Is code review now the bottleneck?

Statement: The bottleneck of software delivery is no longer writing code, it’s now reviewing code.

Collin: As Brian noted earlier in the day, developers only spend about 14% of their time writing code, so code was never really the bottleneck. The real bottlenecks are decision-making, prioritization, product design, support, and operations.

Jesse: AI is an amplifier. Whatever your organization’s bottleneck used to be, it still is. For Twilio, deep work has been an issue for a long time and still is. The conversation has shifted to “agent experience,” but the fix is the same: improve developer experience. Meetings, coordination tax, decision-making, these have always been the problems.

Brian: Code review is still a bottleneck, and it’s increasing as more code is produced. But at Microsoft, some features take two or more years from planning to customer delivery. Going from two days to three days on review isn’t the long pole. Planning and prioritization are.

Rafe: The perceived bottleneck of code review is what’s interesting. When writing a PR took two days, a review delay was annoying but tolerable. When writing takes 10 minutes, the human process of finding a reviewer and waiting feels unbearable. The interruptions of human processes, unless they provide clear value, are grating.

Jesse: Twilio’s data showed their highest-quartile AI developers took a ~14-point hit in perceived code review turnaround, but their actual median merge time decreased by multiple hours. It’s pure perception.

Is risk the only thing holding back adoption?

Statement: The only thing holding engineers back on AI adoption is unnecessary worry about risk.

Eirini: Engineers feel deeply accountable for what ships. When something is high-risk or hard to undo, they pull back on how much they leverage agentic velocity. That’s not irrational, it’s human behavior that’s hard to get over.

Brian: Risk is a big reason, but not the only one. Over a fifth of developers in our research cited concern about introducing defects and vulnerabilities as a top barrier.

Jesse: I question whether AI is actually increasing risk, since engineers were already introducing bugs before AI.

Rafe: The risk landscape has completely shifted. If we had a butter knife and now you have a chainsaw, the risks are different. The risks that you won’t be able to cut down a tree have gone way, way down. But the risks that you might really make a mess are a lot higher. We have people using these tools who don’t really know how a chainsaw is different than a butter knife — “Look, someone just gave me something, cool.” And on the other hand, engineers who are like, “I don’t want to use the chainsaw unless I really understand everything about how it works.”

Collin: Beyond risk, there are basic enablement problems: helping people choose tools, configure them, and get started. Those aren’t AI-specific, they’re just adoption problems.

Are AI adoption problems really culture problems?

Statement: Most AI adoption problems are really culture and management problems, not tooling problems.

Rafe: Culture and management problems are real, but sometimes the next blocker is a tooling problem, or a process problem. Process is a form of tooling. At Etsy, with a 20-year-old codebase and deeply tenured engineers, stated and unstated processes create friction. It’s all of the above. But if you’re a leader and you’re not giving people time to learn, you are doing it wrong. I used to ask engineers, “Are you working as fast as you can?” “I don’t know.” “Are you learning anything from what you’re doing?” “No.” Then you’re working as fast as you can. You’re not taking any time for that. Making space for learning solves 90% of these problems.

Collin: Most problems are human problems. Tool impediments exist, but the bigger barriers are incentives, anxiety, learning, and space to learn. Leaders who give people time to learn demonstrate they understand how the world actually works.

Eirini: The transformation is massive. Bottlenecks have shifted, processes aren’t working as expected, metrics no longer mean the same thing, and people are having an identity crisis. It can’t be just a tooling problem. Culture, motivation, and infrastructure all need to adjust.

Jesse: There’s a compounding effect. For people who haven’t started, it’s getting harder to start. I told my VP a couple weeks ago, “I have to quit to be able to keep up with the industry.” That’s the day job. So if folks haven’t gotten started, I can only imagine how much more daunting it’s getting.

Brian: Everything is moving so fast that what you learned yesterday is irrelevant tomorrow. Long-term burnout and fear of falling behind are basic human challenges. No matter how complex the technology, it’s always a human problem at the end of the day.

Eirini: Teams that did two-week group learning sprints (everyone uses agents for everything, no exceptions) saw much better results in adoption, engagement, and outcomes than teams where individuals learned on their own.

—

There’s a common throughline in these discussions: AI is reshaping the task mix of software work, but not eliminating the need for engineers. Additionally, the biggest risks and opportunities sit in how leaders design roles, measure impact, and consider the full PDLC, not in whether AI can generate more code.

To go deeper, you can watch the full session here and explore the rest of the DX Annual recordings on the Sessions page.

This week’s featured DevProd job openings. See more open roles here.

• Ashby is hiring an Staff Platform Engineer | Remote

• BambooHR is hiring a VP of Engineering | Utah (Hybrid)

• Cashea is hiring an Infrastructure & Developer Productivity Platform Engineering Manager | Remote

• Figma is hiring a Staff Software Engineer, Developer Experience | Remote; US

• Leidos is hiring a Platform Engineer | Remote; US

• Morgan Stanely is hiring an AI Platform Engineer - Vice President | New York

• Weave is hiring a Senior Platform Engineer, Data Infrastructure | Remote; US

That’s it for this week. Thanks for reading.

Share

Subscribe now

Hi everyone 👋 I’m Brian - I’ve recently joined DX as an Applied Scientist. I previously led research for EngThrive, Microsoft’s company-wide initiative for measuring and improving developer experience. I’ve been part of this newsletter before and am now excited and honored to be a regular contributor.

For my first issue, I’m summarizing a new paper I co-authored with Rudrajit Choudhuri, Eirini Kalliamvakou, and Thomas Zimmermann, published in ACM Queue: The AI-Native Developer.

In this study, we explore how AI is reshaping software engineering by analyzing survey data from over 1,300 developers and interviews with 22 AI-fluent practitioners. We specifically look at what developers want from AI and how AI fluency develops, and discuss three possible futures for the developer role.

What developers actually want from AI

To understand where developers want AI involved—and where they don’t—we mapped their daily work across four dimensions: value, identity, accountability, and demands. This revealed three clusters. Core Work (coding, debugging, system design, testing) scored high on all four. Ops & Coordination (DevOps, documentation, stakeholder communication) scored high on value and demands, but weakly on identity. People & AI Building (mentoring, AI feature integration) scored moderately across the board, with relatively strong identity alignment.

These clusters map onto what we call an “AI opportunity space”, a plot of developers’ openness to AI support against their actual AI usage. Core Work and Ops & Coordination tasks both landed in the Build and Improve zones. Developers were open to deeper AI assistance for everything from cross-artifact debugging to environment provisioning, but reported that current tools hadn’t caught up. This suggests the barrier is less reluctance and more trust.

For production-facing work, developers consistently emphasized predictability, transparency, verifiability, and human oversight.

The De-prioritize zone was the most revealing. Relational work like stakeholder communication and customer interactions, landed here consistently. Developers saw little role for AI beyond preparation. They wanted to retain final voice and responsibility.

How AI fluency actually develops

In our interviews, we observed a consistent four-stage journey from initial resistance to full orchestration.

The Skeptic starts with rational resistance. For skeptics, AI suggestions interrupted flow and created extra work. What shifted skeptics wasn’t persuasion; it was urgency. Developers saw AI fluency becoming a competitive advantage and pushed through the frustration. One put it plainly: “Either you have to embrace the AI, or you get out of your career.”

The Explorer emerges when AI starts to genuinely help. A cryptic error decoded, a familiar task automated, mundane work handled. Each small win chips away at skepticism.

The Collaborator integrates AI into the workflow earlier rather than using it only as a fallback. They treat iteration as part of the workflow rather than a recovery mechanism when AI output fails.

The Strategist orchestrates multiple agents in parallel: one defining test cases, another implementing, a third reviewing for security. They set context, review output, and decide what to delegate next. Strategists don’t just prompt, they front-load context. Some even have the AI “interview” them to surface missing requirements and business goals before a single line of code is generated. They have moved from being code producers to architects of intent. Ask them about the future, and they’ll tell you AI will write 90–100% of code within two years. What’s striking is that this prospect energizes rather than threatens them.

The tensions that don’t have easy answers

Even the most optimistic strategists we interviewed raised concerns.

The deepest was what we call the learning paradox. Mid-career developers wondered whether they were getting better at software engineering or “just better at prompting.” What does mastery look like when the skill is orchestration rather than implementation?

Deskilling was a related concern. Senior developers learned their craft through years of writing, debugging, and refactoring code. If junior developers delegate from day one, what foundational skills might they skip? Several interviewees questioned whether current onboarding approaches would prepare juniors for AI-augmented work or leave them dependent on tools they don’t fully understand.

Accountability runs underneath both. Developers put their names on code they didn’t directly write. Review and verification become the primary mechanisms for maintaining understanding and accountability, but as velocity increases, can verification norms keep pace, or will the pressure to ship quietly erode the guardrails that make delegation safe?

Three futures, and the choices that decide which we get

The paper closes by extending current patterns into three plausible futures.

Human craft at AI speed. Developers consciously choose to remain hands-on, using AI as an accelerator rather than a substitute. Tasks compress, but the shape of the work remains recognizable.

Orchestration and blended work. Software engineering is defined less by writing code and more by orchestrating intelligent systems and exercising judgment. Verification, mentoring, and design decisions take primary place; implementation gets delegated.

The Clerical Coder. In the paper’s dystopian future, AI generates all production code. Humans rubber-stamp at an industrial scale, accountable for outcomes they don’t fully understand. This is the rise of the “Software Signatory”, where you are rewarded for the volume of approvals rather than the depth of your craft. As the paper warns: If we reward speed without craft, we get volume without depth. When failures happen, the human reviewer is blamed.

There is no single trajectory here. Different teams will land in different places depending on their context and incentives. The outcomes are shaped by what organizations choose to reward and protect.

Final thoughts

AI adoption has increased, and the gap between how developers actually work and how they want to work has not closed. If anything, it has widened.

That suggests the framing of AI as a productivity tool—one that reduces toil and frees up time for meaningful work—may be too simple. The tasks developers find tedious aren’t necessarily the ones they trust AI to handle. And the tasks they trust AI to handle aren’t necessarily the ones creating the gap.

The more useful question for leaders isn’t “what can we automate?” but “are our tools earning the trust to touch what developers find meaningful?” and “what about the work is worth protecting?”

In case you missed it, we just released the DX Annual recording library. You can watch the recordings by clicking through the Sessions page here. We’ll release recordings on the podcast soon as well, stay tuned.

That’s it for this week. Thanks for reading.

Share

Subscribe now

If you weren’t able to attend DX Annual last month, or if you want to revisit specific sessions, all recordings are now available at dxannual.com.

DX Annual brought together ~500 senior engineering leaders from companies like Microsoft, Airbnb, Uber, Vanguard, Dell, and BNY for a day of practitioner-led sessions on what’s working and what isn’t as AI is integrated across the software development lifecycle.

Below are the key themes that emerged across sessions, along with links to the recordings worth watching for each.

The bottleneck of software delivery is no longer writing code

Microsoft’s research shows engineers spend only ~14% of their time writing code. AI has accelerated that 14%, but the other 86%—code review, testing, documentation, deployment approvals—is where meaningful gains are waiting. Vanguard found engineers are 30% faster at coding with AI, but end-to-end cycle time hadn’t moved until they started applying AI across the full lifecycle. Twilio found that top-quartile AI users saw a decline in review turnaround time even though merge time decreased by hours.

Watch: Keynote with Abi Noda & Brian Houck | Vanguard session | Closing panel

AI token spend is top of mind and organizations are measuring it differently

One organization (300–400 engineers) shared their token spend reached $128,000/week and is climbing toward $150,000. They operate with no caps, believing the opportunity cost of slowing adoption outweighs the bill. Some individual developers spend up to $20,000/month. Despite the headline number, their cost per pull request is actually decreasing as throughput increases.

1Password framed the AI token bill as the new cloud bill—something that must be managed with the same rigor as AWS spend. Their advice: stop using high-powered models for simple tasks, and forward-project consumption to negotiate better per-token rates with providers.

Etsy cautioned against top-down usage mandates that incentivize “tokenmaxxing”: employees generating unnecessary activity just to hit metrics.

Microsoft argued that leaders have a responsibility to create headroom to learn, which includes spending tokens on experimentation that may not produce immediate value. The fastest learners will have a competitive advantage. (They acknowledged Microsoft is in a unique position here.)

Watch: Opening panel | Closing panel

Structured training and skill quality matter more than tool choice

Indeed put ~2,000 engineers through a structured, tool-agnostic AI course. Engineers who completed it saw a 36% reduction in coding time; those who didn’t saw zero change—despite using the same tools at 97% adoption. Another big tech company took a complementary approach: narrowed tool options to two, created one-click setup, and built a champion network of ~50 engineers running peer-led onboarding sessions.

Similarly, Microsoft observed that most engineers, when under pressure, revert to single-threaded, pairing-style workflows rather than orchestrating teams of agents.

Watch: Indeed session | Closing panel

Leaders are incorporating more outcome-focused metrics to measure AI impact

Uber is evolving their measurement to include feature velocity, which captures value delivered regardless of whether a human or agent wrote the code. Dropbox shared that they adopted feature velocity as their primary measure of engineering effectiveness for the same reason: it ties directly to outcomes, not activity.

Some organizations are measuring AI through three areas, covering cost management (spend by team and project), engineering effectiveness (how well AI is being used), and value impact (feature velocity, quality indicators).

Watch: Uber session | Dropbox session

Productivity gains require daily, sustained AI usage

Airbnb segmented engineers by daily AI usage and found the returns aren’t linear. Engineers using AI 4+ hours per day more than doubled their output versus pre-AI baselines. Casual users saw only modest improvement. Organization-wide, Airbnb is seeing 65% higher throughput and 59% AI-authored code with no mandate in place. Additionally, Intercom showed what sustained usage looks like at scale: 95.9% of PRs authored by Claude, throughput doubled in 9 months, defect backlog down over 50%.

Watch: Airbnb session | Intercom session

PMs and designers are shipping code, and organizations are adapting

Airbnb has 2× as many AI users as developers—PMs, data scientists, designers, even their finance team independently onboarded themselves to VS Code. Mercari saw a 45.7% drop in accounting tasks and 60% reduction in help desk workload from AI tools operated by non-engineering teams. Vanguard is embedding AI across PMs, designers, and QA, treating the full team as the unit of productivity improvement.

Watch: Airbnb session | Mercari session | Vanguard session

All sessions

For the full list of recordings, visit the Sessions page at dxannual.com.

DX Annual was an invite-only event. Stay tuned for 2027 dates to be released soon.

Share

Subscribe now

In an opening session at DX Annual, Abi hosted a discussion with Tim Bozarth (CVP, CoreAI at Microsoft), Nancy Wang (CTO, 1Password), and Taroon Mandhana (CTO of AI & Teamwork at Atlassian) to learn how they’re adapting to the impact AI is having now and in the future. The panel covered topics such as org design, managing costs, and how the developer role is changing.

Below is a lightly edited excerpt from the discussion. You can watch the recording here. Stay tuned for the full DX Annual recording library to be released next week.

What does an AI-native SDLC look like?

Tim: If we use a five-stage frame: plan, create, validate, deploy, operate. Create is already changing. AI is incredibly good at writing a line of code. Whether it’s good at building complex systems remains to be seen. Historically, roughly 80% of engineering time goes to operate, 10 to 15% to create, and the remainder splits across plan, validate, and deploy. In the most effective teams, that’s inverting: plan and validate now consume the majority of time, because create and operate are compressing. Those are the stages where humans are tastemakers, where understanding of craft matters most. But don’t delegate validate to AI yet. We still need humans in the loop for important systems. And don’t delegate security to AI. Use it for pen testing, use it for red teams, use it to battle-harden your system, but don’t trust it to deliver secure products on its own.

“Roughly 80% of engineering time goes to operate. In the most effective teams, that’s inverting: plan and validate now consume the majority of time.”

Nancy: We’ve stopped writing full-length PRDs at 1Password. Teams build prototypes and put them in front of customers instead. That’s eliminated almost half of the back-and-forth where engineering asks product, “How do you handle this edge case?” You’ve already answered that during the plan and validate phase. But think about the SDLC as a pipe: adding more to the front means you add more to the back. More makers writing PRs means more code reviews, more reliability and maintainability work. We’re running an experiment with a reinforcement learning lab to build a DevOps agent customized to our environment, trained on real incident data and how our engineers respond. The goal is to delegate a significant portion of operate to an agent.

Taroon: The area with the most untapped potential is operate. Our engineers spend an insane amount of time responding to alerts, customer issues, and incidents. We’re seeing agents start to respond to alerts and only wake up a human if it’s real. Post-incident reviews are getting automated. 50% of simple vulnerabilities at Atlassian, things like library version bumps, are now resolved using AI. Accessibility bugs that were taking a backseat are getting done much faster. A lot of this is being run in the background by central dev infra teams to give time back to engineers.

Has AI changed how your engineering orgs are structured?

Taroon: We haven’t restructured the org. What’s changed is how teams form. For zero-to-one projects, we’ve moved to squads of 3 to 4 people. That would have felt too small a year ago, but AI has compressed the building part enough that the bottleneck is now alignment and decision-making. You don’t need 8 engineers when 3 can move faster, as long as they have the context and the authority to make calls without waiting on a chain of approvals. The org chart looks the same. The way work actually happens inside it looks very different.

Tim: Similar story. We haven’t redrawn boxes on an org chart. What we’ve done is shift to eight-week cycles with small, mission-specific v-teams. The point of those teams isn’t sustained delivery. It’s speed of learning. You form a team around a specific question, give them 8 weeks, and then decide whether to continue, absorb the work into a larger team, or stop. That cadence lets us stay responsive without constantly reorganizing.

Nancy: At 1Password, the biggest shift has been in planning horizons. We used to plan 12 to 18 months out. Now we’re down to a single quarter. When the tools and the capabilities are changing this fast, anything beyond 90 days is guesswork. That compression hasn’t required a reorg, but it has required a completely different operating rhythm from leadership. You have to be comfortable making decisions with less certainty and revisiting them more often.

Do any of your organizations mandate AI usage?

Tim: No. We track daily active AI use across Microsoft, and we invest heavily in enablement, training, and incentives to make adoption easy.

“We don’t set targets for usage itself. The metrics we actually care about are speed, ease, and quality.”

Activity is a diagnostic signal. If adoption is low in a particular team, that tells us something is blocking them, and we go figure out what it is.

Taroon: Same philosophy. We have a central team that ensures easy access to tools and removes friction from onboarding. Beyond that, what’s worked is organic champions. In groups of 100 to 200 engineers, someone naturally emerges who’s excited about this stuff and starts showing others what’s possible. We amplify those wins across the company. That’s been far more effective than any top-down mandate.

Nancy: We built a guild of AI champions at 1Password. The thing that actually moves adoption is celebrating concrete examples. When someone uses AI to pull a launch date forward by two weeks, and you tell that story publicly, it does more than any mandate ever could. People see the result and want to figure out how to get there themselves.

How are you managing AI costs?

Taroon: I’m on my third budget forecast since January. Token costs are volatile, and the models and pricing are shifting underneath you constantly. It’s the hardest part of financial planning I’ve dealt with in a while. We literally started to think about managing it like AWS COGS cost because it requires that level of rigor and sophistication.

Nancy: We built an internal SaaS cost management tool that maps token spend by repo and project. Without that visibility, you’re flying blind. That lets us map token spend back to intent, so we know what tokens are actually being spent on. We’re also looking at guardrails and abstractions, like mapping token usage by build volume to a CI automation service, so if something goes out of whack, you can catch it. Having frameworks and hypotheses about how tokens are being spent in your organization can prevent surprise conversations with your CFO. My recommendation to other leaders: negotiate forward-projected commitments with your model providers. If you can commit to volume, you can significantly reduce your per-token cost.

“Treat your AI token bill the same way you’d negotiate a cloud contract.”

Tim: One thing I’d add is that not every token needs to produce direct value. Some of this spend is learning and experimentation. The fastest learners will win, and that means we are going to spend tokens on things that do not create value, but they create learning amongst the workforce. Teams need room to try things that don’t immediately pay off, and leaders need to create headroom for that when they’re reporting to boards.

How do you define the skills profile of a great engineer 3-5 years from now?

Tim: Projecting three to five years out right now is hallucinating. What we focus on is what’s durable. And the pattern we see consistently, inside Microsoft, with our customers, in the startup community, is the maker’s mindset. You’re not attached to a specific tool. You’re oriented toward an objective, toward building something that creates a business outcome, and you drive toward it with whatever you can. That’s what we’re looking for in new hires and encouraging in existing engineers through how we structure incentives, rewards, and how we talk about the speed of experimentation and learning.

Nancy: I’ve shifted between product and engineering roles throughout my career, and the lines between them are blurring fast. What I’d look for is generalists with strong product instincts. The best advice I give new grads right now is to try to span the entire product development lifecycle through the software development lifecycle. Don’t specialize too early. The engineers who are seeing the most impact from AI are the ones who can move across codebases, work across craft, and operate at a higher level of abstraction. They’re not just writing code faster. They’re making better decisions about what to build and how to ship it, because they understand the full picture.

Taroon: I’d add that agency is becoming as important as technical depth. The willingness to step up, have the right conversations, and make decisions without waiting to be told. When AI compresses the time it takes to build something, the differentiator becomes the person who can figure out what to build next and move on it. That’s not a technical skill. It’s a disposition. And it’s something we’re starting to look for explicitly in how we evaluate engineers.

Where are the boundaries when non-engineers contribute code?

Taroon: Designers at Atlassian are submitting PRs. That’s real, it’s happening today, and it’s genuinely useful. The fidelity of early conversations has gone up dramatically because instead of debating a static spec, you’re looking at something interactive. But I want to be honest about the friction. Engineers are escalating daily about quality issues in those contributions. Prototyping by non-engineers is a clear unlock. Shipping to production is a different bar. The teams that are most comfortable accepting contributions from non-engineering roles are the ones with robust test suites and deployment checks already in place. If your right-of-code processes are weak, AI-assisted contributions from anyone, engineers included, are going to cause problems.

More broadly, we’re seeing patterns of duplication and tech debt increasing as people quickly produce features. The maintainability of the code is suffering. It’s prompted us to go back to standardized approaches and more right-of-code quality checks.

Nancy: Our CX associates are now generating PRs for front-end test coverage across browsers and mobile clients. These are people who know the product deeply from a customer perspective, and they’re using AI to translate that knowledge into actual test code. Engineering’s role has shifted. Instead of writing those tests themselves, they’re building the testing harnesses and review processes to evaluate the contributions. It’s a different kind of engineering work, but it’s just as important, and arguably higher leverage.

Tim: There’s a second dimension to this that I think gets overlooked. Yes, more people can now take an idea in their head and turn it into something you can interact with. That’s the prototyping side. But the non-engineers who are getting the most out of AI aren’t contributing code to products. They’re using it to optimize how they work: how they gather and distill information, how they communicate, how they run their own workflows. That doesn’t go to production in the traditional sense, but it goes to production in how the business operates. And that’s happening right now.

You can listen to the full discussion here.

This week’s featured DevProd job openings. See more open roles here.

• Ashby is hiring an Staff Platform Engineer | Remote

• Cashea is hiring an Infrastructure & Developer Productivity Platform Engineering Manager | Remote

• Figma is hiring a Staff Software Engineer, Developer Experience | Remote; US

• Leidos is hiring a Platform Engineer | Remote; US

• Weave is hiring a Senior Platform Engineer, Data Infrastructure | Remote; US

That’s it for this week. Thanks for reading.

Share

Subscribe now

Over the past 16 months, DX has been running a longitudinal study on AI’s impact on engineering velocity across a sample of more than 400 engineering organizations. We found that as AI tool usage increased by an average of 65%, median PR throughput increased by just under 8%. Most organizations are landing in the 5–15% range—a meaningful gain, but far below the 3x or 10x expectations many leaders are being held to.

That gap raises some questions. Why aren’t the gains higher? Where is reclaimed time actually going? And how should leaders be thinking about where to invest next?

This is what and I discussed to kick off the mainstage program at the inaugural DX Annual, which brought together roughly 500 senior engineering and developer productivity leaders for a day focused on navigating the AI era. Brian is a well-known developer productivity researcher, and someone we’ve referenced often in this newsletter—from past interviews with him (here and here) to summaries of papers he has co-authored (like this one and this one). In this conversation, Brian asked me about the study methodology and what sparked this research, as well as what’s limiting gains, how to measure AI’s impact, and how leaders can unlock greater gains.

Today we’re releasing the full report. You can download it here: AI and Engineering Velocity: A Longitudinal Analysis.

What follows is a lightly edited transcript of my conversation with Brian at DX Annual.

Brian: To kick us off, Abi—you’re sitting on one of the largest datasets on how engineers work in the real world. This new report digs into what’s happening as AI adoption scales across the industry. Before we get into the findings, what motivated this research?

Abi: Thanks, Brian. A little background on this study: Nearly every conversation I’ve had with folks over the past few months has started the same way. My CEO, our executives, are expecting astronomical gains because there’s so much hype about AI in the media, but that’s not what we’re seeing on the ground. How do we close that gap or set realistic expectations? How do we even know what to aim for?

I’d bet everyone reading this has experienced that to some extent. So at DX, we started asking ourselves the same question. What can we see in the data about what companies are actually seeing? That’s what we’ve been digging into. This is just the beginning; there’s a lot more to unpack and a lot of nuance, but at least we now have a first look at what’s happened over the past 16 months as AI adoption has matured across companies.

Brian: I’d love to nerd out on research methodology for a minute. Walk us through how DX actually investigated this. How did you design the study and decide which companies to include?

Abi: Our customer network at this point is about 500 companies. For this study, we took a sample of those. We looked for companies that had reached a point of maturity in their developers’ adoption of AI: we defined that as over 75% monthly active usage of AI coding tools. In almost all cases, there’s been a significant rise in adoption over the past 12 months. We also narrowed the scope to companies with over 100 engineers and excluded companies that had gone through major liquidity events, M&A, IPOs, or regulatory changes that could have had an impact.

In terms of methodology, it’s really hard to single out causality. What is AI and what are confounding factors? One approach I see a lot is cross-sectional analysis: comparing developers who use AI to developers who don’t. There are flaws in that approach because oftentimes the developers using AI were already the ones coding the most. They always look better.

What we did instead is look at longitudinal data at the organizational level. As AI adoption has matured, what has been the impact on organizational velocity and throughput?

Brian: What were your hypotheses, and what did you actually find?

Abi: As a developer myself, I had some gut-feel hypotheses, but honestly, from a study standpoint, we really didn’t know. We were pushing the team, asking “what’s the answer?” We wanted to know just as much as you probably do.

What we ultimately found is that most organizations fall in the 10-15% range in terms of PR throughput increase. The actual median was around 8%, the mean was around 11%. So a real gain, but well below what most leaders expect.

It’s much more modest than many CEOs, who are talking to their CEO friends about the 10x gains they’re supposedly delivering, would expect. In our engagements with companies and customers, it wasn’t completely surprising. There are a lot of factors underlying that, which we’ll get into.

Brian: One of the central themes of the SPACE framework is that developer productivity is nuanced and about much more than counting activities. So given that, why did you choose PR throughput as your primary measure?

Abi: Measuring velocity or productivity is inherently challenging. We did look across a number of different metrics, including the full spectrum of SPACE and the DX Core 4. But for what we wanted to focus on and publish, we felt PR throughput was the most relevant and practical right now, mostly because that’s what we see most organizations talking about and focusing on. We wanted to meet the world where it’s at.

We also explored our proprietary metric TrueThroughput, which uses AI to weight each PR by relative size and complexity so it takes some of the noise out. We liked that signal, but again, for relevance and practicality, PR throughput is a more accessible metric. We have more data on other metrics and how those have been affected, and we’ll eventually publish those as well.

Brian: I suspect that, for many, a 5%, 7%, 10% increase in PR throughput matches what they’ve been feeling. But as you mentioned, when I talk to business leaders, they’re often expecting 20, 30, 40, 50% increases in productivity. So why do you think the gains are lower than what people outside the industry might expect?

Abi: Our team conducted follow-up interviews to really dig into that question. There were a number of factors. These are just the top themes, not the full list.

The top one probably wouldn’t surprise most of us: coding is not the primary bottleneck for engineers. Brian, at Microsoft, you published a study on where engineering time is spent, and only about 14% of developer time is spent coding. It’s a very small part of where our time and money goes. So if AI is only optimizing that slice, it partially explains why the enterprise velocity gains are limited.

We also heard how AI tools and new automation introduce new bottlenecks. Things like review burden, technical debt, cognitive debt—which is a newer concept we can talk about. We also heard about challenges with adoption—a lot of social friction and cultural clashes that are inhibiting full adoption. There were a few more, but those were the major themes.

Brian: You don’t have to go far out in the distribution before you see some organizations with substantially higher gains. Any time the median and mean are that different, it implies outliers. So what set apart some of the companies with disproportionately high gains from the more typical experience?

Abi: We don’t have a great answer to this yet. Our first focus was on why the gains aren’t higher. The next question is: for those where gains are higher, why?

There are two parts to how we want to approach that. One is that we’re working to tease out whether the companies at the top of the range are seeing real, material gains or superficial, inflated numbers. We want to peel back the onion.

Two, assuming the gains are real, we want to understand what strategies and factors are driving those results. I think a lot of it boils down to what I’ve seen anecdotally: an all-in culture, a fully bought-in organization with centralized rollout and championing of AI tools. It’s more of a cultural shift toward infusing AI into the entire SDLC, not just coding, with an aligned push to drive adoption.

Brian: That aligns with some recent research I published showing that even something like leadership advocacy significantly increased the gains people saw from AI. You said something earlier that I think is really important and often misunderstood: software engineering is so much more than coding. Coding may be central to our identity as developers, but it’s only about 14% of our day.

Brian: One of the things I’ve been seeing in my research is that, yes, we’re producing 5, 7, 10, 15% more pull requests, but we’re doing it substantially more efficiently. For hands-on keyboard time spent coding, we’re producing about 40% more PRs per hour. That hints that we are reclaiming some coding time. Do you have any idea where that time is getting reinvested?

Abi: This has been a similarly perplexing question. For the past six months, I’ve heard customers asking: we’re seeing meaningful self-reported time savings from developers, but we’re not seeing that show up in our output and throughput. Where’s the disconnect? Where’s that time going?

We don’t know yet—this requires more research. But one preliminary hypothesis is that if developers only spend 14% of their time coding, then it makes sense that the time they’re recouping is ratably distributed across their activities. Only 14% of the time savings go back into coding, which would explain why we don’t see one-to-one output gains mapping to time savings.

Another hypothesis ties back to why the gains aren’t higher: some of the time savings come with side effects that require more time, like overseeing the work, QAing it, reviewing it. There’s also the lag factor: what do developers do while they’re waiting for agents to produce the code?

So those are some of the hypotheses, but we don’t have a clear answer yet.

Brian: There’s a reason we don’t just measure lines of code as a good measure of productivity. Writing more code isn’t always the right answer. As we increase velocity, what are some of the potential unwanted side effects you’ve been seeing?

Abi: Quality and cost are the two I hear customers talking about constantly. Cost is something we’re all trying to wrangle or at least get visibility into. Quality is something we all understand is an underlying risk, but I think it’s still early days. We did see Amazon go very public with this, but there’s some delay between the technical debt we may be creating and the consequences that follow.

The biggest thing I’ve been talking to customers about is a cultural risk I’ve been calling false velocity. I see this in so many organizations right now. And to some extent, even developer productivity leaders can fall into this trap, being so focused on showing off how much faster and more prolific we are with AI, without asking whether it’s leading to meaningful improvement.

You probably have engineers in your organization showing off really impressive things they can do with Claude or whatever tool. But are we asking: is your team’s product velocity actually increasing? Leaders are talking about how many more PRs and lines of code they’re generating, but is their roadmap actually accelerating? Is the quality of their products and code sustainable? There’s a real risk of being so focused on demonstrating what we can do that we’re not paying attention to whether we’re actually getting better—whether we’re materially improving our businesses.

Brian: So what’s your recommendation to engineering leaders who want to apply AI beyond just code generation? Where should they be looking?

That’s the biggest question I’m hearing from leaders right now. A lot of folks are at a point where they’ve rolled out one or more popular coding tools. Adoption is in a fairly satisfactory place, and they’re asking: what next? Especially if you’re sitting at that 10% number and asking how to get further.

The things I’m hearing about and seeing are, first, looking left and right of code. We’ve accelerated coding to some extent, but if that’s only 14% of where our time and money goes, what about the other 86%? How do we accelerate and optimize that?

Second, beyond thinking about the SDLC, there’s the question of how we improve the developer experience more deeply. Things like deep work and documentation—the friction points developers experience. How can we leverage AI to materially improve those?

And the third theme is this idea—there are many names for it—autonomous engineering, async engineering, background agents. I think today most of us are focused on leveraging AI to accelerate human work. Humans are still in the cockpit, steering and monitoring the work of the agents. The question is: how do we complement that acceleration with augmentation? Meaning more autonomous agents that are truly augmenting your human workforce and working in parallel, not just underneath your developers. Those are the three big themes.

Brian: You and I are both huge fans of Dr. Margaret-Anne Storey’s work. She’s been talking a lot about cognitive debt and the human cost of these AI transformations. What have you been looking at in that area?

Abi: I’m a big fan of that research. I think it falls under what I was talking about earlier in terms of risks and the time delay; we’re not always thinking about what the consequences might be 12 months from now if we adopt certain ways of working.

The loss of human understanding of the systems we’re building is a really interesting risk. There are different takes on how material it is. Some people argue it doesn’t matter if humans don’t deeply understand the systems, because they can use AI to quickly regain that understanding when needed. Others argue: no, if you need to call the mechanic, they should have mastery over the machine. I don’t think we know yet, but it’s a valid question.

Brian: We’ve hinted at it throughout this conversation—are we actually delivering innovation faster, or are we just moving bottlenecks around? My first reaction to an unanswered question is always: how do we measure it? Any thoughts on how leaders should be thinking about measurement as AI transforms how we work?

Abi: Our approach to measurement—some things stay the same and some things need to evolve. How we think about the overall software organization, things like quality and velocity, I think those stay the same. And especially if we’re trying to understand how things are changing with AI, you need consistent measures pre, post, and during this transformation.

But there are also new tools, new ways of working, new workflows being born every week. Measuring these new ways of working requires new approaches.

I’ll touch on two things. One is that I think increasingly there’s a need to separate how you’re measuring and framing AI’s impact. You should be thinking about acceleration—how much faster are our humans—as one bucket. And augmentation—how much more capacity are we generating with agents—as a second bucket. Thinking of those as two separate components in your overall formula is useful.

The question of how to measure agents is especially interesting. We’ve been exploring this idea of agent hourly rate: the human-equivalent hours an agent delivers, divided by its cost. That gives you a return-on-investment figure you can compare to human hourly rate, which is an interesting comparison point.

Something really new—not even in our minds last September—is this idea of agent experience. In the same way DX was founded on the idea of measuring developer effectiveness by going to developers and getting signal from them, we’ve applied a similar approach to agents. How do you measure AI agent effectiveness? You go to the agents and get feedback from them on where their bottlenecks and constraints are. We’ve just rolled that out. We’re surveying agents, which is pretty wild.

Brian: That is wild. To wrap up our conversation, give me one quick finding. What’s one thing you’ve learned about making agents more effective?

Abi: You have to ask them, just like we ask our developers. It’s really early days, but we’ve initially begun measuring four factors. For example, we ask the agents: how were the requirements you were given? How easily were you able to understand the codebase you were working in? How well were you steered by the human you were pairing with? We get quantitative metrics from that as well as qualitative feedback—the agent explains where it had challenges or bottlenecks. That’s the idea.

Brian: I love it—the same principles of measurement apply. That is unfortunately all the time we have, but I think we covered a lot of ground.

Abi: A special thank you to Brian, and everyone who attended DX Annual. Stay tuned for the session recordings, coming soon.

This week’s featured DevProd job openings. See more open roles here.

• Cashea is hiring an Infrastructure & Developer Productivity Platform Engineering Manager | Remote

• BNY is hiring an SVP, Application Development Manager | Pittsburgh, PA

• Figma is hiring a Staff Software Engineer, Developer Experience | Remote; US

• Leidos is hiring a Platform Engineer | Remote; US

• Weave is hiring a Senior Platform Engineer, Data Infrastructure | Remote; US

That’s it for this week. Thanks for reading.

Share

Subscribe now

Last week we held our first-ever DX Annual event in San Francisco, bringing together nearly 500 engineering and platform leaders for a full day focused on developer productivity and AI. Thank you to everyone who joined us. Session recordings will be available in the coming weeks.

Abi: This week we have a guest post from Dr. Margaret-Anne Storey, a Professor of Computer Science and a Canada Research Chair in Human and Social Aspects of Software Engineering. Margaret-Anne is one of the most widely published researchers on developer productivity, having co-authored the SPACE and DevEx frameworks amongst many other works.

Margaret-Anne: Earlier this year I published two posts exploring how generative and agentic AI may be quietly shifting where the most significant risks in software development lie, away from technical debt and code quality, and toward something harder to see and measure: the erosion of shared understanding across teams. This is what I refer to as cognitive debt. The response to these posts surprised me, as practitioners confirmed that cognitive debt was a significant challenge they were facing. They also proposed concrete suggestions for recognizing and mitigating cognitive debt. I’m combining both posts below, with light edits, as they tell a connected story.

For readers who want to go deeper, two papers extend these ideas. In Theory of Troubleshooting, co-authored with Arty Starr, we ground the cognitive debt concerns in cognitive science, showing how making sense of unexpected system behavior places considerable demands on working memory and attention, and how prolonged troubleshooting leads to cognitive fatigue with real implications for developer well-being. In From Technical Debt to Cognitive and Intent Debt I propose a Triple Debt Model that adds a third dimension to this framework: intent debt, the erosion of externalized rationale that both developers and AI agents need to work with to safely maintain and evolve a codebase.

How generative and agentic AI shift concern from technical debt to cognitive debt

The term technical debt is often used to refer to the accumulation of design or implementation choices that later make the software harder and more costly to understand, modify, or extend over time. Technical debt nicely captures that “human understanding” also matters, but the words “technical debt” conjure up the notion that the accrued debt is a property of the code and effort needs to be spent on removing that debt from code.

Cognitive debt, a term gaining traction recently, instead communicates the notion that the debt compounded from going fast lives in the brains of the developers and affects their lived experiences and abilities to “go fast” or to make changes. Even if AI agents produce code that could be easy to understand, the humans involved may have simply lost the plot and may not understand what the program is supposed to do, how their intentions were implemented, or how to possibly change it. Where cognitive load is what developers experience in the moment, cognitive debt is a project-level property, capturing how a team loses understanding over time.

Cognitive debt is likely a much bigger threat than technical debt, as AI and agents are adopted. Peter Naur reminded us some decades ago that a program is more than its source code. Rather a program is a theory that lives in the minds of the developer(s) capturing what the program does, how developer intentions are implemented, and how the program can be changed over time. Usually this theory is not just in the minds of one developer but fragments of this theory are distributed across the minds of many, if not thousands, of other developers.

I saw this dynamic play out vividly in an entrepreneurship course I taught recently. Student teams were building software products over the semester, moving quickly to ship features and meet milestones. But by weeks 7 or 8, one team hit a wall. They could no longer make even simple changes without breaking something unexpected. When I met with them, the team initially blamed technical debt: messy code, poor architecture, hurried implementations. But as we dug deeper, the real problem emerged: no one on the team could explain why certain design decisions had been made or how different parts of the system were supposed to work together. The code might have been messy, but the bigger issue was that the theory of the system, their shared understanding, had fragmented or disappeared entirely. They had accumulated cognitive debt across their team faster than technical debt, and it paralyzed them.

This dynamic echoes a classic lesson from Fred Brooks’ Mythical Man-Month. Adding more agents to a project may add more coordination overhead, invisible decisions, and thus cognitive load. Of course, agents can also be used to manage cognitive load by summarizing what changes have been made and how, but the core constraints of human memory and working capacity will be stretched with the push for speed at all costs. The reluctance to slow down and to do the work that Kent Beck calls “make the hard change easy” is what will lead to cognitive debt and cognitive load in the future.

In a breakout session at a recent Future of Software Engineering Retreat (arranged by Martin Fowler and Thoughtworks) we discussed how developers need to slow down and use practices such as pair programming, refactoring, and test-driven development to address technical debt AND cognitive debt. By slowing down and following these practices, cognitive debt can also be reduced and shared understanding across developers and teams rebuilt.

But what can teams do concretely as AI and agents become more prevalent? First, they may need to recognize that velocity without understanding is not sustainable. Teams should establish cognitive debt mitigation strategies. For example, they may wish to require that at least one human on the team fully understands each AI-generated change before it ships, document not just what changed but why, and create regular checkpoints where the team rebuilds shared understanding through code reviews, retrospectives, or knowledge-sharing sessions.

Second, we need better ways to detect cognitive debt before it becomes crippling. Warning signs include: team members hesitating to make changes for fear of unintended consequences, increased reliance on “tribal knowledge” held by just one or two people, or a growing sense that the system is becoming a black box. These may be signals that the shared theory is eroding.

Finally, this phenomenon demands serious research attention. How do we measure cognitive debt? What practices are most effective at preventing or reducing it in AI-augmented development environments? How does cognitive debt scale across distributed teams or open-source projects where the “theory” must be reconstructed by newcomers? As generative and agentic AI reshape how software is built, understanding and managing cognitive debt may be one of the most important challenges our field faces.

I explored these questions further in a recent keynote at the ICSE Technical Debt Conference and Panel. Cognitive debt tends not to announce itself through failing builds or subtle bugs after deployment, but rather shows up through a silent loss of shared theory. As generative and agentic AI accelerate development, protecting that shared theory of what the software does and how it can change may matter more for long-term software health than any single metric of speed or output.

Discussion: What I’m hearing about cognitive debt (so far)

After publishing the original post (above) on cognitive debt, it sparked thoughtful discussion across different communities. I’ve synthesized what I’m hearing below, and am connecting it to other reflections I’ve been reading.

There’s a growing concern about shared understanding

Several practitioners, including Simon Willison and others on a Hacker News discussion of a Martin Fowler article, describe experiencing cognitive debt directly. They talk about getting lost in their own projects and finding it harder to confidently add new features. They can move faster, but they lose the deeper sensemaking that connects decisions to intent, and intent to code.

This is not just about code quality. It is about whether individual developers and product teams can maintain a coherent mental model of what the system is doing and why.

Across these discussions, one theme is consistent: velocity can outpace understanding.

Cognitive debt hurts developers, not just the software

Technical debt lives in the code. Cognitive debt lives in people.

When shared understanding erodes, the pain shows up in:

• Loss of confidence when making changes

• Heavier review burden

• Debugging friction

• Slower onboarding

• Stress and fatigue

The software may be “working”, but the theory of the system becomes harder to access and keep track of. The cost is not only structural. It is experiential.

Siddhant Khare has written about AI fatigue. Steve Yegge reflects on burnout emerging from AI-accelerated development. Annie Vella eloquently writes about the emotional and cognitive experience of uncertainty when systems become harder to reason about. These perspectives reinforce that this is not just an engineering discipline issue, but one that affects how developers feel and function.

Cognitive debt, like technical debt, must be repaid

Martin Fowler notes that, like technical debt, cognitive debt must eventually be repaid. I agree.

But rebuilding lost knowledge requires restoring the distributed theory of the system. That includes capturing intent, the rationale behind decisions, key constraints, and how the architecture supports change. That theory is not stored in code alone. It is distributed across people, documentation, tests, conversations, tooling, and increasingly, AI agents.

Repayment means maintaining all of these, not just refactoring code or updating architecture documents. Under pressure to move quickly, whether in startups racing to learn or in large organizations pushing AI adoption, that repayment can feel expensive and easy to defer.

“This is just engineering”, but incentives are changing

Several commenters, including Michael Würsch, argue that cognitive debt reflects a failure of good engineering discipline. Clear specifications, rigorous reviews, extensive testing, and explicit architecture documentation should prevent knowledge loss.

In principle, I agree. But in practice, the incentives are shifting. AI lowers the cost of producing structure. It becomes easier for structure to evolve faster than shared understanding can stabilize. Even disciplined teams must consciously throttle or shape their practices to keep understanding aligned with change.

Specifications and documents are not sufficient if they are not living artifacts that teams actively engage with.

Emerging mitigation strategies

Encouragingly, many readers shared how they are mitigating cognitive debt.

They describe:

• More rigorous review practices

• Writing tests that capture intent

• Updating design documents continuously

• Treating prototypes as disposable

Some also describe using AI to reduce the cost of these practices, and even to support cognitive tracking, dependency management, and explanation.

Used deliberately, AI may help make cognitive work more visible rather than obscuring it.

The open question: How will high-performing teams adapt?

High-performing teams have always managed technical debt intentionally. As AI is adopted by startups and large companies, the question becomes how teams will manage cognitive debt.

How will they shape socio-technical practices and tools to externalize intent and sustain shared understanding? How will they use Generative and Agentic AI not only to accelerate code production, but to maintain their collective theory?

As AI reduces technical friction, shared understanding may become the bottleneck on performance.

I am continuing to watch how this evolves. If you are seeing mitigation practices that work in real teams, I would love to learn from them. As mentioned above, check out the article that goes deeper into how to recognize and mitigate cognitive debt and also proposes using the concept of intent debt to capture when decisions and the why behind a system are not captured for future humans and agents to refer to.

This week’s featured DevProd job openings. See more open roles here.

• Cashea is hiring an Infrastructure & Developer Productivity Platform Engineering Manager | Remote

• BNY is hiring an SVP, Application Development Manager | Pittsburgh, PA

• Figma is hiring a Staff Software Engineer, Developer Experience | Remote; US

• Leidos is hiring a Platform Engineer | Remote; US

• Reddit is hiring a Staff Software Engineer, Developer Experience | Remote; US

• Weave is hiring a Senior Platform Engineer, Data Infrastructure | Remote; US

That’s it for this week. Thanks for reading.

Share

Subscribe now

📅 Live research readout: Sign up here to join Abi for a readout and interactive discussion about the impact of AI on engineering velocity.

Since releasing our Q4 2025 AI Impact Report, reasoning models have improved, agentic workflows have gained traction, and teams are finding new ways to integrate AI into how they build.

Each quarter, we analyze the prior quarter’s data to benchmark AI adoption rates, productivity impact, and how teams are using these tools. For this edition, we expanded our dataset by over 40% more data points to give you a more representative picture.

We’ve compiled this analysis into our AI-Assisted Engineering: Q1 2026 Impact Report, which we are releasing today.

The full report covers industry adoption rates (now at 93%), tooling benchmarks, and the growing risks of “Shadow AI.” You can read the full report here. Below are two previews of interesting shifts we’ve observed since last quarter.

Engineering managers are shipping 4x more code than six months ago

Perhaps the most dramatic structural shift we observed is the changing nature of the engineering manager role.

In Q4, we noted that engineering managers who used AI daily were shipping twice as many pull requests over the period of time the sample was taken. That figure has doubled again: engineering managers are now shipping 4x as much code.

Why this matters: The definition of a developer is expanding. As agentic tools and AI assistants drastically lower the barrier to context-switching and boilerplate generation, managers are finding it far easier to get back into the codebase without sacrificing their leadership responsibilities. Coupled with an industry-wide trend of flattening organizational structures, AI is enabling the true return of the player-coach, allowing EMs to stay technically sharp and contribute directly to the product.

The question then is: if EMs can meaningfully contribute code again, should org structures evolve to reflect that? From what we’re hearing, it’s top of mind for many leaders, but most are still in “watch and learn” mode. No one we’ve spoken to has made structural changes based on this shift yet, but we’ll continue to monitor this trend.

Junior engineers now save more time with AI than Staff+ engineers

In our Q4 2025 report, Staff+ engineers who used AI daily were regaining more time than any other tenure band. Just three months later, the data tells a different story.

Junior engineers who use AI daily are now saving 4.9 hours per week, edging out daily Staff+ users who are saving 4.8 hours.

While time savings alone doesn’t tell a full story—in fact, this shift should come with heightened attention to quality, maintainability, and security—it is interesting to see this change in work impact among junior engineers.

Why this matters: Staff+ engineers, who naturally spend more time mentoring and reviewing rather than writing net-new code, are still seeing benefits, but junior engineers have officially taken the lead in total time saved. We provide a look at AI’s impact on quality in the full report; we’ll continue to pay close attention to quality measures amongst junior developer cohorts in the future.

What else is in the report?

The full Q1 2026 report provides comprehensive data and visual benchmarks on:

• Rust-specific improvements: How improvements in model reasoning and agentic loops have caused Rust to leapfrog other modern languages in AI-driven time savings.

• Quality volatility: Why some companies are seeing change failure rates swing by 2% (a 50% increase in defects), and why automated testing is no longer optional.

• The shadow AI risk: How developers are bypassing enterprise guardrails, and why acceptable use policies are critical for ensuring safe use of AI

• Small vs. large enterprises: Why companies with fewer than 200 developers are outpacing large enterprises in efficiency gains.

Final thoughts

As always, a reminder: there is no “average” experience with AI impact. Looking at industry trends and averages like those in this report can help contextualize your performance and do some pattern-matching, but I want to caution against using these averages as a guide to what your performance should look like. The truth is that AI impact is very asymmetrical and looks very different for each organization. Some are doing very well when it comes to throughput, but quality is suffering; some have done a great job of using AI for migrations, but are struggling to incorporate AI in testing and release processes. Averages abstract away the nuances of each company’s transformation journey. The best way to use this information is to enhance—not replace—your own metrics and AI strategy.

This week’s featured DevProd job openings. See more open roles here.

• Cashea is hiring an Infrastructure & Developer Productivity Platform Engineering Manager | Remote

• BNY is hiring an SVP, Application Development Manager | Pittsburgh, PA

• Figma is hiring a Staff Software Engineer, Developer Experience | Remote; US

• Leidos is hiring a Platform Engineer | Remote; US

• Reddit is hiring a Staff Software Engineer, Developer Experience | Remote; US

• Weave is hiring a Senior Platform Engineer, Data Infrastructure | Remote; US

That’s it for this week. Thanks for reading.

Share

In this episode, I’m joined by Eleanor Millman, Senior Staff Product Manager, and Mina Tawadrous, Associate Director of Platform Engineering at SiriusXM, to discuss how platform teams can scale prioritization without relying on revenue.

We talk through how SiriusXM moved beyond RICE to build a custom framework for internal platforms, using weighted factors like developer speed, reliability, cost, and trust to guide decisions across teams.

We also explore their concept of “assumptions as code,” in which teams store and reuse assumptions in a central repository to reduce misalignment and improve decision-making, with AI helping to surface and validate those assumptions.

We close with how this system is shaping SiriusXM’s 2026 prioritization approach and what it signals about a broader shift toward builder-driven product development.

Some takeaways:

Prioritization breaks without a shared system

• Prioritization does not scale naturally across teams. What works for one team breaks down at the org level with multiple stakeholders and competing requests

• Platform teams lack a clear revenue signal. Unlike product teams, they must prioritize based on indirect impact

• A shared framework aligns decisions. Without it, prioritization defaults to local optimization and noise

RICE is a starting point, not a solution

• Standard frameworks miss key dimensions for platform teams. Urgency and indirect impact are not captured well

• “Impact” needs to be decomposed. SiriusXM broke it into developer speed, reliability, cost, security, and more

• The framework must evolve over time. Iteration was critical to making it useful in practice

Weighting forces real tradeoffs

• You cannot prioritize everything at once. Increasing one dimension (like cost) necessarily deprioritizes others

• Assigning weights makes decisions explicit. Leaders must commit to what matters this quarter

• The output drives alignment across teams. A single prioritized list reduces cross-team conflicts

Data and conversation work together

• The framework creates a place to attach data. Metrics like reliability scores inform prioritization decisions

• Disagreements surface quickly. Teams can see where assumptions or inputs differ

• Conversations, not just scores, drive alignment. The value comes from debating inputs, not just ranking outputs

Assumptions are the real bottleneck

• Most disagreements come from hidden assumptions. Teams often believe they are aligned when they are not

• Assumptions can be conflicting, invisible, or stale. All three create friction in decision-making

• Making assumptions explicit improves clarity. It becomes easier to validate or challenge them

Storing assumptions as code scales learning

• Assumptions are stored in a central repository. User research and data become reusable across teams

• This reduces duplicated effort. Teams don’t need to rediscover the same insights repeatedly

• It creates a shared source of truth. Assumptions become visible, versioned, and easier to update

In this episode, we cover:

(00:00) Intro

(01:17) Mina’s role and path into platform engineering

(02:03) Eleanor’s background and shift into product

(03:15) Scaling prioritization across platform engineering teams

(05:41) Aligning platform priorities with stakeholders

(09:08) Evolving RICE into a platform-specific prioritization framework

(11:33) Iterating on the prioritization framework over time

(16:57) How the framework, data, and conversations drive alignment

(19:06) Storing assumptions as code in a central repository

(26:47) Resolving assumption conflicts with user interviews

(30:47) How stored assumptions integrate with AI workflows

(35:30) Standard mode and different user personas

(37:20) The industry shift towards builders

(41:04) The challenges of platform engineering

(43:36) How SiriusXM is prioritizing in 2026

Referenced:

• Measuring AI code assistants and agents

• SiriusXM

• VMware

• How SiriusXM revamped their platform and developer experience

• RICE Scoring Model | Prioritization Method Overview

• The evaporating cloud: A tool for resolving workplace conflict

Subscribe now

📅 Live research readout: Sign up here to join Abi for a readout and interactive discussion about an upcoming report that looks at the impact of AI on engineering velocity.

Onboarding new hires has always been an expensive and time-consuming process, and an area where AI has the opportunity to have a meaningful impact. In Q4 2025, when we looked at Time to 10th PR (a measure we use to track ramp-up time), we saw AI already having a dramatic effect. In some companies, Time to 10th PR was cut in half: from 91 days with no AI usage to 49 days with daily AI use.

We revisit this data quarterly. In our most recent analysis, we’re seeing that trend continue: onboarding time is faster today than it was in Q4 of last year.

For this latest cut, we analyzed data from a random sample of 400 companies during the period from October 2025 to February of 2026. We measured the average number of days between a developer’s start date and their 10th merged PR. We specifically looked at an aggregate of engineers who showed daily use of AI and compared how ramp-up time changed from October 2025 to February 2026.

The dataset includes large, global organizations with 500+ developers, spanning both tech and non‑tech companies.

As of April 2026, the current average Time to 10th PR across our dataset is 33 days, down from 39 days in Q4 2025. That represents a further ~15% decrease quarter-over-quarter, and more than a 50% reduction since Q1 2024, before AI usage was widespread. The change this quarter isn’t as dramatic as the 50% reduction we saw last year, but it is still meaningful. This trend begs the question: will onboarding times continue to fall, or are we approaching a natural floor where many of the remaining steps are not easily compressed by AI? As more agentic solutions are devised for onboarding processes, we may see this figure continue to decline.

Part of the improvement we’re seeing since last quarter comes from the fact that AI is no longer an optional, individual choice in many organizations. Developers are guided to use AI to ask questions about the codebase, to explain architectural decisions, and to propose draft changes that senior engineers can review. The result is that the “figuring things out” phase is shorter.

At the same time, Time to 10th PR is a narrow measure. It tells us how quickly someone reaches a specific milestone, but it does not, on its own, tell us about the quality of those changes, the amount of rework they generate, or the depth of understanding new hires have of the systems they’re touching. That will be something we explore more in future analyses.

Stay tuned for the full report coming soon.

This week’s featured DevProd job openings. See more open roles here.

• Awardco is hiring an Engineering Manager, Cloud Platform | San Francisco, CA

• BNY is hiring an SVP, Application Development Manager | Pittsburgh, PA

• Figma is hiring a Staff Software Engineer, Developer Experience | Remote; US

• Leidos is hiring a Platform Engineer | Remote; US

• Reddit is hiring a Staff Software Engineer, Developer Experience | Remote; US

• Weave is hiring a Senior Platform Engineer, Data Infrastructure | Remote; US

That’s it for this week. Thanks for reading.

Share

In this special episode of Engineering Enablement, I welcome back Jesse Adametz, this time as host.

In our conversation, we explore how AI is showing up across the SDLC, not just in code generation, and how it is shifting bottlenecks across the development process. We unpack what “AI readiness” actually means in practice, and why it often comes down to developer experience fundamentals like documentation, environments, and feedback loops.

We also discuss why enablement matters more than tool choice, how teams are thinking about measuring ROI, and what changes as background agents become more common. Finally, we explore how the role of the engineer may evolve, what questions teams are still trying to answer, and the challenges of non-engineers contributing to codebases.

Some takeaways:

AI is expanding beyond coding into the full SDLC

• The focus has shifted from code generation to the entire software lifecycle. Teams are applying AI to planning, prototyping, review, and documentation—not just writing code.

AI readiness is a developer experience problem

• The biggest blockers to AI adoption are long-standing DX gaps. Missing documentation, inconsistent environments, weak CI, and unclear system boundaries all limit effectiveness.

• Tool choice is not the primary driver of success. Models and tools are evolving too quickly for this to be a durable advantage.

• Some organizations are formalizing AI enablement as a function. Dedicated teams are emerging to drive adoption and share practices.

Measuring AI ROI is messy and still evolving

• Correlation vs causation makes attribution difficult. High AI usage often correlates with already high-performing engineers.

• Longitudinal analysis is more reliable than snapshots. Tracking changes over time gives better insight into impact.

• Token spend introduces real cost considerations. AI creates a direct, variable cost that organizations must evaluate.

AI impact falls into two buckets: amplification and augmentation

• Amplification improves human productivity. This includes higher throughput, time savings, and better developer experience.

• Augmentation extends capacity beyond humans. Agents begin to act as additional “headcount,” completing work independently.

• These require different measurement approaches. Amplification focuses on human output, while augmentation focuses on agent output relative to cost.

Background agents shift how work gets done and where bottlenecks appear

• Agents enable work to happen outside the human loop. Tasks can be completed asynchronously and proactively.

• This changes the developer role. Engineers move toward reviewing, guiding, and orchestrating agent output.

• Human workflows can become the bottleneck. If agents produce work faster than humans can process it, the constraint shifts.

• This reframes productivity. The question becomes where human involvement adds the most value.

Specs and documentation are becoming critical infrastructure

• AI makes documentation a core dependency. It directly impacts the quality of outputs.

• Poor documentation leads to poor results. Agents can duplicate systems or make incorrect assumptions without context.

• Documentation is shifting from optional to essential. It is now foundational for both human and AI productivity.

In this episode, we cover:

(00:00) Intro

(02:12) Where AI is showing up across the SDLC

(05:53) AI readiness and its link to developer experience

(08:23) Why enablement, education, and experimentation matter more than tool choice

(13:05) The case for a dedicated enablement team

(14:50) Measuring AI ROI: challenges and tradeoffs

(19:46) Background agents and token spend

(24:12) Measuring agent output with PR throughput

(26:58) How the engineer role might change

(31:01) Specs and documentation in the age of AI

(33:11) Non-engineers writing code

(35:30) What’s changing in the SDLC and open questions

Referenced:

• Measuring AI code assistants and agents

• Lessons from Twilio’s multi-year platform consolidation

• The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win

• How Claude remembers your project - Claude Code Docs

• specIsJustCode : r/ProgrammerHumor

Subscribe now

📅 Live research readout: Sign up here to join Abi for a readout and interactive discussion about an upcoming report that looks at the impact of AI on engineering velocity.

Each quarter, DX publishes data on how AI is being used at 500+ organizations and the impact it’s having. One metric we’ve been following is the percentage of code that’s written by AI—today’s newsletter shares a preview of what we’re seeing.

Currently, we measure the “percentage of AI-generated code” by asking developers directly how much of their merged code they believe is written by AI. This measure is captured quarterly by DX. In Q1 2026, developers at more than 500 organizations reported their average percentage of AI-authored code, along with how frequently they use AI tools (daily, weekly, or monthly). We aggregated these self-reported percentages to calculate the mean share of merged AI-generated code for each usage segment and overall.

We ran the previous Q4 analysis in the same way. In the future, we will collect this data and compare against a new system-based measure that automatically tracks the percentage of AI-generated code.

For this quarter, here’s a preview of what we’re seeing. Stay tuned for the full report later this month.

AI-generated code reaching production sees a slight increase

Since last quarter, the average share of merged code authored by AI has moved from 22% to 27.4%. While that’s directionally up, it’s not as meaningful of a change as we had expected.

The stability around 30% could be read two ways. On one hand, it may simply mean that current models and tooling are producing roughly the same proportion of merge-ready code as they were last quarter. However, there’s reason to believe something else is going on. The second half of 2025—particularly November and December—is widely regarded as a turning point for AI-assisted development. Models like Opus 4.5 represented a significant leap in capability, and some have gone so far as to say that anything before November 2025 shouldn’t even be used as a baseline, given how much changed in a short window.

If that’s true, then the more likely explanation for the stability around 30% is that most teams haven’t yet fully adapted to take advantage of those improvements. New models don’t automatically translate to more merged AI code; developers still need to update their workflows, build trust in the output, and find the right use cases for more capable tools.

There’s some evidence to support this interpretation. The daily users segment saw the largest increase in AI-generated code, moving from 24.1% to 30.8%. Meanwhile, weekly and monthly users, who are less likely to have adjusted their habits, saw smaller increases.

Final thoughts

This analysis focuses on how much merged code is AI-generated, but it doesn’t answer questions like whether a higher percentage of AI-generated code is associated with changes in quality. It also begs the question of whether there are certain types of organizations, or groups of developers, that are merging more vs. less AI-generated code.

We’ll explore these questions, and others like them, in the full Q1 AI Impact Report.

Additionally, in future reports we’ll compare this self-reported measure against a telemetry-based metric that tracks AI-generated code directly from version control systems. This will not only provide another lens on AI’s impact, but also give us a more granular view of how patterns are changing over time and where those changes are concentrated.

Stay tuned.

This week’s featured DevProd job openings. See more open roles here.

• Amplitude is hiring an Infrastructure Engineer | Lindon, UT

• Awardco is hiring an Engineering Manager, Cloud Platform | San Francisco, CA

• BNY is hiring an SVP, Application Development Manager | Pittsburgh, PA

• Figma is hiring a Staff Software Engineer, Developer Experience | Remote; US

• Leidos is hiring a Platform Engineer | Remote; US

• Lob is hiring a Staff Platform Engineer | Remote; US

• Weave is hiring a Senior Platform Engineer, Data Infrastructure | Remote; US

That’s it for this week. Thanks for reading.

Share

Subscribe now

Last week I hosted a live AMA about measuring the impact of AI and where organizations are with their rollout today. (Special thanks to Jesse Adametz for hosting the discussion.) One of the questions that came up had to do with measuring ROI: specifically, how should we think about correlation vs. causation? Leaders want to be careful about whether they should attribute changes in metrics to AI tools.

I answered the question live—you can download and watch the full recording here—or read this week’s newsletter for my take.

Many organizations make this mistake: They compare developers who use AI heavily to those who use it less, notice that the heavy users have higher throughput, and then conclude that AI must be the cause. The problem with this kind of analysis—“do people who use AI more have higher code throughput?”—is that in many cases, the developers who use AI more are the ones who were already coding more in the first place.

That’s why most of the time, longitudinal analysis (looking at how things change over time) is more informative. But it’s also harder to do. You need clean data over a long enough period, and you still have to account for confounding factors. For example, one big confound right now is the heightened pressure in many companies to increase throughput. Leaders are simultaneously pushing for more output and more AI usage. Because of this, some of the increase in throughput is likely due to this pressure itself—a classic case of Goodhart’s law, where once a metric becomes a target, it stops being a good measure.

When thinking about the ROI of AI more broadly, it’s useful to break it into two buckets:

• Amplification: How much more productive are humans thanks to AI? Here we look at:

  • Throughput (are engineers shipping more by using AI?)

  • Time saved (how much time do developers feel they’re saving in specific workflows?)

  • Developer experience scores (are AI tools improving the overall developer experience?) We can then convert improvements in developer experience into time savings, for example using something like a developer experience index.

• Augmentation: To what extent are you actually extending your engineering capacity by using agents, as if they were additional headcount? One unit we like to use here is human-equivalent hours: how much work are agents delivering, how much would that have taken a human, and then you divide the cost by those human-equivalent hours. Dividing the cost by the human-equivalent hours gives you an effective agent hourly rate (dollars per equivalent hour). If that rate is low, it indicates a high-ROI place to invest.

This amplification/augmentation framing is also helpful when talking to executives. You can say: to some extent we’re amplifying our humans, and to some extent we’re augmenting our workforce with agents.

Another common and related question that tends to come up in this discussion: if we agree that lines of code is a poor metric, what should we use instead to measure AI’s impact?

LOC is a noisy metric for a simple reason: A low-effort change can involve many lines of code, and a high-effort change can involve very few lines of code. That was already true before AI, and it becomes even worse with AI-generated code, which tends to produce more lines than a human might. This makes LOC even more inflated and noisy.

For these reasons—both pre- and post-AI—we’ve preferred metrics like PR throughput. It’s still imperfect, but it gives a more normalized view of “change throughput”: how many atomic changes are we pushing through the system? That makes it a less noisy high-level signal than raw LOC.

At DX, we’ve also developed a metric called TrueThroughput, which is a weighted version of PR throughput. It incorporates lines of code as one of several inputs to weight PRs, but LOC is not the sole or primary metric. Even so, all of these metrics are imperfect; you’re really choosing between different degrees of imperfection. In that space, LOC is significantly less useful than PR throughput as a primary signal. This lines up with broader industry experience as well. Many large tech companies that have studied this problem in depth have also converged on some form of change throughput as their preferred signal for tracking impact, rather than relying on raw lines of code.

To summarize my perspective on the correlation vs causation question: a simpler way to frame AI’s ROI is to look at how much it amplifies your existing developers, and how much it augments your organization with agent-driven capacity.

Download and rewatch the full AMA discussion here. My response to this specific question starts at 16:50.

This week’s featured DevProd job openings. See more open roles here.

• Amplitude is hiring an Engineering Manager, Cloud Platform | San Francisco, CA

• BNY is hiring an SVP, Application Development Manager | Pittsburgh, PA

• Figma is hiring a Staff Software Engineer, Developer Experience | Remote; US

• Lob is hiring a Staff Platform Engineer | Remote; US

• Mastercard is hiring a Vice President, Software Engineering | O’Fallon, MO; Boston, MA

• Plaid is hiring a Software Engineer - Platform | New York, NY

• Vercel is hiring a DX Engineer | Hybrid (Austin, New York City, San Francisco)

That’s it for this week. Thanks for reading.

Share

Subscribe now

Last week, we shared data showing that the current productivity gains from AI in engineering are more modest than many headlines imply. To better understand this result, we’ve been interviewing developers across a range of companies to hear what they’re experiencing, and to dig into the reasons behind these more modest gains.

This newsletter summarizes the key themes we’re hearing in those conversations so far, along with direct anonymous quotes from the interviews we performed. This work is ongoing, so if you have something to add, please share your perspective in the comments or on LinkedIn. We’d love to hear from you.

Why are the real‑world gains from AI more modest than many people expect?

Developers we spoke with pointed to several reasons. The most common: coding is a relatively small share of how engineers spend their time, so even meaningful acceleration there has limited impact on overall throughput. Beyond that: new bottlenecks are being created, and organizational and social friction hampers AI’s potential impact.

Here’s what we heard.

1. Coding isn’t the main bottleneck

The most common theme we heard: AI does accelerate developers’ ability to write code, but coding only represents a percentage of the work engineers actually do. Even if AI cut coding time in half, it would only affect a fraction of the total time spent delivering a feature.

• A lot of the slowdown isn’t from our tools, but from either other teams, services or processes.

• Many senior devs say: “writing code is the easy part”... The bottleneck is the human side: alignment, planning, scoping, code reviews, etc.

• AI can help developers write code fast, but writing faster code is not always more productivity. Most developers code only a small proportion of their time. There might be an expectation that this percentage is higher from those who write the headlines, but the reality is that most of the time developers spend is on navigating process and quality, change management, regulation, etc.

• The easy tasks are a little easier. The tedious tasks are a little less annoying. Larger/more complicated tasks — I may be able to shave off hours to a day… but that’s not happening every day, and that doesn’t necessarily mean I ship 3x more PRs. Just means a 4d task can take 3.

2. Partial SDLC automation creates new bottlenecks, so AI gains cap out

AI has sped up code generation, but that has created or exposed bottlenecks downstream, particularly in code review and validation. More code is being produced, but the processes for checking that code haven’t scaled with it, and developers report that the time saved writing is often consumed by the extra scrutiny that AI-generated code requires.

• AI can very significantly speed up initial engineering time, but often that saved time is spent on extended reviews, fact checking or issue remediation, resulting in net-zero productivity gain.

• The opportunities for LLMs to save large amounts of time on new projects is offset by time spent, sometimes ineffectively, attempting to use them on existing projects whose complexity and intricacies lead the LLM to make changes that are at minimum difficult to review even for experienced owners of the code.

• People do not have all parts of SDLC AI-enabled, wherever it does not exist becomes a bottleneck (planning, code review, verification). When people are trying to remove those bottlenecks, there are often lack of guardrails that creates misdirection and churned motions.

3. Social friction is a barrier to adoption

Polarization between pro- and anti-AI engineers, ambiguity about when and how AI should be used, and the absence of peer champions all slow the rate at which teams develop effective workflows with these tools.

• There’s misalignment amongst engineers – the camps are pretty polarized in terms of pro-AI and anti-AI. When anti-AI engineers have status and loud voices it is challenging for everyone to know how to interact with ai tooling.

• AI tooling isn’t discussed enough. Engineers are often unsure whether it’s high status or low status to be talking about using AI, so they just don’t. And people are reluctant to state their true views. If people say what they might really believe, they might seem crazy. In many organizations it is not socially acceptable to suggest that we re-evaluate changes to the code review process for example.

• Having at least one other teammate who is bought in to using AI tools is essential, being an isolated solo-adopter does not allow you to materialize the gains in a meaningful way. Software development is a team sport.

4. Tooling and skill gaps are limiting gains

These two factors are hard to separate: immature tools make the learning curve steeper, and developers who are early in that curve get less out of the tools. Both are limiting gains, though developers who shared this view generally see it as a temporary problem.

On the skill side, using AI effectively is its own discipline. On the tooling side, AI assistants don’t always slot cleanly into existing developer workflows. The integrations that would make them feel native to how teams already work aren’t there yet, and getting agents to operate autonomously on real infrastructure remains an unsolved problem for many.

On skill gaps:

• Someone just starting out with these tools is not as effective as someone who has incorporated them into their workflows for much longer. I have 1000+ hours of purely agentic coding experience and I have so much to learn. Many beginners may have just a few hours or tens of hours. Learning to agentically code is a skill.

• Most of the task isn’t the actual coding, it’s articulating a fuzzy problem in a clear, easily understandable way.

• This is more short term, but the pace of change in the tooling (e.g., the emergence of skills) means I’m spending a fair amount of time getting to know how to use the tools appropriately; which usually means going down various rabbit holes to discover how NOT to use them. That creates some (temporary?) friction that means I might fall back to my own skills when there’s production pressures.

Note: Leaders can distribute DX’s prompting guide and advanced prompting guide internally with developers as a way to help them build a foundational practice with AI.

On tool maturity:

• The tools aren’t mature enough to fit into existing systems cleanly, and most developers haven’t figured out how to use them well yet. Not because they’re bad engineers — the workflow is just genuinely new and nobody’s handed them a playbook.

• The difference between frontier models and models even a few months back can be significant.

• Most developers are used to working inside well-defined tools — your IDE, your terminal, your CI pipeline. Everything has a clear place and a clear trigger. AI assistants don’t work that way… So you end up with higher adoption among developers who naturally gravitate toward that kind of open-ended tooling, and much lower adoption among developers who just want something that fits cleanly into what they’re already doing.

5. Most AI tools lack important context

AI tools perform well on problems that are self-contained and well-documented. But most real engineering work isn’t like that. The context that matters—why a system was designed a certain way, what the implicit rules are, what business constraints apply—typically isn’t written down. It lives in people’s heads. Until that knowledge is made explicit and accessible, AI will keep hitting a ceiling on the kinds of problems it can reliably help with.

• The bigger issue I keep running into is that most codebases and systems aren’t set up for AI to actually help. Not an architecture problem — more that the knowledge of how things work lives in people’s heads. Why this service behaves this way, what the implicit contract between these two systems is, why that design decision got made three years ago — none of that is written down anywhere. An AI assistant can’t reason over a Slack thread from an archived channel or the mental model of the engineer who built it. Until that context gets surfaced into something concrete, you’re always going to hit a ceiling on what the tooling can do.

• AI doesn’t have the context I do on making those decisions, as LLMs are built of massive stats, not direct knowledge of the niche ends of what I’m accomplishing.

• I would (anecdotally) put forth that a lot of my work with AI is trying to get it setup to do the work with proper guardrails. It’s not safe to let run loose on our code-bases or production facing infrastructure as its not deterministic.

• AI models are generally good at understanding what we are trying to achieve, especially when they have access to the workspace or codebase. They can often provide useful solutions for specific use cases, particularly when the problem is related to foundational aspects of a technology or programming language. However, what they often lack is a deeper understanding of the business context and the various factors that influence business logic. That broader context is important to fully understand the bigger picture of a project. In some cases, this limitation can be mitigated by carefully crafting prompts, but when the business logic becomes complex or the prompts become too large for relatively simple tasks, people may step back and revert to more traditional approaches.

Other observations about the current state of AI tools

While speaking with developers, we heard a few other interesting themes about what they’re currently experiencing.

1. Documentation robustness is improving, but its quality is yet to be determined.

• The biggest quality gain I’m seeing is in improved documentation. Developers hate writing docs, so it will always be the last thing they tackle, or the first to get cut... if they tackle it at all. At least for me, I found our documentation getting much more robust thanks to AI being able to take that task off our plate. And it’s not just great for the developers to have access to these docs. Giving the AI agents access to it provides a quicker context boost to know how to tackle the next project.

• Using it to write code documentation leads to terribly incorrect information, an active detriment to future work, creating more work. This is really painful. It’s just not its forte.

2. AI is especially powerful in greenfield projects.

• Could you whip up an app from scratch 3x faster with AI? I would believe that. And if I had to just guess, that’s where I think a lot of the hype comes from. Someone says “look I created this thing from scratch in 1 day!” But, I think it is known that AI isn’t quite as proficient when working on a mature, large codebase...with legacy code and lots of nuance. Which is where most ICs are living in day to day, I think.

• While multi-tasking on my main efforts, I was able to build an entire new microservice from scratch to tests an idea and provide a proof of concept of a new paradigm (this has massive comprehension debt, I haven’t even looked at the code, but the service does what I expect it to).

• It has helped a lot with doing things like fast PoCs to check if something will work.

3. There’s concern that developers are offloading critical thinking.

• I would say I’m losing muscle memory on how to do things (e.g. terminal commands like checking git history for when a file changed, checking what version of a dependency is installed, etc.). I’ve grown dependent on the AI to remember the right commands for me.

• I think if someone blindly uses AI for everything, he/she slowly forgets day to day things.

• AI helps speed up some tasks but may result in folks offloading their thinking. Particularly amongst junior developers, the ‘improved initial speed’ can come at the expense of skill development. Overall speed improvement is negligible.

• I have also found myself having to explain to more junior engineers why what AI said about a problem isn’t the correct answer for us in a particular situation. AI empowers more junior engineers to work longer without asking engineering questions of the more senior folks. That is a sharp double-edged sword that I am still learning how to work with when mentoring people.

• [For me, AI’s] handiness is augmented by me knowing when it’s wrong due to my engineering experience.

Final thoughts

Two takeaways stand out to me from these conversations. First, the gains from AI code assistants are structurally limited by how much of the job is actually coding. Microsoft’s research puts coding at around 16% of a developer’s time. Even dramatic acceleration to code generation can only move the overall needle so much.

Second, it’s the human aspects of the software development lifecycle that are the biggest bottlenecks. Handoffs between teams, the skill required to effectively prompt and manage agents, and the review burden that AI-generated code creates—these are where time is going, and where the next wave of productivity gains may come from.

For leaders, this raises a couple of important questions. Which parts of the SDLC beyond coding are candidates for AI assistance? And are the human conditions for capturing AI’s upside in place?

The 10% figure is not the ceiling. But closing the gap will likely require focusing on the parts of the job that AI hasn’t touched yet.

If you have something to add, please share in the comments or on LinkedIn.

This week’s featured DevProd job openings. See more open roles here.

• Amplitude is hiring an Engineering Manager, Cloud Platform | San Francisco, CA

• BNY is hiring an SVP, Application Development Manager | Pittsburgh, PA

• Figma is hiring a Staff Software Engineer, Developer Experience | Remote; US

• Lob is hiring a Staff Platform Engineer | Remote; US

• Plaid is hiring a Software Engineer - Platform | New York, NY

• Vercel is hiring a DX Engineer | Hybrid (Austin, New York City, San Francisco)

• Zillow is hiring a Senior Product Manager, Developer Experience | Remote; US

That’s it for this week. Thanks for reading.

Share