Subscribe now

🗓 Join me and Brian Houck on July 23 for a readout of this report, where we’ll discuss new findings from DX’s data on AI tool usage, spend, and impact across 500+ organizations. Register here.

We are excited to announce our Q2 2026 AI impact report.

When we first began tracking the impact of AI on engineering teams, our primary goal was to measure AI cohorts against historical baselines to answer the question of what happens to software output after adoption. With industry-wide AI adoption exceeding 90%, comparing AI users against a non-user control group is no longer a viable measurement strategy.

Engineering leaders are now under immense pressure to justify exponentially-increasing AI budgets. The data from our Q2 report reveals that while AI is delivering objective gains in velocity, those gains are highly uneven.

Download the full analysis here.

In the new report, we’ve uncovered a number of critical trends, including:

1. Over 50% of code is now generated by AI. This metric has accelerated rapidly, increasing from 34% in Q1 2026 to 52% in Q2 2026. This steep trajectory indicates that once AI tools are deployed, the code they generate rapidly scales across codebases, frequently moving through reviews, dependencies, and shared workflows.

2. Quality may be declining. During the same period that AI adoption has increased, median pull request sizes have nearly doubled. Increases in PR size can serve as an early indicator of technical debt, as higher code volumes generally correlate with increased complexity and potential for bugs. This trend can also introduce additional friction in the review process, as more lines of code generated means more lines of code to review.

3. Some aspects of developer experience are declining. The Developer Experience Index (DXI) dropped from 67 to 65 over four quarters. AI is improving some aspects of the developer experience—documentation quality, code maintainability, onboarding speed—while creating new friction in others: larger PRs, slower reviews, less incremental delivery. In aggregate, the net effect is currently negative. Velocity metrics alone will tell you things are improving. Developer experience metrics will tell you whether that’s actually true.

4. AI is making codebases easier to understand, but it’s also making the code it generates harder to trust. The Q2 data highlights a striking divergence between two historically correlated software quality metrics. Specifically, from Q1 2026, Code Maintainability improved by 3.8%, whereas Change Confidence decreased by 6.1%. Code Maintainability indicates how easily developers can understand the codebase, while Change Confidence measures their trust that modifications won’t cause production failures. Traditionally, highly maintainable code results in higher confidence when making changes. However, this data reveals a new tension: although AI helps developers understand the code in front of them, they exhibit less trust in the code they are pushing to production.

5. Saved time isn’t converting into innovation. AI users are now saving an estimated 4 to 6 hours per week. However, the innovation ratio, defined as the percentage of time spent on building new features versus maintenance and overhead, has remained flat over the same period of study. This flat trend indicates that the time saved by AI is not currently converting into increased capacity for creating new value. Leaders should keep a close eye on this metric over time. Ideally, innovation ratio will increase as AI frees up engineers to work on more new features.

6. AI spend is accelerating faster than outcomes. Median quarterly organizational AI spend climbed from ~$1.5K to ~$44K over four quarters. Tech-sector spend increased nearly 28x. These numbers will draw scrutiny. Leaders who cannot connect this investment to downstream outcomes (feature velocity, innovation ratio, quality) may face increasingly difficult budget conversations in the back half of 2026.

What this means for leaders

The Q2 2026 data indicates that the industry is shifting from base AI deployment to evaluating concrete return on investment. As AI expenditures accelerate, engineering leaders must shift their focus from simply acquiring AI tools to optimizing the surrounding development pipelines and resolving systemic bottlenecks. To achieve true ROI, leaders must ensure that saved hours are reinvested into product innovation rather than absorbed by existing organizational friction.

To explore the full data and benchmark your team against 500+ organizations on measures of throughput, quality, and AI tooling cost, download the full report here.

That’s it for this week. Thanks for reading.

-Justin

Share

Subscribe now

🗓 Join DX Deputy CTO, Justin Reock and Distinguished Scientist, Brian Houck on July 23 for a readout of the Q2 State of AI Impact in Engineering Report. Register here.

I recently sat down with Thomas Newton, VP of Engineering at UKG, to discuss how his team built a system to guide AI adoption and assess whether it’s translating into meaningful engineering outcomes. What was particularly interesting was how UKG provides teams with a metrics dashboard that managers can use to have better coaching conversations, and ultimately help developers become more effective with AI tools.

For this week’s newsletter, Thomas is describing how their approach works.

Here’s Thomas.

Thomas: As a leader in our industry, we previously faced a question that most engineering organizations are grappling with right now: how do you measure whether AI usage is translating into more meaningful work shipped?

This was a leadership priority with one goal from the start: make sure that the right data, in a digestible format, landed with the people closest to the work—the managers.

The result is what we call the Manager AI Adoption Dashboard: a set of visuals that combine AI usage patterns, delivery outcomes, and spend into a single picture that engineering managers can use to coach their teams, guide adoption, and have better conversations about how work is getting done.

Starting with experimentation and adoption

Our journey using AI tools in product development started the way most do. We experimented with several tools (GitHub Copilot, Windsurf, and others) before making a significant push toward Claude.

Giving our engineers access to these tools was a good place to start, but our managers needed visibility. They could feel the productivity gains anecdotally, but the data was missing. We wanted to better understand where AI was helping teams, and which workflows were creating the most impact.

The shift to a consumption-based model made this need even more urgent. Unlike fixed-cost seat licenses, consumption pricing means the meter is always running. Leaders needed to understand not just whether teams were using AI, but whether the investment was producing returns.

Deciding what to measure

One of the earliest decisions we made was to anchor the dashboard around TrueThroughput, a metric developed by DX that goes beyond raw pull request counts to account for the relative complexity and size of work delivered.

TrueThroughput uses AI to classify the complexity of different tasks, giving you a size-adjusted throughput number. Think of it like a weighted GPA versus an unweighted GPA. Both are useful, but the weighted version tells you whether someone is delivering meaningful work or just merging a lot of five-second fixes.

Instead of indexing on consumption, the dashboard creates a balanced view of AI impact by correlating consistent AI usage, measured in days of use, with TrueThroughput. These metrics help us answer whether consistent use of AI is helping teams deliver more meaningful work.

The dashboard combines three signals:

• AI usage: How consistently someone is using AI tools in their workflow.

• TrueThroughput: The volume and complexity of work delivered.

• Spend: Awareness of AI investment and consumption patterns.

Inside the dashboard

The dashboard was built around a quadrant view. The horizontal axis tracks consistent days of AI use over a 30-day window: fewer than 15 days puts you on the left, more than 15 on the right. The vertical axis tracks TrueThroughput.

Example for illustration purpose only:

That creates four zones:

• Exploring (bottom left): Lower AI adoption, developing throughput. Teams discovering what problems AI might solve.

• Learning (bottom right): High AI adoption, building throughput. Expected pattern as teams explore different applications over 2-3 months.

• Efficient (top left): High throughput without heavy AI adoption. Some roles and workflows don’t need AI.

• Amplified (top right): High AI adoption, high throughput. Patterns here show which applications create real impact.

Each dot on the chart represents an individual. Bubble size reflects spend. And the views are drillable: from the entire organization down to a business unit, a team, and ultimately an individual manager’s direct reports.

What the data revealed

Since rolling out the dashboard, the results have been striking. Initially, as one might expect with newly introduced tooling, the bottom-left quadrant was packed, meaning that a large portion of our engineering organization hadn’t touched AI tools at all. Within four months,, that quadrant was nearly empty; less than 1% of the organization remained in the low-AI, high-throughput zone. It’s been exciting to watch the whole organization shift on this.

“Less than 1% of the organization remained in the low-AI, high-throughput zone.”

We also saw some things in the data that confirmed what we’d hypothesized. For example, the group seeing the biggest throughput gains were our senior and principal engineers, at ~20-30% above what other roles saw. That’s intuitive, but it was interesting to see it in the data. Senior engineers already have strong instincts for where AI can help and where it can’t. They were often quicker to identify high-leverage opportunities and incorporate the tools into existing workflows.

One pattern caught us off guard: managers and directors started writing code again. Our leadership population started doing more direct hands-on coding. They have more assistants, they can multitask better, whatever the reason, it’s a clear trend. That pattern extends beyond engineering: our product managers and designers have started leaning into AI tools too, getting comfortable with the terminal, creating digital artifacts, and contributing in ways that show up in delivery metrics. When we started this, it wasn’t what we anticipated, but it’s a trend that is now more commonly observed and discussed.

How managers use the dashboard

The real value of the dashboard is the quality of the conversations the data enables.

For managers, the first question often focused on adoption: “How do we get you from left to right?” Are you using it daily, is it part of your workflow, or are you still finding your footing with it? But over time, the conversation became less about adoption itself and more about impact. Where is AI helping? Which workflows are working well? Where is it creating leverage, and where is it not?

When a manager saw an engineer with high AI usage but flat throughput, the right response wasn’t to question the spend. It was to ask what they were working on; maybe they were ramping up on new AI workflows, or in a role where their AI-assisted work didn’t produce code commits. One example is heavy operation roles where productivity didn’t always result in commits into GitHub.

The best way to head off gaming or surveillance concerns is proactive communication. Don’t let people fill in the blanks on what the dashboards are for or why they exist. Be very clear. The dashboard is a conversation starter, not a performance management system. If teams interpret the dashboards as a tool for punishment or reward, they’ll have every incentive to game the numbers.

“The dashboard is a conversation starter, not a performance management system.”

At UKG, the framing has been consistent from day one: the dashboard exists to help managers empower their team, understand the work, understand how AI is being applied to that work, and help people lean into a new way of working.

Keeping AI spend in check without leading with cost

AI spend is visible on the dashboard, but it’s deliberately not the headline metric. It’s an awareness layer, something managers should be conscious of, not something that drives the conversation.

We use what I call “circuit breakers”; daily budget controls that flag when usage spikes above a threshold. But we haven’t yet landed on a firm benchmark for what reasonable per-engineer spend looks like.

The range is just too wide right now. Some teams are considering multi-agent orchestration and long-running loops that transform an entire codebase overnight. Others are using AI to finish a feature today. At this stage, it’s really hard to generalize.

I expect spending patterns will stabilize as the initial learning ramp flattens. Your first couple of prompts will probably be expensive. You’re still learning how to use the models correctly, you’re playing, you’re understanding how to use this radical new thing. But in as little as a few months, you will start to see averages form. We’re still in an experimentation phase, and for now, the approach is pragmatic: watch carefully, trust manager judgment, and make sure the investment is going toward the right outcomes.

What comes next

We’re already thinking about the limits of what the current dashboard captures. TrueThroughput is powerful for teams that ship code, but it misses productivity gains in operations, SRE, and other roles where AI is being used to correlate incidents, search logs, and accelerate incident resolution, work that never ends up in a pull request.

We have teams where an engineer uses AI to search past incidents during a live outage and correlate similar patterns to get to a resolution faster. That’s enormously productive, but it won’t show up in throughput. We’re trying to think through what the next level of digital footprints looks like, the metrics that capture the full picture of AI-enabled productivity, not just the code-commit slice of it.

The dashboard is a living system designed to evolve as we learn more about what effective AI-enabled engineering looks like.

Final thoughts:

For engineering leaders at other organizations who haven’t yet started measuring AI adoption, my advice is simple: Measure something. Metrics you have access to might differ, but collect some form of data, figure out what makes sense for your organization, and don’t make it a binary decision based on the data itself. The data should enable leaders to have further conversations.

That’s an important takeaway: The tools are powerful, and the data is illuminating, but the transformation happens in the conversations between managers and their teams.

If you have questions about UKG’s approach, or just want to hear more from Thomas, make sure to follow or connect with him on LinkedIn.

That’s it for this week. Thanks for reading.

-Abi

Share

In this episode of the Engineering Enablement podcast, I sit down with Sejal Amin, Chief Technology Officer at Priceline, and Pedro Gutierrez, Senior Director of Software Engineering, to discuss how Priceline adopted a product operating model and the role developer experience played in making that transformation successful.

We explore why the company moved away from a project-based approach, how DX metrics and developer feedback helped uncover organizational bottlenecks, and why a phased rollout, clear communication, and empowered engineering managers were critical to building trust and improving developer experience. We also discuss creating a dedicated developer experience team, lessons learned throughout the transformation, and how Priceline’s product operating model has helped the organization adapt to AI-driven software development.

Some takeaways:

Developer experience as a driver of organizational change

• Developer experience data can reveal organizational problems that traditional engineering metrics miss. At Priceline, DX signals uncovered organizational bottlenecks—including handoffs, dependencies, and team friction—that ultimately led the company to adopt a product operating model.

• Developer experience should be treated as a strategic capability, not just an engineering metric. Rather than measuring developer satisfaction in isolation, Priceline used DX insights to guide structural changes that improved autonomy, delivery, and engineering culture.

Adopting a product operating model

• Reducing dependencies gives teams greater ownership. Priceline shifted from a project-based organization to cross-functional product teams, reducing handoffs and giving teams the people and capabilities needed to own outcomes end to end.

• Autonomy requires visibility into team health. DX metrics gave engineering managers a clear view of the obstacles affecting their teams, allowing them to improve local workflows while staying aligned with broader organizational goals.

Turning developer feedback into action

• Developer experience surveys should lead to action—not just measurement. Managers reviewed survey results, completed a triage process, created quarterly action plans, and measured whether those improvements had an impact in the next survey cycle.

• Small workflow improvements can have an outsized impact. DX data helped teams reclaim focus time, identify tooling regressions after migrations, surface cross-team dependencies, and address day-to-day friction before it became systemic.

Building trust in developer experience metrics

• Clear communication is essential for adoption. Leaders consistently reinforced that DX metrics existed to improve teams rather than evaluate individuals, helping build confidence in the process from the outset.

• Trust grows when developers see meaningful change. Acting on feedback quarter after quarter encouraged greater participation, strengthened psychological safety, and made developer experience part of the organization’s culture.

The evolving role of engineering managers

• Engineering managers became owners of developer experience. Managers were expected to understand DX data, improve their team’s DXI each quarter, and make developer experience part of their regular operating rhythm.

• Developer experience became part of everyday engineering leadership. DX metrics were discussed openly in all-hands meetings and other forums, making developer experience a visible measure of organizational health rather than a one-time initiative.

Preparing engineering organizations for AI

• AI changes where bottlenecks occur—not whether they exist. As AI accelerated code generation, Priceline used its product operating model and developer experience data to identify where constraints had shifted and respond accordingly.

• A strong operating model helps organizations adapt to AI. Autonomous teams, continuous measurement, and visibility into developer workflows allowed Priceline to embrace AI while continuing to improve flow across the software development lifecycle.

In this episode, we cover:

(00:00) Intro

(01:07) Meet Sejal Amin and Pedro Gutierrez

(01:47) How Priceline’s developer experience journey began

(04:55) Lessons from Priceline’s first developer experience surveys

(06:55) How DX improved Priceline’s developer experience surveys

(09:47) Identifying the causes of organizational slowness

(12:33) How the product operating model changed the way Priceline works

(14:10) Priceline’s phased rollout with DX

(18:14) How DX insights drove organizational changes

(19:33) Why Priceline improved developer experience before org change was complete

(22:18) How clear communication builds trust

(24:25) Early results from Priceline’s Core Four

(25:38) Creating a culture of continuous feedback to build trust

(27:40) What has changed in the engineering manager role

(30:10) Resources for learning about the product operating model

(32:40) What Pedro learned from implementing DX

(34:51) The developer experience team

(35:59) How AI tools have impacted Priceline’s teams

(37:20) How the product operating model supports AI-driven development

(39:13) Final advice for engineering leaders

Where to find Sejal Amin:

• LinkedIn: https://www.linkedin.com/in/sejal-amin

Where to find Pedro Gutierrez:

• LinkedIn: https://www.linkedin.com/in/pedro-gutierrez-b6605422

Where to find Justin Reock:

• LinkedIn: https://www.linkedin.com/in/justinreock

Referenced:

• Measuring developer productivity with the DX Core 4

• Transformed: Moving to the Product Operating Model (Silicon Valley Product Group)

• Team Topologies

• Flow Framework

• Project to Product: How to Survive and Thrive in the Age of Digital Disruption with the Flow Framework

Subscribe now

🗓 Join me on July 23 for a readout of the upcoming State of AI Impact in Engineering: Q2 Report. We’ll discuss new findings from DX’s data on AI tool usage, spend, and impact across 500+ organizations. Register here.

Every once in a while, several independent papers arrive at roughly the same time and collectively tell a bigger story than any one of them does alone. This week, I’m sharing five recent papers that have significantly influenced how I’m thinking about AI and software engineering.

Each paper tackles a different question. Some measure the productivity impact of AI coding assistants. Others examine how those gains propagate through the software delivery process, explore what developers actually want from future AI systems, or reconsider the kinds of debt we should be paying attention to in an AI-assisted world.

Despite coming from different research groups and using very different methodologies, they all seem to be converging on the same underlying story.

AI is compressing the upstream work of software engineering. The more I sat with these papers, the less I found myself asking, “Is AI making developers faster?” and the more I found myself asking, “What happens after the code is written?” Are we actually shipping more value? Where do the new bottlenecks emerge? And what are the costs if understanding can’t keep pace with generation?

After reading these five papers, I came away with one overarching conclusion: we’re generating code faster than we’re generating the systems needed to safely understand, verify, and deliver it.

A quick note on disclosure: three of these papers come from people I know and work with extensively. None of the papers are mine.

Here they are, in the order I’d recommend reading them.

1. GitHub Copilot and Developer Productivity

Paper: Heilman, A., Kyllo, A., Murphy-Hill, E. GitHub Copilot and Developer Productivity: An Observational Dose-Response Analysis.

The first paper I want to highlight tackles the familiar question of whether GitHub Copilot makes developers more productive, but it does so with one of the more clever research designs I’ve seen.

Rather than simply comparing Copilot users to non-users (which are getting harder and harder to find), the authors control for Active Coding Time (i.e., how much time developers spend actively engaging with development tools) and examine how productivity changes within the same engineer over 43 weeks across a population of 16,223 developers.

The payoff of this design is that it compares engineers to themselves rather than to one another. Using that approach, the authors found that weeks with the highest Copilot usage were associated with ~40% more completed PRs per hour of coding time than weeks with no usage.

The relationship showed a clear dose-response pattern (a way to do a causal analysis, once everyone is already using the tools). More Copilot engagement was associated with more PR throughput, although the gains appeared to level off at very high usage.

The authors ran seven robustness and falsification tests to rule out alternative explanations (team-level effects, generic AI engagement, PR slicing, shifts toward easier work). The positive association remained remarkably consistent.

Interestingly, the gains were not concentrated in tiny PRs. The strongest effects were observed for larger PRs (7+ files), arguing against the idea that developers are simply breaking work into smaller units.

It’s a thoughtful analysis and shows that we’re not just coding more, we’re increasing coding efficiency as well. These findings anchor many of the studies that follow in this roundup.

2. Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools

Paper: Demirer, M., Musolff, L., Yang, L. Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools.

The next paper I’m highlighting was published by the National Bureau of Economic Research. It analyzes AI adoption across 100,000+ GitHub developers and asks a more nuanced question than Heilman’s: when AI makes individual coding steps faster, how much of that gain actually survives all the way to shipped software?

The authors examine how AI productivity gains propagate through a hierarchy of software development: lines of code → files → commits → pull requests → projects/repos → releases.

They found that AI is clearly increasing coding activity, and the gains grow with each generation of tools. They estimate roughly +40% more commits from autocomplete, growing to +140% from interactive coding agents, and finally +180% from autonomous agents.

However, those gains fall off significantly as work moves through the software delivery process. The largest effects are seen in code generation, but smaller effects appear in repos touched, small still in releases shipped, and ultimately software consumed by users. Even with very large increases in coding activity, the effect on shipped software is much smaller, topping out at roughly +30% more releases. This is illustrated in figure 2 below.

One of the findings I found most interesting is that they estimate a low elasticity of substitution (~0.25) between AI-generated output and human effort. That’s an economics concept that measures how replaceable human work is with AI output. As a methodology nerd, and someone with an economics degree, I found this particularly clever — they infer this elasticity from how AI productivity gains attenuate across the delivery process. Their estimate suggests AI and human work are still largely complements rather than substitutes, with substantial human effort still required to review, integrate, validate, and ship software.

One open question is whether the observed fall-off through the delivery process is some fundamental limit of software engineering, or simply the fact that organizations have not yet adapted their processes to an agentic world.

If Heilman tells you Copilot is making engineers measurably faster, this paper asks the harder question: faster at what, exactly?

3. The Impact of AI Coding Assistants on Software Engineering

Paper: Vella, A., Blincoe, K. The Impact of AI Coding Assistants on Software Engineering: A Longitudinal Study.

The next study I want to highlight is unique because it isn’t just a snapshot in time, it’s a six-month longitudinal study of 95 professional software engineers. It also calls into question a relationship that we’ve long believed to be a bedrock of developer experience.

The study was done using two questionnaires six months apart, mixed-methods, with reflexive thematic analysis on the open-ended responses.

Vella found that productivity perceptions were stable and strongly positive over time. 84% of study participants reported improvement at both time points. Consistent with the first two studies in this round-up, the story of accelerated throughput is real and persistent.

The really striking finding is what the authors call the productivity-experience paradox. Among the matched cohort, the share of engineers reporting worse DevEx on at least one dimension nearly doubled in just six months, from 14% to 27%. Flow state was the most vulnerable; cognitive load eroded modestly; feedback loops actually improved.

More importantly: while the cross-sectional correlations between DevEx and productivity were strong, the change scores didn’t correlate. Productivity and developer experience appear to be decoupling over time in AI-assisted workflows. For those of us who’ve spent years working with the SPACE and DevEx frameworks, that’s worth sitting with.

While this study didn’t have a particularly large population, the findings were significant and rigorously validated, proving that a study doesn’t have to be massive if the strength of results is strong enough. This longitudinal design is rare and valuable, and the productivity-experience decoupling is the kind of finding worth replicating in larger populations.

4. To Copilot and Beyond: 22 AI Systems Developers Want Built

Paper: Choudhuri, R., Badea, C., Bird, C., Butler, J., DeLine, R., Houck, B. AI Where It Matters: Where, Why, and How Developers Want AI Support in Daily Work.

Last year I published a paper called AI Where It Matters, and my co-authors ended up writing a 2nd paper based on the original survey responses (860 Microsoft developers across roles, domains, and geographies). The paper outlines a roadmap of 22 AI tools that developers want beyond just code generation, centered around a concept they call “bounded delegation.” A lot of this echoes what the rest of this round-up is circling:

• The “right-shift” burden. Because AI is speeding up code generation, it’s creating a massive bottleneck downstream. Devs are getting flooded with more code to review, more production incidents to debug, and documentation that falls behind faster than ever.

• The move to verification. Developers don’t want more code-generation assistants; they want AI embedded into verification tasks — tools that automatically assemble log/trace “case files” for on-call incidents, PR reviewers that catch complex business logic flaws before human review, change-aware test generation that knows which assertions actually matter.

• “Bounded delegation.” There is a strict boundary around where developers want AI to stop. Developers want AI to absorb the tedious “assembly work” surrounding their craft (updating docs, writing edge-case unit tests), but never the core logic, architecture, or critical decision-making. Notably, developers drew this line even for tasks they acknowledged AI could plausibly handle — suggesting it’s not just about capability gaps and won’t move just because models improve.

• Four non-negotiable guardrails. For future AI tools to be adopted, developers say they must enforce explicit authority scoping (no auto-approvals), clear data provenance, explicit uncertainty signaling (the AI must admit when it doesn’t know something), and least-privilege security access.

You can check out both papers and an interactive website here: aka.ms/ai-where-it-matters

5. From Technical Debt to Cognitive and Intent Debt

Paper: Storey, M. From Technical Debt to Cognitive and Intent Debt: Rethinking software health in the age of AI

I’ve saved this for last because I think this is the most important paper I’ve read in a long time. Margaret-Anne Storey makes a generational argument: the metaphor we’ve used for decades to think about software health—technical debt—is no longer sufficient. AI is reducing technical debt (through refactoring, test generation, automated review) while quietly accelerating the accumulation of two other forms of debt that matter more in this era.

Technical debt lives in code. It accumulates when implementation decisions compromise future changeability. AI is genuinely helping here.

Cognitive debt lives in people. It accumulates when a team’s shared understanding of a system erodes faster than it’s replenished. When AI generates the code, developers may accept it without building the same mental model they would have built by writing it themselves. Multiply that across a team and over time, and you get “an accumulation of not knowing.”

Intent debt lives in artifacts. It accumulates when the goals, constraints, and rationale that guide a system—the things both humans and AI agents need to work safely—are unclear, unwritten, or forgotten. As more development is AI-assisted, intent debt becomes a first-order constraint on what AI can actually do for you.

The three debts interact and compound. Intent debt causes cognitive debt; cognitive debt causes technical debt; technical debt amplifies cognitive debt. Managing software system health requires attention to all three layers, not just the one easiest to measure.

The four practical implications Storey draws are worth reading in full, but the headline is: treat understanding as a deliverable. Just as working code is a product of software development, shared understanding should be treated as a first-class deliverable, not something that happens as a side effect of writing code.

Final thoughts

Read together, these five papers say something stronger than any of them say individually. AI is genuinely making code generation faster, and the per-engineer efficiency gains are real (Heilman). But those gains don’t survive the trip to shipped software at anywhere near the same magnitude (Demirer). The bottleneck has moved downstream to review, integration, verification, and understanding. Developers feel it, they’re explicitly asking for tools to address those bottlenecks while refusing to delegate the parts of the job they consider craft (Choudhuri). The lived experience of working this way is more uneven than the productivity numbers suggest, with flow and cognitive load eroding even as throughput holds (Vella). And the deepest cost may be one we don’t yet measure: the slow erosion of shared understanding, which is what makes any system safe to change (Storey).

The bottleneck has moved. Our tools, metrics, and team designs haven’t moved with it yet. That’s where the next several years of work in our field are going to happen.

That’s it for this week. And make sure to sign up for my upcoming live research readout covering new findings on AI’s impact, where we’ll discuss data from both DX and the broader industry.

-Brian

Share

Subscribe now

🗓 Join me on July 23 for a readout of the upcoming State of AI Impact in Engineering: Q2 Report. We’ll discuss new findings from DX’s data on AI tool usage, spend, and impact across 500+ organizations. Register here.

I’ve spent a lot of time over the years thinking about developer productivity metrics. The longer I do this work, the more convinced I become that the hardest part isn’t measuring software engineering. It’s improving it.

That’s where I think the best measurement systems distinguish themselves. They don’t just tell you whether things are getting better or worse. They guide you to interventions that actually improve how developers work. If your dashboard isn’t changing decisions, it’s not creating value.

That idea sits at the heart of EngThrive: Make It Fast and Easy to Do Great Work, a paper I recently co-authored with Tim Bozarth, David Liu, and Dean Carignan. The paper describes the measurement and improvement system I helped build at Microsoft, but what has stuck with me most aren’t the dashboards or the framework. They’re the stories.

One team intentionally “gamed” a productivity metric and improved onboarding for an entire year. Another protected developers’ focus time and discovered that the biggest gains came from work they weren’t even trying to improve. A third gave every developer two unexpected days off and found that the lost “productivity” disappeared within weeks while the wellbeing benefits lasted for months.

Three different organizations. Three different problems. Three different interventions. Yet they all taught the same lesson: the best measurement systems don’t just tell you what’s happening. They help you figure out what to do next.

Before we get to those stories, though, it’s worth remembering how easy it is to measure confidently and be wrong.

In the first two months of mandatory remote work at Microsoft in early 2020, pull requests per developer jumped more than 20%, and the company’s stock price rose more than 15%. By those measures, things looked great. During that same period, however, 78% of developers reported feeling burned out. Three signals from the same quarter, pointing in two different directions. Any one of them on its own would have told a confident but completely misleading story.

That is the trap EngThrive was built to avoid. Organizations often measure activities—pull requests, commits, tasks completed—and quietly treat them as proxies for outcomes like delivery speed, software quality, or developer effectiveness. The two are not the same.

EngThrive instead organizes measurement around outcome dimensions like Speed, Ease, and Quality, with Thriving serving as a guardrail: if an intervention makes developers faster but leaves them burned out, we don’t consider it a success. We use a handful of outcome-oriented North Star metrics supported by diagnostic metrics that help explain why those outcomes move.

With that frame in mind, here are three case studies that fundamentally changed how I think about measuring, and improving, developer productivity.

Case one: Improving one outcome changed four

“Too many meetings” is one of the most frequently cited workplace challenges among software engineers. So in late 2025, Microsoft’s CoreAI organization launched an initiative to protect developers’ focus time. Rather than simply banning meetings, leaders set an explicit target: lift the bottom 20% of developers to at least 25 hours of focus time per week.

To achieve this goal, teams did a handful of sensible things. They removed low-quality meetings, they clustered meetings together to form larger uninterrupted blocks of time, and they explicitly blocked time on their calendars to do focus work.

Within eight weeks, the results showed up across multiple dimensions. Focus time increased by 2.1 hours per developer per week, roughly twice the improvement seen in the control group. Bad Developer Days, a composite measure of daily developer friction, fell by 25%. PR velocity increased by 13%, about four times the control group. Taken together, the productivity gains were roughly equivalent to adding the output of 350 developers.

What surprised me wasn’t that focus time improved. It was that improving focus time seemed to improve things we weren’t directly trying to change.

Only about half of the reduction in Bad Developer Days could be explained by the additional focus time itself. Teams appeared to be using their newly protected capacity to pay down technical debt and eliminate other sources of friction that had been generating bad days in the first place. Creating focus time didn’t just help developers concentrate. It gave teams the space to improve the system that had been interrupting them.

That’s exactly why I think outcome-oriented measurement matters. If we had only measured focus time, we would have concluded that developers gained two extra hours each week. Looking across multiple dimensions revealed something much more interesting: the intervention triggered improvements well beyond its original goal.

One team lead summarized the lesson better than I could:

Metrics led to questions, questions led to improvements, improvements reinforced the metric’s value.

Case two: When gaming the metric is the right answer

One of the most common objections to productivity metrics is that people will game them. I worry about that too, but I increasingly think it’s also one of the best tests of whether a metric is well designed. The best metrics are ones where gaming them is indistinguishable from genuine improvement.

Time-to-First-PR is my favorite example.

One organization of roughly 4,000 developers decided to “game” the metric on purpose by assigning every new hire a trivial pull request on their first day. As a gaming exercise, it worked exactly as intended. Time-to-First-PR improved by 30%.

The surprise came later.

Those same developers went on to complete 23% more pull requests over their first year than the control group. Interviews explained why. The first pull request was never really about the code. It was about setting up the development environment, learning the team’s tools and review process, and becoming comfortable contributing. By forcing all of that to happen in the first week, the organization didn’t just improve a metric. It accelerated onboarding.

A separate AI-assisted onboarding tool, FirstMate, arrived at the same conclusion from a different direction. By automating environment setup and helping new hires navigate an unfamiliar codebase, it reduced Time-to-First-PR by 65%.

That’s the lesson I keep coming back to:

A well-designed metric shouldn’t be easy to game. It should be difficult to improve without doing something genuinely valuable. When that happens, gaming the metric and improving the system become the same thing.

Case three: A cost that turned out to be free

During the burnout crisis of 2020, one organization tried something that looked reckless on a Speed dashboard: it gave every developer two unexpected days off, called Health Days.

At first, the metric behaved exactly as you would expect. Pull request output dropped during those two days.

Then something surprising happened.

Within two weeks, the “lost” pull requests had all been made up. The apparent productivity cost disappeared. The burnout relief, however, lasted another 14 weeks.

To me, this is the clearest example of why we treat Thriving as a guardrail rather than just another metric. If we had only looked at Speed, Health Days would have appeared to be a costly intervention and might never have been attempted again. Looking across multiple dimensions told a completely different story. The intervention was effectively free from a Speed perspective while delivering a sustained improvement in developer wellbeing.

I’ll also acknowledge an important limitation. This wasn’t a controlled experiment; it was one organization’s experience. But it’s consistent with a pattern we’ve seen repeatedly: leaders often assume wellbeing and productivity exist in tension, when in practice the trade-off is frequently much smaller than expected (or doesn’t exist at all).

The lesson isn’t that every organization should schedule Health Days. It’s that looking across outcome dimensions gives leaders permission to try interventions that a single productivity metric would immediately reject.

Why this matters for engineering leaders

The common thread across all three stories isn’t focus time, onboarding, or Health Days. It’s that none of those interventions came from optimizing a single activity metric. They came from measuring outcomes, looking across dimensions, and using supporting metrics to understand why those outcomes changed.

That’s the distinction I hope readers take away from the EngThrive work. A good measurement system isn’t just a reporting system. It’s a learning system. It doesn’t simply tell leaders whether things are getting better or worse. It helps them discover interventions they wouldn’t have tried otherwise, understand why they worked, and build confidence in repeating them.

Activity metrics still have an important role to play, but not as the destination. They’re clues. They help explain why an outcome changed, not whether it mattered in the first place.

Ultimately, I think that’s the shift engineering organizations need to make. Stop asking, “What should we measure?” Start asking, “What decisions are we trying to make, and what measurements would help us make them better?” The metrics should serve the intervention, not become the intervention.

None of this is the work of one person, or even four. EngThrive is the product of a large team that has spent years building the platform, the research, the surveys, and the discipline behind these results. If these stories are useful, the credit belongs to them.

This week’s featured DevProd job openings. See more open roles here.

• Ashby is hiring an Staff Platform Engineer | Remote

• Carta is hiring a Senior Software Engineer II, Developer Experience | Santa Clara, CA; San Francisco, CA; New York, NY

• Figma is hiring a Staff Software Engineer, Developer Experience | Remote; US

• GM is hiring a Principal Software Engineer, Developer Experience | Austin, Texas

• Morgan Stanley is hiring an AI Platform Engineer - Vice President | New York

• Notion is hiring a Software Engineer, Developer Experience | Hybrid; Hyderabad, India

• Vercel is hiring a Sr. Engineering Manager, Platform | New York City, San Francisco

That’s it for this week. Thanks for reading.

-Brian

Share

In this closing panel from DX Annual, Rafe Colburn, Chief Product and Technology Officer at Etsy; Jesse Adametz, Senior Director of Engineering, Platform Engineering at Twilio; Eirini Kalliamvakou, Research Advisor at GitHub; Collin Green, Senior Staff UX Researcher at Google; and Brian Houck, Senior Principal Applied Scientist at Microsoft debate some of the biggest questions surrounding AI and engineering productivity.

They discuss whether AI will reduce the need for engineers, how AI is affecting technical debt, the future role of software engineers in an agentic world, and whether organizations should mandate AI adoption. They also explore how bottlenecks are shifting across the software development lifecycle, the challenges facing junior engineers, and why learning, culture, and change management may ultimately matter more than the tools themselves.

Some takeaways:

AI is changing software engineering, but not eliminating the need for engineers

• The panel largely rejected the idea that an AI-first SDLC means dramatically fewer engineers. As the cost of building software decreases, demand for software is likely to increase, creating new opportunities rather than eliminating the need for technical talent.

• Several panelists argued that the role of software engineers will evolve rather than disappear. The tasks that make up the job may change, but organizations will continue to need people who can solve problems, make decisions, and build systems.

Technical debt remains a tradeoff, not just an AI problem

• Panelists disagreed on whether AI is creating technical debt faster than it can remove it. Some argued that AI is accelerating both code generation and technical debt, while others believed the underlying business pressures that create technical debt remain largely unchanged.

• The discussion also introduced the idea of cognitive debt. As engineers rely more heavily on AI-generated code, understanding and maintaining systems may become more difficult even if development velocity increases.

The future engineer may work at a higher level of abstraction

• Several panelists predicted that engineers will spend less time writing code directly and more time defining intent, setting constraints, providing context, and validating results. Rather than replacing engineering work, AI may shift it to a different level of abstraction.

• The panel also pushed back on the idea that engineers will simply become managers of agents. Effective AI use still requires technical judgment, communication skills, and careful oversight.

Mandates rarely create meaningful AI adoption

• Most panelists opposed the idea that organizations should mandate AI usage. Instead, they emphasized enablement, reducing friction, and helping developers discover value through their own workflows.

• Usage metrics can easily become the wrong goal. The group cautioned against treating AI usage itself as a performance metric, arguing that outcomes matter more than activity.

Junior engineers remain essential to the future of the profession

• The panel strongly rejected the idea that organizations will no longer need junior engineers. Today’s junior engineers become tomorrow’s senior engineers, making talent development critical to the long-term health of the industry.

• Several speakers also noted that newer engineers may bring valuable AI-native perspectives. Just as previous technology shifts rewarded developers who grew up with new tools, the next generation may help shape how AI is used in practice.

The biggest AI adoption challenges are human, not technical

• While tooling matters, the panel repeatedly returned to learning, culture, incentives, and change management as the biggest barriers to successful AI adoption. Engineers are navigating rapid technological change, shifting workflows, and new expectations about their role.

• Organizations that create space for learning appear to see stronger results. The panel highlighted examples where teams learned together, experimented together, and achieved better adoption outcomes than individuals working in isolation.

In this episode, we cover:

(00:00) Intro

(01:16) Why an AI-first SDLC doesn’t mean fewer engineers

(03:09) The debate over AI and technical debt

(07:40) AI-generated code and the future role of engineers

(14:16) Why mandating AI use doesn’t necessarily lead to better outcomes

(20:43) Predictions for the future of junior engineers

(23:22) Where the bottlenecks are in the SDLC now

(28:25) How risk influences AI use

(32:38) Why the human side is the biggest AI adoption challenge

Referenced:

• Etsy

• GitHub

• Microsoft

• Twilio

• Google

• Stewart Reichling

• What is the SPACE framework and when should you use it?

Indeed increased AI coding tool adoption from roughly 25% to 97% across its engineering organization, but getting engineers to use the tools was only part of the challenge.

In this session from DX Annual, Michael Redding, Principal Product Manager, and Jeff Davis, VP of Core Infrastructure at Indeed, explain how the company used structured training, leadership support, and ongoing community engagement to help more than 2,000 engineers build practical AI skills. They share why an early train-the-trainer model fell short, how they redesigned their approach around hands-on learning, and what they learned about balancing adoption, measurement, and psychological safety.

They also discuss the impact of the program on coding time, the role of continuous enablement after formal training ended, and how Indeed is preparing for the next phase of AI adoption, including agentic workflows and AI-powered coaching.

Some takeaways:

Indeed started with a productivity problem, not an AI problem

• At the beginning of 2025, Indeed’s DX survey showed that only about half of developer time was being spent on new features and innovation. The remaining 48% was consumed by maintenance, upgrades, incident response, and other forms of engineering overhead.

• The company’s AI strategy focused on two goals: reducing overhead work and increasing output during coding time. The long-term objective was to double engineering productivity by shrinking non-value-added work while helping engineers produce more during the time they spend building.

AI Coding Essentials succeeded where AI Coding Ambassadors fell short

• Indeed’s first enablement effort, AI Coding Ambassadors, used a train-the-trainer model built around roughly 60 AI champions across the organization. While ambassadors maintained high levels of engagement, adoption among their teammates declined after the program ended.

• The company responded by launching AI Coding Essentials (AICE), a structured training program designed for all engineers. The experience convinced the team that direct, hands-on learning was far more effective than relying on knowledge to spread organically through teams.

Indeed treated AI upskilling as a company-wide investment

• Training more than 2,000 engineers required significant organizational commitment and leadership support. Michael estimated the investment at roughly $3–4 million in engineering time across the company.

• Rather than mandating AI usage, Indeed strongly encouraged completion of the training itself. Managers were given visibility into participation, while engineers retained flexibility in how and whether they ultimately incorporated AI into their workflows.

AI adoption increased from 25% to 97%

• Despite offering AI tools, training resources, and executive support, weekly AI usage remained stuck around 25% at the start of 2025. The challenge was not tool access but helping engineers develop practical skills and confidence.

• By the time of the presentation, weekly AI tool usage had reached approximately 97%. The company also successfully navigated multiple tool transitions, moving from Cody and Copilot to newer agentic tools such as Claude Code, Cursor, Windsurf, and Amp.

Structured training produced measurable results

• Engineers who completed AI Coding Essentials reduced coding time by roughly 35–36%, while engineers who did not complete the training saw little change. Across the broader organization, coding time decreased by roughly 20%.

• Indeed measured coding time as the period between a developer picking up a Jira ticket and opening a diff in GitLab. The company continued to see benefits months after training ended, especially as newer frontier models became available.

Community and continuous enablement kept momentum going

• Indeed reinforced learning through coding forums, office hours, hackathons, Slack communities, and its AI Showcase recognition program. More than 100 unique community posts were being shared monthly in the company’s primary AI channel.

• The goal was to make AI learning continuous rather than event-based. Engineers had multiple ways to share discoveries, get help, and learn from peers long after formal training concluded.

The next challenge is moving beyond coding

• Indeed is now focused on agentic workflows, AI coaching, and expanding enablement beyond software engineering. Product managers, designers, researchers, and other R&D functions are becoming part of the company’s AI adoption strategy.

• As coding becomes faster, bottlenecks are beginning to shift elsewhere in the development lifecycle. The team is already monitoring signs that code review and other downstream activities may become the next constraints on engineering throughput.

In this episode, we cover:

(00:00) Intro

(01:05) Indeed’s DX survey from January 2025

(02:30) The two-part strategy to double engineering productivity

(04:21) How Indeed increased AI adoption from 25% to 97%

(15:40) Results from Indeed’s AI training program

(18:33) How Indeed sustains AI adoption and learning

(23:06) What’s next for AI enablement at Indeed

(24:41) Q&A: How coding time was calculated

(25:25) Q&A: How Indeed uses AI playbooks

(26:40) Q&A: Balancing asynchronous and live AI training

(28:22) Q&A: Psychological safety during AI adoption

(31:44) Q&A: Why AI adoption spikes after the holidays

(33:20) Q&A: The metrics Indeed tracked

(35:22) Q&A: Where the time savings are going

(36:54) Q&A: Reaching engineers who skipped the training

(38:08) Closing thoughts

Referenced:

• Indeed

• Claude Code | Anthropic’s agentic coding system

• Cursor

• Windsurf

• Amp Code

• The Complete Guide to Building Skills for Claude | Anthropic

• Measuring developer productivity with the DX Core 4

In this session from DX Annual, Uma Namasivayam, Senior Director of Engineering Productivity at Dropbox, shares how the company’s developer productivity efforts evolved from improving developer experience to preparing for the agentic era.

He explains how Dropbox approached AI adoption across its engineering organization, the impact it had on developer productivity, and why faster code generation is creating new bottlenecks in areas such as code review, validation, and CI/CD. He also discusses Dropbox’s efforts to rethink engineering systems, measurement, and workflows, including the development of agentic tooling and new metrics designed to move beyond PR throughput and toward product velocity.

Some takeaways:

Dropbox’s productivity journey started before AI

• DXI helped Dropbox identify productivity problems as system problems rather than talent problems. When the company began measuring developer experience in 2023, it found significant variation across teams in DXI scores, PR throughput, and cycle time.

• Measuring developer experience created a framework for prioritizing investments. The team used DXI to identify friction across areas such as debugging, documentation, and build systems while giving leadership a common language for discussing productivity.

AI adoption required more than access to tools

• Dropbox combined executive support, developer segmentation, enablement, and strong guardrails to drive adoption. Different teams and developer roles were matched with different tools and workflows based on their needs.

• The approach helped Dropbox increase AI adoption from roughly 30% to 100% within six months. During the same period, PR throughput doubled and developer satisfaction with AI tools increased significantly.

Engineers used their extra capacity to tackle neglected work

• As AI increased throughput, engineers naturally pulled maintenance work, migrations, and technical debt from the backlog. Dropbox saw significant growth in these categories without any specific direction from leadership.

• The additional capacity was often reinvested into engineering health. Teams used the opportunity to address long-standing issues that had accumulated over time rather than focusing exclusively on new feature development.

The next challenges are scale, trust, and measurement

• Dropbox believes the move to agentic engineering creates three major challenges: scale, validation and trust, and measurement. Existing development systems were not designed for a world where AI dramatically increases code throughput.

• As code generation accelerates, bottlenecks are shifting toward code review, validation, and CI/CD systems. The company is already seeing pressure move downstream in the software development lifecycle.

Agentic engineering requires redesigning the entire system

• Uma compared the transition to the shift from steam-powered factories to electric factories. The biggest gains came from redesigning the entire system rather than simply replacing one technology with another.

• Dropbox is investing in agentic workflows across the SDLC and building Nova as an orchestration layer. The company is evaluating roughly 30 development steps, and one in twelve pull requests is already being generated by Nova.

PR throughput is becoming a less useful measure of productivity

• Dropbox believes traditional engineering metrics need to evolve alongside AI. As agentic workflows become more common, measuring productivity through pull request volume alone provides an incomplete picture of engineering output.

• The company is increasingly focused on metrics such as AI contribution, loaded cost per PR, agentic workflow coverage, work distribution, and time to ship. The goal is to better connect engineering activity to customer value and business outcomes.

In this episode, we cover:

(00:00) Intro

(00:57) The beginning of Dropbox’s DX journey

(02:34) AI adoption at Dropbox: what made it work

(04:46) The results of Dropbox’s AI adoption efforts

(05:39) What the results mean for the business

(06:55) The phases of AI adoption and where they are now

(08:00) The new bottlenecks

(09:16) Three challenges Dropbox faces moving into agentic engineering

(10:05) How Dropbox is redesigning the SDLC for agentic engineering

(15:46) The new metrics that matter

(19:16) Final takeaways

Referenced:

• Dropbox

• Developer Experience Index (DXI) | DX

• DX Core 4 Productivity Framework

• Cursor

• Claude Code | Anthropic’s agentic coding system

• JetBrains

• Visual Studio Code

• Jira | Project Management for the AI Era | Atlassian

• GitHub

Subscribe now

🗓 Join me on July 23 for a readout of the upcoming Q2 2026 AI Impact Report. We’ll discuss new findings from DX’s data on AI tool usage, spend, and impact across 500+ organizations. Register here.

When AI coding tools started delivering meaningful results, a predictable question followed from CTOs and engineering leaders: how do we measure the impact? There is a strong instinct to assume that the frameworks built over the last decade no longer apply, and that the age of AI demands a fundamentally different measurement architecture.

I’d push back on that instinct. The evidence suggests the opposite is closer to the truth.

While AI represents a massive paradigm shift in how software is built, it does not alter what engineering organizations are ultimately trying to accomplish. Foundational engineering principles still map to high-level outcomes. How quickly is value delivered? How easy is it for developers to do their work effectively? How stable are the systems? And, ultimately, what is the business impact of the work? Rather than rendering these categories obsolete, the introduction of AI makes anchoring to a stable, outcome-oriented framework more critical than ever.

Engineering leaders are under unprecedented pressure to justify the massive budgets being poured into AI tooling. When executives demand proof that an AI investment is paying off, the immediate temptation is to reach for a shiny new metric that isolates the tool itself. But that is exactly where the risk lies.

The DX Core 4 framework (speed, effectiveness, quality, and business impact) is built around answering these persistent questions. It was designed to give engineering leaders a durable measurement architecture that survives new technology cycles. AI is a significant shift in workflow, but because the framework anchors to macro outcomes rather than the mechanics of coding, it remains stable. If anything, the rise of AI makes this type of durable framework more important, not less.

This article makes three related arguments:

1. First, the high-level dimensions of engineering productivity remain remarkably stable, even as AI transforms how software is built.

2. Second, AI-specific telemetry should be treated as diagnostic context rather than a replacement for outcome-oriented measurement.

3. Finally, while many traditional engineering metrics remain valuable, the behaviors that generate them are changing, and disentangling those signals requires triangulating across the layered structure of diagnostic, system, and outcome metrics.

The Core 4 holds (and here’s why that matters)

The value of anchoring to these four overarching dimensions—speed, effectiveness, quality, and business impact—is that they synthesize key principles from DORA, SPACE, and DevEx into a unified methodology. Core 4 inherits DORA’s focus on delivery outcomes, SPACE’s insistence that productivity is multidimensional, and DevEx’s emphasis on the lived experience of developers—and combines them into a four-dimension framework optimized for executive decision-making.

AI doesn’t change what engineering organizations are trying to accomplish. What it does is make the signals noisier.

As AI coding assistants become standard and agentic workflows begin handling multi-step tasks autonomously, traditional activity metrics shift in ways that can easily mislead. Pull request counts spike, cycle times compress, and code volumes bloat. Engineering leaders who chase these surface-level fluctuations without anchoring to a balanced, outcome-oriented framework risk optimizing for sheer motion rather than actual progress.

This is precisely where a high-level outcome framework proves its utility. I’m using the Core 4 as the specific example here, but the same logic applies to any mature measurement framework aligned to the principles of SPACE. By focusing on outcomes that matter, regardless of how code gets written, the model remains insulated from technology disruptions. This structural design looks increasingly necessary as developer workflows continue to evolve away from manual synthesis and toward intent-driven architecture.

Activity vs. outcome: The role of AI telemetry

To be clear, focusing on measuring stable macro outcomes does not mean engineering leaders should ignore AI adoption and usage. Tracking how developers engage with AI tools is incredibly valuable, but it is critical to understand what those metrics are telling us.

AI adoption, token usage, and the number of tasks assigned to agents are examples of diagnostic telemetry. Like more traditional operational metrics such as pull request size, build duration, or meeting load, they provide visibility into how work is being performed rather than whether it is producing better outcomes.

One way to think about this distinction is illustrated in the image below, whether AI-specific or traditional, helps explain the mechanics of software delivery and the dynamics of the engineering system. By contrast, outcome-oriented frameworks evaluate whether those operating patterns are ultimately translating into better engineering results.

Specialized measurement frameworks can help organize these diagnostic signals. For example, DX’s AI Measurement Framework combines AI-specific telemetry around utilization and cost with outcome-oriented metrics to evaluate AI’s overall impact on engineering organizations. These two classes of measurement answer fundamentally different questions: “How is work being performed?” versus “Is the engineering organization delivering better outcomes?”

The value of tracking AI activity is that it helps us understand the shifting patterns that lead to our outcomes. For example, if a team’s AI adoption spikes to 90%, that metric alone doesn’t prove success. Instead, it serves as a lens to interpret changes in the Core 4: did that spike in adoption correlate with an increase in speed? Did it negatively impact quality via a higher change failure rate? Or did it inadvertently degrade developer effectiveness by introducing new code-review bottlenecks?

Tracking AI telemetry tells us how the work is changing. Tracking the core dimensions tells us if that change is actually delivering results.

When leaders are tasked with proving AI investment ROI, they cannot do it by pointing to adoption spikes or token volume. A high utilization rate means nothing if software delivery stalls or system stability crashes. Outcome-based developer experience metrics aren’t just a way to measure engineering anymore, they may be the most reliable ledger for proving AI value.

PR throughput in the AI era

Of the key metrics within the Core 4, PR throughput has attracted the most debate, both before and after the arrival of AI.

The criticism of PR throughput is entirely fair at the individual level. Not all PRs are created equal in terms of size, complexity, or value. DX developed a methodology called TrueThroughput, which uses AI to normalize these variations by weighting PRs based on actual complexity. Yet, even with that kind of normalization in place, the metric is a poor instrument for evaluating any individual developer’s contribution. I’ve argued this myself, and I’d stand by it. Using PR throughput to assess individuals is the wrong application of the metric.

At the system level, though, it remains one of the most useful signals available. The reason is that it doesn’t just measure output, it measures engineering flow. Whether code in a pull request was written by a human or generated by an AI agent, if it’s moving through review, CI, and deployment without friction, the metric reflects that. If it’s stalling—because review is bottlenecked, builds are flaky, or deployment processes are slow—the metric surfaces that too. PR throughput is a signal for whether an engineering system can move work through, regardless of where that work originates.

It also occupies a unique position among the Core 4 metrics. Unlike measures such as Change Failure Rate or DXI, which continue to evaluate enduring organizational outcomes, PR throughput is directly tied to the mechanics of software delivery. As workflows evolve from code-first to intent-first development, the role of the pull request itself may change substantially, making PR throughput more susceptible to reinterpretation than most other metrics in the framework.

In our own longitudinal research at DX, we found that AI coding tools produced roughly a 7.8% increase in PR throughput across organizations that had adopted them. That’s a real and meaningful signal. It’s also a useful corrective to more optimistic claims about AI’s productivity impact. The gains are real; they tend to be more modest than headline figures suggest, and they vary considerably across different types of work.

The majority of code shipped in production is still written by humans, though that share is shifting. Our research showed that during the first quarter of 2026, the percentage of code generated by AI that reaches production is 27.4% of production code on average. For most engineering organizations today, pull requests remain the primary unit of software delivery, making PR throughput one of the clearest indicators of engineering system flow.

If, and when, the transition to intent-first workflows materializes, the field will likely need a metric that captures innovation velocity as a higher level of abstraction. The Idea-to-Customer velocity metric introduced in the recent EngThrive framework paper is one implementation worth watching as a future key metric for the speed dimension. But even in that future, PR throughput will likely remain a crucial secondary metric for diagnosing system flow.

Evolving the interpretation, not the framework

To recap, the top-level dimensions of the DX Core 4 are stable and as meaningful as ever. The key metrics that support them also continue to hold.

What is changing is the diagnostic layer beneath them, the operational signals that have always helped explain how engineering systems produce those outcomes. AI doesn’t change what good looks like at the outcome level, but it does change the mechanisms that generate many of our familiar diagnostic metrics. The same number can now be produced by very different combinations of human and AI behavior, which means individual diagnostic metrics are noisier than they used to be, and the signals they do provide may relate to outcomes in different ways than they used to.

Take, for example:

• PR Merge Rate: Historically, a high merge rate signaled a highly aligned team shipping clean, uncontroversial work. In an agentic workflow, does a 95% merge rate mean the AI is flawless? Or does it mean your human developers are rubber-stamping machine-generated code because they’re too overwhelmed to properly review it?

• Time-to-10th-PR: This is currently one of my favorite onboarding metrics because it is highly predictive of a new hire’s long-term success and speed-to-productivity. But its utility faces an unresolved question: if an AI onboarding assistant can help an engineer generate and ship 10 PRs by their second afternoon, does that metric still capture true structural onboarding health? Or does it just track how quickly someone learned to use AI tools?

This is the core challenge. The data points themselves have not changed, but the behaviors that generate them have. AI activity metrics, such as tool adoption or token counts, provide critical context for understanding why traditional engineering metrics move the way they do, but they do not replace those metrics.

Triangulating between diagnostic metrics, engineering system metrics, and high-level outcome metrics is what lets us translate how teams work into whether they’re achieving what they set out to. Building a map of these new patterns—how to interpret them, and what outcomes they predict—will be critical work for engineering teams and researchers moving forward.

Final thoughts

The instinct to reach for entirely new metrics in this age of AI is understandable. AI is genuinely reshaping how software gets built, and it is reasonable to question whether existing measurement frameworks can keep pace.

But our research and data show that the core dimensions of productivity have held up, not because they anticipated AI specifically, but because they were designed around enduring organizational outcomes rather than any particular workflow or technology. Speed, effectiveness, quality, and business impact remain the right questions to ask, whether code is written by a developer at a terminal or generated by an autonomous agent.

What has changed is not what we should measure, but how we should interpret it. AI-specific telemetry provides valuable diagnostic context for understanding how work is evolving, but it does not replace outcome-oriented measurement. Likewise, familiar engineering metrics such as PR throughput, merge rates, or onboarding velocity continue to provide meaningful signals, even as the behaviors that generate those signals shift.

The priority for engineering leaders is not to rebuild their measurement architecture from scratch. It is to learn to interpret existing frameworks through a new lens, one that recognizes the growing role of AI while remaining anchored to the outcomes that ultimately matter.

The framework is stable. The interpretation is where the real work begins.

That’s it for this week. Thanks for reading.

Share

In this session from DX Annual, Christopher Sanson, Product Lead, AI Developer Experience, and Madison Capps, Engineering Manager, Infrastructure at Airbnb, challenge some of the most common assumptions about AI. Is AI primarily about replacing humans? Do organizations need mandates to drive adoption? And are the productivity gains really as small as some studies suggest?

Using examples from Airbnb’s own AI journey, they share how the company achieved widespread adoption of agentic AI through AirChat, community enablement, and internal tooling rather than top-down mandates. They also discuss the impact AI is having on developer productivity, how non-developers are increasingly using coding tools, and how teams are rethinking product development in an AI-first world.

Finally, Madison takes a deeper look at the infrastructure powering Airbnb’s AI strategy, including AirChat CLI, the AirChat SDK, and AirChat Remote, along with the company’s vision for asynchronous agent workflows and the next generation of AI-powered development.

Some takeaways:

AI adoption at scale

• Successful AI adoption does not require mandates. Airbnb achieved 97% weekly usage and 90% daily usage of agentic AI tools among engineers without tying adoption to performance reviews or quotas. Christopher argued that the best adoption comes when developers choose to use AI because it genuinely helps them work faster and better.

• Treat internal AI tools like products, not internal infrastructure. Airbnb built a recognizable brand around AirChat, invested in onboarding and workshops, created internal marketing materials, and focused heavily on user experience. That product mindset helped turn AirChat into a company-wide platform rather than just another engineering tool.

• Community-driven learning scales better than centralized training. AI champions, train-the-trainer programs, hackathons, workshops, and active peer-to-peer learning channels allowed knowledge to spread organically across the company. Over time, the AI community became larger and more active than the team managing the platform itself.

Productivity gains are accelerating

• Developers are spending more time actively coding. Christopher challenged the idea that engineers only spend a small percentage of their time writing code. As coding becomes faster and easier with agentic AI, developers can spend more of their week building software rather than working around implementation bottlenecks.

• The most active AI users see the largest productivity gains. Airbnb found that developers who spent four or more hours per day working with agentic AI dramatically increased their output. The relationship between AI usage and productivity became stronger as engineers learned how to incorporate agents into their daily workflows.

• PR throughput increased by 65% after the introduction of agentic AI. Airbnb’s data suggests that productivity gains extend well beyond the single-digit improvements often cited in industry studies. Developers who heavily embraced agentic AI moved from industry-average output to some of the highest throughput levels measured internally.

• AI-authored code is becoming mainstream. Roughly 59% of Airbnb’s code is now primarily authored by AI, and more than half of developers report that AI generates the majority of the code they work with. Christopher argued that this shift is happening far faster than most organizations realize.

AI is spreading beyond engineering

• The addressable market for AI is much larger than developers alone. Airbnb initially expected adoption to level off around its engineering population. Instead, usage continued growing as product managers, designers, finance teams, and operations teams began integrating agentic AI into their work.

• People will learn new workflows when the value is obvious. Some non-engineering teams adopted VS Code and terminal-based tools simply because they provided the best access to agentic AI capabilities. Rather than resisting technical tools, employees were willing to learn them in exchange for meaningful productivity gains.

• Domain experts are increasingly building their own AI-powered solutions. Airbnb’s internal platforms allow teams to create specialized applications tailored to their own workflows. This shifts more problem-solving into the hands of the people closest to the business problem.

Rethinking how work gets done

• Many existing processes were designed around expensive software development. Product reviews, lengthy requirements documents, and sequential handoffs evolved in a world where implementation was slow and costly. AI changes those economics and creates opportunities to redesign workflows from first principles.

• AI enables faster movement from ideas to prototypes. Rather than spending weeks refining specifications before building anything, teams can generate multiple prototypes quickly, test ideas earlier, and iterate before committing significant resources.

• Smaller teams can collaborate earlier and move faster. Airbnb sees opportunities to reduce handoffs between product managers, designers, and engineers by bringing teams together earlier in the process and using AI to accelerate exploration and execution.

Building for asynchronous AI

• Current agentic AI tooling still creates friction. Managing multiple sessions, handling long-running tasks, maintaining context, and switching between workflows remain cumbersome despite major advances in model capabilities.

• The next frontier is asynchronous agent workflows. Rather than interacting with a single agent in real time, developers are increasingly orchestrating multiple agents working in parallel, often across long-running tasks that continue without constant supervision.

• Airbnb is investing in infrastructure, not just models. AirChat CLI, migration tooling, the AirChat SDK, and AirChat Remote were all built around the belief that future gains will come from workflow orchestration, platform capabilities, and developer experience as much as from improvements in foundation models.

Preparing for an AI-first future

• Organizations should build for where developer workflows are heading. Madison described Airbnb’s approach as continuously forecasting how engineers are likely to work in the near future and investing in the infrastructure required to support those workflows before they become mainstream.

• AI-first architecture will become increasingly important. As throughput rises and more work is delegated to agents, teams will need stronger guardrails, scalable platforms, and systems designed specifically to support AI-assisted development.

• The biggest bottlenecks are shifting away from code generation. As AI reduces implementation costs, constraints move elsewhere in the system. Coordination, validation, infrastructure, and workflow management are becoming the new challenges organizations must solve.

In this episode, we cover:

(00:00) Intro

(01:37) Myth #1: AI is about replacing humans

(03:22) Myth #2: You need mandates to drive AI adoption

(05:21) AirChat, agentic AI, and Airbnb’s adoption strategy

(08:07) Myth #3: AI has little impact on productivity

(09:33) Airbnb’s increase in coding time and PR throughput

(14:20) Myth #4: AI coding tools are just for coders

(15:39) How non-developers are using coding tools

(17:24) Rethinking product development in an AI-first world

(20:30) Myth #5: Vibe coding isn’t coding

(22:16) Unsolved problems in agentic AI tooling and how Airbnb is addressing them

(26:30) Airbnb’s overall AI philosophy in practice

(29:15) Using agentic AI to accelerate code migrations

(30:18) AirChat SDK: How Airbnb enables teams to build AI-powered applications

(33:17) AirChat Remote and asynchronous agent workflows

(36:07) Predictions for what’s next

Where to find Christopher Sanson:

• LinkedIn: https://www.linkedin.com/in/christophersanson

Where to find Madison Capps:

• LinkedIn: https://www.linkedin.com/in/madison-capps-66950625

Referenced:

• Airbnb

• Steve Jobs’s Bicycles for the Mind

• Jennifer St Pierre

• Justin Reock

• AI-generated merged code holds steady at ~30%

• Andrej Karpathy’s post on X

In this session from DX Annual, Rebecca Fitzhugh, Lead Principal Engineer at Atlassian, moderates a panel featuring Nidhi Allipuram, Vice President, Enterprise Developer Experience and Platform at Nationwide, Jai Schniepp, Senior Director, DevX Product Management at Comcast, Brent Foster, Vice President and Head of Architecture and Strategy at TD Bank, and Praveena Patchipulusu, Vice President of Engineering at HPE.

Together, they discuss how large enterprises are approaching AI adoption, what it takes to build an AI-first software development lifecycle, and how engineering leaders are balancing speed, security, governance, and developer experience. They also share their perspectives on the changing role of engineers, human accountability, and how organizations can prepare for the future of software engineering.

Some takeaways:

Building an AI-first software development lifecycle

• AI adoption is becoming a redesign effort, not a tooling effort. Several panelists argued that the biggest opportunity is not simply adding AI assistants to existing workflows but rethinking the software development lifecycle itself. Rather than treating AI as a coding tool, organizations are beginning to integrate it into requirements gathering, design, testing, code reviews, and deployment.

• Training and organizational support matter more than tool selection. Nationwide found that productivity gains came less from introducing new tools and more from providing engineers with training, coaching, playbooks, and time to learn. Teams consistently reported that air cover, psychological safety, and opportunities to experiment were more valuable than access to additional AI products.

• Successful adoption requires systems, not mandates. Organizations cannot simply tell teams to “go use AI.” Several panelists described building AI champion programs, governance models, embedded coaching, and structured learning opportunities that help teams develop new habits and scale adoption across large enterprises.

Keeping humans accountable

• Humans remain responsible for outcomes regardless of who writes the code. Every panelist emphasized that accountability does not shift to AI. Whether code is generated by an engineer, a copilot, or an agent, humans remain responsible for validating outputs, making decisions, and owning the results delivered to customers.

• Validation is becoming more important than approval. Traditional approval processes may matter less than ensuring the right people validate assumptions, outcomes, and risks. Teams are increasingly focused on creating workflows where humans review and challenge AI-generated work rather than simply acting as signoff gates.

• Decision-making is becoming a core engineering skill. As AI takes over more implementation work, engineers are spending more time evaluating tradeoffs, validating outputs, and making judgment calls. The ability to make good decisions quickly may become a larger differentiator than the ability to manually write code.

Security and governance in an AI-powered world

• Shift-left practices become even more important with AI. Security, compliance, and quality checks are being pushed earlier into the development process. Rather than relying on reviews at the end of the pipeline, organizations are embedding guardrails directly into workflows and development platforms.

• AI-generated infrastructure introduces new challenges. The conversation extended beyond application code to infrastructure. As AI increasingly generates Terraform, YAML, and cloud configuration files, organizations must build policy-driven validation and security controls to prevent vulnerabilities from entering production environments.

• Context is both a powerful asset and a potential risk. One of AI’s greatest strengths is its ability to use organizational knowledge and historical context. At the same time, exposing that information to AI systems creates new security concerns, making governance and access controls increasingly important.

The changing role of the engineer

• Engineers are becoming orchestrators rather than implementers. As AI takes over more boilerplate work, engineers are expected to focus more on system design, architecture, critical thinking, and coordinating work across humans, agents, and platforms. Success increasingly depends on defining intent and evaluating outcomes rather than writing every line of code manually.

• Role boundaries are becoming less rigid. The panel described a future where engineers, product managers, designers, and other builders work more closely together. AI is making it easier for individuals to contribute across traditional functional boundaries, creating smaller teams with broader responsibilities.

• Critical thinking and creativity become more valuable. While AI can accelerate execution, it cannot replace human judgment and problem framing. Several panelists argued that creativity, curiosity, and the ability to think differently about problems will become increasingly important as AI capabilities continue to improve.

Rethinking developer experience

• Developer experience is becoming workflow experience. The focus is shifting from individual tools toward creating trusted workflows that help teams move from idea to production more quickly. Organizations are increasingly measuring success by how effectively teams can deliver outcomes rather than by how efficiently they write code.

• Developer experience now includes agent experience. As AI agents become active participants in software delivery, organizations must consider how agents consume context, operate within guardrails, and interact with development platforms. Designing effective systems now means thinking about both human and AI users.

• Breaking down silos creates better outcomes. Several panelists argued that AI provides an opportunity to reduce friction between product managers, designers, developers, and security teams. The organizations that benefit most may be those that remove barriers between disciplines and enable more collaborative ways of working.

Preparing for the future

• The time to experiment is now. Every panelist encouraged organizations to begin learning through direct experience rather than waiting for the technology to mature. Teams that develop AI skills, workflows, and governance practices today will be better positioned as the technology continues to evolve.

• Institutional knowledge may become a competitive advantage. Large enterprises possess decades of documentation, decisions, diagrams, and expertise that often remain difficult to access. Several speakers highlighted the opportunity to unlock that knowledge and make it useful through AI-powered systems.

• Fundamentals still matter. Despite rapid technological change, the panel repeatedly returned to the same conclusion: strong engineering fundamentals, sound judgment, accountability, security practices, and critical thinking remain essential regardless of how much AI enters the software development process.

In this episode, we cover:

(00:00) Intro

(02:28) The AI journey across TD Bank, Comcast, and HPE

(05:59) Inside Nationwide’s AI-assisted development lifecycle

(10:04) Reimagining the software development lifecycle with AI

(11:32) Security, governance, and human accountability

(15:27) Embedding security and guardrails into AI workflows

(17:55) How AI is changing the role of an engineer

(21:52) What developer experience looks like in the AI era

(26:55) What software engineering may look like in 2030

(32:47) How to prepare for the AI-driven future

Where to find Rebecca Fitzhugh:

• LinkedIn: https://www.linkedin.com/in/rmfitzhugh

• X: https://x.com/RebeccaFitzhugh

Where to find Jai Schniepp:

• LinkedIn: https://www.linkedin.com/in/jessicaschniepp

Where to find Nidhi Allipuram:

• LinkedIn: https://www.linkedin.com/in/nidhi-allipuram

Where to find Brent Foster:

• LinkedIn: https://www.linkedin.com/in/engineeringthefuture

• Website: https://brentfoster.me

Where to find Praveena Patchipulusu:

• LinkedIn: https://www.linkedin.com/in/praveena-patchipulusu-158741

Referenced:

• Atlassian

• TD Bank

• Comcast Corporation

• Hewlett Packard Enterprise (HPE)

• Nationwide

• GitHub Spec Kit

• Abi Noda

As AI becomes embedded in software development, many of the metrics that engineering organizations have relied on for years are starting to break down.

In this session from DX Annual, Uber’s Ty Smith and Abhishek Tibrewal share how their approach to measuring AI’s impact on developer productivity has evolved over time. They walk through the different phases of their measurement journey, from adoption and engagement to measuring impact, ROI, and agentic value, explaining what they chose to measure at each stage, what worked, what failed, and how their thinking changed along the way.

They also discuss the role of qualitative feedback before telemetry existed, the challenge of identifying meaningful engagement signals, why “developer years saved” failed as an ROI metric, and how AI agents forced them to rethink traditional productivity measurements. Finally, they introduce Uber’s emerging framework built around feature velocity and explore the unanswered questions that remain as software development becomes increasingly agent-driven.

Some takeaways:

Why AI breaks traditional productivity metrics

• Many measurement frameworks were built for a world where humans wrote most of the code. As AI agents become more capable, metrics that once provided useful signals can quickly become misleading.

• Teams should expect their metrics to break. Uber’s measurement journey required repeatedly revisiting assumptions as AI-assisted development evolved into agentic workflows.

Start with stakeholder questions

• The best metrics answer real business questions. Uber worked backward from questions about productivity, ROI, investment priorities, and business value instead of collecting data for its own sake.

• Measurement should support decision-making. Metrics influence budgets, tooling investments, enablement efforts, and long-term strategy.

• Use qualitative signals before telemetry exists

• Qualitative feedback can be the fastest path to insight. Before AI tooling generated reliable telemetry, Uber relied on surveys, interviews, and experience sampling to understand adoption and guide investments.

• Behavioral questions are more useful than perception questions. Asking developers what they actually did produced stronger signals than asking whether they found AI helpful.

• Measure engagement through behavior, not demographics

• Behavioral patterns revealed insights that demographics could not. Role, tenure, and organization offered limited signal compared to how engineers actually used AI tools.

• A small group of AI power users emerged early. Studying usage patterns helped Uber identify engineers who were engaging deeply with AI and generating outsized results.

Correlation is not causation

• High AI usage does not automatically prove AI caused higher productivity. The most productive engineers are often the first to adopt new tools.

• Rigorous analysis matters when making investment decisions. Uber used causal methods to better understand the true impact of AI-assisted development.

Why measuring AI ROI is difficult

• Developer years saved sounded compelling but failed as an ROI metric. The approach created anxiety around replacement, required constant recalibration, and did not answer the business questions leadership cared about most.

• Business leaders ultimately care about outcomes. Time saved is useful context, but value creation, customer impact, and business results matter more.

• PRs measure activity, features measure value

• Agentic AI exposes the limitations of activity-based metrics. A single agent task can generate many pull requests without creating meaningful customer value.

• Feature velocity became Uber’s new North Star. The goal shifted from measuring engineering output to measuring whether valuable capabilities were actually being delivered.

Building an AI-native measurement framework

• Feature velocity works alongside supporting metrics. Flow efficiency, quality, and capability expansion help create a more complete picture of AI’s impact.

• PR classification provides important context. Understanding the type and complexity of work helps distinguish meaningful progress from routine maintenance and toil.

• The future belongs to outcome-based metrics

• The most durable metrics are tied to business outcomes rather than engineering activity. As AI becomes more autonomous, output alone becomes a less reliable signal.

• Many important questions remain unanswered. Organizations still need better ways to measure judgment, autonomy, technical debt, and the value created by increasingly agent-driven software development.

In this episode, we cover:

(00:00) Intro

(01:30) Steve Yegge’s 8 stages of AI-assisted development

(03:22) Uber’s shift to a generative AI-powered company

(04:20) Uber’s pre-AI productivity metrics

(06:55) Important questions from stakeholders that previous metrics didn’t answer

(08:25) How Uber measures AI before telemetry exists

(11:11) Metrics used to measure adoption

(12:49) Measuring engagement

(14:30) Measuring impact

(16:32) The challenge of measuring AI ROI

(19:32) Rethinking adoption, engagement, and impact for agentic AI

(26:01) The new north star: Feature velocity

(28:41) PR classification + feature velocity: the questions it can answer

(33:01) What comes next and what’s still unanswered

(34:30) Lessons learned and what they’d do differently

(37:11) Q&A #1: How Uber defines a feature

(38:50) Q&A #2: Measuring success and AI ROI

Where to find Abhishek Tibrewal

• LinkedIn: https://www.linkedin.com/in/aabhishektibrewal

Where to find Ty Smith:

• LinkedIn: https://www.linkedin.com/in/tyvsmith

Referenced:

• Welcome to Gas Town

• Dara Khosrowshahi (Uber CEO)

Share

🗓 Join Brian Houck and me, this Thursday June 18th for a research briefing on measuring AI agents, revisiting the Core 4, and more. Register here.

In our Q1 AI impact analysis, we found that 27.4% of code was AI-authored. Because the space is changing quickly, the DX Research team reports on this metric quarterly to track changes in AI’s impact on organizations’ ability to create code.

To measure the change in AI-authored code, and the impact on quality, we conducted two analyses:

1. First we measured the percentage of AI-authored code, using self-reported data from developers. We define the metric as code generated by AI without major human rewrites.

   1. As with any self-reported metric, there is potential for bias in both directions—undercounting from fully autonomous workflows and overcounting when developers treat AI use as a performance signal. In the future we’ll share what we’re seeing from DX’s AI Code Insights, which automatically measures the percentage of AI-generated code.

   2. Our sample included DX data from over 400 companies from Q2 (April 2026-June 2026), reported by the average of user responses within each company. We interpret this data as estimates of the proportion of coding workload delegated to AI tools, rather than literal measures of code output. This reflects the assumption that respondents anchor to how often they ask AI to do work rather than measuring how much code AI actually produces.

2. Additionally, to evaluate the downstream impact of AI-authored code, we also looked at PR size using telemetry data from the same cohort over the last year (July 2025-June 2026).

   1. In the future we’ll share further investigations on the impact of increased AI-authored code, as well as the impact of increased PR size.

Here’s what we’re seeing.

AI-authored code is consistent across organization sizes

Our preliminary Q2 findings show that, on average, 51.9% of code is now AI-authored. Newer models, better workflow integration due in large part to usage of CLI tools, AI mandates, and learning curve progress have all contributed to this massive shift. While this indicates that AI is significantly impacting our ability to create code, it says little about the quality of the code being generated.

When segmented by organization size, our finding still holds. The median percentage of code that is AI-authored holds steady at around 50%. This reflects a broader shift in how code is produced, regardless of team size.

Median pull request size has nearly doubled

Because of the dramatic change in AI-authored code, we also looked at whether PR size—one measure for quality—has changed for the same cohort of companies over the past year. Interestingly, our data is showing an equally dramatic change: median PR size nearly doubled, growing from 44 lines to 72 lines per pull request between July 2025 and June 2026.

This finding confirms what many teams would expect: AI tends to generate more lines of code than humans. When the majority of code is machine-produced, that verbosity results in larger pull requests.

More broadly, this metric is becoming one of the most important to watch. Generally, more code can equal more complexity, less portability, and a greater potential for bugs and vulnerabilities. More verbose code can also be more difficult to review and maintain. One of the traits of a skilled engineer is the ability to fully implement a use case with exactly as much code as needed to perform the task. When AI undermines that instinct at scale, the result is not just technical debt. It is compounding cognitive debt across the team as engineers struggle to understand code they did not write.

The critical question for leaders: have review and quality processes kept pace with this volume? There’s been a lot of discussion in engineering leadership communities about how to shift processes to handle code review being the new bottleneck. I also appreciated Camille Fournier’s recent piece sharing guidelines for respectable use of AI, which outlines expectations leaders can set with their teams for using AI. If this is something you’re actively thinking about, please let me know in the comments—I’d love to hear from you.

That’s it for this week. Thanks for reading.

Subscribe now

Michael Galloway leads Platform Engineering at Mercari, while Snehal Shinde leads Cost and Performance Engineering. Together, they have been at the center of Mercari’s effort to become an AI-native company.

In this session from DX Annual, Michael and Snehal share what happened after Mercari’s CEO mandated 100% AI adoption across the organization. While AI accelerated code generation and increased engineering output, the team quickly discovered that their existing dashboards could not answer a simple question: was AI actually improving productivity?

They discuss how Mercari built new visibility into AI usage and software delivery, the bottlenecks they uncovered across the SDLC, why faster coding did not automatically translate into faster delivery, and the lessons they learned rolling out AI at scale. They also share how Mercari is rethinking software development around agents, feedback loops, and new ways of working.

Some takeaways:

Measuring AI impact

• AI adoption alone does not guarantee business value. Mercari found that while AI usage increased rapidly across the organization, existing dashboards could not answer the leadership team’s most important question: whether AI was actually improving productivity.

• Local optimization does not necessarily improve system-wide performance. Engineers reported working faster with AI tools, but end-to-end delivery metrics remained largely unchanged because bottlenecks elsewhere in the software delivery process continued to slow teams down.

• Organizations need visibility into both AI usage and delivery outcomes. Mercari built new dashboards that combined AI tool data with SDLC metrics to better understand adoption, throughput, quality, and operational performance.

The reality of becoming AI-Native

• AI adoption required a cultural transformation, not just a tooling rollout. Mercari’s CEO mandated company-wide AI adoption, but success depended on changing workflows, habits, and expectations across engineering, finance, legal, customer support, and other functions.

• Different teams required different forms of enablement. Employees varied significantly in their technical backgrounds and familiarity with AI tools, making education, workshops, and support systems essential to driving adoption.

• The goal was to rethink work itself. Rather than layering AI onto existing processes, Mercari challenged teams to reconsider what they built, how they built it, and how people worked together.

The bottlenecks AI exposed

• AI revealed problems that already existed inside the organization. Review queues, CI instability, deployment friction, and support requests became more visible as coding accelerated.

• Code generation was not the primary constraint. Engineers often spent more time waiting for approvals, navigating organizational boundaries, and dealing with infrastructure limitations than writing code.

• System complexity amplified AI-related challenges. As AI-generated changes increased, existing architectural complexity and fragile workflows became harder to ignore.

Finding AI workflow opportunities

• Mercari mapped AI opportunities across 33 domains. The AI task force reviewed functional areas across the company to identify where AI could automate work, where it could not, and where the strongest leverage points existed.

• The biggest opportunities extended far beyond engineering. Role-specific workshops helped teams in finance, legal, design, operations, customer support, and other departments find practical AI use cases in their own workflows.

• Early wins created proof points for broader adoption. Mercari saw measurable impact from support bots, accounting workflows, platform support automation, and Socrates, an internal BI agent that made company data easier to query and use.

Rethinking software development

• Faster coding shifted attention upstream. As implementation became easier, planning, specification, and decision-making emerged as larger constraints on delivery speed.

• Agent Spec-Driven Development moves AI earlier in the lifecycle. Mercari began using agents to analyze documentation, code, and organizational knowledge before implementation work started.

• Future workflows will focus more on intent than execution. Teams increasingly define goals, constraints, and success criteria while agents handle larger portions of implementation and validation.

Preparing for an agent-driven future

• Feedback loops matter more than ever. Mercari’s multi-loop SDLC emphasizes rapid validation, iterative learning, and increasingly autonomous agent workflows.

• Behavioral change remains harder than technological change. Organizations must rethink ownership, accountability, and trust before they can fully benefit from agent-based development.

• The path to AI-native development is iterative. Mercari expects continued setbacks and learning cycles, applying what they describe as the Stockdale Paradox: maintaining confidence in the destination while remaining honest about current challenges.

In this episode, we cover:

(00:00) Intro

(01:46) Mercari’s scale and engineering culture

(02:51) DX awards at Mercari

(03:44) Mercari’s push to become AI-native

(06:34) The mandate to rethink everything

(08:02) Mercari’s AI visibility problem and how they solved it

(11:30) Mercari’s early findings on AI implementation

(18:47) Closing the AI awareness gap at Mercari

(21:11) Mapping AI opportunities across Mercari

(31:32) Unpacking the results from the second rollout

(34:14) Agent spec-driven development and what’s next

(37:37) A multi-loop SDLC

(40:50) Some hard lessons

(42:55) Closing thoughts

Where to find Michael Galloway:

• LinkedIn: https://www.linkedin.com/in/michaelroygalloway

• X: https://x.com/michaelgalloway

Where to find Snehal Shinde:

• LinkedIn: https://www.linkedin.com/in/snehal-shinde

Referenced:

• Mercari

• Cursor

• Devin

• Claude Code | Anthropic’s agentic coding system

• GitHub

• Datadog

• Tim Bozarth - Microsoft | LinkedIn

• Airbnb

• Jim Collins - Concepts - The Stockdale Paradox

Kelly Anne Pipe is Head of Developer Experience at Vanguard, and Nicole Scribner is a Director in the firm’s Chief Technology Office focused on engineering enablement and advancement.

In this session from DX Annual, Kelly Anne and Nicole share how Vanguard is expanding its AI strategy beyond software engineering to the entire product development lifecycle. While the company initially focused on tools like GitHub Copilot for engineers, they found that faster coding alone did not significantly improve delivery speed. Product managers, designers, QA teams, and organizational processes were still operating at a different pace.

To address this challenge, Vanguard developed a product team maturity model built around three stages: Augmented, Accelerated, and Autonomized. The framework spans six dimensions, from AI-powered delivery and AI-ready codebases to team autonomy, operations, and responsible AI.

Kelly Anne and Nicole explain how Vanguard is applying the model across more than 800 product teams, the behaviors they believe will enable faster delivery, and the lessons they have learned about measurement, organizational change, dependencies, and scaling AI across the product development lifecycle.

Some takeaways:

Beyond the engineering bubble

• Faster coding does not automatically lead to faster delivery. Vanguard found that while engineers using AI tools reported significant productivity gains, product managers, designers, QA teams, and governance processes were still operating at traditional speeds.

• AI adoption becomes fragmented when it is treated as an engineering initiative. Organizations that focus solely on developer tooling risk creating an “engineering bubble” where one part of the product team accelerates while the rest of the workflow remains unchanged.

• The goal is to optimize the entire product development lifecycle. Vanguard shifted its focus from helping engineers code faster to helping cross-functional product teams move faster from idea to production.

The AI maturity model

• Vanguard built a maturity model around three stages: Augmented, Accelerated, and Autonomized. The framework gives more than 800 product teams a shared language for discussing AI adoption and long-term transformation.

• The model spans six dimensions of AI maturity. These include AI-powered product delivery, AI-ready codebases, agent-powered workflows, AI-augmented operations, team autonomy and enablement, and responsible AI.

• The goal is organizational transformation, not tool adoption. The framework focuses on how entire product teams evolve as AI becomes embedded throughout the product development lifecycle.

Building AI-ready foundations

• AI readiness starts with the fundamentals. Documentation, testing, CI/CD pipelines, architecture decisions, and code quality all become more important when agents are introduced into the development process.

• The codebase becomes the interface between teams and AI agents. Poor documentation, weak test coverage, and slow feedback loops limit the effectiveness of even the most capable AI tools.

• Dependencies become more visible at agent speed. Processes that were merely frustrating for humans become major bottlenecks when AI can complete implementation work in hours rather than days.

Scaling AI beyond engineering

• Every role on the product team needs AI-specific workflows. Vanguard is focused on helping product managers, designers, QA teams, and engineers incorporate AI into their daily work rather than limiting adoption to developers.

• The most valuable opportunities often begin before coding starts. AI can help transform customer conversations, discovery work, requirements, and design artifacts into implementation-ready inputs.

• Agent orchestration changes the role of the human. As agents take on more routine execution work, people increasingly act as orchestrators, reviewers, and strategic decision-makers.

The challenges of adoption and measurement

• Behavior change is harder than deploying tools. Vanguard found that fear, uncertainty, and questions about job security often create bigger barriers to adoption than technology itself.

• Simple productivity metrics can be misleading. Measures such as lines of code generated or time saved per developer do not capture whether AI is creating meaningful business value.

• Organizations need layered measurement strategies. Adoption metrics, process improvements, cycle time, quality, and customer outcomes all need to be considered together to understand AI’s true impact.

Lessons from the AI transition

• Agent speed exposes organizational debt. Slow approvals, review queues, onboarding processes, and governance workflows become much more obvious when implementation work accelerates.

• Responsible AI can accelerate delivery rather than slow it down. Investing in guardrails, governance, security, and automated controls early enables teams to move faster with greater confidence.

• The biggest opportunity is organizational transformation. Vanguard believes the future belongs to companies that redesign entire product teams around AI rather than simply adding AI tools to existing workflows.

In this episode, we cover:

(00:00) Intro

(02:16) The state of AI one year ago at Vanguard

(02:54) The engineering bubble

(05:05) Building an AI maturity model for 800 product teams

(08:24) Dimension 1: AI-powered product delivery

(10:00) Dimension 2: AI-ready codebase

(12:20) Dimension 3: Autonomous agent utilization

(13:00) Dimension 4: AI-augmented operations

(14:00) Dimension 5: Team autonomy and enablement

(16:11) Dimension 6: Responsible AI

(18:15) The people problem: role evolution

(20:00) The measurement problem

(22:55) Lessons learned from rolling out the maturity model

(26:46) What’s ahead

(30:10) Q&A #1: Getting your codebase ready for AI

(32:22) Q&A #2: Audit trails and responsible AI

(34:16) Q&A #3: Vanguard’s maturity model progress

(36:15) Q&A #4: Measuring cycle time across 800 teams

Where to find Nicole Scribner:

• LinkedIn: https://www.linkedin.com/in/nicole-scribner-35b80422a

Where to find Kelly Anne Pipe:

• LinkedIn: https://www.linkedin.com/in/kellyannepipe

Referenced:

• Vanguard

• Jennifer St Pierre - Dell Technologies | LinkedIn

• Mercari

In this session from DX Annual, Eleanor Millman, Senior Staff Product Manager, and Mina Tawadrous, Associate Director of Product Management at SiriusXM, share how their platform engineering organization developed a prioritization framework for platform engineering teams serving hundreds of developers across a complex cloud platform.

They explain how they define and weight platform-specific impact factors, use developer data to refine priorities, and score projects more consistently. They also explore why prioritization debates often stem from conflicting, invisible, or outdated assumptions, and how SiriusXM began treating assumptions like code by documenting, versioning, and reviewing them in source control.

Finally, they demonstrate how AI can surface assumptions, connect initiatives to existing knowledge, and support project scoring while keeping humans in the loop. Throughout the session, they offer a practical framework for making prioritization decisions more transparent, data-driven, and scalable.

Some takeaways:

Building a platform engineering prioritization framework

• Platform engineering requires different prioritization criteria. SiriusXM found that traditional product metrics did not fully capture the value of platform engineering work, leading the team to define platform-specific impact factors around development speed, reliability, security, cost, platform efficiency, user trust, and data-driven decision making.

• A simple scoring model created a shared language for prioritization. The framework combined impact, urgency, effort, and business needs to help teams compare projects consistently and explain why certain initiatives were prioritized over others.

• The framework evolved alongside the organization. As company priorities changed after a major platform launch, SiriusXM adjusted impact factor weights to reflect new goals around cost optimization, technical debt reduction, and data maturity.

Using developer data to guide decisions

• Developer feedback helped shape prioritization. Rather than relying solely on intuition, the team used survey data and other developer insights to determine where additional investment would have the greatest impact.

• Impact factor weights were revisited regularly. Quarterly reviews allowed the team to adjust priorities based on changing business objectives and improvements in areas such as reliability and security.

• Data increased confidence in prioritization decisions. By grounding discussions in evidence, teams were able to align more effectively on where to invest their limited capacity.

Treating assumptions like code

• Many prioritization conflicts stem from assumptions rather than priorities. Teams often disagreed because they were working from different, invisible, or outdated assumptions about users, workflows, and business needs.

• Documenting assumptions improved organizational alignment. SiriusXM began storing assumptions in source control, making them easier to discover, review, update, and validate over time.

• Debates became more productive when assumptions were explicit. Instead of arguing over which project mattered most, teams focused on validating the underlying beliefs that informed their decisions.

Using AI to surface organizational knowledge

• Assumption repositories became difficult to navigate at scale. As more assumptions were documented, it became increasingly difficult for individuals to find relevant context and connections across projects.

• AI helped uncover relationships humans might miss. By searching assumption repositories, OKRs, and prior project data, AI was able to surface relevant information that would otherwise be difficult to discover.

• AI improved information recall rather than replacing judgment. The goal was not automated decision making but helping teams access the knowledge needed to make better decisions.

Building an AI-assisted prioritization workflow

• AI can guide teams through the scoring process. SiriusXM built workflows that ask clarifying questions, surface assumptions, identify relevant organizational context, and generate initial project scores.

• Human validation remains essential. Teams review assumptions, challenge recommendations, and approve updates before information is added back into the system.

• Each prioritization cycle strengthens the knowledge base. New assumptions, decisions, and project context become available for future initiatives, making the system more valuable over time.

Keeping humans in the loop

• The framework is designed to support conversations, not replace them. Scores help teams discuss priorities more objectively, but important decisions still require context and judgment.

• Stakeholder disagreements often reveal useful information. When the framework produces results that feel wrong, the discussion can uncover missing assumptions, incomplete data, or opportunities to improve the model itself.

• The framework continues to evolve. SiriusXM treats both the prioritization model and the supporting AI tools as products that require ongoing iteration, feedback, and refinement.

In this episode, we cover:

(00:00) Intro

(02:58) Building a platform engineering prioritization framework

(04:59) The seven platform engineering impact factors

(09:38) Using impact factors to score projects

(13:11) Using developer data to refine priorities

(16:33) Three ways assumptions fail

(17:40) Assumptions as code

(21:00) New problems created by assumptions as code

(22:00) Using AI to surface assumptions

(23:44) Building an AI-powered feedback loop

(25:44) Inside the AI prioritization tool

(28:18) Three steps to build your own framework

(30:02) Q&A #1: Evaluating high-cost projects

(31:30) Q&A #2: The cadence of iteration

(32:10) Q&A #3: When the framework conflicts with a stakeholder’s priorities

(35:26) Q&A #4: Using the framework for non-developers

Where to find Eleanor Millman:

• LinkedIn: https://www.linkedin.com/in/eleanor-millman-98b10350

Where to find Mina Tawadrous:

• LinkedIn: https://www.linkedin.com/in/mina-tawadrous

Referenced:

• AWS

• Databricks

• RICE: Simple prioritization for product managers

• Designing developer experience surveys

• GSB Preserve | View | The Curse of Knowledge

Brian Scanlan is a Senior Principal Systems Engineer at Intercom, where he works on platform engineering, developer productivity, and AI adoption across the company.

In this session from DX Annual, Brian shares how Intercom set out to double engineering throughput and ultimately achieved that goal in nine months. Rather than treating AI as an optional productivity tool, the company standardized on Claude Code, updated performance expectations, invested heavily in enablement, and adopted an agent-first approach to technical work.

Brian explains why Intercom views Claude Code as a platform rather than a tool, how the company is building domain-specific skills and workflows for agents, and why it believes agents should eventually be able to perform any technical task a senior engineer can complete on a laptop.

He also shares the data behind Intercom’s AI adoption efforts, including gains in throughput, reductions in defect backlogs, improvements in code quality, and the growing use of automated pull request approvals. Throughout the talk, Brian offers a practical look at what it takes to scale AI adoption across a large engineering organization and the lessons Intercom has learned along the way.

Some takeaways:

Doubling engineering throughput

• Intercom set a goal to double engineering throughput. Rather than focusing on AI adoption metrics, the company chose a concrete business outcome: doubling merged pull requests per member of the R&D organization.

• The goal was achieved in nine months. Intercom ultimately doubled PR throughput and later tripled it over a 16-month period, with no signs of the trend slowing down.

• Throughput was treated as a reasonable proxy for impact. While Brian acknowledged that every metric has flaws, he argued that organizations adopting AI at scale should expect to see meaningful increases in output.

Managing organizational change

• AI adoption became part of the job. Intercom updated expectations for engineers, designers, and product managers so that using AI tools effectively became part of performance expectations rather than an optional activity.

• The company combined incentives with support. Hackathons, enablement days, dedicated support teams, leadership messaging, and public recognition all helped accelerate adoption.

• Leadership stayed relentlessly on message. Repeating the same goals and expectations across every forum helped create clarity about the direction of the organization.

Why Intercom standardized on Claude Code

• Intercom chose a single AI platform. Rather than allowing teams to fragment across different tools and workflows, the company standardized on Claude Code and invested heavily in making it work well.

• The real value comes from context, not models. Brian argued that domain knowledge, skills, documentation, workflows, and organizational context create far more value than constantly switching between models.

• Agents should be treated like new employees. Intercom’s goal is to onboard agents the same way it would onboard a senior engineer by giving them access, training, tools, documentation, and clear expectations.

Building an agent-first engineering organization

• All technical work is becoming agent-first. Intercom believes that any task a human can perform on a laptop should eventually be accessible to agents.

• The focus is on durable capabilities rather than custom AI infrastructure. Teams are encouraged to build skills, access patterns, and workflows that will remain valuable even as models and tools continue to evolve.

• Agents should solve problems, not just execute commands. Rather than telling agents exactly which skill to run, engineers increasingly describe the problem and allow agents to determine the best workflow.

Skills as organizational knowledge

• Intercom has built hundreds of reusable skills. These skills capture domain expertise, troubleshooting processes, coding standards, operational procedures, and other institutional knowledge.

• High-quality skills create leverage across the organization. Once a skill exists, every engineer can benefit from the expertise embedded within it, even if they were not involved in creating it.

• Skills continuously improve over time. Engineers are encouraged to update skills whenever new knowledge is discovered so that lessons learned become available to everyone.

Measuring the impact of AI adoption

• Nearly all pull requests are now authored by Claude. Brian shared that more than 95% of pull requests are created with AI assistance, while automated pull request approvals continue to grow.

• Saved time is being reinvested into quality. As teams gained efficiency, they spent more time reducing technical debt and fixing defects, leading to a significant reduction in Intercom’s defect backlog.

• Code quality improved alongside throughput. Research conducted with Stanford showed that recent code changes were improving the overall quality of the codebase rather than degrading it.

The future of agentic software development

• Intercom wants agents to participate throughout the software development lifecycle. The company is replacing runbooks, expanding automation, and building remote agent capabilities that move work beyond individual laptops.

• AI adoption has expanded far beyond engineering. More than a thousand employees use Claude Code weekly, including teams in finance, operations, and other business functions.

• The biggest changes may still be ahead. Brian believes AI will reshape planning, team structures, workflows, and engineering roles over the coming years, not just how code is written.

In this episode, we cover:

(00:00) Intro

(02:54) Intercom’s goal of doubling throughput

(07:30) The platform strategy

(09:30) Their agent-first strategy

(10:58) Evergreen capabilities vs custom tooling

(12:28) How Intercom works with agents

(16:43) What the data reveals about AI adoption and impact

(19:20) Using session data to improve AI workflows

(20:20) Cutting the defect backlog in half

(22:44) Inside Intercom’s Claude Code setup

(28:09) Claude Code beyond engineering

(30:49) Q&A #1: Token cost

(32:52) Q&A #2: Preparing for AI pricing changes

(34:14) Q&A #3: Stress testing and auditing skills

(36:31) Q&A #4: Criteria for agents approving PRs

Where to find Brian Scanlan:

• LinkedIn: https://www.linkedin.com/in/scanlanb

• X: https://x.com/brian_scanlan

• Website: https://brian.scanlan.ie

Referenced:

• Intercom

• Software? No Way. We’re an A.I. Company Now! - The New York Times

• Anthropic

• Snowflake

• Linear

• LaunchDarkly

• Fin AI

• Microsoft Copilot

• Cursor

• Claude Code | Anthropic’s agentic coding system

• Steve Yegge (@Steve_Yegge) / Posts / X

• Honeycomb

• Fin Ideas

• Fin CLI | AI Agent Command Line Interface

Subscribe now

🗓 Join Justin Reock and me on June 18th for a research briefing on measuring AI agents, revisiting the Core 4, and more. Register here.

Last month, I had the privilege of attending the inaugural Developer Experience Research Forum at UC Irvine. It brought together researchers and practitioners from across academia and industry for a day of talks, conversations, and the kind of honest debate that only happens when the right people are in the same room.

The day included a panel that I won’t ever forget. For the first time since the SPACE framework was published in February 2021, all six of its authors were together in person: Nicole Forsgren, Margaret-Anne (Peggy) Storey, Chandra Maddila, Thomas Zimmermann, Jenna Butler, and me. I want to start by saying thank you to each of them. Collaborating on SPACE has been one of the most meaningful experiences of my career, and getting to sit alongside these colleagues five years later and take stock of what the framework has become was genuinely moving. I’m grateful to Tom, Iftekhar Ahmed, and the UCI team for making it happen, and to André van der Hoek for his amazing job moderating.

Unfortunately I don’t have a recording to make available to those who were unable to attend, but here is my attempt to capture the highlights of that conversation. The panel was wide-ranging, driven largely by questions from the audience, and covered a lot of ground. I’ll do my best to do it justice.

How SPACE came to be

For those less familiar with the backstory, SPACE did not emerge from a formal research program or a planned initiative. The idea started with Nicole. DORA, which she co-created, provided a measurement framework for software delivery, but there was a need for something that addressed developer productivity more broadly. So she reached out to a handful of colleagues (which thankfully included me), arrived with a few dimensions sketched out in her head, and the rest took shape over a series of Teams calls during what was still largely a remote-work world.

What’s remarkable is that several of us had never met in person before that project. We built the framework together, at a distance, and then watched it travel far beyond anything we had imagined. As I said on the panel:

“We could have never imagined that it was going to sort of grow into the thing it became. I don’t think you should ever hope to do something like that, because you’ll never be able to quite capture it. It’s like... right place, right time, things came together.”

The framework itself is straightforward in concept: five dimensions to consider when thinking about developer productivity. Satisfaction and wellbeing. Performance. Activity. Communication and collaboration. Efficiency and flow. The core argument is that productivity cannot be reduced to a single metric, and that a meaningful measurement approach should draw from at least three dimensions and include at least one perceptual measure. Metrics chosen well will often create productive tension with each other, and that tension is a feature, not a flaw.

Chandra reflected on the process of building it, including a detail I had honestly forgotten: the framework's original working name was not SPACE at all. It was FACTS. Trust was in there from the beginning, under a different label. Looking back on the framework more than five years later, Peggy said:

“I think we did a really good job. I think that the five dimensions have really held up really well. But they’re big — each of those dimensions are such huge concepts. And maybe what we need to do now is look at each of these in turn.”

Activity metrics: newly controversial, newly important

No dimension generated more discussion on the panel than Activity, and that’s not an accident. Activity metrics are the ones most organizations default to because they’re the easiest to instrument. They’re also the ones most prone to misuse.

What made the conversation interesting is that the panel did not argue for abandoning Activity measurement. The argument was more nuanced than that. Jenna put it directly:

"I actually think this is one of the areas where SPACE is newly important again, because you may have seen headlines about what percent of codebases are AI generated at this point. And I'm like, we're back there. We wrote about this a decade ago... some of those activity metrics like lines of code and PRs are newly resurfacing, and people are forgetting that we knew that this wasn't the greatest plan in isolation."

Chandra added that the scale has changed in a way that makes the problem even more acute. A single developer working with a swarm of agents can now generate an extraordinary volume of pull requests. The count alone tells you almost nothing about the quality, the impact, or the experience of the work.

The more useful question is not whether to measure activity, but which activity metrics are worth measuring and what you plan to do with them. I offered an example from my own work: Time-To-First-PR, meaning how long it takes a new hire to check in their first piece of code.

“Obviously easy to game, right? Have a new hire check in a trivial first PR... Turns out when you try to game it, when you explicitly try to have a trivial first check-in, it still leads to positive long-term outcomes. Why? Because that first code check-in has nothing to do with the code. It’s about learning your environment, setting up your system.”

I believe that good metric design involves choosing metrics where gaming them still gets you the outcome you actually want.

The politics of productivity measurement

One of the most candid moments of the panel came in response to an audience question about whether productivity measurement is inherently neutral or whether it inevitably becomes a political tool. The honest answer is that it is both, and you have to design for that reality.

Tom made the point that having five dimensions rather than one makes it structurally harder to play politics with the data. When you look at multiple dimensions simultaneously, you’ll often find they point in different directions, and that tension forces more careful thinking.

Jenna was direct about something that deserves to be said plainly. There is an elephant in the room across the industry right now about how many developers organizations need, and productivity metrics are being watched closely in that context.

"We tend to decouple from products and we're very... hoard-y with our data. We will give them trends. We'll let them know this is what's happening on a broad scale, or doing this had this impact. But we are not allowing individual managers, directors to look at people's information. We protect that because in theory, happy workers are productive workers. People who are terrified are not."

… and Nicole added additional framing that I found useful:

"Some data is better than no data... I know that for many of us here, we really do our best to measure in a way that is very neutral. But I know I'll have execs and other business divisions come to me and they'll say, 'Well, I need this [metric] to go up.' And I was like, 'Amazing. That's not on me. That's on you. I can give you the information and you can figure out if it goes up [or] down and why.'"

One practical safeguard worth noting is that some organizations deliberately bucket metrics together, so that no one can drive up a single number without being held accountable for the others in the cluster. It makes the kind of narrow gaming that distorts incentives structurally harder to do.

The C in SPACE: the most underinvested dimension

If Activity is the dimension that gets the most attention, Communication and Collaboration may be the one that gets the least. That gap is growing more consequential, and as Peggy put it, needs more focus than ever:

“Development is a team sport. And with AI, I don’t think there’s anyone in this room that doesn’t think that collaboration and communication hasn’t changed... If anybody here is thinking of using SPACE, make C one of the first things you look at.”

What makes this particularly important right now is that AI has changed collaboration patterns in ways we are only beginning to understand. Developers report asking questions of their AI tools that they used to ask colleagues. The texture of team communication is shifting. And yet the measurement infrastructure for tracking that dimension barely exists in most organizations. I shared a finding from my SPACE of AI paper that felt particularly relevant:

“C is the only dimension of SPACE that the majority of developers do not believe that AI has improved. Every other dimension showed improvement. Not C.”

That's a significant signal, and I think it points toward where the field needs to invest its energy next.

What would you add to SPACE today?

The question the audience asked that will stay with me longest was a simple one: if you were writing SPACE now, what would you add?

The five dimensions have held up well. The panel was in agreement on that. But five years of AI acceleration, remote and hybrid work, and a rapidly shifting sense of what software development even means has surfaced things the original framework didn’t fully anticipate. While we might not add new dimensions, if we were updating it today, we would add focus for:

• Trust: This was the most consistent answer across the panel, acting as a foundational bedrock for satisfaction and performance.

• Cognitive and Intent Debt: Based on Peggy’s recent work, are we losing overall understanding of codebases as AI writes more of them?

• Deskilling: The worry that relying heavily on automated tools will cause core engineering capabilities to atrophy over time.

• AI Addiction: Within the wellbeing dimension, tracking addiction-like behaviors with generative AI tools.

Ultimately, as Chandra mentioned, the strength of SPACE is that organizations can dial up or down different dimensions to meet rising needs. You don’t need to change the structure; you just need to rebalance the rubric.

"Things like well-being are very, very, very important. So I think reducing focus a little bit on the activity side... that doesn't fundamentally change what SPACE is. You can just use SPACE but rebalance the rubric."

We acknowledge that the framework was never designed to be a perfect model. But, as Peggy put it, that doesn’t mean it isn’t valuable.

"Some models are wrong, some are useful. It was supposed to change the conversation. It was supposed to make people think about the different aspects of productivity. And I think it did that."

That feels right to me. The framework was designed to change the conversation. I think it did. The work now is to keep refining what we measure within that space, with the same care we brought to defining it in the first place.

Final thoughts

It was a remarkable day for me. I’m grateful to UC Irvine for hosting it, to my co-authors for showing up, and to everyone in that room for asking the hard questions. If the industry continues to ask them with this much rigor and honesty, I think the next five years will be even more interesting than the last.

This week’s featured DevProd job openings. See more open roles here.

• Ashby is hiring an Staff Platform Engineer | Remote

• BambooHR is hiring a VP of Engineering | Utah (Hybrid)

• Cashea is hiring an Infrastructure & Developer Productivity Platform Engineering Manager | Remote

• Figma is hiring a Staff Software Engineer, Developer Experience | Remote; US

• Morgan Stanely is hiring an AI Platform Engineer - Vice President | New York

That’s it for this week. Thanks for reading.

Share

Abi Noda is joined live at DX Annual by three engineering leaders shaping AI adoption at scale: Tim Bozarth, Corporate Vice President in Microsoft’s CoreAI division; Nancy Wang, CTO of 1Password; and Taroon Mandhana, CTO of AI and Teamwork at Atlassian. Together, they discuss how AI is changing engineering organizations, from team structures and planning cycles to hiring, governance, and measurement.

The panel explores how the profile of a great engineer is evolving, why smaller cross-functional teams are becoming more effective, and what happens when product managers, designers, and customer support teams start contributing code. They also share why they are encouraging AI adoption through enablement, training, and local champions rather than mandates, and how AI is shifting more of the software development lifecycle toward planning and validation.

Finally, they discuss where human judgment remains essential, how to measure adoption and manage token usage, and how to connect AI investments to business outcomes while preserving room for experimentation and learning.

Some takeaways:

Rethink team structures for faster learning

• Smaller teams are becoming more effective for zero-to-one work. AI reduces the cost of implementation, making alignment and rapid iteration the primary bottlenecks when searching for product-market fit.

• Planning cycles are getting shorter. Instead of locking in 12- to 18-month roadmaps, teams are shifting toward quarterly planning to adapt more quickly to changing technology and market conditions.

• Large-scale org changes can wait. Several panelists emphasized that experimentation and learning should come before major structural redesigns.

The best engineers think like makers

• A maker’s mindset matters more than mastery of a specific tool. The most effective engineers stay focused on outcomes and use whatever tools help them build valuable products.

• Product and engineering skills are converging. Strong engineers increasingly combine technical depth with product judgment, customer empathy, and design sensibility.

• Agency is becoming a defining trait. Engineers who can work across functions, navigate ambiguity, and drive decisions are gaining leverage as AI handles more of the implementation work.

Expect more people to participate in software creation

• Prototypes are replacing long-form requirements documents. Interactive demos often lead to faster and more productive conversations than detailed specifications.

• Product managers, designers, and customer support teams are starting to write code. AI is lowering the barrier for non-engineers to contribute directly to the software development lifecycle.

• Quality systems become more important as more people contribute code. Robust tests, review processes, and deployment safeguards are essential when more people can generate production code.

Drive adoption through enablement, not mandates

• Outcomes matter more than activity. The goal is not maximizing AI usage for its own sake, but improving speed, ease, and product quality.

• Local champions accelerate adoption. Teams learn fastest when respected peers demonstrate practical ways to use AI in real workflows.

• Activity metrics are diagnostic, not the objective. Low usage can signal where teams need more training, support, or better tools.

The AI-native SDLC shifts work toward planning and validation

• Plan and validate are becoming the highest-leverage activities. As AI accelerates code generation, more human effort shifts toward defining what to build and evaluating whether it meets expectations.

• Operations and incident response are ripe for automation. Engineering teams are beginning to use AI to triage alerts, investigate incidents, write postmortems, and reduce the time spent on routine operational work.

• Human judgment remains essential. Leaders were unanimous that critical decisions around quality, security, and accountability still require people in the loop.

Measure outcomes, costs, and learning

• Token usage is becoming the new cloud bill. Engineering leaders are applying FinOps-style discipline to monitor and forecast AI spending.

• North Star metrics provide a clearer signal. Examples include idea-to-value, innovation time, and product quality.

• Experimentation deserves budget. Some AI usage will not generate immediate ROI, but it can create learning that compounds into long-term competitive advantage.

In this episode, we cover:

(00:00) Intro

(01:08) Introducing the panelists

(02:16) AI’s impact on engineering team structures and planning cycles

(05:00) How the role of the engineer is changing and what makes a great engineer

(10:11) The opportunities and challenges of non-engineers writing code

(15:26) Encouraging AI adoption without mandating it

(21:25) What an AI-native SDLC looks like and why human judgment still matters

(30:56) Measuring AI adoption, token usage, and ROI

(37:06) How to tie AI investments to business outcomes

Where to find Nancy Wang:

• LinkedIn: https://www.linkedin.com/in/wangnancy

Where to find Taroon Mandhana:

• LinkedIn: https://www.linkedin.com/in/taroonm

Where to find Tim Bozarth:

• LinkedIn: https://www.linkedin.com/in/tbozarth

Where to find Abi Noda:

• LinkedIn: https://www.linkedin.com/in/abinoda

Referenced:

• DX Core 4 Productivity Framework

• Microsoft

• 1Password

• Atlassian

• Jira

• Confluence

• Loom

• Rovo

• Amazon Operating Cadence - Working Backwards

Jennifer St Pierre is Senior Vice President of Developer Experience and Transformation at Dell Technologies, where she leads the strategy for how Dell’s Infrastructure Solutions Group builds, operates, and evolves software.

In this session from DX Annual, Jen argues that the biggest challenge in adopting agentic AI is not the technology itself, but the people transition behind it. Drawing on lessons from earlier shifts like Agile, DevOps, and cloud adoption, she explains why organizations that treat AI as a simple tooling rollout may get compliance, but not commitment.

Jen outlines five leadership imperatives for navigating the transition: building a shared understanding of why change is happening, defining a clear future state, clarifying how roles will evolve, creating psychological safety for experimentation, and aligning metrics and organizational structures with new ways of working. Throughout the talk, she emphasizes that while AI may generate code, humans remain responsible for direction, judgment, and meaning.

Some takeaways:

Treat AI adoption as a people transformation

• Technology transitions are really people transitions. The hardest part of adopting agentic AI is not the tooling itself, but helping engineers understand how their roles, workflows, and career paths will evolve.

• Compliance is not the same as commitment. Organizations that treat AI as a tooling rollout may achieve adoption metrics, but they will struggle to build the trust and engagement needed for lasting change.

• Every major platform shift follows a familiar pattern. New technologies create excitement, skepticism, fear, and eventually productivity gains that become the new normal.

Build a shared understanding of why AI adoption matters

• Start with an honest explanation of why the change is happening. If developers do not understand the business rationale, they are likely to assume the initiative is primarily about cost reduction.

• Shared understanding does not require universal agreement. It means everyone is working from the same candid view of market pressures, strategic goals, and organizational intent.

• Framing shapes emotional response. Positioning AI as a way to help engineers focus on more strategic work creates a very different reaction than simply saying it will increase productivity.

Define a clear future state

• A vague vision creates fear. When people cannot picture what their work will look like in 12 to 18 months, they tend to imagine replacement, stagnation, or obsolescence.

• Role clarity is essential. Teams need to understand what skills will matter, how performance will be measured, and which responsibilities will increase or diminish.

• Specificity beats slogans. Concrete expectations about how AI will be used help people see where they fit in the new model.

Create psychological safety for experimentation

• Teams need permission to make mistakes. AI adoption requires experimentation, and experimentation inevitably involves missteps and imperfect results.

• Psychological safety helps teams surface problems earlier. When engineers feel safe speaking up, leaders get better information and can address issues before they escalate.

• Silence is expensive. Leaders who discourage candor risk making decisions based on filtered or incomplete information.

Align metrics and organizational structures

• Old metrics can reinforce old behaviors. Measuring lines of code or heroic firefighting may encourage exactly the habits AI should help organizations move beyond.

• Metrics and structure must evolve together. Governance, incentives, funding, and performance systems need to support the behaviors leaders want to see.

• Transformation should survive without constant reminders. If the desired behaviors disappear as soon as leaders stop talking about them, the change has not yet become embedded.

Lead the transformation intentionally

• Leaders must model the change themselves. Using AI tools, sharing lessons learned, and being transparent about failures builds credibility.

• Career paths must be made explicit. Engineers want to know how they can continue to grow and whether deep technical expertise will remain valuable.

• AI may generate code, but humans generate direction. Judgment, context, and meaning remain the most valuable contributions people bring to software development.

In this episode, we cover:

(00:00) Intro

(00:13) Why every major technology shift is ultimately a people transition

(05:00) AI-generated code and the evolving role of software engineers

(07:43) The importance of developing a shared understanding

(12:00) Defining a clear future state and how engineering roles will evolve

(19:12) How psychological safety enables experimentation and honest feedback

(22:41) Why metrics and organizational structure must evolve for the age of AI

(25:40) Why leaders must drive AI transformation intentionally

Where to find Jennifer St Pierre:

• LinkedIn: https://www.linkedin.com/in/jennifer-st-pierre-4935a81

Referenced:

• Measuring developer productivity with the DX Core 4

• Understand team effectiveness