Subscribe now

🗓 Join our live research panel on September 24 as we introduce CAFE(S), a framework for improving AI agent effectiveness through better context. Register here.

Some of the most publicized AI failures of the last few years came from the legal profession. Lawyers have now been sanctioned, more than once, for filing briefs containing citations to cases that did not exist. It is tempting to blame these hallucinations entirely on model failure, but that offers an incomplete picture.

A brief was drafted, reviewed, and filed. Somewhere along that chain, an unsupported claim produced by a probabilistic language model became something a human was willing to represent as fact. The hallucination was the visible failure, but the root cause was the information environment surrounding the model.

A better model might hallucinate less often, but it won’t eliminate the need for the information and safeguards required to separate the plausible from the true.

That distinction is critical, and it’s showing up everywhere software gets built.

Picking up where the code left off

Recently, I wrote about what AI is doing to software quality, using a definition of quality that some colleagues and I developed back in 2020. The conclusion I landed on was that code is becoming a late place to look. More of what determines quality is settled before any code exists.

I ended that piece with five sentences that have stuck with me since:

Reuse requires knowing what already exists in the codebase. Review requires knowing what a change was meant to accomplish. Evolvability requires knowing why the system is structured the way it is. Handoff requires knowing what the previous person was thinking. Serving user needs requires knowing what problem the software was supposed to solve in the first place.

None of that knowledge necessarily lives in the code. It lives in tickets, design docs, runbooks, architecture decision records, and Slack threads. A fair amount of it lives nowhere at all except in somebody’s head.

When we hand a task to an agent, we hand over some representation of that knowledge with it. That representation is the context. And when context fails, it does not fail randomly, the same shapes keep recurring.

We have a vocabulary for this, just not for context

Software engineers have a rich language for recurring problems in code. Say code smell and everyone knows you mean something that isn’t a bug, but reliably signals trouble ahead. Long method. Feature envy. Shotgun surgery. Those names are useful precisely because they convert a vague unease into something you can point at in a review.

We have almost nothing equivalent for context when working with agents. When an agent produces something wrong, the diagnosis usually stops at “the model got confused,” which explains nothing and suggests no action.

So let me offer a starting vocabulary. Call them context smells. Here are six I keep running into. None of them are model limitations, and all of them survive a model upgrade.

Confident hallucination. The model asserts something untrue with exactly the fluency it uses for things that are true. We treat this as a model defect, and partly it is. But a model can only separate supported claims from unsupported ones if something in its context gives it a way to do that. The sanctioned lawyers weren’t undone by fabrication alone. They were undone by a workflow in which nothing downstream of the fabrication was designed to catch it.

Specification ambiguity. This is the one I find most underappreciated. Faced with an obviously underdetermined request, an agent will often stop and ask for clarification. Faced with a semi-ambiguous request, it does something considerably more dangerous. It picks an interpretation and proceeds. If that first read is wrong, every subsequent step compounds it, and the work looks confident the entire way down. Nothing was missing from the context. It simply admitted more than one reasonable reading, and nobody noticed which one got chosen.

Stale guidance. Context that accurately describes a system that no longer exists. The runbook that was correct two migrations ago. The architecture doc written before the service was split. A human reading these usually senses the mismatch and goes looking for why. An agent has no such reflex, and treats them as current.

Lost in the middle. More context is not reliably better. Model performance degrades as input grows, and it degrades unevenly, with material buried in the middle of a long context getting overlooked in favor of what sits at the edges. This one is genuinely counterintuitive, because nothing is missing. The agent has everything it needs and still fails, because the signal is competing with too much else for attention. Agents, like people, can get distracted.

Lost in the details. Context that supplies plenty of implementation detail and none of the reasoning behind it. The agent optimizes local mechanics beautifully while losing track of what the task was actually for. Margaret-Anne Storey calls the underlying condition intent debt, the gradual disappearance of recorded rationale from the artifacts a system leaves behind. It’s the difference between knowing that a timeout is set to 300 milliseconds and knowing whether that number came from a benchmark, a support incident, or somebody’s afternoon guess.

Weekend runaway. An agent given a task with no definition of success may never discover that it’s finished. So it retries, widens scope, and keeps exploring an objective that may not even be achievable. The result is consumed time, consumed tokens, and occasionally a genuinely startling bill on Monday morning. This may be the least intellectually interesting failure on the list, but maybe the one most likely to get a leader’s attention.

None of these are new

Read that list again and notice how little of it is actually about AI.

Ambiguous specifications have been the central concern of requirements engineering for decades. Documentation that drifts out of sync with the system it describes is one of the oldest complaints in software maintenance. Signal drowning in noise is the founding problem of information retrieval. Rationale vanishing from the artifacts a team leaves behind is what knowledge management has always been about.

None of these challenges are new. Bad context has always been a normal fact-of-life for software organizations. What kept it from being catastrophic is that humans quietly repaired it, constantly, without anyone noticing.

A new engineer handed an ambiguous ticket walks over and asks what it means. Handed a runbook that doesn’t match production, they notice the discrepancy and go find someone who knows. Handed two documents that contradict each other, they work out which one smells current, or they ask in a channel and get an answer in four minutes. Every organization I’ve studied runs on documentation that would fail inspection, and it works anyway, because a layer of human judgment sits between the bad context and the actual work.

Agents don’t reliably have that same repair loop, especially as we race to make them more autonomous. Handed the same ambiguous ticket, the same stale runbook, the same contradictory pair of documents, they proceed. Confidently, quickly, and at a scale that used to be impossible.

I don’t think AI created a context problem. I think it withdrew the error correction that had been hiding one. The information environment we’ve been tolerating was always this bad. If that’s right, it may also help explain one of DORA’s more interesting findings about AI adoption. Their 2025 research describes AI as an amplifier. It tends to magnify the strengths and weaknesses of the systems already surrounding agents. That’s usually framed as an organizational capability problem, but I wonder if part of it is an information problem. Teams with clearer specifications, better documentation, and more accessible organizational knowledge may simply have less damage for the human buffer to absorb in the first place.

What you can do

Most of what I’ve described sounds like it needs instrumentation we don’t have yet. It doesn’t (at least not to start).

Before handing a task to an agent, ask whether a competent new colleague could complete it with what you’re making available. Not a senior engineer who already knows your system. A capable person with no additional access to what’s in your head.

That test is deliberately low-tech, and it catches a surprising amount. The specification that admits two readings. The runbook describing last year’s architecture. The fifty-page dump with the one important constraint on page thirty-one. The task with no stated definition of done.

A competent new colleague would probably stop at each of those. Which interpretation did you mean? Is this runbook still current? Which of these fifty pages actually matters? What does done look like? Those questions are the human repair loop in action. The more autonomous we make agents, the less we should assume that loop will save bad context.

Naming these patterns is a start, but only a start. Context smells give us a vocabulary for recognizing trouble. They don’t give us a systematic way to evaluate the quality of context before an agent starts working. That second problem is considerably harder, and it’s where I’ve been spending most of my time lately. Over the coming weeks, I’ll be publishing a framework describing the dimensions of context quality, followed by an approach to measuring those dimensions.

For now, though, the question I’d put to any team frustrated with their agents isn’t which model they should be using.

It’s whether they’d have given the same task to a new hire and expected them to get it right.

That’s it for this week. Thanks for reading.

-Brian

Share

In this episode of Engineering Enablement, I sit down with Robert Lucero, Chief Architect at Okta, to discuss how agentic AI is changing identity and access management. We explore how organizations should define agent identities, why authentication and authorization remain foundational, and how sandboxing, fine-grained permissions, and just-in-time access let agents operate autonomously without creating unacceptable risk.

Robert shares what Okta has learned from driving AI adoption among security-minded engineers, what makes a repository ready for AI agents, and why strong testing, CI, and review processes matter even more as AI generates more code. We also discuss AI’s role in software validation, the need for human judgment, and whether AI will ultimately give the advantage to security teams or attackers.

Some takeaways:

Identity becomes critical when AI agents move beyond content generation

• AI agents need access to documents, source code, CI systems, ticketing tools, and other resources to perform useful work. Identity systems determine which resources they can access and what actions they can take.

• Non-determinism makes tightly scoped access especially important. Organizations need dynamic policies, fine-grained permissions, and visibility when agents attempt to operate beyond their boundaries.

The definition of an agent identity is still evolving

• Organizations have not yet settled on whether every agent or ephemeral workload needs its own identity. An agent might instead operate under an orchestrator, a service account, or the identity of a human owner.

• The fundamentals of identity remain the same. Agents still need to be authenticated before organizations can apply authorization policies, governance, and access controls.

Treat AI agents like new hires

• Start agents in a sandbox with tightly scoped access and clearly defined tasks. Like new employees, agents may seek out tools or information when their environment is missing something they need.

• Rely on technical controls rather than assuming an agent has earned trust. Sandboxing, guardrails, and just-in-time access can limit the blast radius of unpredictable behavior.

Identity systems are a control plane, not an orchestration layer

• The identity layer governs what agents can access, trigger, and call. The workflows connecting coding, review, testing, and deployment agents will likely be orchestrated elsewhere.

• Identity systems can provide essential visibility into agent relationships. A relational record of agent-to-agent and agent-to-resource access makes it possible to investigate why an agent was able to take a particular action.

Successful AI adoption depends on useful outcomes

• Okta’s security-minded engineers were initially skeptical because early tools did not reliably help them do their jobs. Adoption increased as coding models improved and became more effective in real engineering workflows.

• The value of AI extends far beyond writing code. Engineers can use it to summarize documentation, review product specs, investigate production issues, and respond to customer requests more efficiently.

AI readiness starts with engineering fundamentals

• Repositories need clear instructions, discoverable dependencies, compatible harnesses, and defined development practices before agents can work effectively.

• Strong testing, CI, linting, and code review provide the safety nets AI-generated changes require. Improving a repository for AI agents often makes it easier for human developers to work in as well.

Passing tests does not prove that AI-generated code is correct

• An agent can write tests and produce code that passes them without understanding the intended business outcome. Reliable validation depends on giving agents access to the right requirements and context.

• Human judgment remains essential for determining whether software actually solves the problem. Developers, designers, test engineers, and product owners must still evaluate whether the result meets user and business needs.

In this episode, we cover:

(00:00) Intro

(02:16) Meet Robert Lucero

(02:56) Why identity becomes critical as AI agents access more systems

(06:07) Governing automated access for non-deterministic agents

(08:01) Defining and managing agent identities

(10:32) Authentication and authorization for AI agents

(13:33) Determining how much autonomy to give AI agents

(19:44) Identity systems as the control plane for AI agents

(22:40) Driving AI adoption among security-minded engineers

(25:48) Why AI’s impact on engineering extends beyond coding

(28:44) The three components of AI readiness for repositories

(30:55) AI’s role in software testing and validation

(34:26) Whether AI gives defenders or attackers the advantage

(36:01) Where to find the best food in the country

Where to find Robert Lucero:

• LinkedIn: https://www.linkedin.com/in/rlucero

Where to find Brian Houck:

• LinkedIn: https://www.linkedin.com/in/brianhouck

Referenced:

• State of AI Impact in Engineering Q2 Report 2026

• Okta

• The Hugging Face incident and the road ahead | OpenAI

• GitHub Copilot · Your AI pair programmer

• Claude Code by Anthropic | AI Coding Agent, Terminal, IDE

• Eight Myths on Software Engineering and GenAI | Queue

• Adriana Corona

Subscribe now

🗓 Join DX’s live research panel on September 24 as Brian Houck introduces CAFE(S), a framework for improving AI agent effectiveness through better context. Register here.

As reported in our Q2 AI Impact Report, time saved thanks to AI continues to increase, with developers reporting saving 6.1 hours per week in Q2 2026 compared to 3 hours in Q3 2025. However, it raises the question for engineering leaders:

What are developers doing with that time?

While AI is frequently positioned as a way to reclaim time for innovation, our data reveals a disconnect between AI-driven output and actual shifts in work composition. Organizations are investing in AI with the hope that the time AI saves will directly translate into faster roadmap execution and more innovation. DX tracks this using innovation ratio, the percent of engineering effort dedicated to new capabilities versus maintenance.

We used multivariate regression on a sample of 500+ DX customers to test the innovation ratio against 15 workflow metrics, including AI output, shipping velocity, and operational drag. We analyzed AI output according to how much AI-authored code, agent-delivered work, and PR throughput teams produce. Our analysis shows that while AI output has a powerful link to AI-driven time savings, it does not reliably translate into a higher innovation ratio.

Information-seeking may be the hidden barrier to innovation

While AI tools are delivering measurable speed, our data suggests that reclaiming capacity for new features requires clearing organizational friction, rather than assuming AI will lead to innovation.

1. AI is a lever for speed. The data confirms that AI does what it promises. AI output accounts for 63% of the variation in developer time savings, suggesting that developers with higher AI output save more time.

2. Speed doesn’t convert to innovation. Despite increased time savings, AI output suggests a weak link to innovation (p < 0.01, β=0.16). No single workflow metric emerged as a reliable predictor of innovation, indicating that dedicating more time for new features and capabilities is not a byproduct of moving faster.

3. The impact of operational friction. The clearest signal tied to lower innovation ratio is information-seeking, referring to time lost hunting for context, documentation, or answers (p < 0.01, β=-0.19). It is not a large effect and needs additional validation, but this developer-reported friction represents a significant hypothesis for engineering leaders. It suggests that reducing operational drag may be a more effective path to increasing innovation than AI (or AI alone).

Key takeaways for engineering leaders

1. Decouple AI strategy from innovation goals. AI delivers speed, but there is no evidence it automatically shifts teams toward new-feature work. Treat innovation as a separate objective requiring its own strategy

2. Test the operational friction hypothesis. Focus on reducing time lost to information-seeking. While not yet a proven driver of innovation, it is the most promising lead for improving developer conditions.

3. Use this as a diagnostic. Because our model accounted for only 13% of the differences in innovation ratio, much of what drives new-feature work remains unique to each company. The most valuable next step is running this analysis against your own team’s data; your strongest predictors might differ from the aggregate.

   ---

   That’s it for this week. Thanks for reading.
   -Grace

   Share

Subscribe now

🗓 Join our live research panel on September 24 as we introduce CAFE(S), a framework for improving AI agent effectiveness through better context. Register here.

Two numbers from the 2026 Q2 State of AI Impact report have raised a lot of questions for me about the impact of AI on “quality”.

Across a sample of more than 500 companies, code maintainability improved 3.8% between Q1 and Q2. Over the same period change confidence fell 6.1%. Maintainability asks how easy the codebase is to understand and modify. Change confidence asks whether developers trust that their changes won’t break things.

Those two measures have historically moved together. Code you understand is code you feel safe changing. Now they are moving in opposite directions.

Layering in other findings from the report makes the picture even less clear. Documentation quality, code maintainability, and production debugging had some of the biggest improvements in the quarter—all signals we might expect to provide tailwinds to quality. At the same time, review turnaround declined, incremental delivery fell sharply, and pull requests continued to get larger—all signals that we might expect to provide headwinds.

Perhaps quality isn’t simply going up or down. Perhaps different parts of it are moving in different directions.

To help me frame the problem, I went back to a definition of quality that a couple of colleagues and I developed in 2020, several years before generative AI complicated the picture. I wanted to know whether our definition could help make sense of what we’re seeing now.

Turns out, the places where the data appears most contradictory make considerably more sense once we stop treating “quality” as one thing.

What developers mean by quality, before AI could complicate it

In January 2020, Margaret-Anne Storey, Tom Zimmermann and I surveyed 131 developers and 34 engineering managers at Microsoft about how they define productivity and quality, in their own words. (Paper summary here.) We coded the open-text answers inductively, and five dimensions of quality fell out. We called these dimensions TRUCE, which are defined below:

Six years later, TRUCE gives us a useful lens for interpreting what otherwise looks like contradictory evidence about AI and software quality.

Robustness is unsettled, but we can see it

Start with the dimension we instrument best.

Change failure rate volatility widened in Q2, with many organizations now swinging plus or minus three percentage points against a 4% industry benchmark. The distribution of failed deployment recovery time is leaning longer, even though developers’ experience with production debugging improved slightly.

In other words, teams report getting better at diagnosing production problems while recovery outcomes are becoming more variable.

It’s important not to turn correlation into causation here. This data doesn’t tell us that AI created volatility, or even that AI amplified it. What we can say is that robustness is becoming less predictable during the same period in which AI-authored code has accelerated dramatically.

Robustness itself isn’t mysterious; we know what we’re trying to observe. Many metrics exist to try and measure robustness: failure rates, recovery times, incidents, defects, test outcomes, and more. We can debate exactly which measures are best, but when robustness changes, we have a reasonably mature vocabulary for describing what happened.

Hold onto that, because robustness turns out to be the exception. The dimensions that follow are roughly ordered by how well we can measure them, and as that measurement thins out, something consistent happens: the dimension stops behaving like one thing. Whether those two facts are related is a question I want to come back to.

Evolvability is splitting into two things

TRUCE derived evolvability from developers defining quality as “Easy to test, easy to change.” It’s natural to assume that code you understand is code you can feel safe changing. Recent academic research suggests something different may be happening when we zoom out from individual code changes to the systems they become part of.

In More Code, Less Reuse, Huang and colleagues compared pull requests generated by AI agents with those written by human developers. They found that AI agents were more likely to miss opportunities to reuse existing code, introducing greater redundancy. Interestingly, reviewers still tended to respond neutrally or positively to the AI-generated contributions. The authors describe this as a disconnect between the “surface-level plausibility” of the code and the technical debt accumulating underneath it.

Another recent study makes the distinction even clearer. Needle in the Repo evaluated 23 AI coding configurations on repository-level tasks designed to test maintainability, not just whether the resulting code worked. Across those configurations, 13.3% of outcomes passed every functional test while still failing the study’s structural maintainability criteria. Performance was especially poor on architectural concerns such as dependency control and responsibility decomposition.

Both findings point toward the same distinction. A piece of code can be easy to understand. It can work. Reviewers can even think it looks good. But the system it belongs to can still become harder to evolve.

The emerging pattern is that locally good code doesn’t guarantee a healthy system. Generated code may be perfectly understandable in isolation while duplicating something that already exists, introducing an unnecessary dependency, or putting functionality in a place that makes the broader system harder to change.

And that brings us back to the numbers I started with. Maintainability improved 3.8%. Change confidence fell 6.1%. Perhaps those measures aren’t disagreeing after all. They may simply be looking at different parts of evolvability.

Collaboration is splitting too

One of my favorite definitions of quality from the TRUCE paper came from a developer describing collaboration, one of the least frequently mentioned dimensions:

“Quality is code or solutions that solve a problem and don’t need undue maintenance or lengthy handoff. If I get hit by a bus and the company can still easily use the code I’ve written, I’ve made a quality solution.”

We wrote at the time that enabling collaboration or reuse by others “is often invisible and hard to measure,” which likely explains why it was a less frequently mentioned dimension, “and yet it is very important.” I did not expect that sentence to age the way it has.

This is where the data gets strange. Documentation quality was among the clearest wins in the quarter. That matters for collaboration. Documentation supports handoff, comprehension, and the ability of someone else to pick up your work.

At the same time, review turnaround declined. Incremental delivery, which asks whether developers work in small, testable changes, showed the sharpest decrease of any driver. Median PR size nearly doubled over the year, increasing the amount of unfamiliar code developers need to understand at once.

So we have better documentation attached to larger changes that are harder to review, written increasingly by something that isn’t on the team. Documentation quality and shared understanding aren’t the same thing. AI can document what the code does without transferring the reasoning that led to it. It can make handoff easier while making shared ownership harder.

That’s what makes the bus-factor definition feel so different to me now. AI may be improving the explicit artifacts that support handoff while making the implicit knowledge surrounding the work more difficult to maintain.

The collaboration dimension that only 17% of developers thought to mention in 2020 suddenly looks much more important. And we said at the time that we couldn’t see it very well.

Timeliness is not the same thing as speed

Only 7% of developers named timeliness as a dimension of quality in 2020, the lowest of the five dimensions. It is now the site of perhaps the cleanest decoupling in the data.

Median normalized PR Throughput rose 37% over four quarters. Deployment frequency is up across nearly every segment. However, over the same period, developers’ perceived rate of delivery was flat.

That looks contradictory only if we assume delivery speed and timeliness are the same thing. Timeliness was not intended to be a pipeline property. It was about whether what was needed arrived when it was needed.

AI can increase the amount of code flowing through a system without changing how quickly an engineer feels able to get the thing that matters into a customer’s hands. More PRs can run into slower reviews. Faster implementation can expose bottlenecks in testing, deployment, requirements, or coordination. Additional output can simply create additional work downstream.

The pipeline got measurably faster. The felt experience of delivering on time did not.

Once again, TRUCE makes the apparent contradiction easier to understand. Our delivery-speed metrics and our experience of timeliness are measuring related things, but they aren’t measuring the same thing.

User needs is the dimension we still can’t see

It would be a stretch to say much of anything about AI’s impact on this dimension from the available data. That is itself a finding.

The closest available proxy in the Q2 data is perceived software quality, and it says something counterintuitive. Traditional industries and financial services, the slowest-moving segments on throughput, report the highest perceived quality.

That is self-reported and may say as much about expectations as about software, so I’d stop short of any claim stronger than this: speed and perceived quality are not moving together.

I also went looking for external research on whether the people who use software are experiencing it differently as AI authorship climbs. I came up mostly empty.

There is a great deal on defect rates, vulnerability counts, code quality, and QA workload. We have remarkably detailed telemetry about how much code AI writes, how quickly pull requests move, how frequently teams deploy, and how often changes fail. We know considerably less about whether the resulting software is actually solving people’s problems better.

That’s a striking asymmetry.

One developer in the study told us “Quality is doing what the customer wants.” Four years into this shift, we can measure changes in the machinery of software production with extraordinary precision while remaining largely blind to what may be the most important dimension of quality: did the thing we built serve the people using it?

Quality was never just about the code

Put the five dimensions back together and the story isn’t that AI is improving quality. It also isn’t that AI is destroying it. The story is that “quality” was never a scalar.

Seen this way, the maintainability/change-confidence paradox that started this piece doesn’t look like a paradox anymore. It looks like exactly what we should expect when a multidimensional construct is being pushed in different directions at once.

Six years on, I don’t think AI makes TRUCE less useful. If anything, it makes the framework more useful by reminding us what we’re actually trying to preserve. We still want software that is robust, evolvable, collaborative, timely, and serves the people using it.

What may be changing is where we need to look to understand whether we’ll get those things.

Reuse requires knowing what already exists in the codebase. Review requires knowing what a change was meant to accomplish. Evolvability requires knowing why the system is structured the way it is. Handoff requires knowing what the previous person was thinking. Serving user needs requires knowing what problem the software was supposed to solve in the first place.

None of that knowledge necessarily lives in the code.

Margaret-Anne Storey, my TRUCE co-author, has arrived somewhere similar from a different direction. Her triple debt model separates technical debt, which lives in code, from cognitive debt, which lives in a team’s shared understanding, and intent debt, which lives in the artifacts recording what a system is for.

Her argument is that AI may reduce one kind of debt while accelerating the others, and that code is no longer necessarily the binding constraint on software health.

Her model maps to the findings we just walked through. Technical debt is visible in the evolvability problem. Cognitive debt shows up in the collaboration split. And intent debt helps explain why it is so difficult to know whether a system is meeting real user needs, the dimension we can barely see at all. Hers is a conceptual model rather than an empirical one, so I’d call this convergence rather than confirmation. But two frameworks built for different purposes landed on the same three places.

There’s a small irony worth noting. When Mansi Khemka and I asked developers in 2023 what they most wanted AI help with, authoring and refactoring code ranked fifth, at 25%. Documentation ranked third at 37%.

Documentation is now one of the clearest quality wins in the data. Refactoring is one of the structural signals moving most sharply in the other direction.

I wouldn’t read too much into a single comparison. But it is interesting to me that the dimension holding up best is the one where the tooling matched what developers said they wanted.

Which brings me back to the artifact itself. We have spent decades learning to evaluate code. It is worth asking whether code is still the thing that carries the most information about whether these five dimensions will hold.

Code is not the only artifact worth evaluating

For most of software’s history, code was the currency of software engineering. A developer understood the requirement, carried some model of the architecture in their head, knew what neighboring systems did, made a series of design decisions, and eventually expressed those decisions as code. That made code an unusually information-rich artifact, and it is why evaluating the code was a critical way to evaluate the work.

That is changing. The Q2 report found that 52.7% of code is AI-authored, up from 24% in Q4 2025 and 34% in Q1 2026. Increasingly, a developer doesn’t translate everything they know directly into code. They express an intent, provide some representation of the relevant context, and ask a machine to produce the implementation.

Intent and context aren’t quite the same thing. Intent is what a person ultimately wants to accomplish. That intent must then be translated into context, which is the information that represents the task, its constraints, and the surrounding knowledge. More of the knowledge a developer once carried implicitly now has to be externalized before the work can begin.

TRUCE still gives us language for evaluating the artifact at the end of that chain, and AI doesn’t change that. What concerns me is that the end of the chain is becoming a late place to look. As development shifts from writing implementations toward expressing intent, more of the conditions that determine all five dimensions are set before any code exists.

We know how to ask whether the artifact is good. We’re only beginning to develop the vocabulary for asking whether the context that produced it was good enough to begin with.

That’s it for this week. Thanks for reading.

-Brian

Share

In this episode of Engineering Enablement, I sit down with Max Kanat-Alexander, Executive Distinguished Engineer at Capital One, to discuss how AI is changing software development and why strong engineering fundamentals matter more than ever.

We explore how the skills engineers need are expanding, why AI amplifies both the strengths and weaknesses of the development lifecycle, and how teams should think differently about code review, quality, and testing. Max also shares how leaders can assess whether their organizations are ready for more advanced AI workflows and why we need more research into how junior engineers become senior engineers.

Some takeaways:

AI amplifies the engineering system you already have

• Strong engineering fundamentals matter even more with AI. AI amplifies everything that is good or bad about a software development lifecycle, so teams with strong tooling, CI, testing, and workflows are positioned to see the greatest gains.

• Faster coding doesn’t solve problems elsewhere in the development lifecycle. If engineers spend significant time navigating broken processes, waiting for reviews, or dealing with poor tooling, accelerating code generation only addresses a small part of their work.

The skills engineers need are changing

• Clearly expressing requirements and providing feedback are becoming increasingly important engineering skills. These are capabilities senior engineers have traditionally developed through years of experience, but they become essential much earlier when working effectively with coding agents.

• Deep technical knowledge still helps engineers recognize when something is wrong. Experience with languages, systems, and software design develops the pattern recognition needed to spot a flawed approach without reviewing hundreds of lines of AI-generated code.

• Domain understanding and problem definition are becoming more valuable. As AI gets better at reproducing knowledge found in documentation, engineers can differentiate themselves through understanding what needs to be built and why.

Code review is fundamentally about maintainability

• Human code review shouldn’t be the primary way teams catch bugs. Tests, static analysis, and other automation are more effective and scalable ways to detect defects.

• Human judgment remains important for evaluating maintainability and design. Reviewers can assess whether code is understandable, structured appropriately, and aligned with the longer-term intent of a system.

• The right quality bar depends on the codebase. A disposable script and enterprise software expected to operate for a decade shouldn’t be subjected to identical standards because the consequences and costs of getting them wrong are very different.

Not every change needs the same review process

• Code review is an important tool for developing less experienced engineers. Max says reviewing someone’s code is the most effective way he has found to improve their core software engineering skills.

• Trusted engineers may not need every change reviewed on codebases they deeply understand. That trust should be specific to the combination of the person and the codebase, rather than becoming a blanket exemption from review.

• The risk of the individual change can also determine whether human review is necessary. Teams can evaluate confidence and potential consequences rather than requiring the same review process for every PR.

AI can create a vicious cycle of declining code quality

• AI struggles when it has to build on poorly structured, poorly tested code. When weak review allows its bad output back into the codebase, the code becomes even harder for AI to work with, creating a cycle that can eventually leave teams with systems they can’t understand or maintain.

• AI can also help teams reverse that cycle. Agents can write tests, improve testability, and are particularly effective at refactoring when engineers deliberately direct them toward improving the codebase.

• Refactoring should support forward progress on product goals. Rather than attempting to clean up an entire codebase, teams should improve the parts they need to touch for upcoming work and make the necessary developer experience improvements along the way.

AI readiness depends on how far left rigor extends

• Fixing an existing software development lifecycle is much harder than building a good one from the beginning. Organizations with weak engineering practices often have to start with production problems and work backward through incident management, observability, deployment, testing, and earlier stages of development.

• How far left rigor extends is a useful signal of AI readiness. Organizations without strong controls around production, code review, and testing may introduce significant risk by giving engineers tools that allow them to move much faster.

• Leaders should evaluate the organization they actually have rather than following the AI hype cycle. Max is bullish on AI adoption, but argues that organizations need the engineering foundations to support the additional velocity it creates.

The industry needs a better way to develop senior engineers

• We still know surprisingly little about what reliably turns junior engineers into senior engineers. Mentorship and apprenticeship are widely considered important, but the practices that actually produce experienced engineers have not been deeply studied.

• AI makes understanding that development path more urgent. Senior engineers rely on experience to recognize failure patterns, which raises questions about how newer engineers will acquire that judgment as AI takes on more of the implementation work.

• AI may ultimately increase the need for engineering expertise rather than reduce it. As hundreds of millions more people gain the ability to build software and the rate of change accelerates, Max believes the world will need more experienced engineers to manage that complexity.

In this episode, we cover:

(00:00) Intro

(01:54) Max’s role at Capital One

(02:52) Where to invest in engineering organizations

(06:36) The new entry-level engineering skills to pay attention to

(10:39) Why deepening your understanding still matters

(12:29) The bottlenecks around code review

(19:40) Why human code reviews still have value

(25:20) Why not all PRs need human review

(26:11) The vicious cycle of AI-driven development

(30:13) Using LLMs for refactoring

(33:58) AI readiness and why fixing engineering systems is so hard

(38:42) Why research is needed on creating good senior engineers

(41:00) Why AI will increase the need for engineers

Where to find Max Kanat-Alexander:

• LinkedIn: https://www.linkedin.com/in/mkanat

• X: https://x.com/mkanat

• Blog: https://www.codesimplicity.com

Where to find Brian Houck:

• LinkedIn: https://www.linkedin.com/in/brianhouck

Referenced:

• DX Core 4 Productivity Framework

• Capital One

• Code Review Guidelines at Google

• Code Simplicity by Max Kanat-Alexander

• How Microsoft sees engineering bottlenecks changing with AI

Subscribe now

An engineering leader recently asked me how to measure whether their team is predictable in their software delivery. While organizations collect many engineering metrics (cycle time, throughput, review latency, deployment frequency, etc.), there is rarely a single metric that directly measures predictability.

The central question is:

Can we use the metrics we already collect to quantify delivery predictability?

Higher delivery predictability enables more accurate release planning, resource allocation, dependency management, and stakeholder communication. Improving predictability is therefore not simply about increasing throughput; it is about increasing confidence in engineering plans.

Key observation

Delivery predictability is not another engineering metric; it is a statistical property of a delivery process.

Traditional metrics tell us how much work teams complete or how long work takes. Predictability tells us, given historical performance, how confidently can we forecast future delivery? Predictability is a measure of uncertainty. For example, a team that consistently delivers 20 work items each sprint is highly predictable. A team that alternates between delivering 5 and 35 work items may have the same average throughput, but is far less predictable. The distinction is not average performance. It is variation.

A note on two lenses. “Delivery predictability” can mean two related but distinct things. The first is flow stability: how consistent and low-variance a team’s delivery is over time. The second is commitment reliability: how closely actual delivery matches what was committed (a ratio of committed vs. completed work). This article focuses on flow stability, because it can be derived directly from the metrics most organizations already collect.

Example

Let’s walk through a practical example of two hypothetical teams’ count of completed work items, by sprint:

Both teams deliver exactly the same average throughput. The difference is that Team A is predictable while Team B is not. The remainder of this article describes several statistical techniques that quantify this difference.

What different statistical techniques tell us

The following statistical techniques each answer a different question about delivery predictability using the Team A and Team B example.

Standard deviation

Standard deviation measures how much a team’s delivery typically varies from its average. A low standard deviation means delivery is consistent from sprint to sprint, while a high standard deviation indicates greater volatility.

In our example, both teams average 20 completed work items per sprint. However, Team A’s delivery stays tightly clustered around that average, while Team B swings between 10 and 30. Team B therefore has a much larger standard deviation, despite identical average throughput.

Leadership takeaway: Standard deviation answers the question, “How much does this team’s delivery naturally fluctuate?” It is often the simplest way to quantify delivery consistency.

Coefficient of variation

Standard deviation alone can be misleading when comparing teams of different sizes. A team delivering 80 ± 5 work items is more variable in absolute terms than Team A (20 ± 1.3), but proportionally it is just as predictable.

The coefficient of variation normalizes the standard deviation by the team’s average throughput, allowing comparisons across teams regardless of scale.

Although Teams A and B have the same average in our example, this measure becomes valuable when comparing organizations with very different delivery volumes.

Leadership takeaway: Which teams are most predictable relative to the amount of work they deliver?

Percentiles and distribution shape

The sections above use throughput per sprint. Percentiles are most illuminating on a per-item measure such as cycle time, the number of days a work item takes to go from started to done, so this section shifts to that lens.

Averages describe the typical case, but predictability is usually broken by the atypical case. Percentiles, rather than the mean, show how often delivery runs long and by how much.

Consider two teams that both have a median (P50) cycle time of 4 days. If the first team’s 95th percentile (P95) is 7 days while the second team’s P95 is 21 days, they are not equally predictable. The first team’s slowest work is only modestly slower than its typical work. The second team’s slowest work takes three weeks, even though half its items finish in under four days.

The gap between P50 and P95 is, in effect, a predictability measure of its own. A tight gap means most work behaves like the typical case. A wide gap means the typical case tells you little about the work that actually puts a commitment at risk.

This is why a healthy median can still mask an unpredictable process: it is the tail, not the middle, that causes missed forecasts.

Leadership takeaway: When we commit to a date, are we planning against the typical case or the realistic worst case? Percentiles let us answer, “X% of work of this type finishes within N days,” a far more honest basis for a forecast than an average.

Statistical process control (control charts)

Control charts answer a different question. Rather than measuring overall variability, they determine whether a delivery process has fundamentally changed.

Suppose Team A suddenly delivered 9 work items in Sprint 4 instead of its usual ~20. While the average throughput might not change much immediately, a control chart would likely flag Sprint 4 as outside the team’s normal operating range.

This often indicates that something meaningful occurred, such as a production incident, staffing change, organizational restructuring, or dependency bottleneck.

Leadership takeaway: Did something change that warrants investigation, or is this simply normal variation?

Prediction intervals

Prediction intervals translate historical variability into forward-looking planning confidence. Rather than forecasting that Team A will complete exactly 20 work items next sprint, we estimate the range it is likely to land in.

Using a 95% prediction interval, Team A is forecast to deliver roughly 17 to 23 work items next sprint. Team B, with the identical average, is forecast at 0 to 40.

That second range deserves a moment. Team B’s forecast spans from delivering nothing to delivering twice its average. It is technically correct and completely useless for planning, which is precisely the point. Team B does not have a throughput problem. It has a forecasting problem, and no average will ever surface it. Notice that neither team’s average changed. Only the confidence around that average did.

The practical implication is that interval width, maybe more than the forecast itself, is the thing worth tracking over time. A team whose range narrows is becoming more predictable, even if its average never moves.

Leadership takeaway: How confidently can we forecast future delivery, and how wide is the range we should plan against?

Beyond the spreadsheet: machine learning

Everything above can be done in a spreadsheet. That’s deliberate. But if you have the data volume and the appetite, traditional machine learning (not LLMs) extends three of these techniques in genuinely useful ways.

Conditional forecasting. The prediction intervals above are team-level, with one range for everything the team does. Quantile regression forests and conformal prediction produce intervals conditioned on the specific item. Given its size, component, and the current queue depth, this work finishes in 3 to 12 days with 90% confidence. That’s a better input to a release plan than a team-wide average.

Survival analysis for work in flight. Cycle time statistics only describe items that have already finished, which quietly biases the picture toward work that went well. Survival models handle still-open items correctly and answer the question planners actually ask: what is the probability this item ships before the release date?

Driver attribution. Identifying what explains variation is hard, and the place modeling can really earn its keep. A gradient-boosted model with SHAP values can rank which operational signals (review latency, WIP, incident load, dependency count) account for the most variance in delivery. Treat the output as associational and as a hypothesis generator, not as evidence of cause.

Two caveats. Many teams don’t have enough history for any of this. Eight sprints is a worked example, not a training set. And the second caveat is that you need good workflow data hygiene. A model trained on tickets that sat in “In Review” for three weeks after they were actually merged will confidently forecast fiction.

Start with the spreadsheet.

Recommendation: Avoid a single predictability score

While it may be tempting to distill delivery predictability into a single score, I recommend against doing so. A single number simplifies reporting, but it can also obscure important differences between teams and make it difficult to identify the underlying causes of delivery risk.

For example, teams may all receive similar scores while exhibiting fundamentally different delivery characteristics:

• Consistently fast – High throughput with low variability.

• Consistently slow – Lower throughput, but reliable and predictable.

• Highly variable – Similar average performance, but large sprint-to-sprint fluctuations.

• Steadily improving – Predictability changing over time as the team’s delivery process matures, which a point-in-time score hides entirely.

Rather than reducing predictability to a single KPI, I recommend reporting three complementary views:

This approach preserves the information leaders need for planning while providing engineering teams with actionable insights into where process improvements will have the greatest impact.

Recommendation: Use a multi-step approach

I recommend measuring delivery predictability in four steps:

1. Select a delivery outcome. Choose a metric that represents delivery performance, such as throughput, completed work items, cycle time, or PR completion time.

2. Measure historical variability. Quantify how consistently that outcome is achieved over time using techniques such as standard deviation, percentiles, and control charts.

3. Forecast future delivery. Use prediction intervals to estimate a range of likely outcomes rather than relying on a single point estimate.

4. Identify the drivers of variation. Use existing engineering metrics, such as review latency, incident load, build reliability, focus time, or team composition, to explain changes in predictability and surface opportunities for improvement.

The goal is not to eliminate variation, but to understand it well enough to plan around it.

That’s it for this week. Thanks for reading.

-Brian

Share

Subscribe now

DX’s Q2 AI Impact Report is now available with the latest research on AI’s impact across engineering organizations. Read the full report.

When teams reduce the time it takes to complete a pull request, it’s natural to assume they’ll be able to deliver more work, but this relationship is rarely quantified at scale. A customer recently brought this question to DX’s Research team, asking us to look into the relationship between PR cycle time and PR throughput.

Key takeaway: High-throughput organizations are highly sensitive to long cycle times, potentially indicating that bottlenecks like review wait times are particularly acute for these orgs. For lower-throughput teams, there is no meaningful relationship between cycle time and throughput, suggesting bottlenecks exist elsewhere.

Queuing theory behind cycle time and throughput

Little’s Law, a theorem from queuing theory, helps explain the relationship between three things: how much work is in progress (WIP), how quickly work gets completed, and how long each piece of work takes. For software teams, we can think of these as:

• WIP: the number of pull requests currently open,

• Pull Request (PR) Throughput: number of pull requests merged over a period of time, and

• PR cycle time: how long a pull request takes to move from development through review and merge.

The equation is straightforward. When the amount and type of work stay relatively consistent, work that moves through the process faster can help a team complete more work over time. This relationship is commonly used to explain how software teams improve flow but it hasn’t yet been quantified in the developer productivity space.

Research by Maddila et al. shows that pull requests often spend significant amounts of time waiting, not being actively coded or reviewed, and implies that cycle time isn’t a fixed metric. Teams can recover some of that waiting time by improving the workflow around development and review, without changing the process of writing code.

To understand how PR cycle time relates to throughput, we looked at the relationship between median PR cycle time and average PR throughput per developer across organizations using data from 500+ DX customers from January-July 2026, inclusive.

Cycle time matters more for high-throughput organizations than it does low-throughput organizations

Our analysis suggests that PR cycle time is not a reliable standalone predictor of throughput. In the raw data, organizations with similar review speeds show a wide range of throughput levels, indicating that other factors, like PR size and organizational context, likely play an important role in determining output.

The median trend is relatively flat across most of the distribution, with throughput beginning to decline more noticeably only at longer cycle times. This indicates that reducing cycle time may not automatically increase throughput for every organization, particularly when teams are already operating within a typical throughput range.

Because median results can obscure differences across the performance distribution, we also used quantile regression to look at this relationship at the 25th, 50th, 75th, and 90th percentiles of throughput. This approach allows us to look at whether the association between cycle time and throughput changes for lower-, median-, and higher-throughput organizations.

The results show that the relationship between cycle time and throughput is concentrated among higher-performing organizations.

• At the 25th percentile, there is no statistically significant relationship between cycle time and throughput (p=0.366, β=-1.1).

• At the median, there is a modest but statistically significant negative relationship (p=0.0008, β=-4.0).

• At the 75th and 90th percentiles, the negative relationship becomes substantially stronger and remains statistically significant (p=0.000, β=-11.7; p=0.000, β=-12.8).

In practical terms, high-throughput teams are the most sensitive to cycle time slowdowns. For teams already in the bottom 25% of throughput, cycle time barely matters. For teams in the top 10%, it matters quite a bit.

Final thoughts: Speed is a safeguard but not a universal solution

Across most organizations, throughput stays fairly steady as PR cycle time increases. The relationship between the two becomes more noticeable around 17 hours. Beyond that point, median throughput starts to decline. This makes 17 hours a useful inflection point, not a universal target. For teams below that point, reducing cycle time may not lead to a meaningful increase in throughput. But once cycle times go beyond it, longer review cycles may start to limit teams that have the capacity to deliver more.

Think of the 17-hour mark as a warning sign and not a productivity cutoff. Teams with longer cycle times may benefit from looking more closely at review queues, reviewer availability, PR size, and other sources of waiting. These results show an association, not proof that longer cycle times directly cause lower throughput. Cycle time is most useful when considered alongside quality, work complexity, team structure, and other developer-experience measures, not as a standalone productivity target.

This week’s featured DevProd job openings. See more open roles here.

• Ashby is hiring an Staff Platform Engineer | Remote

• Carta is hiring a Sr. Software Engineer II, Developer Experience | Santa Clara, CA; San Francisco, CA; New York, NY

• Figma is hiring a Staff Software Engineer, Developer Experience | Remote; US

• Mastercard is hiring a Vice President, Software Engineering | New York

• Notion is hiring a Software Engineer, Developer Experience | Hybrid; Hyderabad, India

Share

AI adoption among software developers is approaching 100%, AI-authored code now makes up more than half of merged code, and developers report saving more time with AI every quarter. But those gains aren’t translating evenly into better outcomes.

In this episode of Engineering Enablement, I sit down with Justin Reock, Deputy CTO at DX, to unpack findings from our latest AI Impact Report. We explore where AI is improving engineering velocity and developer experience, where concerns are emerging around PR size, change confidence, and failure rates, and why rising AI spend has yet to produce a comparable increase in innovation.

We also discuss how AI is changing the meaning of code maintainability and where developers’ AI-driven time savings may actually be going.

Some takeaways:

AI adoption is no longer the interesting question

• AI adoption is approaching 100%, making adoption itself a less useful measure of success. DX sees 95% adoption through telemetry, and even developers outside that group are merging AI-generated code into production.

• The more important question is how effectively developers are using AI. As adoption becomes nearly universal, organizations need to look beyond usage and understand how sophisticated their AI usage is and whether or not it is producing meaningful outcomes.

AI is increasing engineering velocity, but the gains need context

• Developers report saving more time with AI each quarter, while PR throughput and deployment frequency are also increasing. Recent causal research gives additional evidence that AI is contributing to the increase in throughput.

• More than half of merged code is now AI-authored. But metrics like time saved, code generated, and PR throughput don’t tell us on their own whether teams are creating more customer value.

Larger PRs create risks that extend beyond code review

• Average PR size has increased from 42 to 72 lines of code, raising concerns about quality and code understanding. Larger PRs are harder to review and revert, more likely to be rubber-stamped, and can move more slowly through the development system.

• Code bloat can also create a vicious cycle of rising AI costs. As agents generate larger codebases, future agents have more code to read into context, consuming additional tokens repeatedly rather than creating only a one-time cost.

Developers can change code more easily while trusting it less

• Code maintainability is improving while change confidence is declining, even though those measures have historically been closely correlated. AI may make code easier to understand and modify while leaving developers less confident that their changes won’t break something in production.

• AI may be changing what “maintainable” code actually means. If developers increasingly rely on agents to understand and modify code, traditional measures of maintainability and ownership may no longer capture the same things they once did.

AI is amplifying differences between engineering organizations

• Change failure rates are becoming more volatile rather than moving uniformly in one direction. Some organizations continue to perform well while others are seeing higher failure rates, suggesting that AI can amplify existing strengths and weaknesses.

• Developer experience is also showing concerning signals despite clear improvements in areas like documentation. DX’s Developer Experience Index has fallen about 2% over two quarters, with declines in drivers such as incremental delivery outweighing some of AI’s benefits.

AI spending is growing much faster than measurable innovation

• AI spend has risen dramatically, reaching roughly 28 times year-ago levels among the largest companies in the dataset. At the same time, developer ramp-up has improved, but the innovation ratio has increased by only about one percentage point.

• Time saved by AI can easily be absorbed by existing organizational friction. Meeting-heavy days, interruptions, build and test wait times, developer environment toil, and review delays remain major sources of lost time. In one real-world case study, reducing unnecessary meetings produced roughly twice the PR-throughput gains achieved through AI.

• Ultimately, AI efficiency only matters if it translates into customer value. Removing friction and increasing throughput are useful, but the larger question is whether those gains allow organizations to ship more valuable software.

In this episode, we cover:

(00:00) Intro

(01:45) How the current AI impact report is tied to Core 4

(03:24) The state of AI adoption

(05:12) How much time AI is saving developers and percentage of AI-authored code

(07:47) AI’s impact on PR throughput and deployment frequency

(11:09) How EMs are shipping more code

(13:02) Why larger PRs may be problematic

(18:21) The growing gap between code maintainability and change confidence

(21:48) How perceived code quality varies by organization size

(23:49) The growing volatility in change failure rates

(28:07) What the Developer Experience Index reveals

(32:20) Cost, dev ramp-up, and innovation ratio

(35:38) Where AI time savings are getting lost

(37:11) Questions and wrap-up

Where to find Justin Reock:

• LinkedIn: https://www.linkedin.com/in/justinreock

Where to find Brian Houck:

• LinkedIn: https://www.linkedin.com/in/brianhouck

Referenced:

• DX Core 4 Productivity Framework

• AI Impact report

• The AI-native developer - by Brian Houck

• GitHub Copilot and Developer Productivity: An Observational Dose-Response Analysis

• Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools | NBER

• The Productivity-Experience Paradox - Annie Vella

• EngThrive: Make It Fast and Easy to Do Great Work

• The AI efficiency plateau - by Brian Houck

• Tradable Quality Hypothesis

Subscribe now

DX’s updated 2026 engineering benchmarks are now available to all customers, with new Core 4 data across industries and geographies. See 2026 benchmarks here.

Many engineering leaders are right when they tell me that industry benchmarks don’t apply to them. They’re just wrong about what that means.

Most have already looked at a benchmark, compared it to their own numbers, and concluded the comparison isn’t useful. The mistake isn’t recognizing that their organization is different—it’s expecting a benchmark to answer a question it was never designed to answer.

Organizational context genuinely matters. Smaller engineering organizations consistently outperform larger ones on many metrics. Technology companies spend more time on new features than traditional enterprises. Mobile engineering has sufficiently different workflows that it warrants its own benchmark segment. Even survey response styles differ systematically across regions, making some absolute comparisons misleading.

The mix of factors goes far beyond things we can easily segment. Every engineering organization has its own governance model, release process, architecture, regulatory requirements, engineering culture, and history. Some require five approvals before deployment; others deploy continuously. Some invest heavily in internal platforms;others rely on commercial tooling. These choices can dramatically affect developer metrics, making two organizations within the same industry or size cohort look very different from each other.

Those differences are real, but they don’t make benchmarking useless. They simply mean we’ve been asking benchmarks to answer the wrong question.

Exec summary

• Engineering leaders are often right that benchmark values don’t directly apply to them, but they are wrong to dismiss benchmarks entirely.

• Benchmarks, internal trends, and benchmark trends each answer a different question:

  • Benchmark levels → Are we normal?

  • Internal trends → Are we improving?

  • Benchmark trends → Are we improving faster than everyone else?

• The third question (“Are we improving faster than everyone else?”) is often the most important when evaluating investments like AI tools or process changes.

• Benchmark trends help separate your results from broader industry tailwinds (e.g., AI adoption, economic shifts)

• Benchmark values are still useful for identifying unusual performance and areas worth investigating.

• DX data shows metrics consistently move in the same direction across very different organizations year over year, suggesting organizations are less unique than they may believe.

• Changing metrics, survey instruments, or org structure mid-window makes it challenging to use benchmarks to measure change

• Benchmark trends won’t prove causation, but they meaningfully reduce uncertainty about what would have happened anyway.

• The real value of benchmarks isn’t knowing if you’re average, it’s reasoning more carefully about change.

Benchmarks and trends answer different questions

Benchmarks and trends answer fundamentally different questions.

• A benchmark answers: Is this normal? That question is often more valuable than we give it credit for. Knowing that your review latency or deployment frequency is unusual can help identify where deeper investigation is warranted, even if the benchmark itself doesn’t explain why. It tells you where you sit within a distribution. That’s a question about levels, and levels are influenced by things you may never be able to change like your industry, your size, your regulatory environment, your architecture, and the countless organizational decisions that shape how engineering gets done.Your own historical trend answers a different question: Are we improving? That’s a question about change. Because you’re comparing yourself to yourself, most of the differences that make your organization unique simply cancel out.

Both questions are valuable, but neither is the question engineering leaders usually care about most. The question they really want answered is: Are we improving faster or slower than everyone else?That’s a fundamentally different question, and it’s one that neither a benchmark nor an internal trend can answer on its own.

Suppose your deployment frequency improves by 15% over the next year. Is that good? If the rest of the industry improved by only 5%, you’re pulling ahead. If everyone else improved by 30%, you’re falling behind. In other words,improvement alone can’t tell you whether you’re pulling ahead or just keeping pace. Likewise, a benchmark can’t answer it by itself either. Knowing you’re at the 60th percentile today doesn’t tell you whether you’ve been gaining ground or losing it.

To answer the question leaders actually care about, you need both.That’s where benchmarks show their value, even for organizations that genuinely are unique. They don’t require your absolute metrics to be directly comparable to another company’s. They simply show you that the change in your organization can be interpreted alongside the change in everyone else’s.

And that, it turns out, is a far easier condition to satisfy.

Benchmark trends as an observational control group

The reason I find benchmark trends so valuable has very little to do with benchmarking. It has to do with causal inference.

Now suppose that 15% improvement in deployment frequency followed an investment in an internal developer platform. Was the platform responsible? Maybe. But maybe AI coding assistants became dramatically better during the same period. Maybe developer workflows improved across the industry. Maybe a slowing economy reduced feature work and increased engineering capacity everywhere.

A simple before-and-after comparison can’t distinguish between those explanations. That’s where benchmark trends really begin to shine. They don’t tell you what your metrics should have been; they tell you what happened to comparable organizations over the same period. They become an observational control group, a way of estimating the background improvement that would likely have occurred even if you had done nothing.

They’re not perfect. Organizations aren’t randomly assigned to different engineering strategies, and no benchmark population is identical to yours. My point is that they don’t have to be. If your deployment frequency improves 15% while comparable organizations improve 5%, that’s evidence that something beyond the broader industry trend may be happening inside your organization.

This is actually how measurement teams already communicate internally, even if they don’t describe it this way. When one engineering organization of roughly 3,000 developers adopted an AI agent to reduce live-site toil, the result wasn’t reported as the absolute changes in incident mitigation time. It was reported that mitigation time improved 2.4x faster than the company as a whole.

Nobody cared whether that organization’s services looked like the company average. The claim wasn’t about absolute performance. It was about relative improvement.

Your organization is less unusual than you think

At this point, there is still a reasonable objection.

Organizations don’t just differ in their absolute metrics, they also respond differently to new tools, new processes, and new ways of working. If every company has its own architecture, engineering culture, governance model, and technical debt, why should we expect benchmark trends to tell us anything useful at all?

I don’t think there’s a complete answer to that question, but I do think there is strong evidence that benchmark trends are more transferable than we expect.

Part of the reason is psychological. Researchers have studied our tendency to believe that our own situation is more unusual than it actually is. It’s called uniqueness bias. Organizations can fall into the same trap. Every company has a list of reasons why their engineering organization is unlike everyone else’s, and many of those reasons are legitimate. Organizational theorists have argued for decades that organizations facing similar environments tend to converge in their structures and practices, despite many local differences.

I’ve seen this in DX’s own benchmark data, and the pattern keeps repeating year after year. In our 2025 benchmarks, change confidence improved across every segment, with the median rising more than 12 points. Cross-team collaboration declined across every segment. In 2026, different metrics told the same kind of story. Documentation improved across every segment and customer focus rose across all of them, while review turnaround and incremental delivery declined across most.

These segments differ by size, by sector, and by geography, and their absolute values differ substantially. Yet year after year they move in the same direction at roughly the same time. If organizational context dominated the way the uniqueness objection assumes, we would expect these trajectories to diverge. Mostly they don’t. Those organizations weren’t identical, but they were responding to many of the same underlying forces.

It should be noted that direction and timing are shared, magnitude isn’t. Segments improved on the same metrics in the same years without improving by the same amounts, and individual organizations within a segment vary more still. Benchmark trends need both properties. Shared direction is what makes the baseline trustworthy. Dispersion around it is where your own signal lives. A control group is useful precisely because it behaves predictably, and the same is true here. Co-movement isn’t evidence that there’s nothing left to detect. It’s what makes detection possible.

This isn’t unique to engineering metrics, either. Economists routinely compare countries with different political systems, cultures, and industrial structures. Healthcare researchers compare hospitals serving very different patient populations. Education researchers compare schools with very different student demographics. None of these comparisons are randomized experiments, and none produce perfect control groups. Yet they’re still valuable because the alternative is to assume that nothing else in the world changed while your intervention took place.

I don’t think benchmark trends eliminate that uncertainty, but I do think they substantially reduce it. They’re not a replacement for understanding your own organization. They’re a way of putting your organization’s improvement into context.

You can’t trend against a moving ruler

Aligning to this way of thinking does change some of the advice we’ve historically given. We’ve encouraged organizations to improve their benchmark position over time, and I still think that’s good advice. Absolute benchmark values provide valuable context. They help answer whether your organization looks unusual relative to similar organizations, identify potential areas of opportunity, and highlight where deeper investigation might be worthwhile.

But once you understand where you stand today, the more interesting question becomes whether you’re improving faster than the background trend.

That distinction matters because organizations don’t improve in isolation. New tools, changing engineering practices, AI adoption, and broader industry shifts all influence engineering metrics over time. Benchmark trends help separate improvements that are happening everywhere from improvements that may reflect something unique about your organization.

This also explains why organizations that don’t trust absolute benchmark values shouldn’t dismiss benchmark data entirely. Even if your architecture, culture, or regulatory environment makes direct comparisons difficult, organizations facing similar external forces can still provide valuable context for understanding how quickly the world around you is changing.

That said, comparing changes only works for the things that hold still. If you reorganized, acquired a company, or changed how you count engineers during the same window, those differences don’t cancel either, and your trend becomes as hard to interpret as your levels were. Changing a metric definition or a survey instrument partway through does the same damage. You can’t trend against a moving ruler, which is why, for trend analysis, consistency in how you measure matters more than precision in what you measure.

None of this makes benchmark cohorts less important. The better your comparison group, the better your estimate of that background trend.

Absolute benchmarks tell you where you are. Internal trends tell you whether you’re improving. Benchmark trends help answer whether you’re improving faster than you would reasonably have expected.

What we still don’t know

I can’t yet prove that benchmark trends consistently produce better decisions than benchmark levels alone. I also don’t know which comparison groups produce the most useful estimates of background improvement, or how similar two organizations need to be before their trends become informative.

Those are empirical questions, and I think they’re worth studying.

My intuition is that benchmark trends won’t eliminate uncertainty. They simply provide another source of evidence. Like any observational control, they’re imperfect. But imperfect evidence can still improve decision making when it’s interpreted appropriately.

Final thoughts

Benchmark trends aren’t a replacement for internal metrics, experiments, or randomized trials. Whenever we can run controlled experiments, we should.

But most engineering organizations don’t make decisions under laboratory conditions. We launch AI coding assistants to everyone at once. We reorganize teams. We change review processes. We invest in developer platforms. We rarely have the luxury of a true control group.

That means engineering leaders spend much of their time making decisions from imperfect evidence.

Benchmark trends make that evidence meaningfully stronger. They don’t eliminate uncertainty. They don’t prove causation. They don’t guarantee that one intervention caused another outcome. They simply provide a better estimate of what might have happened anyway.

To me, that’s the real value of benchmarks. Not because they tell us whether we’re average. But because they help us reason more carefully about change.

That’s it for this week. Thanks for reading.

-Brian

Share

Building on this year’s inaugural event, which brought together 500 senior engineering leaders from companies like Microsoft, Airbnb, Uber, Vanguard, Dell, and BNY, we’re expanding to two cities for 2027.

In both San Francisco and London, DX Annual will be a single-day event with keynotes, fireside chats, and peer roundtables built around how organizations are rethinking developer productivity and how software is delivered with AI. The event will once again focus on curated attendance, meaningful peer connections, and sessions led by practitioners.

Save the dates ↓

• San Francisco — May 6, 2027

• London — September 23, 2027

Register your interest

In this episode of Engineering Enablement, I sit down with Tim Bozarth, Corporate Vice President in Microsoft CoreAI and leader of Microsoft’s Engineering Thrive initiative. We discuss Engineering Thrive, Microsoft’s framework for measuring and improving engineering productivity, and why AI makes outcome-based metrics more important than ever.

Tim shares how AI is reshaping the software development lifecycle, where new bottlenecks are emerging, and why verification and confidence may become more valuable than code generation itself. We also explore why the purpose of engineering remains the same despite rising levels of abstraction, the skills that remain durable in an AI-driven world, and why engineering leaders should focus on outcomes rather than activity metrics.

Some takeaways:

Engineering Thrive gives Microsoft a common language for improving productivity

• Engineering Thrive defines productivity through speed, ease, and quality. Its goal is to make it fast and easy to do great work by identifying friction across the developer experience rather than optimizing isolated activities.

• Speed, ease, and quality should not be treated as opposing goals. Engineering Thrive applies a “do no harm” principle: an improvement in one dimension should not be celebrated if it makes another materially worse.

• The framework helps Microsoft identify bottlenecks and invest where they will have the greatest impact. Tim argues that developer time is one of the company’s most valuable resources, making productivity improvements a strategic investment.

AI makes outcome-based metrics more important, not obsolete

• The industry is returning to activity metrics it rejected years ago. Tim sees renewed attention to lines of code, pull request counts, and similar measures as a search for easy answers to how AI is affecting productivity.

• More engineering activity does not necessarily mean more value. Engineering Thrive instead measures outcomes such as product quality, end-to-end speed, and the amount of time engineers can devote to innovation.

• A web of outcome metrics is harder to game than any individual measure. Looking across speed, ease, quality, and innovation time creates a more durable picture of whether an organization is actually improving.

Planning and validation are becoming the new bottlenecks

• Before AI, most engineering time was spent creating and operating software. Tim estimates that those phases historically accounted for more than 90% of engineering time, with operations alone consuming roughly 70% to 80%.

• On frontier teams, the bottlenecks have already shifted to planning and validation. Code creation is taking dramatically less time, while deciding what to build and determining whether the result is trustworthy are consuming a greater share of the work.

• The create phase may continue to shrink as models and agent harnesses improve. Tim expects planning and validation to remain durable constraints over the next several years, even as other parts of the software development lifecycle become increasingly automated.

The code review bottleneck is really a confidence bottleneck

• Higher PR throughput has increased the amount of change humans must evaluate. Enterprise software still requires a level of trust that teams cannot achieve by simply accepting AI-generated code without review.

• Code review is only one way to establish confidence. Testing, continuous rollouts, feature flags, canaries, and deployment practices all contribute signals about whether a change is reliable.

• The next generation of verification should help humans ask higher-level questions. Rather than inspecting every branch or class, engineers should be able to assess the scope, complexity, impact, and fundamental purpose of a change.

More abstraction does not change the purpose of engineering

• AI will reduce the time engineers spend on low-level implementation details. Tim sees that as another step in the long history of abstractions that allow engineers to spend more time describing system behavior and intended outcomes.

• Great engineers have never been defined by their ability to write a line of code. Their value comes from systems thinking, understanding objectives, and expressing functional and nonfunctional requirements coherently.

• Producing more software increases the need for strong engineering judgment. The faster organizations can build, the more important it becomes to ensure that their systems remain coherent, valid, and reliable.

A maker’s mindset and ability to experiment effectively remain durable advantages

• The maker’s mindset is relentlessly oriented toward producing a valuable final outcome. It combines a clear vision of what should be built with the ability to understand the needs of the customer using it.

• Engineers will need to continuously experiment and evaluate changes in outcomes. Tim argues that this discipline is no longer limited to model trainers; everyone using AI tools must determine whether a new approach actually improves the result.

• Using AI is not itself the goal. The goal is to accomplish more, and sometimes the best way to do that will be not to use AI at all.

Engineering leaders should measure idea-to-value, not PR velocity

• PR velocity says little about the success of an engineering organization. Tim urges leaders to stop evaluating teams through activity measures the industry already recognized as inadequate years ago.

• Idea-to-value connects engineering work to meaningful outcomes. Leaders should also examine how much time engineers spend innovating compared with keeping the lights on or handling corporate overhead.

• Product quality and business results should remain the ultimate measures of success. Focusing on outcomes can improve developer happiness, productivity, margins, and the value delivered by the organization.

In this episode, we cover:

(00:00) Intro

(02:13) What Engineering Thrive is and the problem it solves

(04:46) Why Engineering Thrive isn’t specific to Microsoft

(09:10) The impact of Engineering Thrive at Microsoft

(14:31) Why AI makes outcome-based productivity metrics more important

(18:22) Where AI is creating new bottlenecks in the SDLC

(24:37) Why more abstraction doesn’t change the purpose of engineering

(27:25) The durable skills of good engineers

(33:03) The changing economics of software development

(36:56) Advice for leaders: measure outcomes, not activity

Where to find Tim Bozarth:

• LinkedIn: linkedin.com/in/tbozarth

Where to find Brian Houck:

• LinkedIn: https://www.linkedin.com/in/brianhouck

Referenced:

• DX Core 4 Productivity Framework

• EngThrive: Make It Fast and Easy to Do Great Work: Building a durable model for outcome-oriented engineering measurement

• Quote by Eliyahu M. Goldratt: “Tell me how you measure me and...”

• Jevons paradox - Wikipedia

Subscribe now

DX’s Q2 AI Impact Report is now available with the latest research on AI’s impact across engineering organizations. Read the full report.

Something is straining in the review queue.

Over the past year at Meta, significant lines of code per human-landed diff increased by 106%. Diffs per developer per month rose 51%. More than 80% of that growth came from agentic AI. Meanwhile, the percentage of diffs reviewed within 24 hours is declining. In some large groups, reviewers are staring down thousands of pending reviews.

This isn’t a Meta-specific problem. Across the industry, AI coding tools are producing code faster than humans can meaningfully evaluate it. Our own DX analysis found that AI is increasing both the number of pull requests and the size of each one (median pull request size grew by 64%). Without a corresponding increase in reviewer capacity, the review process will eventually buckle under its own weight.

Unfortunately, we don’t have more hours in the day, and even if we did, we wouldn’t want to spend them reviewing code written by AI. In previous research, we found developers ideally only want to spend about 7% of their time reviewing code. Asking developers to review more isn’t a sustainable answer.

The math doesn’t work.

But before we ask AI to solve this problem, it’s worth asking a different question:

What problem was code review solving before AI arrived?

If the answer were as simple as “finding defects,” then a fully automated review starts to sound inevitable (and appealing).

But if code review was also how teams shared knowledge, built collective ownership, spread architectural understanding, and taught junior engineers how experienced developers think, then the answer becomes much less obvious.

That’s the mistake I think many organizations are about to make, and the reason we need to rethink what code review is actually for.

We’ve known better for years

Here’s an uncomfortable truth: a significant portion of the review burden we’re feeling right now is self-inflicted.

The frustrating part is that none of this is new. The research on what makes code review effective has been unambiguous. Keep changes small. Write a meaningful description of what changed and why. Run automated checks before asking a human to look. Review frequently, in bounded sessions, focused on substance over style. Select reviewers who actually know the code, but avoid concentrating review responsibility on the same small group of experts whenever possible.

A 2016 Microsoft study of 911 developers found that timely feedback, review size, and understanding the motivation for a change were the top three challenges in code review. Those challenges should sound familiar. The research had already identified many of the practices that improve review quality, yet only 26% of developers said they always wrote a detailed description of the code being reviewed. “Bikeshedding”—disputing minor issues while more serious ones went unexamined—remained one of the most common review failures. We didn’t need new guidance. We needed to consistently apply what we already knew.

AI didn’t create this situation. It inherited it, and then amplified it. Larger PRs, higher review volume, less context per change, these aren’t new symptoms. They’re old ones, scaled up.

Before asking AI to fix your code review process, ask whether your team has built the habits that make code review effective in the first place. Small, well-explained changes. Protected reviewer time. Automated routine checks so humans can focus on judgment.

AI can absolutely improve code review. But it can’t compensate for a review culture that was already struggling. It doesn’t eliminate bad review habits. It amplifies them.

AI can help, if we use it wisely

Once the fundamentals are in place, AI has a real role to play in code review. That’s exactly what we found in our AI Where It Matters research. Developers don’t want code review to disappear. They want AI to remove the parts of review that don’t require human judgment so reviewers can spend more time on the parts that do.

What they want AI to do: catch security and compliance issues, flag high-risk changes, generate test scaffolding, surface the impact of a change across the codebase, and handle the high-volume routine so human attention can go where it matters. As one developer put it: “Should be able to detect high risk changes and derisk them.”

What they explicitly don’t want: AI that auto-merges, auto-commits, or takes final accountability. “I don’t want AI to just act as a red-light / green-light. It should raise issues… and still require human review.” Developers aren’t asking for a replacement, they’re asking for a better collaborator.

Interestingly, one of the most sophisticated production deployments I’ve seen tries to walk that line.

Meta’s RADAR (Risk Aware Diff Auto Review) system automates review for a carefully selected subset of low-to-medium risk changes while routing higher-risk diffs to human reviewers. It combines static analysis, machine learning, LLM-based review, and deterministic validation before anything lands.

The results are striking: more than 535,000 diffs reviewed, over 331,000 landed, a revert rate roughly one-third that of non-RADAR diffs, a production incident rate one-fiftieth as high, and a 3.3x faster median time to close (roughly a 70% reduction).

RADAR isn’t simply “AI reviewing code.” It’s a carefully engineered system built around the principle that scarce human attention should be reserved for changes where human judgment and accountability matter most.

Just as importantly, the RADAR team also acknowledges a trade-off. Automated review can dramatically improve efficiency, but as automation expands, the knowledge transfer provided by human review could suffer. They identify this as something engineering organizations should actively monitor.

That’s the distinction I think many organizations miss. AI shouldn’t eliminate human review. It should make human review more valuable. AI-enabled review should have discipline around it: clear eligibility criteria, thoughtful risk stratification, and a deliberate decision about which changes deserve human attention, and why.

If you’re evaluating an AI review system, don’t start by asking, “Does it work?” Start by asking, “How does it maximize the time and value of human judgment?”

Don’t lose what review was actually doing

Here’s the part that gets left out of the AI review conversation: code review was never just about finding defects.

A landmark Microsoft study found that while most developers identified defect detection as a primary motivation for code review, defect-related comments made up only 14% of actual review comments. In practice, code review serves many other purposes. More than half of developers said they use reviews to explore alternative solutions, while many also pointed to knowledge transfer and gaining awareness of what their teammates are building.

The visible output of code review is better code. The invisible output is a better engineering organization.

Code review is how teams build shared understanding of a system. It’s how junior developers learn from experienced ones. It’s how architectural intent gets surfaced, questioned, and refined. It’s how one engineer’s mental model gradually becomes the team’s mental model. Organizations don’t become resilient because one person understands a subsystem. They become resilient because many people do.

This is why the stakes around AI review are so high. Automate the review of a diff, and you may have successfully reviewed that diff. But you haven’t transferred any knowledge. You haven’t built shared ownership. You haven’t given a newer engineer a window into how a more experienced teammate reasons about trade-offs. You haven’t surfaced the design rationale that someone will need six months from now when they’re trying to respond to customer feedback.

Developers in our AI Where It Matters research understood this instinctively. One participant wrote, “I can’t fully delegate the final code review to AI—my approval puts my name on it.” Another warned, “Intellectual offloading can result in errors that eventually no one understands.” That’s the slow-moving risk. The gradual erosion of a team’s ability to reason about its own software.

Margaret-Anne Storey’s recent work gives this phenomenon a name. As AI accelerates software development, teams don’t just accumulate technical debt. They accumulate cognitive and intent debt—a growing gap between what the system does and what the organization collectively understands about why it does it. Those debts don’t appear on a dashboard. They surface months later, during an outage, a handoff, or a redesign, when nobody remembers the reasoning that once lived inside a code review conversation. By then, recovering that understanding is far more expensive than preserving it would have been.

This future isn’t inevitable. But it also won’t arrive all at once. It will emerge through a series of individually reasonable decisions: this change is low risk, this review can be automated, this approval can be skipped. Each decision saves a little time. Taken together, they may slowly eliminate one of the primary ways engineering teams build shared understanding.

The challenge isn’t choosing between AI and human review. It’s deciding which parts of code review are too valuable to automate away.

What to actually do

Three things, in order.

Fix the basics first. Audit your current review process. Are pull requests small enough to review meaningfully? Do change descriptions explain why, not just what? Are reviewers protected from overload? Are automated tools already handling the routine work they should (e.g. formatting, linting, and obvious style issues)?

Design AI around human judgment. Developers consistently describe code review as high-value, high-accountability work. They don’t want AI making the decision; they want AI helping them make better ones. That means risk stratification instead of blanket automation. It means AI that surfaces issues, not AI that silently resolves them. It means conservative eligibility thresholds, auditability, and clear human accountability.

Protect what review is actually building. The easiest thing to measure about code review is defects. The most valuable thing it produces is shared understanding. Measure review health beyond throughput. Are junior developers learning? Is architectural knowledge spreading across the team? Are reviewers engaging with substance or simply rubber-stamping? Design your AI review strategy so automation absorbs the routine while humans spend more time on the conversations that create understanding, ownership, and better engineering judgment.

Code review is one of the highest-leverage practices in software engineering, and right now it’s under pressure from every direction. The answer isn’t to make it faster by making it shallower. It’s to get serious about doing it well—with or without AI—and then use AI deliberately, in the places where it earns trust and preserves what the practice was accomplishing all along.

AI should absolutely reduce the time we spend reviewing code. It just shouldn’t reduce the amount we learn from it.

That’s it for this week. Thanks for reading.

-Brian

Share

Subscribe now

DX’s Q2 AI Impact Report is now available with the latest research on AI’s impact across engineering organizations. Read the full report.

A quick heads-up: this issue is a little different from our usual format. Instead of sharing a finding from our research and conversations, this issue is more of a reframe of how to think about what developer metrics are actually for.

Here is a sentence I never expected to write: Developer-experience metrics can measure the impact of Daylight Saving Time.

Not a survey asking developers how they feel about the time change. An actual, measurable shift in engineering behavior, detected using the same kinds of outcome metrics we use to evaluate AI coding assistants, build systems, and code review workflows.

That sounds like an absurd thing to measure. And yet, that’s exactly what we found.

When Daylight Saving Time begins, developers in Seattle (one of the highest-latitude major cities in the continental United States) show a measurable shift relative to developers closer to the equator, where seasonal daylight changes are much smaller. Across three years of data, active coding time increased by roughly 6%, while pull requests completed about 8% faster, driven largely by quicker code reviews.

The most plausible explanation is also the simplest: an extra hour of evening daylight appears to keep people engaged with their work a little longer.

I’d resist reading that as purely good news. More engagement isn’t automatically healthier. If some of that extra time is coming at the expense of sleep or recovery, that’s a trade-off worth measuring, not celebrating.

At first glance, this has nothing to do with software engineering. After all, developer metrics are supposed to measure developer things: faster builds, better IDEs, improved code reviews, AI-assisted coding.

Or so I thought.

The more I’ve worked with developer-experience metrics, the more I’ve come to believe we’ve been thinking about them too narrowly. We often describe them as a way to evaluate developer tools and engineering workflows. But that’s not really what they’re measuring. They’re measuring the experience of doing software engineering. And that experience is shaped by far more than software.

Measuring outcomes, not interventions

When people think about developer-experience metrics, they naturally think about the interventions we introduce: a new AI coding assistant, a faster build system, a different code review process, a new deployment pipeline.

But those aren’t actually what the metrics care about.

Good developer-experience metrics measure outcomes. They tell us whether developers are able to do focused, meaningful, high-quality work. Once you measure outcomes instead of interventions, something interesting happens.

The intervention no longer has to be software. It can be an office redesign. A meeting policy. The weather. Even Daylight Saving Time.

That doesn’t mean engineering leaders suddenly own the weather, facilities, or company policy. But measurement doesn’t have to imply ownership. Sometimes it helps explain why an outcome changed. Other times it gives you evidence to influence the people who do own the lever.

That realization changed how I think about developer metrics. They’re still excellent for evaluating developer tools. They just turn out to be useful for much more.

Once I started looking through this lens, examples kept appearing, not just in my own research, but across entirely different disciplines. And they weren’t limited to software engineering. Researchers have found that higher indoor air pollution correlates to more errors by chess players, warmer classrooms are associated with lower student performance, and high-noise environments measurably degrade memory and motivation. Different domains, different outcomes, but the same underlying lesson: our environment shapes performance.

The difference is that developer-experience metrics give us a language for asking the same kinds of questions about software engineering.

Some of those influences are things organizations can change. Others aren’t. Both leave measurable fingerprints on the developer experience.

The spaces your developers work in

Daylight Saving Time is an unusual example because there isn’t much an engineering leader can do about it. Office space is different. Organizations make decisions about where and how developers work all the time, yet those decisions are often driven by cost, convenience, or intuition rather than evidence.

When we asked developers what they actually value about coming into the office, the answers were overwhelmingly human. The top response, by a wide margin, was other people: seeing colleagues face to face, the conversations that get sparked, the camaraderie. Office design came next (whiteboards, rooms to hash out problems), followed by food and coffee. This isn’t just a list of office perks. It’s a window into the parts of the developer experience that still depend on the physical world.

The specifics ranged from the practical to the personal. One developer’s entire reason for coming in: “I don’t have to fight with my cats.” However they phrased it, the pattern was the same. What draws developers to the office is overwhelmingly human, and largely physical, the parts of the experience that software still hasn’t managed to replace.

Those findings explain why the office matters. Developer metrics can help answer the next question: how much does it matter?

In a previously unpublished analysis, I looked at teams relocating between office spaces. The building itself moved the numbers.

One team relocated into a well-designed, high-performance office and saw developer engagement, measured through active coding time, increase by roughly 13% relative to comparable peer teams. The really interesting part came later. When that same team eventually moved back into their original office, their active coding time returned almost exactly to its previous level.

I also saw the opposite pattern. Teams moving into poorly designed office spaces experienced productivity declines on the order of 10%.

This is observational, not a controlled experiment, so we should be cautious about claiming causality. Office moves often coincide with organizational changes, new teammates, and countless other confounding factors. But the pattern is still striking. The same outcome metrics we might use to evaluate a new AI coding assistant also measured the impact of an office redesign. The intervention changed. The measurement didn’t.

That changes the conversation. Office space is usually treated as a facilities expense to be minimized. But if a better workspace meaningfully improves the developer experience, it becomes a productivity investment instead. A useful rule of thumb is that facilities costs are roughly 10% of payroll. That means an improvement in developer effectiveness on the order of 10% has the potential to offset the entire cost of the workspace, before considering any additional benefits such as hiring, retention, or collaboration.

Interestingly, our earlier hybrid-work research found another version of the same idea. Developers who chose whether to work from home or the office based on the type of work they planned to do reported higher productivity than those choosing primarily for personal convenience. Different environments appear to support different kinds of work, and developer metrics give us a way to test those assumptions rather than simply debate them.

When the weather is a variable

Daylight Saving Time at least has policy debates attached to it. Weather is even simpler. No engineering leader can change it.

And yet it still shows up in the data.

In Seattle, you can often identify winter snow days simply by looking at engineering activity. Active coding time drops by roughly 18%. The reasons are easy to imagine: disrupted commutes, childcare, school closures, or simply the irresistible pull of a rare Pacific Northwest snow day. Untangling those mechanisms is difficult, and I wouldn’t claim a clean causal story.

But the mechanism isn’t really the point.

The point is that the measurement detected the change.

At first glance, measuring something you can’t control might seem pointless. I think the opposite is true. If a team’s delivery slows during a snowstorm, that isn’t necessarily a performance problem to solve. It’s the context that helps explain what happened.

That’s one of the underappreciated benefits of developer-experience metrics. Sometimes their greatest value isn’t telling you what to change. It’s telling you what not to blame.

Knowing that a metric moved because of external circumstances prevents organizations from chasing the wrong explanations, setting unrealistic expectations, or concluding that a team suddenly became less effective when nothing about the team actually changed.

The durable part

There is a reason I keep coming back to these examples, and it is not just that they are fun to share at a dinner party.

It is tempting to think of developer-experience metrics as a way to evaluate developer tools. But that is too narrow. Good developer metrics measure outcomes, not interventions. They tell us whether developers can do focused, meaningful, high-quality work. The intervention itself might be a faster build, a better office, a meeting policy, or even something as unexpected as Daylight Saving Time.

That distinction is what makes these measurement systems durable.

Five years ago, organizations were asking different questions than they are today. Five years from now, they’ll ask different questions again. AI is the dominant topic today, just as cloud development environments, CI/CD, or code review tooling were at other points in time. The interventions evolve. The outcomes we care about—Speed, Ease, Quality, and Thriving—do not.

That’s why I don’t think AI is a special case. It is simply the latest intervention whose impact we want to understand. The questions remain the same: Does it help developers do better work? Does it reduce friction? Does it improve quality? Does it help people thrive?

The tools will continue to change. The interventions will continue to change. The measurement doesn’t have to.

It also leaves an interesting question for another day: if these ideas apply so well to software engineering, how much further do they extend?

That’s it for this week. Thanks for reading.

-Brian

Share

Subscribe now

DX’s Q2 AI Impact Report is now available with the latest research on AI’s impact across engineering organizations. Read the full report.

When an organization invests in an AI coding tool, the next question is almost always some version of “How do we prove it’s working?” It’s a fair question, and one many organizations are wrestling with today.

I was recently asked to weigh in on one such proposal: a metric called “Developer Horsepower,” defined as useful work per day and calculated by multiplying AI-assisted pull requests by an estimate of the human effort each would have required. It’s an intuitive idea, and a genuinely thoughtful attempt at a hard problem. But I think it starts one step too early.

Rather than asking how much human work the AI replaced, I’d ask whether AI has increased the organization’s capacity to deliver innovation (and whether that additional capacity is sustainable). That framing leads to a very different measurement strategy, and one that I believe is both easier to defend and more actionable.

First, separate “what” from “how”

Two different questions get tangled together in most measurement conversations. The first is what to measure. The second is how: not just how you collect the data (surveys versus telemetry) but how strong your evidence needs to be. Is a correlation enough, or do you need a full causal study?

Proving causation is genuinely valuable, and I’d never talk someone out of it. If you’re doing something novel, or you want to publish in a peer-reviewed journal, there are good approaches available, from difference-in-differences designs to dose-response studies. But for most organizations, a full causal study is overkill. There is now substantial causal evidence that AI coding tools can improve coding throughput under many conditions. You don’t need to re-prove that AI increases coding throughput; the field has done that work. In most cases you can measure the correlations in your own environment and lean on the existing causal literature to interpret them. That’s a far lighter lift, and it’s honest (as long as you’re not claiming something the literature doesn’t support).

With that settled, the interesting question is what to measure.

What to measure: innovation capacity, across dimensions

The goal isn’t to isolate the tool. It’s to answer whether your engineering system now has more capacity to deliver innovation, and whether that capacity is sustainable. I’d build that up in layers.

Throughput: Is more total work happening? PR throughput is a perfectly valid system-level signal for whether more work, in aggregate, is moving through your engineering system. It gets criticized, and rightly so, as a measure of any individual developer, because not all PRs are equal in size or complexity. But at the system level it holds up well. If you can normalize for PR complexity, that’s better still (TrueThroughput is one approach to doing that). But even without normalization, aggregate throughput remains a useful directional indicator. I explore this in more depth in the recent article: Revisiting the DX Core 4 in the age of AI

Deployments: Does the work survive to shipped software? This is the step most teams skip, and it’s the most important one. Recent research shows that AI increases coding activity far more than it increases shipped software. In one large study of over 100,000 developers, coding agents raised commit volume by up to 180%, yet the effect on actual releases attenuated to roughly 20-30%. The authors attribute the gap to a “weak link”: the upstream speedup runs into human bottlenecks downstream, in review, integration, and release.

Counting merged PRs alone may overstate impact, because a large share of that activity attenuates before it reaches production. Tracing your throughput increase all the way through to deployments tells you how much of the upstream gain actually survives.

I’d frame the result as innovation capacity: the organization’s realized capacity to deliver innovation. Note the deliberate boundary. Whether that additional capacity translates into business value is a separate question, and often not an engineering one. Deployment frequency is the practical proxy for delivered work; business impact is the true north if you can reach it. Conflating the two is how measurement programs lose credibility.

Innovation Time Ratio: Are the savings being reinvested? AI only creates organizational value if the time it saves is reinvested in higher-value work. This is exactly what a metric like Innovation Time Ratio captures: the share of developer time spent creating new value versus running the business or carrying administrative load. For most organizations, survey data is the logical place to start, unless you already have robust calendar and development telemetry. And self-report is less of a concern here than it first appears, because you’re looking at change over time. Whatever reporting bias exists tends to stay consistent across measurements, so it largely cancels out in the delta.

Quality: A guardrail, not an afterthought. More work moving faster is only progress if it isn’t just churn. A quality signal such as change failure rate or incident mitigation time keeps the speed gains honest. If throughput climbs while failure rates climb with it, you haven’t gained capacity. You’ve moved the cost somewhere less visible.

Satisfaction: The sustainability guardrail. Finally, watch developer satisfaction. Throughput and experience can decouple, and gains bought by burning out your engineers aren’t gains you get to keep. Sustainable capacity requires healthy developers. If higher throughput comes at the expense of satisfaction, you’ve likely borrowed from future capacity rather than increased it.

Taken together, those layers answer the question leadership is actually asking: not “how much code did the tool produce?” but “does our system have more capacity to deliver innovation, and can we sustain it?”

“Developer Horsepower”

Which brings me back to the proposal I was asked about. I understand the appeal of a single “horsepower” number, but ultimately I’d recommend a different approach.

To be clear, my aim isn’t to single out this particular metric. It’s a reasonable attempt at a real problem, and the team that proposed it is asking exactly the right question. I want to use it as a worked example of the kind of probing I do whenever someone hands me a composite score: what is the unit actually measuring, and how much variation is it hiding? Those questions apply to any single-figure productivity metric, not just this one.

My concern isn’t that “Developer Horsepower” is impossible to calculate. It’s that I’m not sure it is answering the question that really needs answering. Leadership doesn’t ultimately care how many “human-equivalent hours” an AI system replaced. They care whether their engineering organization can deliver more innovation, more reliably, and more sustainably than before.

The core problem is that it tries to collapse a multidimensional question into one figure. That runs against a principle the field has largely converged on: engineering productivity can’t be captured by a single metric. A single number simplifies reporting, but it obscures the tradeoffs that matter and makes it difficult to understand what’s actually driving change.

The specific construction compounds this in several ways.

First, the normalization itself is non-standard and difficult to interpret. “Human-equivalent hours” isn’t an industry term, and a figure built on a bespoke conversion is difficult for anyone outside the team to trust or reason about.

More fundamentally, it’s not obvious what those hours are supposed to represent.

One of the more surprising findings from our AI Native Developer research was that developers spend only about 14% of their work week actively writing code. The rest is spread across activities like design, code review, debugging, learning, meetings, planning, documentation, and coordinating with teammates. AI doesn’t simply replace coding time; it changes how engineers spend time across many of those activities. Some work disappears, some shifts, and entirely new work (like reviewing AI-generated code or managing context) emerges.

Trying to convert an AI-assisted pull request into a fixed number of “human hours saved” assumes those relationships are stable and additive. In practice, they’re neither.

Even if “human-equivalent hours” were the right unit, a flat estimate is almost certainly inaccurate. Eight hours per PR treats every pull request as identical, but PRs vary enormously in size, complexity, and review effort.

The research also increasingly suggests that AI and human effort remain largely complements rather than substitutes. Engineers still spend substantial time reviewing, integrating, validating, testing, and coordinating around AI-generated code. A clean “hours replaced” conversion assumes substitution where the evidence points toward augmentation.

Where to start

If this sounds like more than you can measure today, start anyway. You do not need to be able to measure all of this with a mature telemetry stack in order to begin.

Don’t have the telemetry? Use surveys. If you have precise, telemetry-driven data for every commit, review, build, and deployment, you’re in an uncommonly good position. Most organizations aren’t, and that’s fine. Targeted surveys will get you moving quickly, and they’re better suited to some of these dimensions than telemetry is anyway (reinvested time and satisfaction, for instance). The usual objection is self-report bias, but it matters less than people expect here, because you’re watching change over time. Whatever bias exists tends to stay consistent across measurements, so it largely cancels out in the delta. The measure only has to be directionally correct to be useful.

Don’t feel lost if you can only start with a couple of metrics. You don’t need all five dimensions on day one. Pick one or two metrics per dimension, balancing an objective signal with a subjective one, and add more as your instrumentation matures. A speed metric from telemetry paired with a satisfaction question from a survey tells you more than five telemetry metrics alone. Too many metrics dilute focus and make it harder to see what’s actually changing. Start small, learn what moves, and expand deliberately.

As your instrumentation matures, work toward the full picture. The richer your measurement, the more confidently you can answer whether your capacity to deliver innovation is real and sustainable.

The bottom line

The pressure to produce one clean number that proves an AI tool’s horsepower is understandable, but it answers the wrong question. The better question is whether your engineering system has more capacity to deliver innovation, and whether that capacity is sustainable. Measure that across a few well-chosen dimensions, lean on the causal work the field has already done, and you’ll have something far more defensible than any single horsepower figure, and far more useful for deciding what to do next. AI shouldn’t be judged by how much human work it appears to replace. It should be judged by whether it gives your engineering organization a greater and more sustainable capacity to deliver innovation.

This week’s featured DevProd job openings. See more open roles here.

• Ashby is hiring an Staff Platform Engineer | Remote

• Cart is hiring a Sr. Software Engineer II, Developer Experience | Santa Clara, CA; San Francisco, CA; New York, NY

• Cashea is hiring an Infrastructure & Developer Productivity Platform Engineering Manager | Remote

• Figma is hiring a Staff Software Engineer, Developer Experience | Remote; US

• Morgan Stanely is hiring an AI Platform Engineer - Vice President | New York

• Notion is hiring a Software Engineer, Developer Experience | Hybrid; Hyderabad, India

Share

AI coding agents are changing how software gets built, but they’re also forcing us to rethink how we measure engineering effectiveness. Traditional developer experience metrics were designed for humans, not AI agents, so how should engineering leaders adapt?

In this webinar, I’m joined by Brian Houck, Distinguished Scientist at DX and co-author of the SPACE framework, to explore the emerging field of agent experience and how it builds on developer experience rather than replacing it. We discuss how organizations can prepare for AI-assisted software development, how the DX Core 4 applies in the age of AI, why metrics like token usage and PR throughput don’t tell the whole story, and the growing importance of documentation. We also examine the impact AI-driven pressure is having on burnout and cognitive overload.

Throughout the conversation, we share practical guidance for building engineering organizations where both developers and AI agents can do their best work.

Some takeaways:

Agent experience builds on developer experience

• Agent experience focuses on creating the conditions for AI agents to succeed. Brian defines agent experience as the environment surrounding AI agents, including the quality of context, documentation, validation, and feedback they receive. As agents become part of software teams, improving those conditions becomes increasingly important.

• Model quality is only one part of successful AI adoption. Organizations often focus on choosing the best model, but Brian argues that context, clear intent, and effective workflows often have a greater impact on outcomes than incremental improvements in model capability.

• The same systems that help developers often help AI agents. Investments in documentation, development workflows, and engineering platforms create a stronger foundation for both humans and AI to produce high-quality work.

Developer experience and agent experience don’t always align

• Many improvements benefit both developers and AI agents. Better documentation, clearer context, and stronger engineering practices improve outcomes across the board, making existing developer experience investments even more valuable.

• Optimizing for one doesn’t automatically optimize for the other. Brian explains that organizations will increasingly encounter situations where workflows that help AI agents introduce friction for developers, or vice versa.

• Organizations should measure both independently. Rather than assuming every AI optimization improves the developer experience, engineering leaders should evaluate where the two reinforce each other and where they diverge.

Preparing for AI requires organizational change

• Successful AI adoption requires more than coding tools. Justin and Brian describe AI readiness as a combination of developer tooling, engineering platforms, and organizational practices rather than a single technology decision.

• The DX Core 4 still provides a useful foundation. Instead of abandoning existing engineering metrics, organizations should reinterpret them for AI-assisted development while continuing to focus on business outcomes rather than activity.

• Validation becomes more important as generation becomes easier. As AI produces more code, engineering organizations need stronger review, testing, and verification processes to ensure quality keeps pace with productivity.

Documentation becomes infrastructure for AI agents

• Documentation is no longer just for people. AI agents rely on high-quality documentation to understand systems, follow conventions, and complete work accurately, making documentation a core engineering asset rather than an afterthought.

• Not all documentation delivers equal value. Brian highlights that the biggest returns come from documenting information that helps agents understand systems, architecture, and engineering intent rather than simply producing more documentation.

• Capturing organizational knowledge improves both human and AI performance. Teams that make important context explicit reduce repeated questions, improve onboarding, and enable AI agents to work more effectively.

Engineering metrics need to evolve with AI

• Token usage is a cost metric, not a productivity metric. Brian cautions against treating token consumption as a measure of engineering effectiveness because it reflects AI usage rather than business value or software quality.

• PR throughput tells only part of the story. Larger pull requests and faster code generation may indicate increased AI adoption, but they can also increase review complexity and cognitive load if organizations measure throughput in isolation.

• Outcome metrics matter more than activity metrics. Justin emphasizes measuring whether engineering teams deliver value, improve quality, and create better developer experiences instead of rewarding raw AI utilization.

AI changes how engineering work feels—not just how it’s done

• AI pressure is contributing to burnout and cognitive overload. Brian describes growing pressure to move faster with AI while simultaneously reviewing larger code changes and maintaining confidence in increasingly AI-generated systems.

• Software engineering is much more than writing code. Even as AI accelerates code generation, engineers remain responsible for judgment, communication, system design, validation, and building trust in what gets shipped.

• The long-term challenge is balancing speed with confidence. Organizations that move faster than their ability to verify AI-generated work risk increasing technical debt, developer stress, and uncertainty rather than creating sustainable productivity gains.

In this episode, we cover:

(00:00) Intro

(01:26) Justin’s new role at DX

(03:53) What agent experience is and why engineering leaders should care

(08:55) How to improve agent experience at the platform level

(11:25) How agent experience and developer experience influence each other

(14:41) Preparing engineering teams for agentic work

(21:38) Why the DX Core 4 still matters in the age of AI

(27:23) What PR throughput actually measures

(32:47) The limits of token metrics

(37:10) What the data shows about documentation and developer experience

(39:32) Improving documentation for AI agents

(40:43) AI-washing, burnout, and cognitive overload

(45:35) Brian’s upcoming research on agent experience

Where to find Brian Houck:

• LinkedIn: https://www.linkedin.com/in/brianhouck

Where to find Justin Reock:

• LinkedIn: https://www.linkedin.com/in/justinreock

Referenced:

• DX Core 4 Productivity Framework

• The SPACE of AI | Queue

• Sara Chizari on LinkedIn

• The Middle Loop - Annie Vella

• gastownhall/gastown: Gas Town - multi-agent workspace manager · GitHub

• DORA, SPACE, and DevEx: Which framework should you use?

• AI Impact Report: Q1 2026

• AI-authored code has nearly doubled, but so has PR size

• EngThrive: Make It Fast and Easy to Do Great Work: Building a durable model for outcome-oriented engineering measurement

• Google’s Project Aristotle - Psychological Safety

• Zapier

Subscribe now

🗓 Join me and Brian Houck on July 23 for a readout of this report, where we’ll discuss new findings from DX’s data on AI tool usage, spend, and impact across 500+ organizations. Register here.

We are excited to announce our Q2 2026 AI impact report.

When we first began tracking the impact of AI on engineering teams, our primary goal was to measure AI cohorts against historical baselines to answer the question of what happens to software output after adoption. With industry-wide AI adoption exceeding 90%, comparing AI users against a non-user control group is no longer a viable measurement strategy.

Engineering leaders are now under immense pressure to justify exponentially-increasing AI budgets. The data from our Q2 report reveals that while AI is delivering objective gains in velocity, those gains are highly uneven.

Download the full analysis here.

In the new report, we’ve uncovered a number of critical trends, including:

1. Over 50% of code is now generated by AI. This metric has accelerated rapidly, increasing from 34% in Q1 2026 to 52% in Q2 2026. This steep trajectory indicates that once AI tools are deployed, the code they generate rapidly scales across codebases, frequently moving through reviews, dependencies, and shared workflows.

2. Quality may be declining. During the same period that AI adoption has increased, median pull request sizes have nearly doubled. Increases in PR size can serve as an early indicator of technical debt, as higher code volumes generally correlate with increased complexity and potential for bugs. This trend can also introduce additional friction in the review process, as more lines of code generated means more lines of code to review.

3. Some aspects of developer experience are declining. The Developer Experience Index (DXI) dropped from 67 to 65 over four quarters. AI is improving some aspects of the developer experience—documentation quality, code maintainability, onboarding speed—while creating new friction in others: larger PRs, slower reviews, less incremental delivery. In aggregate, the net effect is currently negative. Velocity metrics alone will tell you things are improving. Developer experience metrics will tell you whether that’s actually true.

4. AI is making codebases easier to understand, but it’s also making the code it generates harder to trust. The Q2 data highlights a striking divergence between two historically correlated software quality metrics. Specifically, from Q1 2026, Code Maintainability improved by 3.8%, whereas Change Confidence decreased by 6.1%. Code Maintainability indicates how easily developers can understand the codebase, while Change Confidence measures their trust that modifications won’t cause production failures. Traditionally, highly maintainable code results in higher confidence when making changes. However, this data reveals a new tension: although AI helps developers understand the code in front of them, they exhibit less trust in the code they are pushing to production.

5. Saved time isn’t converting into innovation. AI users are now saving an estimated 4 to 6 hours per week. However, the innovation ratio, defined as the percentage of time spent on building new features versus maintenance and overhead, has remained flat over the same period of study. This flat trend indicates that the time saved by AI is not currently converting into increased capacity for creating new value. Leaders should keep a close eye on this metric over time. Ideally, innovation ratio will increase as AI frees up engineers to work on more new features.

6. AI spend is accelerating faster than outcomes. Median quarterly organizational AI spend climbed from ~$1.5K to ~$44K over four quarters. Tech-sector spend increased nearly 28x. These numbers will draw scrutiny. Leaders who cannot connect this investment to downstream outcomes (feature velocity, innovation ratio, quality) may face increasingly difficult budget conversations in the back half of 2026.

What this means for leaders

The Q2 2026 data indicates that the industry is shifting from base AI deployment to evaluating concrete return on investment. As AI expenditures accelerate, engineering leaders must shift their focus from simply acquiring AI tools to optimizing the surrounding development pipelines and resolving systemic bottlenecks. To achieve true ROI, leaders must ensure that saved hours are reinvested into product innovation rather than absorbed by existing organizational friction.

To explore the full data and benchmark your team against 500+ organizations on measures of throughput, quality, and AI tooling cost, download the full report here.

That’s it for this week. Thanks for reading.

-Justin

Share

Subscribe now

🗓 Join DX Deputy CTO, Justin Reock and Distinguished Scientist, Brian Houck on July 23 for a readout of the Q2 State of AI Impact in Engineering Report. Register here.

I recently sat down with Thomas Newton, VP of Engineering at UKG, to discuss how his team built a system to guide AI adoption and assess whether it’s translating into meaningful engineering outcomes. What was particularly interesting was how UKG provides teams with a metrics dashboard that managers can use to have better coaching conversations, and ultimately help developers become more effective with AI tools.

For this week’s newsletter, Thomas is describing how their approach works.

Here’s Thomas.

Thomas: As a leader in our industry, we previously faced a question that most engineering organizations are grappling with right now: how do you measure whether AI usage is translating into more meaningful work shipped?

This was a leadership priority with one goal from the start: make sure that the right data, in a digestible format, landed with the people closest to the work—the managers.

The result is what we call the Manager AI Adoption Dashboard: a set of visuals that combine AI usage patterns, delivery outcomes, and spend into a single picture that engineering managers can use to coach their teams, guide adoption, and have better conversations about how work is getting done.

Starting with experimentation and adoption

Our journey using AI tools in product development started the way most do. We experimented with several tools (GitHub Copilot, Windsurf, and others) before making a significant push toward Claude.

Giving our engineers access to these tools was a good place to start, but our managers needed visibility. They could feel the productivity gains anecdotally, but the data was missing. We wanted to better understand where AI was helping teams, and which workflows were creating the most impact.

The shift to a consumption-based model made this need even more urgent. Unlike fixed-cost seat licenses, consumption pricing means the meter is always running. Leaders needed to understand not just whether teams were using AI, but whether the investment was producing returns.

Deciding what to measure

One of the earliest decisions we made was to anchor the dashboard around TrueThroughput, a metric developed by DX that goes beyond raw pull request counts to account for the relative complexity and size of work delivered.

TrueThroughput uses AI to classify the complexity of different tasks, giving you a size-adjusted throughput number. Think of it like a weighted GPA versus an unweighted GPA. Both are useful, but the weighted version tells you whether someone is delivering meaningful work or just merging a lot of five-second fixes.

Instead of indexing on consumption, the dashboard creates a balanced view of AI impact by correlating consistent AI usage, measured in days of use, with TrueThroughput. These metrics help us answer whether consistent use of AI is helping teams deliver more meaningful work.

The dashboard combines three signals:

• AI usage: How consistently someone is using AI tools in their workflow.

• TrueThroughput: The volume and complexity of work delivered.

• Spend: Awareness of AI investment and consumption patterns.

Inside the dashboard

The dashboard was built around a quadrant view. The horizontal axis tracks consistent days of AI use over a 30-day window: fewer than 15 days puts you on the left, more than 15 on the right. The vertical axis tracks TrueThroughput.

Example for illustration purpose only:

That creates four zones:

• Exploring (bottom left): Lower AI adoption, developing throughput. Teams discovering what problems AI might solve.

• Learning (bottom right): High AI adoption, building throughput. Expected pattern as teams explore different applications over 2-3 months.

• Efficient (top left): High throughput without heavy AI adoption. Some roles and workflows don’t need AI.

• Amplified (top right): High AI adoption, high throughput. Patterns here show which applications create real impact.

Each dot on the chart represents an individual. Bubble size reflects spend. And the views are drillable: from the entire organization down to a business unit, a team, and ultimately an individual manager’s direct reports.

What the data revealed

Since rolling out the dashboard, the results have been striking. Initially, as one might expect with newly introduced tooling, the bottom-left quadrant was packed, meaning that a large portion of our engineering organization hadn’t touched AI tools at all. Within four months,, that quadrant was nearly empty; less than 1% of the organization remained in the low-AI, high-throughput zone. It’s been exciting to watch the whole organization shift on this.

“Less than 1% of the organization remained in the low-AI, high-throughput zone.”

We also saw some things in the data that confirmed what we’d hypothesized. For example, the group seeing the biggest throughput gains were our senior and principal engineers, at ~20-30% above what other roles saw. That’s intuitive, but it was interesting to see it in the data. Senior engineers already have strong instincts for where AI can help and where it can’t. They were often quicker to identify high-leverage opportunities and incorporate the tools into existing workflows.

One pattern caught us off guard: managers and directors started writing code again. Our leadership population started doing more direct hands-on coding. They have more assistants, they can multitask better, whatever the reason, it’s a clear trend. That pattern extends beyond engineering: our product managers and designers have started leaning into AI tools too, getting comfortable with the terminal, creating digital artifacts, and contributing in ways that show up in delivery metrics. When we started this, it wasn’t what we anticipated, but it’s a trend that is now more commonly observed and discussed.

How managers use the dashboard

The real value of the dashboard is the quality of the conversations the data enables.

For managers, the first question often focused on adoption: “How do we get you from left to right?” Are you using it daily, is it part of your workflow, or are you still finding your footing with it? But over time, the conversation became less about adoption itself and more about impact. Where is AI helping? Which workflows are working well? Where is it creating leverage, and where is it not?

When a manager saw an engineer with high AI usage but flat throughput, the right response wasn’t to question the spend. It was to ask what they were working on; maybe they were ramping up on new AI workflows, or in a role where their AI-assisted work didn’t produce code commits. One example is heavy operation roles where productivity didn’t always result in commits into GitHub.

The best way to head off gaming or surveillance concerns is proactive communication. Don’t let people fill in the blanks on what the dashboards are for or why they exist. Be very clear. The dashboard is a conversation starter, not a performance management system. If teams interpret the dashboards as a tool for punishment or reward, they’ll have every incentive to game the numbers.

“The dashboard is a conversation starter, not a performance management system.”

At UKG, the framing has been consistent from day one: the dashboard exists to help managers empower their team, understand the work, understand how AI is being applied to that work, and help people lean into a new way of working.

Keeping AI spend in check without leading with cost

AI spend is visible on the dashboard, but it’s deliberately not the headline metric. It’s an awareness layer, something managers should be conscious of, not something that drives the conversation.

We use what I call “circuit breakers”; daily budget controls that flag when usage spikes above a threshold. But we haven’t yet landed on a firm benchmark for what reasonable per-engineer spend looks like.

The range is just too wide right now. Some teams are considering multi-agent orchestration and long-running loops that transform an entire codebase overnight. Others are using AI to finish a feature today. At this stage, it’s really hard to generalize.

I expect spending patterns will stabilize as the initial learning ramp flattens. Your first couple of prompts will probably be expensive. You’re still learning how to use the models correctly, you’re playing, you’re understanding how to use this radical new thing. But in as little as a few months, you will start to see averages form. We’re still in an experimentation phase, and for now, the approach is pragmatic: watch carefully, trust manager judgment, and make sure the investment is going toward the right outcomes.

What comes next

We’re already thinking about the limits of what the current dashboard captures. TrueThroughput is powerful for teams that ship code, but it misses productivity gains in operations, SRE, and other roles where AI is being used to correlate incidents, search logs, and accelerate incident resolution, work that never ends up in a pull request.

We have teams where an engineer uses AI to search past incidents during a live outage and correlate similar patterns to get to a resolution faster. That’s enormously productive, but it won’t show up in throughput. We’re trying to think through what the next level of digital footprints looks like, the metrics that capture the full picture of AI-enabled productivity, not just the code-commit slice of it.

The dashboard is a living system designed to evolve as we learn more about what effective AI-enabled engineering looks like.

Final thoughts:

For engineering leaders at other organizations who haven’t yet started measuring AI adoption, my advice is simple: Measure something. Metrics you have access to might differ, but collect some form of data, figure out what makes sense for your organization, and don’t make it a binary decision based on the data itself. The data should enable leaders to have further conversations.

That’s an important takeaway: The tools are powerful, and the data is illuminating, but the transformation happens in the conversations between managers and their teams.

If you have questions about UKG’s approach, or just want to hear more from Thomas, make sure to follow or connect with him on LinkedIn.

That’s it for this week. Thanks for reading.

-Abi

Share

In this episode of the Engineering Enablement podcast, I sit down with Sejal Amin, Chief Technology Officer at Priceline, and Pedro Gutierrez, Senior Director of Software Engineering, to discuss how Priceline adopted a product operating model and the role developer experience played in making that transformation successful.

We explore why the company moved away from a project-based approach, how DX metrics and developer feedback helped uncover organizational bottlenecks, and why a phased rollout, clear communication, and empowered engineering managers were critical to building trust and improving developer experience. We also discuss creating a dedicated developer experience team, lessons learned throughout the transformation, and how Priceline’s product operating model has helped the organization adapt to AI-driven software development.

Some takeaways:

Developer experience as a driver of organizational change

• Developer experience data can reveal organizational problems that traditional engineering metrics miss. At Priceline, DX signals uncovered organizational bottlenecks—including handoffs, dependencies, and team friction—that ultimately led the company to adopt a product operating model.

• Developer experience should be treated as a strategic capability, not just an engineering metric. Rather than measuring developer satisfaction in isolation, Priceline used DX insights to guide structural changes that improved autonomy, delivery, and engineering culture.

Adopting a product operating model

• Reducing dependencies gives teams greater ownership. Priceline shifted from a project-based organization to cross-functional product teams, reducing handoffs and giving teams the people and capabilities needed to own outcomes end to end.

• Autonomy requires visibility into team health. DX metrics gave engineering managers a clear view of the obstacles affecting their teams, allowing them to improve local workflows while staying aligned with broader organizational goals.

Turning developer feedback into action

• Developer experience surveys should lead to action—not just measurement. Managers reviewed survey results, completed a triage process, created quarterly action plans, and measured whether those improvements had an impact in the next survey cycle.

• Small workflow improvements can have an outsized impact. DX data helped teams reclaim focus time, identify tooling regressions after migrations, surface cross-team dependencies, and address day-to-day friction before it became systemic.

Building trust in developer experience metrics

• Clear communication is essential for adoption. Leaders consistently reinforced that DX metrics existed to improve teams rather than evaluate individuals, helping build confidence in the process from the outset.

• Trust grows when developers see meaningful change. Acting on feedback quarter after quarter encouraged greater participation, strengthened psychological safety, and made developer experience part of the organization’s culture.

The evolving role of engineering managers

• Engineering managers became owners of developer experience. Managers were expected to understand DX data, improve their team’s DXI each quarter, and make developer experience part of their regular operating rhythm.

• Developer experience became part of everyday engineering leadership. DX metrics were discussed openly in all-hands meetings and other forums, making developer experience a visible measure of organizational health rather than a one-time initiative.

Preparing engineering organizations for AI

• AI changes where bottlenecks occur—not whether they exist. As AI accelerated code generation, Priceline used its product operating model and developer experience data to identify where constraints had shifted and respond accordingly.

• A strong operating model helps organizations adapt to AI. Autonomous teams, continuous measurement, and visibility into developer workflows allowed Priceline to embrace AI while continuing to improve flow across the software development lifecycle.

In this episode, we cover:

(00:00) Intro

(01:07) Meet Sejal Amin and Pedro Gutierrez

(01:47) How Priceline’s developer experience journey began

(04:55) Lessons from Priceline’s first developer experience surveys

(06:55) How DX improved Priceline’s developer experience surveys

(09:47) Identifying the causes of organizational slowness

(12:33) How the product operating model changed the way Priceline works

(14:10) Priceline’s phased rollout with DX

(18:14) How DX insights drove organizational changes

(19:33) Why Priceline improved developer experience before org change was complete

(22:18) How clear communication builds trust

(24:25) Early results from Priceline’s Core Four

(25:38) Creating a culture of continuous feedback to build trust

(27:40) What has changed in the engineering manager role

(30:10) Resources for learning about the product operating model

(32:40) What Pedro learned from implementing DX

(34:51) The developer experience team

(35:59) How AI tools have impacted Priceline’s teams

(37:20) How the product operating model supports AI-driven development

(39:13) Final advice for engineering leaders

Where to find Sejal Amin:

• LinkedIn: https://www.linkedin.com/in/sejal-amin

Where to find Pedro Gutierrez:

• LinkedIn: https://www.linkedin.com/in/pedro-gutierrez-b6605422

Where to find Justin Reock:

• LinkedIn: https://www.linkedin.com/in/justinreock

Referenced:

• Measuring developer productivity with the DX Core 4

• Transformed: Moving to the Product Operating Model (Silicon Valley Product Group)

• Team Topologies

• Flow Framework

• Project to Product: How to Survive and Thrive in the Age of Digital Disruption with the Flow Framework

Subscribe now

🗓 Join me on July 23 for a readout of the upcoming State of AI Impact in Engineering: Q2 Report. We’ll discuss new findings from DX’s data on AI tool usage, spend, and impact across 500+ organizations. Register here.

Every once in a while, several independent papers arrive at roughly the same time and collectively tell a bigger story than any one of them does alone. This week, I’m sharing five recent papers that have significantly influenced how I’m thinking about AI and software engineering.

Each paper tackles a different question. Some measure the productivity impact of AI coding assistants. Others examine how those gains propagate through the software delivery process, explore what developers actually want from future AI systems, or reconsider the kinds of debt we should be paying attention to in an AI-assisted world.

Despite coming from different research groups and using very different methodologies, they all seem to be converging on the same underlying story.

AI is compressing the upstream work of software engineering. The more I sat with these papers, the less I found myself asking, “Is AI making developers faster?” and the more I found myself asking, “What happens after the code is written?” Are we actually shipping more value? Where do the new bottlenecks emerge? And what are the costs if understanding can’t keep pace with generation?

After reading these five papers, I came away with one overarching conclusion: we’re generating code faster than we’re generating the systems needed to safely understand, verify, and deliver it.

A quick note on disclosure: three of these papers come from people I know and work with extensively. None of the papers are mine.

Here they are, in the order I’d recommend reading them.

1. GitHub Copilot and Developer Productivity

Paper: Heilman, A., Kyllo, A., Murphy-Hill, E. GitHub Copilot and Developer Productivity: An Observational Dose-Response Analysis.

The first paper I want to highlight tackles the familiar question of whether GitHub Copilot makes developers more productive, but it does so with one of the more clever research designs I’ve seen.

Rather than simply comparing Copilot users to non-users (which are getting harder and harder to find), the authors control for Active Coding Time (i.e., how much time developers spend actively engaging with development tools) and examine how productivity changes within the same engineer over 43 weeks across a population of 16,223 developers.

The payoff of this design is that it compares engineers to themselves rather than to one another. Using that approach, the authors found that weeks with the highest Copilot usage were associated with ~40% more completed PRs per hour of coding time than weeks with no usage.

The relationship showed a clear dose-response pattern (a way to do a causal analysis, once everyone is already using the tools). More Copilot engagement was associated with more PR throughput, although the gains appeared to level off at very high usage.

The authors ran seven robustness and falsification tests to rule out alternative explanations (team-level effects, generic AI engagement, PR slicing, shifts toward easier work). The positive association remained remarkably consistent.

Interestingly, the gains were not concentrated in tiny PRs. The strongest effects were observed for larger PRs (7+ files), arguing against the idea that developers are simply breaking work into smaller units.

It’s a thoughtful analysis and shows that we’re not just coding more, we’re increasing coding efficiency as well. These findings anchor many of the studies that follow in this roundup.

2. Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools

Paper: Demirer, M., Musolff, L., Yang, L. Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools.

The next paper I’m highlighting was published by the National Bureau of Economic Research. It analyzes AI adoption across 100,000+ GitHub developers and asks a more nuanced question than Heilman’s: when AI makes individual coding steps faster, how much of that gain actually survives all the way to shipped software?

The authors examine how AI productivity gains propagate through a hierarchy of software development: lines of code → files → commits → pull requests → projects/repos → releases.

They found that AI is clearly increasing coding activity, and the gains grow with each generation of tools. They estimate roughly +40% more commits from autocomplete, growing to +140% from interactive coding agents, and finally +180% from autonomous agents.

However, those gains fall off significantly as work moves through the software delivery process. The largest effects are seen in code generation, but smaller effects appear in repos touched, small still in releases shipped, and ultimately software consumed by users. Even with very large increases in coding activity, the effect on shipped software is much smaller, topping out at roughly +30% more releases. This is illustrated in figure 2 below.

One of the findings I found most interesting is that they estimate a low elasticity of substitution (~0.25) between AI-generated output and human effort. That’s an economics concept that measures how replaceable human work is with AI output. As a methodology nerd, and someone with an economics degree, I found this particularly clever — they infer this elasticity from how AI productivity gains attenuate across the delivery process. Their estimate suggests AI and human work are still largely complements rather than substitutes, with substantial human effort still required to review, integrate, validate, and ship software.

One open question is whether the observed fall-off through the delivery process is some fundamental limit of software engineering, or simply the fact that organizations have not yet adapted their processes to an agentic world.

If Heilman tells you Copilot is making engineers measurably faster, this paper asks the harder question: faster at what, exactly?

3. The Impact of AI Coding Assistants on Software Engineering

Paper: Vella, A., Blincoe, K. The Impact of AI Coding Assistants on Software Engineering: A Longitudinal Study.

The next study I want to highlight is unique because it isn’t just a snapshot in time, it’s a six-month longitudinal study of 95 professional software engineers. It also calls into question a relationship that we’ve long believed to be a bedrock of developer experience.

The study was done using two questionnaires six months apart, mixed-methods, with reflexive thematic analysis on the open-ended responses.

Vella found that productivity perceptions were stable and strongly positive over time. 84% of study participants reported improvement at both time points. Consistent with the first two studies in this round-up, the story of accelerated throughput is real and persistent.

The really striking finding is what the authors call the productivity-experience paradox. Among the matched cohort, the share of engineers reporting worse DevEx on at least one dimension nearly doubled in just six months, from 14% to 27%. Flow state was the most vulnerable; cognitive load eroded modestly; feedback loops actually improved.

More importantly: while the cross-sectional correlations between DevEx and productivity were strong, the change scores didn’t correlate. Productivity and developer experience appear to be decoupling over time in AI-assisted workflows. For those of us who’ve spent years working with the SPACE and DevEx frameworks, that’s worth sitting with.

While this study didn’t have a particularly large population, the findings were significant and rigorously validated, proving that a study doesn’t have to be massive if the strength of results is strong enough. This longitudinal design is rare and valuable, and the productivity-experience decoupling is the kind of finding worth replicating in larger populations.

4. To Copilot and Beyond: 22 AI Systems Developers Want Built

Paper: Choudhuri, R., Badea, C., Bird, C., Butler, J., DeLine, R., Houck, B. AI Where It Matters: Where, Why, and How Developers Want AI Support in Daily Work.

Last year I published a paper called AI Where It Matters, and my co-authors ended up writing a 2nd paper based on the original survey responses (860 Microsoft developers across roles, domains, and geographies). The paper outlines a roadmap of 22 AI tools that developers want beyond just code generation, centered around a concept they call “bounded delegation.” A lot of this echoes what the rest of this round-up is circling:

• The “right-shift” burden. Because AI is speeding up code generation, it’s creating a massive bottleneck downstream. Devs are getting flooded with more code to review, more production incidents to debug, and documentation that falls behind faster than ever.

• The move to verification. Developers don’t want more code-generation assistants; they want AI embedded into verification tasks — tools that automatically assemble log/trace “case files” for on-call incidents, PR reviewers that catch complex business logic flaws before human review, change-aware test generation that knows which assertions actually matter.

• “Bounded delegation.” There is a strict boundary around where developers want AI to stop. Developers want AI to absorb the tedious “assembly work” surrounding their craft (updating docs, writing edge-case unit tests), but never the core logic, architecture, or critical decision-making. Notably, developers drew this line even for tasks they acknowledged AI could plausibly handle — suggesting it’s not just about capability gaps and won’t move just because models improve.

• Four non-negotiable guardrails. For future AI tools to be adopted, developers say they must enforce explicit authority scoping (no auto-approvals), clear data provenance, explicit uncertainty signaling (the AI must admit when it doesn’t know something), and least-privilege security access.

You can check out both papers and an interactive website here: aka.ms/ai-where-it-matters

5. From Technical Debt to Cognitive and Intent Debt

Paper: Storey, M. From Technical Debt to Cognitive and Intent Debt: Rethinking software health in the age of AI

I’ve saved this for last because I think this is the most important paper I’ve read in a long time. Margaret-Anne Storey makes a generational argument: the metaphor we’ve used for decades to think about software health—technical debt—is no longer sufficient. AI is reducing technical debt (through refactoring, test generation, automated review) while quietly accelerating the accumulation of two other forms of debt that matter more in this era.

Technical debt lives in code. It accumulates when implementation decisions compromise future changeability. AI is genuinely helping here.

Cognitive debt lives in people. It accumulates when a team’s shared understanding of a system erodes faster than it’s replenished. When AI generates the code, developers may accept it without building the same mental model they would have built by writing it themselves. Multiply that across a team and over time, and you get “an accumulation of not knowing.”

Intent debt lives in artifacts. It accumulates when the goals, constraints, and rationale that guide a system—the things both humans and AI agents need to work safely—are unclear, unwritten, or forgotten. As more development is AI-assisted, intent debt becomes a first-order constraint on what AI can actually do for you.

The three debts interact and compound. Intent debt causes cognitive debt; cognitive debt causes technical debt; technical debt amplifies cognitive debt. Managing software system health requires attention to all three layers, not just the one easiest to measure.

The four practical implications Storey draws are worth reading in full, but the headline is: treat understanding as a deliverable. Just as working code is a product of software development, shared understanding should be treated as a first-class deliverable, not something that happens as a side effect of writing code.

Final thoughts

Read together, these five papers say something stronger than any of them say individually. AI is genuinely making code generation faster, and the per-engineer efficiency gains are real (Heilman). But those gains don’t survive the trip to shipped software at anywhere near the same magnitude (Demirer). The bottleneck has moved downstream to review, integration, verification, and understanding. Developers feel it, they’re explicitly asking for tools to address those bottlenecks while refusing to delegate the parts of the job they consider craft (Choudhuri). The lived experience of working this way is more uneven than the productivity numbers suggest, with flow and cognitive load eroding even as throughput holds (Vella). And the deepest cost may be one we don’t yet measure: the slow erosion of shared understanding, which is what makes any system safe to change (Storey).

The bottleneck has moved. Our tools, metrics, and team designs haven’t moved with it yet. That’s where the next several years of work in our field are going to happen.

That’s it for this week. And make sure to sign up for my upcoming live research readout covering new findings on AI’s impact, where we’ll discuss data from both DX and the broader industry.

-Brian

Share

Subscribe now

🗓 Join me on July 23 for a readout of the upcoming State of AI Impact in Engineering: Q2 Report. We’ll discuss new findings from DX’s data on AI tool usage, spend, and impact across 500+ organizations. Register here.

I’ve spent a lot of time over the years thinking about developer productivity metrics. The longer I do this work, the more convinced I become that the hardest part isn’t measuring software engineering. It’s improving it.

That’s where I think the best measurement systems distinguish themselves. They don’t just tell you whether things are getting better or worse. They guide you to interventions that actually improve how developers work. If your dashboard isn’t changing decisions, it’s not creating value.

That idea sits at the heart of EngThrive: Make It Fast and Easy to Do Great Work, a paper I recently co-authored with Tim Bozarth, David Liu, and Dean Carignan. The paper describes the measurement and improvement system I helped build at Microsoft, but what has stuck with me most aren’t the dashboards or the framework. They’re the stories.

One team intentionally “gamed” a productivity metric and improved onboarding for an entire year. Another protected developers’ focus time and discovered that the biggest gains came from work they weren’t even trying to improve. A third gave every developer two unexpected days off and found that the lost “productivity” disappeared within weeks while the wellbeing benefits lasted for months.

Three different organizations. Three different problems. Three different interventions. Yet they all taught the same lesson: the best measurement systems don’t just tell you what’s happening. They help you figure out what to do next.

Before we get to those stories, though, it’s worth remembering how easy it is to measure confidently and be wrong.

In the first two months of mandatory remote work at Microsoft in early 2020, pull requests per developer jumped more than 20%, and the company’s stock price rose more than 15%. By those measures, things looked great. During that same period, however, 78% of developers reported feeling burned out. Three signals from the same quarter, pointing in two different directions. Any one of them on its own would have told a confident but completely misleading story.

That is the trap EngThrive was built to avoid. Organizations often measure activities—pull requests, commits, tasks completed—and quietly treat them as proxies for outcomes like delivery speed, software quality, or developer effectiveness. The two are not the same.

EngThrive instead organizes measurement around outcome dimensions like Speed, Ease, and Quality, with Thriving serving as a guardrail: if an intervention makes developers faster but leaves them burned out, we don’t consider it a success. We use a handful of outcome-oriented North Star metrics supported by diagnostic metrics that help explain why those outcomes move.

With that frame in mind, here are three case studies that fundamentally changed how I think about measuring, and improving, developer productivity.

Case one: Improving one outcome changed four

“Too many meetings” is one of the most frequently cited workplace challenges among software engineers. So in late 2025, Microsoft’s CoreAI organization launched an initiative to protect developers’ focus time. Rather than simply banning meetings, leaders set an explicit target: lift the bottom 20% of developers to at least 25 hours of focus time per week.

To achieve this goal, teams did a handful of sensible things. They removed low-quality meetings, they clustered meetings together to form larger uninterrupted blocks of time, and they explicitly blocked time on their calendars to do focus work.

Within eight weeks, the results showed up across multiple dimensions. Focus time increased by 2.1 hours per developer per week, roughly twice the improvement seen in the control group. Bad Developer Days, a composite measure of daily developer friction, fell by 25%. PR velocity increased by 13%, about four times the control group. Taken together, the productivity gains were roughly equivalent to adding the output of 350 developers.

What surprised me wasn’t that focus time improved. It was that improving focus time seemed to improve things we weren’t directly trying to change.

Only about half of the reduction in Bad Developer Days could be explained by the additional focus time itself. Teams appeared to be using their newly protected capacity to pay down technical debt and eliminate other sources of friction that had been generating bad days in the first place. Creating focus time didn’t just help developers concentrate. It gave teams the space to improve the system that had been interrupting them.

That’s exactly why I think outcome-oriented measurement matters. If we had only measured focus time, we would have concluded that developers gained two extra hours each week. Looking across multiple dimensions revealed something much more interesting: the intervention triggered improvements well beyond its original goal.

One team lead summarized the lesson better than I could:

Metrics led to questions, questions led to improvements, improvements reinforced the metric’s value.

Case two: When gaming the metric is the right answer

One of the most common objections to productivity metrics is that people will game them. I worry about that too, but I increasingly think it’s also one of the best tests of whether a metric is well designed. The best metrics are ones where gaming them is indistinguishable from genuine improvement.

Time-to-First-PR is my favorite example.

One organization of roughly 4,000 developers decided to “game” the metric on purpose by assigning every new hire a trivial pull request on their first day. As a gaming exercise, it worked exactly as intended. Time-to-First-PR improved by 30%.

The surprise came later.

Those same developers went on to complete 23% more pull requests over their first year than the control group. Interviews explained why. The first pull request was never really about the code. It was about setting up the development environment, learning the team’s tools and review process, and becoming comfortable contributing. By forcing all of that to happen in the first week, the organization didn’t just improve a metric. It accelerated onboarding.

A separate AI-assisted onboarding tool, FirstMate, arrived at the same conclusion from a different direction. By automating environment setup and helping new hires navigate an unfamiliar codebase, it reduced Time-to-First-PR by 65%.

That’s the lesson I keep coming back to:

A well-designed metric shouldn’t be easy to game. It should be difficult to improve without doing something genuinely valuable. When that happens, gaming the metric and improving the system become the same thing.

Case three: A cost that turned out to be free

During the burnout crisis of 2020, one organization tried something that looked reckless on a Speed dashboard: it gave every developer two unexpected days off, called Health Days.

At first, the metric behaved exactly as you would expect. Pull request output dropped during those two days.

Then something surprising happened.

Within two weeks, the “lost” pull requests had all been made up. The apparent productivity cost disappeared. The burnout relief, however, lasted another 14 weeks.

To me, this is the clearest example of why we treat Thriving as a guardrail rather than just another metric. If we had only looked at Speed, Health Days would have appeared to be a costly intervention and might never have been attempted again. Looking across multiple dimensions told a completely different story. The intervention was effectively free from a Speed perspective while delivering a sustained improvement in developer wellbeing.

I’ll also acknowledge an important limitation. This wasn’t a controlled experiment; it was one organization’s experience. But it’s consistent with a pattern we’ve seen repeatedly: leaders often assume wellbeing and productivity exist in tension, when in practice the trade-off is frequently much smaller than expected (or doesn’t exist at all).

The lesson isn’t that every organization should schedule Health Days. It’s that looking across outcome dimensions gives leaders permission to try interventions that a single productivity metric would immediately reject.

Why this matters for engineering leaders

The common thread across all three stories isn’t focus time, onboarding, or Health Days. It’s that none of those interventions came from optimizing a single activity metric. They came from measuring outcomes, looking across dimensions, and using supporting metrics to understand why those outcomes changed.

That’s the distinction I hope readers take away from the EngThrive work. A good measurement system isn’t just a reporting system. It’s a learning system. It doesn’t simply tell leaders whether things are getting better or worse. It helps them discover interventions they wouldn’t have tried otherwise, understand why they worked, and build confidence in repeating them.

Activity metrics still have an important role to play, but not as the destination. They’re clues. They help explain why an outcome changed, not whether it mattered in the first place.

Ultimately, I think that’s the shift engineering organizations need to make. Stop asking, “What should we measure?” Start asking, “What decisions are we trying to make, and what measurements would help us make them better?” The metrics should serve the intervention, not become the intervention.

None of this is the work of one person, or even four. EngThrive is the product of a large team that has spent years building the platform, the research, the surveys, and the discipline behind these results. If these stories are useful, the credit belongs to them.

This week’s featured DevProd job openings. See more open roles here.

• Ashby is hiring an Staff Platform Engineer | Remote

• Carta is hiring a Senior Software Engineer II, Developer Experience | Santa Clara, CA; San Francisco, CA; New York, NY

• Figma is hiring a Staff Software Engineer, Developer Experience | Remote; US

• GM is hiring a Principal Software Engineer, Developer Experience | Austin, Texas

• Morgan Stanley is hiring an AI Platform Engineer - Vice President | New York

• Notion is hiring a Software Engineer, Developer Experience | Hybrid; Hyderabad, India

• Vercel is hiring a Sr. Engineering Manager, Platform | New York City, San Francisco

That’s it for this week. Thanks for reading.

-Brian

Share