<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd" xmlns:googleplay="http://www.google.com/schemas/play-podcasts/1.0"><channel><title><![CDATA[Engineering Enablement]]></title><description><![CDATA[Research and perspectives on developer productivity. ]]></description><link>https://newsletter.getdx.com</link><image><url>https://substackcdn.com/image/fetch/$s_!upog!,w_256,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fabb0ddc7-a50e-4130-b210-b90b7896ba15_110x106.png</url><title>Engineering Enablement</title><link>https://newsletter.getdx.com</link></image><generator>Substack</generator><lastBuildDate>Sat, 21 Nov 2026 21:07:46 GMT</lastBuildDate><atom:link href="https://newsletter.getdx.com/feed" rel="self" type="application/rss+xml"/><copyright><![CDATA[Abi Noda]]></copyright><language><![CDATA[en]]></language><webMaster><![CDATA[abinoda@substack.com]]></webMaster><itunes:owner><itunes:email><![CDATA[abinoda@substack.com]]></itunes:email><itunes:name><![CDATA[Abi Noda]]></itunes:name></itunes:owner><itunes:author><![CDATA[Abi Noda]]></itunes:author><googleplay:owner><![CDATA[abinoda@substack.com]]></googleplay:owner><googleplay:email><![CDATA[abinoda@substack.com]]></googleplay:email><googleplay:author><![CDATA[Abi Noda]]></googleplay:author><itunes:block><![CDATA[Yes]]></itunes:block><item><title><![CDATA[What to do when developers say no to AI]]></title><description><![CDATA[The four types of no&#8217;s you get from developers, and how to respond as a leader.]]></description><link>https://newsletter.getdx.com/p/what-to-do-when-developers-say-no</link><guid isPermaLink="false">https://newsletter.getdx.com/p/what-to-do-when-developers-say-no</guid><dc:creator><![CDATA[Eirini Kalliamvakou]]></dc:creator><pubDate>Wed, 18 Nov 2026 10:41:42 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/a1c7a224-7f1e-4467-a8b5-fe390cacfdbe_2400x1260.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><em><strong>Welcome to the latest issue of Engineering Enablement,</strong> a weekly newsletter sharing research and perspectives on developer productivity.</em></p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.getdx.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe now&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.getdx.com/subscribe?"><span>Subscribe now</span></a></p><p>Our latest webinar with Snowflake&#8217;s Nathan Hunt explores where AI-generated capacity may be going and how to reinvest it in higher-value work. Watch the discussion<strong> <a href="https://getdx.com/webinar/ai-generated-capacity-where-does-the-time-go-us-emea/">here.</a></strong></p><div><hr></div><p><strong>Scott:</strong> Software developer Brett Chalupa <a href="https://www.youtube.com/watch?v=2ZU3j4GQ4K8">went viral three months ago for a video</a> where he explained why he refuses to use AI in any capacity. &#8220;Whatever comes of this decision &#8211; if I get fired, if I can&#8217;t find a job, if my career as I know it is over &#8211; so be it,&#8221; he said.</p><p><a href="https://leaddev.com/ai/meet-the-developers-rejecting-ai">Chalupa is far from alone</a>. There are plenty of developers who would rather risk their career than allow AI models to shape the way they work. This puts leaders in a tricky position. With adoption already near universal, and optimism that AI tools can unlock new levels of productivity and creativity from software developers, industry hype is at an all-time high.</p><p>But what is driving that refusal? This week, my colleague <span class="mention-wrap" data-attrs="{&quot;name&quot;:&quot;Eirini Kalliamvakou&quot;,&quot;id&quot;:554376261,&quot;type&quot;:&quot;user&quot;,&quot;url&quot;:null,&quot;photo_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5c463cb7-9819-4cdf-a4d4-2c3540b9c64d_2083x2083.jpeg&quot;,&quot;uuid&quot;:&quot;73763b5d-9ece-443d-bb1b-f7c4f184d64d&quot;}" data-component-name="MentionToDOM"></span> lays out the four types of &#8220;no&#8221; engineering leaders often hear, and how to respond in each case.</p><div><hr></div><p><strong>Eirini:</strong> I joined DX recently as an Applied Scientist, after many years of studying how developers work, most recently at GitHub. What I enjoy most is the moment a pattern starts to make sense: why developers do what they do, and what it means for the people leading them. Right now developers&#8217; work is changing faster than I&#8217;ve ever seen, which makes this a great time to be doing research, and I&#8217;m excited to share what I&#8217;m learning here, early and as I go.</p><p>For my first issue I wanted to write about the developers saying no to AI, and why their reluctance is worth listening to.</p><p>The last time DX&#8217;s data looked at AI adoption across hundreds of organizations, it had hit an <a href="https://getdx.com/report/ai-assisted-engineering-q1-impact-report/">all-time high at 93%</a>. But that doesn&#8217;t mean that all developers are on board.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Adhl!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F007d4367-5a3e-4076-b53e-f08302e07558_4300x3184.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Adhl!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F007d4367-5a3e-4076-b53e-f08302e07558_4300x3184.jpeg 424w, https://substackcdn.com/image/fetch/$s_!Adhl!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F007d4367-5a3e-4076-b53e-f08302e07558_4300x3184.jpeg 848w, https://substackcdn.com/image/fetch/$s_!Adhl!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F007d4367-5a3e-4076-b53e-f08302e07558_4300x3184.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!Adhl!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F007d4367-5a3e-4076-b53e-f08302e07558_4300x3184.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Adhl!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F007d4367-5a3e-4076-b53e-f08302e07558_4300x3184.jpeg" width="1456" height="1078" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/007d4367-5a3e-4076-b53e-f08302e07558_4300x3184.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1078,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:837217,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://newsletter.getdx.com/i/225327042?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F007d4367-5a3e-4076-b53e-f08302e07558_4300x3184.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!Adhl!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F007d4367-5a3e-4076-b53e-f08302e07558_4300x3184.jpeg 424w, https://substackcdn.com/image/fetch/$s_!Adhl!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F007d4367-5a3e-4076-b53e-f08302e07558_4300x3184.jpeg 848w, https://substackcdn.com/image/fetch/$s_!Adhl!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F007d4367-5a3e-4076-b53e-f08302e07558_4300x3184.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!Adhl!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F007d4367-5a3e-4076-b53e-f08302e07558_4300x3184.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>That reluctance &#8211; or even outright refusal &#8211; could even be coming from developers who use AI every day. It may be shadow AI usage, holding back on using AI in particular work, giving up on it partway through a task, or weighing it up and deciding the trade doesn&#8217;t add up for that piece of work.</p><p>When adoption is what we track, a developer who says no can look like the last part of the curve still to close. Everett Rogers, who spent decades studying <a href="https://en.wikipedia.org/wiki/Diffusion_of_innovations">how innovations spread</a>, has warned about exactly this reading. He called it <strong>individual-blame bias</strong>: when people don&#8217;t adopt, we look for what&#8217;s wrong with them, and rarely ask what the innovation, or the system around it, failed to give them. That second question is where a no becomes useful for a leader.</p><h2>What a no can tell you</h2><p>Adoption might not mean impact, but it still matters at the start of an AI transformation. For that reason alone, a no that stalls adoption is worth understanding.</p><p>In an AI rollout, the organization adopts a tool, while each developer changes how they work: how they learn it, how much they check, where they let it in and where they keep it out. Early on, it makes sense for people to work much of this out for themselves. In a pilot, relying on intuition and existing experience is part of the point.</p><p>As the rollout widens, that working-out is supposed to move into the organization&#8217;s processes, systems and workflows. When it doesn&#8217;t, each developer keeps figuring it out alone, and some of them, quite reasonably, start to say no.</p><p>For engineering leaders looking to scale AI impact beyond just individual use, a no can tell you where to focus next.</p><p>One exclusion before going further. Some developers refuse to engage with AI on principle, for ethical or environmental reasons. They&#8217;re entitled to that, and there&#8217;s nothing in it for a leader to fix, so everything below is about everyone else.</p><h2>Four kinds of no</h2><p>Through my own research, recent studies of developers <a href="https://arxiv.org/abs/2607.00533">at Microsoft</a> and <a href="https://drops.dagstuhl.de/storage/00lipics/lipics-vol394-esem2026/LIPIcs.ESEM.2026.9/LIPIcs.ESEM.2026.9.pdf">across 16 organizations</a>, and developers&#8217; own accounts, the no&#8217;s tend to come down to four things. On a dashboard they will all look the same, but up close they are very different.</p><ol><li><p><strong>&#8220;I don&#8217;t know how, and there&#8217;s no room to figure it out while I do my other work.&#8221;</strong> The early stretch with AI tools is messy, and it resets when a new generation of tools shows up. Results are uneven, the tools interrupt the way people are used to working, and nobody starts out knowing what good instructions or good context look like. <a href="https://doi.org/10.1145/3807961">The developers we interviewed who became highly fluent</a> learned it by getting burnt and adjusting, and every one of them started out skeptical. What carried them through was urgency, and some fear of being left behind. Part of not knowing how to work with AI is also not yet seeing what a developer&#8217;s work becomes once an agent does some of what they used to do. Many developers try, find it slower and more frustrating than the way they already know, and go back to what works. That&#8217;s a sensible response to a learning curve that doesn&#8217;t come with time set aside for it.</p></li><li><p><strong>&#8220;I&#8217;ve weighed it, and it doesn&#8217;t add up.&#8221;</strong> This no often comes from people who use AI well. They see real gains, and they also see where the costs are: more output to review, skills they practise less, less of the work they enjoy, and expectations that rise because the tools exist. So they decide, task by task, where the trade off is worth it and where it isn&#8217;t. Of the four types of no, this is the one most easily mistaken for resistance, and it is the most deliberate. These developers are already fluent, and their sense of where it pays off and where it doesn&#8217;t is something the organization can learn from.</p></li><li><p><strong>&#8220;I can&#8217;t stand behind it.&#8221;</strong> Developers&#8217; names are on what ships, so they weigh how risky a change is, how reversible, and how much checking it will take. When they hold back, it is usually because they can&#8217;t check what was produced well enough to own it, or because there is more of it than they can review. <a href="https://survey.stackoverflow.co/2026/ai/data">Developers tend to trust AI output when it is easy to verify</a>, and much of the work that matters has edge cases and exceptions. The instinct here is right, and it gets stronger as more of the code in front of them was written by something else.</p></li><li><p><strong>&#8220;It was decided without me.&#8221;</strong> Sometimes the no is about how AI arrived. Which tool, how much use is expected, which work it&#8217;s required for. At the far end are usage targets and leaderboards. Being told which tool to use can feel like being told how to do your job, and much of this no would show up with any mandate delivered that way, AI or not.</p></li></ol><p>Most reluctance is a mix of the above, and the same developer can give a different reason for different work. Someone who has never started and someone who uses AI every day can give the same no. So it is more useful to look at the reasons than to sort the people, and only the people saying no can tell you which reasons are theirs.</p><h2>The layer underneath</h2><p>What strikes me when developers explain a no is how rarely they are talking about the tool. They talk about what they own, what they are good at, and who decides how their work gets done. In my research for <a href="https://doi.org/10.1145/3807961">The AI-Native Developer</a>, the developers deciding where AI was applied in their work were protecting the parts that made it worth doing.</p><p>That is close to what motivation research has described for a long time. <a href="https://doi.org/10.1037/0003-066X.55.1.68">Self-Determination Theory</a>, one of the most studied accounts of motivation at work, describes the three needs people have: autonomy, competence, and relatedness. When those needs are met, people commit to a change because they see the point of it. When they aren&#8217;t, you can still get compliance, but not the kind of commitment the later benefits depend on.</p><p>We already talk about AI readiness in rollouts, usually in terms of systems, data, procurement and security. Each of those can stall a transformation, and nobody rolls out without checking them. I think motivation belongs on that list. It&#8217;s harder to see, and it can block progress too. But notice how we respond. When a system isn&#8217;t ready, we investigate. When a developer says no, we are more likely to push.</p><p>You can hear the first two needs in what developers say: &#8220;it was decided without me&#8221; is about having a say in how you work (autonomy), and &#8220;I don&#8217;t know how&#8221; is about being good at the work and staying good at it (competence). The third need, connection to the people around you (relatedness), rarely comes up as a reason, but it shows up elsewhere: developers describing AI as making their work easier and lonelier. That deserves its own investigation.</p><h2>The gap behind each no</h2><p>Each kind of no can point to a gap: something the developer is compensating for because the organization hasn&#8217;t put it in place yet. As long as individuals are compensating, whatever they gain mostly stays with them, and it&#8217;s hard to see at the organization level. Here is where you can start looking for each kind of no.</p><p><strong>Room to learn.</strong> Learning how to leverage AI in daily work is something that needs to happen regularly, alongside the evolution of tools and the emergence of new practices. When learning is left to each person, the organization pays for it many times over. Every developer faces each challenge separately, and what one of them figures out rarely reaches the next. Closing this gap means two things. Time to learn inside the work, so it isn&#8217;t squeezed in around it. And a way for what people learn to travel, so the patterns that work become shared practice. Some organizations, <a href="https://stackoverflow.blog/2026/07/02/ai-coding-chaos-into-a-repeatable-playbook/">Snowflake among them</a>, run regular focus weeks and turn what their engineers found into patterns everyone can use.</p><p><strong>Accountability.</strong> Today the decision about how much review a change needs is usually left to the developer, one change at a time, while the amount to review keeps growing. What&#8217;s missing is a decision the organization makes and curates over time: which kinds of work can move with lighter review, which still needs someone who understands the change fully, and what gets checked automatically before a person looks. I&#8217;d make that call with the same questions developers already ask themselves: how risky the work is, and how reversible.</p><p><strong>Work design.</strong> This gap has two parts. One is what the fourth no is about: decisions made without the people they affect. The other is the processes that need to adapt. As more of the code comes from AI, review grows, handoffs shift, and the time saved on writing code goes into directing and checking AI output, work that rarely shows up in anyone&#8217;s plan. Both parts land on the developer, and neither has an owner.</p><p><strong>Fit.</strong> Often there is no coordinated effort to identify, for the organization as a whole, where AI pays off and for which work. So each developer runs that evaluation alone, and what they find never adds up to a view the organization can use. The result is uneven use across teams. Closing this gap means collecting what people have found about where AI works for each type of task (writing tests, say, or migrating legacy code), and accepting that for some kinds of work the answer will be that AI doesn&#8217;t pay off yet. It also means asking again as the tools change, because the answer for a given kind of task won&#8217;t stay the same for long.</p><h2>Where to start</h2><p>Here&#8217;s where I think there is leverage for leaders.</p><p><strong>Listen for moments.</strong> It is always good advice to have good listening mechanisms for your developers. In this case you want to ask, in a blameless way, about examples of choosing not to use AI for something, and why. Asking about a specific moment gets you the situation and the reason, and it works whether the person uses AI every day, or hardly at all. The reasons will point to the gaps. If use is tracked or required, how you ask matters as much as what you ask. People need to be able to answer honestly, so think about anonymity and about who does the asking.</p><p><strong>Redesign the work with the engineers who do it.</strong> Start from the challenges the no&#8217;s brought up. If you can, bring in the engineers who ran into those challenges and held back. They will give you more nuanced requirements, and they will show you constraints you wouldn&#8217;t otherwise see. For that to work, the redesign has to be allowed to conclude that some work isn&#8217;t ready for AI yet.</p><h2>Final thoughts</h2><p>The developers saying no are often the first to run into what a rollout hasn&#8217;t put in place yet. That makes their reluctance one of the earliest signals a leader gets, and it&#8217;s worth treating the way we&#8217;d treat any other readiness problem: by finding out what&#8217;s behind it.</p><p>It also helps to remember who is saying no. Often it&#8217;s people who care about the work and want to keep doing it well, which is exactly who you want shaping how AI fits into it. Handled well, a no is where a more useful conversation about AI can start.</p><div><hr></div><h2><strong>One more thing</strong></h2><p><strong><a href="https://getdx.com/customers/workleap-uses-dx-for-the-ai-sdlc/">We published an article earlier this week</a></strong> about how the HR tech company Workleap quadrupled engineering velocity across 120 developers by using DX telemetry to measure and defend AI investments.<br><br>&#8220;Without the data from DX, executive <strong>discussions around our $3 million token spend would have been purely defensive</strong>. Having real-time data allowed us to prove that moving faster was delivering tangible value to the business.&#8221; said Guillaume Roy, Chief Product and Technology Officer, Workleap.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://getdx.com/customers/workleap-uses-dx-for-the-ai-sdlc/&quot;,&quot;text&quot;:&quot;Read about it&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://getdx.com/customers/workleap-uses-dx-for-the-ai-sdlc/"><span>Read about it</span></a></p>]]></content:encoded></item><item><title><![CDATA[Inside the DevEx Metrics Compass with André Meyer]]></title><description><![CDATA[Researcher Andr&#233; Meyer shares how teams can measure AI&#8217;s impact beyond adoption, choose meaningful productivity metrics, and understand what makes developers&#8217; work rewarding.]]></description><link>https://newsletter.getdx.com/p/inside-the-devex-metrics-compass</link><guid isPermaLink="false">https://newsletter.getdx.com/p/inside-the-devex-metrics-compass</guid><dc:creator><![CDATA[Brian Houck]]></dc:creator><pubDate>Thu, 12 Nov 2026 16:49:03 GMT</pubDate><enclosure url="https://api.substack.com/feed/podcast/224203641/b547bf69079195a3b8c33f8d3c7b7c87.mp3" length="0" type="audio/mpeg"/><content:encoded><![CDATA[<p>Listen and watch now on <strong><a href="https://youtu.be/vt-ZB-i3d34">YouTube</a>, <a href="https://podcasts.apple.com/us/podcast/engineering-enablement-by-abi-noda/id1619140476">Apple</a>, and <a href="https://open.spotify.com/show/3NxjyIsuxeDMQtisDqBy7D">Spotify</a></strong>.</p><p>Are we measuring the impact of AI on engineering, or just how much it&#8217;s being used?</p><p>In this episode of <em>Engineering Enablement</em>, I talk with researcher Andr&#233; Meyer about his DevEx Metrics Compass, why traditional productivity metrics often miss what matters, and how to measure the value AI actually delivers.</p><p>We also revisit Andr&#233;&#8217;s <em>Today Was a Good Day</em> study to explore what makes developers&#8217; work rewarding, how AI is changing that experience, and practical ways to balance focused work, collaboration, and growing code review demands.</p><div id="youtube2-vt-ZB-i3d34" class="youtube-wrap" data-attrs="{&quot;videoId&quot;:&quot;vt-ZB-i3d34&quot;,&quot;startTime&quot;:null,&quot;endTime&quot;:null}" data-component-name="Youtube2ToDOM"><div class="youtube-inner"><iframe src="https://www.youtube-nocookie.com/embed/vt-ZB-i3d34?rel=0&amp;autoplay=0&amp;showinfo=0&amp;enablejsapi=0" frameborder="0" loading="lazy" gesture="media" allow="autoplay; fullscreen" allowautoplay="true" allowfullscreen="true" width="728" height="409"></iframe></div></div><h2><strong>Some takeaways: </strong></h2><h3><strong>Why traditional metrics fall short</strong></h3><ul><li><p><strong>AI adoption and output metrics don&#8217;t demonstrate business value.</strong> Metrics like AI usage, acceptance rates, and pull request counts reveal activity, not outcomes.</p></li><li><p><strong>Collaboration is largely overlooked in developer productivity measurement.</strong> Developers spend a significant portion of their time communicating and collaborating, yet few metrics capture its impact.</p></li><li><p><strong>Measuring customer and business value remains a major challenge.</strong> Connecting engineering data to customer satisfaction, feature adoption, and business outcomes requires looking beyond traditional development metrics.</p></li></ul><h3><strong>Choosing the right DevEx metrics</strong></h3><ul><li><p><strong>The DevEx Metrics Compass maps more than 120 metrics from research and industry.</strong> It helps teams navigate existing measurement frameworks and identify relevant metrics.</p></li><li><p><strong>There&#8217;s no universal set of DevEx metrics.</strong> Organizations should select measures based on their specific goals, challenges, and level of measurement maturity.</p></li><li><p><strong>More metrics don&#8217;t necessarily produce better insights.</strong> Focus on a manageable set of priorities, with more detailed metrics available to investigate specific problems.</p></li></ul><h3><strong>Building a more effective measurement system</strong></h3><ul><li><p><strong>Telemetry tells you what changed, but not necessarily why.</strong> Pair quantitative data with developer surveys and qualitative feedback to understand the full picture.</p></li><li><p><strong>Metrics should be evaluated and retired as needs evolve.</strong> Regularly assess whether they remain actionable, inform decisions, or show signs of being gamed.</p></li><li><p><strong>Measure AI&#8217;s impact across multiple dimensions.</strong> Look beyond activity to developer satisfaction, quality, efficiency, and business outcomes, tracking changes over time.</p></li></ul><h3><strong>What makes a good day for developers</strong></h3><ul><li><p><strong>Developers find satisfaction in making meaningful progress. </strong>They report better workdays when they can create something valuable, encounter fewer blockers, and have time to focus.</p></li><li><p><strong>AI doesn&#8217;t necessarily diminish the satisfaction developers get from their work.</strong> Developers can still enjoy creating software when they retain control over what gets built and how decisions are made.</p></li><li><p><strong>Not all context switching is harmful.</strong> Observational research revealed that the cost depends on the nature of the task switch, something automated telemetry alone can&#8217;t capture.</p></li></ul><h3><strong>Protecting focus time and improving collaboration</strong></h3><ul><li><p><strong>The timing of meetings matters as much as their frequency.</strong> Scattered meetings fragment the workday and make sustained focus difficult.</p></li><li><p><strong>Teams should coordinate focused work and collaboration time.</strong> Shared schedules can reduce interruptions while preserving opportunities to communicate and solve problems together.</p></li><li><p><strong>AI-generated code is increasing the burden of peer review.</strong> Clear PR expectations, author accountability, and risk-based review practices can help reduce cognitive load and protect code quality.</p></li></ul><h2><strong>In this episode, we cover:</strong></h2><p>(<a href="https://www.youtube.com/watch?v=vt-ZB-i3d34">00:00</a>) Intro</p><p>(<a href="https://www.youtube.com/watch?v=vt-ZB-i3d34&amp;t=162s">02:42</a>) Why AI dashboards fail to capture actual value</p><p>(<a href="https://www.youtube.com/watch?v=vt-ZB-i3d34&amp;t=308s">05:08</a>) What the DevEx Metrics Compass is and why it was created</p><p>(<a href="https://www.youtube.com/watch?v=vt-ZB-i3d34&amp;t=418s">06:58</a>) Why measuring collaboration matters</p><p>(<a href="https://www.youtube.com/watch?v=vt-ZB-i3d34&amp;t=587s">09:47</a>) Determining the right metrics for your organization</p><p>(<a href="https://www.youtube.com/watch?v=vt-ZB-i3d34&amp;t=723s&amp;pp=0gcJCWMAwfN6Pr3D">12:03</a>) Distinguishing measurement gaps from gaps in public reporting</p><p>(<a href="https://www.youtube.com/watch?v=vt-ZB-i3d34&amp;t=875s">14:35</a>) Why some metrics are harder to act on at scale</p><p>(<a href="https://www.youtube.com/watch?v=vt-ZB-i3d34&amp;t=1091s">18:11</a>) Why customer and business value are difficult to measure</p><p>(<a href="https://www.youtube.com/watch?v=vt-ZB-i3d34&amp;t=1203s">20:03</a>) How to know when it&#8217;s time to retire a metric</p><p>(<a href="https://www.youtube.com/watch?v=vt-ZB-i3d34&amp;t=1460s">24:20</a>) Hierarchies of metrics</p><p>(<a href="https://www.youtube.com/watch?v=vt-ZB-i3d34&amp;t=1651s">27:31</a>) How to measure AI&#8217;s impact beyond activity metrics</p><p>(<a href="https://www.youtube.com/watch?v=vt-ZB-i3d34&amp;t=1820s&amp;pp=0gcJCWMAwfN6Pr3D">30:20</a>) Andr&#233;&#8217;s Good Day study: What makes a good workday for developers</p><p>(<a href="https://www.youtube.com/watch?v=vt-ZB-i3d34&amp;t=1916s">31:56</a>) What observing developers revealed about context switching</p><p>(<a href="https://www.youtube.com/watch?v=vt-ZB-i3d34&amp;t=2068s">34:28</a>) What makes developers&#8217; work rewarding in the age of AI</p><p>(<a href="https://www.youtube.com/watch?v=vt-ZB-i3d34&amp;t=2187s">36:27</a>) The tradeoffs of delegating coding to AI</p><p>(<a href="https://www.youtube.com/watch?v=vt-ZB-i3d34&amp;t=2281s">38:01</a>) Balancing collaboration with focused work</p><p>(<a href="https://www.youtube.com/watch?v=vt-ZB-i3d34&amp;t=2531s">42:11</a>) Two ways to improve developers&#8217; workdays</p><p><strong>Where to find Andre&#769; Meyer-Baron:</strong></p><p>&#8226; LinkedIn: <a href="https://www.linkedin.com/in/andrenmeyer">https://www.linkedin.com/in/andrenmeyer</a></p><p>&#8226; Website: <a href="https://andre-meyer.ch">https://andre-meyer.ch</a> and <a href="http://www.devexcompass.com">www.devexcompass.com</a></p><p><strong><span>Where to find Brian Houck:</span></strong></p><p><span>&#8226; LinkedIn: </span><a href="https://www.linkedin.com/in/brianhouck"><span>https://www.linkedin.com/in/brianhouck</span></a></p><h2><strong>Referenced:</strong></h2><p>&#8226; <a href="https://getdx.com/whitepaper/the-cafes-framework/">CAFE(S): Your Agent Is Only as Good as Its Context</a></p><p>&#8226; <a href="https://queue.acm.org/doi/fullHtml/10.1145/3831360">Too Many DevEx Metrics, Too Little Guidance</a></p><p>&#8226; <a href="https://www.microsoft.com/en-us/research/wp-content/uploads/2019/04/devtime-preprint-TSE19.pdf">Today was a Good Day: The Daily Life of Software Developers</a></p><p>&#8226; <a href="https://devexcompass.com/">DevEx Metrics Compass</a></p><p>&#8226; <a href="https://www.linkedin.com/in/gailcmurphy/">Gail Murphy | LinkedIn</a></p><p>&#8226; <a href="https://www.linkedin.com/in/thomas-fritz-aa53a8/">Thomas Fritz | LinkedIn</a></p><p>&#8226; <a href="https://www.linkedin.com/in/patrickomeyer">Patrick Meyer | LinkedIn</a></p><p>&#8226; <a href="https://space-framework.com/">The SPACE Framework</a></p><p>&#8226; <a href="https://arxiv.org/abs/2603.22106">From Technical Debt to Cognitive and Intent Debt: Rethinking Software Health in the Age of AI</a></p>]]></content:encoded></item><item><title><![CDATA[The 10x org is in your grasp]]></title><description><![CDATA[Part III: Setting the groundwork for a truly 10x organization]]></description><link>https://newsletter.getdx.com/p/the-10x-org-is-in-your-grasp</link><guid isPermaLink="false">https://newsletter.getdx.com/p/the-10x-org-is-in-your-grasp</guid><dc:creator><![CDATA[Justin Reock]]></dc:creator><pubDate>Wed, 11 Nov 2026 11:07:07 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/24df9d87-f262-4826-8a35-e81ba5c57b49_2400x1260.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><em><strong>Welcome to the latest issue of Engineering Enablement,</strong> a weekly newsletter sharing research and perspectives on developer productivity.</em></p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.getdx.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe now&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.getdx.com/subscribe?"><span>Subscribe now</span></a></p><p>Join us <strong>tomorrow</strong> for DX&#8217;s next webinar. <span class="mention-wrap" data-attrs="{&quot;name&quot;:&quot;Eirini Kalliamvakou&quot;,&quot;id&quot;:554376261,&quot;type&quot;:&quot;user&quot;,&quot;url&quot;:null,&quot;photo_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5c463cb7-9819-4cdf-a4d4-2c3540b9c64d_2083x2083.jpeg&quot;,&quot;uuid&quot;:&quot;9a41e08e-3eb3-467f-ad1f-71c1ddca88cc&quot;}" data-component-name="MentionToDOM"></span> will be discussing <em><strong>where AI-related time savings are going</strong></em>, with <span class="mention-wrap" data-attrs="{&quot;name&quot;:&quot;Annie Vella&quot;,&quot;id&quot;:14002203,&quot;type&quot;:&quot;user&quot;,&quot;url&quot;:null,&quot;photo_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0a3968d5-8ea1-408f-89f4-6e342bea5f3a_144x144.png&quot;,&quot;uuid&quot;:&quot;cf92a82a-a62e-4516-a2b5-d38da94dc9d7&quot;}" data-component-name="MentionToDOM"></span> and Nathan Hunt, a Principal Software Engineer at Snowflake. <strong><a href="https://getdx.com/webinar/ai-generated-capacity-where-does-the-time-go/?utm_source=newsletter">Sign up today.</a></strong></p><div><hr></div><p><strong>Scott:</strong> For the last two weeks, our Field CTO Justin Reock has been laying the groundwork for how to 10x your engineering organization in the age of AI. Starting by <a href="https://newsletter.getdx.com/p/you-cant-find-10x-developers">dispelling the myth of the 10x developer</a> (hopefully for good this time) and then <a href="https://newsletter.getdx.com/p/why-dora-wasnt-enough-and-what-came">charting the evolution of developer metrics frameworks</a>, now we need to bring it all together.</p><p>This week, Justin lays out the foundations for a healthy engineering organization in the age of AI, and which metrics you need to track to unlock the 10x potential of your teams.</p><p>Over to you, Justin.</p><div><hr></div><p><strong>Justin:</strong> In our last two installments, we&#8217;ve established that <a href="https://newsletter.getdx.com/p/you-cant-find-10x-developers">10x organizations are built on environments</a> supporting deep focus, and that traditional metrics borrowed from the factory floor only tell part of the story. </p><p>We also know that when organizations measure the wrong things, they fall victim to Goodhart&#8217;s Law, where well-intentioned measures backfire, leading teams to optimize for the metric rather than the actual business outcome.</p><p>To build a high-performing engineering culture that avoids these pitfalls, engineering leaders must align their continuous improvement strategies behind a unified, holistic set of metrics.</p><h3>The Developer Experience Index as a roadmap</h3><p>Relying solely on quantitative data is an incomplete strategy, while relying purely on qualitative surveys is difficult and costly to execute. The solution is a developer-centric approach that measures the lived experience of engineers, as well as the efficiency of the surrounding platform, culture, and environment.</p><p><a href="https://newsletter.getdx.com/p/why-dora-wasnt-enough-and-what-came">Last week, I charted the path from DORA, SPACE, and DevEx to the  DX Core 4</a>, a comprehensive distillation of the three mainstream productivity measurement frameworks.</p><p>Rolling this framework out requires a shift in how management views productivity. It is not just about installing a dashboard, it&#8217;s about recognizing how foundational drivers such as local iteration, incident response, documentation, and dev environments funnel directly into the productivity outcomes sought by leadership.</p><p>When you improve these specific drivers, the Developer Experience Index (DXI) rises, which directly influences the organization&#8217;s critical outcomes: speed, ease of delivery, quality, engagement, and efficiency. Ultimately, this is about adopting a mindset of continuous improvement, and using the DXI to maintain up-to-date situational awareness of the greatest impediments to productivity, so that they can be intervened upon and remediated.</p><p>Although I much prefer the narrative of optimal developer experience leading to increased throughput and value generation, the DXI can also provide us with a cost-based return on investment (ROI) of these improvements. For every 1-point improvement organizations can achieve in their DXI, they return about 10 hours a year per engineer in reduced waste, toil, and friction.</p><h3>Aligning behind a single set of metrics</h3><p>One of the most powerful aspects of aligning engineering leadership behind a single, standard, agreed upon set of metrics is the ability to align the organization&#8217;s definition of productivity, while preserving the autonomy engineering leaders need to improve their teams.</p><p>Well-intentioned leaders often create their own strategy for measurement and improvement. Some may choose DORA, others SPACE, and many will come up with their own set of metrics. While this may lead to short term improvements, over time these fragmented strategies can cause major organizational misalignment.</p><p>This misalignment can be difficult to detect at the team level, because engineering leaders following their own strategies feel that they are doing the right thing. This trust and autonomy provides a sense of personal satisfaction, which is important, but can create a false sense of direction. In other words, that necessary autonomy can lead to falsely positive sentiments about overall organizational improvement. This is a very precarious position to be in.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!1xIU!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff15a402c-9f97-4977-a1d1-8a19eea49931_650x406.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!1xIU!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff15a402c-9f97-4977-a1d1-8a19eea49931_650x406.png 424w, https://substackcdn.com/image/fetch/$s_!1xIU!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff15a402c-9f97-4977-a1d1-8a19eea49931_650x406.png 848w, https://substackcdn.com/image/fetch/$s_!1xIU!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff15a402c-9f97-4977-a1d1-8a19eea49931_650x406.png 1272w, https://substackcdn.com/image/fetch/$s_!1xIU!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff15a402c-9f97-4977-a1d1-8a19eea49931_650x406.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!1xIU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff15a402c-9f97-4977-a1d1-8a19eea49931_650x406.png" width="650" height="406" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f15a402c-9f97-4977-a1d1-8a19eea49931_650x406.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:406,&quot;width&quot;:650,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!1xIU!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff15a402c-9f97-4977-a1d1-8a19eea49931_650x406.png 424w, https://substackcdn.com/image/fetch/$s_!1xIU!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff15a402c-9f97-4977-a1d1-8a19eea49931_650x406.png 848w, https://substackcdn.com/image/fetch/$s_!1xIU!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff15a402c-9f97-4977-a1d1-8a19eea49931_650x406.png 1272w, https://substackcdn.com/image/fetch/$s_!1xIU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff15a402c-9f97-4977-a1d1-8a19eea49931_650x406.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Credit: Henrik Kniberg, from &#8220;Spotify&#8217;s Engineering Culture&#8221;, 2014</figcaption></figure></div><p>The lower right quadrant of Kniberg&#8217;s cartoon represents the state of misaligned autonomy that engineering organizations often find themselves in. Notice that while leaders are happy, the organization isn&#8217;t moving in the same direction.</p><p>By prescribing the Core 4 as the accepted metric set, and having engineering leaders agree to base their continuous improvement initiatives in response to these signals, the organization can achieve what <a href="https://www.orgtopologies.com/post/aligned-autonomy-at-scale">Spotify refers to as &#8220;aligned autonomy&#8221;</a> behind continuous improvement of their platforms, systems, and culture in which engineers work.</p><h3>How AI changes this</h3><p>With the rapid proliferation of AI agents and assistants, the baseline for many of these metrics are shifting. According to our most recent benchmarks, AI adoption swelled to over 95% weekly active users <a href="https://getdx.com/report/State-of-AI-Impact-in-Engineering-Q2-Report/">across surveyed organizations by Q2 2026</a>. This <a href="https://getdx.com/blog/don-t-measure-ai-code-percentage-on-its-own/">all but eliminates the control group of &#8220;non-AI users.&#8221;</a> AI-authored code now makes up roughly 62% of all new code, marking a 15% jump from just the previous quarter, and there aren&#8217;t clear signs of this slowing down.</p><p>This scale of code generation is driving measurable and significant speed improvements across the industry. Median weekly TrueThroughput, DX&#8217;s proprietary weighted PR throughput metric, grew consistently. Additionally, AI is drastically shrinking the time it takes for new hires to add value. The median time to a new developer&#8217;s 10th merged PR plummeted from 70 days in Q3 2025 to just 30 days in Q2 2026.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!_bT0!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1b98613-2829-40b1-97dd-7bf4321d5d42_2100x1466.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!_bT0!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1b98613-2829-40b1-97dd-7bf4321d5d42_2100x1466.png 424w, https://substackcdn.com/image/fetch/$s_!_bT0!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1b98613-2829-40b1-97dd-7bf4321d5d42_2100x1466.png 848w, https://substackcdn.com/image/fetch/$s_!_bT0!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1b98613-2829-40b1-97dd-7bf4321d5d42_2100x1466.png 1272w, https://substackcdn.com/image/fetch/$s_!_bT0!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1b98613-2829-40b1-97dd-7bf4321d5d42_2100x1466.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!_bT0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1b98613-2829-40b1-97dd-7bf4321d5d42_2100x1466.png" width="1456" height="1016" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a1b98613-2829-40b1-97dd-7bf4321d5d42_2100x1466.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1016,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:172186,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://newsletter.getdx.com/i/224326094?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1b98613-2829-40b1-97dd-7bf4321d5d42_2100x1466.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!_bT0!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1b98613-2829-40b1-97dd-7bf4321d5d42_2100x1466.png 424w, https://substackcdn.com/image/fetch/$s_!_bT0!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1b98613-2829-40b1-97dd-7bf4321d5d42_2100x1466.png 848w, https://substackcdn.com/image/fetch/$s_!_bT0!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1b98613-2829-40b1-97dd-7bf4321d5d42_2100x1466.png 1272w, https://substackcdn.com/image/fetch/$s_!_bT0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1b98613-2829-40b1-97dd-7bf4321d5d42_2100x1466.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>With unprecedented speed comes new friction and escalating costs.</p><p>PR sizes have grown 122% over four quarters, and this verbosity, along with increased throughput, is shifting the bottleneck directly onto human reviewers. Organizations generating high volumes of AI code (80+%) now wait twice as long for a first review compared to organizations generating lower volumes (20-40%). There are numerous factors likely at play here, including the growing size of PRs, and swelling review backlogs.</p><p>Meanwhile, the financial reality of this transformation is setting in. Median AI spend at larger organizations (750+ developers) accelerated sharply, growing 3x quarter-over-quarter. Across all organization sizes, token spend grew roughly 21x over the past four quarters.</p><h3>The true bottlenecks remain human</h3><p>Despite these massive AI investments and the rapid acceleration of code generation, the data reveals a sobering truth: non-AI factors, such as meeting-heavy days and interruptions still outweigh AI benefits.</p><p>When analyzing the annualized distribution of developer time, the cost of meeting-heavy days, which remain stubbornly flat, and the cost of interruption frequency completely eclipse the time savings provided by AI tools.</p><p>AI metrics like token spend and AI-assisted PR percentages measure what is happening, but your Core metrics (like the DX Core 4) measure whether these investments are working.</p><blockquote><p>Although AI is accelerating parts of our pipeline, the underlying organizational physics have not changed.</p></blockquote><p>To build a 10x organization, you must find and fix the bottleneck. That bottleneck was never the speed at which we can write code, it is the environment in which we ask our engineers to review, collaborate, and focus.</p><h3>Fix human bottlenecks to build 10x engineering organizations</h3><p>Building a 10x organization requires an environment that minimizes interruptions to prioritize developer focus. Measuring this success evolved from traditional manufacturing and DORA metrics into the developer-centric SPACE framework.</p><p>To unify measurement, modern teams use the DX Core 4 to optimize flow state, feedback loops, and cognitive load. While AI massively accelerates code generation, it also creates new bottlenecks like inflated pull requests and longer review times.</p><p>Because human friction, like meeting-heavy days, still outweighs AI time savings, leaders must focus on resolving environmental constraints, not just writing code faster.</p><p>-Justin</p><div><hr></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.getdx.com/p/the-10x-org-is-in-your-grasp?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.getdx.com/p/the-10x-org-is-in-your-grasp?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p>]]></content:encoded></item><item><title><![CDATA[Why DORA wasn’t enough, and what came next]]></title><description><![CDATA[Part II: The evolution of developer measurement]]></description><link>https://newsletter.getdx.com/p/why-dora-wasnt-enough-and-what-came</link><guid isPermaLink="false">https://newsletter.getdx.com/p/why-dora-wasnt-enough-and-what-came</guid><dc:creator><![CDATA[Justin Reock]]></dc:creator><pubDate>Wed, 04 Nov 2026 11:03:21 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/d4098e51-4ca6-433d-8248-884d32c7f033_2400x1260.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><em><strong>Welcome to the latest issue of Engineering Enablement,</strong> a weekly newsletter sharing research and perspectives on developer productivity.</em></p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.getdx.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe now&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.getdx.com/subscribe?"><span>Subscribe now</span></a></p><p>DX&#8217;s Q3 AI Impact Report is now available with the latest research on AI&#8217;s impact across engineering organizations. <strong><a href="https://getdx.com/report/state-of-ai-impact-in-engineering-q3-report/?utm_source=newsletter">Read the full report.</a></strong></p><div><hr></div><p><strong>Scott: </strong>Welcome back to Engineering Enablement. This is part two of a three-part series on building 10x organizations by our Field CTO, Justin Reock.</p><p>Two months ago, <a href="https://www.linkedin.com/posts/gbgb-share-7501291218820087808-qWa-/">DORA announced</a> that its 2026 annual survey and report will be the last of its kind, as the team switches to more &#8220;qualitative and observational research&#8221;. Given that news, it feels like a pertinent time to revisit the history of the defining developer measurement framework, and where it fell short.</p><p>Over to you, Justin.</p><div><hr></div><p><strong>Justin:</strong> <a href="https://newsletter.getdx.com/p/you-cant-find-10x-developers">Part one</a> made the case that the original 10x finding was about environments, not individuals. This week, I want to go through the four major measurement frameworks our industry has produced, in the order they appeared, because each one is a direct response to software industry progression and evolution.</p><p>If you walk into a typical engineering organization today and ask how they measure productivity, you&#8217;ll usually get one of three answers. They&#8217;ll tell you they&#8217;re &#8220;doing DORA.&#8221; They&#8217;ll tell you they&#8217;re &#8220;starting to use SPACE.&#8221; Or they&#8217;ll tell you they don&#8217;t measure it. All three answers are signals of where they sit on the maturity curve.</p><h2>DORA: Powerful for DevOps, focused on software delivery</h2><p>DORA metrics came out of the DevOps Research and Assessment program led by Nicole Forsgren, Jez Humble, and Gene Kim. They&#8217;re now more than a decade old, and they remain the most widely deployed framework in the industry.</p><p>DORA&#8217;s strength is that it&#8217;s prescriptive. It tells you exactly what to measure. Where it becomes limited is that it&#8217;s scoped almost entirely to the delivery pipeline. With the exception of some subcomponents of Lead Time for Changes, it tells you little about what happens before the merge to production, i.e., the developer experience, which is where most of a developer&#8217;s time actually goes.</p><p>This is not a direct critique on DORA, but rather a perspective on the limitations and blind spots of DevOps. I&#8217;ve been writing code professionally since the late 90s and I&#8217;m intimately acquainted with what the world looked like before DevOps. The toil of manual deployments. The friction of delivering artifacts to QA teams via the SneakerNet. DevOps has profoundly improved the way we work as software professionals.</p><p>Despite these clear advantages, a widely-accepted critique of DevOps, and by extension DORA metrics, is that they borrow too heavily from manufacturing. DevOps came out of the revolutionary Lean and Toyota Production System practices, and the factory metaphor is genuinely useful for many parts of software delivery. Work in progress, batch size, cycle time, flow efficiency, the idea that small batches moving through a stable pipeline produce better outcomes than large batches moving through an unstable one all map cleanly onto software, and DORA captures it well.</p><p>The metaphor breaks down at the workstation. A factory worker on an assembly line is, by design, doing standardized, repeatable work where the value being created is external and physical. A software engineer is doing the opposite. The work is non-standardized by definition: the value being created is cognitive, and the software engineer is what management research refers to as an opinionated knowledge worker: someone whose output depends not just on their hands, but on their judgment, context, and attention.</p><p>A factory metric like cycle time looks at the external value stream: code moves from idea to production through a pipeline, and we measure how fast it moves and how often it breaks. But there&#8217;s a second value stream running in parallel, invisible to any system metric, which I call the &#8217;internal value stream.&#8217; It&#8217;s what happens inside the developer&#8217;s head between the moment they pick up a task and the moment they ship it. Flow state, context switches, cognitive fatigue, all of that lives in the internal value stream, and none of it shows up on a DORA dashboard.</p><p>The other limitation is that the metrics are easy to game. Deployment frequency goes up if you split your PRs in half. Lead time goes down if you stop blocking merges on tests. Change failure rate goes down if you stop deploying anything risky. Any single-metric system applied to humans will be gamed according to Goodhart&#8217;s Law, and DORA is no exception. If you&#8217;re running DORA in 2026, run it correctly: as a balanced set of signals where moving any one number at the expense of the others is treated as a regression, not a win.</p><h2>SPACE: Comprehensive, flexible, with no prescriptive metrics</h2><p>SPACE arrived in 2021 from Forsgren, Margaret-Anne Storey, Brian Houck, Chandra Maddila, Thomas Zimmerman, and Jenna Butler. It was an explicit attempt to broaden the productivity measurement lens beyond delivery pipelines. The core insight of SPACE is that productivity is multidimensional and that any single dimension, measured in isolation, will mislead you.</p><p>SPACE is the right answer to the question &#8220;what dimensions matter?&#8221; but it&#8217;s the wrong answer to the question &#8220;what should I measure on Monday morning?&#8221; It&#8217;s a powerful normalization framework for designing your own measurement program and scaffolding your culture, but it&#8217;s not a measurement program itself.</p><p>The SPACE framework is arranged into five dimensions of Satisfaction, Performance, Activity, Collaboration, and Efficiency. It&#8217;s meant to help organizations categorize the metrics that matter to them into a balanced set of oppositional dimensions, creating the necessary tension between metrics that prevent gamification and promote a comprehensive understanding of productivity, and the continuous improvement thereof.</p><p>This gets us much closer to understanding both the experiential and systems metrics necessary to model developer experience and its relationship to developer productivity. The internal value stream, however, is highly nuanced and so the DevEx framework arrived to help provide a more granular look at this essential part of the throughput pipeline of software organizations.</p><h2>DevEx: Developer experience predicts developer productivity</h2><p>The DevEx framework, published in 2023 by Forsgren, Storey, Michaela Greiler, and DX cofounder Abi Noda, was the first to put developer experience on equal footing with delivery metrics. It balances three dimensions: feedback loops, cognitive load, and flow state. These are exactly the variables DeMarco and Lister identified in 1984, now operationalized as something you can measure across a modern engineering organization with a combination of survey and system data.</p><p>DevEx is the framework I most often see catalyze DevEx improvements, because it&#8217;s the first one that takes seriously what developers themselves say is slowing them down. Long build times, flaky tests, unclear ownership, broken local environments, the friction that doesn&#8217;t show up in a DORA dashboard but eats half the workweek.</p><p>DevEx&#8217;s limitation is that it&#8217;s heavily self-reported, and self-reported data alone is not enough to create cycles of continuous improvement around developer experience. It&#8217;s powerful to have a signal that, for example, &#8220;build times are slow.&#8221; Coupling that signal with system metrics that show actual build times allows for material improvement of the root problem.</p><p>This is the gap the Core 4 was built to close, while simultaneously distilling the best parts of DORA, SPACE, and DevEx into a single unified framework for improving developer experience and increasing productivity.</p><h2>Core 4: The distillative framework with prescribed metrics</h2><p>The <a href="https://getdx.com/research/benchmarks/">DX Core 4</a> is not a replacement for DORA, SPACE, or DevEx. It&#8217;s a distillation. If you&#8217;re running Core 4, you&#8217;re running pieces of all three.</p><p>Core 4 organizes measurement into four dimensions, each with one primary metric and a few supporting ones.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!M9ue!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbcd4e19-3d83-4004-afa2-ebc24d406602_4200x1608.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!M9ue!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbcd4e19-3d83-4004-afa2-ebc24d406602_4200x1608.png 424w, https://substackcdn.com/image/fetch/$s_!M9ue!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbcd4e19-3d83-4004-afa2-ebc24d406602_4200x1608.png 848w, https://substackcdn.com/image/fetch/$s_!M9ue!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbcd4e19-3d83-4004-afa2-ebc24d406602_4200x1608.png 1272w, https://substackcdn.com/image/fetch/$s_!M9ue!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbcd4e19-3d83-4004-afa2-ebc24d406602_4200x1608.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!M9ue!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbcd4e19-3d83-4004-afa2-ebc24d406602_4200x1608.png" width="1456" height="557" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/bbcd4e19-3d83-4004-afa2-ebc24d406602_4200x1608.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:557,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:249579,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://newsletter.getdx.com/i/223366255?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbcd4e19-3d83-4004-afa2-ebc24d406602_4200x1608.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!M9ue!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbcd4e19-3d83-4004-afa2-ebc24d406602_4200x1608.png 424w, https://substackcdn.com/image/fetch/$s_!M9ue!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbcd4e19-3d83-4004-afa2-ebc24d406602_4200x1608.png 848w, https://substackcdn.com/image/fetch/$s_!M9ue!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbcd4e19-3d83-4004-afa2-ebc24d406602_4200x1608.png 1272w, https://substackcdn.com/image/fetch/$s_!M9ue!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbcd4e19-3d83-4004-afa2-ebc24d406602_4200x1608.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>The first is Speed. The primary metric is weighted PR Throughput, sometimes called PRs per engineer. This is the most controversial choice in the framework, and we (DX) have always been clear that it must never be applied at the individual level or tied to performance reviews. As an aggregate signal of flow through the system, balanced against the other three dimensions, it works.</p><p>The second is Effectiveness, measured by the Developer Experience Index, or DXI. This is a single aggregated score from fourteen standardized survey items covering everything from code review quality, to focus time, to confidence in the deployment process. The DXI is where DevEx gets folded into Core 4, and provides an anchor for continuous improvement of various aspects of developer experience.</p><p>The third is Quality, measured by change failure rate or defect ratio, which is borrowed directly from DORA and serves as the counterweight to Speed. If Speed goes up and Quality goes down, you don&#8217;t have a 10x organization, you have a fragile one.</p><p>The fourth is Impact, measured by the percentage of engineering time spent on new capabilities versus keep-the-lights-on work. This is the metric that translates engineering investment into business outcomes. A team that&#8217;s shipping fast, has a happy developer base, and a low change failure rate, but spends 80% of its time on toil, is not a 10x organization.</p><h2>Implementing Core 4 at different maturity levels</h2><p>Because the Core 4 blends the strengths of its predecessor frameworks, organizations can stand on the shoulders of giants and adopt a comprehensive measurement strategy at any existing maturity stage.</p><p>If you&#8217;re not measuring anything today, you can start with the DXI and the minimum viable system signals necessary to calculate DORA metrics.</p><p>If you&#8217;ve been running DORA for a year or more and your leadership is asking what else matters, layer DevEx on top through a survey instrument like the DXI.</p><p>If you&#8217;re ready to brief a CFO or a board on engineering performance and want something defensible top-to-bottom, the Core 4 is the framework with the cleanest line from engineer friction all the way through to business impact.</p><p>What you should not do is invent your own framework from scratch. We have forty-one years of cumulative research starting from the Coding War Games and ending at Core 4. None of it is hypothetical anymore. The frameworks exist. The benchmarks exist. The choice is which one to start with, not whether to start.</p><p>Next week, in the third and final installment, I&#8217;ll close the series with the operational piece: what it actually looks like to run one of these frameworks across an engineering organization, the failure modes to watch for, and the role AI is playing in changing which of these metrics matters most.</p><p>Until then.</p><p>-Justin</p><div><hr></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.getdx.com/p/why-dora-wasnt-enough-and-what-came?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.getdx.com/p/why-dora-wasnt-enough-and-what-came?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p>]]></content:encoded></item><item><title><![CDATA[Roblox’s approach to tripling AI code review acceptance rates]]></title><description><![CDATA[Andrew Swerdlow shares how Roblox is building trust in AI agents to autonomously review, test, and deploy code and what that means for engineers.]]></description><link>https://newsletter.getdx.com/p/robloxs-approach-to-tripling-ai-code</link><guid isPermaLink="false">https://newsletter.getdx.com/p/robloxs-approach-to-tripling-ai-code</guid><dc:creator><![CDATA[Brian Houck]]></dc:creator><pubDate>Fri, 30 Oct 2026 14:33:04 GMT</pubDate><enclosure url="https://api.substack.com/feed/podcast/221552896/5859fe03bf7134c8a5d98e1f86a49b9d.mp3" length="0" type="audio/mpeg"/><content:encoded><![CDATA[<p>Listen and watch now on <strong><a href="https://youtu.be/j3YuAfNCZGk">YouTube</a>, <a href="https://podcasts.apple.com/us/podcast/engineering-enablement-by-abi-noda/id1619140476">Apple</a>, and <a href="https://open.spotify.com/show/3NxjyIsuxeDMQtisDqBy7D">Spotify</a></strong>.</p><p>In this episode of <em>Engineering Enablement</em>, I talk with Andrew Swerdlow, VP of Engineering at Roblox, about how the company is building trust in AI agents to review, test, and deploy code autonomously. Andrew explains how Roblox turns past code review feedback into rules for agents and why engineering is shifting toward specifications and intent. We explore what that means for the skills engineers need, how Roblox measures AI&#8217;s impact on productivity, and how it manages costs. We also discuss the guardrails needed to let agents safely ship code with minimal human intervention.</p><div id="youtube2-j3YuAfNCZGk" class="youtube-wrap" data-attrs="{&quot;videoId&quot;:&quot;j3YuAfNCZGk&quot;,&quot;startTime&quot;:null,&quot;endTime&quot;:null}" data-component-name="Youtube2ToDOM"><div class="youtube-inner"><iframe src="https://www.youtube-nocookie.com/embed/j3YuAfNCZGk?rel=0&amp;autoplay=0&amp;showinfo=0&amp;enablejsapi=0" frameborder="0" loading="lazy" gesture="media" allow="autoplay; fullscreen" allowautoplay="true" allowfullscreen="true" width="728" height="409"></iframe></div></div><h2><strong>Some takeaways: </strong></h2><h3><strong>Building trust in autonomous development</strong></h3><ul><li><p><strong>Generating code is no longer the biggest bottleneck.</strong> Code review, testing, and deployment still slow delivery. Realizing AI&#8217;s full potential requires automating more of the software development lifecycle.</p></li><li><p><strong>Autonomy has to be earned through trust.</strong> Strong testing infrastructure, production canaries, and automated rollback capabilities are essential for allowing agents to operate independently.</p></li></ul><h3><strong>Organization-specific rules lifted AI review acceptance to nearly 70%</strong></h3><ul><li><p><strong>Generic AI models lack organization-specific knowledge.</strong> Companies need ways to capture their internal engineering practices and make that information available to agents.</p></li><li><p><strong>Code review comments are a valuable source of institutional knowledge.</strong> Roblox mines historical feedback to create rules called exemplars that guide AI-generated code and code reviews.</p></li><li><p><strong>Organization-specific guidance makes AI code review more effective.</strong> Roblox&#8217;s AI-generated review comments now have an acceptance rate of nearly 70%, compared with 50&#8211;55% for human-written comments.</p></li></ul><h3><strong>Humans approve the rules, and about 5% of PRs auto-merge</strong></h3><ul><li><p><strong>Human judgment is shifting from reviewing individual changes to defining the rules.</strong> Engineers can establish and approve guardrails that allow AI to review code without requiring constant human intervention.</p></li><li><p><strong>Roblox is already allowing some code to reach production autonomously.</strong> About 5% of PRs are automatically merged after passing AI code review, programmatic safety checks, and risk assessments.</p></li></ul><h3><strong>Intent, specs, and judgment are becoming the core engineering skills</strong></h3><ul><li><p><strong>Engineering work is shifting from code toward intent, architecture, and specifications.</strong> As agents handle more implementation details, engineers will increasingly collaborate on higher-level designs.</p></li><li><p><strong>Experience and judgment remain essential.</strong> Senior engineers are often better equipped to challenge AI-generated solutions and anticipate problems with scalability, maintainability, and system design.</p></li><li><p><strong>Engineering interviews may need to evaluate process rather than just results.</strong> How efficiently candidates use AI, develop specifications, and arrive at solutions could become more important than traditional coding exercises.</p></li></ul><h3><strong>Roblox tracks accepted, rolled-back, and unattended code</strong></h3><ul><li><p><strong>The amount of AI-generated code is becoming a less meaningful metric.</strong> What matters is whether that code survives in production, delivers value, and requires less human intervention.</p></li><li><p><strong>Roblox is experimenting with new measures of productivity.</strong> Agent autonomy, feature velocity, and developer sentiment help reveal whether AI tools are genuinely improving engineering outcomes.</p></li></ul><h3><strong>Session tracing and caching are the main levers on AI cost</strong></h3><ul><li><p><strong>Observability is essential for understanding AI costs and waste.</strong> Roblox tracks agent sessions, token usage, and code outcomes to identify inefficient workflows and opportunities for improvement.</p></li><li><p><strong>The right infrastructure can dramatically reduce costs.</strong> Roblox reduced the cost of individual code review rule evaluations by 90% through caching and shared context.</p></li><li><p><strong>Flexibility matters in a rapidly changing AI landscape.</strong> Rather than building every capability internally or committing to one vendor, Roblox supports multiple models and tools so engineers can adopt the most effective solutions.</p></li></ul><h2><strong>In this episode, we cover:</strong></h2><p>(<a href="https://www.youtube.com/watch?v=j3YuAfNCZGk">00:00</a>) Intro</p><p>(<a href="https://www.youtube.com/watch?v=j3YuAfNCZGk&amp;t=140s">02:20</a>) Building trust in AI across the software development lifecycle</p><p>(<a href="https://www.youtube.com/watch?v=j3YuAfNCZGk&amp;t=284s">04:44</a>) When human oversight helps and when it gets in the way</p><p>(<a href="https://www.youtube.com/watch?v=j3YuAfNCZGk&amp;t=410s">06:50</a>) What AI agents need to know about your organization</p><p>(<a href="https://www.youtube.com/watch?v=j3YuAfNCZGk&amp;t=525s">08:45</a>) How Roblox turns engineering knowledge into rules for AI agents</p><p>(<a href="https://www.youtube.com/watch?v=j3YuAfNCZGk&amp;t=760s">12:40</a>) How AI code review compares to human code review</p><p>(<a href="https://www.youtube.com/watch?v=j3YuAfNCZGk&amp;t=899s">14:59</a>) Capturing engineering knowledge beyond documentation</p><p>(<a href="https://www.youtube.com/watch?v=j3YuAfNCZGk&amp;t=1094s">18:14</a>) How AI changes what engineers need to understand</p><p>(<a href="https://www.youtube.com/watch?v=j3YuAfNCZGk&amp;t=1217s">20:17</a>) The shift to spec-driven development</p><p>(<a href="https://www.youtube.com/watch?v=j3YuAfNCZGk&amp;t=1404s">23:24</a>) How AI changes how we evaluate engineers</p><p>(<a href="https://www.youtube.com/watch?v=j3YuAfNCZGk&amp;t=1593s">26:33</a>) Rethinking engineering interviews in the age of AI</p><p>(<a href="https://www.youtube.com/watch?v=j3YuAfNCZGk&amp;t=1703s">28:23</a>) How Roblox tracks AI-generated code</p><p>(<a href="https://www.youtube.com/watch?v=j3YuAfNCZGk&amp;t=2156s">35:56</a>) Model routing and choosing the right AI tools</p><p>(<a href="https://www.youtube.com/watch?v=j3YuAfNCZGk&amp;t=2390s">39:50</a>) Measuring AI&#8217;s impact on engineering productivity</p><p>(<a href="https://www.youtube.com/watch?v=j3YuAfNCZGk&amp;t=2684s">44:44</a>) Optimizing AI costs at scale</p><p>(<a href="https://www.youtube.com/watch?v=j3YuAfNCZGk&amp;t=2892s">48:12</a>) Builders at Roblox and the future of software engineering</p><p>(<a href="https://www.youtube.com/watch?v=j3YuAfNCZGk&amp;t=3019s">50:19</a>) Building trust in autonomous code review</p><p>(<a href="https://www.youtube.com/watch?v=j3YuAfNCZGk&amp;t=3191s">53:11</a>) The cultural shift toward autonomous engineering</p><p><strong>Where to find Andrew Swerdlow:</strong></p><p>&#8226; LinkedIn: <a href="https://www.linkedin.com/in/andrewswerdlow/">https://www.linkedin.com/in/andrewswerdlow/</a></p><p>&#8226; X:  <a href="https://x.com/andrewswerdlow">https://x.com/andrewswerdlow</a></p><p>&#8226; Website: <a href="https://andrewswerdlow.net">https://andrewswerdlow.net</a></p><p><strong><span>Where to find Brian Houck:</span></strong></p><p><span>&#8226; LinkedIn: </span><a href="https://www.linkedin.com/in/brianhouck"><span>https://www.linkedin.com/in/brianhouck</span></a></p><h2><strong>Referenced:</strong></h2><p>&#8226; <a href="https://getdx.com/whitepaper/the-cafes-framework/">CAFE(S): Your Agent Is Only as Good as Its Context</a></p><p>&#8226; <a href="https://www.roblox.com/">Roblox</a></p><p>&#8226; <a href="https://www.amazon.com/dp/B0CK45BHLX">Tech Leadership by Andrew Swerdlow</a></p><p>&#8226; <a href="https://margaretstorey.com/">Margaret-Anne Storey</a></p><p>&#8226; <a href="https://www.linkedin.com/in/malteubl/">Malte Ubl - Vercel | LinkedIn</a></p><p>&#8226; <a href="https://grafana.com/">Grafana Labs</a></p><p>&#8226; <a href="https://www.linkedin.com/in/kushan02/">Kushan Mehta | LinkedIn</a></p><p>&#8226; <a href="https://engineering.fb.com/2025/08/06/developer-tools/diff-risk-score-drs-ai-risk-aware-software-development-meta/">Diff Risk Score: AI-driven risk-aware software development - Engineering at Meta</a></p>]]></content:encoded></item><item><title><![CDATA[You can’t find 10x developers, but you can build 10x organizations]]></title><description><![CDATA[Part I: The Coding War Games, a study behind the 10x developer myth.]]></description><link>https://newsletter.getdx.com/p/you-cant-find-10x-developers</link><guid isPermaLink="false">https://newsletter.getdx.com/p/you-cant-find-10x-developers</guid><dc:creator><![CDATA[Justin Reock]]></dc:creator><pubDate>Wed, 28 Oct 2026 10:02:34 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/6a0b5ccd-fe66-48e2-903f-eeb6156f97cb_2400x1260.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><em><strong>Welcome to the latest issue of Engineering Enablement,</strong> a weekly newsletter sharing research and perspectives on developer productivity.</em></p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.getdx.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe now&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.getdx.com/subscribe?"><span>Subscribe now</span></a></p><p>DX&#8217;s Q3 AI Impact Report is now available with the latest research on AI&#8217;s impact across engineering organizations. <strong><a href="https://getdx.com/report/state-of-ai-impact-in-engineering-q3-report/?utm_source=newsletter">Read the full report.</a></strong></p><div><hr></div><p><strong>Scott:</strong> One of the most pervasive myths of the software industry is the existence of a 10x developer. And just when we thought we had debunked the idea of a lone wolf coding maestro delivering at 10 times the rate of their colleagues, AI coding assistants appeared, promising 10x gains for anyone willing to master the tools.</p><p>Back in March, Venkat Venkataramani, VP of application infrastructure at OpenAI,  <a href="https://leaddev.com/ai/openai-says-there-are-easily-1000x-engineers-now">told LeadDev that</a> &#8220;there are easily 1,000x engineers now. I don&#8217;t even know if that&#8217;s the limit. There may be 1,000,000x engineers coming.&#8221; I thought this was a neat piece of marketing for their own Codex tools, but yet again this narrative misses a core principle of software engineering work: that at almost every professional level, building and shipping software is a team sport.</p><p>Picking up on this theme, our Field CTO Justin Reock is kicking off a three-part series this week, adapted from his talk: <a href="https://www.youtube.com/watch?v=4-CSYcjdWvI">Building 10x Organizations with Modern Productivity Metrics.</a> Over to you, Justin.</p><div><hr></div><p><strong>Justin:</strong> I&#8217;m going to start by going back to where the pervasive <a href="https://www.youtube.com/watch?v=4-CSYcjdWvI">10x developer idea</a> actually came from.</p><p>The 10x developer is one of the most repeated and least understood ideas in our industry. It&#8217;s used to justify hiring decisions, set compensation bands, and influence entire management philosophies. But the source material, a 1968 study called <em>&#8220;Exploratory experimental studies comparing online and offline programming performance&#8221;</em> has been largely debunked when looking at individual developers working within a team or organization.</p><p>This pivot originated from a study called &#8220;<em>the Coding War Games</em>,&#8221; run by software researchers Tom DeMarco and Tim Lister starting in 1984, which would form the backbone for their influential book &#8220;Peopleware: Productive Projects and Teams.&#8221;</p><p>More than 600 developers across almost 100 companies were asked to perform the same business task using code in their normal workflow, and asked to measure the time it took to complete the task, as well as defect rates. The spread between the fastest and slowest performers was 11.1x, and that single number contributed to the 10x developer myth.</p><p>Here&#8217;s what almost everyone gets wrong about that finding. The 11.1x spread did not cluster by individual talent, it clustered by company. When DeMarco and Lister grouped participants by employer, the top-performing companies were dramatically faster and produced dramatically fewer defects than the bottom-performing companies, and the gap inside any one company was narrow: only about a 21% difference in productivity between engineers working in the same organization.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!325d!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3975d0d9-b9f9-4d5c-99e2-bf4eb5160b9a_4200x2398.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!325d!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3975d0d9-b9f9-4d5c-99e2-bf4eb5160b9a_4200x2398.png 424w, https://substackcdn.com/image/fetch/$s_!325d!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3975d0d9-b9f9-4d5c-99e2-bf4eb5160b9a_4200x2398.png 848w, https://substackcdn.com/image/fetch/$s_!325d!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3975d0d9-b9f9-4d5c-99e2-bf4eb5160b9a_4200x2398.png 1272w, https://substackcdn.com/image/fetch/$s_!325d!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3975d0d9-b9f9-4d5c-99e2-bf4eb5160b9a_4200x2398.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!325d!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3975d0d9-b9f9-4d5c-99e2-bf4eb5160b9a_4200x2398.png" width="1456" height="831" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3975d0d9-b9f9-4d5c-99e2-bf4eb5160b9a_4200x2398.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:831,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:192431,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://newsletter.getdx.com/i/222219934?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3975d0d9-b9f9-4d5c-99e2-bf4eb5160b9a_4200x2398.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!325d!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3975d0d9-b9f9-4d5c-99e2-bf4eb5160b9a_4200x2398.png 424w, https://substackcdn.com/image/fetch/$s_!325d!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3975d0d9-b9f9-4d5c-99e2-bf4eb5160b9a_4200x2398.png 848w, https://substackcdn.com/image/fetch/$s_!325d!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3975d0d9-b9f9-4d5c-99e2-bf4eb5160b9a_4200x2398.png 1272w, https://substackcdn.com/image/fetch/$s_!325d!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3975d0d9-b9f9-4d5c-99e2-bf4eb5160b9a_4200x2398.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Years of experience, salary, and tenure barely predicted performance. The environment did. A quiet workspace, freedom from interruption, and the ability to reach and maintain flow state were the variables that moved the needle.</p><p>The top performers (middle column below) differed dramatically in their responses to questions about their working environment, and ultimately, their developer experience:</p><ul><li><p>Top performers had almost twice as much dedicated working space.</p></li><li><p>Nearly twice as many of the top performers reported that it was acceptably quiet.</p></li><li><p>Three times as many top performers reported that it was acceptably private.</p></li><li><p>Five times as many<strong> </strong>top performers were allowed to silence their phones (equivalent to Teams, Slack, etc. today), and three times as many were allowed to divert their calls.</p></li><li><p>Twice as many low performers indicated that they were often interrupted needlessly.</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!2Th5!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8b7b55d4-bb48-40a1-8abf-8f9479fbe27b_2048x1477.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!2Th5!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8b7b55d4-bb48-40a1-8abf-8f9479fbe27b_2048x1477.png 424w, https://substackcdn.com/image/fetch/$s_!2Th5!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8b7b55d4-bb48-40a1-8abf-8f9479fbe27b_2048x1477.png 848w, https://substackcdn.com/image/fetch/$s_!2Th5!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8b7b55d4-bb48-40a1-8abf-8f9479fbe27b_2048x1477.png 1272w, https://substackcdn.com/image/fetch/$s_!2Th5!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8b7b55d4-bb48-40a1-8abf-8f9479fbe27b_2048x1477.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!2Th5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8b7b55d4-bb48-40a1-8abf-8f9479fbe27b_2048x1477.png" width="1456" height="1050" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/8b7b55d4-bb48-40a1-8abf-8f9479fbe27b_2048x1477.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1050,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!2Th5!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8b7b55d4-bb48-40a1-8abf-8f9479fbe27b_2048x1477.png 424w, https://substackcdn.com/image/fetch/$s_!2Th5!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8b7b55d4-bb48-40a1-8abf-8f9479fbe27b_2048x1477.png 848w, https://substackcdn.com/image/fetch/$s_!2Th5!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8b7b55d4-bb48-40a1-8abf-8f9479fbe27b_2048x1477.png 1272w, https://substackcdn.com/image/fetch/$s_!2Th5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8b7b55d4-bb48-40a1-8abf-8f9479fbe27b_2048x1477.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>That&#8217;s the inversion I want to focus on. The idea of 10x developers, in the way most people use the phrase, are largely fictional. However, 10x organizations are real, measurable, and reproducible. The conditions that produce them are knowable. We&#8217;ve just spent forty years optimizing the wrong variables.</p><h2>Why this keeps mattering</h2><p>Every few years, our industry rediscovers developer productivity, gives it a new name, and proceeds as if no one has thought about it before. I sometimes call this YAPP: Yet Another Productivity Philosophy. Agile is YAPP. DevOps is YAPP. Platform Engineering is YAPP. AI-assisted engineering is the current YAPP.</p><p>Each one arrived with the implicit claim that the previous philosophy got it wrong. Each one ended up rediscovering the same underlying truth that DeMarco and Lister already had data for in 1984. It was already substantiated by the work of W. Edwards Deming, who famously said:</p><p style="text-align: center;">&#8220;90-95% of the productivity of an organization is <strong>determined by the system and not the worker.&#8221;</strong></p><p>Most of the variance in developer output, even with AI augmentation, is not about the developer,it&#8217;s about friction. It&#8217;s about how many times a day someone gets interrupted, how long the builds or tests take, how quickly a review is turned around, how confident they feel deploying on a Friday afternoon, whether the test suite is trustworthy, and whether they spend their week on work they consider meaningful.</p><h2>What the 10x organization actually looks like</h2><p>A few patterns appear consistently across the highest-performing engineering organizations I&#8217;ve worked with at DX, and they map cleanly onto what the original Coding War Games found.</p><p>The first is <strong>protected focus time</strong>, with emphasis on &#8220;protected.&#8221; The top performers in the 1984 study had quiet, defensible workspaces. The 2026 equivalent is meeting load, notification load, and the structural ability to spend two-to-four hours on a single problem without context-switching. Engineers at 10x organizations report meaningfully more deep work hours per week than engineers at median organizations, and the gap is not small.</p><p>The second is a <strong>smooth and efficient inner loop</strong>. The inner loop is everything a developer touches between writing a line of code and getting feedback on it: editor, local build, local test, local run. When the inner loop is fast and reliable, developers stay in flow. When it&#8217;s slow or flaky, every change becomes a context switch.</p><p>The third is <strong>trust and psychological safety.</strong> The Coding War Games organizations that performed best gave their engineers real latitude over how the work got done. That hasn&#8217;t changed. Teams that can experiment in safe-to-fail environments, work with AI without fear of displacement, and understand the worth of the value they create for the organization consistently outperform teams that can&#8217;t. This would be validated in <a href="https://rework.withgoogle.com/intl/en/guides/understand-team-effectiveness">Google&#8217;s Project Aristotle</a>, which valued psychological safety as a critical component of developer experience, and a strong leading indicator of team productivity.</p><h2>Where this series is going</h2><p>Next week, in part two, I&#8217;ll get into the measurement problem this exposes.</p><p>We have four established frameworks for measuring developer productivity now: DORA, SPACE, DevEx, and the <a href="https://getdx.com/news/introducing-the-dx-core-4/">DX Core 4</a>. Each one was a response to progress and evolution in productivity theory. Understanding why they evolved makes it easier to understand how to apply them to your organization.</p><p>Then, in part three, I&#8217;ll get specific about what a measurement program actually looks like when it&#8217;s deployed at scale, and how to keep them from collapsing into a single-metric trap.</p><div><hr></div><h2>One more thing</h2><p>We published an article about how the travel booking company trivago is attempting to 10x its organization with AI tooling. One of the big wins was consolidating usage data across multiple AI coding tools into one dashboard to evaluate how different developer cohorts leverage AI, and then scaling the approaches that work best.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Gige!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61456338-dc4f-43b8-8afe-a007170d5ec4_2400x1254.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Gige!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61456338-dc4f-43b8-8afe-a007170d5ec4_2400x1254.png 424w, https://substackcdn.com/image/fetch/$s_!Gige!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61456338-dc4f-43b8-8afe-a007170d5ec4_2400x1254.png 848w, https://substackcdn.com/image/fetch/$s_!Gige!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61456338-dc4f-43b8-8afe-a007170d5ec4_2400x1254.png 1272w, https://substackcdn.com/image/fetch/$s_!Gige!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61456338-dc4f-43b8-8afe-a007170d5ec4_2400x1254.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Gige!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61456338-dc4f-43b8-8afe-a007170d5ec4_2400x1254.png" width="1456" height="761" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/61456338-dc4f-43b8-8afe-a007170d5ec4_2400x1254.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:761,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:95810,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://newsletter.getdx.com/i/222219934?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61456338-dc4f-43b8-8afe-a007170d5ec4_2400x1254.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!Gige!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61456338-dc4f-43b8-8afe-a007170d5ec4_2400x1254.png 424w, https://substackcdn.com/image/fetch/$s_!Gige!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61456338-dc4f-43b8-8afe-a007170d5ec4_2400x1254.png 848w, https://substackcdn.com/image/fetch/$s_!Gige!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61456338-dc4f-43b8-8afe-a007170d5ec4_2400x1254.png 1272w, https://substackcdn.com/image/fetch/$s_!Gige!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61456338-dc4f-43b8-8afe-a007170d5ec4_2400x1254.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://getdx.com/customers/trivago-measures-ai-roi-with-dx/?utm_source=newsletter&quot;,&quot;text&quot;:&quot;Read more&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://getdx.com/customers/trivago-measures-ai-roi-with-dx/?utm_source=newsletter"><span>Read more</span></a></p><div><hr></div><p>That&#8217;s it for this week. Thanks for reading.<br>-Justin</p>]]></content:encoded></item><item><title><![CDATA[AI made developers faster. Is their judgment keeping up?]]></title><description><![CDATA[Is AI just making developers faster, or is it opening up entirely new possibilities?]]></description><link>https://newsletter.getdx.com/p/ai-made-developers-faster-is-their</link><guid isPermaLink="false">https://newsletter.getdx.com/p/ai-made-developers-faster-is-their</guid><dc:creator><![CDATA[Brian Houck]]></dc:creator><pubDate>Wed, 21 Oct 2026 10:03:02 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/8ecb32d1-fda1-4890-9433-b347ee5a96dc_2400x1260.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><em><strong>Welcome to the latest issue of Engineering Enablement,</strong> a weekly newsletter sharing research and perspectives on developer productivity.</em></p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.getdx.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe now&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.getdx.com/subscribe?"><span>Subscribe now</span></a></p><p>DX&#8217;s Q3 AI Impact Report is now available with the latest research on AI&#8217;s impact across engineering organizations. <a href="https://getdx.com/report/state-of-ai-impact-in-engineering-q3-report/?utm_source=newsletter">Read the full report.</a></p><div><hr></div><p><strong>Scott: </strong>I&#8217;ve been sensing a real anxiety amongst developers over the past year or so around skill and knowledge atrophy. We know that AI tools like Cursor and Claude make developers faster &#8211; or at least <em>feel </em>faster &#8211; but at what cost?</p><p>Brian and his fellow researchers have now given that anxiety a name: the hollow orchestrator. A developer can easily find themselves approving more work each month, without becoming better equipped to judge it.</p><p>To dig into that trend and more findings from his latest study into how AI is impacting 331 developers at Microsoft, I&#8217;ll hand things over to Brian.</p><div><hr></div><p><strong>Brian:</strong> When factories were first electrified, almost nothing happened. Owners pulled the steam engine out, dropped an electric motor in its place, and left the machines where they were, still arranged around the old central driveshaft. Productivity barely moved, for decades. The payoff came later, once the work itself had been rebuilt around what electricity could actually do, with a small motor on every machine, instead of one dynamo turning the whole floor.</p><p>I found myself thinking about that period when looking at our <a href="http://getdx.com/report/state-of-ai-impact-in-engineering-q3-report/">Q3 State of AI Impact in Engineering</a>. Drawing on data from more than 500 engineering organizations, it&#8217;s clear that the engine is spinning faster. Median weekly PR Throughput grew 75.5% over the last four quarters. But that doesn&#8217;t mean we&#8217;re delivering more value. In fact, the share of time going to new capabilities rather than maintenance and support, rose just two percentage points, from 57% to 59%.</p><p>Our speed metrics don&#8217;t tell us what work has become possible that wasn&#8217;t before. A task we regularly do can get faster with AI. But, a task can also become cheap enough to attempt for the first time. Those are very different kinds of gains.</p><p>To explore that distinction, I teamed up with researchers at Microsoft and Oregon State University: Rudrajit Choudhuri, Jenna Butler, Christian Bird, Carmen Badea, Anita Sarma, and I invited 4,000 randomly sampled Microsoft developers to participate in a study. We analyzed 331 responses, which became the basis <a href="https://arxiv.org/abs/2610.23708">for our new paper</a>.</p><p>We asked developers a seemingly simple question: <em>Is there anything you can now do at work because of AI that you couldn&#8217;t do before?</em></p><p>Their answers were then sorted into three zones:</p><p><strong>Faster </strong>was familiar work like writing documentation or authoring tests, just now done more quickly. 45% of the responses fit into this bucket.</p><p><strong>Further </strong>was work that had been out of reach, either because it needed skills or capacity the developer didn&#8217;t have, or because it was never worth the effort. 36.9% of tasks landed here.</p><p><strong>Afresh</strong> was work that didn&#8217;t exist before, largely built around directing agents and setting the standards their output has to meet. That accounted for 12.4%. The remaining 5.7% named no specific task.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!eQ0m!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F007dbd1b-71ae-4d13-b5e6-8e88d755a79a_4200x1884.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!eQ0m!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F007dbd1b-71ae-4d13-b5e6-8e88d755a79a_4200x1884.png 424w, https://substackcdn.com/image/fetch/$s_!eQ0m!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F007dbd1b-71ae-4d13-b5e6-8e88d755a79a_4200x1884.png 848w, https://substackcdn.com/image/fetch/$s_!eQ0m!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F007dbd1b-71ae-4d13-b5e6-8e88d755a79a_4200x1884.png 1272w, https://substackcdn.com/image/fetch/$s_!eQ0m!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F007dbd1b-71ae-4d13-b5e6-8e88d755a79a_4200x1884.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!eQ0m!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F007dbd1b-71ae-4d13-b5e6-8e88d755a79a_4200x1884.png" width="1456" height="653" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/007dbd1b-71ae-4d13-b5e6-8e88d755a79a_4200x1884.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:653,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:232414,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://newsletter.getdx.com/i/221168024?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F007dbd1b-71ae-4d13-b5e6-8e88d755a79a_4200x1884.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!eQ0m!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F007dbd1b-71ae-4d13-b5e6-8e88d755a79a_4200x1884.png 424w, https://substackcdn.com/image/fetch/$s_!eQ0m!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F007dbd1b-71ae-4d13-b5e6-8e88d755a79a_4200x1884.png 848w, https://substackcdn.com/image/fetch/$s_!eQ0m!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F007dbd1b-71ae-4d13-b5e6-8e88d755a79a_4200x1884.png 1272w, https://substackcdn.com/image/fetch/$s_!eQ0m!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F007dbd1b-71ae-4d13-b5e6-8e88d755a79a_4200x1884.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>These zones are not invisible to our dashboards. A feature that was previously out of reach can still produce pull requests. A new agent workflow can still have a cycle time.</p><p>What those measures don&#8217;t tell us is whether we accelerated familiar work, made something previously impractical worth doing, or created a new kind of engineering responsibility. We can count the output without understanding what AI changed.</p><h2>Faster: familiar work, accelerated</h2><p>The Faster accounts were largely unsurprising, and several were refreshingly blunt about it.</p><blockquote><p>&#8220;<em>I could do everything I currently do without AI; however, I can do my work faster</em>&#8220; (P5).</p></blockquote><p>Documentation drafted in minutes, root cause analysis under pressure, test coverage in an afternoon. What caught my attention in this zone wasn&#8217;t the speed. It was the reorganization around the speed.</p><blockquote><p>&#8220;<em>I used to only do one thing at a time. But now I will do two or three while waiting for AI to respond to each problem</em>&#8220; (P274).</p></blockquote><p>Faster implementation still has to get through review, testing, and delivery. Those constraints remain important. But some developers described the work they could now do that they would never previously have attempted, either because they didn&#8217;t have the time, or skills to do it.</p><h2>Further: the work that used to be out of reach</h2><p>For engineering leaders struggling to articulate the return on investment from an exponentially increasing token spend, this is a good place to start.</p><p>About 24% of responses described working past a limit the developer had before AI. As one back-end engineer told us:</p><blockquote><p><em>&#8220;I can cover end-to-end now. Previously, I was limited to only the backend</em>&#8220; (P28).</p></blockquote><p>Another 12.7% of all responses described work developers could have done, but could never justify doing.. One response lays the economics out completely:</p><blockquote><p>&#8220;<em>I recently automated </em>[a task]<em> in an hour that saved me 80 hours of manual work. I never would have done those 80 hours, so those items would never have been updated. Writing the automation without AI would have probably taken me 30 hours, which would still have been too hard to justify the investment.</em>&#8220; (P197)</p></blockquote><p>No one reclaimed 80 hours, because no one was going to spend them. The gain was getting useful work done that would otherwise have stayed undone. This is an entirely different kind of productivity gain. We tend to measure how much work gets done, or how quickly it gets done, but rarely ask what work has become worth doing in the first place. A maintenance task that was too expensive to justify yesterday might suddenly be worthwhile today. And that&#8217;s a benefit our traditional productivity measures don&#8217;t necessarily help us recognize.</p><h2>Afresh: new work created around AI</h2><p>The smallest zone, 12.4% of responses, captures new responsibilities emerging around AI tools. Supervising agents and building context management systems for them are examples of tasks that never existed before AI. For leaders tasked with AI transformation, these tasks offer a useful glimpse of how the work itself is changing.</p><p>Some of it is role change: &#8220;My role has shifted from executing every step myself to orchestrating work streams and evaluating outcomes&#8221; (P116). On one team, &#8220;more than 90% of the code is AI-written. We act as a protective layer, as a manager of agents&#8221; (P114).</p><p>Just because an agent stops working does not mean the task is finished. Maybe the agent got stuck, maybe it didn&#8217;t know what &#8220;done&#8221; looked like, maybe it misunderstood the assignment completely. As one respondent put it, &#8220;<em>Right now AI sort of just stops. There&#8217;s some probabilistic mechanism that determines it is time to await a human reply</em>.&#8221; Developers described building tools to check an agent&#8217;s output against explicit criteria for a completed task. That shifts some of their work from producing the implementation to defining what &#8220;done&#8221; means and checking whether the agent actually got there.</p><p>Developers are also building tools to make AI&#8217;s output easier for people to evaluate. One participant was working on a tool to break large code changes into &#8220;<em>smaller logically-grouped chunks</em>&#8221; (P316). Rather than expecting someone to make sense of a 90-file pull request all at once, the tool would organize related changes for review. The new work is not simply reviewing more code. It&#8217;s building tools that make large, agent-generated changes easier to review.</p><p>This is the work around AI, not simply work handed to AI. Someone has to define what good looks like, recognize when an agent has missed it, and build the context and checks that make the next attempt better. Some of this is opportunity. Some is the cost of making AI useful. But as AI expands what developers can do, how do we make sure their judgment keeps up?</p><h2>Two learners, one loop</h2><p>Across all three zones, developers described learning from AI while also teaching it how their work should be done. They used it to take on unfamiliar problems, then captured their own expertise in specifications, examples, tests, and standing instructions.</p><p>Drawing on <a href="https://doi.org/10.2307/j.ctvjf9vz4">Vygotsky&#8217;s zone of proximal development</a>, we describe this as two-way scaffolding. The developer builds understanding and the agent benefits from better guidance in subsequent work. It&#8217;s not that every interaction retrains the model, but context can improve, use cases sharpen, and skills develop.</p><p>The focal point is what we call the <strong>skill&#8211;judgment nexus</strong>. AI can provide skills beyond what it can reliably judge, and developers must bridge that gap with their own judgment. As AI&#8217;s capabilities expand, developers need to grow their judgment alongside them.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!_R2M!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73274638-627a-4dba-9d74-912d87b75eee_1920x1080.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!_R2M!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73274638-627a-4dba-9d74-912d87b75eee_1920x1080.png 424w, https://substackcdn.com/image/fetch/$s_!_R2M!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73274638-627a-4dba-9d74-912d87b75eee_1920x1080.png 848w, https://substackcdn.com/image/fetch/$s_!_R2M!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73274638-627a-4dba-9d74-912d87b75eee_1920x1080.png 1272w, https://substackcdn.com/image/fetch/$s_!_R2M!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73274638-627a-4dba-9d74-912d87b75eee_1920x1080.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!_R2M!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73274638-627a-4dba-9d74-912d87b75eee_1920x1080.png" width="1456" height="819" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/73274638-627a-4dba-9d74-912d87b75eee_1920x1080.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:819,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:282433,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://newsletter.getdx.com/i/221168024?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73274638-627a-4dba-9d74-912d87b75eee_1920x1080.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!_R2M!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73274638-627a-4dba-9d74-912d87b75eee_1920x1080.png 424w, https://substackcdn.com/image/fetch/$s_!_R2M!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73274638-627a-4dba-9d74-912d87b75eee_1920x1080.png 848w, https://substackcdn.com/image/fetch/$s_!_R2M!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73274638-627a-4dba-9d74-912d87b75eee_1920x1080.png 1272w, https://substackcdn.com/image/fetch/$s_!_R2M!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73274638-627a-4dba-9d74-912d87b75eee_1920x1080.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>When a developer takes the time to understand a solution, they become better able to judge the next attempt. When they capture what good looks like, subsequent agent work can benefit from that guidance. That is the loop we want to support: greater capability accompanied by growing judgment, rather than increasing dependence.</p><h2>The hollow-orchestrator</h2><p>The loop can also stall. A developer may approve more work each month without becoming better equipped to judge it. We describe the risk as becoming a <strong>hollow orchestrator</strong>. It&#8217;s the state of keeping responsibility for work you can no longer reliably understand.</p><p>One participant described their domain knowledge as <em>&#8220;the bulwark against fabrications or hallucinations impacting my work,</em>&#8221; but worried that &#8220;<em>as my own domain knowledge gets stale, the chances that I miss something increase</em>&#8221; (P320).</p><p>The uncomfortable part is how slowly this might happen. Speed metrics won&#8217;t catch it. They&#8217;ll look good right up until they don&#8217;t.</p><p>Which is why I treat quality as a guardrail, and why one small number in last week&#8217;s Q3 report has my attention. Change failure rate had been largely flat over the last year, but the median quarter-over-quarter change increased by roughly +0.5 percentage points from Q2 to Q3 this year. That&#8217;s a relatively small movement, and there are many possible explanations. But it&#8217;s the number I&#8217;ll be watching most closely next quarter.</p><p>We don&#8217;t have evidence that declining developer judgment is behind this increase. But as AI takes on more of the work, our ability to recognize and prevent failures becomes increasingly important. If that ability starts to erode, we need measures that help us recognize it before the consequences compound.</p><p>Capacity bought by eroding judgment is borrowed from next year.</p><h2>Where to start</h2><p>There&#8217;s two things you can do here, one is cheap and one is harder.</p><p>Ask your developers whether there is anything they can now do at work because of AI that they couldn&#8217;t do before. Use the three zones as a lens to view their answers, then ask what made the work possible and why it is valuable. Ask again in six months to see how those experiences are changing. The goal isn&#8217;t to move everyone toward the afresh zone; it is to understand what AI is changing about the work.</p><p>The harder one is protecting the loop. The paper lays out nine scaffolding functions adapted from the tutoring and cognitive apprenticeship literature. Three are worth calling out here:</p><ul><li><p><strong>Show, don&#8217;t just instruct</strong>. &#8220;Handle errors better&#8221; tells an agent almost nothing. A pair of examples, one that meets your team&#8217;s standard and one that nearly does, tells it where the line actually falls.</p></li><li><p><strong>Size delegation to what you can verify</strong>. Split work so the part you hand over matches the part you can check. The failure mode is delegating more than you can verify, then signing off because you assume the agent got it right, not because you&#8217;ve verified the result.</p></li><li><p><strong>Judge AI&#8217;s work against an independent standard.</strong> Use requirements, tests, evaluations, and reviewers to challenge the result. A different model can help, but another agent agreeing is not enough. The checks still need to establish whether the work meets the intended behavior.</p></li></ul><h2>Final thoughts</h2><p>I don&#8217;t think we need to stop measuring speed. I think we need to stop relying on it to explain everything. Faster describes familiar work accelerated. Further asks what has become possible. Afresh makes visible the new responsibilities involved in working with AI models. Together, they give us a better set of questions, not a replacement productivity score.</p><p>Those questions belong alongside another: are developers building the judgment to sustain their expanded reach? It&#8217;s possible to become more capable with AI without becoming more capable of evaluating what it produces. The paper&#8217;s two-way scaffolding practices are intended to help close that gap.</p><p>The factory owners couldn&#8217;t have told you everything electricity would eventually make possible, in the same way that I can&#8217;t tell you what AI will. But we can ask about more than the speed of our existing work.</p><p>As we continue in our AI transformation journeys, it&#8217;s not &#8220;how much faster are our developers with AI?&#8221; but &#8220;what can this organization do now that it couldn&#8217;t before, and are we building the judgment to keep doing it?&#8221;</p><div><hr></div><h2>One more thing</h2><p>We just published an article about how the travel booking company trivago is attempting to 10x its engineering team with AI tooling. One of the big wins was consolidating usage data across multiple AI coding tools into one dashboard to evaluate how different developer cohorts leverage AI, and then scaling the approaches that work best.</p><p>&#8220;Anyone can maximize token usage. We wanted to maximize output. DX moves us from counting tokens to counting outcomes: pull requests shipped, cycle time, review load, so productivity gains show up in what we ship, not just in what we spend,&#8221; Carolina Muradas, Strategy and Operations Lead at trivago, said.</p><p><a href="https://getdx.com/customers/trivago-measures-ai-roi-with-dx/">Here&#8217;s all the details.</a></p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.getdx.com/p/ai-made-developers-faster-is-their?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.getdx.com/p/ai-made-developers-faster-is-their?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><p></p>]]></content:encoded></item><item><title><![CDATA[The CAFE(S) framework: Improving AI agent effectiveness]]></title><description><![CDATA[Brian Houck and his co-authors explore how the CAFE(S) framework helps engineering teams improve context quality, reduce costly AI failures, and make agents more effective.]]></description><link>https://newsletter.getdx.com/p/the-cafes-framework-improving-ai</link><guid isPermaLink="false">https://newsletter.getdx.com/p/the-cafes-framework-improving-ai</guid><dc:creator><![CDATA[Brian Houck]]></dc:creator><pubDate>Fri, 16 Oct 2026 14:21:30 GMT</pubDate><enclosure url="https://api.substack.com/feed/podcast/220404919/9d708e59ab8c191cdf46753b74c8f7d9.mp3" length="0" type="audio/mpeg"/><content:encoded><![CDATA[<p>Listen and watch now on <strong><a href="https://youtu.be/V32mImG3Lg8">YouTube</a>, <a href="https://podcasts.apple.com/us/podcast/engineering-enablement-by-abi-noda/id1619140476">Apple</a>, and <a href="https://open.spotify.com/show/3NxjyIsuxeDMQtisDqBy7D">Spotify</a></strong>.</p><p>As AI agents become more capable, context quality is increasingly the limiting factor in their performance. In this episode of <em>Engineering Enablement</em>, I&#8217;m joined by my CAFE(S) co-authors Max Kanat-Alexander (Capital One), Eirini Kalliamvakou (DX), and Margaret-Anne Storey (University of Victoria) to introduce our framework for evaluating whether agents have the context they need to do their jobs well.</p><p>We introduce the five dimensions of CAFE(S): Clarity, Actionability, Fidelity, Efficiency, and Security. We walk through common context failures, how poor context accrues technical and cognitive debt, what it costs in developer productivity and operating spend, how AI is changing the skills engineers need, and why context review could become as important as code review.</p><div id="youtube2-V32mImG3Lg8" class="youtube-wrap" data-attrs="{&quot;videoId&quot;:&quot;V32mImG3Lg8&quot;,&quot;startTime&quot;:null,&quot;endTime&quot;:null}" data-component-name="Youtube2ToDOM"><div class="youtube-inner"><iframe src="https://www.youtube-nocookie.com/embed/V32mImG3Lg8?rel=0&amp;autoplay=0&amp;showinfo=0&amp;enablejsapi=0" frameborder="0" loading="lazy" gesture="media" allow="autoplay; fullscreen" allowautoplay="true" allowfullscreen="true" width="728" height="409"></iframe></div></div><h2><strong>Some takeaways: </strong></h2><h3><strong>Why context quality matters for AI agents</strong></h3><ul><li><p><strong>Improving models and harnesses isn&#8217;t enough to maximize AI agent effectiveness.</strong> The quality of the context agents receive is just as important, yet organizations rarely evaluate it systematically.</p></li><li><p><strong>Poor context creates compounding costs and risks.</strong> Ambiguous instructions, irrelevant information, and insecure context can lead to runaway token usage, incorrect outputs, security breaches, and additional developer toil.</p></li></ul><h3><strong>The five dimensions of the CAFE(S) framework</strong></h3><ul><li><p><strong>CAFE(S) defines five properties of high-quality context: clarity, actionability, fidelity, efficiency, and security.</strong> Each addresses a distinct failure mode and helps identify where context needs improvement.</p></li><li><p><strong>Clarity may be the most important dimension because ambiguity is often invisible to the person providing instructions.</strong> Agents can&#8217;t reliably infer unstated assumptions, and an incorrect interpretation can compound throughout a task.</p></li></ul><h3><strong>Why model failures are often context failures</strong></h3><ul><li><p><strong>Agents frequently make mistakes because critical knowledge exists only in developers&#8217; heads.</strong> Without documented requirements, constraints, and decisions, agents may build unwanted features or take actions that conflict with the team&#8217;s intent.</p></li><li><p><strong>Security failures can stem from giving agents access to information they shouldn&#8217;t have.</strong> One real-world example involved an agent finding an unrelated API credential and deleting a production database.</p></li></ul><h3><strong>How poor context contributes to technical, cognitive, and intent debt</strong></h3><ul><li><p><strong>Undocumented goals, constraints, and decisions create intent debt that can become more costly with AI agents.</strong> Agents need access to the reasoning behind decisions, including what a system should not do.</p></li><li><p><strong>Better context can help prevent debt, but successful agent workflows can also encourage complacency.</strong> Teams may stop documenting important decisions when agents appear to be working well.</p></li></ul><h3><strong>Creating high-quality context is becoming a core engineering skill</strong></h3><ul><li><p><strong>AI fluency involves learning to anticipate what agents need rather than correcting mistakes after they happen.</strong> More experienced developers proactively define goals, constraints, and success criteria based on lessons from previous failures.</p></li><li><p><strong>Skills traditionally associated with senior engineers may become essential at every level. </strong>Clearly communicating intent and requirements has long been part of senior engineering roles, but AI agents are making those skills increasingly important for all developers.</p></li></ul><h3><strong>Why context review could become as important as code review</strong></h3><ul><li><p><strong>Shared context deserves more scrutiny because its effects multiply across agent interactions.</strong> A mistake in a widely used instructions file can be repeated in every session, while an improvement can make every agent more effective.</p></li><li><p><strong>Context review should be proportional to the importance and reach of the information.</strong> Critical shared instructions may need formal review, while lightweight documentation updates should remain easy to make.</p></li><li><p><strong>Reviewing context can also build shared understanding across engineering teams.</strong> Like code review, the process can help developers align on requirements, decisions, and the intended behavior of their systems.</p></li></ul><h2><strong>In this episode, we cover:</strong></h2><p>(<a href="https://www.youtube.com/watch?v=V32mImG3Lg8">00:00</a>) Intro</p><p>(<a href="https://www.youtube.com/watch?v=V32mImG3Lg8&amp;t=78s">01:18</a>) Meet the CAFE(S) authors</p><p>(<a href="https://www.youtube.com/watch?v=V32mImG3Lg8&amp;t=193s">03:13</a>) Introducing CAFE(S): Why context quality matters</p><p>(<a href="https://www.youtube.com/watch?v=V32mImG3Lg8&amp;t=392s">06:32</a>) Common context failure modes</p><p>(<a href="https://www.youtube.com/watch?v=V32mImG3Lg8&amp;t=510s">08:30</a>) The five dimensions of high-quality context</p><p>(<a href="https://www.youtube.com/watch?v=V32mImG3Lg8&amp;t=753s">12:33</a>) When model failures are actually context failures</p><p>(<a href="https://www.youtube.com/watch?v=V32mImG3Lg8&amp;t=1104s">18:24</a>) Why the panel prioritizes clarity in CAFE(S)</p><p>(<a href="https://www.youtube.com/watch?v=V32mImG3Lg8&amp;t=1354s">22:34</a>) How context quality relates to technical, cognitive, and intent debt</p><p>(<a href="https://www.youtube.com/watch?v=V32mImG3Lg8&amp;t=1535s">25:35</a>) How human interventions reveal context failures</p><p>(<a href="https://www.youtube.com/watch?v=V32mImG3Lg8&amp;t=1691s">28:11</a>) How context quality relates to AI fluency</p><p>(<a href="https://www.youtube.com/watch?v=V32mImG3Lg8&amp;t=1828s">30:28</a>) How AI is changing what makes a good engineer</p><p>(<a href="https://www.youtube.com/watch?v=V32mImG3Lg8&amp;t=2028s">33:48</a>) The case for context review as a discipline</p><p>(<a href="https://www.youtube.com/watch?v=V32mImG3Lg8&amp;t=2324s">38:44</a>) Wrap-up</p><p><strong>Where to find Margaret-Anne Storey:</strong></p><p>&#8226; LinkedIn: <a href="https://www.linkedin.com/in/margaret-anne-storey">https://www.linkedin.com/in/margaret-anne-storey</a></p><p>&#8226; X:  <a href="https://x.com/margaretstorey">https://x.com/margaretstorey</a></p><p>&#8226; Website: <a href="https://margaretstorey.com">https://margaretstorey.com</a></p><p><strong>Where to find Max Kanat-Alexander:</strong></p><p>&#8226; LinkedIn: <a href="https://www.linkedin.com/in/mkanat">https://www.linkedin.com/in/mkanat</a></p><p>&#8226; X: <a href="https://x.com/mkanat">https://x.com/mkanat</a></p><p>&#8226; Website: <a href="https://www.codesimplicity.com">https://www.codesimplicity.com</a></p><p><strong>Where to find Eirini Kalliamvakou:</strong></p><p>&#8226; LinkedIn: <a href="https://www.linkedin.com/in/eirini-kalliamvakou-1016865">https://www.linkedin.com/in/eirini-kalliamvakou-1016865</a></p><p>&#8226; X: <a href="https://x.com/irina_kAl">https://x.com/irina_kAl</a></p><p><strong>Where to find Brian Houck:</strong></p><p>&#8226; LinkedIn: <a href="https://www.linkedin.com/in/brianhouck">https://www.linkedin.com/in/brianhouck</a></p><h2><strong>Referenced:</strong></h2><p>&#8226; <a href="https://getdx.com/whitepaper/the-cafes-framework/">CAFE(S): Your Agent Is Only as Good as Its Context</a></p><p>&#8226; <a href="https://nicolefv.com/">Nicole Forsgren</a></p><p>&#8226; <a href="https://annievella.com/posts/the-productivity-experience-paradox/">The Productivity-Experience Paradox - Annie Vella</a></p><p>&#8226; <a href="https://startup-studio.ai/">Startup Studio</a></p><p>&#8226; <a href="https://arxiv.org/abs/2603.22106">From Technical Debt to Cognitive and Intent Debt: Rethinking Software Health in the Age of AI</a></p><p>&#8226; <a href="https://getdx.com/blog/the-ai-native-developer/">The AI-native developer</a></p><p>&#8226; <a href="https://codescene.com/">CodeScene</a></p>]]></content:encoded></item><item><title><![CDATA[The gap between code velocity and trust is growing]]></title><description><![CDATA[Throughput is up 75.5% in a year, but organizations with the most AI-authored code wait 105% longer for a first review, according to our latest State of AI in Engineering data.]]></description><link>https://newsletter.getdx.com/p/the-gap-between-code-velocity-and</link><guid isPermaLink="false">https://newsletter.getdx.com/p/the-gap-between-code-velocity-and</guid><dc:creator><![CDATA[Grace Fu]]></dc:creator><pubDate>Wed, 14 Oct 2026 10:47:59 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/165cbf75-6cb7-4cad-8eed-b5ef90c87a3b_2400x1254.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><em><strong>Welcome to the latest issue of Engineering Enablement,</strong> a weekly newsletter sharing research and perspectives on developer productivity.</em></p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.getdx.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe now&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.getdx.com/subscribe?"><span>Subscribe now</span></a></p><div><hr></div><p>Today, we published our <strong><a href="https://getdx.com/report/state-of-ai-impact-in-engineering-q3-report/?utm_source=newsletter">Q3 State of AI Impact in Engineering Report</a>,</strong> with new data on how AI coding tools continue to shift the role of software engineers. </p><p>This is the first time I have felt closer to answering a nagging question: If AI is making engineers more productive, where does that earned time go?</p><p>Our data suggests that the answer largely is on maintenance and fixes, yet satisfaction with code maintainability tasks only improved by 0.6% this quarter. Which brings me to another question: why? Is it because the AI tools we have today are uniquely suited to these tasks, or is that just where most of us decided to point them first?</p><p>To dig into that question, and a few others, I&#8217;ll hand over to <span class="mention-wrap" data-attrs="{&quot;name&quot;:&quot;Grace Fu&quot;,&quot;id&quot;:496785640,&quot;type&quot;:&quot;user&quot;,&quot;url&quot;:null,&quot;photo_url&quot;:&quot;https://substackcdn.com/image/fetch/$s_!6JzB!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7927597d-e123-4494-bcea-2d77312569eb_907x907.jpeg&quot;,&quot;uuid&quot;:&quot;4017a4a3-6ddc-4491-91ae-32a60c9ed4fb&quot;}" data-component-name="MentionToDOM"></span>, the report&#8217;s author, to talk you through some of those key findings.</p><div><hr></div><p><strong>Grace:</strong> In my experience, the adoption of digital tools follows a familiar arc. Initial excitement leads to significant early gains, followed by a flattening of value recognition as bottlenecks appear elsewhere. Looking at our Q3 State of AI in Engineering report, I was fully anticipating some kind of plateau.</p><p>The data surprised me. Median weekly TrueThroughput, our complexity-weighted PR throughput metric, is still climbing, up 75.5% over the last four quarters. Velocity isn&#8217;t showing any signs of slowing down. What is struggling to keep up is the rest of the SDLC &#8211; the review, testing, and verification that turn code into something a team can trust.</p><h3>Review capacity is a confirmed bottleneck</h3><p>In Q3 2026, organizations reporting the highest AI code volume waited 105% longer for a first review than those with the lowest.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!BsN_!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87b9e819-859a-483d-9305-cc9a8f588f29_2048x1363.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!BsN_!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87b9e819-859a-483d-9305-cc9a8f588f29_2048x1363.png 424w, https://substackcdn.com/image/fetch/$s_!BsN_!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87b9e819-859a-483d-9305-cc9a8f588f29_2048x1363.png 848w, https://substackcdn.com/image/fetch/$s_!BsN_!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87b9e819-859a-483d-9305-cc9a8f588f29_2048x1363.png 1272w, https://substackcdn.com/image/fetch/$s_!BsN_!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87b9e819-859a-483d-9305-cc9a8f588f29_2048x1363.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!BsN_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87b9e819-859a-483d-9305-cc9a8f588f29_2048x1363.png" width="1456" height="969" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/87b9e819-859a-483d-9305-cc9a8f588f29_2048x1363.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:969,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!BsN_!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87b9e819-859a-483d-9305-cc9a8f588f29_2048x1363.png 424w, https://substackcdn.com/image/fetch/$s_!BsN_!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87b9e819-859a-483d-9305-cc9a8f588f29_2048x1363.png 848w, https://substackcdn.com/image/fetch/$s_!BsN_!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87b9e819-859a-483d-9305-cc9a8f588f29_2048x1363.png 1272w, https://substackcdn.com/image/fetch/$s_!BsN_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87b9e819-859a-483d-9305-cc9a8f588f29_2048x1363.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>PR size seems to be a big part of the picture. Median PR size is up 122% year over year, so there&#8217;s simply more code to reviewWhat our data can&#8217;t tell us is whether something else is going on, and our team has two theories.</p><p>The first is that AI-generated changesets take more effort to review, which ties up reviewers and pushes everything else back in the queue. The second is that reviewers hesitate to pick up large, unfamiliar, or AI-generated diffs because of the cognitive friction and the fear of missing a non-obvious bug, which delays the first review directly.</p><h3>Individual relief versus systemic friction</h3><p>Developers are telling us that AI is making parts of their work meaningfully better. Our data tells us where the relief is concentrated. We measure this through the Developer Experience Index (DXI), and the figures below are quarter-over-quarter changes in the factors that make it up. </p><p>While all factors saw increases from last quarter, these comparisons capture a small point in time, and small shifts at this granularity can look more meaningful than they are. These gains should be read as directional rather than as evidence of a sustained trend until they hold across additional quarters.</p><p>Satisfaction with isolated, diagnostic tasks has risen &#8211; production debugging is up 13.3% and documentation is up 10.4%. These tasks have clear inputs, verifiable outputs, and short feedback loops, tasks that AI tools typically perform well on.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!3Eyy!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e07fbbb-fe80-4634-bd35-c1aa5a23e8fb_2048x1363.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!3Eyy!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e07fbbb-fe80-4634-bd35-c1aa5a23e8fb_2048x1363.png 424w, https://substackcdn.com/image/fetch/$s_!3Eyy!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e07fbbb-fe80-4634-bd35-c1aa5a23e8fb_2048x1363.png 848w, https://substackcdn.com/image/fetch/$s_!3Eyy!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e07fbbb-fe80-4634-bd35-c1aa5a23e8fb_2048x1363.png 1272w, https://substackcdn.com/image/fetch/$s_!3Eyy!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e07fbbb-fe80-4634-bd35-c1aa5a23e8fb_2048x1363.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!3Eyy!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e07fbbb-fe80-4634-bd35-c1aa5a23e8fb_2048x1363.png" width="1456" height="969" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/9e07fbbb-fe80-4634-bd35-c1aa5a23e8fb_2048x1363.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:969,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!3Eyy!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e07fbbb-fe80-4634-bd35-c1aa5a23e8fb_2048x1363.png 424w, https://substackcdn.com/image/fetch/$s_!3Eyy!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e07fbbb-fe80-4634-bd35-c1aa5a23e8fb_2048x1363.png 848w, https://substackcdn.com/image/fetch/$s_!3Eyy!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e07fbbb-fe80-4634-bd35-c1aa5a23e8fb_2048x1363.png 1272w, https://substackcdn.com/image/fetch/$s_!3Eyy!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9e07fbbb-fe80-4634-bd35-c1aa5a23e8fb_2048x1363.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>The smallest gains are in the tasks that require holding a whole system in view. Code maintainability (+0.6%), clear project direction (+2.0%), and the end-to-end experience of moving code safely into production, measured as perceived ease of delivery, improved by only 3%. AI is seeming to help developers navigate existing points of friction faster, but not able to remove more complex systemic friction.</p><h3>The innovation plateau</h3><p>Despite reported time savings, the share of time spent on new features has barely moved. Innovation ratio has increased by only one percentage point, a statistically significant (p = 0.002) but small gain.</p><p>We previously hypothesized that time saved from AI is going toward addressing existing backlogs, and new telemetry from our AI Code Insights daemon appears to validate this.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!ZYbw!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F345e34e0-ad3e-48bc-889b-2f4e20a0a6ca_2048x1235.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ZYbw!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F345e34e0-ad3e-48bc-889b-2f4e20a0a6ca_2048x1235.png 424w, https://substackcdn.com/image/fetch/$s_!ZYbw!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F345e34e0-ad3e-48bc-889b-2f4e20a0a6ca_2048x1235.png 848w, https://substackcdn.com/image/fetch/$s_!ZYbw!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F345e34e0-ad3e-48bc-889b-2f4e20a0a6ca_2048x1235.png 1272w, https://substackcdn.com/image/fetch/$s_!ZYbw!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F345e34e0-ad3e-48bc-889b-2f4e20a0a6ca_2048x1235.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!ZYbw!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F345e34e0-ad3e-48bc-889b-2f4e20a0a6ca_2048x1235.png" width="1456" height="878" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/345e34e0-ad3e-48bc-889b-2f4e20a0a6ca_2048x1235.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:878,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!ZYbw!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F345e34e0-ad3e-48bc-889b-2f4e20a0a6ca_2048x1235.png 424w, https://substackcdn.com/image/fetch/$s_!ZYbw!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F345e34e0-ad3e-48bc-889b-2f4e20a0a6ca_2048x1235.png 848w, https://substackcdn.com/image/fetch/$s_!ZYbw!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F345e34e0-ad3e-48bc-889b-2f4e20a0a6ca_2048x1235.png 1272w, https://substackcdn.com/image/fetch/$s_!ZYbw!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F345e34e0-ad3e-48bc-889b-2f4e20a0a6ca_2048x1235.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>About 33% of sessions go to maintenance and 23.3% to fixes, against only 11% on new capability. Code review is among the smallest use cases. That fits the flat ratio, and it&#8217;s the clearest direct evidence we have that AI usage is concentrated in keeping existing systems running, not towards innovation.</p><h3>A framework for context quality</h3><p>This quarter&#8217;s report also reframes many of these friction points using <strong><a href="https://getdx.com/research/cafe-s-your-agent-is-only-as-good-as-its-context/?utm_source=newsletter">our newly released CAFE(S) framework</a></strong>, which can be used to evaluate the quality of the context supplied to AI systems across five dimensions: Clarity, Actionability, Fidelity, Efficiency, and Security.</p><p>Our working hypothesis is that many of these friction areas are being amplified by context constraints, not model capabilities. When agents operate without sufficient architectural intent or clear boundaries, the resulting code is functional but costly to verify, and that cost surfaces downstream, in review queues and delivery friction.</p><p>In our report, we link these industry trends to CAFE(S) diagnoses and provide some recommendations for how to address growing bottlenecks. Some of my top recommendations are:</p><ol><li><p><strong>Make intent explicit and machine-readable.</strong> Unstated rationale, constraints, and goals are invisible while humans do the work but extremely costly once agents do. Put architectural constraints, ownership boundaries, and acceptance criteria in repos and in a form a model can load.</p></li><li><p><strong>Re-plan around delivery capacity, not generation capacity. </strong>Consistent throughput growth is promising, but it&#8217;s arriving alongside longer review waits and bigger PRs. Plan roadmap commitments against the full friction picture, and treat review and verification investment as what makes continued growth safe, not as overhead.</p></li></ol><p>To read more about the trends we&#8217;re uncovering and recommendations for engineering leaders moving forward, <strong><a href="https://getdx.com/report/state-of-ai-impact-in-engineering-q3-report/">download the full report here.</a></strong></p><div><hr></div><p>That&#8217;s it for this week. Thanks for reading.</p><p>-Scott</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.getdx.com/p/the-gap-between-code-velocity-and?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.getdx.com/p/the-gap-between-code-velocity-and?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p>]]></content:encoded></item><item><title><![CDATA[Don’t measure AI Code Percentage on its own]]></title><description><![CDATA[Treat it like a lens, not a KPI.]]></description><link>https://newsletter.getdx.com/p/dont-measure-ai-code-percentage-on</link><guid isPermaLink="false">https://newsletter.getdx.com/p/dont-measure-ai-code-percentage-on</guid><dc:creator><![CDATA[Scott Carey]]></dc:creator><pubDate>Wed, 07 Oct 2026 09:02:53 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/337081ce-10b4-4641-8536-7c7c966af563_2400x1260.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><em><strong>Welcome to the latest issue of Engineering Enablement,</strong> a weekly newsletter sharing research and perspectives on developer productivity.</em></p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.getdx.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe now&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.getdx.com/subscribe?"><span>Subscribe now</span></a></p><div><hr></div><p>Hi everyone, I&#8217;m Scott, the new Managing Editor at DX.</p><p>Each week I&#8217;ll share what I am seeing from across the industry, with a particular focus on AI enablement, metrics, and software team dynamics. You&#8217;ll still regularly hear from <a href="https://www.linkedin.com/in/brianhouck/">Brian Houck</a>, <a href="https://www.linkedin.com/in/justinreock/">Justin Reock</a>, <a href="https://www.linkedin.com/in/eirini-kalliamvakou-1016865/">Eirini Kalliamvakou</a>, <a href="https://www.linkedin.com/in/gratiana-fu/">Gratiana Fu</a>, and our extended research community.</p><p>This week, I&#8217;m going to hand things over to Brian, who&#8217;s been thinking about the popular AI code percentage metric, and why, in a world where almost every developer has adopted AI coding tools, it needs to be applied as a lens and not a KPI.</p><div><hr></div><p><strong>Brian: </strong>The question of whether developers are using AI, and whether it is writing a lot of their code, is settled.</p><p><a href="https://newsletter.getdx.com/p/the-state-of-ai-impact-in-engineering">Across more than 500 companies</a>, weekly active AI usage now exceeds 95%. And the percentage of code being generated by AI keeps climbing. In Q3 of 2025, an average of 19% of all merged code was AI-authored. By Q2 of 2026, that figure had reached 52%.</p><p>When AI was new, the percentage of AI-authored code was a useful proxy for adoption. Moving from 10% to 30% meant developers were working differently than they had been, and it was fair to read that as progress. However, adoption is no longer an open question, so the metric has lost the role it was originally intended for.</p><p>That doesn&#8217;t mean AI Code Percentage (AICP) should come out of our measurement systems. It means being clear about its role in them.</p><h2>A useful research variable can be a bad target</h2><p>This distinction matters to me as a researcher. I want to know how much code is being generated by AI.</p><p>AICP lets us split developers or teams into cohorts and ask much more interesting questions. As AI-authored code increases, what happens to:</p><ul><li><p>PR throughput and cycle time?</p></li><li><p>Change failure rate and other quality measures?</p></li><li><p>Code review wait time?</p></li><li><p>Time spent on new capabilities versus maintenance?</p></li><li><p>Developer experience?</p></li><li><p>Developers&#8217; confidence in the changes they&#8217;re shipping?</p></li></ul><p>Most of those map onto dimensions <a href="https://newsletter.getdx.com/p/revisiting-the-dx-core-4">the Core 4</a> already tracks: Speed, Effectiveness, Quality, Impact. AICP isn&#8217;t one of them. It&#8217;s the variable you segment by in order to answer why those dimensions may be changing, which makes it an input to better understanding engineering performance, rather than a measure of engineering performance itself.</p><p>In research terms, AICP is often more interesting as an independent variable than a dependent variable. For engineering leaders, I&#8217;d put it more simply:</p><p><strong>AI Code Percentage is more useful for explaining outcomes than as an outcome itself.</strong></p><p>Being a good diagnostic doesn&#8217;t make it a good target. The test I apply to any metric someone wants to use as a goal is whether gaming it still produces the outcome you wanted. Time-to-First-PR passes this test. Push a new hire to open a trivial PR in their first week and you&#8217;ve still forced the onboarding environment to work, which was the thing you cared about. Gaming it and doing it properly are hard to tell apart, and that&#8217;s what makes it a good target.</p><p>AICP fails badly. Tell a team its AICP needs to reach 60% and they can deliver that within a quarter by routing work through agents that would have been a ten-line human change. Benchmarking also falls apart. Suppose the measurement was perfect and you knew your organization sat at 40%, while a peer sat at 60%. You&#8217;d have learned that two companies produce code differently. You still wouldn&#8217;t know which one gets more value from AI, and the gap on its own gives you no reason to move in either direction.</p><h2>Activity metrics aren&#8217;t the problem</h2><p>One of the central ideas behind the <a href="https://queue.acm.org/doi/10.1145/3454122.3454124">SPACE framework</a> was that developer productivity cannot be reduced to a single measure, and that activity should not be confused with productivity.</p><p>Activity data isn&#8217;t useless. Quite the opposite. Commits, pull requests, deployments and other observable behaviors can tell us a great deal about how a software engineering system works.</p><p>Though on the other side of the argument is lines-of-code, which has earned its reputation as a meaningless activity metric. Producing twice as much code doesn&#8217;t mean you&#8217;ve produced twice as much value, or even that you&#8217;ve done twice as much work. Often the best solution is the one that needs the least code.</p><p>AICP sidesteps that trap. It isn&#8217;t a count, it&#8217;s a rate. Since it is a proportion, more code doesn&#8217;t inherently mean a higher score, and a larger team doesn&#8217;t automatically outperform a smaller one. The question shifts from &#8220;How much code are we producing?&#8221; to &#8220;How much of our code production involves AI?&#8221;</p><p>AICP does share one limitation with lines of code: a higher number doesn&#8217;t necessarily mean a better outcome. It tells us how much of our code production involves AI, not how effectively we&#8217;re using it.</p><h2>The same AI Code Percentage can describe very different ways of working</h2><p>This problem gets more complicated as we move from autocomplete toward agents.</p><p>Imagine two teams that both report 60% AI-authored code. On the first team, developers use AI primarily as sophisticated autocomplete. They decide what to build, decompose the problem, write most of the implementation logic, inspect suggestions as they&#8217;re produced, and retain the useful ones. On the second team, developers delegate entire tasks. They describe the desired outcome, an agent explores the repository, implements a solution, runs tests and submits a pull request.</p><p>Both teams might produce the same AICP, but the role AI plays in their work is radically different.</p><p>This is why <strong>autonomy </strong>is becoming a more interesting dimension of AI adoption. Knowing that AI produced a piece of code doesn&#8217;t tell us how much of the surrounding task AI performed, how much human guidance it required, or where human judgment entered the process.</p><h2>Generated isn&#8217;t the same as shipped</h2><p>Suppose an AI agent generates 1,000 lines of code. A developer accepts 800 of them, rewrites 300 during review, removes another 200 before merge, and six weeks later half of what&#8217;s left has been replaced.</p><p>How much AI-authored code was there? The answer depends heavily on <em>when</em> you measure it.</p><p>Attributing code to AI is inherently messy. AI generates code that humans modify. Humans move and refactor AI-generated code. Suggestions get partially accepted. Developers work across multiple tools and interfaces. Agents behave differently from IDE autocomplete.</p><p>This is why I think <strong>survivorship</strong> is an increasingly important companion to AI Code Percentage. A useful way to think about AI output is as a funnel:</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!p7di!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1f017331-db09-4e98-a5a9-c8c0bae08c5a_2508x158.png 424w, https://substackcdn.com/image/fetch/$s_!p7di!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1f017331-db09-4e98-a5a9-c8c0bae08c5a_2508x158.png 848w, https://substackcdn.com/image/fetch/$s_!p7di!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1f017331-db09-4e98-a5a9-c8c0bae08c5a_2508x158.png 1272w, https://substackcdn.com/image/fetch/$s_!p7di!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1f017331-db09-4e98-a5a9-c8c0bae08c5a_2508x158.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!p7di!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1f017331-db09-4e98-a5a9-c8c0bae08c5a_2508x158.png" width="1456" height="92" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/1f017331-db09-4e98-a5a9-c8c0bae08c5a_2508x158.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:92,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:46887,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://newsletter.getdx.com/i/219110134?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1f017331-db09-4e98-a5a9-c8c0bae08c5a_2508x158.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!p7di!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1f017331-db09-4e98-a5a9-c8c0bae08c5a_2508x158.png 424w, https://substackcdn.com/image/fetch/$s_!p7di!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1f017331-db09-4e98-a5a9-c8c0bae08c5a_2508x158.png 848w, https://substackcdn.com/image/fetch/$s_!p7di!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1f017331-db09-4e98-a5a9-c8c0bae08c5a_2508x158.png 1272w, https://substackcdn.com/image/fetch/$s_!p7di!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1f017331-db09-4e98-a5a9-c8c0bae08c5a_2508x158.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div></figure></div><p><a href="https://arxiv.org/abs/2601.16809?utm_source=chatgpt.com">Recent research</a> is beginning to look at exactly this question. A 2026 study of more than 200,000 code units across 201 open-source projects examined the subsequent fate of agent-authored code rather than simply measuring how much was generated.</p><p>The researchers found meaningful differences in how AI and human-authored code evolved after merge. More interestingly, they found significant differences in survivorship between different AI tools. The survival rate of code from Claude was nearly twice that of code from Devin.</p><h2>How I&#8217;d actually use it</h2><p>At DX, we think AICP should be treated as a lens, not a KPI. Read it alongside throughput, cycle time, quality, time allocation, and developer experience. If AICP rises, look for whether teams are delivering faster, maintaining quality, and spending less time on work they find tedious. If those metrics don&#8217;t improve, perhaps AI is exposing new bottlenecks. The patterns won&#8217;t establish cause and effect on their own, but they tell you where to investigate.</p><p>Review it quarterly, not weekly. This is the part I see missed most often. AICP describes a structural property of how a team works, and structural properties move slowly. A metric earns a weekly slot when you expect it to respond to something you did last sprint. AICP mostly won&#8217;t. Watching it closely generates noise, and it creates a temptation to force the line to move upwards.</p><p>As agentic development matures, we&#8217;ll need measures AICP was never designed to provide: how much autonomy agents are given, <a href="https://newsletter.getdx.com/p/introducing-cafes-a-framework-for">the quality of the context</a> and requirements they receive, how reliably they accomplish what the developer actually wanted, and what happens to their output after it merges.</p><p>AICP has a place in that system. It just shouldn&#8217;t be the point of it.</p><p>Don&#8217;t ask, &#8220;How do we increase our AI Code Percentage?&#8221; Ask, &#8220;What changes when our AI Code Percentage increases?&#8221;</p><p>A lens only works if you&#8217;re looking through it at something else.</p><div><hr></div><p>That&#8217;s it for this week. Thanks for reading.<br>-Scott</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.getdx.com/p/dont-measure-ai-code-percentage-on?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.getdx.com/p/dont-measure-ai-code-percentage-on?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><p></p>]]></content:encoded></item><item><title><![CDATA[How Meta reduced diff authoring time by 40%]]></title><description><![CDATA[Meta researcher Moritz Beller shares how Meta measures developer productivity and how AI is changing engineers&#8217; work, metrics, and testing.]]></description><link>https://newsletter.getdx.com/p/how-meta-reduced-diff-authoring-time</link><guid isPermaLink="false">https://newsletter.getdx.com/p/how-meta-reduced-diff-authoring-time</guid><dc:creator><![CDATA[Brian Houck]]></dc:creator><pubDate>Fri, 02 Oct 2026 14:40:54 GMT</pubDate><enclosure url="https://api.substack.com/feed/podcast/218108782/32da7f4f8b157f1e5c5f5d76f7454ce3.mp3" length="0" type="audio/mpeg"/><content:encoded><![CDATA[<p>Listen and watch now on <strong><a href="https://youtu.be/d1gFQ_6OWAQ">YouTube</a>, <a href="https://podcasts.apple.com/us/podcast/engineering-enablement-by-abi-noda/id1619140476">Apple</a>, and <a href="https://open.spotify.com/show/3NxjyIsuxeDMQtisDqBy7D">Spotify</a></strong>.</p><p>In this episode of Engineering Enablement, I talk with Meta researcher Moritz Beller about how Meta measures developer productivity. Moritz explains diff authoring time and how the company uses it to evaluate tools and guide engineering decisions. We explore how AI is changing engineers&#8217; work and revealing gaps in traditional metrics. We also revisit his highly influential &#8220;Mind the Gap&#8221; study and discuss the promise and risks of AI agents for testing.</p><div id="youtube2-d1gFQ_6OWAQ" class="youtube-wrap" data-attrs="{&quot;videoId&quot;:&quot;d1gFQ_6OWAQ&quot;,&quot;startTime&quot;:null,&quot;endTime&quot;:null}" data-component-name="Youtube2ToDOM"><div class="youtube-inner"><iframe src="https://www.youtube-nocookie.com/embed/d1gFQ_6OWAQ?rel=0&amp;autoplay=0&amp;showinfo=0&amp;enablejsapi=0" frameborder="0" loading="lazy" gesture="media" allow="autoplay; fullscreen" allowautoplay="true" allowfullscreen="true" width="728" height="409"></iframe></div></div><h2><strong>Some takeaways: </strong></h2><h3><strong>Diff authoring time enables more precise productivity measurement</strong></h3><ul><li><p><strong>Diff authoring time measures the active work involved in creating, testing, and reviewing a specific code change.</strong> This gives Meta more precision than metrics averaged across an entire day.</p></li><li><p><strong>No single diff metric provides a complete picture of productivity.</strong> Meta considers authoring time alongside throughput, diff size, and quality to make changes in the data easier to interpret.</p></li><li><p><strong>Meta does not use diff authoring time to evaluate individual developers.</strong> It is used at the team, organizational, and company levels to identify trends, regressions, and opportunities for improvement.</p></li></ul><h3><strong>A/B experiments help Meta prioritize engineering investments</strong></h3><ul><li><p><strong>Diff-level measurement allows Meta to quantify whether changes to developer tools and frameworks save engineers time.</strong> This helps leaders decide which improvements deserve further investment.</p></li><li><p><strong>Automatic memoization in the React compiler reduced diff authoring time by roughly 30% compared with implementing caching manually.</strong> Results of that magnitude suggest that foundational improvements can be more valuable than small interface optimizations.</p></li></ul><h3><strong>AI is changing how engineers spend their time</strong></h3><ul><li><p><strong>Meta&#8217;s year-over-year diff authoring time has fallen by more than 40%, while developers are producing more and larger diffs.</strong> Moritz views the company-wide pattern as a strong signal of AI&#8217;s impact.</p></li><li><p><strong>It is not yet clear where all the saved authoring time goes.</strong> Moritz believes engineers are spending more time gathering context and performing work adjacent to coding, while relatively little time goes into writing prompts.</p></li><li><p><strong>As code becomes cheaper to generate, intent becomes more valuable.</strong> The important question is increasingly whether the finished code accurately reflects what the developer intended to build.</p></li></ul><h3><strong>Traditional telemetry misses important engineering work</strong></h3><ul><li><p><strong>Activities like whiteboarding, brainstorming, and architectural alignment are difficult to connect to code changes through conventional telemetry.</strong> Meeting transcripts and other AI-generated records could help make that work more visible.</p></li><li><p><strong>Cheaper code generation does not eliminate the need for architecture.</strong> Skipping a design document can simply transfer the cost to reviewers, who must reconstruct the intended design from the implementation.</p></li></ul><h3><strong>Developer productivity is both measured and perceived</strong></h3><ul><li><p><strong>Moritz&#8217;s &#8220;Mind the Gap&#8221; study connected automatically measured activity with developers&#8217; perceptions of their own productivity.</strong> Time spent coding was an important predictor, but sleep, interruptions, and on-call responsibilities also mattered.</p></li><li><p><strong>A modern version of the study would need to account for agentic work.</strong> Moritz would examine how developers interact with agents, how much they trust them, and whether managing parallel work creates cognitive overload.</p></li><li><p><strong>Higher output does not necessarily improve developer experience.</strong> Brian points to research showing that output can rise while flow, cognitive load, and overall experience remain flat or worsen.</p></li></ul><h3><strong>AI makes testing easier but not necessarily safer</strong></h3><ul><li><p><strong>Agents make it inexpensive to generate unit and end-to-end tests.</strong> This could fill gaps left by developers who previously did little testing.</p></li><li><p><strong>Passing tests can create false confidence when both the implementation and tests reflect the same misunderstanding.</strong> The agent may satisfy its interpretation of an underspecified request without delivering what the developer actually intended.</p></li><li><p><strong>Specification-driven development could become a new form of test-driven development.</strong> Defining correctness before implementation may matter more as agents take on more of the coding.</p></li></ul><h3><strong>Collaboration remains one of the hardest parts of productivity to measure</strong></h3><ul><li><p><strong>Interpersonal dynamics, alignment, and knowledge sharing are central to engineering productivity but poorly captured by existing metrics.</strong> Faster implementation also makes it easier for teams to unknowingly duplicate one another&#8217;s work.</p></li><li><p><strong>Reducing low-quality meetings can produce major throughput gains, but eliminating collaboration creates different problems.</strong> Teams still need enough interaction to generate ideas, share context, and stay aligned.</p></li><li><p><strong>Small human interactions can have measurable value.</strong> Brian&#8217;s research found that informal conversation before and after meetings predicted self-reported productivity better than internet quality.</p></li></ul><h2><strong>In this episode, we cover:</strong></h2><p>(<a href="https://www.youtube.com/watch?v=d1gFQ_6OWAQ">00:00</a>) Intro</p><p>(<a href="https://www.youtube.com/watch?v=d1gFQ_6OWAQ&amp;t=130s">02:10</a>) Moritz&#8217;s role at Meta</p><p>(<a href="https://www.youtube.com/watch?v=d1gFQ_6OWAQ&amp;t=241s">04:01</a>) Measuring diff authoring time</p><p>(<a href="https://www.youtube.com/watch?v=d1gFQ_6OWAQ&amp;t=500s">08:20</a>) Measuring A/B experiments</p><p>(<a href="https://www.youtube.com/watch?v=d1gFQ_6OWAQ&amp;t=731s&amp;pp=0gcJCWMAwfN6Pr3D">12:11</a>) What diff authoring time reveals</p><p>(<a href="https://www.youtube.com/watch?v=d1gFQ_6OWAQ&amp;t=885s">14:45</a>) Planning and leadership reporting</p><p>(<a href="https://www.youtube.com/watch?v=d1gFQ_6OWAQ&amp;t=1116s">18:36</a>) Why Meta measures teams, not individuals</p><p>(<a href="https://www.youtube.com/watch?v=d1gFQ_6OWAQ&amp;t=1353s">22:33</a>) Where developer time goes with AI</p><p>(<a href="https://www.youtube.com/watch?v=d1gFQ_6OWAQ&amp;t=1566s">26:06</a>) AI&#8217;s impact on junior and senior developers</p><p>(<a href="https://www.youtube.com/watch?v=d1gFQ_6OWAQ&amp;t=1638s">27:18</a>) Capturing invisible work with AI</p><p>(<a href="https://www.youtube.com/watch?v=d1gFQ_6OWAQ&amp;t=1772s">29:32</a>) The &#8220;Mind the Gap&#8221; study</p><p>(<a href="https://www.youtube.com/watch?v=d1gFQ_6OWAQ&amp;t=2020s">33:40</a>) Revisiting &#8220;Mind the Gap&#8221; in 2026</p><p>(<a href="https://www.youtube.com/watch?v=d1gFQ_6OWAQ&amp;t=2312s">38:32</a>) AI agents for software testing</p><p>(<a href="https://www.youtube.com/watch?v=d1gFQ_6OWAQ&amp;t=2509s">41:49</a>) What remains hard to measure</p><p><strong>Where to find Moritz Beller:</strong></p><p>&#8226; LinkedIn: <a href="https://www.linkedin.com/in/inventitech">https://www.linkedin.com/in/inventitech</a></p><p>&#8226; X: <a href="https://x.com/Inventitech">https://x.com/Inventitech</a></p><p>&#8226; GitHub: <a href="https://github.com/Inventitech">https://github.com/Inventitech</a></p><p>&#8226; Website: <a href="https://inventitech.com">https://inventitech.com</a></p><p><strong><span>Where to find Brian Houck:</span></strong></p><p><span>&#8226; LinkedIn: </span><a href="https://www.linkedin.com/in/brianhouck"><span>https://www.linkedin.com/in/brianhouck</span></a></p><h2><strong>Referenced:</strong></h2><p>&#8226; <a href="https://getdx.com/report/State-of-AI-Impact-in-Engineering-Q2-Report">State of AI Impact in Engineering Q2 Report 2026</a></p><p>&#8226; <a href="https://arxiv.org/abs/2606.00438">GitHub Copilot and Developer Productivity: An Observational Dose-Response Analysis</a></p><p>&#8226; <a href="https://qb4.acm.org/detail.cfm?id=3807966">From Technical Debt to Cognitive and Intent Debt</a></p><p>&#8226; <a href="https://arxiv.org/abs/2012.07428">Mind the Gap: On the Relationship Between Automatically Measured and Self-Reported Productivity</a></p><p>&#8226; <a href="https://arxiv.org/abs/2605.23135">The Impact of AI Coding Assistants on Software Engineering: A Longitudinal Study</a></p><p>&#8226; <a href="https://dl.acm.org/doi/10.1145/2786805.2786843">When, how, and why developers (do not) test in their IDEs | Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering</a></p><p>&#8226; <a href="https://getdx.com/guide/dora-space-devex/">DORA, SPACE, and DevEx: Which framework should you use</a></p><p>&#8226; <a href="https://arxiv.org/abs/2008.11147">A Tale of Two Cities: Software Developers Working from Home During the COVID-19 Pandemic</a></p>]]></content:encoded></item><item><title><![CDATA[Do you really need another framework? ]]></title><description><![CDATA[Where CAFE(S) fits with your existing measurement and evaluation stack.]]></description><link>https://newsletter.getdx.com/p/do-you-really-need-another-framework</link><guid isPermaLink="false">https://newsletter.getdx.com/p/do-you-really-need-another-framework</guid><dc:creator><![CDATA[Brian Houck]]></dc:creator><pubDate>Wed, 30 Sep 2026 10:01:34 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/e59f71f7-7d4f-43c7-abae-b2c86a8c9f7c_2400x1260.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><em><strong>Welcome to the latest issue of Engineering Enablement,</strong> a weekly newsletter sharing research and perspectives on developer productivity.</em></p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.getdx.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe now&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.getdx.com/subscribe?"><span>Subscribe now</span></a></p><div><hr></div><p>Whenever a new evaluation framework shows up, the first reasonable question is which existing one it competes with. That reflex is healthy. New frameworks often overlap with things you&#8217;re already measuring, and every addition comes with some cost. Anyone who has watched a measurement program collapse under its own weight has learned to ask what is being displaced before adopting something new.</p><p>I wanted to answer that question for my <a href="https://getdx.com/whitepaper/the-cafes-framework/">recently published CAFE(S) framework</a>, because it doesn&#8217;t compete with what you&#8217;re already running. CAFE(S) is a framework for evaluating the quality of the context we give AI agents across five dimensions: Clarity, Actionability, Fidelity, Efficiency, and (Security). It asks whether the information agents have to work with is good enough for the tasks they&#8217;re trying to accomplish.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://newsletter.getdx.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Engineering Enablement! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p><a href="https://www.atlassian.com/devops/frameworks/dora-metrics">DORA</a> looks at your delivery system. <a href="https://queue.acm.org/doi/10.1145/3454122.3454124">SPACE</a>, <a href="https://getdx.com/research/measuring-developer-productivity-with-the-dx-core-4/">DX Core 4</a>, and <a href="https://queue.acm.org/doi/10.1145/3819080">EngThrive</a> look at developers and the outcomes they&#8217;re working toward. Retrieval and model evals assess what the AI retrieved and produced. OWASP looks at the risk surface. Context and harness engineering guidance tells you how to build the environment the agent works in.</p><p>What none of your existing frameworks were built to look at is the information the agent was handed in the first place. <a href="https://newsletter.getdx.com/p/your-agent-doesnt-have-a-model-problem">I&#8217;ve argued that AI didn&#8217;t create a context problem,</a> but pushed an existing issue past its breaking point.</p><p>This isn&#8217;t an argument to drop what you&#8217;re already running, or to blindly bolt one more framework onto the stack. The rest of this piece looks at where CAFE(S) sits alongside existing frameworks you may be using. In doing so, I hope to make the boundaries between all of them a little clearer.</p><h3>Delivery and experience metrics tell you something is wrong, not where</h3><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!xmm3!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97bcef15-63f6-4f71-bf26-e93eccc8d4d0_2512x2000.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!xmm3!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97bcef15-63f6-4f71-bf26-e93eccc8d4d0_2512x2000.png 424w, https://substackcdn.com/image/fetch/$s_!xmm3!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97bcef15-63f6-4f71-bf26-e93eccc8d4d0_2512x2000.png 848w, https://substackcdn.com/image/fetch/$s_!xmm3!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97bcef15-63f6-4f71-bf26-e93eccc8d4d0_2512x2000.png 1272w, https://substackcdn.com/image/fetch/$s_!xmm3!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97bcef15-63f6-4f71-bf26-e93eccc8d4d0_2512x2000.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!xmm3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97bcef15-63f6-4f71-bf26-e93eccc8d4d0_2512x2000.png" width="1456" height="1159" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/97bcef15-63f6-4f71-bf26-e93eccc8d4d0_2512x2000.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1159,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:157822,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://newsletter.getdx.com/i/217846456?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97bcef15-63f6-4f71-bf26-e93eccc8d4d0_2512x2000.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!xmm3!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97bcef15-63f6-4f71-bf26-e93eccc8d4d0_2512x2000.png 424w, https://substackcdn.com/image/fetch/$s_!xmm3!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97bcef15-63f6-4f71-bf26-e93eccc8d4d0_2512x2000.png 848w, https://substackcdn.com/image/fetch/$s_!xmm3!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97bcef15-63f6-4f71-bf26-e93eccc8d4d0_2512x2000.png 1272w, https://substackcdn.com/image/fetch/$s_!xmm3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F97bcef15-63f6-4f71-bf26-e93eccc8d4d0_2512x2000.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>DORA, SPACE, DX Core 4, and EngThrive are all anchored to outcomes, which is exactly why they&#8217;ve survived several technology cycles and <a href="https://newsletter.getdx.com/p/revisiting-the-dx-core-4">why I don&#8217;t expect AI to invalidate any of them</a>.</p><p>But an outcome moving in the wrong direction is a problem, not a diagnosis. Change failure rate climbing, DXI sliding, or innovation rate stagnating tells you where to start asking questions, not what caused the change.</p><p>Core 4 and EngThrive both explicitly carry diagnostic dimensions alongside the outcome ones, so this isn&#8217;t a clean lagging vs. leading split, but those diagnostics aren&#8217;t focused on the assembled context as a first-class engineering artifact.</p><p>We&#8217;re still trying to accomplish the same top-level outcomes in our agentic workflows as we were before. What has changed is the diagnostic surface, requiring us to look in new places for answers when we&#8217;re not achieving them.</p><blockquote><p><strong>Ask yourself: </strong>when a delivery or experience metric moves the wrong way, can you tell whether the information your agents worked from had anything to do with it?</p></blockquote><h3>Retrieval evals tell you whether retrieval worked, not whether the context worked</h3><p>Retrieval-augmented generation (<a href="https://aclanthology.org/2024.eacl-demo.16/">RAG) evaluation frameworks</a> measure the quality of the information retrieved and the responses generated from it, which touches on the Fidelity and Efficiency dimensions of CAFE(S). If you run RAG evals well, you already have two of the five properties partially covered.</p><p>The other three (Clarity, Actionability, and Security) are outside their frame. A retrieval eval does not ask whether the task was stated in a way that admits a single interpretation, whether anyone defined what finished looks like, or whether attacker-controlled content made it into the window.</p><p>There&#8217;s an important difference in what gets evaluated. A retrieval eval scores the information your retriever fetched. The agent reasons over the whole assembled context, which also includes the human&#8217;s prompt, the root <a href="http://agents.md">AGENTS.md</a> file, tool output, prior turns, and whatever memory the harness decided to carry forward. Most of that never passed through your retriever, so most of it never appears in your eval.</p><blockquote><p><strong>Ask yourself:</strong> are you evaluating the passages your retriever returned, or everything the agent actually read?</p></blockquote><h3>OWASP evaluates security risk, not context quality</h3><p><a href="https://genai.owasp.org/llm-top-10/">OWASP</a> goes much deeper on security risk than CAFE(S) does. The &#8216;S&#8217; pillar is one property among five; OWASP provides focused security guidance. If you&#8217;re choosing between them for security depth, you shouldn&#8217;t choose CAFE(S). But you shouldn&#8217;t have to choose between them.</p><p>CAFE(S) puts security inside a broader quality model for context. That&#8217;s useful because security and usefulness aren&#8217;t always separable. Oversharing, for example, can be both a security failure and an efficiency failure. Trimming unnecessary context can simultaneously reduce token spend and shrink the attack surface.</p><p>Failures of the first four dimensions of CAFE(S) make agents less effective. Failures of security make them unsafe.</p><blockquote><p><strong>Ask yourself: </strong>are you evaluating your context for both usefulness and safety?</p></blockquote><h3>Context engineering tells you how to build context, not whether it&#8217;s good</h3><p><a href="https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents">Anthropic&#8217;s context engineering guidance</a>, <a href="https://martinfowler.com/articles/harness-engineering.html">Birgitta B&#246;ckeler&#8217;s harness engineering</a> guidance, and <a href="https://openai.com/index/harness-engineering/">OpenAI&#8217;s harness engineering</a> guide all offer practices for building better context and harnesses. CAFE(S) starts at the other end: given the context an agent actually received, was it any good?</p><p>That&#8217;s the difference between prescriptive and evaluative. You can follow good context engineering practices and still end up with context that is ambiguous, incomplete, stale, bloated, or unsafe. Conversely, CAFE(S) doesn&#8217;t tell you which retrieval strategy, memory architecture, or harness design to use. It gives you properties for evaluating the result.</p><blockquote><p><strong>Ask yourself:</strong> do you have a definition of good context that is independent of how you built it?</p></blockquote><h3>TRUCE evaluates code quality, not context quality</h3><p><a href="https://doi.org/10.1145/3528579.3529177">TRUCE</a> is a multidimensional framework for evaluating code quality and is the closest conceptual sibling to CAFE(S).</p><p>CAFE(S) applies a similar idea to a different engineering artifact. As context becomes a first-class input to agentic software development, its quality matters independent of the quality of the code that eventually comes out.</p><p>TRUCE gives us a language for thinking about the quality of code. CAFE(S) is intended to do the same for context.</p><blockquote><p><strong>Ask yourself: </strong>if context is becoming a first-class engineering artifact, do you have a way to evaluate its quality?</p></blockquote><h2>Final thoughts</h2><p>So, do you need another framework? Maybe that&#8217;s the wrong question.</p><p>DORA, SPACE, retrieval evals, OWASP, TRUCE, and CAFE(S) aren&#8217;t competing ways to measure the same thing. They answer different questions about different parts of the engineering system.</p><p>CAFE(S) adds one that has become increasingly important in agentic development: was the context we gave the agent any good?</p><p>It doesn&#8217;t replace what you&#8217;re already using. It fills the gaps between them.</p><div><hr></div><p>That&#8217;s it for this week. Thanks for reading.</p><p>-Brian</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.getdx.com/p/do-you-really-need-another-framework?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.getdx.com/p/do-you-really-need-another-framework?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p>]]></content:encoded></item><item><title><![CDATA[Introducing CAFE(S): A framework for defining AI context quality]]></title><description><![CDATA[Five durable properties for evaluating the context we hand to AI agents]]></description><link>https://newsletter.getdx.com/p/introducing-cafes-a-framework-for</link><guid isPermaLink="false">https://newsletter.getdx.com/p/introducing-cafes-a-framework-for</guid><dc:creator><![CDATA[Brian Houck]]></dc:creator><pubDate>Thu, 24 Sep 2026 16:42:16 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/9af4c190-cbbd-46b5-b69b-da0dd6d46b77_2400x1260.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><em><strong>Welcome to the latest issue of Engineering Enablement,</strong> a weekly newsletter sharing research and perspectives on developer productivity.</em></p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.getdx.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe now&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.getdx.com/subscribe?"><span>Subscribe now</span></a></p><p>&#128467; <a href="https://getdx.com/webinar/cafes-framework-improving-agent-effectiveness?utm_source=newsletter">Join DX&#8217;s live research panel today</a> as we introduce CAFE(S), a framework for improving AI agent effectiveness through better context. <a href="https://getdx.com/webinar/cafes-framework-improving-agent-effectiveness?utm_source=newsletter">Register here.</a></p><div><hr></div><p><span>This special issue marks the release of </span><strong><span>CAFE(S)</span></strong><span>, a new framework for evaluating the quality of context assembled for AI systems. I had the privilege of co-authoring it with Max Kanat-Alexander (Capital One), Eirini Kalliamvakou (DX, formerly GitHub), Margaret-Anne Storey (University of Victoria), and Nicole Forsgren (Google). </span></p><p><span>It is out now in </span><em><span>ACM Queue</span></em><span>, and you can </span><strong><a href="https://getdx.com/whitepaper/the-cafes-framework/"><span>download the paper here.</span></a></strong></p><h2><span>Low-quality context now exacts a high price</span></h2><p><span>For most of software&#8217;s history, knowledge management has been a major source of developer pain. Engineers have struggled with out-of-date documentation, unclear specifications, and conflicting requirements for decades. Poor context might have been a drag on developer experience, but that pain could be absorbed because a developer could go over and ask a peer for clarification.</span></p><p><span>What&#8217;s changed isn&#8217;t that context started mattering. It&#8217;s that the consequences now arrive immediately, at scale, and usually with a number attached.</span></p><p><strong><span>Tokens</span></strong><span>. Every token sent to an agent is paid for, and context is where tokens accumulate. When Notion engineered its context for reuse through prompt caching, costs dropped roughly 90 percent with no loss in output quality, because reading cached context is far cheaper than retransmitting the same information.</span></p><p><strong><span>Liability</span></strong><span>. In February 2024, the British Columbia Civil Resolution Tribunal found Air Canada liable for a refund its customer service chatbot promised and its actual policy did not provide. Air Canada argued the chatbot was effectively responsible for its own statements. The tribunal disagreed. The damages were modest. The precedent was not.</span></p><p><strong><span>Security</span></strong><span>. EchoLeak (CVE-2025-32711) was the first documented prompt injection to achieve real-world data exfiltration from a production AI system. A single crafted email, never opened by the user, caused Microsoft 365 Copilot to pull internal documents into its working context and leak them. Untrusted information became part of the agent&#8217;s context, and the agent acted on it, faithfully.</span></p><h2><span>Translating human intent for intelligent models</span></h2><p><span>Our creative ideas aren&#8217;t automatically absorbed by intelligent models. They need to be translated before an agent can act on them. This simple pipeline describes how human intent becomes agent behavior.</span></p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!C6yb!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7b6fe269-6999-4641-81d0-e20fff742aea_1974x286.png 424w, https://substackcdn.com/image/fetch/$s_!C6yb!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7b6fe269-6999-4641-81d0-e20fff742aea_1974x286.png 848w, https://substackcdn.com/image/fetch/$s_!C6yb!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7b6fe269-6999-4641-81d0-e20fff742aea_1974x286.png 1272w, https://substackcdn.com/image/fetch/$s_!C6yb!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7b6fe269-6999-4641-81d0-e20fff742aea_1974x286.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!C6yb!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7b6fe269-6999-4641-81d0-e20fff742aea_1974x286.png" width="1456" height="211" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/7b6fe269-6999-4641-81d0-e20fff742aea_1974x286.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:211,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:27335,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://newsletter.getdx.com/i/215696862?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7b6fe269-6999-4641-81d0-e20fff742aea_1974x286.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!C6yb!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7b6fe269-6999-4641-81d0-e20fff742aea_1974x286.png 424w, https://substackcdn.com/image/fetch/$s_!C6yb!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7b6fe269-6999-4641-81d0-e20fff742aea_1974x286.png 848w, https://substackcdn.com/image/fetch/$s_!C6yb!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7b6fe269-6999-4641-81d0-e20fff742aea_1974x286.png 1272w, https://substackcdn.com/image/fetch/$s_!C6yb!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7b6fe269-6999-4641-81d0-e20fff742aea_1974x286.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div></figure></div><p><span>Intent is what a person ultimately wants to accomplish. That </span><strong><span>intent</span></strong><span> must then be translated into </span><strong><span>context</span></strong><span>, which is the information that represents the task, its constraints, and the surrounding knowledge. A </span><strong><span>harness</span></strong><span> applies that context through orchestration, retrieval, tools, memory, and execution logic before passing it to a </span><strong><span>model</span></strong><span>, which supplies the underlying intelligence that produces a result.</span></p><p><span>Every layer matters, and three of them have serious disciplines behind them. Machine learning research improves models. Harness engineering improves how those models are orchestrated. Product and requirements work improves how we create and refine intent in the first place. The translation from intent into context is the one that gets relatively little attention, despite increasingly determining whether an otherwise capable agent succeeds or fails.</span></p><p><span>Some disciplines get close, but answer different questions. Knowledge management asks whether the information exists and is maintained. Information retrieval asks whether it can be found. Context engineering asks how to select and assemble it for a model. But none of those tell you whether the context itself is fit for the task at hand.</span></p><p><span>That&#8217;s the gap CAFE(S) is meant to fill.</span></p><h2><span>The five properties</span></h2><p><span>CAFE(S) names five properties of good context. The first four ask whether the context lets agents do good work. The fifth, set slightly apart in parentheses, asks whether the context is safe to use at all.</span></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!8OU4!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F30d87c8e-b359-4033-a5a8-d9880b3d13d2_4200x1156.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!8OU4!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F30d87c8e-b359-4033-a5a8-d9880b3d13d2_4200x1156.png 424w, https://substackcdn.com/image/fetch/$s_!8OU4!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F30d87c8e-b359-4033-a5a8-d9880b3d13d2_4200x1156.png 848w, https://substackcdn.com/image/fetch/$s_!8OU4!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F30d87c8e-b359-4033-a5a8-d9880b3d13d2_4200x1156.png 1272w, https://substackcdn.com/image/fetch/$s_!8OU4!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F30d87c8e-b359-4033-a5a8-d9880b3d13d2_4200x1156.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!8OU4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F30d87c8e-b359-4033-a5a8-d9880b3d13d2_4200x1156.png" width="1456" height="401" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/30d87c8e-b359-4033-a5a8-d9880b3d13d2_4200x1156.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:401,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:136897,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://newsletter.getdx.com/i/215696862?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F30d87c8e-b359-4033-a5a8-d9880b3d13d2_4200x1156.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!8OU4!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F30d87c8e-b359-4033-a5a8-d9880b3d13d2_4200x1156.png 424w, https://substackcdn.com/image/fetch/$s_!8OU4!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F30d87c8e-b359-4033-a5a8-d9880b3d13d2_4200x1156.png 848w, https://substackcdn.com/image/fetch/$s_!8OU4!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F30d87c8e-b359-4033-a5a8-d9880b3d13d2_4200x1156.png 1272w, https://substackcdn.com/image/fetch/$s_!8OU4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F30d87c8e-b359-4033-a5a8-d9880b3d13d2_4200x1156.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><ul><li><p><strong><span>Clarity</span></strong><span>: can the agent interpret the request as intended? Clear context resolves the same way for the author and for the agent. When it is ambiguous, the agent may choose an incorrect interpretation, often silently, and every step after that compounds the choice.</span></p></li><li><p><strong><span>Actionability</span></strong><span>: can the agent proceed, and know when it&#8217;s done? Actionable context states the objective, relevant constraints, and what &#8220;done&#8221; looks like. Without those, agents spiral into execution loops, stop early, or declare success against a target they inferred rather than one they were given.</span></p></li><li><p><strong><span>Fidelity</span></strong><span>: can the agent trust that this context is true? Faithful context is accurate, current, internally consistent, and drawn from sources worth trusting. When it isn&#8217;t, agents reason confidently from stale guidance, or fill a gap with a guess that then becomes indistinguishable from fact.</span></p></li><li><p><strong><span>Efficiency</span></strong><span>: can the agent focus on what matters? Efficient context carries a high signal-to-noise ratio. Irrelevant material doesn&#8217;t merely take up room, it competes for the model&#8217;s attention, which is how the one detail that mattered ends up lost in the middle. And every unnecessary token is paid for.</span></p></li><li><p><strong><span>Security</span></strong><span>: should the agent have this context at all? Secure context is safe and appropriate for the agent to access and process. Failures here look like prompt injection, data leakage, and policy violations rather than merely disappointing results.</span></p></li></ul><p><span>A fair question is why we stopped at five, when timeliness, cost, and provenance are all plausible candidates. Our bar was that a property earns its own pillar only if it isn&#8217;t derivable from the others, can be acted on independently, and names a failure the others miss. Most fold in cleanly. Stale context is context that&#8217;s no longer true, which makes timeliness a facet of fidelity, and cost is a downstream consequence of efficiency rather than a property of the artifact.</span></p><p><span>What&#8217;s left is genuinely independent. Context can be clear but not faithful, describing the system as it was rather than as it is. It can be efficient and faithful but not actionable, never saying what finishing looks like. A single bad instruction can fail several pillars at once, but each one names something you can reason about, review for, and fix on its own.</span></p><p><span>Security sits in parentheses because it evaluates the same artifact while asking a different kind of question about it. Failures of the first four make agents less effective. Failures of security make them unsafe.</span></p><h2><span>Putting CAFE(S) into practice</span></h2><p><span>Understanding the five properties is a start. The value comes from treating the context you hand to agents as a first-class engineering artifact, which means someone owns it, someone reviews it, and it gets maintained as the system changes.</span></p><p><span>First, the information must be </span><strong><span>accessible</span></strong><span>; the best context is useless if it is out of the agent&#8217;s reach. Second, the system must be capable of </span><strong><span>cutting through the noise</span></strong><span> to retrieve the exact slice of data needed across scattered systems, without exhausting token budgets or context windows. Finally, the knowledge must </span><strong><span>exist in the first place</span></strong><span>. No retrieval architecture or context engineering can surface the undocumented hallway decisions and unrecorded conversations that live only in people&#8217;s heads. Only after information is captured, connected, and retrievable do the properties of CAFE(S) become the operative standard for quality.</span></p><p><span>Once you&#8217;re past those, here are the highest-leverage moves against each property.</span></p><ul><li><p><strong><span>For clarity and actionability</span></strong><span>: introduce a review discipline for shared context files like AGENTS.md or checked-in specs. You wouldn&#8217;t merge unreviewed code. Review these the same way, and scale the scrutiny to how widely they get reused. A repository-wide file shaping thousands of sessions deserves far more attention than a working note between two engineers.</span></p></li><li><p><strong><span>For fidelity</span></strong><span>: assign clear ownership and a review cadence to critical context sources, including runbooks, architecture decision records, and API schemas. Information decays as systems evolve, and an owner is what gets stale guidance pruned before an agent turns it into a confident mistake. Ownership should follow teams rather than individuals, so context survives reorgs.</span></p></li><li><p><strong><span>For efficiency</span></strong><span>: put guidance where it&#8217;s actually used. The same words can be high quality in one location and harmful in another. An instruction about running the test suite belongs in a root AGENTS.md, because nearly every session needs it. An instruction about one rarely touched file does not, because it gets injected into thousands of sessions that have nothing to do with it. Move that instruction into a comment inside the file it describes and it becomes high quality, available exactly when it&#8217;s relevant. Nothing about the words changed. Only their location did.</span></p></li><li><p><strong><span>For security</span></strong><span>: hold a hard boundary between trusted instructions and untrusted data. Emails, tickets, and pull request content are inputs, not instructions. Treating them as instructions is the mechanism behind the attack I described earlier.</span></p></li></ul><p><span>The paper goes considerably further on what individuals, teams, and organizations can do to improve context quality along the dimensions of CAFE(S).</span></p><h2><span>Final thoughts</span></h2><p><span>CAFE(S) provides a definition of context quality, not a measurement system. Our goal is to give engineering teams a shared vocabulary to discuss context, review it as a first-class asset, and intervene before bad context causes failures. While empirical metrics and automated evaluations will follow, meaningful measurement first requires a precise definition of what high-quality context actually looks like.</span></p><p><span>As software development shifts toward greater human-agent collaboration, high-quality context becomes a fundamental prerequisite for agent autonomy. Clear, actionable, faithful, efficient, and secure context creates the baseline conditions where delegating complex work becomes reasonable, though never fully guaranteed. CAFE(S) doesn&#8217;t claim good context is sufficient to hand an agent the keys; it simply shows why, without it, you shouldn&#8217;t.</span></p><div><hr></div><p><span>You can read the full paper here: </span><strong><a href="https://getdx.com/whitepaper/the-cafes-framework/"><span>CAFE(S): Your Agent Is Only as Good as Its Context</span></a></strong></p><div><hr></div><p>That&#8217;s it for this week. Thanks for reading.<br>-Brian</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.getdx.com/p/introducing-cafes-a-framework-for?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.getdx.com/p/introducing-cafes-a-framework-for?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><p></p>]]></content:encoded></item><item><title><![CDATA[Your agent doesn't have a model problem]]></title><description><![CDATA[The context problems humans have been quietly fixing for decades and what happens when agents inherit them.]]></description><link>https://newsletter.getdx.com/p/your-agent-doesnt-have-a-model-problem</link><guid isPermaLink="false">https://newsletter.getdx.com/p/your-agent-doesnt-have-a-model-problem</guid><dc:creator><![CDATA[Brian Houck]]></dc:creator><pubDate>Wed, 16 Sep 2026 10:03:38 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/0fc84c32-104b-45b9-a67e-8b1ab0477321_2400x1260.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><em><strong><span>Welcome to the latest issue of Engineering Enablement, </span></strong><span>a weekly newsletter sharing research and perspectives on developer productivity.</span></em></p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.getdx.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe now&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.getdx.com/subscribe?"><span>Subscribe now</span></a></p><p><span>&#128467; </span><a href="https://getdx.com/webinar/cafes-framework-improving-agent-effectiveness/#content"><span>Join our live research panel on September 24</span></a> <span>as we introduce CAFE(S), a framework for improving AI agent effectiveness through better context. </span><a href="https://getdx.com/webinar/cafes-framework-improving-agent-effectiveness?utm_source=newsletter"><span>Register here.</span></a></p><div><hr></div><p><span>Some of the most publicized AI failures of the last few years came from the legal profession. </span><a href="https://natlawreview.com/article/more-sanctions-inquiries-against-lawyers-judges-cite-hallucinations"><span>Lawyers have now been sanctioned</span></a><span>, more than once, for filing briefs containing citations to cases that did not exist. It is tempting to blame these hallucinations entirely on model failure, but that offers an incomplete picture.</span></p><p><span>A brief was drafted, reviewed, and filed. Somewhere along that chain, an unsupported claim produced by a probabilistic language model became something a human was willing to represent as fact. The hallucination was the visible failure, but the root cause was the information environment surrounding the model.</span></p><p><span>A better model might hallucinate less often, but it won&#8217;t eliminate the need for the information and safeguards required to separate the plausible from the true.</span></p><p><span>That distinction is critical, and it&#8217;s showing up everywhere software gets built.</span></p><h3><span>Picking up where the code left off</span></h3><p><span>Recently, I wrote about </span><a href="https://newsletter.getdx.com/p/the-quality-paradox-of-ai-generated"><span>what AI is doing to software quality,</span></a><span> using a definition of quality that some colleagues and I developed back in 2020. The conclusion I landed on was that code is becoming a late place to look. More of what determines quality is settled before any code exists.</span></p><p><span>I ended that piece with five sentences that have stuck with me since:</span></p><blockquote><p><em><span>Reuse requires knowing what already exists in the codebase. Review requires knowing what a change was meant to accomplish. Evolvability requires knowing why the system is structured the way it is. Handoff requires knowing what the previous person was thinking. Serving user needs requires knowing what problem the software was supposed to solve in the first place.</span></em></p></blockquote><p><span>None of that knowledge necessarily lives in the code. It lives in tickets, design docs, runbooks, architecture decision records, and Slack threads. A fair amount of it lives nowhere at all except in somebody&#8217;s head.</span></p><p><span>When we hand a task to an agent, we hand over some representation of that knowledge with it. That representation is the context. And when context fails, it does not fail randomly, the same shapes keep recurring.</span></p><h3><span>We have a vocabulary for this, just not for context</span></h3><p><span>Software engineers have a rich language for recurring problems in code. Say </span><strong><a href="https://wiki.c2.com/?CodeSmell"><span>code smell</span></a></strong><span> and everyone knows you mean something that isn&#8217;t a bug, but reliably signals trouble ahead. Long method. Feature envy. Shotgun surgery. Those names are useful precisely because they convert a vague unease into something you can point at in a review.</span></p><p><span>We have almost nothing equivalent for context when working with agents. When an agent produces something wrong, the diagnosis usually stops at &#8220;the model got confused,&#8221; which explains nothing and suggests no action.</span></p><p><span>So let me offer a starting vocabulary. Call them </span><em><span>context smells</span></em><span>. Here are six I keep running into. None of them are model limitations, and all of them survive a model upgrade.</span></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!XWM9!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F246d658b-a5f4-4a97-8848-3f259a82deaf_2048x720.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!XWM9!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F246d658b-a5f4-4a97-8848-3f259a82deaf_2048x720.png 424w, https://substackcdn.com/image/fetch/$s_!XWM9!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F246d658b-a5f4-4a97-8848-3f259a82deaf_2048x720.png 848w, https://substackcdn.com/image/fetch/$s_!XWM9!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F246d658b-a5f4-4a97-8848-3f259a82deaf_2048x720.png 1272w, https://substackcdn.com/image/fetch/$s_!XWM9!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F246d658b-a5f4-4a97-8848-3f259a82deaf_2048x720.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!XWM9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F246d658b-a5f4-4a97-8848-3f259a82deaf_2048x720.png" width="1456" height="512" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/246d658b-a5f4-4a97-8848-3f259a82deaf_2048x720.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:512,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:59066,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://newsletter.getdx.com/i/213041998?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F246d658b-a5f4-4a97-8848-3f259a82deaf_2048x720.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!XWM9!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F246d658b-a5f4-4a97-8848-3f259a82deaf_2048x720.png 424w, https://substackcdn.com/image/fetch/$s_!XWM9!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F246d658b-a5f4-4a97-8848-3f259a82deaf_2048x720.png 848w, https://substackcdn.com/image/fetch/$s_!XWM9!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F246d658b-a5f4-4a97-8848-3f259a82deaf_2048x720.png 1272w, https://substackcdn.com/image/fetch/$s_!XWM9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F246d658b-a5f4-4a97-8848-3f259a82deaf_2048x720.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><strong><span>Confident hallucination</span></strong><span>. The model asserts something untrue with exactly the fluency it uses for things that are true. We treat this as a model defect, and partly it is. But a model can only separate supported claims from unsupported ones if something in its context gives it a way to do that. The </span><a href="https://en.wikipedia.org/wiki/Mata_v._Avianca,_Inc."><span>sanctioned lawyers</span></a><span> weren&#8217;t undone by fabrication alone. They were undone by a workflow in which nothing downstream of the fabrication was designed to catch it.</span></p><p><strong><span>Specification ambiguity</span></strong><span>. This is the one I find most underappreciated. Faced with an obviously underdetermined request, an agent will often stop and ask for clarification. Faced with a semi-ambiguous request, it does something considerably more dangerous. </span><a href="https://arxiv.org/abs/2304.14399v1"><span>It picks an interpretation and proceeds</span></a><span>. If that first read is wrong, every subsequent step compounds it, and the work looks confident the entire way down. Nothing was missing from the context. It simply admitted more than one reasonable reading, and nobody noticed which one got chosen.</span></p><p><strong><span>Stale guidance</span></strong><span>. Context that accurately describes </span><a href="https://engineering.fb.com/2026/04/06/developer-tools/how-meta-used-ai-to-map-tribal-knowledge-in-large-scale-data-pipelines/"><span>a system that no longer exists</span></a><span>. The runbook that was correct two migrations ago. The architecture doc written before the service was split. A human reading these usually senses the mismatch and goes looking for why. An agent has no such reflex, and treats them as current.</span></p><p><strong><span>Lost in the middle</span></strong><span>. More context is not reliably better. Model performance degrades as input grows, and it degrades unevenly, </span><a href="https://doi.org/10.1162/tacl_a_00638"><span>with material buried in the middle of a long context</span></a><span> getting overlooked in favor of what sits at the edges. This one is genuinely counterintuitive, because nothing is missing. The agent has everything it needs and still fails, because the signal is competing with too much else for attention. Agents, like people, can get distracted.</span></p><p><strong><span>Lost in the details</span></strong><span>. Context that supplies plenty of implementation detail and none of the reasoning behind it. The agent optimizes local mechanics beautifully while losing track of what the task was actually for. Margaret-Anne Storey calls the underlying condition </span><a href="https://queue.acm.org/detail.cfm?id=3807966"><span>intent debt</span></a><span>, the gradual disappearance of recorded rationale from the artifacts a system leaves behind. It&#8217;s the difference between knowing that a timeout is set to 300 milliseconds and knowing whether that number came from a benchmark, a support incident, or somebody&#8217;s afternoon guess.</span></p><p><strong><span>Weekend runaway</span></strong><span>. An agent given a task with no definition of success may never discover that it&#8217;s finished. So it retries, widens scope, and </span><a href="https://arxiv.org/abs/2503.13657"><span>keeps exploring an objective that may not even be achievable</span></a><span>. The result is consumed time, consumed tokens, and occasionally a genuinely startling bill on Monday morning. This may be the least intellectually interesting failure on the list, but maybe the one most likely to get a leader&#8217;s attention.</span></p><h3><span>None of these are new</span></h3><p><span>Read that list again and notice how little of it is actually about AI.</span></p><p><span>Ambiguous specifications have been the central concern of requirements engineering for decades. Documentation that drifts out of sync with the system it describes is one of the oldest complaints in software maintenance. Signal drowning in noise is the founding problem of information retrieval. Rationale vanishing from the artifacts a team leaves behind is what knowledge management has always been about.</span></p><p><span>None of these challenges are new.  Bad context has always been a normal fact-of-life for software organizations. What kept it from being catastrophic is that humans quietly repaired it, constantly, without anyone noticing.</span></p><p><span>A new engineer handed an ambiguous ticket walks over and asks what it means. Handed a runbook that doesn&#8217;t match production, they notice the discrepancy and go find someone who knows. Handed two documents that contradict each other, they work out which one smells current, or they ask in a channel and get an answer in four minutes. Every organization I&#8217;ve studied runs on documentation that would fail inspection, and it works anyway, because a layer of human judgment sits between the bad context and the actual work.</span></p><p><span>Agents don&#8217;t reliably have that same repair loop, especially as we race to make them more autonomous. Handed the same ambiguous ticket, the same stale runbook, the same contradictory pair of documents, they proceed. Confidently, quickly, and at a scale that used to be impossible.</span></p><p><span>I don&#8217;t think AI created a context problem. I think it withdrew the error correction that had been hiding one. The information environment we&#8217;ve been tolerating was always this bad. If that&#8217;s right, it may also help explain one of DORA&#8217;s more interesting findings about AI adoption. </span><a href="https://dora.dev/research/2025/"><span>Their 2025 research</span></a><span> describes AI as an amplifier. It tends to magnify the strengths and weaknesses of the systems already surrounding agents. That&#8217;s usually framed as an organizational capability problem, but I wonder if part of it is an information problem. Teams with clearer specifications, better documentation, and more accessible organizational knowledge may simply have less damage for the human buffer to absorb in the first place.</span></p><h3><span>What you can do</span></h3><p><span>Most of what I&#8217;ve described sounds like it needs instrumentation we don&#8217;t have yet. It doesn&#8217;t (at least not to start).</span></p><p><span>Before handing a task to an agent, ask whether a competent new colleague could complete it with what you&#8217;re making available. Not a senior engineer who already knows your system. A capable person with no additional access to what&#8217;s in your head.</span></p><p><span>That test is deliberately low-tech, and it catches a surprising amount. The specification that admits two readings. The runbook describing last year&#8217;s architecture. The fifty-page dump with the one important constraint on page thirty-one. The task with no stated definition of done.</span></p><p><span>A competent new colleague would probably stop at each of those. Which interpretation did you mean? Is this runbook still current? Which of these fifty pages actually matters? What does done look like? Those questions are the human repair loop in action. The more autonomous we make agents, the less we should assume that loop will save bad context.</span></p><p><span>Naming these patterns is a start, but only a start. Context smells give us a vocabulary for recognizing trouble. They don&#8217;t give us a systematic way to evaluate the quality of context before an agent starts working. That second problem is considerably harder, and it&#8217;s where I&#8217;ve been spending most of my time lately. Over the coming weeks, I&#8217;ll be publishing a framework describing the dimensions of context quality, followed by an approach to measuring those dimensions.</span></p><p><span>For now, though, the question I&#8217;d put to any team frustrated with their agents isn&#8217;t which model they should be using.</span></p><p><span>It&#8217;s whether they&#8217;d have given the same task to a new hire and expected them to get it right.</span></p><div><hr></div><p>That&#8217;s it for this week. Thanks for reading.</p><p>-Brian</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.getdx.com/p/your-agent-doesnt-have-a-model-problem?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.getdx.com/p/your-agent-doesnt-have-a-model-problem?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p>]]></content:encoded></item><item><title><![CDATA[How Okta sets guardrails and context for AI agents]]></title><description><![CDATA[Okta Chief Architect Robert Lucero explains how organizations can securely deploy AI agents while improving engineering productivity, testing, and software delivery.]]></description><link>https://newsletter.getdx.com/p/how-okta-governs-ai-agents-at-enterprise</link><guid isPermaLink="false">https://newsletter.getdx.com/p/how-okta-governs-ai-agents-at-enterprise</guid><dc:creator><![CDATA[Brian Houck]]></dc:creator><pubDate>Fri, 11 Sep 2026 15:26:03 GMT</pubDate><enclosure url="https://api.substack.com/feed/podcast/214945154/730bcabfb8f2312341627e1d34c47850.mp3" length="0" type="audio/mpeg"/><content:encoded><![CDATA[<p>Listen and watch now on <strong><a href="https://youtu.be/KgA5wy4BINk">YouTube</a>, <a href="https://podcasts.apple.com/us/podcast/engineering-enablement-by-abi-noda/id1619140476">Apple</a>, and <a href="https://open.spotify.com/show/3NxjyIsuxeDMQtisDqBy7D">Spotify</a></strong>.</p><p><span>In this episode of Engineering Enablement, I sit down with Robert Lucero, Chief Architect at Okta, to discuss how agentic AI is changing identity and access management. We explore how organizations should define agent identities, why authentication and authorization remain foundational, and how sandboxing, fine-grained permissions, and just-in-time access let agents operate autonomously without creating unacceptable risk.</span></p><p><span>Robert shares what Okta has learned from driving AI adoption among security-minded engineers, what makes a repository ready for AI agents, and why strong testing, CI, and review processes matter even more as AI generates more code. We also discuss AI&#8217;s role in software validation, the need for human judgment, and whether AI will ultimately give the advantage to security teams or attackers.</span></p><div id="youtube2-KgA5wy4BINk" class="youtube-wrap" data-attrs="{&quot;videoId&quot;:&quot;KgA5wy4BINk&quot;,&quot;startTime&quot;:null,&quot;endTime&quot;:null}" data-component-name="Youtube2ToDOM"><div class="youtube-inner"><iframe src="https://www.youtube-nocookie.com/embed/KgA5wy4BINk?rel=0&amp;autoplay=0&amp;showinfo=0&amp;enablejsapi=0" frameborder="0" loading="lazy" gesture="media" allow="autoplay; fullscreen" allowautoplay="true" allowfullscreen="true" width="728" height="409"></iframe></div></div><h2><strong>Some takeaways: </strong></h2><h3><strong><span>Identity becomes critical when AI agents move beyond content generation</span></strong></h3><ul><li><p><strong><span>AI agents need access to documents, source code, CI systems, ticketing tools, and other resources to perform useful work.</span></strong><span> Identity systems determine which resources they can access and what actions they can take.</span></p></li><li><p><strong><span>Non-determinism makes tightly scoped access especially important.</span></strong><span> Organizations need dynamic policies, fine-grained permissions, and visibility when agents attempt to operate beyond their boundaries.</span></p></li></ul><h3><strong><span>The definition of an agent identity is still evolving</span></strong></h3><ul><li><p><strong><span>Organizations have not yet settled on whether every agent or ephemeral workload needs its own identity.</span></strong><span> An agent might instead operate under an orchestrator, a service account, or the identity of a human owner.</span></p></li><li><p><strong><span>The fundamentals of identity remain the same.</span></strong><span> Agents still need to be authenticated before organizations can apply authorization policies, governance, and access controls.</span></p></li></ul><h3><strong><span>Treat AI agents like new hires</span></strong></h3><ul><li><p><strong><span>Start agents in a sandbox with tightly scoped access and clearly defined tasks.</span></strong><span> Like new employees, agents may seek out tools or information when their environment is missing something they need.</span></p></li><li><p><strong><span>Rely on technical controls rather than assuming an agent has earned trust.</span></strong><span> Sandboxing, guardrails, and just-in-time access can limit the blast radius of unpredictable behavior.</span></p></li></ul><h3><strong><span>Identity systems are a control plane, not an orchestration layer</span></strong></h3><ul><li><p><strong><span>The identity layer governs what agents can access, trigger, and call.</span></strong><span> The workflows connecting coding, review, testing, and deployment agents will likely be orchestrated elsewhere.</span></p></li><li><p><strong><span>Identity systems can provide essential visibility into agent relationships.</span></strong><span> A relational record of agent-to-agent and agent-to-resource access makes it possible to investigate why an agent was able to take a particular action.</span></p></li></ul><h3><strong><span>Successful AI adoption depends on useful outcomes</span></strong></h3><ul><li><p><strong><span>Okta&#8217;s security-minded engineers were initially skeptical because early tools did not reliably help them do their jobs.</span></strong><span> Adoption increased as coding models improved and became more effective in real engineering workflows.</span></p></li><li><p><strong><span>The value of AI extends far beyond writing code.</span></strong><span> Engineers can use it to summarize documentation, review product specs, investigate production issues, and respond to customer requests more efficiently.</span></p></li></ul><h3><strong><span>AI readiness starts with engineering fundamentals</span></strong></h3><ul><li><p><strong><span>Repositories need clear instructions, discoverable dependencies, compatible harnesses, and defined development practices before agents can work effectively.</span></strong></p></li><li><p><strong><span>Strong testing, CI, linting, and code review provide the safety nets AI-generated changes require.</span></strong><span> Improving a repository for AI agents often makes it easier for human developers to work in as well.</span></p></li></ul><h3><strong><span>Passing tests does not prove that AI-generated code is correct</span></strong></h3><ul><li><p><strong><span>An agent can write tests and produce code that passes them without understanding the intended business outcome.</span></strong><span> Reliable validation depends on giving agents access to the right requirements and context.</span></p></li><li><p><strong><span>Human judgment remains essential for determining whether software actually solves the problem.</span></strong><span> Developers, designers, test engineers, and product owners must still evaluate whether the result meets user and business needs.</span></p></li></ul><h2><strong>In this episode, we cover:</strong></h2><p>(<a href="https://www.youtube.com/watch?v=KgA5wy4BINk">00:00</a>) Intro</p><p>(<a href="https://www.youtube.com/watch?v=KgA5wy4BINk&amp;t=136s">02:16</a>) Meet Robert Lucero</p><p>(<a href="https://www.youtube.com/watch?v=KgA5wy4BINk&amp;t=176s">02:56</a>) Why identity becomes critical as AI agents access more systems</p><p>(<a href="https://www.youtube.com/watch?v=KgA5wy4BINk&amp;t=367s">06:07</a>) Governing automated access for non-deterministic agents</p><p>(<a href="https://www.youtube.com/watch?v=KgA5wy4BINk&amp;t=481s">08:01</a>) Defining and managing agent identities</p><p>(<a href="https://www.youtube.com/watch?v=KgA5wy4BINk&amp;t=632s">10:32</a>) Authentication and authorization for AI agents</p><p>(<a href="https://www.youtube.com/watch?v=KgA5wy4BINk&amp;t=813s">13:33</a>) Determining how much autonomy to give AI agents</p><p>(<a href="https://www.youtube.com/watch?v=KgA5wy4BINk&amp;t=1184s">19:44</a>) Identity systems as the control plane for AI agents</p><p>(<a href="https://www.youtube.com/watch?v=KgA5wy4BINk&amp;t=1360s">22:40</a>) Driving AI adoption among security-minded engineers</p><p>(<a href="https://www.youtube.com/watch?v=KgA5wy4BINk&amp;t=1548s">25:48</a>) Why AI&#8217;s impact on engineering extends beyond coding</p><p>(<a href="https://www.youtube.com/watch?v=KgA5wy4BINk&amp;t=1724s">28:44</a>) The three components of AI readiness for repositories</p><p>(<a href="https://www.youtube.com/watch?v=KgA5wy4BINk&amp;t=1855s">30:55</a>) AI&#8217;s role in software testing and validation</p><p>(<a href="https://www.youtube.com/watch?v=KgA5wy4BINk&amp;t=2066s">34:26</a>) Whether AI gives defenders or attackers the advantage</p><p>(<a href="https://www.youtube.com/watch?v=KgA5wy4BINk&amp;t=2161s">36:01</a>) Where to find the best food in the country</p><p><strong><span>Where to find Robert Lucero:</span></strong></p><p><span>&#8226; LinkedIn: </span><a href="https://www.linkedin.com/in/rlucero"><span>https://www.linkedin.com/in/rlucero</span></a></p><p><strong><span>Where to find Brian Houck:</span></strong></p><p><span>&#8226; LinkedIn: </span><a href="https://www.linkedin.com/in/brianhouck"><span>https://www.linkedin.com/in/brianhouck</span></a></p><h2><strong>Referenced:</strong></h2><p><span>&#8226; </span><a href="https://getdx.com/report/State-of-AI-Impact-in-Engineering-Q2-Report"><span>State of AI Impact in Engineering Q2 Report 2026</span></a></p><p><span>&#8226; </span><a href="https://www.okta.com/"><span>Okta</span></a></p><p><span>&#8226; </span><a href="https://openai.com/index/hugging-face-incident-and-the-road-ahead/"><span>The Hugging Face incident and the road ahead | OpenAI</span></a></p><p><span>&#8226; </span><a href="https://github.com/features/copilot"><span>GitHub Copilot &#183; Your AI pair programmer</span></a></p><p><span>&#8226; </span><a href="https://claude.com/product/claude-code"><span>Claude Code by Anthropic | AI Coding Agent, Terminal, IDE</span></a></p><p><span>&#8226; </span><a href="https://queue.acm.org/doi/10.1145/3807963"><span>Eight Myths on Software Engineering and GenAI | Queue</span></a></p><p><span>&#8226; </span><a href="https://www.linkedin.com/in/adriana-corona/"><span>Adriana Corona</span></a></p>]]></content:encoded></item><item><title><![CDATA[AI accelerates output, not innovation]]></title><description><![CDATA[AI output explains 63% of developer time savings but shows a weak link to increased innovation ratio.]]></description><link>https://newsletter.getdx.com/p/ai-accelerates-output-not-innovation</link><guid isPermaLink="false">https://newsletter.getdx.com/p/ai-accelerates-output-not-innovation</guid><dc:creator><![CDATA[Grace Fu]]></dc:creator><pubDate>Wed, 09 Sep 2026 10:03:41 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/d4e1b8f4-50be-481f-ab99-a577b227a255_2400x1260.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><em><strong><span>Welcome to the latest issue of Engineering Enablement,</span></strong><span> a weekly newsletter sharing research and perspectives on developer productivity.</span></em></p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.getdx.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe now&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.getdx.com/subscribe?"><span>Subscribe now</span></a></p><p><span>&#128467; </span><a href="https://getdx.com/webinar/cafes-framework-improving-agent-effectiveness?utm_source=newsletter"><span>Join DX&#8217;s live research panel on September 24</span></a><span> as Brian Houck introduces CAFE(S), a framework for improving AI agent effectiveness through better context. </span><a href="https://getdx.com/webinar/cafes-framework-improving-agent-effectiveness?utm_source=newsletter"><span>Register here.</span></a></p><div><hr></div><p><span>As reported in our </span><a href="https://getdx.com/report/State-of-AI-Impact-in-Engineering-Q2-Report/?utm_source=newsletter"><span>Q2 AI Impact Report</span></a><span>, time saved thanks to AI continues to increase, with developers reporting saving 6.1 hours per week in Q2 2026 compared to 3 hours in Q3 2025. However, it raises the question for engineering leaders:</span></p><p><em><span>What are developers doing with that time?</span></em></p><p><span>While AI is frequently positioned as a way to reclaim time for innovation, our data reveals a disconnect between AI-driven output and actual shifts in work composition.</span><em><span> </span></em><span>Organizations are investing in AI with the hope that the time AI saves will directly translate into faster roadmap execution and more innovation. DX tracks this using innovation ratio, the percent of engineering effort dedicated to new capabilities versus maintenance.</span></p><p><span>We used multivariate regression on a sample of 500+ DX customers to test the innovation ratio against 15 workflow metrics, including AI output, shipping velocity, and operational drag. We analyzed AI output according to how much AI-authored code, agent-delivered work, and PR throughput teams produce.</span><em><span> </span></em><span>Our analysis shows that while</span><strong><span> AI output has a powerful link to AI-driven time savings, it does not reliably translate into a higher innovation ratio.</span></strong></p><h3><span>Information-seeking may be the hidden barrier to innovation</span></h3><p><span>While AI tools are delivering measurable speed, our data suggests that reclaiming capacity for new features requires clearing organizational friction, rather than assuming AI will lead to innovation.</span></p><ol><li><p><strong><span>AI is a lever for speed</span></strong><span>. The data confirms that AI does what it promises. AI output accounts for 63% of the variation in developer time savings, suggesting that developers with higher AI output save more time.</span></p></li><li><p><strong><span>Speed doesn&#8217;t convert to innovation</span></strong><span>. Despite increased time savings, AI output suggests a weak link to innovation (p &lt; 0.01, &#946;=0.16). No single workflow metric emerged as a reliable predictor of innovation, indicating that dedicating more time for new features and capabilities is not a byproduct of moving faster.</span></p></li><li><p><strong><span>The impact of operational friction</span></strong><span>. The clearest signal tied to lower innovation ratio is information-seeking, referring to time lost hunting for context, documentation, or answers (p &lt; 0.01, &#946;=-0.19). It is not a large effect and needs additional validation, but this developer-reported friction represents a significant hypothesis for engineering leaders. It suggests that reducing operational drag may be a more effective path to increasing innovation than AI (or AI alone).</span></p></li></ol><blockquote></blockquote><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!BkZh!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f375706-f86b-47d4-8845-042a16d3a02b_2048x1448.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!BkZh!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f375706-f86b-47d4-8845-042a16d3a02b_2048x1448.jpeg 424w, https://substackcdn.com/image/fetch/$s_!BkZh!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f375706-f86b-47d4-8845-042a16d3a02b_2048x1448.jpeg 848w, https://substackcdn.com/image/fetch/$s_!BkZh!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f375706-f86b-47d4-8845-042a16d3a02b_2048x1448.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!BkZh!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f375706-f86b-47d4-8845-042a16d3a02b_2048x1448.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!BkZh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f375706-f86b-47d4-8845-042a16d3a02b_2048x1448.jpeg" width="1456" height="1029" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2f375706-f86b-47d4-8845-042a16d3a02b_2048x1448.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1029,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!BkZh!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f375706-f86b-47d4-8845-042a16d3a02b_2048x1448.jpeg 424w, https://substackcdn.com/image/fetch/$s_!BkZh!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f375706-f86b-47d4-8845-042a16d3a02b_2048x1448.jpeg 848w, https://substackcdn.com/image/fetch/$s_!BkZh!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f375706-f86b-47d4-8845-042a16d3a02b_2048x1448.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!BkZh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f375706-f86b-47d4-8845-042a16d3a02b_2048x1448.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h2><span>Key takeaways for engineering leaders</span></h2><ol><li><p><strong><span>Decouple AI strategy from innovation goals</span></strong><span>. AI delivers speed, but there is no evidence it automatically shifts teams toward new-feature work. Treat innovation as a separate objective requiring its own strategy</span></p></li><li><p><strong><span>Test the operational friction hypothesis</span></strong><span>. Focus on reducing time lost to information-seeking. While not yet a proven driver of innovation, it is the most promising lead for improving developer conditions.</span></p></li><li><p><strong><span>Use this as a diagnostic.</span></strong><span> Because our model accounted for only 13% of the differences in innovation ratio, much of what drives new-feature work remains unique to each company. The most valuable next step is running this analysis against your own team&#8217;s data; your strongest predictors might differ from the aggregate.</span></p><div><hr></div><p>That&#8217;s it for this week. Thanks for reading.<br>-Grace</p><p></p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.getdx.com/p/ai-accelerates-output-not-innovation?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.getdx.com/p/ai-accelerates-output-not-innovation?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><p></p></li></ol>]]></content:encoded></item><item><title><![CDATA[The quality paradox of AI-generated code]]></title><description><![CDATA[What a five-part definition of quality from 2020 tells us about software development in the age of AI.]]></description><link>https://newsletter.getdx.com/p/the-quality-paradox-of-ai-generated</link><guid isPermaLink="false">https://newsletter.getdx.com/p/the-quality-paradox-of-ai-generated</guid><dc:creator><![CDATA[Brian Houck]]></dc:creator><pubDate>Wed, 02 Sep 2026 10:03:27 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/7ba4ba74-fbe8-4295-8b68-7c0cb012b63c_2400x1260.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><em><strong><span>Welcome to the latest issue of Engineering Enablement, </span></strong><span>a weekly newsletter sharing research and perspectives on developer productivity.</span></em></p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.getdx.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe now&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.getdx.com/subscribe?"><span>Subscribe now</span></a></p><p>&#128467; <a href="https://getdx.com/webinar/cafes-framework-improving-agent-effectiveness/#content">Join our live research panel on September 24</a> as we introduce CAFE(S), a framework for improving AI agent effectiveness through better context. <a href="https://getdx.com/webinar/cafes-framework-improving-agent-effectiveness?utm_source=newsletter">Register here.</a></p><div><hr></div><p><span>Two numbers from the </span><a href="https://getdx.com/report/state-of-ai-impact-in-engineering-q2-report/?utm_source=newsletter"><span>2026 Q2 State of AI Impact report</span></a><span> have raised a lot of questions for me about the impact of AI on &#8220;quality&#8221;.</span></p><p><span>Across a sample of more than 500 companies, </span><em><span>code maintainability</span></em><span> improved 3.8% between Q1 and Q2. Over the same period </span><em><span>change confidence</span></em><span> fell 6.1%. Maintainability asks how easy the codebase is to understand and modify. Change confidence asks whether developers trust that their changes won&#8217;t break things.</span></p><p><span>Those two measures have historically moved together. Code you understand is code you feel safe changing. Now they are moving in opposite directions.</span></p><p><span>Layering in other findings from the report makes the picture even less clear. Documentation quality, code maintainability, and production debugging had some of the biggest improvements in the quarter&#8212;all signals we might expect to provide tailwinds to quality. At the same time, review turnaround declined, incremental delivery fell sharply, and pull requests continued to get larger&#8212;all signals that we might expect to provide headwinds.</span></p><p><span>Perhaps </span><em><span>quality </span></em><span>isn&#8217;t simply going up or down. Perhaps different parts of it are moving in different directions.</span></p><p><span>To help me frame the problem, I went back to a definition of quality that a couple of colleagues and I developed in 2020, several years before generative AI complicated the picture. I wanted to know whether our definition could help make sense of what we&#8217;re seeing now.</span></p><p><span>Turns out, the places where the data appears most contradictory make considerably more sense once we stop treating &#8220;quality&#8221; as one thing.</span></p><h3><span>What developers mean by quality, before AI could complicate it</span></h3><p><a href="https://arxiv.org/pdf/2111.04302"><span>In January 2020, Margaret-Anne Storey, Tom Zimmermann and I surveyed 131 developers and 34 engineering managers</span></a><span> at Microsoft about how they define productivity and quality, in their own words. </span><a href="https://newsletter.getdx.com/p/define-productivity"><span>(Paper summary here.)</span></a><span> We coded the open-text answers inductively, and five dimensions of quality fell out. We called these dimensions TRUCE, which are defined below:</span></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!SSUr!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5a9425f-e1ed-42c9-b649-9f8fc4b74da5_1830x1074.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!SSUr!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5a9425f-e1ed-42c9-b649-9f8fc4b74da5_1830x1074.png 424w, https://substackcdn.com/image/fetch/$s_!SSUr!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5a9425f-e1ed-42c9-b649-9f8fc4b74da5_1830x1074.png 848w, https://substackcdn.com/image/fetch/$s_!SSUr!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5a9425f-e1ed-42c9-b649-9f8fc4b74da5_1830x1074.png 1272w, https://substackcdn.com/image/fetch/$s_!SSUr!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5a9425f-e1ed-42c9-b649-9f8fc4b74da5_1830x1074.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!SSUr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5a9425f-e1ed-42c9-b649-9f8fc4b74da5_1830x1074.png" width="1456" height="855" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a5a9425f-e1ed-42c9-b649-9f8fc4b74da5_1830x1074.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:855,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:67150,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://newsletter.getdx.com/i/211766445?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5a9425f-e1ed-42c9-b649-9f8fc4b74da5_1830x1074.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!SSUr!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5a9425f-e1ed-42c9-b649-9f8fc4b74da5_1830x1074.png 424w, https://substackcdn.com/image/fetch/$s_!SSUr!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5a9425f-e1ed-42c9-b649-9f8fc4b74da5_1830x1074.png 848w, https://substackcdn.com/image/fetch/$s_!SSUr!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5a9425f-e1ed-42c9-b649-9f8fc4b74da5_1830x1074.png 1272w, https://substackcdn.com/image/fetch/$s_!SSUr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5a9425f-e1ed-42c9-b649-9f8fc4b74da5_1830x1074.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><span>Six years later, TRUCE gives us a useful lens for interpreting what otherwise looks like contradictory evidence about AI and software quality.</span></p><h3><span>Robustness is unsettled, but we can see it</span></h3><p><span>Start with the dimension we instrument best.</span></p><p><span>Change failure rate volatility widened in Q2, with many organizations now swinging plus or minus three percentage points against a 4% industry benchmark. The distribution of failed deployment recovery time is leaning longer, even though developers&#8217; experience with production debugging improved slightly.</span></p><p><span>In other words, teams report getting better at diagnosing production problems while recovery outcomes are becoming more variable.</span></p><p><span>It&#8217;s important not to turn correlation into causation here. This data doesn&#8217;t tell us that AI created volatility, or even that AI amplified it. What we can say is that robustness is becoming less predictable during the same period in which AI-authored code has accelerated dramatically.</span></p><p><span>Robustness itself isn&#8217;t mysterious; we know what we&#8217;re trying to observe. Many metrics exist to try and measure robustness: failure rates, recovery times, incidents, defects, test outcomes, and more. We can debate exactly which measures are best, but when robustness changes, we have a reasonably mature vocabulary for describing what happened.</span></p><p><span>Hold onto that, because robustness turns out to be the exception. The dimensions that follow are roughly ordered by how well we can measure them, and as that measurement thins out, something consistent happens: the dimension stops behaving like one thing. Whether those two facts are related is a question I want to come back to.</span></p><h3><span>Evolvability is splitting into two things</span></h3><p><span>TRUCE derived evolvability from developers defining quality as &#8220;</span><em><span>Easy to test, </span><strong><span>easy to change</span></strong></em><span>.&#8221;  It&#8217;s natural to assume that code you understand is code you can feel safe changing. Recent academic research suggests something different may be happening when we zoom out from individual code changes to the systems they become part of.</span></p><p><span>In </span><em><a href="https://arxiv.org/abs/2601.21276"><span>More Code, Less Reuse</span></a></em><span>, Huang and colleagues compared pull requests generated by AI agents with those written by human developers. They found that AI agents were more likely to miss opportunities to reuse existing code, introducing greater redundancy. Interestingly, reviewers still tended to respond neutrally or positively to the AI-generated contributions. The authors describe this as a disconnect between the &#8220;surface-level plausibility&#8221; of the code and the technical debt accumulating underneath it.</span></p><p><span>Another recent study makes the distinction even clearer. </span><em><a href="https://arxiv.org/abs/2603.27745"><span>Needle in the Repo</span></a></em><span> evaluated 23 AI coding configurations on repository-level tasks designed to test maintainability, not just whether the resulting code worked. Across those configurations, 13.3% of outcomes passed every functional test while still failing the study&#8217;s structural maintainability criteria. Performance was especially poor on architectural concerns such as dependency control and responsibility decomposition.</span></p><p><span>Both findings point toward the same distinction. A piece of code can be easy to understand. It can work. Reviewers can even think it looks good. But the system it belongs to can still become harder to evolve.</span></p><p><span>The emerging pattern is that locally good code doesn&#8217;t guarantee a healthy system. Generated code may be perfectly understandable in isolation while duplicating something that already exists, introducing an unnecessary dependency, or putting functionality in a place that makes the broader system harder to change.</span></p><p><span>And that brings us back to the numbers I started with. Maintainability improved 3.8%. Change confidence fell 6.1%. Perhaps those measures aren&#8217;t disagreeing after all. They may simply be looking at different parts of evolvability.</span></p><h3><span>Collaboration is splitting too</span></h3><p><span>One of my favorite definitions of quality from the TRUCE paper came from a developer describing collaboration, one of the least frequently mentioned dimensions: </span></p><blockquote><p><em><span>&#8220;Quality is code or solutions that solve a problem and don&#8217;t need undue maintenance or lengthy handoff. If I get hit by a bus and the company can still easily use the code I&#8217;ve written, I&#8217;ve made a quality solution.&#8221;</span></em></p></blockquote><p><span>We wrote at the time that enabling collaboration or reuse by others &#8220;is often invisible and hard to measure,&#8221; which likely explains why it was a less frequently mentioned dimension, &#8220;and yet it is very important.&#8221; I did not expect that sentence to age the way it has.</span></p><p><span>This is where the data gets strange. Documentation quality was among the clearest wins in the quarter. That matters for collaboration. Documentation supports handoff, comprehension, and the ability of someone else to pick up your work.</span></p><p><span>At the same time, review turnaround declined. Incremental delivery, which asks whether developers work in small, testable changes, showed the sharpest decrease of any driver. Median PR size nearly doubled over the year, increasing the amount of unfamiliar code developers need to understand at once.</span></p><p><span>So we have better documentation attached to larger changes that are harder to review, written increasingly by something that isn&#8217;t on the team. Documentation quality and shared understanding aren&#8217;t the same thing. AI can document what the code does without transferring the reasoning that led to it. It can make handoff easier while making shared ownership harder.</span></p><p><span>That&#8217;s what makes the bus-factor definition feel so different to me now. AI may be improving the explicit artifacts that support handoff while making the implicit knowledge surrounding the work more difficult to maintain.</span></p><p><span>The collaboration dimension that only 17% of developers thought to mention in 2020 suddenly looks much more important. And we said at the time that we couldn&#8217;t see it very well.</span></p><h3><span>Timeliness is not the same thing as speed</span></h3><p><span>Only 7% of developers named timeliness as a dimension of quality in 2020, the lowest of the five dimensions. It is now the site of perhaps the cleanest decoupling in the data.</span></p><p><span>Median normalized PR Throughput rose 37% over four quarters. Deployment frequency is up across nearly every segment. However, over the same period, developers&#8217; perceived rate of delivery was flat.</span></p><p><span>That looks contradictory only if we assume delivery speed and timeliness are the same thing. Timeliness was not intended to be a pipeline property. It was about whether what was needed arrived when it was needed.</span></p><p><span>AI can increase the amount of code flowing through a system without changing how quickly an engineer feels able to get the thing that matters into a customer&#8217;s hands. More PRs can run into slower reviews. Faster implementation can expose bottlenecks in testing, deployment, requirements, or coordination. Additional output can simply create additional work downstream.</span></p><p><span>The pipeline got measurably faster. The felt experience of delivering on time did not.</span></p><p><span>Once again, TRUCE makes the apparent contradiction easier to understand. Our delivery-speed metrics and our experience of timeliness are measuring related things, but they aren&#8217;t measuring the same thing.</span></p><h3><span>User needs is the dimension we still can&#8217;t see</span></h3><p><span>It would be a stretch to say much of anything about AI&#8217;s impact on this dimension from the available data. That is itself a finding.</span></p><p><span>The closest available proxy in the Q2 data is perceived software quality, and it says something counterintuitive. Traditional industries and financial services, the slowest-moving segments on throughput, report the highest perceived quality.</span></p><p><span>That is self-reported and may say as much about expectations as about software, so I&#8217;d stop short of any claim stronger than this: speed and perceived quality are not moving together.</span></p><p><span>I also went looking for external research on whether the people who use software are experiencing it differently as AI authorship climbs. I came up mostly empty.</span></p><p><span>There is a great deal on defect rates, vulnerability counts, code quality, and QA workload. We have remarkably detailed telemetry about how much code AI writes, how quickly pull requests move, how frequently teams deploy, and how often changes fail. We know considerably less about whether the resulting software is actually solving people&#8217;s problems better.</span></p><p><span>That&#8217;s a striking asymmetry.</span></p><p><span>One developer in the study told us &#8220;</span><em><span>Quality is doing what the customer wants.</span></em><span>&#8221; Four years into this shift, we can measure changes in the machinery of software production with extraordinary precision while remaining largely blind to what may be the most important dimension of quality: did the thing we built serve the people using it?</span></p><h3><span>Quality was never just about the code</span></h3><p><span>Put the five dimensions back together and the story isn&#8217;t that AI is improving quality. It also isn&#8217;t that AI is destroying it. The story is that &#8220;quality&#8221; was never a scalar.</span></p><p><span>Seen this way, the maintainability/change-confidence paradox that started this piece doesn&#8217;t look like a paradox anymore. It looks like exactly what we should expect when a multidimensional construct is being pushed in different directions at once.</span></p><p><span>Six years on, I don&#8217;t think AI makes TRUCE less useful. If anything, it makes the framework more useful by reminding us what we&#8217;re actually trying to preserve. We still want software that is robust, evolvable, collaborative, timely, and serves the people using it.</span></p><p><span>What may be changing is where we need to look to understand whether we&#8217;ll get those things.</span></p><p><span>Reuse requires knowing what already exists in the codebase. Review requires knowing what a change was meant to accomplish. Evolvability requires knowing why the system is structured the way it is. Handoff requires knowing what the previous person was thinking. Serving user needs requires knowing what problem the software was supposed to solve in the first place.</span></p><p><span>None of that knowledge necessarily lives in the code.</span></p><p><span>Margaret-Anne Storey, my TRUCE co-author, has arrived somewhere similar from a different direction. Her </span><a href="https://queue.acm.org/detail.cfm?id=3807966"><span>triple debt model</span></a><span> separates technical debt, which lives in code, from cognitive debt, which lives in a team&#8217;s shared understanding, and intent debt, which lives in the artifacts recording what a system is for.</span></p><p><span>Her argument is that AI may reduce one kind of debt while accelerating the others, and that code is no longer necessarily the binding constraint on software health.</span></p><p><span>Her model maps to the findings we just walked through. Technical debt is visible in the evolvability problem. Cognitive debt shows up in the collaboration split. And intent debt helps explain why it is so difficult to know whether a system is meeting real user needs, the dimension we can barely see at all. Hers is a conceptual model rather than an empirical one, so I&#8217;d call this convergence rather than confirmation. But two frameworks built for different purposes landed on the same three places.</span></p><p><span>There&#8217;s a small irony worth noting. When </span><a href="https://aka.ms/aipaper"><span>Mansi Khemka and I asked developers in 2023</span></a><span> what they most wanted AI help with, authoring and refactoring code ranked fifth, at 25%. Documentation ranked third at 37%.</span></p><p><span>Documentation is now one of the clearest quality wins in the data. Refactoring is one of the structural signals moving most sharply in the other direction.</span></p><p><span>I wouldn&#8217;t read too much into a single comparison. But it is interesting to me that the dimension holding up best is the one where the tooling matched what developers said they wanted.</span></p><p><span>Which brings me back to the artifact itself. We have spent decades learning to evaluate code. It is worth asking whether code is still the thing that carries the most information about whether these five dimensions will hold.</span></p><h3><span>Code is not the only artifact worth evaluating</span></h3><p><span>For most of software&#8217;s history, code was the currency of software engineering. A developer understood the requirement, carried some model of the architecture in their head, knew what neighboring systems did, made a series of design decisions, and eventually expressed those decisions as code. That made code an unusually information-rich artifact, and it is why evaluating the code was a critical way to evaluate the work.</span></p><p><span>That is changing. The Q2 report found that 52.7% of code is AI-authored, up from 24% in Q4 2025 and 34% in Q1 2026. Increasingly, a developer doesn&#8217;t translate everything they know directly into code. They express an intent, provide some representation of the relevant context, and ask a machine to produce the implementation.</span></p><p><span>Intent and context aren&#8217;t quite the same thing. Intent is what a person ultimately wants to accomplish. That intent must then be translated into context, which is the information that represents the task, its constraints, and the surrounding knowledge. More of the knowledge a developer once carried implicitly now has to be externalized before the work can begin.</span></p><p><span>TRUCE still gives us language for evaluating the artifact at the end of that chain, and AI doesn&#8217;t change that. What concerns me is that the end of the chain is becoming a late place to look. As development shifts from writing implementations toward expressing intent, more of the conditions that determine all five dimensions are set before any code exists.</span></p><p><span>We know how to ask whether the artifact is good. We&#8217;re only beginning to develop the vocabulary for asking whether the context that produced it was good enough to begin with.</span></p><div><hr></div><p>That&#8217;s it for this week. Thanks for reading.</p><p>-Brian</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.getdx.com/p/the-quality-paradox-of-ai-generated?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.getdx.com/p/the-quality-paradox-of-ai-generated?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p>]]></content:encoded></item><item><title><![CDATA[Code Quality & AI Readiness at Capital One]]></title><description><![CDATA[Max Kanat-Alexander, Executive Distinguished Engineer at Capital One, discusses how AI is reshaping software development and why strong engineering fundamentals matter more than ever.]]></description><link>https://newsletter.getdx.com/p/how-capital-one-assesses-ai-readiness</link><guid isPermaLink="false">https://newsletter.getdx.com/p/how-capital-one-assesses-ai-readiness</guid><dc:creator><![CDATA[Brian Houck]]></dc:creator><pubDate>Wed, 26 Aug 2026 11:31:48 GMT</pubDate><enclosure url="https://api.substack.com/feed/podcast/212730058/1af29867e3619ccf46328c74fd14afa7.mp3" length="0" type="audio/mpeg"/><content:encoded><![CDATA[<p>Listen and watch now on <strong><a href="https://youtu.be/gEuD4bP17uc">YouTube</a>, <a href="https://podcasts.apple.com/us/podcast/engineering-enablement-by-abi-noda/id1619140476">Apple</a>, and <a href="https://open.spotify.com/show/3NxjyIsuxeDMQtisDqBy7D">Spotify</a></strong>.</p><p><span>In this episode of Engineering Enablement, I sit down with Max Kanat-Alexander, Executive Distinguished Engineer at Capital One, to discuss how AI is changing software development and why strong engineering fundamentals matter more than ever.</span></p><p><span>We explore how the skills engineers need are expanding, why AI amplifies both the strengths and weaknesses of the development lifecycle, and how teams should think differently about code review, quality, and testing. Max also shares how leaders can assess whether their organizations are ready for more advanced AI workflows and why we need more research into how junior engineers become senior engineers.</span></p><div id="youtube2-gEuD4bP17uc" class="youtube-wrap" data-attrs="{&quot;videoId&quot;:&quot;gEuD4bP17uc&quot;,&quot;startTime&quot;:null,&quot;endTime&quot;:null}" data-component-name="Youtube2ToDOM"><div class="youtube-inner"><iframe src="https://www.youtube-nocookie.com/embed/gEuD4bP17uc?rel=0&amp;autoplay=0&amp;showinfo=0&amp;enablejsapi=0" frameborder="0" loading="lazy" gesture="media" allow="autoplay; fullscreen" allowautoplay="true" allowfullscreen="true" width="728" height="409"></iframe></div></div><h2><strong>Some takeaways: </strong></h2><h3><strong><span>AI amplifies the engineering system you already have</span></strong></h3><ul><li><p><strong><span>Strong engineering fundamentals matter even more with AI.</span></strong><span> AI amplifies everything that is good or bad about a software development lifecycle, so teams with strong tooling, CI, testing, and workflows are positioned to see the greatest gains.</span></p></li><li><p><strong><span>Faster coding doesn&#8217;t solve problems elsewhere in the development lifecycle.</span></strong><span> If engineers spend significant time navigating broken processes, waiting for reviews, or dealing with poor tooling, accelerating code generation only addresses a small part of their work.</span></p></li></ul><h3><strong><span>The skills engineers need are changing</span></strong></h3><ul><li><p><strong><span>Clearly expressing requirements and providing feedback are becoming increasingly important engineering skills.</span></strong><span> These are capabilities senior engineers have traditionally developed through years of experience, but they become essential much earlier when working effectively with coding agents.</span></p></li><li><p><strong><span>Deep technical knowledge still helps engineers recognize when something is wrong.</span></strong><span> Experience with languages, systems, and software design develops the pattern recognition needed to spot a flawed approach without reviewing hundreds of lines of AI-generated code.</span></p></li><li><p><strong><span>Domain understanding and problem definition are becoming more valuable.</span></strong><span> As AI gets better at reproducing knowledge found in documentation, engineers can differentiate themselves through understanding what needs to be built and why.</span></p></li></ul><h3><strong><span>Code review is fundamentally about maintainability</span></strong></h3><ul><li><p><strong><span>Human code review shouldn&#8217;t be the primary way teams catch bugs.</span></strong><span> Tests, static analysis, and other automation are more effective and scalable ways to detect defects.</span></p></li><li><p><strong><span>Human judgment remains important for evaluating maintainability and design.</span></strong><span> Reviewers can assess whether code is understandable, structured appropriately, and aligned with the longer-term intent of a system.</span></p></li><li><p><strong><span>The right quality bar depends on the codebase.</span></strong><span> A disposable script and enterprise software expected to operate for a decade shouldn&#8217;t be subjected to identical standards because the consequences and costs of getting them wrong are very different.</span></p></li></ul><h3><strong><span>Not every change needs the same review process</span></strong></h3><ul><li><p><strong><span>Code review is an important tool for developing less experienced engineers.</span></strong><span> Max says reviewing someone&#8217;s code is the most effective way he has found to improve their core software engineering skills.</span></p></li><li><p><strong><span>Trusted engineers may not need every change reviewed on codebases they deeply understand.</span></strong><span> That trust should be specific to the combination of the person and the codebase, rather than becoming a blanket exemption from review.</span></p></li><li><p><strong><span>The risk of the individual change can also determine whether human review is necessary.</span></strong><span> Teams can evaluate confidence and potential consequences rather than requiring the same review process for every PR.</span></p></li></ul><h3><strong><span>AI can create a vicious cycle of declining code quality</span></strong></h3><ul><li><p><strong><span>AI struggles when it has to build on poorly structured, poorly tested code.</span></strong><span> When weak review allows its bad output back into the codebase, the code becomes even harder for AI to work with, creating a cycle that can eventually leave teams with systems they can&#8217;t understand or maintain.</span></p></li><li><p><strong><span>AI can also help teams reverse that cycle.</span></strong><span> Agents can write tests, improve testability, and are particularly effective at refactoring when engineers deliberately direct them toward improving the codebase.</span></p></li><li><p><strong><span>Refactoring should support forward progress on product goals.</span></strong><span> Rather than attempting to clean up an entire codebase, teams should improve the parts they need to touch for upcoming work and make the necessary developer experience improvements along the way.</span></p></li></ul><h3><strong><span>AI readiness depends on how far left rigor extends</span></strong></h3><ul><li><p><strong><span>Fixing an existing software development lifecycle is much harder than building a good one from the beginning.</span></strong><span> Organizations with weak engineering practices often have to start with production problems and work backward through incident management, observability, deployment, testing, and earlier stages of development.</span></p></li><li><p><strong><span>How far left rigor extends is a useful signal of AI readiness.</span></strong><span> Organizations without strong controls around production, code review, and testing may introduce significant risk by giving engineers tools that allow them to move much faster.</span></p></li><li><p><strong><span>Leaders should evaluate the organization they actually have rather than following the AI hype cycle.</span></strong><span> Max is bullish on AI adoption, but argues that organizations need the engineering foundations to support the additional velocity it creates.</span></p></li></ul><h3><strong><span>The industry needs a better way to develop senior engineers</span></strong></h3><ul><li><p><strong><span>We still know surprisingly little about what reliably turns junior engineers into senior engineers.</span></strong><span> Mentorship and apprenticeship are widely considered important, but the practices that actually produce experienced engineers have not been deeply studied.</span></p></li><li><p><strong><span>AI makes understanding that development path more urgent.</span></strong><span> Senior engineers rely on experience to recognize failure patterns, which raises questions about how newer engineers will acquire that judgment as AI takes on more of the implementation work.</span></p></li><li><p><strong><span>AI may ultimately increase the need for engineering expertise rather than reduce it.</span></strong><span> As hundreds of millions more people gain the ability to build software and the rate of change accelerates, Max believes the world will need more experienced engineers to manage that complexity.</span></p></li></ul><h2><strong>In this episode, we cover:</strong></h2><p>(<a href="https://www.youtube.com/watch?v=gEuD4bP17uc">00:00</a>) Intro</p><p>(<a href="https://www.youtube.com/watch?v=gEuD4bP17uc&amp;t=114s">01:54</a>) Max&#8217;s role at Capital One</p><p>(<a href="https://www.youtube.com/watch?v=gEuD4bP17uc&amp;t=172s">02:52</a>) Where to invest in engineering organizations</p><p>(<a href="https://www.youtube.com/watch?v=gEuD4bP17uc&amp;t=396s">06:36</a>) The new entry-level engineering skills to pay attention to</p><p>(<a href="https://www.youtube.com/watch?v=gEuD4bP17uc&amp;t=639s">10:39</a>) Why deepening your understanding still matters</p><p>(<a href="https://www.youtube.com/watch?v=gEuD4bP17uc&amp;t=749s">12:29</a>) The bottlenecks around code review</p><p>(<a href="https://www.youtube.com/watch?v=gEuD4bP17uc&amp;t=1180s">19:40</a>) Why human code reviews still have value</p><p>(<a href="https://www.youtube.com/watch?v=gEuD4bP17uc&amp;t=1520s">25:20</a>) Why not all PRs need human review</p><p>(<a href="https://www.youtube.com/watch?v=gEuD4bP17uc&amp;t=1571s">26:11</a>) The vicious cycle of AI-driven development</p><p>(<a href="https://www.youtube.com/watch?v=gEuD4bP17uc&amp;t=1813s">30:13</a>) Using LLMs for refactoring</p><p>(<a href="https://www.youtube.com/watch?v=gEuD4bP17uc&amp;t=2038s">33:58</a>) AI readiness and why fixing engineering systems is so hard</p><p>(<a href="https://www.youtube.com/watch?v=gEuD4bP17uc&amp;t=2322s">38:42</a>) Why research is needed on creating good senior engineers</p><p>(<a href="https://www.youtube.com/watch?v=gEuD4bP17uc&amp;t=2460s">41:00</a>) Why AI will increase the need for engineers</p><p><strong><span>Where to find Max Kanat-Alexander:</span></strong></p><p><span>&#8226; LinkedIn: </span><a href="https://www.linkedin.com/in/mkanat"><span>https://www.linkedin.com/in/mkanat</span></a></p><p><span>&#8226; X: </span><a href="https://x.com/mkanat"><span>https://x.com/mkanat</span></a></p><p><span>&#8226; Blog: </span><a href="https://www.codesimplicity.com">https://www.codesimplicity.com</a></p><p><strong><span>Where to find Brian Houck:</span></strong></p><p><span>&#8226; LinkedIn: </span><a href="https://www.linkedin.com/in/brianhouck"><span>https://www.linkedin.com/in/brianhouck</span></a></p><h2><strong>Referenced:</strong></h2><p><span>&#8226; </span><a href="https://getdx.com/corefour"><span>DX Core 4 Productivity Framework</span></a></p><p><span>&#8226; </span><a href="https://www.capitalone.com/"><span>Capital One</span></a></p><p><span>&#8226; </span><a href="https://google.github.io/eng-practices/review/"><span>Code Review Guidelines at Google</span></a></p><p><span>&#8226; </span><a href="https://www.amazon.com/dp/1449313892?lv=shuf&amp;channelId=500&amp;plpRedirect=mhFallback"><span>Code Simplicity by Max Kanat-Alexander</span></a></p><p><span>&#8226; </span><a href="https://getdx.com/podcast/how-microsoft-sees-engineering-bottlenecks-changing-with-ai/"><span>How Microsoft sees engineering bottlenecks changing with AI</span></a></p>]]></content:encoded></item><item><title><![CDATA[Can “Predictable Delivery” be measured?]]></title><description><![CDATA[Five statistical techniques for turning delivery history into forecasting confidence.]]></description><link>https://newsletter.getdx.com/p/can-predictable-delivery-be-measured</link><guid isPermaLink="false">https://newsletter.getdx.com/p/can-predictable-delivery-be-measured</guid><dc:creator><![CDATA[Brian Houck]]></dc:creator><pubDate>Fri, 21 Aug 2026 10:03:23 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/32476bde-2e45-4b1f-8301-94cff44d0a13_2400x1260.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><em><strong><span>Welcome to the latest issue of Engineering Enablement,</span></strong><span> a weekly newsletter sharing research and perspectives on developer productivity.</span></em></p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.getdx.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe now&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.getdx.com/subscribe?"><span>Subscribe now</span></a></p><div><hr></div><p><span>An engineering leader recently asked me how to measure whether their team is predictable in their software delivery. While organizations collect many engineering metrics (cycle time, throughput, review latency, deployment frequency, etc.), there is rarely a single metric that directly measures predictability.</span></p><p><span>The central question is:</span></p><blockquote><p><strong><span>Can we use the metrics we already collect to quantify delivery predictability?</span></strong></p></blockquote><p><span>Higher delivery predictability enables more accurate release planning, resource allocation, dependency management, and stakeholder communication. Improving predictability is therefore not simply about increasing throughput; it is about increasing confidence in engineering plans.</span></p><h3><span>Key observation</span></h3><p><span>Delivery predictability is not another engineering metric; it is a statistical property of a delivery process.</span></p><p><span>Traditional metrics tell us how much work teams complete or how long work takes. Predictability tells us, given historical performance, how confidently can we forecast future delivery? Predictability is a measure of uncertainty. For example, a team that consistently delivers 20 work items each sprint is highly predictable. A team that alternates between delivering 5 and 35 work items may have the same average throughput, but is far less predictable. The distinction is not average performance. It is variation.</span></p><p><span>A note on two lenses. &#8220;Delivery predictability&#8221; can mean two related but distinct things. The first is flow stability: how consistent and low-variance a team&#8217;s delivery is over time. The second is commitment reliability: how closely actual delivery matches what was committed (a ratio of committed vs. completed work). This article focuses on flow stability, because it can be derived directly from the metrics most organizations already collect.</span></p><h3><span>Example</span></h3><p><span>Let&#8217;s walk through a practical example of two hypothetical teams&#8217; count of completed work items, by sprint:</span></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Ztij!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f7df4f5-980e-4ffc-b5f5-5c0194466eeb_1722x562.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Ztij!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f7df4f5-980e-4ffc-b5f5-5c0194466eeb_1722x562.png 424w, https://substackcdn.com/image/fetch/$s_!Ztij!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f7df4f5-980e-4ffc-b5f5-5c0194466eeb_1722x562.png 848w, https://substackcdn.com/image/fetch/$s_!Ztij!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f7df4f5-980e-4ffc-b5f5-5c0194466eeb_1722x562.png 1272w, https://substackcdn.com/image/fetch/$s_!Ztij!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f7df4f5-980e-4ffc-b5f5-5c0194466eeb_1722x562.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Ztij!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f7df4f5-980e-4ffc-b5f5-5c0194466eeb_1722x562.png" width="1456" height="475" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/8f7df4f5-980e-4ffc-b5f5-5c0194466eeb_1722x562.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:475,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:30221,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://newsletter.getdx.com/i/210658582?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f7df4f5-980e-4ffc-b5f5-5c0194466eeb_1722x562.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!Ztij!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f7df4f5-980e-4ffc-b5f5-5c0194466eeb_1722x562.png 424w, https://substackcdn.com/image/fetch/$s_!Ztij!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f7df4f5-980e-4ffc-b5f5-5c0194466eeb_1722x562.png 848w, https://substackcdn.com/image/fetch/$s_!Ztij!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f7df4f5-980e-4ffc-b5f5-5c0194466eeb_1722x562.png 1272w, https://substackcdn.com/image/fetch/$s_!Ztij!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f7df4f5-980e-4ffc-b5f5-5c0194466eeb_1722x562.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><span>Both teams deliver exactly the same average throughput. The difference is that Team A is predictable while Team B is not. The remainder of this article describes several statistical techniques that quantify this difference.</span></p><h3><span>What different statistical techniques tell us</span></h3><p><span>The following statistical techniques each answer a different question about delivery predictability using the Team A and Team B example.</span></p><h4><span>Standard deviation</span></h4><p><span>Standard deviation measures how much a team&#8217;s delivery typically varies from its average. A low standard deviation means delivery is consistent from sprint to sprint, while a high standard deviation indicates greater volatility.</span></p><p><span>In our example, both teams average 20 completed work items per sprint. However, Team A&#8217;s delivery stays tightly clustered around that average, while Team B swings between 10 and 30. Team B therefore has a much larger standard deviation, despite identical average throughput.</span></p><p><strong><span>Leadership takeaway:</span></strong><em><span> </span></em><span>Standard deviation answers the question, &#8220;How much does this team&#8217;s delivery naturally fluctuate?&#8221; It is often the simplest way to quantify delivery consistency.</span></p><h4><span>Coefficient of variation</span></h4><p><span>Standard deviation alone can be misleading when comparing teams of different sizes. A team delivering 80 &#177; 5 work items is more variable in absolute terms than Team A (20 &#177; 1.3), but proportionally it is just as predictable.</span></p><p><span>The coefficient of variation normalizes the standard deviation by the team&#8217;s average throughput, allowing comparisons across teams regardless of scale.</span></p><p><span>Although Teams A and B have the same average in our example, this measure becomes valuable when comparing organizations with very different delivery volumes.</span></p><p><strong><span>Leadership takeaway:</span></strong><em><span> </span></em><span>Which teams are most predictable relative to the amount of work they deliver?</span></p><h4><span>Percentiles and distribution shape</span></h4><p><span>The sections above use throughput per sprint. Percentiles are most illuminating on a per-item measure such as cycle time, the number of days a work item takes to go from </span><em><span>started</span></em><span> to </span><em><span>done</span></em><span>, so this section shifts to that lens.</span></p><p><span>Averages describe the typical case, but predictability is usually broken by the atypical case. Percentiles, rather than the mean, show how often delivery runs long and by how much.</span></p><p><span>Consider two teams that both have a median (P50) cycle time of 4 days. If the first team&#8217;s 95th percentile (P95) is 7 days while the second team&#8217;s P95 is 21 days, they are not equally predictable. The first team&#8217;s slowest work is only modestly slower than its typical work. The second team&#8217;s slowest work takes three weeks, even though half its items finish in under four days.</span></p><p><span>The gap between P50 and P95 is, in effect, a predictability measure of its own. A tight gap means most work behaves like the typical case. A wide gap means the typical case tells you little about the work that actually puts a commitment at risk.</span></p><p><span>This is why a healthy median can still mask an unpredictable process: it is the tail, not the middle, that causes missed forecasts.</span></p><p><strong><span>Leadership takeaway</span></strong><span>: When we commit to a date, are we planning against the typical case or the realistic worst case? Percentiles let us answer, &#8220;X% of work of this type finishes within N days,&#8221; a far more honest basis for a forecast than an average.</span></p><h4><span>Statistical process control (control charts)</span></h4><p><span>Control charts answer a different question. Rather than measuring overall variability, they determine whether a delivery process has fundamentally changed.</span></p><p><span>Suppose Team A suddenly delivered 9 work items in Sprint 4 instead of its usual ~20. While the average throughput might not change much immediately, a control chart would likely flag Sprint 4 as outside the team&#8217;s normal operating range.</span></p><p><span>This often indicates that something meaningful occurred, such as a production incident, staffing change, organizational restructuring, or dependency bottleneck.</span></p><p><strong><span>Leadership takeaway:</span></strong><span> Did something change that warrants investigation, or is this simply normal variation?</span></p><h4><span>Prediction intervals</span></h4><p><span>Prediction intervals translate historical variability into forward-looking planning confidence. Rather than forecasting that Team A will complete exactly 20 work items next sprint, we estimate the range it is likely to land in.</span></p><p><span>Using a 95% prediction interval, Team A is forecast to deliver roughly 17 to 23 work items next sprint. Team B, with the identical average, is forecast at 0 to 40.</span></p><p><span>That second range deserves a moment. Team B&#8217;s forecast spans from delivering nothing to delivering twice its average. It is technically correct and completely useless for planning, which is precisely the point. Team B does not have a throughput problem. It has a forecasting problem, and no average will ever surface it. Notice that neither team&#8217;s average changed. Only the confidence around that average did.</span></p><p><span>The practical implication is that interval width, maybe more than the forecast itself, is the thing worth tracking over time. A team whose range narrows is becoming more predictable, even if its average never moves.</span></p><p><strong><span>Leadership takeaway</span></strong><span>: How confidently can we forecast future delivery, and how wide is the range we should plan against?</span></p><h3><span>Beyond the spreadsheet: machine learning</span></h3><p><span>Everything above can be done in a spreadsheet. That&#8217;s deliberate. But if you have the data volume and the appetite, traditional machine learning (not LLMs) extends three of these techniques in genuinely useful ways.</span></p><p><strong><span>Conditional forecasting</span></strong><span>. The prediction intervals above are team-level, with one range for everything the team does. Quantile regression forests and conformal prediction produce intervals conditioned on the specific item. Given its size, component, and the current queue depth, this work finishes in 3 to 12 days with 90% confidence. That&#8217;s a better input to a release plan than a team-wide average.</span></p><p><strong><span>Survival analysis </span></strong><span>for work in flight. Cycle time statistics only describe items that have already finished, which quietly biases the picture toward work that went well. Survival models handle still-open items correctly and answer the question planners actually ask: what is the probability this item ships before the release date?</span></p><p><strong><span>Driver attribution</span></strong><span>. Identifying what explains variation is hard, and the place modeling can really earn its keep. A gradient-boosted model with SHAP values can rank which operational signals (review latency, WIP, incident load, dependency count) account for the most variance in delivery. Treat the output as associational and as a hypothesis generator, not as evidence of cause.</span></p><p><span>Two caveats. Many teams don&#8217;t have enough history for any of this. Eight sprints is a worked example, not a training set. And the second caveat is that you need good workflow data hygiene. A model trained on tickets that sat in &#8220;In Review&#8221; for three weeks after they were actually merged will confidently forecast fiction.</span></p><p><span>Start with the spreadsheet.</span></p><h3><span>Recommendation: Avoid a single predictability score</span></h3><p><span>While it may be tempting to distill delivery predictability into a single score, I recommend against doing so. A single number simplifies reporting, but it can also obscure important differences between teams and make it difficult to identify the underlying causes of delivery risk.</span></p><p><span>For example, teams may all receive similar scores while exhibiting fundamentally different delivery characteristics:</span></p><ul><li><p><strong><span>Consistently fast</span></strong><span> &#8211; High throughput with low variability.</span></p></li><li><p><strong><span>Consistently slow</span></strong><span> &#8211; Lower throughput, but reliable and predictable.</span></p></li><li><p><strong><span>Highly variable</span></strong><span> &#8211; Similar average performance, but large sprint-to-sprint fluctuations.</span></p></li><li><p><strong><span>Steadily improving</span></strong><span> &#8211; Predictability changing over time as the team&#8217;s delivery process matures, which a point-in-time score hides entirely.</span></p></li></ul><p><span>Rather than reducing predictability to a single KPI, I recommend reporting three complementary views:</span></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Vz2E!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd679f649-1997-42f8-91ff-9134fb68df08_2400x858.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Vz2E!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd679f649-1997-42f8-91ff-9134fb68df08_2400x858.png 424w, https://substackcdn.com/image/fetch/$s_!Vz2E!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd679f649-1997-42f8-91ff-9134fb68df08_2400x858.png 848w, https://substackcdn.com/image/fetch/$s_!Vz2E!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd679f649-1997-42f8-91ff-9134fb68df08_2400x858.png 1272w, https://substackcdn.com/image/fetch/$s_!Vz2E!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd679f649-1997-42f8-91ff-9134fb68df08_2400x858.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Vz2E!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd679f649-1997-42f8-91ff-9134fb68df08_2400x858.png" width="1456" height="521" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/d679f649-1997-42f8-91ff-9134fb68df08_2400x858.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:521,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:213647,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://newsletter.getdx.com/i/210658582?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd679f649-1997-42f8-91ff-9134fb68df08_2400x858.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!Vz2E!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd679f649-1997-42f8-91ff-9134fb68df08_2400x858.png 424w, https://substackcdn.com/image/fetch/$s_!Vz2E!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd679f649-1997-42f8-91ff-9134fb68df08_2400x858.png 848w, https://substackcdn.com/image/fetch/$s_!Vz2E!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd679f649-1997-42f8-91ff-9134fb68df08_2400x858.png 1272w, https://substackcdn.com/image/fetch/$s_!Vz2E!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd679f649-1997-42f8-91ff-9134fb68df08_2400x858.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><span>This approach preserves the information leaders need for planning while providing engineering teams with actionable insights into where process improvements will have the greatest impact.</span></p><h3><span>Recommendation: Use a multi-step approach</span></h3><p><span>I recommend measuring delivery predictability in four steps:</span></p><ol><li><p><strong><span>Select a delivery outcome</span></strong><span>. Choose a metric that represents delivery performance, such as throughput, completed work items, cycle time, or PR completion time.</span></p></li><li><p><strong><span>Measure historical variability</span></strong><span>. Quantify how consistently that outcome is achieved over time using techniques such as standard deviation, percentiles, and control charts.</span></p></li><li><p><strong><span>Forecast future delivery</span></strong><span>. Use prediction intervals to estimate a range of likely outcomes rather than relying on a single point estimate.</span></p></li><li><p><strong><span>Identify the drivers of variation</span></strong><span>. Use existing engineering metrics, such as review latency, incident load, build reliability, focus time, or team composition, to explain changes in predictability and surface opportunities for improvement.</span></p></li></ol><p><span>The goal is not to eliminate variation, but to understand it well enough to plan around it.</span></p><div><hr></div><p>That&#8217;s it for this week. Thanks for reading.</p><p>-Brian</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.getdx.com/p/can-predictable-delivery-be-measured?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.getdx.com/p/can-predictable-delivery-be-measured?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p>]]></content:encoded></item><item><title><![CDATA[Is there a relationship between cycle time and PR throughput?]]></title><description><![CDATA[There&#8217;s a closer relationship between the two in high-throughput organizations. In low-throughput organizations, there are other variables at play.]]></description><link>https://newsletter.getdx.com/p/is-there-a-relationship-between-cycle</link><guid isPermaLink="false">https://newsletter.getdx.com/p/is-there-a-relationship-between-cycle</guid><dc:creator><![CDATA[Brian Houck]]></dc:creator><pubDate>Wed, 19 Aug 2026 10:01:00 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/b634985d-db98-4738-ae3d-aa1c5eccd060_2400x1260.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><em><strong><span>Welcome to the latest issue of Engineering Enablement, </span></strong><span>a weekly newsletter sharing research and perspectives on developer productivity.</span></em></p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.getdx.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe now&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.getdx.com/subscribe?"><span>Subscribe now</span></a></p><p><span>DX&#8217;s Q2 AI Impact Report is now available with the latest research on AI&#8217;s impact across engineering organizations. </span><a href="https://getdx.com/resources/?utm_source=newsletter">Read the full report.</a></p><div><hr></div><p><span>When teams reduce the time it takes to complete a pull request, it&#8217;s natural to assume they&#8217;ll be able to deliver more work, but this relationship is rarely quantified at scale. A customer recently brought this question to DX&#8217;s Research team, asking us to look into the relationship between PR cycle time and PR throughput.</span></p><blockquote><p><strong><span>Key takeaway: </span></strong><span>High-throughput organizations are highly sensitive to long cycle times, potentially indicating that bottlenecks like review wait times are particularly acute for these orgs. For lower-throughput teams, there is no meaningful relationship between cycle time and throughput, suggesting bottlenecks exist elsewhere.</span></p></blockquote><h3><span>Queuing theory behind cycle time and throughput</span></h3><p><a href="https://en.wikipedia.org/wiki/Little%27s_law"><span>Little&#8217;s Law</span></a><span>, a theorem from queuing theory, helps explain the relationship between three things: how much work is in progress (WIP), how quickly work gets completed, and how long each piece of work takes. For software teams, we can think of these as:</span></p><ul><li><p><span>WIP: the number of pull requests currently open,</span></p></li><li><p><span>Pull Request (PR) Throughput: number of pull requests merged over a period of time, and</span></p></li><li><p><span>PR cycle time: how long a pull request takes to move from development through review and merge.</span></p></li></ul><p><span>The equation is straightforward. When the amount and type of work stay relatively consistent, work that moves through the process faster can help a team complete more work over time. This relationship is commonly used to explain how software teams improve flow but it hasn&#8217;t yet been quantified in the developer productivity space.</span></p><p><a href="https://dl.acm.org/doi/10.1145/3544791"><span>Research by Maddila et al.</span></a><span> shows that pull requests often spend significant amounts of time waiting, not being actively coded or reviewed, and implies that cycle time isn&#8217;t a fixed metric. Teams can recover some of that waiting time by improving the workflow around development and review, without changing the process of writing code.</span></p><p><span>To understand how PR cycle time relates to throughput, we looked at the relationship between median PR cycle time and average PR throughput per developer across organizations using data from 500+ DX customers from January-July 2026, inclusive.</span></p><h3><span>Cycle time matters more for high-throughput organizations than it does low-throughput organizations</span></h3><p><span>Our analysis suggests that PR cycle time is not a reliable standalone predictor of throughput. In the raw data, organizations with similar review speeds show a wide range of throughput levels, indicating that other factors, like PR size and organizational context, likely play an important role in determining output.</span></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!dNMV!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F447a4744-154e-4ccc-b7d5-575359cb2d8c_4200x2654.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!dNMV!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F447a4744-154e-4ccc-b7d5-575359cb2d8c_4200x2654.png 424w, https://substackcdn.com/image/fetch/$s_!dNMV!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F447a4744-154e-4ccc-b7d5-575359cb2d8c_4200x2654.png 848w, https://substackcdn.com/image/fetch/$s_!dNMV!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F447a4744-154e-4ccc-b7d5-575359cb2d8c_4200x2654.png 1272w, https://substackcdn.com/image/fetch/$s_!dNMV!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F447a4744-154e-4ccc-b7d5-575359cb2d8c_4200x2654.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!dNMV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F447a4744-154e-4ccc-b7d5-575359cb2d8c_4200x2654.png" width="1456" height="920" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/447a4744-154e-4ccc-b7d5-575359cb2d8c_4200x2654.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:920,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:1201873,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://newsletter.getdx.com/i/211734214?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F447a4744-154e-4ccc-b7d5-575359cb2d8c_4200x2654.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!dNMV!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F447a4744-154e-4ccc-b7d5-575359cb2d8c_4200x2654.png 424w, https://substackcdn.com/image/fetch/$s_!dNMV!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F447a4744-154e-4ccc-b7d5-575359cb2d8c_4200x2654.png 848w, https://substackcdn.com/image/fetch/$s_!dNMV!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F447a4744-154e-4ccc-b7d5-575359cb2d8c_4200x2654.png 1272w, https://substackcdn.com/image/fetch/$s_!dNMV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F447a4744-154e-4ccc-b7d5-575359cb2d8c_4200x2654.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><span>The median trend is relatively flat across most of the distribution, with throughput beginning to decline more noticeably only at longer cycle times. This indicates that reducing cycle time may not automatically increase throughput for every organization, particularly when teams are already operating within a typical throughput range.</span></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!IacL!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73faa85b-8f8c-4c75-a8b1-4fcf57632255_4200x2884.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!IacL!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73faa85b-8f8c-4c75-a8b1-4fcf57632255_4200x2884.png 424w, https://substackcdn.com/image/fetch/$s_!IacL!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73faa85b-8f8c-4c75-a8b1-4fcf57632255_4200x2884.png 848w, https://substackcdn.com/image/fetch/$s_!IacL!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73faa85b-8f8c-4c75-a8b1-4fcf57632255_4200x2884.png 1272w, https://substackcdn.com/image/fetch/$s_!IacL!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73faa85b-8f8c-4c75-a8b1-4fcf57632255_4200x2884.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!IacL!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73faa85b-8f8c-4c75-a8b1-4fcf57632255_4200x2884.png" width="1456" height="1000" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/73faa85b-8f8c-4c75-a8b1-4fcf57632255_4200x2884.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1000,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:1397532,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://newsletter.getdx.com/i/211734214?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73faa85b-8f8c-4c75-a8b1-4fcf57632255_4200x2884.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!IacL!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73faa85b-8f8c-4c75-a8b1-4fcf57632255_4200x2884.png 424w, https://substackcdn.com/image/fetch/$s_!IacL!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73faa85b-8f8c-4c75-a8b1-4fcf57632255_4200x2884.png 848w, https://substackcdn.com/image/fetch/$s_!IacL!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73faa85b-8f8c-4c75-a8b1-4fcf57632255_4200x2884.png 1272w, https://substackcdn.com/image/fetch/$s_!IacL!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F73faa85b-8f8c-4c75-a8b1-4fcf57632255_4200x2884.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><span>Because median results can obscure differences across the performance distribution, we also used quantile regression to look at this relationship at the 25th, 50th, 75th, and 90th percentiles of throughput. This approach allows us to look at whether the association between cycle time and throughput changes for lower-, median-, and higher-throughput organizations.</span></p><p><span>The results show that the relationship between cycle time and throughput is concentrated among higher-performing organizations.</span></p><ul><li><p><span>At the 25th percentile, there is no statistically significant relationship between cycle time and throughput (p=0.366, &#946;=-1.1).</span></p></li><li><p><span>At the median, there is a modest but statistically significant negative relationship (p=0.0008, &#946;=-4.0).</span></p></li><li><p><span>At the 75th and 90th percentiles, the negative relationship becomes substantially stronger and remains statistically significant (p=0.000, &#946;=-11.7; p=0.000, &#946;=-12.8).</span></p></li></ul><p><span>In practical terms, high-throughput teams are the most sensitive to cycle time slowdowns. For teams already in the bottom 25% of throughput, cycle time barely matters. For teams in the top 10%, it matters quite a bit.</span></p><h2><span>Final thoughts: Speed is a safeguard but not a universal solution</span></h2><p><span>Across most organizations, throughput stays fairly steady as PR cycle time increases. The relationship between the two becomes more noticeable around 17 hours. Beyond that point, median throughput starts to decline. This makes 17 hours a useful inflection point, not a universal target. For teams below that point, reducing cycle time may not lead to a meaningful increase in throughput. But once cycle times go beyond it, longer review cycles may start to limit teams that have the capacity to deliver more.</span></p><p><span>Think of the 17-hour mark as a warning sign and not a productivity cutoff. Teams with longer cycle times may benefit from looking more closely at review queues, reviewer availability, PR size, and other sources of waiting. These results show an association, not proof that longer cycle times directly cause lower throughput. Cycle time is most useful when considered alongside quality, work complexity, team structure, and other developer-experience measures, not as a standalone productivity target.</span></p><div><hr></div><p><span>This week&#8217;s featured DevProd job openings. See more </span><a href="https://getdx.com/resources/devex-jobs/">open roles here</a><span>.</span></p><ul><li><p><strong>Ashby</strong><span> is hiring an </span><a href="https://jobs.ashbyhq.com/Ashby/0f5dbf59-687b-4d88-88a7-73ee0a66b48d?utm_source=PRgMeEgv1Z">Staff Platform Engineer</a><span> | Remote</span></p></li><li><p><strong>Carta</strong><span> is hiring a </span><a href="https://www.linkedin.com/jobs/view/4404135082">Sr. Software Engineer II, Developer Experience</a><span> | Santa Clara, CA; San Francisco, CA; New York, NY</span></p></li><li><p><strong>Figma</strong><span> is hiring a </span><a href="https://job-boards.greenhouse.io/figma/jobs/5790627004?gh_jid=5790627004&amp;gh_src=db0ijm3x4us">Staff Software Engineer, Developer Experience</a><span> | Remote; US</span></p></li><li><p><strong>Mastercard</strong> is hiring a <a href="https://mastercard.wd1.myworkdayjobs.com/CorporateCareers/job/New-York-City-New-York/Vice-President--Software-Engineering_R-286632">Vice President, Software Engineering</a> | New York</p></li><li><p><strong>Notion</strong><span> is hiring a </span><a href="https://jobs.ashbyhq.com/notion/49bdf081-6e20-4323-8c73-6d6b19544ff5">Software Engineer, Developer Experience</a><span> | Hybrid; Hyderabad, India</span></p></li></ul><div><hr></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.getdx.com/p/is-there-a-relationship-between-cycle?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.getdx.com/p/is-there-a-relationship-between-cycle?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p>]]></content:encoded></item></channel></rss>