In this closing panel from DX Annual, Rafe Colburn, Chief Product and Technology Officer at Etsy; Jesse Adametz, Senior Director of Engineering, Platform Engineering at Twilio; Eirini Kalliamvakou, Research Advisor at GitHub; Collin Green, Senior Staff UX Researcher at Google; and Brian Houck, Senior Principal Applied Scientist at Microsoft debate some of the biggest questions surrounding AI and engineering productivity.

They discuss whether AI will reduce the need for engineers, how AI is affecting technical debt, the future role of software engineers in an agentic world, and whether organizations should mandate AI adoption. They also explore how bottlenecks are shifting across the software development lifecycle, the challenges facing junior engineers, and why learning, culture, and change management may ultimately matter more than the tools themselves.

Some takeaways:

AI is changing software engineering, but not eliminating the need for engineers

• The panel largely rejected the idea that an AI-first SDLC means dramatically fewer engineers. As the cost of building software decreases, demand for software is likely to increase, creating new opportunities rather than eliminating the need for technical talent.

• Several panelists argued that the role of software engineers will evolve rather than disappear. The tasks that make up the job may change, but organizations will continue to need people who can solve problems, make decisions, and build systems.

Technical debt remains a tradeoff, not just an AI problem

• Panelists disagreed on whether AI is creating technical debt faster than it can remove it. Some argued that AI is accelerating both code generation and technical debt, while others believed the underlying business pressures that create technical debt remain largely unchanged.

• The discussion also introduced the idea of cognitive debt. As engineers rely more heavily on AI-generated code, understanding and maintaining systems may become more difficult even if development velocity increases.

The future engineer may work at a higher level of abstraction

• Several panelists predicted that engineers will spend less time writing code directly and more time defining intent, setting constraints, providing context, and validating results. Rather than replacing engineering work, AI may shift it to a different level of abstraction.

• The panel also pushed back on the idea that engineers will simply become managers of agents. Effective AI use still requires technical judgment, communication skills, and careful oversight.

Mandates rarely create meaningful AI adoption

• Most panelists opposed the idea that organizations should mandate AI usage. Instead, they emphasized enablement, reducing friction, and helping developers discover value through their own workflows.

• Usage metrics can easily become the wrong goal. The group cautioned against treating AI usage itself as a performance metric, arguing that outcomes matter more than activity.

Junior engineers remain essential to the future of the profession

• The panel strongly rejected the idea that organizations will no longer need junior engineers. Today’s junior engineers become tomorrow’s senior engineers, making talent development critical to the long-term health of the industry.

• Several speakers also noted that newer engineers may bring valuable AI-native perspectives. Just as previous technology shifts rewarded developers who grew up with new tools, the next generation may help shape how AI is used in practice.

The biggest AI adoption challenges are human, not technical

• While tooling matters, the panel repeatedly returned to learning, culture, incentives, and change management as the biggest barriers to successful AI adoption. Engineers are navigating rapid technological change, shifting workflows, and new expectations about their role.

• Organizations that create space for learning appear to see stronger results. The panel highlighted examples where teams learned together, experimented together, and achieved better adoption outcomes than individuals working in isolation.

In this episode, we cover:

(00:00) Intro

(01:16) Why an AI-first SDLC doesn’t mean fewer engineers

(03:09) The debate over AI and technical debt

(07:40) AI-generated code and the future role of engineers

(14:16) Why mandating AI use doesn’t necessarily lead to better outcomes

(20:43) Predictions for the future of junior engineers

(23:22) Where the bottlenecks are in the SDLC now

(28:25) How risk influences AI use

(32:38) Why the human side is the biggest AI adoption challenge

Referenced:

• Etsy

• GitHub

• Microsoft

• Twilio

• Google

• Stewart Reichling

• What is the SPACE framework and when should you use it?

Indeed increased AI coding tool adoption from roughly 25% to 97% across its engineering organization, but getting engineers to use the tools was only part of the challenge.

In this session from DX Annual, Michael Redding, Principal Product Manager, and Jeff Davis, VP of Core Infrastructure at Indeed, explain how the company used structured training, leadership support, and ongoing community engagement to help more than 2,000 engineers build practical AI skills. They share why an early train-the-trainer model fell short, how they redesigned their approach around hands-on learning, and what they learned about balancing adoption, measurement, and psychological safety.

They also discuss the impact of the program on coding time, the role of continuous enablement after formal training ended, and how Indeed is preparing for the next phase of AI adoption, including agentic workflows and AI-powered coaching.

Some takeaways:

Indeed started with a productivity problem, not an AI problem

• At the beginning of 2025, Indeed’s DX survey showed that only about half of developer time was being spent on new features and innovation. The remaining 48% was consumed by maintenance, upgrades, incident response, and other forms of engineering overhead.

• The company’s AI strategy focused on two goals: reducing overhead work and increasing output during coding time. The long-term objective was to double engineering productivity by shrinking non-value-added work while helping engineers produce more during the time they spend building.

AI Coding Essentials succeeded where AI Coding Ambassadors fell short

• Indeed’s first enablement effort, AI Coding Ambassadors, used a train-the-trainer model built around roughly 60 AI champions across the organization. While ambassadors maintained high levels of engagement, adoption among their teammates declined after the program ended.

• The company responded by launching AI Coding Essentials (AICE), a structured training program designed for all engineers. The experience convinced the team that direct, hands-on learning was far more effective than relying on knowledge to spread organically through teams.

Indeed treated AI upskilling as a company-wide investment

• Training more than 2,000 engineers required significant organizational commitment and leadership support. Michael estimated the investment at roughly $3–4 million in engineering time across the company.

• Rather than mandating AI usage, Indeed strongly encouraged completion of the training itself. Managers were given visibility into participation, while engineers retained flexibility in how and whether they ultimately incorporated AI into their workflows.

AI adoption increased from 25% to 97%

• Despite offering AI tools, training resources, and executive support, weekly AI usage remained stuck around 25% at the start of 2025. The challenge was not tool access but helping engineers develop practical skills and confidence.

• By the time of the presentation, weekly AI tool usage had reached approximately 97%. The company also successfully navigated multiple tool transitions, moving from Cody and Copilot to newer agentic tools such as Claude Code, Cursor, Windsurf, and Amp.

Structured training produced measurable results

• Engineers who completed AI Coding Essentials reduced coding time by roughly 35–36%, while engineers who did not complete the training saw little change. Across the broader organization, coding time decreased by roughly 20%.

• Indeed measured coding time as the period between a developer picking up a Jira ticket and opening a diff in GitLab. The company continued to see benefits months after training ended, especially as newer frontier models became available.

Community and continuous enablement kept momentum going

• Indeed reinforced learning through coding forums, office hours, hackathons, Slack communities, and its AI Showcase recognition program. More than 100 unique community posts were being shared monthly in the company’s primary AI channel.

• The goal was to make AI learning continuous rather than event-based. Engineers had multiple ways to share discoveries, get help, and learn from peers long after formal training concluded.

The next challenge is moving beyond coding

• Indeed is now focused on agentic workflows, AI coaching, and expanding enablement beyond software engineering. Product managers, designers, researchers, and other R&D functions are becoming part of the company’s AI adoption strategy.

• As coding becomes faster, bottlenecks are beginning to shift elsewhere in the development lifecycle. The team is already monitoring signs that code review and other downstream activities may become the next constraints on engineering throughput.

In this episode, we cover:

(00:00) Intro

(01:05) Indeed’s DX survey from January 2025

(02:30) The two-part strategy to double engineering productivity

(04:21) How Indeed increased AI adoption from 25% to 97%

(15:40) Results from Indeed’s AI training program

(18:33) How Indeed sustains AI adoption and learning

(23:06) What’s next for AI enablement at Indeed

(24:41) Q&A: How coding time was calculated

(25:25) Q&A: How Indeed uses AI playbooks

(26:40) Q&A: Balancing asynchronous and live AI training

(28:22) Q&A: Psychological safety during AI adoption

(31:44) Q&A: Why AI adoption spikes after the holidays

(33:20) Q&A: The metrics Indeed tracked

(35:22) Q&A: Where the time savings are going

(36:54) Q&A: Reaching engineers who skipped the training

(38:08) Closing thoughts

Referenced:

• Indeed

• Claude Code | Anthropic’s agentic coding system

• Cursor

• Windsurf

• Amp Code

• The Complete Guide to Building Skills for Claude | Anthropic

• Measuring developer productivity with the DX Core 4

In this session from DX Annual, Uma Namasivayam, Senior Director of Engineering Productivity at Dropbox, shares how the company’s developer productivity efforts evolved from improving developer experience to preparing for the agentic era.

He explains how Dropbox approached AI adoption across its engineering organization, the impact it had on developer productivity, and why faster code generation is creating new bottlenecks in areas such as code review, validation, and CI/CD. He also discusses Dropbox’s efforts to rethink engineering systems, measurement, and workflows, including the development of agentic tooling and new metrics designed to move beyond PR throughput and toward product velocity.

Some takeaways:

Dropbox’s productivity journey started before AI

• DXI helped Dropbox identify productivity problems as system problems rather than talent problems. When the company began measuring developer experience in 2023, it found significant variation across teams in DXI scores, PR throughput, and cycle time.

• Measuring developer experience created a framework for prioritizing investments. The team used DXI to identify friction across areas such as debugging, documentation, and build systems while giving leadership a common language for discussing productivity.

AI adoption required more than access to tools

• Dropbox combined executive support, developer segmentation, enablement, and strong guardrails to drive adoption. Different teams and developer roles were matched with different tools and workflows based on their needs.

• The approach helped Dropbox increase AI adoption from roughly 30% to 100% within six months. During the same period, PR throughput doubled and developer satisfaction with AI tools increased significantly.

Engineers used their extra capacity to tackle neglected work

• As AI increased throughput, engineers naturally pulled maintenance work, migrations, and technical debt from the backlog. Dropbox saw significant growth in these categories without any specific direction from leadership.

• The additional capacity was often reinvested into engineering health. Teams used the opportunity to address long-standing issues that had accumulated over time rather than focusing exclusively on new feature development.

The next challenges are scale, trust, and measurement

• Dropbox believes the move to agentic engineering creates three major challenges: scale, validation and trust, and measurement. Existing development systems were not designed for a world where AI dramatically increases code throughput.

• As code generation accelerates, bottlenecks are shifting toward code review, validation, and CI/CD systems. The company is already seeing pressure move downstream in the software development lifecycle.

Agentic engineering requires redesigning the entire system

• Uma compared the transition to the shift from steam-powered factories to electric factories. The biggest gains came from redesigning the entire system rather than simply replacing one technology with another.

• Dropbox is investing in agentic workflows across the SDLC and building Nova as an orchestration layer. The company is evaluating roughly 30 development steps, and one in twelve pull requests is already being generated by Nova.

PR throughput is becoming a less useful measure of productivity

• Dropbox believes traditional engineering metrics need to evolve alongside AI. As agentic workflows become more common, measuring productivity through pull request volume alone provides an incomplete picture of engineering output.

• The company is increasingly focused on metrics such as AI contribution, loaded cost per PR, agentic workflow coverage, work distribution, and time to ship. The goal is to better connect engineering activity to customer value and business outcomes.

In this episode, we cover:

(00:00) Intro

(00:57) The beginning of Dropbox’s DX journey

(02:34) AI adoption at Dropbox: what made it work

(04:46) The results of Dropbox’s AI adoption efforts

(05:39) What the results mean for the business

(06:55) The phases of AI adoption and where they are now

(08:00) The new bottlenecks

(09:16) Three challenges Dropbox faces moving into agentic engineering

(10:05) How Dropbox is redesigning the SDLC for agentic engineering

(15:46) The new metrics that matter

(19:16) Final takeaways

Referenced:

• Dropbox

• Developer Experience Index (DXI) | DX

• DX Core 4 Productivity Framework

• Cursor

• Claude Code | Anthropic’s agentic coding system

• JetBrains

• Visual Studio Code

• Jira | Project Management for the AI Era | Atlassian

• GitHub

In this session from DX Annual, Christopher Sanson, Product Lead, AI Developer Experience, and Madison Capps, Engineering Manager, Infrastructure at Airbnb, challenge some of the most common assumptions about AI. Is AI primarily about replacing humans? Do organizations need mandates to drive adoption? And are the productivity gains really as small as some studies suggest?

Using examples from Airbnb’s own AI journey, they share how the company achieved widespread adoption of agentic AI through AirChat, community enablement, and internal tooling rather than top-down mandates. They also discuss the impact AI is having on developer productivity, how non-developers are increasingly using coding tools, and how teams are rethinking product development in an AI-first world.

Finally, Madison takes a deeper look at the infrastructure powering Airbnb’s AI strategy, including AirChat CLI, the AirChat SDK, and AirChat Remote, along with the company’s vision for asynchronous agent workflows and the next generation of AI-powered development.

Some takeaways:

AI adoption at scale

• Successful AI adoption does not require mandates. Airbnb achieved 97% weekly usage and 90% daily usage of agentic AI tools among engineers without tying adoption to performance reviews or quotas. Christopher argued that the best adoption comes when developers choose to use AI because it genuinely helps them work faster and better.

• Treat internal AI tools like products, not internal infrastructure. Airbnb built a recognizable brand around AirChat, invested in onboarding and workshops, created internal marketing materials, and focused heavily on user experience. That product mindset helped turn AirChat into a company-wide platform rather than just another engineering tool.

• Community-driven learning scales better than centralized training. AI champions, train-the-trainer programs, hackathons, workshops, and active peer-to-peer learning channels allowed knowledge to spread organically across the company. Over time, the AI community became larger and more active than the team managing the platform itself.

Productivity gains are accelerating

• Developers are spending more time actively coding. Christopher challenged the idea that engineers only spend a small percentage of their time writing code. As coding becomes faster and easier with agentic AI, developers can spend more of their week building software rather than working around implementation bottlenecks.

• The most active AI users see the largest productivity gains. Airbnb found that developers who spent four or more hours per day working with agentic AI dramatically increased their output. The relationship between AI usage and productivity became stronger as engineers learned how to incorporate agents into their daily workflows.

• PR throughput increased by 65% after the introduction of agentic AI. Airbnb’s data suggests that productivity gains extend well beyond the single-digit improvements often cited in industry studies. Developers who heavily embraced agentic AI moved from industry-average output to some of the highest throughput levels measured internally.

• AI-authored code is becoming mainstream. Roughly 59% of Airbnb’s code is now primarily authored by AI, and more than half of developers report that AI generates the majority of the code they work with. Christopher argued that this shift is happening far faster than most organizations realize.

AI is spreading beyond engineering

• The addressable market for AI is much larger than developers alone. Airbnb initially expected adoption to level off around its engineering population. Instead, usage continued growing as product managers, designers, finance teams, and operations teams began integrating agentic AI into their work.

• People will learn new workflows when the value is obvious. Some non-engineering teams adopted VS Code and terminal-based tools simply because they provided the best access to agentic AI capabilities. Rather than resisting technical tools, employees were willing to learn them in exchange for meaningful productivity gains.

• Domain experts are increasingly building their own AI-powered solutions. Airbnb’s internal platforms allow teams to create specialized applications tailored to their own workflows. This shifts more problem-solving into the hands of the people closest to the business problem.

Rethinking how work gets done

• Many existing processes were designed around expensive software development. Product reviews, lengthy requirements documents, and sequential handoffs evolved in a world where implementation was slow and costly. AI changes those economics and creates opportunities to redesign workflows from first principles.

• AI enables faster movement from ideas to prototypes. Rather than spending weeks refining specifications before building anything, teams can generate multiple prototypes quickly, test ideas earlier, and iterate before committing significant resources.

• Smaller teams can collaborate earlier and move faster. Airbnb sees opportunities to reduce handoffs between product managers, designers, and engineers by bringing teams together earlier in the process and using AI to accelerate exploration and execution.

Building for asynchronous AI

• Current agentic AI tooling still creates friction. Managing multiple sessions, handling long-running tasks, maintaining context, and switching between workflows remain cumbersome despite major advances in model capabilities.

• The next frontier is asynchronous agent workflows. Rather than interacting with a single agent in real time, developers are increasingly orchestrating multiple agents working in parallel, often across long-running tasks that continue without constant supervision.

• Airbnb is investing in infrastructure, not just models. AirChat CLI, migration tooling, the AirChat SDK, and AirChat Remote were all built around the belief that future gains will come from workflow orchestration, platform capabilities, and developer experience as much as from improvements in foundation models.

Preparing for an AI-first future

• Organizations should build for where developer workflows are heading. Madison described Airbnb’s approach as continuously forecasting how engineers are likely to work in the near future and investing in the infrastructure required to support those workflows before they become mainstream.

• AI-first architecture will become increasingly important. As throughput rises and more work is delegated to agents, teams will need stronger guardrails, scalable platforms, and systems designed specifically to support AI-assisted development.

• The biggest bottlenecks are shifting away from code generation. As AI reduces implementation costs, constraints move elsewhere in the system. Coordination, validation, infrastructure, and workflow management are becoming the new challenges organizations must solve.

In this episode, we cover:

(00:00) Intro

(01:37) Myth #1: AI is about replacing humans

(03:22) Myth #2: You need mandates to drive AI adoption

(05:21) AirChat, agentic AI, and Airbnb’s adoption strategy

(08:07) Myth #3: AI has little impact on productivity

(09:33) Airbnb’s increase in coding time and PR throughput

(14:20) Myth #4: AI coding tools are just for coders

(15:39) How non-developers are using coding tools

(17:24) Rethinking product development in an AI-first world

(20:30) Myth #5: Vibe coding isn’t coding

(22:16) Unsolved problems in agentic AI tooling and how Airbnb is addressing them

(26:30) Airbnb’s overall AI philosophy in practice

(29:15) Using agentic AI to accelerate code migrations

(30:18) AirChat SDK: How Airbnb enables teams to build AI-powered applications

(33:17) AirChat Remote and asynchronous agent workflows

(36:07) Predictions for what’s next

Where to find Christopher Sanson:

• LinkedIn: https://www.linkedin.com/in/christophersanson

Where to find Madison Capps:

• LinkedIn: https://www.linkedin.com/in/madison-capps-66950625

Referenced:

• Airbnb

• Steve Jobs’s Bicycles for the Mind

• Jennifer St Pierre

• Justin Reock

• AI-generated merged code holds steady at ~30%

• Andrej Karpathy’s post on X

In this session from DX Annual, Rebecca Fitzhugh, Lead Principal Engineer at Atlassian, moderates a panel featuring Nidhi Allipuram, Vice President, Enterprise Developer Experience and Platform at Nationwide, Jai Schniepp, Senior Director, DevX Product Management at Comcast, Brent Foster, Vice President and Head of Architecture and Strategy at TD Bank, and Praveena Patchipulusu, Vice President of Engineering at HPE.

Together, they discuss how large enterprises are approaching AI adoption, what it takes to build an AI-first software development lifecycle, and how engineering leaders are balancing speed, security, governance, and developer experience. They also share their perspectives on the changing role of engineers, human accountability, and how organizations can prepare for the future of software engineering.

Some takeaways:

Building an AI-first software development lifecycle

• AI adoption is becoming a redesign effort, not a tooling effort. Several panelists argued that the biggest opportunity is not simply adding AI assistants to existing workflows but rethinking the software development lifecycle itself. Rather than treating AI as a coding tool, organizations are beginning to integrate it into requirements gathering, design, testing, code reviews, and deployment.

• Training and organizational support matter more than tool selection. Nationwide found that productivity gains came less from introducing new tools and more from providing engineers with training, coaching, playbooks, and time to learn. Teams consistently reported that air cover, psychological safety, and opportunities to experiment were more valuable than access to additional AI products.

• Successful adoption requires systems, not mandates. Organizations cannot simply tell teams to “go use AI.” Several panelists described building AI champion programs, governance models, embedded coaching, and structured learning opportunities that help teams develop new habits and scale adoption across large enterprises.

Keeping humans accountable

• Humans remain responsible for outcomes regardless of who writes the code. Every panelist emphasized that accountability does not shift to AI. Whether code is generated by an engineer, a copilot, or an agent, humans remain responsible for validating outputs, making decisions, and owning the results delivered to customers.

• Validation is becoming more important than approval. Traditional approval processes may matter less than ensuring the right people validate assumptions, outcomes, and risks. Teams are increasingly focused on creating workflows where humans review and challenge AI-generated work rather than simply acting as signoff gates.

• Decision-making is becoming a core engineering skill. As AI takes over more implementation work, engineers are spending more time evaluating tradeoffs, validating outputs, and making judgment calls. The ability to make good decisions quickly may become a larger differentiator than the ability to manually write code.

Security and governance in an AI-powered world

• Shift-left practices become even more important with AI. Security, compliance, and quality checks are being pushed earlier into the development process. Rather than relying on reviews at the end of the pipeline, organizations are embedding guardrails directly into workflows and development platforms.

• AI-generated infrastructure introduces new challenges. The conversation extended beyond application code to infrastructure. As AI increasingly generates Terraform, YAML, and cloud configuration files, organizations must build policy-driven validation and security controls to prevent vulnerabilities from entering production environments.

• Context is both a powerful asset and a potential risk. One of AI’s greatest strengths is its ability to use organizational knowledge and historical context. At the same time, exposing that information to AI systems creates new security concerns, making governance and access controls increasingly important.

The changing role of the engineer

• Engineers are becoming orchestrators rather than implementers. As AI takes over more boilerplate work, engineers are expected to focus more on system design, architecture, critical thinking, and coordinating work across humans, agents, and platforms. Success increasingly depends on defining intent and evaluating outcomes rather than writing every line of code manually.

• Role boundaries are becoming less rigid. The panel described a future where engineers, product managers, designers, and other builders work more closely together. AI is making it easier for individuals to contribute across traditional functional boundaries, creating smaller teams with broader responsibilities.

• Critical thinking and creativity become more valuable. While AI can accelerate execution, it cannot replace human judgment and problem framing. Several panelists argued that creativity, curiosity, and the ability to think differently about problems will become increasingly important as AI capabilities continue to improve.

Rethinking developer experience

• Developer experience is becoming workflow experience. The focus is shifting from individual tools toward creating trusted workflows that help teams move from idea to production more quickly. Organizations are increasingly measuring success by how effectively teams can deliver outcomes rather than by how efficiently they write code.

• Developer experience now includes agent experience. As AI agents become active participants in software delivery, organizations must consider how agents consume context, operate within guardrails, and interact with development platforms. Designing effective systems now means thinking about both human and AI users.

• Breaking down silos creates better outcomes. Several panelists argued that AI provides an opportunity to reduce friction between product managers, designers, developers, and security teams. The organizations that benefit most may be those that remove barriers between disciplines and enable more collaborative ways of working.

Preparing for the future

• The time to experiment is now. Every panelist encouraged organizations to begin learning through direct experience rather than waiting for the technology to mature. Teams that develop AI skills, workflows, and governance practices today will be better positioned as the technology continues to evolve.

• Institutional knowledge may become a competitive advantage. Large enterprises possess decades of documentation, decisions, diagrams, and expertise that often remain difficult to access. Several speakers highlighted the opportunity to unlock that knowledge and make it useful through AI-powered systems.

• Fundamentals still matter. Despite rapid technological change, the panel repeatedly returned to the same conclusion: strong engineering fundamentals, sound judgment, accountability, security practices, and critical thinking remain essential regardless of how much AI enters the software development process.

In this episode, we cover:

(00:00) Intro

(02:28) The AI journey across TD Bank, Comcast, and HPE

(05:59) Inside Nationwide’s AI-assisted development lifecycle

(10:04) Reimagining the software development lifecycle with AI

(11:32) Security, governance, and human accountability

(15:27) Embedding security and guardrails into AI workflows

(17:55) How AI is changing the role of an engineer

(21:52) What developer experience looks like in the AI era

(26:55) What software engineering may look like in 2030

(32:47) How to prepare for the AI-driven future

Where to find Rebecca Fitzhugh:

• LinkedIn: https://www.linkedin.com/in/rmfitzhugh

• X: https://x.com/RebeccaFitzhugh

Where to find Jai Schniepp:

• LinkedIn: https://www.linkedin.com/in/jessicaschniepp

Where to find Nidhi Allipuram:

• LinkedIn: https://www.linkedin.com/in/nidhi-allipuram

Where to find Brent Foster:

• LinkedIn: https://www.linkedin.com/in/engineeringthefuture

• Website: https://brentfoster.me

Where to find Praveena Patchipulusu:

• LinkedIn: https://www.linkedin.com/in/praveena-patchipulusu-158741

Referenced:

• Atlassian

• TD Bank

• Comcast Corporation

• Hewlett Packard Enterprise (HPE)

• Nationwide

• GitHub Spec Kit

• Abi Noda

As AI becomes embedded in software development, many of the metrics that engineering organizations have relied on for years are starting to break down.

In this session from DX Annual, Uber’s Ty Smith and Abhishek Tibrewal share how their approach to measuring AI’s impact on developer productivity has evolved over time. They walk through the different phases of their measurement journey, from adoption and engagement to measuring impact, ROI, and agentic value, explaining what they chose to measure at each stage, what worked, what failed, and how their thinking changed along the way.

They also discuss the role of qualitative feedback before telemetry existed, the challenge of identifying meaningful engagement signals, why “developer years saved” failed as an ROI metric, and how AI agents forced them to rethink traditional productivity measurements. Finally, they introduce Uber’s emerging framework built around feature velocity and explore the unanswered questions that remain as software development becomes increasingly agent-driven.

Some takeaways:

Why AI breaks traditional productivity metrics

• Many measurement frameworks were built for a world where humans wrote most of the code. As AI agents become more capable, metrics that once provided useful signals can quickly become misleading.

• Teams should expect their metrics to break. Uber’s measurement journey required repeatedly revisiting assumptions as AI-assisted development evolved into agentic workflows.

Start with stakeholder questions

• The best metrics answer real business questions. Uber worked backward from questions about productivity, ROI, investment priorities, and business value instead of collecting data for its own sake.

• Measurement should support decision-making. Metrics influence budgets, tooling investments, enablement efforts, and long-term strategy.

• Use qualitative signals before telemetry exists

• Qualitative feedback can be the fastest path to insight. Before AI tooling generated reliable telemetry, Uber relied on surveys, interviews, and experience sampling to understand adoption and guide investments.

• Behavioral questions are more useful than perception questions. Asking developers what they actually did produced stronger signals than asking whether they found AI helpful.

• Measure engagement through behavior, not demographics

• Behavioral patterns revealed insights that demographics could not. Role, tenure, and organization offered limited signal compared to how engineers actually used AI tools.

• A small group of AI power users emerged early. Studying usage patterns helped Uber identify engineers who were engaging deeply with AI and generating outsized results.

Correlation is not causation

• High AI usage does not automatically prove AI caused higher productivity. The most productive engineers are often the first to adopt new tools.

• Rigorous analysis matters when making investment decisions. Uber used causal methods to better understand the true impact of AI-assisted development.

Why measuring AI ROI is difficult

• Developer years saved sounded compelling but failed as an ROI metric. The approach created anxiety around replacement, required constant recalibration, and did not answer the business questions leadership cared about most.

• Business leaders ultimately care about outcomes. Time saved is useful context, but value creation, customer impact, and business results matter more.

• PRs measure activity, features measure value

• Agentic AI exposes the limitations of activity-based metrics. A single agent task can generate many pull requests without creating meaningful customer value.

• Feature velocity became Uber’s new North Star. The goal shifted from measuring engineering output to measuring whether valuable capabilities were actually being delivered.

Building an AI-native measurement framework

• Feature velocity works alongside supporting metrics. Flow efficiency, quality, and capability expansion help create a more complete picture of AI’s impact.

• PR classification provides important context. Understanding the type and complexity of work helps distinguish meaningful progress from routine maintenance and toil.

• The future belongs to outcome-based metrics

• The most durable metrics are tied to business outcomes rather than engineering activity. As AI becomes more autonomous, output alone becomes a less reliable signal.

• Many important questions remain unanswered. Organizations still need better ways to measure judgment, autonomy, technical debt, and the value created by increasingly agent-driven software development.

In this episode, we cover:

(00:00) Intro

(01:30) Steve Yegge’s 8 stages of AI-assisted development

(03:22) Uber’s shift to a generative AI-powered company

(04:20) Uber’s pre-AI productivity metrics

(06:55) Important questions from stakeholders that previous metrics didn’t answer

(08:25) How Uber measures AI before telemetry exists

(11:11) Metrics used to measure adoption

(12:49) Measuring engagement

(14:30) Measuring impact

(16:32) The challenge of measuring AI ROI

(19:32) Rethinking adoption, engagement, and impact for agentic AI

(26:01) The new north star: Feature velocity

(28:41) PR classification + feature velocity: the questions it can answer

(33:01) What comes next and what’s still unanswered

(34:30) Lessons learned and what they’d do differently

(37:11) Q&A #1: How Uber defines a feature

(38:50) Q&A #2: Measuring success and AI ROI

Where to find Abhishek Tibrewal

• LinkedIn: https://www.linkedin.com/in/aabhishektibrewal

Where to find Ty Smith:

• LinkedIn: https://www.linkedin.com/in/tyvsmith

Referenced:

• Welcome to Gas Town

• Dara Khosrowshahi (Uber CEO)

Michael Galloway leads Platform Engineering at Mercari, while Snehal Shinde leads Cost and Performance Engineering. Together, they have been at the center of Mercari’s effort to become an AI-native company.

In this session from DX Annual, Michael and Snehal share what happened after Mercari’s CEO mandated 100% AI adoption across the organization. While AI accelerated code generation and increased engineering output, the team quickly discovered that their existing dashboards could not answer a simple question: was AI actually improving productivity?

They discuss how Mercari built new visibility into AI usage and software delivery, the bottlenecks they uncovered across the SDLC, why faster coding did not automatically translate into faster delivery, and the lessons they learned rolling out AI at scale. They also share how Mercari is rethinking software development around agents, feedback loops, and new ways of working.

Some takeaways:

Measuring AI impact

• AI adoption alone does not guarantee business value. Mercari found that while AI usage increased rapidly across the organization, existing dashboards could not answer the leadership team’s most important question: whether AI was actually improving productivity.

• Local optimization does not necessarily improve system-wide performance. Engineers reported working faster with AI tools, but end-to-end delivery metrics remained largely unchanged because bottlenecks elsewhere in the software delivery process continued to slow teams down.

• Organizations need visibility into both AI usage and delivery outcomes. Mercari built new dashboards that combined AI tool data with SDLC metrics to better understand adoption, throughput, quality, and operational performance.

The reality of becoming AI-Native

• AI adoption required a cultural transformation, not just a tooling rollout. Mercari’s CEO mandated company-wide AI adoption, but success depended on changing workflows, habits, and expectations across engineering, finance, legal, customer support, and other functions.

• Different teams required different forms of enablement. Employees varied significantly in their technical backgrounds and familiarity with AI tools, making education, workshops, and support systems essential to driving adoption.

• The goal was to rethink work itself. Rather than layering AI onto existing processes, Mercari challenged teams to reconsider what they built, how they built it, and how people worked together.

The bottlenecks AI exposed

• AI revealed problems that already existed inside the organization. Review queues, CI instability, deployment friction, and support requests became more visible as coding accelerated.

• Code generation was not the primary constraint. Engineers often spent more time waiting for approvals, navigating organizational boundaries, and dealing with infrastructure limitations than writing code.

• System complexity amplified AI-related challenges. As AI-generated changes increased, existing architectural complexity and fragile workflows became harder to ignore.

Finding AI workflow opportunities

• Mercari mapped AI opportunities across 33 domains. The AI task force reviewed functional areas across the company to identify where AI could automate work, where it could not, and where the strongest leverage points existed.

• The biggest opportunities extended far beyond engineering. Role-specific workshops helped teams in finance, legal, design, operations, customer support, and other departments find practical AI use cases in their own workflows.

• Early wins created proof points for broader adoption. Mercari saw measurable impact from support bots, accounting workflows, platform support automation, and Socrates, an internal BI agent that made company data easier to query and use.

Rethinking software development

• Faster coding shifted attention upstream. As implementation became easier, planning, specification, and decision-making emerged as larger constraints on delivery speed.

• Agent Spec-Driven Development moves AI earlier in the lifecycle. Mercari began using agents to analyze documentation, code, and organizational knowledge before implementation work started.

• Future workflows will focus more on intent than execution. Teams increasingly define goals, constraints, and success criteria while agents handle larger portions of implementation and validation.

Preparing for an agent-driven future

• Feedback loops matter more than ever. Mercari’s multi-loop SDLC emphasizes rapid validation, iterative learning, and increasingly autonomous agent workflows.

• Behavioral change remains harder than technological change. Organizations must rethink ownership, accountability, and trust before they can fully benefit from agent-based development.

• The path to AI-native development is iterative. Mercari expects continued setbacks and learning cycles, applying what they describe as the Stockdale Paradox: maintaining confidence in the destination while remaining honest about current challenges.

In this episode, we cover:

(00:00) Intro

(01:46) Mercari’s scale and engineering culture

(02:51) DX awards at Mercari

(03:44) Mercari’s push to become AI-native

(06:34) The mandate to rethink everything

(08:02) Mercari’s AI visibility problem and how they solved it

(11:30) Mercari’s early findings on AI implementation

(18:47) Closing the AI awareness gap at Mercari

(21:11) Mapping AI opportunities across Mercari

(31:32) Unpacking the results from the second rollout

(34:14) Agent spec-driven development and what’s next

(37:37) A multi-loop SDLC

(40:50) Some hard lessons

(42:55) Closing thoughts

Where to find Michael Galloway:

• LinkedIn: https://www.linkedin.com/in/michaelroygalloway

• X: https://x.com/michaelgalloway

Where to find Snehal Shinde:

• LinkedIn: https://www.linkedin.com/in/snehal-shinde

Referenced:

• Mercari

• Cursor

• Devin

• Claude Code | Anthropic’s agentic coding system

• GitHub

• Datadog

• Tim Bozarth - Microsoft | LinkedIn

• Airbnb

• Jim Collins - Concepts - The Stockdale Paradox

Kelly Anne Pipe is Head of Developer Experience at Vanguard, and Nicole Scribner is a Director in the firm’s Chief Technology Office focused on engineering enablement and advancement.

In this session from DX Annual, Kelly Anne and Nicole share how Vanguard is expanding its AI strategy beyond software engineering to the entire product development lifecycle. While the company initially focused on tools like GitHub Copilot for engineers, they found that faster coding alone did not significantly improve delivery speed. Product managers, designers, QA teams, and organizational processes were still operating at a different pace.

To address this challenge, Vanguard developed a product team maturity model built around three stages: Augmented, Accelerated, and Autonomized. The framework spans six dimensions, from AI-powered delivery and AI-ready codebases to team autonomy, operations, and responsible AI.

Kelly Anne and Nicole explain how Vanguard is applying the model across more than 800 product teams, the behaviors they believe will enable faster delivery, and the lessons they have learned about measurement, organizational change, dependencies, and scaling AI across the product development lifecycle.

Some takeaways:

Beyond the engineering bubble

• Faster coding does not automatically lead to faster delivery. Vanguard found that while engineers using AI tools reported significant productivity gains, product managers, designers, QA teams, and governance processes were still operating at traditional speeds.

• AI adoption becomes fragmented when it is treated as an engineering initiative. Organizations that focus solely on developer tooling risk creating an “engineering bubble” where one part of the product team accelerates while the rest of the workflow remains unchanged.

• The goal is to optimize the entire product development lifecycle. Vanguard shifted its focus from helping engineers code faster to helping cross-functional product teams move faster from idea to production.

The AI maturity model

• Vanguard built a maturity model around three stages: Augmented, Accelerated, and Autonomized. The framework gives more than 800 product teams a shared language for discussing AI adoption and long-term transformation.

• The model spans six dimensions of AI maturity. These include AI-powered product delivery, AI-ready codebases, agent-powered workflows, AI-augmented operations, team autonomy and enablement, and responsible AI.

• The goal is organizational transformation, not tool adoption. The framework focuses on how entire product teams evolve as AI becomes embedded throughout the product development lifecycle.

Building AI-ready foundations

• AI readiness starts with the fundamentals. Documentation, testing, CI/CD pipelines, architecture decisions, and code quality all become more important when agents are introduced into the development process.

• The codebase becomes the interface between teams and AI agents. Poor documentation, weak test coverage, and slow feedback loops limit the effectiveness of even the most capable AI tools.

• Dependencies become more visible at agent speed. Processes that were merely frustrating for humans become major bottlenecks when AI can complete implementation work in hours rather than days.

Scaling AI beyond engineering

• Every role on the product team needs AI-specific workflows. Vanguard is focused on helping product managers, designers, QA teams, and engineers incorporate AI into their daily work rather than limiting adoption to developers.

• The most valuable opportunities often begin before coding starts. AI can help transform customer conversations, discovery work, requirements, and design artifacts into implementation-ready inputs.

• Agent orchestration changes the role of the human. As agents take on more routine execution work, people increasingly act as orchestrators, reviewers, and strategic decision-makers.

The challenges of adoption and measurement

• Behavior change is harder than deploying tools. Vanguard found that fear, uncertainty, and questions about job security often create bigger barriers to adoption than technology itself.

• Simple productivity metrics can be misleading. Measures such as lines of code generated or time saved per developer do not capture whether AI is creating meaningful business value.

• Organizations need layered measurement strategies. Adoption metrics, process improvements, cycle time, quality, and customer outcomes all need to be considered together to understand AI’s true impact.

Lessons from the AI transition

• Agent speed exposes organizational debt. Slow approvals, review queues, onboarding processes, and governance workflows become much more obvious when implementation work accelerates.

• Responsible AI can accelerate delivery rather than slow it down. Investing in guardrails, governance, security, and automated controls early enables teams to move faster with greater confidence.

• The biggest opportunity is organizational transformation. Vanguard believes the future belongs to companies that redesign entire product teams around AI rather than simply adding AI tools to existing workflows.

In this episode, we cover:

(00:00) Intro

(02:16) The state of AI one year ago at Vanguard

(02:54) The engineering bubble

(05:05) Building an AI maturity model for 800 product teams

(08:24) Dimension 1: AI-powered product delivery

(10:00) Dimension 2: AI-ready codebase

(12:20) Dimension 3: Autonomous agent utilization

(13:00) Dimension 4: AI-augmented operations

(14:00) Dimension 5: Team autonomy and enablement

(16:11) Dimension 6: Responsible AI

(18:15) The people problem: role evolution

(20:00) The measurement problem

(22:55) Lessons learned from rolling out the maturity model

(26:46) What’s ahead

(30:10) Q&A #1: Getting your codebase ready for AI

(32:22) Q&A #2: Audit trails and responsible AI

(34:16) Q&A #3: Vanguard’s maturity model progress

(36:15) Q&A #4: Measuring cycle time across 800 teams

Where to find Nicole Scribner:

• LinkedIn: https://www.linkedin.com/in/nicole-scribner-35b80422a

Where to find Kelly Anne Pipe:

• LinkedIn: https://www.linkedin.com/in/kellyannepipe

Referenced:

• Vanguard

• Jennifer St Pierre - Dell Technologies | LinkedIn

• Mercari

In this session from DX Annual, Eleanor Millman, Senior Staff Product Manager, and Mina Tawadrous, Associate Director of Product Management at SiriusXM, share how their platform engineering organization developed a prioritization framework for platform engineering teams serving hundreds of developers across a complex cloud platform.

They explain how they define and weight platform-specific impact factors, use developer data to refine priorities, and score projects more consistently. They also explore why prioritization debates often stem from conflicting, invisible, or outdated assumptions, and how SiriusXM began treating assumptions like code by documenting, versioning, and reviewing them in source control.

Finally, they demonstrate how AI can surface assumptions, connect initiatives to existing knowledge, and support project scoring while keeping humans in the loop. Throughout the session, they offer a practical framework for making prioritization decisions more transparent, data-driven, and scalable.

Some takeaways:

Building a platform engineering prioritization framework

• Platform engineering requires different prioritization criteria. SiriusXM found that traditional product metrics did not fully capture the value of platform engineering work, leading the team to define platform-specific impact factors around development speed, reliability, security, cost, platform efficiency, user trust, and data-driven decision making.

• A simple scoring model created a shared language for prioritization. The framework combined impact, urgency, effort, and business needs to help teams compare projects consistently and explain why certain initiatives were prioritized over others.

• The framework evolved alongside the organization. As company priorities changed after a major platform launch, SiriusXM adjusted impact factor weights to reflect new goals around cost optimization, technical debt reduction, and data maturity.

Using developer data to guide decisions

• Developer feedback helped shape prioritization. Rather than relying solely on intuition, the team used survey data and other developer insights to determine where additional investment would have the greatest impact.

• Impact factor weights were revisited regularly. Quarterly reviews allowed the team to adjust priorities based on changing business objectives and improvements in areas such as reliability and security.

• Data increased confidence in prioritization decisions. By grounding discussions in evidence, teams were able to align more effectively on where to invest their limited capacity.

Treating assumptions like code

• Many prioritization conflicts stem from assumptions rather than priorities. Teams often disagreed because they were working from different, invisible, or outdated assumptions about users, workflows, and business needs.

• Documenting assumptions improved organizational alignment. SiriusXM began storing assumptions in source control, making them easier to discover, review, update, and validate over time.

• Debates became more productive when assumptions were explicit. Instead of arguing over which project mattered most, teams focused on validating the underlying beliefs that informed their decisions.

Using AI to surface organizational knowledge

• Assumption repositories became difficult to navigate at scale. As more assumptions were documented, it became increasingly difficult for individuals to find relevant context and connections across projects.

• AI helped uncover relationships humans might miss. By searching assumption repositories, OKRs, and prior project data, AI was able to surface relevant information that would otherwise be difficult to discover.

• AI improved information recall rather than replacing judgment. The goal was not automated decision making but helping teams access the knowledge needed to make better decisions.

Building an AI-assisted prioritization workflow

• AI can guide teams through the scoring process. SiriusXM built workflows that ask clarifying questions, surface assumptions, identify relevant organizational context, and generate initial project scores.

• Human validation remains essential. Teams review assumptions, challenge recommendations, and approve updates before information is added back into the system.

• Each prioritization cycle strengthens the knowledge base. New assumptions, decisions, and project context become available for future initiatives, making the system more valuable over time.

Keeping humans in the loop

• The framework is designed to support conversations, not replace them. Scores help teams discuss priorities more objectively, but important decisions still require context and judgment.

• Stakeholder disagreements often reveal useful information. When the framework produces results that feel wrong, the discussion can uncover missing assumptions, incomplete data, or opportunities to improve the model itself.

• The framework continues to evolve. SiriusXM treats both the prioritization model and the supporting AI tools as products that require ongoing iteration, feedback, and refinement.

In this episode, we cover:

(00:00) Intro

(02:58) Building a platform engineering prioritization framework

(04:59) The seven platform engineering impact factors

(09:38) Using impact factors to score projects

(13:11) Using developer data to refine priorities

(16:33) Three ways assumptions fail

(17:40) Assumptions as code

(21:00) New problems created by assumptions as code

(22:00) Using AI to surface assumptions

(23:44) Building an AI-powered feedback loop

(25:44) Inside the AI prioritization tool

(28:18) Three steps to build your own framework

(30:02) Q&A #1: Evaluating high-cost projects

(31:30) Q&A #2: The cadence of iteration

(32:10) Q&A #3: When the framework conflicts with a stakeholder’s priorities

(35:26) Q&A #4: Using the framework for non-developers

Where to find Eleanor Millman:

• LinkedIn: https://www.linkedin.com/in/eleanor-millman-98b10350

Where to find Mina Tawadrous:

• LinkedIn: https://www.linkedin.com/in/mina-tawadrous

Referenced:

• AWS

• Databricks

• RICE: Simple prioritization for product managers

• Designing developer experience surveys

• GSB Preserve | View | The Curse of Knowledge

Brian Scanlan is a Senior Principal Systems Engineer at Intercom, where he works on platform engineering, developer productivity, and AI adoption across the company.

In this session from DX Annual, Brian shares how Intercom set out to double engineering throughput and ultimately achieved that goal in nine months. Rather than treating AI as an optional productivity tool, the company standardized on Claude Code, updated performance expectations, invested heavily in enablement, and adopted an agent-first approach to technical work.

Brian explains why Intercom views Claude Code as a platform rather than a tool, how the company is building domain-specific skills and workflows for agents, and why it believes agents should eventually be able to perform any technical task a senior engineer can complete on a laptop.

He also shares the data behind Intercom’s AI adoption efforts, including gains in throughput, reductions in defect backlogs, improvements in code quality, and the growing use of automated pull request approvals. Throughout the talk, Brian offers a practical look at what it takes to scale AI adoption across a large engineering organization and the lessons Intercom has learned along the way.

Some takeaways:

Doubling engineering throughput

• Intercom set a goal to double engineering throughput. Rather than focusing on AI adoption metrics, the company chose a concrete business outcome: doubling merged pull requests per member of the R&D organization.

• The goal was achieved in nine months. Intercom ultimately doubled PR throughput and later tripled it over a 16-month period, with no signs of the trend slowing down.

• Throughput was treated as a reasonable proxy for impact. While Brian acknowledged that every metric has flaws, he argued that organizations adopting AI at scale should expect to see meaningful increases in output.

Managing organizational change

• AI adoption became part of the job. Intercom updated expectations for engineers, designers, and product managers so that using AI tools effectively became part of performance expectations rather than an optional activity.

• The company combined incentives with support. Hackathons, enablement days, dedicated support teams, leadership messaging, and public recognition all helped accelerate adoption.

• Leadership stayed relentlessly on message. Repeating the same goals and expectations across every forum helped create clarity about the direction of the organization.

Why Intercom standardized on Claude Code

• Intercom chose a single AI platform. Rather than allowing teams to fragment across different tools and workflows, the company standardized on Claude Code and invested heavily in making it work well.

• The real value comes from context, not models. Brian argued that domain knowledge, skills, documentation, workflows, and organizational context create far more value than constantly switching between models.

• Agents should be treated like new employees. Intercom’s goal is to onboard agents the same way it would onboard a senior engineer by giving them access, training, tools, documentation, and clear expectations.

Building an agent-first engineering organization

• All technical work is becoming agent-first. Intercom believes that any task a human can perform on a laptop should eventually be accessible to agents.

• The focus is on durable capabilities rather than custom AI infrastructure. Teams are encouraged to build skills, access patterns, and workflows that will remain valuable even as models and tools continue to evolve.

• Agents should solve problems, not just execute commands. Rather than telling agents exactly which skill to run, engineers increasingly describe the problem and allow agents to determine the best workflow.

Skills as organizational knowledge

• Intercom has built hundreds of reusable skills. These skills capture domain expertise, troubleshooting processes, coding standards, operational procedures, and other institutional knowledge.

• High-quality skills create leverage across the organization. Once a skill exists, every engineer can benefit from the expertise embedded within it, even if they were not involved in creating it.

• Skills continuously improve over time. Engineers are encouraged to update skills whenever new knowledge is discovered so that lessons learned become available to everyone.

Measuring the impact of AI adoption

• Nearly all pull requests are now authored by Claude. Brian shared that more than 95% of pull requests are created with AI assistance, while automated pull request approvals continue to grow.

• Saved time is being reinvested into quality. As teams gained efficiency, they spent more time reducing technical debt and fixing defects, leading to a significant reduction in Intercom’s defect backlog.

• Code quality improved alongside throughput. Research conducted with Stanford showed that recent code changes were improving the overall quality of the codebase rather than degrading it.

The future of agentic software development

• Intercom wants agents to participate throughout the software development lifecycle. The company is replacing runbooks, expanding automation, and building remote agent capabilities that move work beyond individual laptops.

• AI adoption has expanded far beyond engineering. More than a thousand employees use Claude Code weekly, including teams in finance, operations, and other business functions.

• The biggest changes may still be ahead. Brian believes AI will reshape planning, team structures, workflows, and engineering roles over the coming years, not just how code is written.

In this episode, we cover:

(00:00) Intro

(02:54) Intercom’s goal of doubling throughput

(07:30) The platform strategy

(09:30) Their agent-first strategy

(10:58) Evergreen capabilities vs custom tooling

(12:28) How Intercom works with agents

(16:43) What the data reveals about AI adoption and impact

(19:20) Using session data to improve AI workflows

(20:20) Cutting the defect backlog in half

(22:44) Inside Intercom’s Claude Code setup

(28:09) Claude Code beyond engineering

(30:49) Q&A #1: Token cost

(32:52) Q&A #2: Preparing for AI pricing changes

(34:14) Q&A #3: Stress testing and auditing skills

(36:31) Q&A #4: Criteria for agents approving PRs

Where to find Brian Scanlan:

• LinkedIn: https://www.linkedin.com/in/scanlanb

• X: https://x.com/brian_scanlan

• Website: https://brian.scanlan.ie

Referenced:

• Intercom

• Software? No Way. We’re an A.I. Company Now! - The New York Times

• Anthropic

• Snowflake

• Linear

• LaunchDarkly

• Fin AI

• Microsoft Copilot

• Cursor

• Claude Code | Anthropic’s agentic coding system

• Steve Yegge (@Steve_Yegge) / Posts / X

• Honeycomb

• Fin Ideas

• Fin CLI | AI Agent Command Line Interface

Abi Noda is joined live at DX Annual by three engineering leaders shaping AI adoption at scale: Tim Bozarth, Corporate Vice President in Microsoft’s CoreAI division; Nancy Wang, CTO of 1Password; and Taroon Mandhana, CTO of AI and Teamwork at Atlassian. Together, they discuss how AI is changing engineering organizations, from team structures and planning cycles to hiring, governance, and measurement.

The panel explores how the profile of a great engineer is evolving, why smaller cross-functional teams are becoming more effective, and what happens when product managers, designers, and customer support teams start contributing code. They also share why they are encouraging AI adoption through enablement, training, and local champions rather than mandates, and how AI is shifting more of the software development lifecycle toward planning and validation.

Finally, they discuss where human judgment remains essential, how to measure adoption and manage token usage, and how to connect AI investments to business outcomes while preserving room for experimentation and learning.

Some takeaways:

Rethink team structures for faster learning

• Smaller teams are becoming more effective for zero-to-one work. AI reduces the cost of implementation, making alignment and rapid iteration the primary bottlenecks when searching for product-market fit.

• Planning cycles are getting shorter. Instead of locking in 12- to 18-month roadmaps, teams are shifting toward quarterly planning to adapt more quickly to changing technology and market conditions.

• Large-scale org changes can wait. Several panelists emphasized that experimentation and learning should come before major structural redesigns.

The best engineers think like makers

• A maker’s mindset matters more than mastery of a specific tool. The most effective engineers stay focused on outcomes and use whatever tools help them build valuable products.

• Product and engineering skills are converging. Strong engineers increasingly combine technical depth with product judgment, customer empathy, and design sensibility.

• Agency is becoming a defining trait. Engineers who can work across functions, navigate ambiguity, and drive decisions are gaining leverage as AI handles more of the implementation work.

Expect more people to participate in software creation

• Prototypes are replacing long-form requirements documents. Interactive demos often lead to faster and more productive conversations than detailed specifications.

• Product managers, designers, and customer support teams are starting to write code. AI is lowering the barrier for non-engineers to contribute directly to the software development lifecycle.

• Quality systems become more important as more people contribute code. Robust tests, review processes, and deployment safeguards are essential when more people can generate production code.

Drive adoption through enablement, not mandates

• Outcomes matter more than activity. The goal is not maximizing AI usage for its own sake, but improving speed, ease, and product quality.

• Local champions accelerate adoption. Teams learn fastest when respected peers demonstrate practical ways to use AI in real workflows.

• Activity metrics are diagnostic, not the objective. Low usage can signal where teams need more training, support, or better tools.

The AI-native SDLC shifts work toward planning and validation

• Plan and validate are becoming the highest-leverage activities. As AI accelerates code generation, more human effort shifts toward defining what to build and evaluating whether it meets expectations.

• Operations and incident response are ripe for automation. Engineering teams are beginning to use AI to triage alerts, investigate incidents, write postmortems, and reduce the time spent on routine operational work.

• Human judgment remains essential. Leaders were unanimous that critical decisions around quality, security, and accountability still require people in the loop.

Measure outcomes, costs, and learning

• Token usage is becoming the new cloud bill. Engineering leaders are applying FinOps-style discipline to monitor and forecast AI spending.

• North Star metrics provide a clearer signal. Examples include idea-to-value, innovation time, and product quality.

• Experimentation deserves budget. Some AI usage will not generate immediate ROI, but it can create learning that compounds into long-term competitive advantage.

In this episode, we cover:

(00:00) Intro

(01:08) Introducing the panelists

(02:16) AI’s impact on engineering team structures and planning cycles

(05:00) How the role of the engineer is changing and what makes a great engineer

(10:11) The opportunities and challenges of non-engineers writing code

(15:26) Encouraging AI adoption without mandating it

(21:25) What an AI-native SDLC looks like and why human judgment still matters

(30:56) Measuring AI adoption, token usage, and ROI

(37:06) How to tie AI investments to business outcomes

Where to find Nancy Wang:

• LinkedIn: https://www.linkedin.com/in/wangnancy

Where to find Taroon Mandhana:

• LinkedIn: https://www.linkedin.com/in/taroonm

Where to find Tim Bozarth:

• LinkedIn: https://www.linkedin.com/in/tbozarth

Where to find Abi Noda:

• LinkedIn: https://www.linkedin.com/in/abinoda

Referenced:

• DX Core 4 Productivity Framework

• Microsoft

• 1Password

• Atlassian

• Jira

• Confluence

• Loom

• Rovo

• Amazon Operating Cadence - Working Backwards

Jennifer St Pierre is Senior Vice President of Developer Experience and Transformation at Dell Technologies, where she leads the strategy for how Dell’s Infrastructure Solutions Group builds, operates, and evolves software.

In this session from DX Annual, Jen argues that the biggest challenge in adopting agentic AI is not the technology itself, but the people transition behind it. Drawing on lessons from earlier shifts like Agile, DevOps, and cloud adoption, she explains why organizations that treat AI as a simple tooling rollout may get compliance, but not commitment.

Jen outlines five leadership imperatives for navigating the transition: building a shared understanding of why change is happening, defining a clear future state, clarifying how roles will evolve, creating psychological safety for experimentation, and aligning metrics and organizational structures with new ways of working. Throughout the talk, she emphasizes that while AI may generate code, humans remain responsible for direction, judgment, and meaning.

Some takeaways:

Treat AI adoption as a people transformation

• Technology transitions are really people transitions. The hardest part of adopting agentic AI is not the tooling itself, but helping engineers understand how their roles, workflows, and career paths will evolve.

• Compliance is not the same as commitment. Organizations that treat AI as a tooling rollout may achieve adoption metrics, but they will struggle to build the trust and engagement needed for lasting change.

• Every major platform shift follows a familiar pattern. New technologies create excitement, skepticism, fear, and eventually productivity gains that become the new normal.

Build a shared understanding of why AI adoption matters

• Start with an honest explanation of why the change is happening. If developers do not understand the business rationale, they are likely to assume the initiative is primarily about cost reduction.

• Shared understanding does not require universal agreement. It means everyone is working from the same candid view of market pressures, strategic goals, and organizational intent.

• Framing shapes emotional response. Positioning AI as a way to help engineers focus on more strategic work creates a very different reaction than simply saying it will increase productivity.

Define a clear future state

• A vague vision creates fear. When people cannot picture what their work will look like in 12 to 18 months, they tend to imagine replacement, stagnation, or obsolescence.

• Role clarity is essential. Teams need to understand what skills will matter, how performance will be measured, and which responsibilities will increase or diminish.

• Specificity beats slogans. Concrete expectations about how AI will be used help people see where they fit in the new model.

Create psychological safety for experimentation

• Teams need permission to make mistakes. AI adoption requires experimentation, and experimentation inevitably involves missteps and imperfect results.

• Psychological safety helps teams surface problems earlier. When engineers feel safe speaking up, leaders get better information and can address issues before they escalate.

• Silence is expensive. Leaders who discourage candor risk making decisions based on filtered or incomplete information.

Align metrics and organizational structures

• Old metrics can reinforce old behaviors. Measuring lines of code or heroic firefighting may encourage exactly the habits AI should help organizations move beyond.

• Metrics and structure must evolve together. Governance, incentives, funding, and performance systems need to support the behaviors leaders want to see.

• Transformation should survive without constant reminders. If the desired behaviors disappear as soon as leaders stop talking about them, the change has not yet become embedded.

Lead the transformation intentionally

• Leaders must model the change themselves. Using AI tools, sharing lessons learned, and being transparent about failures builds credibility.

• Career paths must be made explicit. Engineers want to know how they can continue to grow and whether deep technical expertise will remain valuable.

• AI may generate code, but humans generate direction. Judgment, context, and meaning remain the most valuable contributions people bring to software development.

In this episode, we cover:

(00:00) Intro

(00:13) Why every major technology shift is ultimately a people transition

(05:00) AI-generated code and the evolving role of software engineers

(07:43) The importance of developing a shared understanding

(12:00) Defining a clear future state and how engineering roles will evolve

(19:12) How psychological safety enables experimentation and honest feedback

(22:41) Why metrics and organizational structure must evolve for the age of AI

(25:40) Why leaders must drive AI transformation intentionally

Where to find Jennifer St Pierre:

• LinkedIn: https://www.linkedin.com/in/jennifer-st-pierre-4935a81

Referenced:

• Measuring developer productivity with the DX Core 4

• Understand team effectiveness

Jason Valentino is Head of Software Engineering Strategy at BNY, where he oversees developer tooling, DevEx, platform workflows, and software delivery governance across more than 8,000 engineers.

In this session from DX Annual, Jason shares how BNY moved beyond AI coding assistants to rethink the entire software delivery lifecycle. He explains how his team identified bottlenecks across the SDLC, prioritized automation opportunities, and applied AI to planning, peer review, testing, change management, and compliance workflows.

Jason also discusses what it takes to scale AI inside a highly regulated enterprise, including rewriting policies, partnering closely with risk and audit teams, and building a culture that encourages experimentation and rapid sharing of ideas.

Some takeaways:

Start with the 3X stress test

• Ask what breaks if engineering throughput triples. Jason’s team began by assuming AI would dramatically increase the volume of pull requests, code reviews, and releases, then identified which systems and processes would become bottlenecks.

• Map every step of the SDLC. BNY listed each task across planning, coding, testing, peer review, and release governance to understand which steps were manual, partially automated, or already well-instrumented.

• Use developer sentiment to prioritize investments. By combining workflow analysis with DX survey data, the team focused on the areas causing the most friction rather than chasing the latest AI use case.

Apply AI in three distinct ways

• Use IDE and CLI tools to amplify individual developers. Tools like Claude Code, Windsurf, and Codex help engineers move faster while still working within established guardrails.

• Deploy autonomous agents for repetitive work. BNY’s “digital workers” handle tasks like access requests, backlog grooming, and other low-value activities that engineers would rather avoid.

• Embed AI directly into workflows. The biggest gains come when AI is triggered automatically as part of code review, change management, and testing rather than relying on developers to invoke tools manually.

Use small automations to compound over time

• Automate the tedious parts of planning. BNY added AI capabilities to Jira to draft stories and epics, lint requirements, and assign confidence scores.

• Turn one automation into the next. Once a high-quality story exists, it becomes the foundation for generating test cases and other downstream artifacts.

• Look for highly manual actions. Jason recommends watching how teams actually work and identifying repetitive tasks that are prime candidates for automation.

Rebuild governance for an AI-assisted world

• Rewrite policies and controls. Existing language around code review, approvals, and software delivery often assumes humans perform every step and must be updated to reflect AI-assisted workflows.

• Bring risk and audit teams in early. Rather than presenting finished solutions for approval, BNY collaborates with governance partners while designing new approaches.

• Codify deterministic rules. AI can handle routine work automatically, while larger or riskier changes are routed to humans for additional oversight.

Treat duplication as a feature, not a bug

• Expect multiple teams to solve the same problem. In a large organization, some overlap is inevitable when thousands of people are experimenting with AI.

• Use show-and-tell to surface innovation. BNY hosts weekly sessions where teams demonstrate what they’ve built and share lessons learned.

• Consolidate the best ideas. Once similar solutions emerge, platform leaders can combine the strongest features into shared capabilities.

Create a culture that rewards experimentation

• Start saying yes. Jason’s advice to engineering leaders is to lower barriers and put promising ideas in front of users quickly.

• Treat internal tools like products. Successful experiments are documented, shared, and iterated on rather than left as one-off hacks.

• Make engineering fun again. For Jason, one of the biggest wins of the past year has been seeing teams energized by the opportunity to solve meaningful problems with AI.

In this episode, we cover:

(00:00) Intro

(01:20) Early results from AI coding tools at BNY

(04:08) The 3X stress test: What breaks if engineering throughput triples?

(06:56) Three ways to apply AI across the SDLC: IDE and CLI tools

(08:07) Using autonomous AI agents for repetitive engineering tasks

(09:16) Embedding AI directly into SDLC workflows

(12:27) Why leaders should encourage experimentation and “start saying yes”

(15:00) Q&A: How platform and productivity teams are evolving to support AI

(16:33) Q&A: Rewriting policies and controls for AI-assisted software delivery

(17:52) Q&A: How AI is affecting software quality and test ownership

(19:00) Q&A: What Jason is most proud of: Practical examples of AI across the SDLC

(20:30) Q&A: How BNY handles duplicated work across AI initiatives

(22:30) Q&A: How BNY uses AI to support regulatory and compliance work

(23:30) Q&A: Automating code reviews and change tickets

(25:55) Q&A: How increased AI-driven throughput is affecting on-call and reliability

(27:11) Q&A: How BNY works with risk and audit partners to move quickly with AI

(29:01) Q&A: How BNY scales successful AI use cases across the organization

(30:42) Q&A: What Jason is most proud of after BNY’s busiest year with AI

Where to find Jason Valentino:

• LinkedIn: https://www.linkedin.com/in/jasonvalentino

Referenced:

• AI-assisted engineering: Q4 impact report

• Measuring AI code assistants and agents

• Measuring developer productivity with the DX Core 4

• Windsurf

• Claude Code by Anthropic | AI Coding Agent, Terminal, IDE

• Codex | AI Coding Agent

Recorded live at DX Annual, Abi Noda, co-founder and CEO of DX, joins Brian Houck of Microsoft to share an early look at DX’s new research on AI’s impact on engineering velocity.

Drawing on data from a sample of DX customers, they discuss what companies are actually seeing as AI adoption matures. Most organizations in the study saw pull request throughput increase by 10 to 15 percent—far more modest than the 10x gains often promised in industry headlines.

They explore why coding remains only a small part of developer work, where time saved by AI may be going, and the unintended consequences of moving faster, from shifting bottlenecks to “false velocity.” Abi also shares how engineering leaders are applying AI beyond coding and how DX is evolving its measurement framework to account for both human and agent productivity.

Some takeaways:

Most organizations are seeing modest gains from AI

• PR throughput is increasing by about 10 to 15 percent. Across DX’s sample, most organizations saw measurable improvements, but the gains were far smaller than the 10x productivity increases often cited in industry headlines.

• The median improvement was closer to 8 percent. While some organizations saw larger gains, the typical impact was more incremental than transformational.

• Even modest gains can be meaningful at scale. A 10 percent increase in throughput can represent a significant improvement when applied across hundreds or thousands of engineers.

Coding is only one part of the productivity equation

• Developers spend only about 14 percent of their time writing code. If AI primarily accelerates coding, its impact on overall engineering velocity will naturally be constrained.

• The biggest bottlenecks often lie elsewhere. Planning, reviews, testing, documentation, and coordination still consume the majority of engineering time.

• Time savings do not map neatly to output gains. Organizations can see meaningful reductions in coding effort without a proportional increase in pull request volume.

Why productivity gains are lower than many leaders expected

• Coding is not the primary bottleneck. Improving a small slice of the development process only moves the overall system so far.

• Automation creates new bottlenecks. Faster code generation can increase pressure on reviews, QA, and technical oversight.

• Social friction slows adoption. Skepticism, inconsistent usage, and unrealistic expectations can limit the benefits of AI tools.

• Tool and skill gaps compound over time. Engineers need both the right tools and the knowledge to use them effectively.

• AI tools still lack context. Limited understanding of business logic and codebase nuances can reduce output quality.

Beware of false velocity

• More code does not necessarily mean more business value. Teams can increase pull request counts without meaningfully accelerating roadmap delivery.

• Quality and cost remain critical concerns. Organizations are closely monitoring technical debt, token spend, and long-term maintainability.

• Faster output can create delayed consequences. The full impact of AI-generated code may not become apparent until months later.

The biggest opportunities lie beyond coding

• The remaining 86 percent of engineering work is the next frontier. Leaders are applying AI to planning, documentation, incident response, and other parts of the SDLC.

• Autonomous agents can augment human capacity. Instead of simply speeding up developers, organizations are exploring how agents can work in parallel.

• Developer experience still matters. Improving focus time, documentation, and workflow friction can amplify the benefits of AI.

Measurement frameworks are evolving

• Some metrics remain constant. Velocity, quality, and developer experience are still essential signals.

• Acceleration and augmentation should be measured separately. Leaders need to distinguish between human productivity gains and work performed autonomously by agents.

• Agent experience is an emerging concept. DX is beginning to survey AI agents directly to understand their constraints, bottlenecks, and effectiveness.

Cognitive debt is a new concern

• AI can reduce understanding while increasing output. Developers may ship code more quickly while building a weaker mental model of the systems they maintain.

• Short-term efficiency can create long-term costs. Reduced comprehension may make future debugging and maintenance more difficult.

• The long-term effects are still uncertain. Engineering leaders are only beginning to understand the human consequences of AI-assisted development.

In this episode, we cover:

(00:00) Intro

(00:53) What motivated DX’s research into AI’s impact on engineering velocity

(02:36) How DX designed the study and selected companies

(04:54) What DX’s data reveals about AI’s impact on engineering throughput

(06:31) Why PR throughput was the most practical metric to publish

(08:21) Why AI productivity gains are lower than many leaders expected

(10:24) How an all-in culture can amplify AI productivity gains

(12:35) Why it’s hard to track where AI-generated time savings are going

(15:04) Unintended consequences of AI-driven productivity gains

(17:12) Why leaders should look beyond coding to the rest of the SDLC

(19:43) Cognitive debt and the human costs of AI-assisted development

(21:33) How DX’s AI measurement framework is evolving

(24:42) How to make agents more effective

Where to find Brian Houck:

• LinkedIn: https://www.linkedin.com/in/brianhouck/

Where to find Abi Noda:

• LinkedIn: https://www.linkedin.com/in/abinoda

Referenced:

• DX Core 4 Productivity Framework

• DORA, SPACE, and DevEx: Which framework should you use?

• Time Warp: The Gap Between Developers’ Ideal vs Actual Workweeks in an AI-Driven Era - Microsoft • Research

• How Generative and Agentic AI Shift Concern from Technical Debt to Cognitive Debt

• Measuring AI code assistants and agents

In this episode, I’m joined by Eleanor Millman, Senior Staff Product Manager, and Mina Tawadrous, Associate Director of Platform Engineering at SiriusXM, to discuss how platform teams can scale prioritization without relying on revenue.

We talk through how SiriusXM moved beyond RICE to build a custom framework for internal platforms, using weighted factors like developer speed, reliability, cost, and trust to guide decisions across teams.

We also explore their concept of “assumptions as code,” in which teams store and reuse assumptions in a central repository to reduce misalignment and improve decision-making, with AI helping to surface and validate those assumptions.

We close with how this system is shaping SiriusXM’s 2026 prioritization approach and what it signals about a broader shift toward builder-driven product development.

Some takeaways:

Prioritization breaks without a shared system

• Prioritization does not scale naturally across teams. What works for one team breaks down at the org level with multiple stakeholders and competing requests

• Platform teams lack a clear revenue signal. Unlike product teams, they must prioritize based on indirect impact

• A shared framework aligns decisions. Without it, prioritization defaults to local optimization and noise

RICE is a starting point, not a solution

• Standard frameworks miss key dimensions for platform teams. Urgency and indirect impact are not captured well

• “Impact” needs to be decomposed. SiriusXM broke it into developer speed, reliability, cost, security, and more

• The framework must evolve over time. Iteration was critical to making it useful in practice

Weighting forces real tradeoffs

• You cannot prioritize everything at once. Increasing one dimension (like cost) necessarily deprioritizes others

• Assigning weights makes decisions explicit. Leaders must commit to what matters this quarter

• The output drives alignment across teams. A single prioritized list reduces cross-team conflicts

Data and conversation work together

• The framework creates a place to attach data. Metrics like reliability scores inform prioritization decisions

• Disagreements surface quickly. Teams can see where assumptions or inputs differ

• Conversations, not just scores, drive alignment. The value comes from debating inputs, not just ranking outputs

Assumptions are the real bottleneck

• Most disagreements come from hidden assumptions. Teams often believe they are aligned when they are not

• Assumptions can be conflicting, invisible, or stale. All three create friction in decision-making

• Making assumptions explicit improves clarity. It becomes easier to validate or challenge them

Storing assumptions as code scales learning

• Assumptions are stored in a central repository. User research and data become reusable across teams

• This reduces duplicated effort. Teams don’t need to rediscover the same insights repeatedly

• It creates a shared source of truth. Assumptions become visible, versioned, and easier to update

In this episode, we cover:

(00:00) Intro

(01:17) Mina’s role and path into platform engineering

(02:03) Eleanor’s background and shift into product

(03:15) Scaling prioritization across platform engineering teams

(05:41) Aligning platform priorities with stakeholders

(09:08) Evolving RICE into a platform-specific prioritization framework

(11:33) Iterating on the prioritization framework over time

(16:57) How the framework, data, and conversations drive alignment

(19:06) Storing assumptions as code in a central repository

(26:47) Resolving assumption conflicts with user interviews

(30:47) How stored assumptions integrate with AI workflows

(35:30) Standard mode and different user personas

(37:20) The industry shift towards builders

(41:04) The challenges of platform engineering

(43:36) How SiriusXM is prioritizing in 2026

Referenced:

• Measuring AI code assistants and agents

• SiriusXM

• VMware

• How SiriusXM revamped their platform and developer experience

• RICE Scoring Model | Prioritization Method Overview

• The evaporating cloud: A tool for resolving workplace conflict

In this special episode of Engineering Enablement, I welcome back Jesse Adametz, this time as host.

In our conversation, we explore how AI is showing up across the SDLC, not just in code generation, and how it is shifting bottlenecks across the development process. We unpack what “AI readiness” actually means in practice, and why it often comes down to developer experience fundamentals like documentation, environments, and feedback loops.

We also discuss why enablement matters more than tool choice, how teams are thinking about measuring ROI, and what changes as background agents become more common. Finally, we explore how the role of the engineer may evolve, what questions teams are still trying to answer, and the challenges of non-engineers contributing to codebases.

Some takeaways:

AI is expanding beyond coding into the full SDLC

• The focus has shifted from code generation to the entire software lifecycle. Teams are applying AI to planning, prototyping, review, and documentation—not just writing code.

AI readiness is a developer experience problem

• The biggest blockers to AI adoption are long-standing DX gaps. Missing documentation, inconsistent environments, weak CI, and unclear system boundaries all limit effectiveness.

• Tool choice is not the primary driver of success. Models and tools are evolving too quickly for this to be a durable advantage.

• Some organizations are formalizing AI enablement as a function. Dedicated teams are emerging to drive adoption and share practices.

Measuring AI ROI is messy and still evolving

• Correlation vs causation makes attribution difficult. High AI usage often correlates with already high-performing engineers.

• Longitudinal analysis is more reliable than snapshots. Tracking changes over time gives better insight into impact.

• Token spend introduces real cost considerations. AI creates a direct, variable cost that organizations must evaluate.

AI impact falls into two buckets: amplification and augmentation

• Amplification improves human productivity. This includes higher throughput, time savings, and better developer experience.

• Augmentation extends capacity beyond humans. Agents begin to act as additional “headcount,” completing work independently.

• These require different measurement approaches. Amplification focuses on human output, while augmentation focuses on agent output relative to cost.

Background agents shift how work gets done and where bottlenecks appear

• Agents enable work to happen outside the human loop. Tasks can be completed asynchronously and proactively.

• This changes the developer role. Engineers move toward reviewing, guiding, and orchestrating agent output.

• Human workflows can become the bottleneck. If agents produce work faster than humans can process it, the constraint shifts.

• This reframes productivity. The question becomes where human involvement adds the most value.

Specs and documentation are becoming critical infrastructure

• AI makes documentation a core dependency. It directly impacts the quality of outputs.

• Poor documentation leads to poor results. Agents can duplicate systems or make incorrect assumptions without context.

• Documentation is shifting from optional to essential. It is now foundational for both human and AI productivity.

In this episode, we cover:

(00:00) Intro

(02:12) Where AI is showing up across the SDLC

(05:53) AI readiness and its link to developer experience

(08:23) Why enablement, education, and experimentation matter more than tool choice

(13:05) The case for a dedicated enablement team

(14:50) Measuring AI ROI: challenges and tradeoffs

(19:46) Background agents and token spend

(24:12) Measuring agent output with PR throughput

(26:58) How the engineer role might change

(31:01) Specs and documentation in the age of AI

(33:11) Non-engineers writing code

(35:30) What’s changing in the SDLC and open questions

Referenced:

• Measuring AI code assistants and agents

• Lessons from Twilio’s multi-year platform consolidation

• The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win

• How Claude remembers your project - Claude Code Docs

• specIsJustCode : r/ProgrammerHumor

Developer productivity is often treated as a tooling problem or a sentiment problem. In reality, it’s neither. It’s a socio-technical systems problem that spans engineering foundations, leadership alignment, organizational design, and culture.

In this episode, I’m joined by Uma Namasivayam, Senior Director, Engineering Productivity at Dropbox, to explore how Dropbox approaches developer experience at scale. We talk about why productivity needs to be framed as a business problem, how executive alignment creates the conditions for meaningful change, and what it takes to treat developer experience as a real product with developers as customers.

We also dig into Dropbox’s approach to AI adoption. Uma shares why strong foundations, such as build, test, and observability, are prerequisites for AI to actually accelerate work, how Dropbox encourages daily AI use without mandating a single tool, and where build-versus-buy decisions break down at scale.

We close with an honest look at what remains unsolved: how to connect gains in developer productivity and AI-driven capacity to real business outcomes, and where engineering leaders should focus next in 2026.

Some takeaways:

Developer productivity is a socio-technical problem

• Productivity cannot be solved through tooling alone; it spans engineering systems, leadership behavior, organizational structure, and people practices.

• Problems like build and test are engineering problems, while problems like focus time and interruptions are people problems, and both matter equally.

• Treating productivity as a system forces tradeoffs to be explicit, rather than hidden inside isolated tooling initiatives.

Executive alignment matters more than any single metric

• Top-down sponsorship creates permission to act, especially when productivity work cuts across org boundaries.

• A shared framework creates alignment, not answers; its value is giving leaders and engineers a common language.

• System metrics matter more than single metrics, because productivity improvements rarely move one dimension in isolation.

• Distributed accountability makes productivity a company problem, not a developer experience team problem.

Developer experience works best when treated as a product discipline

• Developers are customers, and their experience must be understood through both qualitative feedback and quantitative signals.

• Good system metrics do not guarantee good developer experience, which is why sentiment and perception matter.

• DX surveys surface where systems break differently for different teams, such as desktop, mobile, and web developers.

• Continuous feedback loops are essential, combining surveys, direct conversations, and usage data.

• Internal communication is part of the product, reinforcing to developers that their feedback leads to real change.

Prioritization requires structure, not intuition

• Finite capacity makes prioritization unavoidable, even in large, well-resourced engineering orgs.

• Segmenting developer populations clarifies tradeoffs, since different teams experience different bottlenecks.

• DX survey data provides a defensible starting point, but prioritization still requires judgment.

• Leadership-level stack ranking helps resolve conflicts, especially when multiple teams compete for attention.

• Frameworks make hard decisions easier to explain, even when they do not make them easy.

AI and developer experience must advance in parallel

• AI accelerates work, while developer experience reduces friction, and both are required for sustained gains.

• Foundational systems act as plumbing, enabling trust in speed, quality, and safety.

• Without strong CI, testing, and observability, faster code creation increases risk instead of value.

• Trust in guardrails enables confidence in AI-assisted development, especially at scale.

AI adoption succeeds through choice, not mandates

• Early organic adoption revealed real developer needs, rather than forcing a single tool.

• Different teams require different AI tools, particularly for mobile, desktop, and large-repo workflows.

• Supporting multiple tools increased adoption, rather than reducing it.

• Daily use depends on fitting AI into existing workflows, not adding extra steps.

• Habits matter more than access, which is why SDLC-level integration is critical.

Build vs. buy decisions change at scale

• Many AI tools fail when tested at large-company scale, despite working well in smaller contexts.

• Cost and performance become gating factors, not feature completeness.

• Internal platforms can abstract complexity, enabling teams to build AI workflows safely and consistently.

• Shared internal platforms unlock reuse, allowing teams to innovate without rebuilding infrastructure.

• Speed of iteration remains the primary differentiator, even when building in-house.

In this episode, we cover:

(00:00) Intro

(00:45) Dropbox’s engineering org

(01:59) Why developer productivity is a business problem

(04:08) The role of executive sponsorship in developer productivity

(06:02) How DX’s Core Four framework created a shared language

(08:13) Treating developer experience as a product

(11:30) How Dropbox prioritizes developer experience work

(14:20) The challenge of tying developer experience to business outcomes

(16:38) How AI and developer experience intersect at Dropbox

(18:35) The prerequisites for AI adoption to accelerate work

(20:26) How Dropbox encourages daily AI use

(23:12) AI use beyond code completion

(25:00) Managing AI tool demand at scale

(27:56) Early results from Dropbox’s AI efforts

(30:05) Progress on developer experience at Dropbox

(32:55) Advice for organizations investing in developer experience

(34:25) Capacity tradeoffs for developer experience

(35:59) The unanswered questions around AI and capacity in 2026

Referenced:

• DX Core 4 Productivity Framework

• Dropbox.com

As we close out 2025, I wanted to step back and take stock of what we have actually learned about AI adoption in engineering organizations. Not just where usage has increased, but where impact is real, where it is overstated, and what questions remain unanswered.

In this special year-end episode, I’m joined by Brian Houck from Microsoft, Collin Green and Ciera Jaspan from Google, and Eirini Kalliamvakou from GitHub. Together, we unpack the research each of them worked on this year and explore how leading organizations are thinking about AI measurement, developer experience, and long-term productivity. We talk candidly about why measuring AI’s impact is so difficult, why familiar metrics like lines of code keep resurfacing despite their flaws, and how multidimensional approaches like SPACE and DORA offer a more realistic lens.

We also look ahead to 2026. We discuss how AI is beginning to reshape the identity of the developer, how junior engineers’ skill sets may evolve, where agentic workflows are gaining traction, and why some of the most widely shared AI studies were misunderstood. This episode is an honest conversation about moving past hype and toward a more grounded, evidence-based approach to AI adoption in engineering teams.

Some takeaways:

Measuring AI impact requires multiple lenses

• There is no single metric that can capture AI’s impact. Developer productivity and experience are inherently multidimensional, requiring trade-offs to be evaluated across speed, quality, collaboration, and meaning.

• Frameworks like SPACE and DORA help avoid metric tunnel vision. They encourage teams to examine complementary signals rather than optimizing one dimension at the expense of others.

• Measurement must reflect systems, not tools. AI does not operate in isolation; its impact depends on organizational context, workflows, and existing engineering practices.

Why familiar metrics keep failing us

• Lines of code remains a deeply misleading metric. AI tends to generate verbose code, making raw output a poor proxy for productivity, quality, or long-term maintainability.

• More code does not equal better outcomes. Excess code can increase maintenance burden, technical debt, and cognitive load over time.

• Easy-to-measure metrics are often the most dangerous. Their simplicity makes them attractive during periods of uncertainty, even when they obscure what is actually changing.

The limits of tracking AI-generated code

• Measuring the percentage of AI-generated code oversimplifies reality. AI may write, delete, refactor, or reorganize code in ways that raw percentages fail to capture.
  AI-generated code does not inherently signal higher risk. In some contexts, AI output may be more consistent or higher quality than human-written code.

• These metrics are better used as supporting signals, not goals. They can inform budgeting, experimentation, or adoption patterns but should not drive performance targets.

How AI is reshaping the role of the developer

• Developers are shifting from implementers to orchestrators. Advanced AI users spend more time framing problems, setting context, and validating outcomes than writing raw code.

• AI fluency is becoming a core skill. Knowing how to guide, correct, and collaborate with agents is increasingly important.

• Adoption follows a progression. Developers tend to move from skepticism to exploration, collaboration, and eventually strategic use as expectations recalibrate.

What this means for junior engineers

• Skill development may accelerate rather than disappear. Junior engineers may practice delegation, planning, and system-level thinking earlier by working with AI agents.

• Technical fundamentals still matter. Understanding architecture, requirements, and failure modes remains essential for supervising AI-generated work.

• Interpersonal skills risk being deprioritized. Managing agents is not the same as managing people, raising concerns about how collaboration skills develop over time.

AI is not just a productivity tool

• Creativity and innovation benefit from friction. Research suggests that exposing decision points and seams can create space for new ideas rather than faster repetition.

• Automating everything is not always desirable. Removing all toil may reduce opportunities for learning, insight, and creative problem-solving.

• AI should augment thinking, not replace it. Tools that surface trade-offs and choices can support better outcomes than those that simply eliminate effort.

High-leverage AI use cases focus on toil

• Developers spend only about 14% of their time writing code. Optimizing coding alone rarely leads to large productivity gains.

• The biggest opportunities lie in removing friction. Documentation, compliance tasks, incident response, flaky tests, and knowledge discovery consistently rank as top pain points.

• AI excels at work developers dislike but must still do. Automating dull, repetitive tasks can improve satisfaction and free time for meaningful work.

Why leadership and change management matter

• AI adoption is a human problem before it is a technical one. Organizations that understand developer pain points deploy AI more effectively.
  Agentic workflows amplify organizational differences. Teams with strong experimentation cultures and feedback loops move faster and with less friction.

• Culture determines outcomes. How leaders communicate expectations, normalize experimentation, and support learning shapes whether AI adoption succeeds or stalls.

Looking ahead to 2026

• Task parallelization is an emerging frontier. Developers are beginning to use agents to explore multiple solution paths simultaneously.

• Collaboration with agents will redefine productivity. Teams, not just individuals, will increasingly work alongside AI systems.

• Research must evolve with the work itself. New workflows will require new metrics, new telemetry, and new ways of understanding impact.

Lessons from the METR paper

• Context matters more than headlines suggest. Results showing slower performance often reflected expert developers working in familiar codebases.

• AI may help most where familiarity is lowest. New domains, unfamiliar systems, and onboarding scenarios show different outcomes.

• Media oversimplification distorts understanding. Nuance is critical when interpreting AI research, especially as studies move into real-world environments.

In this episode, we cover:

(00:00) Intro

(02:35) Introducing the panel and the focus of the discussion

(04:43) Why measuring AI’s impact is such a hard problem

(05:30) How Microsoft approaches AI impact measurement

(06:40) How Google thinks about measuring AI impact

(07:28) GitHub’s perspective on measurement and insights from the DORA report

(10:35) Why lines of code is a misleading metric

(14:27) The limitations of measuring the percentage of code generated by AI

(18:24) GitHub’s research on how AI is shaping the identity of the developer

(21:39) How AI may change junior engineers’ skill sets

(24:42) Google’s research on using AI and creativity

(26:24) High-leverage AI use cases that improve developer experience

(32:38) Open research questions for AI and developer productivity in 2026

(35:33) How leading organizations approach change and agentic workflows

(38:02) Why the METR paper resonated and how it was misunderstood

Referenced:

• Measuring AI code assistants and agents

• Kiro

• Claude Code - AI coding agent for terminal & IDE

• SPACE framework: a quick primer

• DORA | State of AI-assisted Software Development 2025

• Martin Fowler - by Gergely Orosz - The Pragmatic Engineer

• Seamful AI for Creative Software Engineering: Use in Software Development Workflows | IEEE Journals & Magazine | IEEE Xplore

• AI Where It Matters: Where, Why, and How Developers Want AI Support in Daily Work - Microsoft Research

• Unpacking METR’s findings: Does AI slow developers down?

• DX Annual 2026

AI engineering tools are evolving fast. Every month brings new coding assistants, debugging agents, and automation capabilities. I want to help engineering leaders take advantage of that innovation while avoiding costly experiments that distract from real product work.

In this episode, Abi Noda and I share a practical, data-driven approach to evaluating AI tools. I walk through how to shortlist tools by use case, design structured trials that reflect real work, select representative participants, and measure impact using baselines and proven frameworks. My goal is to give you a way to test and adopt AI tools with confidence and a clear return on investment.

Some takeaways:

Data-driven evaluations are essential

• Structured, measurable trials prevent bias. Without them, decisions are driven by novelty hype or a few loud voices.

• Define a clear business outcome first (reduce toil, improve delivery speed, or raise code quality).

• Evaluations must inform real decisions, not just check a procurement box.

Choose the right set of tools to evaluate

• Group tools by use case and interaction mode (chat, agentic IDEs, code review assistants, etc.) to ensure fair comparisons.

• Match shortlist size to org capacity to support multiple cohorts and reliable results.

• Multi-vendor strategies reduce lock-in in a rapidly shifting market.

Re-evaluations are essential, not optional

• Incumbent tools must be retested as capabilities evolve and new challengers emerge.

• Triggers for re-evaluation include major feature launches, organic developer adoption of a new tool, and upcoming renewal cycles.

• Every challenger tool evaluation requires a baseline of the incumbent, so you can compare like-for-like.

• A cadence of every 8–14 months ensures decisions reflect the current reality, not the past purchase.

Design trials around research questions

• Start with a hypothesis. It keeps experiments aligned to actual goals.

• Developer sentiment is necessary but insufficient without measurable outcomes.

• Success criteria must be defined in advance to avoid subjective decision-making.

Select representative participants

• Diverse cohorts reveal real impact across languages, teams, and seniority levels.

• Include skeptical and late adopters to uncover onboarding and enablement needs.

• Volunteer-only trials distort results and won’t scale to full org rollout.

Run evaluations long enough to capture true behavior

• Eight to twelve weeks is the minimum to get past the novelty phase and into sustained usage.

• Align evaluation windows to procurement cycles so insights guide buying decisions.

• Short trials lead to false signals and either inflate enthusiasm or create false negativity.

Use self-reported time savings carefully

• Self-reporting is a strong early indicator of perceived usefulness.

• Humans misremember time, often benchmarking against recent AI use.

• Treat CSAT and time savings as directional, not the final truth.

• Objective metrics validate real ROI, including throughput, quality, and innovation time.

Expect variation rather than a single winner

• Different tools shine in different contexts, so multiple standards are often the best path.

• Continuous re-evaluation is required as capabilities evolve every quarter.

• The right goal isn’t the “best tool”, but the best tool for each problem space.

In this episode, we cover:

(00:00) Intro: Running a data-driven evaluation of AI tools

(02:36) Challenges in evaluating AI tools

(06:11) How often to reevaluate AI tools

(07:02) Incumbent tools vs challenger tools

(07:40) Why organizations need disciplined evaluations before rolling out tools

(09:28) How to size your tool shortlist based on developer population

(12:44) Why tools must be grouped by use case and interaction mode

(13:30) How to structure trials around a clear research question

(16:45) Best practices for selecting trial participants

(19:22) Why support and enablement are essential for success

(21:10) How to choose the right duration for evaluations

(22:52) How to measure impact using baselines and the AI Measurement Framework

(25:28) Key considerations for an AI tool evaluation

(28:52) Q&A: How reliable is self-reported time savings from AI tools?

(32:22) Q&A: Why not adopt multiple tools instead of choosing just one?

(33:27) Q&A: Tool performance differences and avoiding vendor lock-in

Where to find Laura Tacho:

• LinkedIn: https://www.linkedin.com/in/lauratacho/

• X: https://x.com/rhein_wein

• Website: https://lauratacho.com/

• Laura’s course (Measuring Engineering Performance and AI Impact): https://lauratacho.com/developer-productivity-metrics-course

Where to find Abi Noda:

• LinkedIn: https://www.linkedin.com/in/abinoda

• Substack: ​​https://substack.com/@abinoda

Referenced:

• Measuring AI code assistants and agents

• QCon conferences

• DX Core 4 engineering metrics

• DORA’s 2025 research on the impact of AI

• Unpacking METR’s findings: Does AI slow developers down?

• METR’s study on how AI affects developer productivity

• Claude Code

• Cursor

• Windsurf

• Do newer AI-native IDEs outperform other AI coding assistants?

In this episode of Engineering Enablement, I sit down with Nathen Harvey, who leads research at DORA, to explore how teams should really think about measuring the impact of AI. We talk about why traditional delivery metrics can give leaders a false sense of confidence and how AI acts as an amplifier, accelerating healthy systems while intensifying existing friction and failure.

We examine findings from the 2025 DORA research on AI-assisted software development alongside DX’s Q4 AI Impact report and unpack where the data aligns and where meaningful gaps emerge. We also dig into how AI is reshaping engineering systems themselves, changing workflows, feedback loops, and team dynamics in ways leaders need to understand to achieve real, sustainable impact.

Some takeaways:

DORA metrics alone cannot measure AI impact

• The four “key” DORA metrics only reflect delivery outcomes, not system behavior. They show where teams end up, not how they got there.

• DORA now measures five software delivery performance metrics, not four.

• These metrics function like a compass rather than a diagnostic tool.

• Delivery performance metrics are leading indicators of organizational health but lagging indicators of engineering practices.

AI acts as an organizational amplifier

• AI does not fix systems; it intensifies what already exists. Strong practices compound while weak practices become more painful.

• Healthy teams experience faster flow while unhealthy systems accumulate more visible friction.

• AI makes hidden bottlenecks impossible to ignore.

The five DORA software delivery performance metrics

• DORA divides delivery performance into throughput and instability categories.

• Throughput metrics include lead time for changes, deployment frequency, and failed deployment recovery time.

• Instability metrics include change fail rate and deployment rework rate.

DX Q4 2025 AI Impact report insights

• Junior engineers adopt AI more heavily than senior engineers. This shifts how work is distributed across teams.

• Senior engineers often capture more measurable time savings despite lower visible usage.

• DX found widespread experimentation with non-enterprise AI tools.

• Engineers reported high AI usage even when enterprise telemetry showed no activity.

• Shadow experimentation reflects weak or unclear organizational AI guidance.

The DORA AI Capabilities Model

• Successful AI adoption depends on team and organizational capabilities, not tool selection.

• A clear and communicated AI stance reduces uncertainty and speeds adoption.

• A healthy internal data ecosystem prevents AI usage from being blocked by silos.

• Internal policies and documentation must be accessible to both humans and AI systems.

• Strong version control provides rollback safety when AI-generated code diverges.

• Small batch work improves both AI output quality and system stability.

• User-centered thinking ensures AI effort aligns with real human outcomes.

• High-quality internal platforms allow improvements to scale across teams.

AI shifts where work breaks

• AI accelerates code creation but moves constraints downstream.

• Code review becomes the dominant bottleneck under AI-assisted development.

• Increased code volume without improved review systems slows overall throughput.

• Bottlenecks become more visible, not less, as AI usage grows.

Measuring AI ROI requires human signals

• Dashboards cannot capture where work feels slow or painful.

• Leaders need direct conversations with engineers about friction and workflow breakdowns.

• Qualitative insight exposes failure points that metrics cannot surface.

In this episode, we cover:

(00:00) Intro

(00:55) Why the four key DORA metrics aren’t enough to measure AI impact

(03:44) The shift from four to five DORA metrics and why leaders need more than dashboards

(06:20) The one-sentence takeaway from the 2025 DORA report

(07:38) How AI amplifies both strengths and bottlenecks inside engineering systems

(08:58) What DX data reveals about how junior and senior engineers use AI differently

(10:33) The DORA AI Capabilities Model and why AI success depends on how it’s used

(18:24) How a clear and communicated AI stance improves adoption and reduces friction

(23:02) Why talking to your teams still matters

Where to find Nathen Harvey:

• LinkedIn: https://www.linkedin.com/in/nathen

Where to find Laura Tacho:

• LinkedIn: https://www.linkedin.com/in/lauratacho/

• X: https://x.com/rhein_wein

• Website: https://lauratacho.com/

• Laura’s course (Measuring Engineering Performance and AI Impact) https://lauratacho.com/developer-productivity-metrics-course

Referenced:

• DORA | State of AI-assisted Software Development 2025

• Steve Fenton - Octonaut | LinkedIn

• AI-assisted engineering: Q4 impact report