Uber’s journey of measuring AI impact on developer productivity

Engineering Enablement by DX

0:00

-41:17

Uber’s journey of measuring AI impact on developer productivity

How Uber evolved its approach to measuring AI’s impact on engineering, why traditional productivity metrics are breaking down, and what new frameworks may be needed in an agent-driven future.

Jun 22, 2026

Listen and watch now on YouTube, Apple, and Spotify.

As AI becomes embedded in software development, many of the metrics that engineering organizations have relied on for years are starting to break down.

In this session from DX Annual, Uber’s Ty Smith and Abhishek Tibrewal share how their approach to measuring AI’s impact on developer productivity has evolved over time. They walk through the different phases of their measurement journey, from adoption and engagement to measuring impact, ROI, and agentic value, explaining what they chose to measure at each stage, what worked, what failed, and how their thinking changed along the way.

They also discuss the role of qualitative feedback before telemetry existed, the challenge of identifying meaningful engagement signals, why “developer years saved” failed as an ROI metric, and how AI agents forced them to rethink traditional productivity measurements. Finally, they introduce Uber’s emerging framework built around feature velocity and explore the unanswered questions that remain as software development becomes increasingly agent-driven.

Some takeaways:

Why AI breaks traditional productivity metrics

Many measurement frameworks were built for a world where humans wrote most of the code. As AI agents become more capable, metrics that once provided useful signals can quickly become misleading.
Teams should expect their metrics to break. Uber’s measurement journey required repeatedly revisiting assumptions as AI-assisted development evolved into agentic workflows.

Start with stakeholder questions

The best metrics answer real business questions. Uber worked backward from questions about productivity, ROI, investment priorities, and business value instead of collecting data for its own sake.
Measurement should support decision-making. Metrics influence budgets, tooling investments, enablement efforts, and long-term strategy.
Use qualitative signals before telemetry exists
Qualitative feedback can be the fastest path to insight. Before AI tooling generated reliable telemetry, Uber relied on surveys, interviews, and experience sampling to understand adoption and guide investments.
Behavioral questions are more useful than perception questions. Asking developers what they actually did produced stronger signals than asking whether they found AI helpful.
Measure engagement through behavior, not demographics
Behavioral patterns revealed insights that demographics could not. Role, tenure, and organization offered limited signal compared to how engineers actually used AI tools.
A small group of AI power users emerged early. Studying usage patterns helped Uber identify engineers who were engaging deeply with AI and generating outsized results.

Correlation is not causation

High AI usage does not automatically prove AI caused higher productivity. The most productive engineers are often the first to adopt new tools.
Rigorous analysis matters when making investment decisions. Uber used causal methods to better understand the true impact of AI-assisted development.

Why measuring AI ROI is difficult

Developer years saved sounded compelling but failed as an ROI metric. The approach created anxiety around replacement, required constant recalibration, and did not answer the business questions leadership cared about most.
Business leaders ultimately care about outcomes. Time saved is useful context, but value creation, customer impact, and business results matter more.
PRs measure activity, features measure value
Agentic AI exposes the limitations of activity-based metrics. A single agent task can generate many pull requests without creating meaningful customer value.
Feature velocity became Uber’s new North Star. The goal shifted from measuring engineering output to measuring whether valuable capabilities were actually being delivered.

Building an AI-native measurement framework

Feature velocity works alongside supporting metrics. Flow efficiency, quality, and capability expansion help create a more complete picture of AI’s impact.
PR classification provides important context. Understanding the type and complexity of work helps distinguish meaningful progress from routine maintenance and toil.
The future belongs to outcome-based metrics
The most durable metrics are tied to business outcomes rather than engineering activity. As AI becomes more autonomous, output alone becomes a less reliable signal.
Many important questions remain unanswered. Organizations still need better ways to measure judgment, autonomy, technical debt, and the value created by increasingly agent-driven software development.