A New Approach To Measuring Developer Productivity

From the creators of DX, DORA, and SPACE.

Abi Noda

May 19, 2023

This is the latest issue of my newsletter. Each week I cover research, opinion, or practice in the field of developer productivity and experience.

This is a special issue to announce my just-published paper which defines a new approach to measuring developer productivity.

The premise for this paper is that today, many organizations attempt to measure productivity through metrics like lead time or pull requests completed. These methods, however, fail to capture the full story around productivity or drive actual improvements.

At DX, our team includes the world’s foremost experts on developer productivity including Margaret-Anne Storey (author of SPACE) and Nicole Forsgren (creator of DORA). Over the past several years, we’ve conducted rigorous research on developer experience and learned a lot from our experience applying our research at DX.

Recently, we arrived at something we finally felt we could share – a new framework for measuring productivity which centers on the developer experience. By capturing qualitative insights from developers, and augmenting these insights with data from our systems, we can unlock clear signals on productivity and how to improve.

Continue reading on for a summary of the paper. For a deeper dive, check out the full paper as well as our interview on The Pragmatic Engineer.

My summary of the paper

Leaders have long sought to improve the productivity of their developers, however knowing how to measure or define productivity has remained elusive. Past approaches, such as measuring the output of developers or the time it takes them to complete tasks, have failed to account for the complex and diverse activities that developers perform.

Therefore, the question has remained: what should leaders measure and focus on to improve the productivity of their developers? The solution we recommend: improve productivity by measuring and focusing on developer experience.

What is developer experience (DevEx)?

Developer experience encompasses how developers “feel about, think about, and value their work”. In other words, developer experience is about the lived experience of developers and the points of friction they encounter in their everyday work.

For example, interruptions, unrealistic deadlines, and friction in development tools negatively affect the developer experience. Conversely, having clear tasks, well-organized code, and pain-free releases improve the developer experience. Developer experience not only affects developer productivity, but also satisfaction, engagement, employee retention, and business performance.

Our DevEx framework distills the factors affecting developer experience into three dimensions: feedback loops, cognitive load, and flow state. Leaders can surface opportunities to improve productivity by measuring and focusing on metrics within the three dimensions.

1. Feedback loops refer to the speed and quality of responses to actions performed. Fast feedback loops allow developers to complete their work quickly with minimal friction. Slow feedback loops, by contrast, interrupt the development process, leading to frustration and delays as developers wait or decide to switch tasks. Organizations should shorten feedback loops by identifying areas in which development tools can be accelerated (e.g. build and test processes, development environment setup), or human hand-off processes improved.

2. Cognitive load encompasses the amount of mental processing required for a developer to perform a task. When cognitive load remains high as a result of problems such as poorly documented code or systems, developers need to devote extra time and effort to complete tasks and avoid mistakes. To improve developer experience, teams and organizations should aim to reduce cognitive load by finding ways to eliminate unnecessary hurdles in the development process.

3. Flow state is a mental state in which a person performing an activity is fully immersed in a feeling of energized focus, full involvement, and enjoyment. Frequent experiences of flow state at work lead to higher productivity, innovation, and employee development. Similarly, studies have shown that developers who enjoy their work perform better and produce higher quality products. Teams and organizations should focus on creating the optimal conditions for flow state.

What to measure

The three dimensions described above provide a foundation for designing useful measurements. We recommend selecting topics to measure across the three dimensions (e.g., automated tests, codebase complexity, and time for deep work), measuring both perceptions and workflows, and including KPIs or “North Star Metrics” to track overall outcomes.

How to measure

Developer experience must be measured by capturing developers’ perceptions as well as their workflows within their tools and processes. We recommend collecting data from both people and systems in order to gain full visibility into their software delivery processes. However, we explicitly recommend starting with surveys as a practical way to capture holistic measures quickly.

Here are some recommendations for organizations conducting surveys:

Design surveys carefully. Poorly designed survey questions lead to inaccurate and unreliable results. At minimum, survey questions should be based on well-defined constructs, and rigorously tested in interviews for consistent interpretation. When possible, consult experts with significant experience in survey development.
Break down results by team and persona. A common mistake made by organizational leaders is to focus on company-wide results instead of data broken-down by team and persona (e.g. role, tenure, seniority). However, developer experience is highly contextual and can differ radically across teams or roles. Focusing only on aggregate results can lead to overlooking problems that affect small but important populations within the company, such as mobile developers.
Compare results against benchmarks. Comparative analysis can help contextualize data and help drive action. For example, developer sentiment toward tech debt is commonly negative, making it difficult to identify problems or gauge their scale. But teams with lower sentiment scores than their peers and organizations with lower scores than their industry competitors flag notable opportunities for improvement.
Mix in transactional surveys. In addition to periodic surveys, organizations can use transactional surveys to collect feedback based on specific touchpoints. For example, Platform teams can use transactional surveys to prompt developers for feedback when a specific error occurs during the installation of a CLI tool. Transactional surveys provide a continuous stream of feedback and can generate higher quality responses due to the timeliness of their posed questions.
Watch out for survey fatigue. Many organizations struggle to sustain high participation rates in surveys over time. A lack of follow-up action commonly causes developers to feel that repeatedly responding to surveys is not a worthwhile exercise. It is therefore critical that leaders and teams follow up on surveys. While a quarterly or semi-annual survey cadence is optimal for most organizations, for some, benefits may accrue from administering surveys more frequently.

That’s it for this week! If this email was forwarded to you, subscribe here so you don’t miss any future posts:

Share your thoughts and feedback anytime by replying to this email.

-Abi