The Impact of Developer Experience

A study from Microsoft, GitHub, and DX that quantifies the impact of an improved developer experience.

This is the latest issue of my newsletter. Each week I cover research, opinion, or practice in the field of developer productivity and experience. 

---

This is a special issue to announce a new paper I co-authored with Nicole Forsgren, Eirini Kalliamvakou, Michaela Greiler, Brian Houck, and Margaret-Anne Storey. 

Earlier this year, we published a paper about a new approach to measuring and improving developer productivity, focusing on the developer experience. Our new paper studies the impact of developer experience at three levels: individual, team, and organization.

Read my summary of the paper below. You can also join the discussion on LinkedIn — I’d love to hear from you.

My summary of the paper

Software engineers are well-aware of how their work environment can help or hinder their ability to do great work. There’s a clear difference, for example, between developing, testing, and deploying code in an environment that’s optimized for those activities versus one that is not. When leaders measure and understand the developer experience, they can gain valuable insights into the challenges that are getting in developers’ way. 

While this is intuitive, there’s a need for empirical research on the impacts of developer experience. Such studies can give organizations a better grasp on what they should expect from these initiatives.

Methodology

The design of this study was based on work design theory (WDT), which is a framework that focuses on how the design of work impacts various outcomes. In this study, the work characteristics that we focused on were the three dimensions of developer experience — flow state, feedback loops, and cognitive load. We looked at how these dimensions impact outcomes at the developer, team, and organizational level. 

This study was conducted with a survey of professional software developers. Multiple different techniques were used for analyzing the data, including: 

• A partial least squares analysis (PLS), which is a statistical method for predictive modeling and exploring relationships between variables. This determined whether each of the three dimensions — flow state, feedback loops, and low cognitive load — positively impact outcomes at the developer, team, or organizational level.

• A likelihood analysis, which was used to analyze the outcomes businesses can expect from improving specific developer experience factors. 

• An importance-performance matrix analysis (IPMA), which is a technique for identifying items that have a high impact and performance relative to the dependent variable under consideration. In other words, we looked at developer experience factors 

Here are the key findings:  

What businesses can expect from improving developer experience 

The likelihood analysis showed us what outcomes businesses can expect from improving different developer experience factors. 

Flow state

Developers who had a significant amount of time carved out for deep work felt 50% more productive, compared to those without dedicated time. In other words, dedicating time to deep work is a practice that pays high dividends in terms of allowing developers to be truly productive. This includes carving out time to focus and ensuring that their environment supports uninterrupted work.

Developers who find their work engaging feel they are 30% more productive, compared to those who found their work boring. To improve engagement, leaders may consider rethinking the distribution of tasks between individuals in a team, or teams within an organization. For example, are the same developers continually working on less desirable projects and tasks which could lead to burnout? Are teams tasked regularly with activities they find boring or divorced from the company’s mission and customers? 

Cognitive load

Developers who report a high degree of understanding the code they work with feel 42% more productive than those who report low to no understanding. It’s an all too familiar pattern when teams need to move fast and overlook making their code clear, simple, or well documented. While that is sometimes necessary it can really hinder the team’s productivity long-term, so tooling and conventions that help code be understandable within and across teams can future-proof productivity. 

Developers who find their tools and work processes intuitive and easy to use feel they are  50% more innovative, compared to those with opaque or hard-to-understand processes. Unintuitive tools and processes can be both a time sink and a source of frustration — in either case, a severe hindrance to individuals’ and teams’ creativity. 

Feedback loops 

Developers who report fast code review turnaround times feel 20% more innovative compared to developers who report slow turnaround times. Code reviews that are completed quickly allow developers and teams to move to their next idea quickly, creating the ground for coming up with the next great thing. 

Teams that provide quick responses to developers' questions report 50% less tech debt than teams where responses are slow. It pays off to document repeated developer questions and/or put tooling in place so that developers can easily and quickly navigate to the response they need and integrate good coding practices and solutions as they write their code, creating less tech debt.

Factors that have an outsized impact 

The importance-performance matrix analysis identified items that have a high impact and performance relative to outcomes at two levels: the developer and the organizational level. We identified:

• To improve developer outcomes, deep work and engaging work have the biggest potential impact. 

• To improve organizational outcomes, several items have the potential for big impact: deep work, engaging work, intuitive processes, and intuitive developer tools.

These provide clues into what leaders can focus on if they want to improve outcomes at the developer or organizational level. For example, leaders looking to improve developer outcomes like productivity, learning, and creativity, may consider thinking about ways that they can provide opportunities for deep work. This can include encouraging focus time for both individual developers and coordinated focus time among teams, like days with few or no meetings. They could also look for opportunities to create engaging work and for learning, such as hack days. Additionally, the analysis shows that deep and engaging work provides outsized support for organizational outcomes like innovation, retention, profitability and broader organizational goals. 

Other ways to improve organizational outcomes could include looking for ways to streamline and clarify processes, or provide access to intuitive, easy-to-use developer tools. 

How to apply the findings

There are five important steps we outline in the paper for driving developer experience improvements: 

1. Get data on the current developer experience. As a first step, you have to understand what the developer experience is like at your organization. For organizations that are early on their DevEx journey, this means collecting new data to reveal their current biggest pain points, as well as knowing their current abilities to make changes. (You can use or adapt this survey guide or use a dedicated solution like DX.) If this is your first time collecting data about DevEx, this becomes your baseline. If you have already been doing some DevEx work and collecting data, you can integrate this data and update your metrics.  

2. Set goals based on your DevEx data. Use your DevEx data to inform your goals and investments. These can be informed by current business priorities, your DevEx data, and what you learned in this paper. 

3. Set your goals up for success. Once you’ve reviewed your data and set your goals, set north star metrics (or OKRs) to help communicate the business impact of your work. 

4. Share progress and validate investments. Share results with developers, as well as DevEx and business leaders to evaluate and discuss the value of your investments. Reflect on what investments delivered impact, as well as what surprised you and what you learned. This practice can increase confidence from others that your investments are making an impact.

5. Repeat. In general you will want to repeat this process of data collection, tracking progress, and setting goals every 3-6 months. 

Final thoughts

The most important contribution of this study is the evidence it provides: improving developer experience creates positive outcomes for individuals, teams, and organizations. The findings from this paper may serve as helpful evidence for leaders advocating for investments in developer experience at their company. 

---

That’s it for this week. If you are interested in beginning to measure developer experience in your organization, download our survey template for help getting started. 

-Abi