SE Research in 2022

The key ideas shared this year and the issues you enjoyed most.

This was my first year writing SE Research — quite a year to be writing about developer productivity. After booming for the best part of two decades, a number of factors coincided to transform the tech landscape and caused many companies to shift their attention towards being more productive. 

My hope in producing SE Research is to help you discover research-based and reliable ways of improving developer productivity. Thank you for being part of this newsletter — writing this has become one of my favorite weekly activities because there is so much research in this space to learn from. In the coming year I’ll experiment with new formats, but paper summaries will continue to be core to this newsletter. 

For now, here’s a summary of the key ideas covered in 2022. 

Summary

What is developer productivity and how do we measure it? 

• In theory, productivity attempts to measure the value created by investing in an organization. However, our industry isn’t aligned on what this really means: developers and managers define productivity differently,¹ and developers have differing views on what it means to be productive amongst each other as well.² 

• We do know that productivity is multi-dimensional can’t be boiled down to one metric.³ 

• Measurement must take into account personal perception. That means measuring whether a developer feels productive and whether they perceive a tool or process to be effective.³

• There are different frameworks for understanding productivity, including the SPACE framework (Satisfaction and well-being, Performance, Activity, Collaboration and communication, and Efficiency and flow),³ and Google’s QUANTS (Quality of the code, Attention from engineers, Intellectual complexity, Tempo and velocity, and Satisfaction).⁴ These frameworks show you where to look, and they also make it clear that organizations should select metrics that cover all parts of productivity. Before choosing a metric, understand what you intend to achieve from measuring and who will be taking action on the data. 

• Satisfaction is consistently included in these frameworks as satisfaction and productivity are positively and bi-directionally correlated.⁵ Unhappiness can also cause lower mental performance and lower focus, and can lead to lower quality code.⁶

• Organizations can capture metrics using survey-based data or system-based data.⁷ Survey-based measures can provide a holistic view of the value stream through continuous or periodic data collection. System-based data provides a continuous view of the value stream, although limited to what can feasibly be collected and correlated.

How do we influence productivity? 

• One path to improving productivity is to reduce waste.⁸ As an industry, we often use terms like friction, cognitive load, toil, and undifferentiated lifting to refer to waste. There are many types of waste, including waiting/multitasking, unnecessarily complex solutions, rework, and mismanaging the backlog. 

• Different studies have looked at what factors impact productivity, such as:

  • Technical debt. Almost a quarter of developers’ working time is reported as wasted due to technical debt. This wasted time is most commonly spent performing additional testing.⁹ To address technical debt, consider one of the following strategies: clear end-to-end ownership, empowered teams to tackle tech debt, or lightweight checks and balances.¹⁰

  • Code quality. In a Google-based study, satisfaction with code quality had the strongest causal relationship with perceived productivity.¹¹ Note that poor code quality is also a type of technical debt.¹⁰ One strategy for improving code quality is to focus on code ownership.¹²

  • Flaky tests. Flaky tests hinder CI and lead to a loss of productivity. Developers who experience flaky tests more frequently are also more likely to ignore potential genuine test failures.¹³

  • Happiness. Because satisfaction and productivity are linked, it is also important to understanding what makes developers unhappy. Some of the top causes of unhappiness include being stuck in problem solving, bad code quality and coding practice, feeling inadequate with work, and bad decision making.¹⁴

• Because every company is different, metrics can be used to understand what is impacting productivity most within an organization. 

References

1. Storey, M., Houck, B., Zimmerman, T. How Developers and Managers Define and Trade Productivity for Quality.

2. Meyer, A., Murphy, G., Fritz, T., Zimmerman, T. Developers’ Diverging Perceptions of Productivity.

3. Forsgren, N., Storey, M., Madilla, C., Zimmerman, T., Houck, B., Butler, J. The SPACE of Developer Productivity: There’s more to it than you think. 

4. Jaspan, C. Measuring Engineering Productivity. (Note: I’ll summarize this article in a future newsletter issue.)

5. Storey, M., Zimmerman, T., Bird, C., Czerwonka, J., Murphy, B., Kalliamvakou, E. Towards a Theory of Software Developer Job Satisfaction and Perceived Productivity.

6. Graziotin, D., Fagerholm, F., Wang, X., Abrahamsson, P. What Happens When Software Developers Are Unhappy?

7. Forsgren, N., Kersten, M. DevOps Metrics: Your Biggest Mistake Might Be Collecting the Wrong Data. 

8. Sedano, T., Ralph, P., Péraire, C., Software Development Waste.

9. Baker, T., Martini, A., Bosch, J. Technical Debt Cripples Productivity. 

10. Cochran, T., Nygard, C., Collins, K., Govande, K., Kick, R., Smith, R. Accumulation of Tech Debt.

11. Cheng, L., Murphy-Hill, E., Canning, M., Jaspan, C., Green, C., Knight, A., Zhang, N., Kammer, E. What Improves Developer Productivity at Google? Code Quality. 

12. Greiler, M., Herzig, K., Czerwonka, J. Code Ownership and Software Quality: A Replication Study. 

13. Parry, O., Kapfhammer, G., Hilton, M., McMinn, P. The Developer Experience of Flaky Tests.

14. Graziotin, D., Fagerholm, F., Wang, X., Abrahammson, P. On the Happiness of Software Developers. 

TL;DR for the most popular issues

1. What Distinguishes Great Software Engineers?

The top five attributes that distinguish great engineers: 

• Being a competent coder,

• Maximizing the current value of their work,

• Practicing informed decision making,

• Enabling others to make decisions, and

• Continuous learning.

2. Developer Productivity at Google

• Perceived productivity is causally related to satisfaction with code quality. 

• Perceived productivity is also causally related to satisfaction with technical debt. 

• Engineers who reported their tools and infrastructure were not innovative were more likely to report lower productivity. 

• Engineers hindered by shifting project priorities were likely to report lower productivity.

3. Addressing Tech Debt

• There are many types of technical debt, including code quality, testing, coupling, unused features, out of date libraries, etc. Gather feedback from developers experiencing the technical debt to understand what the problem really is. 

• To identify whether technical debt is becoming a bottleneck, look at signals such as value lead time, impact to the end user (latency in the systems, customer onboarding time), developer satisfaction, ability to onboard new developers, and degradation in non-functional measures. 

• Strategies such as clear end-to-end ownership, empowered teams to tackle and monitor technical debt, and lightweight process (automated checks or architectural peer review), will help address technical debt. 

---

That’s it for this week! Share your thoughts and feedback anytime by replying to this email.  

If this email was forwarded to you, subscribe here so you don’t miss any future posts: 

See you in 2023!

-Abi