Why it’s Difficult to Measure Developer Productivity

Software engineering is creative and there are outside forces that influence productivity.

This is the latest issue of my newsletter. Each week I cover the latest research and perspectives on developer productivity.

---

This week I read A Human-Centered Approach to Developer Productivity, the first article in a new series on developer productivity by Ciera Jaspan and Collin Green. Here, the authors explore the reasons why it is difficult to measure developer productivity, and set the stage for a more human-centered approach.

My summary of the article

Engineering leaders frequently want to understand whether productivity is up, down, or stable. They want “up” and “down” to map to “good” and “bad.” Unfortunately the task is not so simple. 

Why is it so difficult to measure developer productivity? The authors suggest two reasons: 

Engineers are humans; this is not a robot-powered factory

One problem that makes the measurement challenge difficult is that there are many outside forces that influence how productive a human will be at a task. An outside force could be the complexity of the work (and whether it is necessarily that complex), the interactions with others to get the job done, or the organizational design. There are also factors that specifically affect developers, including flaky tests, build speeds, and technical debt. 

Software engineering is not predictable

Software engineering is a creative endeavor. It is not about the production of uniform, interchangeable outputs. Therefore: 

• Throughput cannot be conflated with productivity.

• We cannot assume the product that was built is the right product. It may have performance issues or it may not scale; there are quality, reliability, and practicality aspects of creating software. 

• We cannot assume that when work doesn’t result in output, it has no value. Problem solving is a core part of software engineering and it is hard to measure. 

Attempts to quantify work productivity by borrowing methods from operating machinery are not suited to software engineering.

Author recommendations 

Measuring productivity is complex, however there are still important reasons for trying. The authors recommend leaders measure productivity “in a holistic and multifaceted way, not in a reductionist, unidimensional way.” Select more than one metric, consider short- and long-term productivity, and remind stakeholders that they are measuring a fundamentally human system. 

Final thoughts

Previously we learned about Google’s Goals/Signals/Metrics framework for selecting metrics. This article may serve as a helpful reminder for those approaching the last step, choosing metrics: consider the different aspects affecting developers’ jobs and collect a holistic set of metrics.

---

That’s it for this week! Share your thoughts and feedback anytime by replying to this email.  

If this email was forwarded to you, subscribe here so you don’t miss any future posts: 

-Abi