Build Times and Developer Productivity
Even modest improvements to build times are helpful.
This is the latest issue of my newsletter. Each week I cover the latest research and perspectives on developer productivity.
This week I read Build Latency, Predictability, and Developer Productivity, a paper by Google researchers Ciera Jaspan and Collin Green. This paper is part of their ongoing series, Developer Productivity for Humans, which has previously covered topics such as the challenges of measuring productivity, productivity in remote teams, and managing tech debt.
This paper specifically shares findings from a Google-based study on the benefits of improving build times. For leaders deciding whether to invest in improving their organization’s build processes, this study provides evidence that even moderate improvements to build latency lead to productivity improvements. It also demonstrates the importance of nudging developers’ behavior when making changes to development tools, in order to achieve the expected productivity improvements.
My summary of the paper
One of the goals of the Developer Productivity for Humans series is to emphasize the human aspects of software engineering and encourage leaders to consider these aspects when attempting to improve productivity. In line with this goal, the study discussed in this paper explores the issue of improving build times with a focus on understanding the human elements that influence the relationship between build time improvements and productivity.
“On the surface, build latency is a purely technical problem. But humans experience and respond to it in interesting ways: forming expectations, making choices, and organizing work around build latency and similar factors.”
The researchers focused on three questions for this study:
How fast do builds need to be for developers to stay on task and be productive?
Are there changes that can be made to build systems to improve productivity besides just “making builds faster”?
How much of a productivity improvement can we reasonably expect by reducing build times?
Here’s what they found:
There’s no magic number for how fast builds need to be
While waiting for builds to complete, developers will often work on another project, check email, get a coffee, or go get lunch. The researchers wanted to understand whether there is a specific threshold for build times where developers are more likely to stay on task.
To investigate this, the researchers tracked developer actions within their tools to understand how long it takes for developers to go off task during a build, and how long it takes for them to return to their task once the build completes.
The authors initially thought that they would find a “magic number,” such as “if the build takes less than 𝑥 seconds, developers are more likely to stay on task.” However, they did not discover a specific magic number. Instead, they found that every improvement to build latency contributes to developers staying on task and returning to tasks more quickly.
“While it’s disappointing to not have an ideal target number, it’s also an opportunity. Every change to build latency can increase the likelihood of developers staying on task, although if there are longer build latencies, one would need a proportionally larger change to see an impact.”
Providing developers with estimated build times can improve productivity
To understand how developers think about their builds and what task they will focus on while waiting for builds to complete, the researchers ran an experience sampling study for two weeks. Developers were prompted to respond to a survey when starting a build. The survey asked developers how long they expected their build to take, and what activities they engaged in while waiting.
This study revealed that developers decide whether to go off task and what to work on based on how much time they think the build will take, not how much time they actually have. For example, if a developer thinks a build will take 60 minutes, they might go get lunch. If they think it will take a few minutes, they might go do a short code review.
However, developers regularly overestimate or underestimate the time they think a build will take, which causes disruptions in their workflow. Despite builds feeling disruptive, the issue here is the developers’ expectation of how long a build will take. If build latencies were consistent, or if the build system informed the developer about the expected time to complete, developers could make better decisions about how to use their time as they wait.
“Reducing build latency is not our only lever to improve productivity here; we can also improve developers’ ability to make decisions around build latency as it is.”
Even modest build latency improvements are helpful
The researchers also sought to identify the real-world benefits of reducing build times. They were inspired by a team at Google that is responsible for build machines: the team was exploring whether to use a more expensive system, and wanted to understand whether the investment was worth it.
To evaluate this, the researchers performed a blind experiment: 15% of developers were selected to have their builds supported by upgraded machines (the experiment group). This group experienced only slightly faster builds, with the median developer having builds improve by just a few seconds. 85% of developers had their builds supported by existing machines (the control group).
Over three months, the researchers tracked whether the experiment group saw changes across the following outcome measures:
Self-reported productivity, self-reported velocity, and satisfaction with build latency as measured through Google’s quarterly surveys.
The number of times developers ran a build each week and the number of lines of code they submitted, measured through logs.
The median wall-clock coding time and median active coding time on the change lists the developer submitted. The wall-clock time is the time from the first edit to the change list being submitted, while the active time is the “fingers on keyboard” time, including time spent in the integrated development environment and also time spent looking up information.
In the study, the experiment group showed slightly higher self-reported productivity and slightly higher self-reported velocity. Satisfaction with build latency also increased in this group. All increases were statistically significant increases over the control group.
Evaluating the behavioral metrics, researchers saw no changes to either of the groups for the first two months of running the study. In the third group, however, they saw a slight improvement for the experiment group: that group, on average, ran one more build a week and submitted 24 more lines of code per week. Additionally, the developers in the experiment group were faster to complete small and medium changes (11% faster active time and 14% faster wall-clock time). The delayed change in behavior was a surprise, so the researchers extended the experiment for two additional months. The behavior was sustained.
The delayed change in behavior surprised the researchers, but they attributed it to developers adapting to the faster build latencies.
Overall, an incremental change in build latency has several effects on developers’ behavior and perception (albeit at a delay) that can and should be considered benefits of faster builds.
Final thoughts
This paper is helpful in several ways:
1. It gives leaders evidence that even modest improvements to build times are helpful. Developers with upgraded machines were able to fit in one more build a week, which meant just a few more lines of code submitted each week. They showed slightly higher self-reported productivity, velocity, and satisfaction with build tools. Additionally, their slightly faster iteration time resulted in overall velocity improvements for smaller changes.
2. It surfaced the insight that developers need to be aware of the faster process in order to optimize around it. If they don’t notice the change, they may not update the way they use the systems.
3. The researchers also suggested that, even if you can’t improve build times, you can improve developer productivity by making these processes more predictable. In the case of builds, we can improve predictability by either making builds take similar lengths of time or by informing developers of how long they are expected to take.
“Build latency is one narrow example, but a general approach to improving productivity falls out of this discussion: if you can’t make a process faster or easier, at least make it more predictable so that developers can optimize around it.”
That’s it for this week! Share your thoughts and feedback anytime by replying to this email.
If this email was forwarded to you, subscribe here so you don’t miss any future posts:
-Abi