What Makes a Code Change Easier to Review?

Change description and change size are key components of reviewability.

This is the latest issue of my newsletter. Each week I cover the latest research and perspectives on developer productivity.

Join us next week for a live discussion on the developer portals market with Chris Westerhold from Thoughtworks. Sign up here.

---

This week I read What Makes a Code Change Easier to Review, a study that examined the factors that lead to better reviewability of code changes. When code changes are easy to review, everyone benefits: authors receive feedback faster, and reviewers are able to stay on track. Prior to this study, however, researchers had not yet explored how to make code changes easier to review.

I recently shared another paper that focused on improving the code review process by addressing stale pull requests. This study provides a different technique for solving the same problem: by focusing on improving reviewability. (Note: the authors refer to this strategy as “reducing reviewer overhead.”)

My summary of the paper

The goal for this study was to define what makes a code change easier or harder to review. The study was carried out in multiple stages:

• Initially, the researchers reviewed existing literature to gain a foundational understanding of the aspects that reviewers look for in code changes. 

• Then they sought to learn what reviewers associate with code change reviewability. They did this by first interviewing developers, where developers were asked to review a real-world code change and then answer questions about what made the change more or less straightforward to review. Then, they used a tool that automatically asks for feedback from reviewers after reviewing a code change (see image below). These in-the-moment surveys asked reviewers to rate the overall reviewability of a code change, as well as to share what aspects of the code change contribute to its reviewability. 

From this, researchers identified the following aspects to have the biggest effect on reviewability:

1. The quality of the change description. Change description was the most frequently reported factor that developers feel affect reviewability. Most of the developers in the study stressed that it’s essential to include the motivation behind a specific change, along with what is being changed, in the description. 

2. The size of the change. The consensus among developers is that the smaller the change, the more reviewable it is. Previous research has supported this: smaller changes have been found to have a higher chance of acceptance. 

3. Commit history. This study supported the findings from prior research that the larger the number of commits in a code change, the lower the chance of its acceptance. Additionally, the message that summarizes the change plays a role how reviewable the PR is. Developers said that commit messages should be self-explanatory and focused, and cannot have non-descriptive idioms such as “Fix typo” or “Update”. 

The researchers also explored whether the three identified components of reviewability (change description, PR size, and commit history) impact the chances a code change gets accepted. Reviewers largely agreed that they would talk with the developer who created the change and give them a chance to fix the issues before accepting it. In other words, reviewability aspects are not the principle drivers behind the overall decision made on whether to accept the code change. In general, these are easily fixable and will cause a reviewer to only temporarily reject a change. 

Final thoughts

I appreciate that this study investigated a specific part of the code review process, while also being clear about the potential impact that improving this area can have. Improving reviewability is not transformative, but it is something leaders can focus on to reduce reviewer overhead and the amount of times that PRs are temporarily rejected. Additionally, this paper provides empirical evidence on the actions developers can take to make code changes easier to review. This paper may be useful for leaders looking for a non-technical approach to improve the code review process. 

---

That’s it for this week. If you know someone who might like this issue, consider sharing it with them:

-Abi