Understanding the Experience of Code Review

Common obstacles in the code review process.

This is the latest issue of my newsletter. Each week I cover the latest research and perspectives on developer productivity.

---

This week I read Understanding the Experience of Code Review: Misalignments, Attention, and Units of Analysis, a paper by researchers at Lund University in Sweden that identifies common problems that frustrate developers in the code review process. Leaders may find this paper helpful to spark discussions and identify ways to improve the code review process within their teams. 

My summary of the paper

Modern code review (which is defined as “tool-supported” code review) has been found to have a positive impact on software quality. However, the process requires a lot of effort and is not always effective. While previous work has explored different reasons why the code review process may at times be ineffective, the researchers here sought to understand the experience of code review from the developers’ perspective. 

The researchers conducted a mixed-methods study at two multinational companies. They started with a series of semi-structured interviews to learn about the developers’ experience of code review, and then analyzed the themes that surfaced. Then, they conducted a follow-up survey to understand the prevalence of certain obstacles in the code review process. 

Here’s what the study found:

Misalignments in the code review process 

After conducting interviews, the researchers categorized the different types of misalignments they surfaced into two high-level themes: misalignments between the tool and the review task, and misalignments from the process. They used a follow-up survey to better understand the prevalence of these problems.

Here are some of the common misalignments the researchers identified in the code review process: 

Process-related misalignments refers to situations where processes don’t fully support the task of a reviewer. 

• Process-completion was the top problem theme reported by developers. This refers to the problem where the expected time to review a task and the actual time for completing that task are misaligned. Both authors and reviewers feel frustrated when reviews take longer than they should. 

• Another commonly reported problem: there is confusion about who is responsible for following up on comments within the review. In interviews, the researchers found that some developers described the responsibility as being up to the author, whereas others described it as being up to the reviewer. This finding connects to previous work which identified “organization of work” as a top challenge in the review process. 

Tool-related misalignments refer to situations where the code review tool doesn’t fully support the intended task of the reviewer. On average, developers experience tool-task misalignment every fourth code review. 

• In the study, researchers identified that the most common problem within tool-task misalignment is the problem of not having the information needed within a code review tool to complete a review. This causes the developer to leave the review environment and pull the change into their editing environment to find the information needed. This finding relates to previous work which has identified “code understanding” as a challenge in the review process. 

• Another common problem that developers experience: the conversation needed for the review task is not supported by the code review tool. Sometimes the best path for moving a code review forward is to have a white board session or otherwise meet and talk through it. Code review tools typically do not support or facilitate these types of conversations. This finding relates to previous work which has identified offline discussions as a coping strategy to resolve communication issues in the review process.

Final thoughts

Most advice for leaders looking to improve their team’s code review process centers around improving the reviews themselves, by way of checklists or tips for reviewers to give feedback more appropriately. These are all useful, but I appreciate that this paper provides a different perspective. It looks at problems with the processes and tools involved that are either hampering—or at least not fully supporting—the tasks involved in code review. Leaders may use this paper as a starting point to learn whether these problems exist in their teams’ review process. 

---

That’s it for this week! If you’re finding this newsletter valuable, share it with a friend: 

If you’re interested in more research, send me a connection request or message on LinkedIn with the note “research” and I’ll share a downloadable bundle of my top 10 favorite papers of this year.