Measuring Engineering Productivity

Google's framework for selecting metrics.

This is the latest issue of my newsletter. Each week I cover the latest research and perspectives on developer productivity.

This week I read Measuring Engineering Productivity by engineering productivity researcher Ciera Jaspan. This is a chapter from the book Software Engineering at Google that describes the rationale for measuring productivity, as well as Google’s framework for selecting metrics. 

---

Update: I interviewed Ciera and Collin from Google’s Engineering Productivity Research team. We cover the paper Ciera wrote, as well as other findings from their research on engineering productivity. Listen to it here:

---

My summary of the paper

Why measure engineering productivity at all? 

Before choosing metrics, first understand your reasons for measuring. Jaspan opens the chapter with a simple explanation: one reason for measuring is to scale the scope of the business by making engineering more productive, rather than by hiring more engineers. The goal of metrics is then to identify inefficiencies in engineering processes and decide which areas to invest in. 

The rest of the chapter outlines a process for selecting meaningful metrics.

Selecting meaningful metrics

Google uses the Goals/Signals/Metrics (GSM) framework to guide metric creation. 

1. Start by defining the goal you’re trying to accomplish. Your goal should be phrased in terms of what you want to understand at a high level and should not contain references to specific ways to measure it. 

Aim to cover different parts of engineering productivity with goals. Jaspan explains that Google divides productivity into five components, which they refer to with the mnemonic “QUANTS”: Quality of the code, Attention from engineers, Intellectual complexity, Tempo and velocity, and Satisfaction. Each component has an associated goal. (See the table below for examples.) 

2. Define signals. A signal is how we know if we’ve accomplished the goal. Every goal should have one or more signal. Not all signals are measurable.  

3. Select metrics. These are proxies for the signal. Some signals might have multiple metrics. 

Selecting metrics also includes deciding how to measure them. In the example below, surveys and logs data are used. Jaspan writes, “Qualitative metrics are metrics, too! Consider having a survey mechanism for tracking longitudinal metrics about engineers’ beliefs. Qualitative metrics should also align with the quantitative metrics; if they do not, it is likely the quantitative metrics that are incorrect.” 

Following the GSM framework is a great way to clarify the goals for why you are measuring your software process and how it will actually be measured. 

Consider whether the metrics are worth capturing 

Jaspan also shares the questions Google uses to help determine whether these engineering productivity metrics are worth measuring: 

• If the data supports your expected result, what action will be taken? If no action will be taken, there is no point in capturing that metric. 

• If we get a negative result, will appropriate action be taken? If there are other inputs that will override a negative result, it might not be worth measuring. 

• Who is going to decide to take action on the result, and when would they do it? “The goal of measuring our software process is to help people make business decisions. It’s important to understand who that individual is, including what form of data convinces them.” 

Final thoughts

I recently read a post by Will Larson that articulates the common reasons for measuring engineering. Of the reasons Will discusses, today’s issue is most relevant for leaders looking to “measure to optimize,” or in other words measure to understand what needs to be done to improve productivity. 

The Goals/Signals/Metrics framework is an important tool that can help engineering teams get clear on what they want to understand and how they’ll use that information — before going down the rabbit hole with metrics. 

---

Curious about how industry leaders measure engineering productivity?

Dive into DX’s report which unveils the approaches taken by top-tier companies such as LinkedIn, Pfizer, and Peloton for measuring developer productivity.

---

If you’re finding this newsletter valuable, consider sharing it with friends, or subscribing if you haven’t already.

-Abi