How Google Measures and Manages Tech Debt

A behind-the-scenes look at how Google approaches technical debt.

This is the latest issue of my newsletter. Each week I cover the latest research and perspectives on developer productivity.

---

This week I read Defining, Measuring, and Managing Technical Debt, a paper by Google researchers Ciera Jaspan and Collin Green. This is part of the "Developer Productivity for Humans Series," which also includes papers on the challenges of measuring developer productivity and lessons learned about productivity during the pandemic.

Over the past several years, Google made a deliberate effort to better  manage technical debt. Of course, managing technical debt is difficult without being able to define or measure it. This paper describes Google’s approach to defining and measuring technical debt, along with their efforts to manage it. 

My summary of the paper 

Google’s effort to better manage technical debt was motivated by the results of their quarterly engineering satisfaction survey. This survey asks engineers to indicate the extent to which they are hindered by unnecessary complexity and technical debt. For several years, the organization has seen that the percentage of engineers who feel hindered is substantial. This sparked the effort to investigate what technical debt is, how it might be measured, and how it could be better managed. 

Defining technical debt

The researchers wanted to understand technical debt as engineers chose to use the term, rather than defining it for them. They started by interviewing subject matter experts at the company, asking about the underlying causes of the technical debt they encountered as well as what mitigations would be appropriate to fix the debt. These insights were then incorporated into the quarterly engineering survey by allowing engineers to select multiple root causes and mitigations for technical debt. The researchers performed a factor analysis to discover patterns in the responses, and then reran the survey the next quarter with the refined response options. 

The researchers identified 10 categories of technical debt, listed in order of the most to least frequently reported as a hindrance to productivity: 

1. Migration is needed or in progress: This may be motivated by the need to scale, due to mandates, to reduce dependencies, or to avoid deprecated technology. 

2. Documentation on project and application programming interfaces (APIs): Information on how your project works is hard to find, missing or incomplete, or may include documentation on APIs or inherited code. 

3. Testing: Poor test quality or coverage, such as missing tests or poor test data, results in fragility, flaky tests, or lots of rollbacks. 

4. Code quality: Product architecture or code within a project was not well designed. It may have been rushed or a prototype/demo. 

5. Dead and/or abandoned code: Code/features/projects were replaced or superseded but not removed. 

6. Code degradation: The code base has degraded or not kept up with changing standards over time. The code may be in maintenance mode, in need of refactoring or updates. 

7. Team lacks necessary expertise: This may be due to staffing gaps and turnover or inherited orphaned code/projects. 

8. Dependencies: Dependencies are unstable, rapidly changing, or trigger rollbacks. 

9. Migration was poorly executed or abandoned: This may have resulted in maintaining two versions. 

10. Release process: The rollout and monitoring of production needs to be updated, migrated, or maintained.

While the authors emphasize that they don’t expect this ordering to apply to other companies, they do believe the list itself is generalizable. 

Measuring technical debt

Google’s quarterly engineering survey enables them to measure the rate at which engineers encounter and are hindered by each type of technical debt. The authors note that they’ve found slicing data by particular product areas especially useful: “we’ve found that engineers working on machine learning systems face different types of technical debt when compared to engineers who build and maintain back-end services. Slicing this data allows us to target technical debt interventions based on the toolchain that engineers are working in or to target specific areas of the company.” 

However, recognizing that quarterly surveys are a lagging indicator of technical debt, the researchers sought to develop metrics based on engineering log data that could capture the presence of tech debt. They focused on three types of tech debt—code degradation, teams lacking expertise, and migrations being needed or in progress—which were selected because each would require very different types of metrics.

They explored 117 metrics for these three forms of technical debt, using a linear regression to determine whether each metric could predict an engineer’s perceptions of technical debt. Their results were “disappointing, to say the least. No single metric predicted reports of technical debt from engineers.” 

While unable to find leading indicators with log data, the researchers continued to measure technical debt through their survey. They added the following questions to their survey to help identify the types of technical debt teams struggle with: 

• To what extent has your team deliberately incurred technical debt in the past three months?

• How often do you feel that incurring technical debt was the right decision? 

• How much did your team invest in reducing existing technical debt and maintaining your code? 

• How well does your team’s process for managing technical debt work?

Managing technical debt

A coalition was formed at Google to help leaders systematically address technical debt within their teams. Their efforts included the following: 

1. Creating a technical debt management framework to help teams establish good practices. The framework includes ways to inventory technical debt, assess the impact of technical debt management practices, define roles for individuals to advance practices, and adopt measurement strategies and tools.

2. Creating a technical debt management maturity model and accompanying technical debt maturity assessment that evaluates and characterizes an organization’s technical debt management process and helps grow its capabilities by guiding it to a relevant set of well-established practices for leads, managers, and individual contributors. The model characterizes a team’s maturity at one of four levels, as listed in the table below. 

3. Organizing classroom instruction and self-guided courses to evangelize best practices and community forums to drive continual engagement and sharing of resources. This work also includes a technical talk series with live (and recorded) sessions from internal and external speakers.

4. Tooling that supports the identification and management of technical debt (for example, indicators of poor test coverage, stale documentation, and deprecated dependencies). “While these metrics may not be perfect indicators, they can allow teams who already believe they have a problem to track their progress toward fixing it.”

These efforts led to a significant reduction in technical debt, with a majority of Google engineers reporting only slight or no hindrance from technical debt, marking the largest trend shift the organization has seen in the five years of running their survey.

Final thoughts

This paper presents a process that leaders can follow to improve the productivity of their teams. The first part of the paper describes the categories of technical debt (some of which overlap with the categories described in this Thoughtworks article), and the second part of the paper explores how these categories may be measured. Leaders can use the insights from those sections to measure and determine whether their teams are struggling with technical debt and, if so, what types of tech debt they’re struggling with. Then, the final part of this paper provides several tactics that may help reduce tech debt. 

---

That’s it for this week! If you’re finding this newsletter valuable, share it with a friend:

-Abi