An Inside Look at DORA
Q&A with Nathen Harvey on the DORA research program and common pitfalls of the metrics.
Every once in a while I’ll interview a researcher or practitioner focused on developer productivity and share the takeaways in this newsletter. My goal is to bring a different lens for viewing specific areas of research and share what experts are actively exploring.
Today’s issue is an interview with Nathen Harvey, who helps lead DORA at Google. Nathen has been part of DORA since it was acquired by Google Cloud in 2018, and in this conversation he shares how the annual reports are conducted as well as challenges he sees in how organizations are adopting parts of DORA’s work today.
Below you’ll find a summarized version of our conversation. You can also listen to the full conversation here.
What is DORA?
DORA means DevOps Research and Assessment, but the word is used to refer to many different things. DORA was a company, it’s a research program, it’s a report. All of this originated from the work of Dr. Nicole Forsgren, whose research led to the founding of DORA the company. Google Cloud acquired the company in 2018 and has since maintained its commitment to running this research study that seeks to answer that question, “how do teams get better at delivering and operating software?”
Many leaders are likely also aware of the “DORA metrics,” which were the four key metrics presented in the book Accelerate. The metrics are:
Time to restore service,
Change failure rate,
Lead time, and
Deployment frequency.
There was also a fifth metric introduced in recent years: reliability.
These were presented as measurable outcomes of improving software delivery. It is also worth noting that Nicole has since left Google Cloud, and today the lead researcher for DORA is Dustin Smith.
How are the annual reports conducted?
Designing the survey. Each year in January we begin working with the researchers to figure out what that year’s report will investigate. We have core areas to investigate every year, including the delivery metrics and the capabilities. We also come up with hypotheses about which new capabilities drive the four key metrics and introduce those.
Then we start writing the questions for the survey. This is something I’ve learned from Nicole and continue to learn from other researchers: the way you ask survey questions is so important. For example, you have to make sure that the question isn't asking two things at the same time. You also need to make sure the questions are interpreted in the right way.
Getting participants. This is an industry-wide survey that aims to have a diverse set of participants. The technical term for the recruitment technique used to get participants is “snowball sampling.” We open the survey and then post on Twitter and LinkedIn, we’ll write blog posts — all encouraging people to participate.
Identifying trends. This is a multi-year study. Over the years we’ve seen a few notable changes, including the introduction of a fifth metric (reliability), the removal of the “elite” cluster, and new clusters called Starting, Flowing, Slowing, and Retiring. Ultimately, we let the data drive what we find and we report on what emerges.
What are the pitfalls of the DORA metrics?
DORA metrics have become widespread, and there are examples where the metrics have been misused. Here are some of the common pitfalls companies face:
1. Comparing teams to each other based on the four key metrics. Different projects have different needs, so we can think more critically about whether a team's metrics should fall in the low, medium, or high performance category given that context.
Pitting teams against each other is also negative culturally: it encourages behaviors like hoarding information and lessons.
2. Setting goals for improving the DORA metrics. Instead set goals to improve the capabilities or factors that drive the DORA metrics.
I've heard leaders say, “by the end of this year, 80% of these teams have to be high performers or elite performers.” There are some challenges with that approach. For one, hitting that metric encourages people to game that metric. You want us to decrease our lead time from commit to running in production? Great. I'll go do a bunch of work over here that's invisible, and then when I know it's good to go I'll put it into the repository and commit it.
3. Not using the metrics to guide improvement at the team level. When the teams doing the work aren’t using the metrics to improve, this defeats the purpose of the metrics.
The signal to look out for is whether the teams doing the work are using the metrics to guide their own improvement or using the metrics to shine when they look up the organizational chart. The former is what you want, the latter is a signal to watch out for.
4. Spending more effort on pulling data into dashboards than on actually improving. Our research is survey-based. Yet we see teams go attempt to pull all of the data from systems.
Unfortunately, too many organizations spend a lot of effort pulling the system data together into beautiful dashboards that nobody looks at. They’ve spent all this capacity building dashboards when that could have been spent toward actually improving their capabilities.
5. Using "industry" as an excuse for not improving. Even companies in well-regulated industries can focus on improvement.
6. Assuming you’re already world-class, so your organization doesn’t need to focus on improving. If software delivery is no longer the constraint, then what is? Identify what is preventing teams from making progress and focus on that.
7. Fixating on the four metrics and forgetting about the capabilities. We don’t get better at those outcomes by focusing on the outcomes. We have to focus on the capabilities that drive those outcomes.
How are DORA and SPACE related?
In 2021, Nicole co-authored a paper called The SPACE of Developer Productivity, which today is frequently referred to as “the SPACE framework.” The paper also provided metrics companies could consider using to measure and improve developer productivity, so many leaders have been trying to figure out which to use — DORA or SPACE — or how to use them together.
DORA and SPACE are actually very complementary. If DORA metrics are outcomes we’re driving towards, what are some inputs to those outcomes? One input might be continuous integration. We can then look to the example metrics provided in SPACE as inspiration, and use those to understand whether we’re improving.
For a deep-dive on all things DORA, listen to the full conversation with Nathen here. You can also follow along with Nathen’s work on LinkedIn, Twitter, or by joining the DORA community.
That’s it for this week! Share your thoughts and feedback anytime by replying to this email.
If this email was forwarded to you, subscribe here so you don’t miss any future posts:
-Abi