How to Misuse & Abuse DORA Metrics
Bryan Finster's paper shares his personal experience rolling out DORA metrics and witnessing how they were misunderstood and misused.
This week I read How to Misuse & Abuse DORA Metrics, a paper by Bryan Finster in the latest issue of the IT Revolution DevOps Enterprise Journal. In this paper, Bryan shares his personal experience of rolling out DORA metrics and witnessing how they were misunderstood and misused. Bryan wrote this paper because he “realized that it wasn’t just [his] organization that was misusing the metrics,” but that other companies across our industry were as well.
You can also listen to my interview with Bryan about his paper here.
My summary of the paper
Background
Bryan was a strong advocate for continuous delivery. When Accelerate came out, it “confirmed a lot of what the DevOps community was already saying.” To help him communicate the value of measuring and improving continuous delivery, he bought six cases of books and distributed them to executives across his company, pointing them to page 19 where the four key metrics known as the “DORA Metrics” were presented.
“Almost immediately things started going south. I started getting feedback that in some teams, people were spending two days every month gathering delivery metrics to report on. We heard about directors and VPs comparing teams based on the metrics. It was just a complete misunderstanding of how to appropriately use the metrics… We unintentionally communicated that measuring human activity is simple instead of complex, and that all we needed to do was set these metrics as goals to measure our way to improvement.”
Others were seeing this problem too. “More commercial tools are including DORA metrics, but the context is being lost and the promised improvements to delivery, quality, culture, and pain are not being realized.” Bryan said his goal in writing the paper was “to help correct the course on DORA metrics usage so that every organization can see the benefits we know to be true.”
DORA metrics explained
DORA metrics are a set of four metrics focused on throughput and stability:
Lead time: time it takes to go from code committed to code successfully running in production.
Deployment frequency: how often code for primary service is deployed to production.
Mean time to restore (MTTR): time it takes to restore service.
Change fail percentage: percentage of changes to production which fail.
These metrics were chosen because: “We have found a valid, reliable way to measure software delivery performance that focuses on global, system-level goals, and measures outcomes that different functions must collaborate in order to improve.” (Accelerate)
Common misuses of the DORA metrics
Bryan describes the common ways companies misuse the metrics:
Focusing too much on speed. “Measuring deployment frequency without using quality metrics as guardrails will result in poor outcomes.”
Setting goals around DORA metrics. “The goal isn’t better DORA metrics… OKRs should be focused on desirable business outcomes.” Choose goals, then choose metrics that align with those goals.
Mistaking measuring DORA metrics as a way to improve. “[DORA metrics] don’t fix things. If we simply get a dashboard and do not buy into using it to identify improvement items, then nothing will get better.”
Using DORA metrics as vanity metrics. “[DORA dashboards] are often used as ‘vanity radiators’ instead of information we can use to help us improve.”
Not including other signals in addition to the four key DORA metrics. “The four key metrics DORA used to correlate behaviors of high-performing organizations are a small subset of the metrics recommended in the book Accelerate. They also represent only one aspect of the health of a system…”
Final thoughts
I think this paper hits at a real problem in our industry right now. The desire for a "silver bullet" for measuring development has led to DORA metrics proliferating across the industry.
DORA metrics have become the "easy button" for measuring success, even though the metrics were designed as indicators of performance across a wide set of practices, not goals in of themselves.
If you're interested in the DORA metrics, make sure you read Accelerate in its entirety so you understand the context in which the metrics are presented, the practices which they promote, and how the metrics are calculated (surprise: they used surveys!).
As with any metric, it's important to make sure you have specific outcomes defined around what and why you want to measure.
Other recommended reads
Challenges of having product managers on a platform team
Jelmer Borst (Platform PM at Picnic Technologies), Will Larson (CTO at Calm), and others
Product managers help platform teams get customer feedback, communicate their team’s priorities, and get adoption. But there are challenges to hiring and retaining PMs that are worth considering as a platform group grows:
It may be more challenging to retain PMs within a platform group. As the PM becomes more senior, they may perceive moving into the consumer side of the business as having a greater impact.
Some teams require platform PMs to be more technical than product engineering teams do.
PMs shouldn’t take over all customer research. Engineers should be included in the customer feedback and solution-ing process as well, just as they should in product engineering teams.
Almost nobody has an effective Staff+ program
Chris Chandler, Software Engineer at Block
This post includes a list of signals that a company doesn’t have an effective program for managing engineers “with 15+ years of experience that don’t want to manage.” Some of the signals include “managers hiring senior folks to be ‘self-directed’ when they actually mean ‘unsupported’” and asking staff engineers to “lead with influence.”
Inequity in job performance feedback
Textio study
Textio’s new study found:
Compared to men, women are twice as likely to report being described as collaborative and nice, seven times more likely to report being described as opinionated, and 11 times more likely to report being described as abrasive.
Black and Latinx employees are more likely to receive job performance feedback that is not actionable (either feedback about their personality, which is less actionable than feedback about someone’s work, or feedback without specific guidance on how to improve).
This matters because people with access to actionable feedback tend to grow faster in their careers.
That’s it for this week! Know someone who would enjoy this newsletter? Consider sharing it with them.
And as always, reach out with thoughts about this newsletter on Twitter at @abinoda, or reply to this email.
-Abi