2023 DORA Report
Key findings and commentary on this year's State of DevOps report.
This is the latest issue of Engineering Enablement, a weekly newsletter covering the data behind world-class engineering organizations. To get articles like this in your inbox every week, subscribe:
This week I’m summarizing the newly released 2023 State of DevOps Report. For those unfamiliar, DORA (DevOps Research and Assessment) is a long-running research program that focuses on helping engineering teams get better at delivering software. Each year, DORA publishes a report based on their research investigating which capabilities are driving software delivery and organizational performance.
I also interviewed Nathen Harvey (who leads DORA at Google) on my podcast this week, so if you’d prefer to listen instead of read, you can find the full interview here. I’ve incorporated some of what I learned from Nathen into today’s summary.
Key takeaways from this year’s State of DevOps report
The DORA research program tries to understand the relationship between different ways of working and relevant engineering outcomes. The “outcomes” the researchers look at fall in two categories: organizational performance and the wellbeing of developers.
While many people are familiar with DORA because of the four measures used to assess software delivery performance, we can see from the model above that these metrics are actually just one part of a broader program to drive meaningful improvements. The real substance of DORA’s research lies in the capabilities.
The annual report is conducted through an industry survey that is promoted online; this year, the report had nearly 3,000 respondents (a substantial 3.6x increase in participation from last year). Prior to each survey, the DORA team determines which outcomes they want to measure against (e.g., organization performance and employee wellbeing), as well as any additional research questions they want to explore as part of the annual study.
Here are my key takeaways from this year’s report:
Teams that focus on the user have higher organizational performance
One of the main findings in this year’s report revolves around “user-centricity,” which refers to how well a team understands their customers and whether they take action on customer feedback. Nathen mentioned that the research team was inspired to explore this topic due to the growing industry interest in platform engineering. In essence, platform engineering teams sometimes don’t understand their users well enough, and end up building things developers don’t need or use. The research team was interested in understanding whether this concept of user-centricity was a driver of performance for both internal and external facing teams.
To study user-centricity, participants were asked about how well their team understands the needs of users, how aligned the team is toward meeting user needs, and whether user feedback is incorporated when prioritizing work.
The study revealed that teams with a strong user focus have 40% higher organizational performance. Here’s the research team’s advice for internal and external-facing teams to apply this finding:
For platform teams, “this shift in focus requires platform engineering teams to understand how developers work today to successfully identify and eliminate areas of friction.”
For product development and delivery teams: “the results show that feature-driven teams failed to achieve top organizational performance. Such teams appear to be overly prioritizing delivery performance to the detriment of both organizational performance and the well-being of employees in the organization.”
Quality documentation amplifies the impact of other capabilities
Documentation refers to the internal written knowledge that people use day-to-day. To study the impact of this topic on performance, the researchers measured the degree to which documentation is reliable, findable, updated, and relevant. Then, they calculated one score for the entire documentation experience.
Documentation is interesting because it amplifies the impact of other capabilities on organizational performance. For example, the study found that quality documentation amplified the impact of continuous integration on organizational performance by 2.4x, continuous delivery by 2.7x, and reliability practices by 1.4x.
Quality documentation was found to positively impact individual job satisfaction and productivity as well.
The technical capabilities that impact performance
DORA’s research also explores whether specific technical capabilities have an effect on the following performance measures (defined here as they’re defined in the report):
Team performance: The ability for an application or service team to create value, innovate, and collaborate.
Organizational performance: The organization should produce not only revenue, but value for customers, as well as for the extended community
Software delivery performance: Teams can safely, quickly, and efficiently change their technology systems.
Operational performance: The service provides a reliable experience for its users.
Teams that have loosely coupled architecture (which are also called loosely coupled teams in the report) are able to make significant changes to their systems without involving other teams. This enables them to move faster; and as shown in the table, it’s the only capability to have an effect on all of DORA’s performance measures.
In our conversation, Nathen also pointed out the significant impact of code review speed on software delivery performance. “We saw that speeding up code reviews led to 50% higher software delivery performance… If your code reviews are already fast or maybe even non-existent, don't try to make them faster. That's not your bottleneck. But where your code reviews are slow, I think you have a big opportunity there.”
Other notable changes since previous reports
1. Team performance is a new construct introduced in the report. In the past, DORA’s research focused on organizational performance. Nathen explained that the way organizational performance is measured is by asking questions such as how profitable their organization is and whether they’re meeting business goals, however some practitioners can be disconnected from those outcomes. Team performance is closer, but still beyond an individual’s remit.
2. MTTR was replaced with Failed Deployment Recovery Time. MTTR has caused some confusion in the community: is the “M” for mean or median? Additionally, the report notes that practitioners seeking to learn more from failures, such as those in the resilience engineering space, are moving past MTTR as a reliable measure for guiding learning and improvement.
3. The way Change Failure Rate is measured changed. In previous years, respondents were presented with six options (0-15%, 16-30%, etc.). This year respondents were presented with a slider so that they could select any value between 0% and 100%. The metric now provides more precision in the answer.
4. “Elite” reemerged as a cluster in the benchmarks. Last year, there wasn’t an elite category that emerged in the research, so it wasn’t included in the report.
Final thoughts
DORA’s annual reports are a great resource for understanding the practices and trends that are benefiting other organizations. I’m always eager to learn about the metrics and how they’re measured, and this year I also thought the insight about user-centricity was especially interesting.
A special thanks to Nathen Harvey for generously sharing his time in the community and on my podcast to discuss DORA’s research and how it might be successfully applied. I always look forward to reading these reports and am glad to see the program continuing strong.
That’s it for this week! If you’re interested in reading a guide for running an internal survey to identify problems impacting developer productivity, send me a connection request with the note “guide.”
-Abi