Listen and watch now on YouTube, Apple, and Spotify.
DORA just released a new report on the impact of generative AI on software development productivity, and the findings were a mix of expected and surprising.
On today’s episode, I’m joined once again by Derek DeBellis, lead researcher on Google’s DORA team, to break down the key insights. We talk about how the survey was developed, why measuring productivity is so complex, and what the data actually tells us about how AI is affecting real-world teams.
Some takeaways:
The research design process
How DORA designed survey questions to validate hypotheses, including the careful iteration of single words.
An explanation of “flow” and why the DORA team decided to use a general definition of flow for the survey
How dev time was broken into two buckets: time spent on toilsome work vs. valuable work—and what those self-reported numbers reveal.
If accuracy matters, measure time as close to the present as possible—same day or “right now”.
You don’t need perfect accuracy to learn a lot from data.
How DORA’s definition of productivity is inspired by the book Slow Productivity, and tied to value creation.
Positive findings from the Gen AI report
Productivity is likely to increase by 2.1% when individual AI adoption is increased by 25%—about 20 minutes.
How a 2.1% increase in productivity per individual could be dramatic across an entire organization.
Interesting and surprising findings from the report
The contradictory finding that the time spent doing valuable work went down with AI, but this is easily explained by AI making the work go faster, not actually reducing the amount of valuable work.
While AI increases personal productivity (e.g., 2.1% boost), it correlates with a 1.5% drop in delivery throughput and a 7.2% decline in stability. The good news is that some of this could be only a short-term period of working with new constraints.
Guidance for measuring AI’s impact on productivity
Make sure that all metrics are aligned with your organization’s goals and multi-faceted as well.
Don’t fall into the common trap of overestimating how much AI tools boost productivity.
Use discrepancies between the current state and goals to inform strategy.
Beware of short-term disorientation when adopting new tools.
In this episode, we cover:
(00:00) Intro: DORA’s new Impact of Gen AI report
(03:24) The methodology used to put together the surveys DORA used for the report
(06:44) An example of how a single word can throw off a question
(07:59) How DORA measures flow
(10:38) The two ways time was measured in the recent survey
(14:30) An overview of experiential surveying
(16:14) Why DORA asks about time
(19:50) Why Derek calls survey results ‘observational data’
(21:49) Interesting findings from the report
(24:17) DORA’s definition of productivity
(26:22) Why a 2.1% increase in individual productivity is significant
(30:00) The report’s findings on decreased team delivery throughput and stability
(32:40) Tips for measuring AI’s impact on productivity
(38:20) Wrap up: understanding the data
Where to find Derek DeBellis:
• LinkedIn: https://www.linkedin.com/in/derekdebellis/
Where to find Abi Noda:
• LinkedIn: https://www.linkedin.com/in/abinoda
Referenced:
Share this post