Engineering Enablement
Engineering Enablement Podcast
Leveraging sentiment data, driving org-wide action, and executive engagement
0:00
Current time: 0:00 / Total time: -24:32
-24:32

Leveraging sentiment data, driving org-wide action, and executive engagement

Twilio’s David Betts on how they take a data-driven approach to driving platform engineering initiatives.

David Betts is a leader on Twilio’s Developer Platform team — a team that I’ve come to know as being exceptionally data-driven in how they drive forward platform initiatives. In this conversation, David and I discuss:

  • Twilio’s journey from traditional metrics to sentiment-driven insights

  • What prompted them to start measuring developer experience

  • Examples of how they use qualitative and quantitative metrics together

  • The steps they take to report on developer experience data

  • What they do to encourage team-level use of data

  • How they justify Platform headcount

  • and more

Listen now on Apple, Spotify, and YouTube.

Some takeaways:

  1. Twilio started measuring developer productivity with the DORA metrics and found value in them. Eventually, they hit a ceiling and stopped being able to surface new actions to drive improvement. When looking for other system metrics they could collect, they came across the idea of taking a sentiment-based approach.

  2. They’d previously run ad hoc developer surveys using Google forms, but didn’t have consistency in their data or in their follow-through. They started using DX, saw 90%+ participation rates, and were able to identify the areas that were causing developers the most friction. This ultimately helped them build their next 12 mos roadmap for Platform.

  3. On the difference between sentiment data and system metrics: Sentiment is easier to get started with, and tends to be more actionable.

  4. How they use both types of data together: they start with sentiment data to identify areas to improve. Once they know where they want to focus, they find 1-3 system metrics they can use to help measure success for improvements in that area. Sentiment data is lagging, so the system-based metrics help them track improvements in real-time.

  5. After each developer survey, the platform team shares results through a slide deck for executives, which gives a high-level of where they’ve improved and where further improvements need to be made. They also send out a Voice of the Developer email to all of engineering, which provides more depth into what people are saying and what the platform team is doing about it.

  6. Many developer experience issues need to be addressed by individual teams, not a centralized Platform group. Two tactics that have helped encourage teams to drive their own improvements: delivering data back to teams, and recognizing improvements made by teams in their Voice of Developer email.

  7. David’s take on the misperception that developer experience is about “nap pods” (or foosball tables, free beer, etc): That perception is a sign of not seeing the value in either developer experience or more broadly in platform engineering. But if you’re in a company where this mindset exists, you can help change that perception by framing results from sentiment data in terms of efficiency gains, quality improvements, etc. "Well, yes, this is sentiment data, but people are self-reporting that they're saving two to four hours by using GitHub Copilot per week, and that equates to four full-time engineers of time saved across our organization."

  8. To justify additional headcount, Twilio’s platform team starts by building their 1-2 year roadmap so they can say “Here’s the capabilities that we can deliver and by when.” When there’s excitement around those projects, they can propose a faster timeline with additional headcount. To summarize, they justify additional headcount by showing how it would accelerate their roadmap.

  9. In addition to using periodic surveys, they use real-time surveys (using DX) to capture in-the-moment feedback with specific tools and workflows. These surveys are triggered when developers complete a workflow or action within a tool, and ask specific questions about their experience. This gives the Platform team a higher volume of feedback, and more specific feedback. It also helps them identify the daily “paper cuts” they might not otherwise be able to surface.

Where to find David Betts:

In this episode we cover:

  • (0:00) Introduction

  • (1:27) Twilio’s developer platform team

  • (2:46) Twilio's approach to release engineering and CD

  • (5:06) How they use sentiment data and telemetry metrics

  • (8:43) Comparing sentiment data and telemetry metrics

  • (11:56) How to take action on sentiment data

  • (15:05) What resonates with execs

  • (18:17) Proving DX value: sentiment, efficiency, and ROI

  • (22:14) Balancing quarterly and real-time developer feedback

Read the full transcript here.

Referenced:

Discussion about this podcast