Applying the DevEx framework
A guest post from Laura Tacho on what the DevEx framework is and how to apply it.
This is the latest issue of my newsletter. Each week I cover the latest research and perspectives on developer productivity.
Last month, Laura Tacho (CTO at DX) and I held a discussion on the DORA, SPACE, and DevEx frameworks. One question that came up after the conversation was whether we had any resources to share on how to apply the DevEx framework.
The DevEx framework, published last year, is an approach for measuring and improving developer productivity that focuses on the developer experience. In today’s issue, I’m sharing Laura’s guide on how to apply the framework.
Here, Laura explains who benefits the most from the framework and how to implement it. This issue will be useful if you are currently dealing with the problem of how to measure or improve productivity, or are already interested in the DevEx framework.
The DevEx framework
Developer experience (DevEx) is a developer-centric approach to improving developer productivity. Instead of focusing wholly on output or activity metrics, DevEx focuses on the lived experiences of developers by measuring the effectiveness of tools and processes through the developers’ lens, and giving them a voice to influence the factors that impact their work.
The DevEx Framework organizes many factors of developer experience into three categories: feedback loops, cognitive load, and flow state.
Feedback Loops: When a developer makes a change, can they get feedback about that change fast enough? Feedback from tooling, like tests or a CI build, and people, like project stakeholders, are equally as important. Slow feedback loops can interrupt or delay the development process.
Cognitive Load: How much stuff do developers need to keep track of in order to complete a task? Complex processes, as well as complex code, can lead to high cognitive load, which slows development down and increases friction.
Flow State: Slow feedback loops and high cognitive load can make it hard to get into a flow state, as well as other factors like unplanned work and a non-optimised meeting schedule. Flow state describes the opportunity to get into a state of energized focus. This doesn’t just mean having blocks of uninterrupted time, but also systems that allow developers to become immersed in their work by reducing friction.
Who should use the DevEx framework?
Teams who are interested in using metrics to improve developer productivity and engagement will benefit most from the DevEx framework.
Platform engineering teams who are responsible for systems that support many engineers can use the DevEx framework to understand where to focus for the most impact
Engineering managers can use it to understand points of friction on their teams
Engineering executives can use the DevEx framework to understand if strategic investments are paying off and keep a pulse on overall engineering organization health
Similar to DORA and SPACE, the DevEx framework is used by all sizes of companies, across many industries.
How do you implement the DevEx framework?
Similar to the SPACE framework, the DevEx framework strongly advocates for including developers’ feedback and experiences in definitions and assessments of productivity. In contrast to SPACE, the DevEx framework is more prescriptive on what to measure, introducing DevEx KPIs, along with a framework to identify potential areas of measurement.
The perceptual measures outlined in the DevEx framework are best collected through a developer experience survey. This allows you to standardize questions and responses in order to track progress over time. A limitation of surveys is that it is often unclear to the respondent how – and if – their response data will be used to noticeably increase their own working conditions.
A countermeasure to potential disengagement is to transparently communicate the plan for the survey data before the survey is administered, answering the questions:
Who will see the data?
What demographic information will be associated with the responses?
What will we do with the data?
For smaller teams, or for environments where survey engagement is forecasted to be very low, you may want to use interviews or feedback opportunities like team retrospectives to collect data related to these perceptual measures. Since these formats are not standardized like surveys, you will need to index and categorize the data in order to track progress over time.
Workflow measures may also be collected via surveys, or your team may opt to ingest the data directly from workflow tools like your ticketing system or source control. The benefit of using a survey to collect workflow data is that you can simplify data collection by using only one method.
A well-designed survey will provide accurate data about workflows. Remember, DORA is based on survey data!
A sample of questions might look like something like this:
Note: Here’s a DevEx survey template to help you get started.
With this data, teams can identify their highest priority drivers.
After identifying which areas are causing developers friction, some teams will then want to bring in quantitative metrics based on data from their engineering systems. For example, you might learn from the survey that developers are unsatisfied with the deployment process. You can then look at Merge to Deploy, a metric calculating the time between a code change being merged and it being deployed, and see that it’s taking developers more than a day to release approved changes.
As another example, survey results might show that developers are frustrated with the code review process. You ask developers for additional feedback on what they’re frustrated with, and learn that some PRs can take more than 24 hours to be reviewed. You then look at the median, P75, and P90 Time in Review, a measure of how long a code change is waiting on review across all of its individual review cycles. You confirm that about 20% of developers’ PRs take over 24 hours to review.
Bringing in quantitative data alongside qualitative insights can help you learn more about a problem you’ve identified. This can also be helpful for tracking improvements at a granular level. In the example above, you can use the Time in Review metric to show progress towards a goal, alongside the developer sentiment metrics gathered in your surveys.
When using qualitative and quantitative metrics together, start with qualitative metrics to figure out where to focus. Once your developers have identified specific areas, you can then use quantitative metrics to drill in deeper.
Quantitative metrics alone lack the context needed to assess whether your developers view something as a problem or a priority, and for this reason, we advise using quantitative metrics in isolation. In the example above, you could look at the Time in Review metric, and not be sure whether what you’re seeing is good or something needs to be improved. Systems data is also not as comprehensive as data coming directly from developers, because each measurement only captures a part of the system.
The best approach is to start with qualitative data to guide your attention, then use quantitative metrics to drill in further. And remember, all measurements should have a purpose.
What’s important to consider when using the DevEx framework?
In a previous issue covering SPACE metrics, we discussed how some leaders may be hesitant to adopt new ways of thinking, based on the latest research. DevEx highlights human attitudes and opinions even more than SPACE, making a strong recommendation that measures of experience, satisfaction, and attitude are critical in order to improve developer productivity.
To be successful with the DevEx framework, it’s crucial that your organization has an intention of using the data to drive impact, and isn’t collecting data for the singular goal of performance assessment, or just out of curiosity. Teams that have been very successful with the DevEx framework have followed this 4-step process to improve developer experience and productivity:
Get feedback from developers to strengthen your understanding of the factors that impede developer productivity and degrade developer experience.
Set a target with the team. Keep this footprint small: 1-2 goals max for any timeframe. Choose how you will track progress against this target.
Drive impact by executing projects and running experiments to change habits, processes, and/or tooling.
Measure again to understand if the challenges have been overcome (use the same method you used in Step 1), and to understand where new challenges might be.
Final thoughts
Engineering leaders understand that increasing developer productivity and engagement leads to better business outcomes. Frameworks like DevEx help teams define productivity and measure it. This gives engineering organizations clarity into where to focus and the ability to quantify the impact of change.
Who’s hiring right now
Find recent DevEx job postings here.
Thanks for reading. If you know someone who might like this issue, consider sharing it with them:
-Abi
The questions examples is a great addition, thanks.
First off, congratulations on the framework. Quick question: In the survey, do you use questions with answer options on a Likert scale? Wouldn’t a tool like Dave Snowden’s Sensemaker work better here? This is because it also captures devs perceptions, feelings, etc. based stories they telll about their experiences. Also, would journaling by devs provide more qualitative data in addition to the survey?