Three-Bucket Framework for Engineering Metrics
Deciding what metrics to track and report to stakeholders.
This is the latest issue of my newsletter. Each week I cover research, opinion, or practice in the field of developer productivity and experience. This week is an article I wrote about selecting metrics to report on.
One of the most common problems engineering leaders face is deciding what metrics to track and report to stakeholders. I receive questions about this frequently:
I was asked by leadership to consider how we measure "FTE productivity" in our engineering organization. How do you handle this kind of thing?
My CEO came to me and asked me: "how do we know if our engineering org is actually good?"
I personally experienced this challenge in my first job as a CTO. Our executive team had started meeting monthly to review each department’s progress. Our CEO wanted more than just the anecdotal release updates I had been providing, and asked me to come up with a hard metric to report.
The problem: I couldn’t think of a good metric to use. Our CEO suggested the number of features we released, but I resisted because I knew it didn’t accurately account for the size of each feature. Alternatives like the number of lines of code or pull requests felt even worse, and I feared losing the respect of our developers if they found out I was representing our work in this way. In the end, I decided to give in and report the number of features released in order to save face with my boss and other executives.
In hindsight, I wish I’d approached this problem differently and today I am able to give concrete advice to leaders that find them themselves in a similar situation. This advice takes the form of reframing the problem, and a simple classification framework for metrics that leaders should be reporting on. I’ve developed this approach based on years of research and experience advising leaders.
If your boss asks for engineering metrics, start by reframing what they’re really asking for. Leaders often fall into a rabbit hole of worrying about whether to measure individuals or teams, the legitimacy of engineering metrics products, or whether engineering can be measured at all. But this misses the big picture: CEOs don’t know or care about the technicalities of engineering measurement; what they really want is a way to have confidence that you’re accountable for the millions of dollars they are spending on engineering.
Once you reframe the problem in such a way where you’re thinking about how to demonstrate strong stewardship, choosing metrics becomes much easier. For example, your CEO is likely under constant pressure around engineering headcount costs, so a clear accounting of projects being worked on along with their business ROI can reassure them and keep them looped in. If stakeholders are worried about whether the “house is in order” in regards to the performance of certain systems or teams, metrics can help assuage concerns.
Ultimately, your goal should be to proactively address these types of concerns by providing a full picture of engineering. This be accomplished by reporting metrics in these three buckets:
Business impact. Your company spends millions of dollars on engineers, which naturally begets the question: “where are we spending this money, and why?”. You should report on current or planned projects alongside data that addresses key questions like: Why are these the right things to build now? How does this project make the business money or otherwise support its goals? Is this project on track or delayed? This type of reporting is often seen as the responsibility of the product team, but only engineering can represent the full set of projects being worked on (e.g. a database migration project is unlikely to be on product’s roadmap).
System performance. Engineering organizations produce software, so stakeholders will want to know about the health and performance of these systems. Are they fast and reliable? Secure and well-maintained? Are users satisfied with them? Useful metrics to report here include things like uptime, number of incidents, and product NPS scores. If you have a dedicated Infra or UXR team, they likely are capturing metrics that fall under this bucket.
Developer effectiveness. Developers cost a lot of money, so stakeholders will naturally be interested in knowing how productive developers are, and how developer effectiveness is being improved. If you have a dedicated DevProd or DevEx team, they likely are capturing metrics that fall under this bucket. Models such as SPACE and DORA are good to check out. I am working on a new DevEx framework with Dr. Nicole Forsgren and Dr. Margarette-Anne Storey which will offer a new approach.
The following table summarizes the three buckets and examples of metrics to use for each one. Metrics can sometimes fall into more than one bucket.
You shouldn’t need fancy tools to capture these metrics–most of the information should already exist through your existing tools and processes, or else manually captured with simple spreadsheets or surveys. It doesn’t matter which specific metrics you use within each bucket or how perfectly things are measured. The important thing is to show intention and effort to your stakeholders, and then continue to iterate on your approach.
There are many metrics being advertised across the internet, which can make the process of choosing metrics overwhelming. I’ve found that categorizing metrics into these buckets forces you to be intellectually honest about why you are using a certain metric: Does the metric actually convey something useful? Or is it just something that was mentioned somewhere?
I see two common mistakes occur when leaders try to come up with metrics without this three-bucket framework:
Some leaders try to count things like lines of code, releases, or merge requests. This feels like a good solution because the data is typically right at our fingertips. But there’s strong consensus amongst researchers and experts that output measures are ineffective and harmful.
Another mistake I see is that leaders sometimes try to use metrics like lead time or cycle time to convey the impact of their organization to other stakeholders. Although these metrics are useful for measuring organizational effectiveness, they don’t tie directly to business impact.
Take a moment to examine your current engineering metrics, and I think you’ll find that you already have metrics for these buckets represented. Consider reorganizing your reports around this framework to create more alignment around the purpose behind each of your metrics. If you’ve recently joined an organization as an engineering executive, this approach can help you strengthen your relationship with stakeholders from the start.
That’s it for this week! If this email was forwarded to you, subscribe here so you don’t miss any future posts:
Share your thoughts and feedback anytime by replying to this email.
-Abi
"If your boss asks for engineering metrics, start by reframing what they’re really asking for."
This here is really the key. One thing that I didn't do the first time I saw myself in such a situation but ended up learning by brute force. Things become way easier after refining this statement.
"I am working on a new DevEx framework with Dr. Nicole Forsgren and Dr. Margarette-Anne Storey which will offer a new approach."
Super interested to see this Abi! Email us as soon as it's live if you can.