Developer Onboarding and Ramp-Up Time
Findings from studies at Google to understand, measure, and improve developer onboarding.
This is the latest issue of my newsletter. Each week I cover the latest research and perspectives on developer productivity.
This week I read Onboarding and Ramp-Up, the latest installment in a series by Collin Green, Ciera Jaspan, and their colleagues. (I’ve covered other papers in this series, including their previous issue on build times and developer productivity.) In this paper, Green and Jaspan share what they’ve learned from their research at Google to understand, measure, and improve developer onboarding.
“Enhancing the new engineer experience has the potential to positively impact engineering productivity, satisfaction, hiring, and retention.”
Here are the key takeaways from the paper:
Understanding the developer onboarding experience
Google researchers first looked to learn about existing onboarding practices so they could better understand what is working well and what isn’t, and how they might act to improve. To study this, they initially worked with an external research vendor to interview engineers from various tech companies. Then, they surveyed new engineers at Google to learn about what they experienced during onboarding and what helped or hindered their ramp-up.
This revealed several insights, some of which are universally applicable to all employees, while others are specific to developers:
The selection of an appropriate starter project that facilitates key learning skills is a major challenge for developers.
Developers with prior work experience would benefit from a more tailored onboarding process that allows them to opt-out of sessions that cover common industry best practices.
Experienced developers preferred to learn about tools, processes, and systems by diving into real tasks on their primary projects, rather than running through sample tutorials or contrived starter projects.
New developers feel better supported when they have a mentor to help guide them while ramping up.
Additionally, Google’s survey data indicated that the three top hindrances to ramping up are: learning a new technology, poor or missing documentation, and finding expertise.
Measuring the impact of changes to onboarding
As Google planned to make improvements to developer onboarding, they wanted the ability to track the impact of their changes. To do this, they explored objective metrics to understand how long it takes for developers to ramp up.
There’s an important point to note here: Google’s intention to was to understand ramp-up time, not to understand developer productivity more holistically during onboarding. (They say that if they had been focused on the latter, they would have reviewed multiple metrics that would capture the different facets of productivity — speed, ease, and quality.) The researchers took this approach because they specifically wanted to measure the impact of different interventions, such as training programs, on ramp-up time.
“Our goal in creating ramp-up-time metrics was to facilitate the evaluation of different interventions (e.g., different orientation and training programs) and the impact of external events (e.g., a global pandemic) on ramp-up times.”
Selecting and validating metrics at Google is a rigorous process. To summarize the steps involved:
Google wanted to avoid scenarios where developers might be individually assess or compared to each other, which may encourage individuals to game the metrics. They also wanted to control for variations in job requirements and work complexity. For these reasons, they chose to validate metrics against engineers’ own perceptions of the extent to which they were ramped up on a variety of key tasks.
The researchers reviewed a set of candidate metrics that might be useful proxies for engineers’ ramp-up speed. They built upon their developer logging system, which ingests logs from multiple developer tools to build a picture of a developers’ behavior during their workday. Ultimately they narrowed the list to three metrics to further evaluate.
Finally, the researchers validated objective metrics against survey data to understand how well the metrics reflected new developers’ more holistic experience of ramping up. For this, the researchers conducted a survey with new engineers, ending up with a sample of more than 3,000 developers.
Two metrics were found to be reasonable proxies for engineers’ ramp-up time: normalized active coding time per LOC, and normalized submitted LOC. That is, developers were coding faster, submitting more LOC to the codebase, and rating themselves as more ramped up on the 17 key engineering tasks as they gained tenure. These metrics were selected to measure the impact of changes to onboarding on ramp-up time.
The researchers add a word of warning before adopting these metrics: “We’ve found these metrics to be highly erratic and noisy on an individual level. These metrics should not be used to evaluate individual ramp-up speed as they’re only valid for evaluating the quality of a training program across a cohort of developers.”
“We’ve found these metrics to be highly erratic and noisy on an individual level. These metrics should not be used to evaluate individual ramp-up speed as they’re only valid for evaluating the quality of a training program across a cohort of developers.”
To demonstrate the sensitivity of these metrics to a substantive change in onboarding practices, Google examined the question of whether ramp-up times were impacted by the shift to remote onboarding practices at the beginning of the pandemic. They found that the shift to remote onboarding resulted in slower ramp-up for engineers on the order of three to six weeks.
Final thoughts
The authors, Collin and Ciera, have spoken about how Google’s approach to measuring productivity includes using qualitative and quantitative metrics together. This paper provides a great example how this works in practice: they began by defining their goals for measurement, and then validated metrics by comparing quantitative information with qualitative information.
That’s it for this week! If you know someone who might like this issue, consider sharing it with them:
P.S. If you’re interested in resources on running internal developer surveys, send me a connection request with the note “surveys”.