Driving Awareness for Internal Developer Tools
The effectiveness of physical newsletters and other tactics for driving awareness of internal tools.
This is the latest issue of my newsletter. Each week I cover the latest research and perspectives on developer productivity.
This week I read Do Developers Learn New Tools on the Toilet?, a paper from Google engineering productivity researchers. This study examined the effectiveness of physical newsletters and other methods in raising awareness about new internal tools. While last week’s paper mainly discussed factors that contribute to longer-term adoption of internal tools, this paper focuses more on helping developers discover the tools that are available to them.
My summary of the paper
Prior to this study, teams at Google attempted to increase awareness and adoption of software tools and practices by printing 1-page “weekly newsletters” and posting them as flyers in company restrooms worldwide (hence the name of the paper). The researchers here wanted to study whether this technique increases tool usage or not. They also sought to compare this technique to others aimed at increasing adoption, as well as identify other factors influencing the usage of tools.
This study was conducted by synthesizing data from both qualitative and quantitative sources. Researchers analyzed logs from 12 development tools to capture usage before and after the publication of corresponding newsletter. Then they contextualized the results by interviewing and surveying the authors and readers of the newsletters.
Here’s what the study found:
Are physical newsletters effective?
The results suggest that all tools but one experienced increased usage due to the newsletter. The figure below illustrates the usage before and after being promoted in the newsletter: the solid black lines represent usage, the solid gray lines indicate the day before the tool was promoted, and the gray dotted line marks 3 weeks post-newsletter. Rows are ordered by relative effect.
The researchers also observe usage patterns that can be attributed to each tools’ purpose. Top plots show (a - f) strong growth after each episode, the three plots on the top-left quadrant (a, c, e) demonstrate sustained growth, and the plots in the right quadrant (b, d, f) exhibit a decline after the newsletter. The tools showing strong growth are designed to be used on a daily basis, whereas others are designed to be used on occasion.
How the physical newsletter compares to other techniques
Some tooling owners use other techniques to promote their tools, and the researchers explored how those techniques compare to the printed newsletter.
Internal social media: Social media posts have a positive impact on tool adoption, but the magnitude of the increase depends on factors like how many followers the developer has. In contrast, the printed newsletter's influence is not limited by an individual's personal network.
Peer learning: Developers learn about tools from their peers. This insight is more relevant for tools that are inherently more visible, as opposed to those being run entirely on one developer’s workstation. One interesting technique surfaced in the study: some developers learned about new tools by seeing other developers have tool badges on their profile pages in the employee directory.
Announcement emails: Tool owners sent targeted emails to relevant mailing lists, resulting in a significant increase in users.
Other spaces for posting flyers: Tool owners posted flyers in other locations as well, such an annual company project fair. However, these efforts had no noticeable impact. This may highlight the importance of aspects such as repeat exposure and higher likelihood of getting developers’ attention when posted in restrooms, and the ability to get in front of the intended audience. Leaders considering whether to print posters or newsletters to increase tool adoption should take these aspects into consideration.
Other factors influencing tool discovery
The researchers also found that the usage or lack of usage of software development tools after the newsletter is partly a consequence of the design of the tool itself, especially in terms of memorability, trialability, breadth of applicability, and usability.
Memorability refers, in this case, to the name of the tool. Data collected from follow-on interviews suggested that names that are easier to remember may help adoption. In this study, “IBLAZE” was referred to as a memorable name because it is a one-letter addition to another standard build tool “BLAZE”, whereas “ChangeTimeZone” was not a memorable name.
Trialability refers to how easy it is to try a tool. One tool in this study, for example, required users to collect a substantial amount of upfront information to determine if the tool would be useful in their context, leading to fewer users adopting the tool.
Breadth of applicability refers to different groups that tools are relevant for. A lower number of users may at the surface seem like an issue, however if the tool is only relevant for a subset of developers, this may not be true.
Usability refers to how easy and intuitive it is to use the tool. Tools that are difficult to use will see drop-offs in usage.
Final thoughts
This study presents several techniques to increase the discoverability of internal tools, including a physical newsletter on developer productivity tools, peer learning, and announcement emails. It also shares other factors influencing usage.
While last week’s paper discussed how to drive longer-term adoption, this one focuses on how leaders can create awareness for a tool. Both papers highlight the importance of investing in usability (or “compatibility”) and observability (where developers learn about new tools from their peers).
That’s it for this week! Share your thoughts and feedback anytime by replying to this email.
If this email was forwarded to you, subscribe here so you don’t miss any future posts:
-Abi