Measuring Flow and Focus
Focus time is a positive predictor of flow.
This is the latest issue of my newsletter. Each week I cover research, opinion, or practice in the field of developer productivity and experience.
This week I read Using Logs Data to Identify When Software Engineers Experience Flow or Focused Work by Google researchers Adam Brown, Sarah D’Angelo, Ben Holtz, Ciera Jaspan, and Collin Green. Here, researchers explored when developers achieve flow and whether it can be measured using a logs-based approach.
My summary of the paper
Flow is often achieved when a person feels focused and fulfilled with their work, and is a positive predictor of developer satisfaction and productivity. The concept has been extensively studied, however previous studies relied on measurement methods that inherently disrupt participants’ work. For this study, researchers sought to develop a metric that could passively detect when an individual was experiencing flow or focused work.
To begin, researchers conducted a preliminary diary study to understand and define how engineers at Google experience flow. Based on the results of the preliminary study, the researchers determined that measuring when engineers experience a state of flow using a logs-based approach alone would not be presently possible. They introduced a new metric, “focus time,” and in the second part of the study explored how well the focus time metric reflects the daily experience of software engineers.
The logs-data leveraged comes from a diverse suite of development and communications tools used by engineers, including email, issue tracking, code repository, internal search, code review, and build/test tooling.
How developers experience flow
Developers were asked questions each day about whether they experienced flow that day, what they were doing, and how long they remained in flow. The researchers found:
Engineers experience flow across a range of tasks and durations, not just coding tasks.
Once established, an engineer’s flow can withstand some amount of distraction. Engineers reported being able to respond to messages from teammates and quickly resume their main task without breaking flow.
Engineers experience of flow is entangled with their judgment of the progress that they make. The same sets of behaviors might be labeled as “flow” or not as “flow” depending on whether they made progress on or finished a task.
Researchers also identified three practices for facilitating flow: 1) schedule management, or preventing dispersed meetings, 2) goal setting so engineers are working on tasks that feel fulfilling, and 3) giving time to “get into flow,” including setting up the workspace, playing music, etc.
These findings suggest that capturing flow states using logs-based metrics is potentially not feasible. With this in mind, the researchers expanded the scope to include focused work and adopted the conceptual model shown here:
The researchers state, “humans achieve flow states if and only if they are doing focused work (i.e., focus is an antecedent to flow), but that they can do focused work without achieving flow.”
Measuring focus time
In this section of the study, the researchers developed a definition for measuring focus time, and validated the metric by comparing it to the subjective experience of engineers.
The focus time metric relies on the concept of task similarity (i.e., performing a number of related actions in a given window of time indicates focus time, whereas performing a number of unrelated actions indicates a lack of focus). This required researchers to define task similarity and train a model to identify similar tasks from events (similar tasks may be editing or reviewing code, for example).
The metric was then validated using diary data. For the same time period where logs-based data was collected, participants were instructed to record the activities they completed. After completing a task, participants would complete a diary record and indicate whether that activity felt in flow or focused.
The data from the logs-based focus time metric was then analyzed using an agreement-based approach with the data from the diary study. The results showed high agreement between focus time and self-reported flow or focus.
Researchers performed a second validation analysis using data from a quarterly survey. Developers were asked to reflect on the previous three months and respond to the question, “How often are you able to reach a high level of focus or achieve flow during development tasks?” (on a 5-point scale). Three quarters of survey results were used, and the surveys used sampling methods, representing a full census of Google’s developer population. Researchers found that the total number of focus sessions an individual had in a quarter was a positive predictor of self-reported flow.
Final thoughts
Flow has been defined as the “optimal experience” and is something leaders generally aim to increase. This study defines a validated metric for predicting flow: focus time.
One other aspect of this study that I find interesting: there’s constant debate about whether the developer productivity data that is self-reported is as accurate as metrics that are derived from our systems. In this study, researchers validated their logs-based data by comparing it to self-reported data.
That’s it for this week! Share your thoughts and feedback anytime by replying to this email.
If this email was forwarded to you, subscribe here so you don’t miss any future posts:
-Abi