Does Experience Matter?
Does Experience Matter?
Years of experience correlates only slightly with programmer efficacy. Self-estimation and learning behaviors are more predictive.
This week I read Correlates of Programmer Efficacy and Their Link to Experience, a recent paper published by researchers at Saarland University in Germany. Despite similar education or backgrounds, programmers can exhibit vast differences in their abilities. This study sought to understand the relationship between programming experience and efficacy.
My summary of the paper
The researchers measured programmer efficacy by having developers complete coding tasks which were assessed for speed and correctness. To understand the link between programmers’ experience and abilities, several research questions were explored:
Do programmers with high efficacy read code differently than others?
Do programmers with different efficacy levels show different levels of cognitive load?
What is the correlation between efficacy and common measures of experience?
Here’s what they found:
1. Programmers with higher efficacy read code in a more efficient manner
Researchers used eye-tracking to understand how different programmers read code during task completion. They found that programmers with higher efficacy tended to have “shorter first fixations, shorter gaze duration, a much lower chance that a token is revisited, and a slightly higher probability that a token is revisited.” That is, these programmers quickly moved and focused on the most relevant parts of the code whereas programmers with lower efficacy spent more time studying the entirety of the code.
Reading code quickly helps programmers complete tasks faster, but can also lead to oversight and error. In prior research, “this was observed with experts missing obvious syntax errors that novices consistently notice.”
2. Programmers with higher efficacy experienced lower cognitive load
Researchers used EEG (electroencephalography) to analyze electrical activity in the brain and measure cognitive load while programmers completed tasks. Programmers with higher efficacy showed lower levels of cognitive load as well as fewer spikes where they were overwhelmed. As an aside, the authors noted that “due to the lower cognitive-load levels, programmers with high efficacy likely can sustain longer periods of work. By contrast, programmers with lower efficacy are more likely to be overwhelmed by constant spikes of cognitive load.”
3. Years of experience does not correlate with programmer efficacy, but self-estimation and indicators of learning eagerness do
The researchers correlated programmer efficacy with commonly used measures of experience such as time (years of programming, years of professional programming), self-estimation (self-reported ability and comparison to peers), and work activities per week (time spent on code reviews, testing, mentoring).
Surprisingly, years of experience showed only a small correlation with efficacy compared to other factors such as self-estimation in comparison to peers and “several questions that capture learning eagerness (time spent learning, mentoring, code review)”.
It is often assumed that more experience equates to greater ability, but this study disassociates the two. “Psychology describes this conscious effort to keep learning as deliberate practice”; learning doesn’t happen on its own. This aligns with the findings of another paper I recently reviewed, which found “continuous learning” to be a key attribute of great engineers.
Final thoughts
The practice of requiring a certain number of “years of experience” in hiring criteria has increasingly come under scrutiny. Critics argue that an individual’s skills can accelerate asymmetrically in certain jobs versus others, or that some skills (e.g., new programming languages) have not been around for long enough to meet the desired experience expressed by companies. This research gives weight to those arguments, while also providing different angles to consider.
That’s it for this week! If you know someone who would enjoy this issue, please consider sharing it with them.
As always, reach out with thoughts about this newsletter on Twitter at @abinoda, or reply to this email.
-Abi