I've seen AI dramatically increase engineering output, but I'm starting to question whether we're measuring the right thing. Faster delivery is valuable, but the bigger opportunity may be tackling problems we previously couldn't justify spending time on. That requires a different kind of judgment from both engineers and managers.
The survey gives a useful account of what developers feel able to attempt. To test the judgment risk, I would add a separate task: give reviewers a plausible change with a known defect, then ask them to explain and challenge it. That would distinguish increased output from retained ability to evaluate the output. The longitudinal comparison seems especially important here: self-reported new capabilities do not, by themselves, establish skill loss.
The electrification comparison is doing a lot of work there, and honestly it earns it — 75% more PRs while the share of time on new stuff moves from 57 to 59 is the most damning pair of numbers in the whole thing. Speed went up. What the work *is* barely budged. "Hollow orchestrator" is a good, unkind name for it: you can approve four hundred diffs a month and still be the person who can't tell which one quietly breaks auth.
Though I'd want the rest of those zone numbers before drawing the moral — the excerpt cuts off right when "further" gets interesting, and that's the bucket where the actual argument lives.
Thanks Nia! Admittedly, it was one of my co-authors who came up with the term, "hollow-orchestrator". I love it though, I think it is so haunting (but accurate).
As for cutting off when things get interesting, I wanted the newsletter to entice the reader to check out the full paper. Seems like it did the trick!
I've seen AI dramatically increase engineering output, but I'm starting to question whether we're measuring the right thing. Faster delivery is valuable, but the bigger opportunity may be tackling problems we previously couldn't justify spending time on. That requires a different kind of judgment from both engineers and managers.
The survey gives a useful account of what developers feel able to attempt. To test the judgment risk, I would add a separate task: give reviewers a plausible change with a known defect, then ask them to explain and challenge it. That would distinguish increased output from retained ability to evaluate the output. The longitudinal comparison seems especially important here: self-reported new capabilities do not, by themselves, establish skill loss.
Clever study design! That would be a good one.
The electrification comparison is doing a lot of work there, and honestly it earns it — 75% more PRs while the share of time on new stuff moves from 57 to 59 is the most damning pair of numbers in the whole thing. Speed went up. What the work *is* barely budged. "Hollow orchestrator" is a good, unkind name for it: you can approve four hundred diffs a month and still be the person who can't tell which one quietly breaks auth.
Though I'd want the rest of those zone numbers before drawing the moral — the excerpt cuts off right when "further" gets interesting, and that's the bucket where the actual argument lives.
Thanks Nia! Admittedly, it was one of my co-authors who came up with the term, "hollow-orchestrator". I love it though, I think it is so haunting (but accurate).
As for cutting off when things get interesting, I wanted the newsletter to entice the reader to check out the full paper. Seems like it did the trick!