As someone who ships AI-assisted code regularly, the survivorship framing is the part I'd underline. Percentage of AI-written code tells me nothing about whether the code survived review, tests, and production. What I actually watch: does the change pass the existing suite, does it introduce regressions my checks can catch, and does it hold up over time. I run eval harnesses with a few hundred sessions and deliberately degraded inputs to see what breaks, and that tells me far more than any adoption percentage. Measure what survives, not what's generated.
Thanks, Brian. Honestly your survivorship framing gave me the language for something I'd been doing without a name for it. Going to steal that for how I talk about my own evals going forward.
Thanks for raising this discussion. We tend to over-rely on metrics that are easy to measure and output-focused, which say little about value delivered/outcomes/real impact. AICP shares that limitation and is in that sense, very similar to a super-flawed metric, lines of code: a higher number doesn't mean a better outcome.
I like survivorship as a companion metric, but it doesn't tell us enough about quality. It only shows how much shipped code stays unmodified. As developers understand the codebase less (i.e., cognitive debt), they may also be less willing/likely to touch AI-generated changes, so code can survive simply because nobody dares to change it. Could it make sense to pair survivorship with rework rate and/or change failure rate?
Thanks for the very thoughtful reply, André! I also believe we assign too much importance to some metrics simply because they’re easy to capture. That’s been a “feature” of activity metrics forever. That’s not to say they don’t have value; we just need to understand their place in a broader measurement system.
I couldn’t agree more about the need to pair these diagnostic metrics with top-level outcome metrics like Change Failure Rate (which I think becomes an increasingly important quality signal as AI adoption grows).
If we are accumulating cognitive debt and becoming less confident in our ability to safely change code, the important question is whether that eventually manifests in worse outcomes. Pairing signals like survivorship or rework with CFR helps us understand whether we’ve actually balanced those concerns against the speed gains we’ve fallen in love with.
Thanks, Brian! Totally agree that usefulness of a metric depends a lot on how we leverage and contextualize them. I am looking forward to seeing how the research and practitioner's solutions on survivorship develops.
As a CS student, the two-teams-at-60% example is the part that stuck with me. The same percentage can mean autocomplete where the developer stays in the loop, or delegated agents where the first human judgment call happens at PR review. That makes me think the more useful question is where human review enters and how well people can spot code that is almost right but wrong. Jumping from 19% to 52% of merged code in under a year makes that skill matter more, not less. Thanks for framing AICP as a lens rather than a target.
I can see AICP continuing to be useful as teams are trying to build the discipline to be ON the loop rather than IN the loop.
It might be interesting to also slice it by type of asset. Harness configuration, spec files, vs code. Or any way that aligns with the intent/goal of the organization.
As someone who ships AI-assisted code regularly, the survivorship framing is the part I'd underline. Percentage of AI-written code tells me nothing about whether the code survived review, tests, and production. What I actually watch: does the change pass the existing suite, does it introduce regressions my checks can catch, and does it hold up over time. I run eval harnesses with a few hundred sessions and deliberately degraded inputs to see what breaks, and that tells me far more than any adoption percentage. Measure what survives, not what's generated.
I LOVE that approach to running evals.
Thanks, Brian. Honestly your survivorship framing gave me the language for something I'd been doing without a name for it. Going to steal that for how I talk about my own evals going forward.
Thanks for raising this discussion. We tend to over-rely on metrics that are easy to measure and output-focused, which say little about value delivered/outcomes/real impact. AICP shares that limitation and is in that sense, very similar to a super-flawed metric, lines of code: a higher number doesn't mean a better outcome.
I like survivorship as a companion metric, but it doesn't tell us enough about quality. It only shows how much shipped code stays unmodified. As developers understand the codebase less (i.e., cognitive debt), they may also be less willing/likely to touch AI-generated changes, so code can survive simply because nobody dares to change it. Could it make sense to pair survivorship with rework rate and/or change failure rate?
Thanks for the very thoughtful reply, André! I also believe we assign too much importance to some metrics simply because they’re easy to capture. That’s been a “feature” of activity metrics forever. That’s not to say they don’t have value; we just need to understand their place in a broader measurement system.
I couldn’t agree more about the need to pair these diagnostic metrics with top-level outcome metrics like Change Failure Rate (which I think becomes an increasingly important quality signal as AI adoption grows).
If we are accumulating cognitive debt and becoming less confident in our ability to safely change code, the important question is whether that eventually manifests in worse outcomes. Pairing signals like survivorship or rework with CFR helps us understand whether we’ve actually balanced those concerns against the speed gains we’ve fallen in love with.
Thanks, Brian! Totally agree that usefulness of a metric depends a lot on how we leverage and contextualize them. I am looking forward to seeing how the research and practitioner's solutions on survivorship develops.
As a CS student, the two-teams-at-60% example is the part that stuck with me. The same percentage can mean autocomplete where the developer stays in the loop, or delegated agents where the first human judgment call happens at PR review. That makes me think the more useful question is where human review enters and how well people can spot code that is almost right but wrong. Jumping from 19% to 52% of merged code in under a year makes that skill matter more, not less. Thanks for framing AICP as a lens rather than a target.
I think there’s an interesting dilemma here.
I can see AICP continuing to be useful as teams are trying to build the discipline to be ON the loop rather than IN the loop.
It might be interesting to also slice it by type of asset. Harness configuration, spec files, vs code. Or any way that aligns with the intent/goal of the organization.