The fake citation isn't the bug, it's the smoke. A model produced a plausible string, sure — but then a human reviewed it, signed it, and filed it as fact. That's three links in the chain, and only one of them is made of math.
The vocabulary point is the one that'll stick with me. "Code smell" earns its keep because it gives you something to point at in a review without having to argue about it from scratch. Right now the entire context conversation is "the agent went weird again," which is not a diagnosis, it's a shrug.
All three, and the other two don’t even throw an error. The model gets the reputation for lying while the review and the filing just nod along and make it official.
Weekend runaway is the one that gets budget approved, and it has the same shape as a security failure. An agent with no definition of done and no ceiling keeps going until the invoice or the incident stops it. The fix I use in my lab is dull: a token budget per session, and a cost alert that a human has to clear before the agent continues. Your new-colleague test catches it too, because a new hire stops and asks.
Spent decades building context (barely good enough) into our software development process. Given human beings aren't really good at writing everything down, it's not surprising to me that the AI has problems with context.
Code smell takes time to learn. It's not like a brand-new programmer (i.e., AI) can look at some code and say, "That doesn't look quite right."
Your review example makes me want to compare the original task with what the agent sees when it starts reviewing. The reason for a change might have been supplied at the beginning and then omitted during summarization. In that case, improving the original specification alone wouldn't fix the review.
This is bang on and why spec driven development works so well with agents. Give an agent a well specified task with clear expectations of architecture and the output is pleasing. A vague prompt without context just brings frustration. All that remains is verification, the other tricky problem!
The fake citation isn't the bug, it's the smoke. A model produced a plausible string, sure — but then a human reviewed it, signed it, and filed it as fact. That's three links in the chain, and only one of them is made of math.
The vocabulary point is the one that'll stick with me. "Code smell" earns its keep because it gives you something to point at in a review without having to argue about it from scratch. Right now the entire context conversation is "the agent went weird again," which is not a diagnosis, it's a shrug.
Three links in the chain, and they all broke!
All three, and the other two don’t even throw an error. The model gets the reputation for lying while the review and the filing just nod along and make it official.
Weekend runaway is the one that gets budget approved, and it has the same shape as a security failure. An agent with no definition of done and no ceiling keeps going until the invoice or the incident stops it. The fix I use in my lab is dull: a token budget per session, and a cost alert that a human has to clear before the agent continues. Your new-colleague test catches it too, because a new hire stops and asks.
Dull but effective, is still effective.
Dull is how I know it’ll still be on in a month. The clever controls get switched off the first time they get in the way of a deadline.
Spent decades building context (barely good enough) into our software development process. Given human beings aren't really good at writing everything down, it's not surprising to me that the AI has problems with context.
Code smell takes time to learn. It's not like a brand-new programmer (i.e., AI) can look at some code and say, "That doesn't look quite right."
Your review example makes me want to compare the original task with what the agent sees when it starts reviewing. The reason for a change might have been supplied at the beginning and then omitted during summarization. In that case, improving the original specification alone wouldn't fix the review.
I also wrote a tutorial on context management, now at 900+ GitHub stars: https://github.com/hardness1020/learn-agent-architecture/tree/main/sections/08-context-management
This is bang on and why spec driven development works so well with agents. Give an agent a well specified task with clear expectations of architecture and the output is pleasing. A vague prompt without context just brings frustration. All that remains is verification, the other tricky problem!