There is a specific failure I keep watching organizations walk into with agentic systems, and it has a sixty-year-old name.
ELIZA ran on a machine with less memory than a modern doorbell. It matched strings against patterns and reflected them back as questions. People confided in it. Some of them, knowing exactly how it worked, still preferred to be left alone with it. Weizenbaum found this so unsettling that he spent much of the rest of his career arguing against the field he had helped start.
The usual reading is that people are gullible. I think that's the wrong lesson, and taking the wrong lesson is what causes the failure.
Fluency is not evidence about mechanism
The right lesson is that surface fluency carries almost no information about the process underneath it. ELIZA's output was locally plausible and its mechanism was fully traceable: you could read the transcript and the rule that fired and see the whole causal chain. The gap between how competent it seemed and how it worked was enormous, and it was invisible from the outside.
We have not fixed this. We have inverted it. Modern systems have genuinely more capable mechanisms and far less traceable ones. The gap is still there; we've just lost the ability to measure it. When a system produces a fluent, well-cited, confidently-wrong answer, nothing in the output distinguishes it from the same system producing a fluent, well-cited, correct one. That is not a prompt problem. It is a structural property of judging processes by their products.
Which means the engineering question is not "does this look right in the demo." It's "what would I have to instrument to find out."
Sparse and strategic, not big and brute
Fluid Concepts and Creative Analogies makes a claim I keep returning to: what general intelligence seems to do is recognize fluid, abstract patterns and reach for strategic analogies through sparse, parallel activation. Not exhaustive search. Not brute-force computation at massive scale. Something that gets to a good-enough structural match fast, on very little, and then commits.
I'm not going to claim that settles anything about how to build systems; the book is a research program, not a blueprint, and the honest position is that we don't know which parts transfer. But it makes a prediction worth holding onto: scale and analogy-making are different axes. A system can get arbitrarily better at the first while barely moving on the second, and because the first improves benchmark scores, you can spend years optimizing in the wrong direction while every dashboard says you're winning.
The practical version of this shows up constantly. A system that has memorized a million instances of a pattern and a system that has grasped the pattern behave identically until you hand them the case that isn't in the million. Then they diverge completely. And "the case that isn't in the million" is not an edge case in production. It's most of Tuesday.
What I actually build for
This leaves me with a few working commitments. None of them are exotic.
Traceability is a feature, not a research luxury. An answer you cannot follow back is an answer you cannot debug, and a system you cannot debug gets quietly abandoned about two quarters after launch, usually without anyone writing down why. Retrieval provenance, tool call logs, and intermediate state are not overhead. They are the difference between a system you can own and a system you can only hope about.
Evaluate on the distribution you'll actually see. Benchmark performance is a measurement of the benchmark. If your held-out set is drawn from the same pile as your training data, you have measured memorization and called it generalization.
Structure the knowledge before you reason over it. This is the least glamorous item and the one that decides the outcome. Most enterprise "AI knowledge" efforts stall because the corpus is sediment: six versions of a policy, four superseded, none saying so. No amount of model quality resolves a contradiction that nothing in the pipeline was assigned to notice.
Be suspicious of your own impressed reaction. If a system's output makes you feel like it understands, that feeling is data about you, not about the system. Weizenbaum's colleagues were domain experts. It didn't help them.
The uncomfortable part
I want these systems to work. My whole reason for going deep on computation was that a single brain cannot hold the coupling between gluons and populations, and I still believe an external substrate is the only way anyone holds a structure that large.
But wanting it is exactly the condition under which you stop checking. The people most likely to accept a plausible-looking answer are the ones who needed it to be true. That was as true in 1966 as it is now, and the machines have gotten much better at producing plausible-looking answers while our instinct for detecting understanding has not improved at all.
So: build the thing, and instrument it like you expect it to fool you. Because you're the easiest person in the room to fool.