Most enterprise "AI knowledge" projects stall in the same place, and it isn't the model.
The demo works. Someone asks a question, the system retrieves three documents, an LLM stitches them into a paragraph, and the room is impressed. Then it meets real users and the failure mode shows up immediately: the answer is fluent, sourced, and wrong — because the three documents contradict each other and nothing in the pipeline was responsible for noticing.
Organizations do not have a knowledge base. They have sediment. Six versions of a policy, four superseded and none saying so. A wiki last touched in 2019 that is authoritative for one section and dangerously stale for the rest. Institutional knowledge that exists only as the reason a particular person always overrides a particular field.
Retrieval is not the hard part
Vector search over a document pile is a commodity. The hard part is everything retrieval assumes has already happened:
- Provenance. Where did this claim come from, and who owned it?
- Supersession. Which of these two conflicting documents wins, and can the system tell?
- Scope. Does this policy apply to this business unit, region, contract type?
- Confidence. Is this a specification, or someone's Slack message that got exported into the corpus?
None of that is model work. It's knowledge engineering — the unglamorous structuring step that determines whether the model has anything true to reason over. This is where knowledge graphs earn their keep over pure vector search: a graph can represent "supersedes" and "applies-to" as first-class edges. An embedding just knows the two documents are similar, which in this case is exactly the wrong conclusion.
Traceability as a design constraint
The systems worth building are the ones where you can follow the reasoning back. Not because interpretability is a virtue in the abstract, but because an answer you cannot trace is an answer you cannot debug, and a system you cannot debug gets quietly abandoned within two quarters of launch.
ELIZA seemed intelligent in the 1960s and was doing string substitution the whole time. The lesson isn't that it was fake — it's that surface fluency tells you almost nothing about the mechanism underneath, and if you don't instrument the mechanism you will be surprised, at scale, later.