A language model has read an enormous amount. Your price list was not part of it. Neither was the 2019 service contract, the internal note on returning custom builds, or the folder that records which customer received which machine. Ask the model anyway and you will almost always get an answer. It reads well. It is invented.
This is where most companies have their first disappointment with language models. It is also where Retrieval-Augmented Generation, usually shortened to RAG, comes in.
Search first, answer second
The idea is unglamorous, which is exactly why it holds up. Before the model is allowed to write a single sentence, the system searches your own documents. The passages that match the question are pulled out and handed to the model as context. Only then does it write an answer, using nothing but what it was given, and it names the source.
An all knowing machine that knows nothing about you becomes a colleague who can read your filing system and tells you where the answer came from. Small difference in the plumbing, very large difference in trust.
The chain, without the jargon
First the documents get cut up. A three hundred page manual is split into manageable passages, because nobody, model included, can use three hundred pages as context at once. Each passage is then turned into a row of numbers, a vector. The vector does not describe the words, it describes the meaning. That is why the search still finds the right passage when somebody asks about a warranty and the document says guarantee.
When a question arrives, it is turned into a vector too, and the system looks for the closest passages. Those go to the model along with the question and one blunt instruction: answer only from this, and if it is not in here, say so.
Where it goes wrong in practice
Not in the model. Almost never in the model. It goes wrong in the documents. If three versions of the same price list sit on the shared drive and none of them carries a date, the system will retrieve the wrong one, and it will do so reliably. If a rule lives only in the heads of two employees and was never written down, the search finds nothing, and no amount of clever technology fixes that.
Which brings us to the most honest sentence we can offer on the subject. A RAG system does not repair a mess of undocumented knowledge. It makes the mess visible, usually in the first week, usually in front of an audience.
Who is allowed to see what
The second thing people underestimate is access rights. Searching across every company document is convenient, right up until the intern asks about salaries and receives a helpful, well sourced answer. Rights have to sit on the search itself, not on the interface. The system searches only what the person asking would already be allowed to open. Anything else is a data leak with a chat window attached.
Should an answer ship without a source?
That leaves the question of what happens when nothing relevant is found. Technically the model could still answer from its general knowledge. We advise against it. A system that cites its sources nine times out of ten and quietly invents the tenth is worse than no system at all, because nobody knows which sentence to trust. Better a plain I cannot find anything on that. It is a useful answer. It leads to a phone call instead of a mistake.
What this could look like in your own company is something we work through in our AI consulting.
Frequently asked questions
Do we need our own trained model?
Usually not. Your knowledge reaches the answer through search, not through training. That is faster, considerably cheaper, and has the pleasant side effect that a corrected document takes effect immediately.
Do our documents stay in house?
That depends on where the search and the model run. Both can be operated in your own data centre. Ask this early, because it shapes the architecture and cannot be bolted on at the end.
How many documents make it worthwhile?
It gets interesting the moment people start asking each other where something is written down. Depending on the company, that happens at a few hundred files.
If you want to judge whether your knowledge base is ready for a step like this, let us talk it through, with no obligation and no sales pressure.
This article is part of our knowledge hub AI and digitalisation for mid sized companies.