Retrieval-augmented generation looks easy in a demo. Load some documents, ask a question and a good answer appears. Then real users arrive with real questions, and it starts missing obvious answers, quoting outdated policies or confidently combining two unrelated documents.
The model is rarely the cause. Most of the work in a dependable system sits around it.
Fix retrieval before you touch the model
If the right passage never reaches the model, no model can answer correctly. So measure retrieval on its own: for a set of real questions, did the correct passage appear in the results? Common improvements include combining keyword and vector search, reranking results, and splitting documents along their real structure instead of at a fixed length. A table torn across two chunks is useless to everyone.
Treat your documents as a data problem
Company knowledge is messy. There are three versions of the same policy, scanned PDFs with poor text, and pages nobody has owned for years. Keep the source, version, date and owner with every passage, so that the system can prefer the current version and show where each answer came from. Sometimes the most useful outcome of a RAG project is learning which documents contradict each other.
Enforce permissions at retrieval
Don’t rely on the model to withhold something it has already been given. Filter by the user’s access rights before passages reach it. Otherwise, a helpful assistant can become a way to read another department’s documents. Test this directly, with accounts at different access levels.
Make refusing an acceptable answer
A system that always answers will sometimes invent one. Give it a clear path to say that the documents don’t cover something, and hand over to a person. Users trust a tool that admits its limits far more than one that is wrong in a confident tone.
Remember that documents can attack
Text inside a retrieved document can contain instructions aimed at the model, a problem known as prompt injection. This matters more once the system can take actions. Treat retrieved text as data, limit what the system can do and test with hostile documents.
Keep an evaluation set
Collect real questions with answers your experts have checked, including awkward ones and questions with no answer. Run them on every change to prompts, models or chunking. Without this, each improvement is only an opinion, and a fix for one question can quietly break ten others.
A sensible order of work
- Choose one user group and one body of documents.
- Collect real questions and build the evaluation set first.
- Clean the documents and record version and owner.
- Build retrieval with permissions, then measure it on its own.
- Add generation with citations and a clear way to refuse.
- Launch to a small group, and add their failed questions to the set.
If you have a pilot that stalled at the demo stage, we can help measure why.