RAG stands for retrieval-augmented generation. It finds the right document in your own files and hands it to the model before the model answers. None of that material was in the training data. Not your policies, not your customer history, not your job notes.
Why can’t a language model answer questions about your company?
A language model generates the next word. That makes it a decent assistant for drafting a blog post or tidying up an email. Ask it what your refund policy says and it has nothing of yours to work from, so it writes a plausible refund policy instead of your one. The paragraph reads fine. It is invented.
The obvious fix is to paste the policy into the prompt. That works for one document. It falls apart at hundreds of thousands of PDFs, where nobody knows which one holds the answer and the right one changes with the question.
RAG is a librarian. Your company is the library, hundreds of thousands of books on the shelves. You walk up and ask what the refund policy is. The librarian knows where to look, pulls the book down and hands it over. Now you have enough to answer the question, and so does the model.
How does RAG work?
Two phases. Ingestion files the documents where they can be found again, retrieval pulls them back out.
- Ingestion. A shipment of 50 new documents arrives and gets sorted across the store, each one wherever it belongs. A vector database does this part, and it is a lesson of its own.
- Retrieval. You ask about the refund policy. The system finds the three or four or five documents that cover it and passes all of them into the prompt.
- Generation. The model reads all five, works out which parts apply and which do not, and writes the final answer.
Nothing about the model changed between a bad answer and a good one. What changed is what was on the desk in front of it.
Does RAG stop hallucinations?
Think of it as an open book exam against a closed book one. Closed book is a model with no RAG. It has to produce something, and sometimes what it produces is invented.
What stops that is the instruction you send with the documents, which carries as much weight as the documents themselves. Do not create any new information. Tell us what you know and what you do not know. When the book is not on the shelf, the model comes back and says the answer is not there. In a closed book exam it could not do that.
RAG vs fine-tuning: which one?
Fine-tuning is what people used to do. Retrain the model on one company’s data, once, then reuse that model everywhere. RAG pulls the information at the moment somebody asks for it.
| RAG | Fine-tuning | |
|---|---|---|
| How it works | Pulls the relevant documents in at question time | Retrains the model on your data, once |
| Effort to set up | Quite easy to implement today | Very expensive |
| Data that changes daily | Picks the change up straight away | Needs another training run |
| Best for | Documents you want pulled when they are needed | Large data sets where the model has to learn patterns |
Data that changes daily is what decides most cases. Bring a book into the library today, take it out tomorrow, and the day after it is not there to be retrieved. A model fine-tuned last quarter still believes the old book is on the shelf. Around 80% of the time RAG on its own is good enough. In the 20% where it is not, fine-tuning is worth a look, and the typical case there is finding patterns across a large set of images.
When should you not use RAG?
General knowledge. Famous people, historical events, how most industries work. Those are average internet answers and the model already has them.
RAG is for the material only you have. Internal documentation. Company policies. Customer history. The commissioning notes from an install three years ago that the crew standing on the roof needs at 9am.
Write down the documents you would be embarrassed to have a model guess at, then check whether they sit anywhere a librarian could find them.
Behind every RAG system that works there is a vector database, the part that tells the librarian which shelf to walk to. That is the next lesson.
