RAG (retrieval augmented generation) means: before the AI answers, a system finds the relevant passages in your documents. The AI writes its answer from those passages, not only from its training. A small business often does not need a system of its own for this, as long as the documents fit into the chat.
A RAG system only pays off with a very large number of documents that change often, or when many people with different permissions need to look things up in them. My AI employees work with knowledge in files: each one has a folder of its own with a personnel file, rules and its own knowledge files. That is a simpler route than RAG; why it is enough for many is in Why ChatGPT Forgets and AI Employees Don't. All the basics about AI are collected on my AI Basics page.
What RAG means
The three words describe the process:
- Retrieval means fetching: the system searches for matching passages.
- Augmented means extended: those passages are added to the question.
- Generation means producing: the language model writes the answer from them.
A language model, a large language model or LLM for short, is the program behind chats such as ChatGPT or Claude. Without RAG it answers from memory, from what it saw during training. With RAG it answers more like in an open-book exam: it gets the relevant pages put in front of it.
Patrick Lewis and colleagues coined the term RAG in a research paper in 2020. Anthropic, the company behind Claude, describes RAG today as a technique that combines information retrieval with text generation. The goal: more accurate answers that rest on evidence. The data is retrieved only when the question is asked, and then lands in the model's context window, the area it has in front of it right now. I explain what the context window is in my article on the context window.
How a RAG system works
Anthropic describes the typical setup in two phases:
Once, in advance, the preparation:
- The documents are split into small chunks, usually no more than a few hundred tokens. A token is a small unit of text, often a word or part of a word.
- A separate model turns each chunk into a series of numbers that captures its meaning. That series of numbers is called an embedding.
- The embeddings go into a vector database. That is a database you can search by similar meaning, not only by identical words.
With every question:
- The system looks for the chunks whose meaning best matches the question.
- Those chunks go to the language model together with the question.
- The model writes the answer. Good systems show where it came from.
One catch that Anthropic names itself: searching by meaning sometimes misses exact matches, such as a specific error code. That is why Anthropic recommends combining it with a classic keyword search.
An example, made up for illustration: a heating installer with eight people has thousands of pages of manufacturer documents, maintenance manuals and warranty terms. A technician asks on their phone: "What service interval applies to boiler X?" The system finds the passage in the manual, and the AI answers with the page reference. Without RAG, the model would guess from its general knowledge, and that is exactly where errors creep in. Why AI guesses in that case is covered in AI Hallucinations: Why AI Makes Things Up.
When a small business needs RAG and when it does not
In 2024, Anthropic published a clear rule of thumb: if your knowledge base is smaller than 200,000 tokens, which Anthropic puts at about 500 pages, you can simply include all of it, without RAG. The figures refer to the models of that time, so check the limit for your own tool. The principle still holds: whatever fits into the context needs no search in front of it.
Some tools already do RAG by themselves. In Claude, a project is a workspace where files and instructions are ready for every chat in it. In Claude projects, according to Anthropic, Claude automatically switches to a RAG mode as soon as the project knowledge approaches the limit of the context window. Then up to ten times as much fits in, and Claude searches the project knowledge with a search tool of its own. This is available on the paid plans Pro, Max, Team and Enterprise.
The table is my recommendation, not a vendor rule. More detail on when a maintained list of frequent questions is enough is in AI Knowledge Management for Companies.
RAG, fine-tuning and knowledge in files
Three approaches often get mixed up.
The first two rows follow Anthropic's glossary. It also states that the Claude API, the interface for developers, does not currently offer fine-tuning. The third row is my own approach and technically the simplest form: the role reads its files itself with every assignment, with no database in front.
What to watch out for
- The answer is only as good as the collection. Anthropic itself says that the effectiveness of RAG depends on the quality of the knowledge base. Outdated or contradictory documents produce outdated or contradictory answers. So tidy up first.
- RAG lowers the risk of invented answers but does not remove it. A model can stretch a match too far or pick the wrong passage. That is why every answer should come with its source.
- Permissions before technology. If not everyone may see everything, the search itself has to filter by permission. Microsoft also names this as a challenge of its own: users may only retrieve content they are authorised to see. A model that has read a source can give it away in an answer. How to plan permissions and separate knowledge spaces is in Build an AI Knowledge Base From Calls.
- Clarify the data path. With confidential documents, the first question is where they are processed. The trade-off between the cloud and your own hardware is in Local AI and Privacy: Cloud or Server?
Frequently asked questions
What does RAG stand for?
RAG stands for retrieval augmented generation: producing an answer after looking things up first. Before answering, the AI gets relevant passages from your documents and writes its answer from them.
What is a RAG system?
A system with two parts: a search that finds relevant chunks in your documents, and a language model that writes the answer from them. Usually there is also a vector database in which the chunks are stored by meaning.
Does a small business need a RAG system?
Usually not right away. As long as your documents fit into the chat or a project, you pass them along directly. A system of your own pays off with very many documents that change often, or when many people with different permissions search them.
What is the difference between RAG and fine-tuning?
RAG passes relevant documents to the model with every question; the model itself stays the same. Fine-tuning trains the model further with additional data. According to Anthropic, RAG helps with up-to-date information and source citations, fine-tuning more with a domain, a task or a writing style.
Does Claude use RAG?
In projects, yes, automatically: when the project knowledge approaches the limit of the context window, Claude switches to a RAG mode according to Anthropic and searches the knowledge with a tool of its own. This applies to the paid plans.
Where to go from here
Write down the ten questions that come up again and again in your business, and check whether the answers are already in your documents. If those documents fit into a project, start there. You can still build a system of your own later. Before that, AI Hallucinations: Why AI Makes Things Up is worth a read, because that is exactly what RAG helps against. You will find every technical term from this article in my AI glossary. How others prepare their knowledge for AI is what we discuss in my community Claude Practitioners.
