What is RAG? Retrieval-augmented generation explained

Retrieval-augmented generation — RAG — is the approach behind most AI features that need to answer questions using information specific to a business: a support assistant that knows your product documentation, an internal tool that can answer questions about your company's policies, a search feature that understands your actual catalogue.

The problem RAG solves

A large language model's knowledge comes entirely from its training data, which is both general and fixed at a point in time. Ask it a question about your company's return policy, and it has no way to know the answer — it was never trained on that document, and even if it had been, that document may have changed since.

The naive fix — paste the entire document into the prompt every time — breaks down fast. Real knowledge bases run to thousands of documents; no model's context window fits that, and even if it did, searching through everything on every query would be slow and expensive.

How RAG actually works

RAG splits the problem into two steps that happen on every query:

1. Retrieval. Given the user's question, search your own data — documents, database records, support tickets, whatever is relevant — for the pieces of information that are actually relevant to answering it. This search typically uses semantic similarity (via a vector database) rather than exact keyword matching, so it can find relevant content even when the user's wording doesn't match the source text exactly.
2. Generation. Pass the retrieved pieces, along with the original question, to the language model as context. The model then answers using that specific information, rather than relying on its general training knowledge.

The practical effect: the model's answers are grounded in your actual, current data, not its fixed training knowledge — and you can update what it "knows" simply by updating the underlying documents, with no retraining required.

What makes retrieval quality the real bottleneck

Most RAG systems that perform poorly in practice have a generation model that is working fine — the problem is upstream, in retrieval. If the system fetches the wrong documents, or misses the one that actually contains the answer, the model will confidently generate a wrong or irrelevant response regardless of how good it is.

This is why serious RAG work spends real effort on:

• Chunking strategy — how source documents get split into retrievable pieces. Too large and irrelevant content dilutes the match; too small and necessary context gets lost.
• Embedding quality — the model used to convert text into the vectors that get compared for similarity search.
• Re-ranking — a second pass that re-orders initial retrieval results by relevance before they reach the generation step, which often improves answer quality more than tuning the generation prompt does.

When RAG is the right approach

RAG fits naturally when:

• The answer depends on information that changes over time (prices, policies, inventory, documentation)
• The knowledge base is too large to fit in a single prompt
• You need the system to cite or point back to a specific source
• Different users or contexts should see different, access-controlled information

It is less suited to tasks that are really about behaviour or style rather than knowledge — teaching a model to always respond in a specific tone, or to follow a specific structured format consistently, is usually closer to a fine-tuning or prompt-engineering problem than a retrieval one. See our RAG vs fine-tuning comparison for that distinction in more depth.

A simple way to picture it

If you think of a language model as a very well-read employee who has never seen your company's internal documents, RAG is the equivalent of handing them exactly the three relevant pages from your handbook right before they answer your question — rather than expecting them to have memorised the entire handbook in advance, or trying to summarise the whole thing into their job training once and hoping it sticks.

If you are weighing whether a specific feature you are considering is actually a RAG problem, tell us what you are trying to build and we will give you a direct answer.