RAG vs fine-tuning: choosing the right approach

"Should we use RAG or fine-tune a model?" is a question we hear often, usually framed as a single choice between two competing techniques. It is worth stopping on that framing, because the two solve different kinds of problem, and the question itself often reveals that neither is actually what the project needs yet.

What each one actually changes

RAG changes what information the model has access to at the moment it answers a question. The model itself is untouched — you are feeding it relevant context pulled from your own data, on every query, and it reasons over that context using its existing general capabilities.

Fine-tuning changes the model itself — its weights are adjusted using a training process on examples you provide, so its default behaviour shifts permanently. The model does not look anything up at query time; whatever it learned during fine-tuning is baked in.

That difference in mechanism is what should drive the choice, not a general sense that one is "more advanced" than the other.

When RAG is the right tool

• The need is knowledge, not behaviour — the model needs to know specific facts (your product catalogue, current policies, internal documents) that change over time.
• You need traceability — the ability to show which source document an answer came from.
• Different users should see different information, based on access control over what gets retrieved.
• The underlying data changes frequently, and retraining a model every time it does is impractical.

When fine-tuning is the right tool

• The need is behaviour or style, not knowledge — you want the model to consistently respond in a specific tone, format or structure, across many different topics.
• You need the model to reliably perform a narrow, well-defined task (a specific classification or extraction job) more accurately or more cheaply than a general-purpose prompt achieves.
• The required knowledge is stable and well-represented in a reasonably-sized set of training examples, rather than a large, changing knowledge base.

When you need both

Some systems genuinely benefit from both — a fine-tuned model that has learned your company's specific tone and response format, combined with RAG to supply it with current factual information it was never trained on. This is more engineering effort than either approach alone, so it is worth confirming the project actually needs both before committing to the combination.

When you need neither

This is the answer more often than either technique's proponents like to admit. A well-written system prompt, a curated set of few-shot examples embedded directly in the prompt, or — most commonly — a conventional rules engine or database query, solves a meaningful share of problems that get initially described as "AI problems." See our piece on where AI automation actually works for how to tell the difference before committing a budget to either approach.

A practical way to decide

Ask: "If I wrote down the answer to this question today and it changed next month, would the system need to know the new answer automatically, or is it acceptable to retrain/redeploy when it changes?"

If the answer needs to update automatically and the underlying knowledge changes often — that is a retrieval problem, and RAG is the natural fit. If the answer is really about *how* the model responds rather than *what* it knows, and that behaviour is stable — that leans toward fine-tuning, if a well-crafted prompt genuinely cannot achieve it.

If you are unsure which category your specific project falls into, describe it to us and we will give you a direct read on which approach — or neither — actually fits.