Choose RAG when the model needs facts it does not have, especially facts that change or need to be cited. Choose fine-tuning when the model already has the knowledge but you need to change how it behaves, its format, tone, or the shape of its output. The costly mistake, which I see repeatedly, is fine-tuning a model to teach it new facts. That is the one job fine-tuning does badly.
What each technique actually does
Retrieval-augmented generation, or RAG, does not change the model at all. At the moment of a question, it looks up relevant information from a knowledge source you control, and hands that information to the model alongside the question. The model then answers using text you placed in front of it. Think of it as an open-book exam: the model is the same, but it now has the right pages open.
Fine-tuning changes the model itself. You continue training it on examples of the behaviour you want, and its weights shift so that it responds differently by default. It is not learning a database of facts you can look up. It is adjusting its instincts: how it phrases things, what structure it returns, which style it adopts, how it handles a particular kind of task. Think of it as training an employee's habits, not handing them a reference document.
That distinction is the whole framework. RAG supplies knowledge at question time. Fine-tuning shapes behaviour ahead of time.
When RAG is the right choice
RAG is the correct tool whenever the problem is fundamentally about knowledge the model needs and does not reliably have. In practice that covers a large share of real business use cases.
- The knowledge changes. Prices, policies, inventory, documentation, this week's decisions. If the truth moves, you want a system where updating the answer means updating a document, not retraining a model. With RAG you change the source and the next answer is already current.
- You need citations. In any setting where an answer must be traceable to a source, RAG is close to mandatory. Because the model answered from retrieved passages, you can show the user exactly where each claim came from. A fine-tuned model cannot cite its training; it simply asserts.
- The knowledge is specific to you. Your internal documents, your product details, your customers' records. This information was never in the model's training and should not be baked into weights that are hard to update and hard to scope by permission.
- You need control over what the model can and cannot say. With retrieval you decide what information is in scope for a given user or query. That is far harder once knowledge is dissolved into weights.
When fine-tuning is the right choice
Fine-tuning earns its keep when the problem is about behaviour, not facts. The model broadly knows enough; you need it to act differently.
- Consistent format or structure. If you need output in a precise shape every single time, and prompting gets you most of the way but not reliably, fine-tuning on many examples of the exact format bakes that behaviour in.
- A specific tone or voice. A brand voice, a domain register, a particular way of writing that you cannot capture in a prompt without a wall of instructions. Examples teach it more reliably than description.
- A narrow, repeated task. Classifying, extracting or transforming in a domain-specific way, where you can show the model thousands of correct examples. Fine-tuning can make a smaller, cheaper model perform a narrow task as well as a large one, which is a real cost win at volume.
- Reducing prompt overhead. If your prompt has grown into pages of instructions and examples, fine-tuning can move that behaviour into the model, cutting per-request cost and latency.
The tell is simple. If you can describe the goal as "behave like this" or "always respond in this shape", fine-tuning is on the table. If the goal is "know this" or "answer from this", it is not.
The expensive mistake: fine-tuning to add knowledge
Here is the trap. A team wants the model to know their internal documents, so they fine-tune it on those documents. It feels intuitive: teach the model our stuff. It goes wrong in every way that matters.
Fine-tuning teaches a model a style of answering, not a reliable, retrievable set of facts. Ask it something specific and it will produce something that sounds right in the correct tone, with no guarantee it is true and no source to check.
Fine-tuning for knowledge produces confident, well-styled answers that are frequently wrong, because the model absorbed the shape of your documents rather than a lookup table of their contents. You cannot cite anything. When a fact changes you must retrain. And you have spent real money and effort to build something less accurate and less maintainable than RAG would have been. If the requirement contains the word "know", reach for retrieval first.
Cost and maintenance trade-offs
Beyond correctness, the two approaches have very different long-term shapes, and the running cost usually matters more than the build cost.
- Updating. RAG updates are cheap and instant: change the document. Fine-tuning updates mean assembling data and retraining, so your knowledge is only as fresh as your last training run.
- Cost profile. RAG adds cost to every request, because retrieval runs and retrieved text lengthens the prompt. Fine-tuning has an upfront training cost, then can lower per-request cost, especially if it lets you use a smaller model or a shorter prompt. At high volume on a stable task, that upfront cost can pay back.
- Debuggability. When RAG gives a bad answer you can inspect what it retrieved and usually see why. When a fine-tuned model misbehaves the cause is buried in weights and far harder to trace.
- They combine. This is not an either-or in the hard cases. Fine-tune a model so it behaves the way you need, then use RAG to feed it current, citable knowledge. Behaviour from fine-tuning, facts from retrieval. That pairing is often the right answer for a mature system.
The one-line rule
Fine-tune for behaviour, retrieve for knowledge. If the requirement is about how the model acts, consider fine-tuning. If it is about what the model knows, especially if that knowledge changes or needs a citation, use RAG. Start with RAG and good prompting, because they are cheaper to change, and reach for fine-tuning only when you have a clear behavioural need that prompting cannot meet.
If you are weighing these options for a real system and want an honest recommendation rather than a default, that is worth a proper conversation. See how we approach AI development or book a free consultation.