RAG vs Fine-tuning

The quickest diagnostic costs an afternoon. Collect twenty or thirty real cases where the model gets it wrong. For each one, paste the information it would need straight into the prompt by hand, then run it again. If the answers become right, the problem is knowledge: the model can do the task once it can see the facts, and retrieval is how you put the facts in front of it automatically. If the answers are still wrong with the correct information sitting in the prompt, the problem is behaviour, and the next move is better instructions, then, only if those fail, training.

Knowledge problems have recognisable symptoms. The model states a price, a policy or a procedure that is wrong or out of date. It invents a plausible feature your product does not have. It says it does not know something your documentation plainly covers. It answers well about anything public and badly about anything internal. Every one of these is about what the model can see, and no amount of training on example answers fixes that durably, because next month the facts will have moved.

Behaviour problems look different. The facts are right but the answer is in the wrong format, too long, in the wrong register, or inconsistent between similar inputs. A classifier puts the same kind of message in different categories on different days. Extraction misses fields that are plainly present. The model ignores a convention that matters to you, however many times the prompt repeats it. Or the output is fine but each call is too slow or too expensive, because the prompt has grown into pages of instructions and examples. These are what fine-tuning addresses, and what prompting often addresses more cheaply.

Many real systems have both, and the order matters. Fix the knowledge problem first, because you cannot judge a model’s behaviour on inputs that are missing the facts it needed. Once retrieval reliably supplies the right passages, the errors that remain are behavioural, and you have a clean picture of what training, or a better prompt, would need to change.

Freshness is where the two differ most plainly. A retrieval system is as current as its index, and a pipeline that re-indexes changed documents, removes deleted ones and keeps versions straight means the next answer reflects today’s content. A fine-tuned model is a snapshot of its training data. It knows nothing about what changed afterwards, and updating it means new examples, a new training run, new evaluation and a new deployment. For information that changes every week, that cycle never catches up.

Citations follow from the same mechanism. A retrieval system knows exactly which passages it put in front of the model, so it can show them, link to them and let a reader check the claim. That changes how much trust an answer deserves and how quickly a wrong one gets caught. A fine-tuned model cannot point to the training example a statement came from, because the knowledge has been blended into its weights. You can train a model to produce citations in a particular format, but only retrieval gives it something real to cite. Evaluate the citations as well: a citation that does not support the sentence it is attached to is worse than none, because it lends false authority.

Retrieval reduces hallucination without eliminating it. Models still misread a passage, merge two passages into a claim neither makes, or ignore the context and answer from training. The controls are well established: instruct the model to answer only from the supplied passages and to say plainly when they do not contain the answer, require a citation for each claim, check automatically whether each claim is supported, and show users a clear "not found" rather than a guess. Fine-tuning gives much weaker control over facts. Training a model on facts does not make it recall them reliably, and published research suggests that training on information the model did not already know can make it more willing to state things it does not know. Use fine-tuning to shape how a model answers, and retrieval to decide what it answers from.

Most of the engineering in a retrieval system is the retrieval. Documents are split into chunks that follow their structure, each chunk is turned into an embedding and stored, and at query time the system finds the chunks closest in meaning to the question. Vector databases do this at scale, either as managed services such as Pinecone or as an extension to a database you already run, such as pgvector for PostgreSQL. Pure semantic search handles exact terms like product codes and names poorly, so good systems combine it with keyword search, filter on metadata such as date, product and permission, and rerank the candidates before choosing what goes into the prompt. A small collection may need no dedicated vector store at all.

For retrieval, data preparation means turning documents into reliable, retrievable passages. Text has to be extracted from PDFs, web pages, spreadsheets, slide decks and sometimes scans, with tables and headings preserved well enough to keep their meaning. Navigation, boilerplate and repeated footers have to go. Chunks have to follow the structure of the document so that each passage carries its own context. Each chunk needs metadata: source, date, owner, version and who is allowed to see it. Then comes the uncomfortable part. When two documents disagree, the system answers from whichever it happens to retrieve, so contradictory or outdated content has to be resolved, and that is an editorial task rather than an engineering one.

For fine-tuning, data preparation means building a set of examples, each an input paired with the ideal output, that represents the task as it will really arrive. Consistency matters more than volume: if two experts would label the same case differently, the model learns the disagreement. The set needs the hard cases and edge cases, not only the easy majority, and a held-out portion the model never trains on, so you can measure it honestly. Writing or reviewing those examples takes the time of people who know the domain, and that time is usually the largest cost in the project. Generating candidate examples with a larger model and having experts review them can help, but check the terms of the model whose outputs you are training on, because some providers restrict that use.

Most of this is ordinary data engineering. Parsers, cleaning scripts, scheduled pipelines, deduplication and validation are typically written in Python, and frameworks such as LangChain provide document loaders, text splitters and integrations with the common vector stores and model APIs, which make a first version quick to assemble. As requirements sharpen, teams often replace parts of a framework with their own code where they need tighter control over chunking, ranking or permissions. That is a normal evolution and nothing to avoid.

Neither approach is a one-off effort. A retrieval pipeline runs for as long as the sources change, and it needs monitoring for failed parses, stale indexes and permissions that have drifted out of sync with the source systems. A fine-tuning dataset needs maintaining as the task evolves, and the tune has to be redone whenever you move to a new base model or the provider retires the one you tuned. In most generative AI development projects, the data side of either approach takes more effort than the model side, and plans that budget the other way round tend to run late.

Hosted models are billed per token, usually with input and output priced separately. Fine-tuning on a hosted platform generally adds a training charge based on the volume of data processed, then bills inference on the tuned model, sometimes at a different rate from the base model. Managed vector stores are usually priced on usage, by storage and queries. Self-hosting an open-weights model swaps per-token charges for the cost of the hardware or cloud instances, which you pay whether or not they are busy, plus the people to run them. Rates, and the list of models that can be tuned at all, change often, so check current rate cards and model your own traffic against them.

Retrieval’s costs are front-loaded into engineering and then spread thinly across each request. Every call carries retrieved passages, so input tokens per call go up. Embeddings must be computed for every chunk when it is indexed and for every query. Storage and the search service cost something, usually modest next to the model calls themselves. The levers are retrieving fewer, better passages through reranking, caching the stable part of the prompt where the provider supports it, and using a smaller model when the retrieved passages make the task easy.

Fine-tuning’s costs concentrate in data preparation and evaluation, then in each training run, and there are usually several runs before one is good enough. The return comes per call: a shorter prompt, because the instructions and examples now live in the weights, and possibly a smaller, cheaper, faster model doing the job. That return scales with volume. At low volume, it rarely repays the preparation. At high volume on a narrow, stable task, it can, and that is the situation in which fine-tuning is most clearly the economic choice.

Count the costs that arrive later. A retrieval system needs its pipeline maintained and its index kept healthy. A fine-tuned model is tied to its base model, and when that base is superseded or retired, the tuning, evaluation and deployment are repeated. Retrieval moves to a new model with a configuration change and an evaluation run. Over a few years, that portability is often worth more than the per-call saving a tune delivered.

With retrieval, your documents stay in your own store, in a location you choose, and only the passages relevant to a given question are sent to the model at query time. That still means passages leave your environment when the model is hosted, so the provider’s terms on retention, on whether API data may be used for training, and on where processing happens all apply. Those terms differ between providers and between products from the same provider, and they change, so read the current ones and choose the contract and region accordingly.

Retrieval can be permission-aware, and in any organisation with access controls it has to be. Store each chunk’s access rules alongside it, filter the search by the identity of the person asking, and keep those rules in sync when permissions change in the source system. Do not rely on instructions in the prompt to withhold information: anything placed in the model’s context should be assumed retrievable by a determined user. The safe rule is that a question can only ever retrieve what the person asking could have opened directly.

Fine-tuning has no equivalent. Everything in the training set becomes, in some diffuse form, available to everyone who can query the model, and there is no way to make one fact visible to one team and hidden from another. Removing something later generally means retraining without it, which is awkward when a person exercises a right to have their data deleted. Hosted fine-tuning also means uploading your training set to the provider, and for closed models the tuned model stays on the provider’s platform. Strip personal data from training sets unless the task genuinely needs it, and treat the training set as sensitive in its own right.

Where data must stay in a particular jurisdiction or inside your own infrastructure, both approaches can be built to comply. The options include regional processing offered by some providers, models hosted in a chosen region through the major cloud platforms, and open-weights models run entirely on your own infrastructure, with retrieval and any fine-tuning happening there as well. The trade-off with self-hosting is operational: you take on serving, scaling and upgrades, and the open models you can run yourself may be less capable than the largest hosted ones for some tasks. Weigh that against the constraint, rather than assuming the constraint is negotiable.

Start with the prompt. Many problems blamed on the model come from instructions that are vague, contradictory or missing the examples that would show the model what good looks like. Prompt engineering done properly is plain, methodical work: state the task and the audience, set out the rules and the edge cases, include a few worked examples of good output, and say what to do when the input does not fit. Version the prompt, keep a test set, and measure each change. A team that has done this rigorously is in a far better position to decide whether training is needed, because it knows what prompting can and cannot do for its task.

Use structured output for format problems. If the complaint is that the model returns the wrong shape, such as missing fields, invalid JSON or free text where a category was needed, constrain it. The major model APIs, including OpenAI and Claude, can hold output to a JSON schema, either directly or through tool definitions, and your code can validate the result and retry on failure. That solves most format problems without any training, and it is often the whole of what people hoped fine-tuning would fix.

Try a smaller model, and route between models. A well-prompted small model handles many classification, extraction and drafting tasks adequately, with lower latency and cost. Where some inputs are harder than others, send the easy ones to a small model and escalate the rest. If your prompts are long because of stable instructions and reference material, prompt caching, where the provider offers it, reduces the cost of repeating them. Each of these attacks the cost and latency arguments for fine-tuning directly, without a training pipeline to maintain.

And before building a retrieval pipeline, check whether you need one. Current models accept long contexts, and if your reference material is a handful of policies or a single product manual, putting all of it in the prompt, with caching for the repeated part, may be simpler and more accurate than chunking and searching it. The limits are real: cost grows with every call, models can attend less reliably to material deep inside a very long prompt, and there is no per-user filtering unless you choose what to include. When the material outgrows the context, or access varies by user, retrieval earns its complexity. Fixing the source documents, so they are correct, current and consistent, helps every one of these options.

The two combine naturally because they change different things. Retrieval supplies the facts, and a tuned model can be better at using them: answering in the house format, citing consistently, declining when the passages do not contain the answer, and keeping a steady tone. A less obvious combination is often more valuable, which is fine-tuning the retrieval side rather than the generator. Training an embedding model or a reranker on examples from your own domain, pairs of real questions and the passages that answer them, can improve what gets retrieved, and better retrieval improves every answer downstream.

Teams who tried to fine-tune knowledge into a model and are moving to retrieval should start from the source material behind the training set rather than the training set itself. Index the original documents, build an evaluation set from real questions, and run the tuned model and a retrieval system with a base model against the same set. Keep the tune only if it adds something behavioural that the base model, with good retrieval and a good prompt, cannot match. The comparison usually settles the question quickly.

Moving the other way, adding fine-tuning to a working retrieval system, is the more natural direction, and the production system supplies the data. Every answer that a reviewer approved or corrected, together with the passages it was given, is a training example of exactly the behaviour you want, drawn from real traffic. Collect them deliberately from the start, with consent and personal data handled properly, and when the time comes to tune, the hardest part of the work is already done.

The sequence that tends to work is prompt first, then structured output, then retrieval with an evaluation set, then fine-tuning only for the behavioural problems that remain, using examples gathered from the system itself. Keep one evaluation set constant across all of these steps, so each change is measured against the same questions and you can tell real improvement from a different sample. Each step is cheaper to undo than the one after it, which is a sound argument for taking them in that order.