RAG vs Fine-tuning

📺 Prefer to watch? 90-second YouTube Short · 💬 Telegram Originally published on software-engineer-blog.com. The single most expensive misconception in applied AI right now is that fine-tuning teaches a model your documents. It doesn't — and entire GPU budgets get torched on this one mistake. The real split is clean, but almost nobody frames it this way: knowledge versus behaviour. Mental model: RAG is a library you hand the model at query time; fine-tuning is teaching the model a writing style. You have a set of internal documents. You want the model to answer questions about them. Two paths show up: Fine-tune the model on those documents. Use RAG — retrieval augmented generation. Most teams pick option 1, expecting the model to "know" the docs. Six weeks later, on GPU credits they can't get back, they realize the model still hallucinates. Then they find out about RAG. Then they argue about which one to use. The argument ends when you stop conflating knowledge with behaviour. RAG doesn't change the model at all. It changes the prompt. Here's the flow: Your documents live in a search index (typically a vector database: Pinecone, Weaviate, Milvus, or a simpler inverted index). A user asks a question. You retrieve the K most relevant chunks from that index. You paste those chunks into the prompt, above or below the user's question. The model reads them and answers. The weights never change. Because the weights never change: Sources are citable. The model can say "according to page 47 of your policy manual." Facts stay current. When your pricing changes, you reindex the new document. You don't retrain. Hallucination is local. If retrieval fails (wrong chunk fetched), you get a confident wrong answer from that chunk. The failure is visible — you can debug the retrieval pipeline. The cost of RAG is the pipeline itself. You now own: A chunking strategy (how do you split documents so the model can read them?) Embeddings (how do you represent each chunk as a vector?) A vector store (where do chunks live, and how do you search them?) Re-ranking or filtering (do the top K results actually matter?) Latency and token overhead (every prompt now includes retrieved chunks). The weak link is retrieval. Fetch the wrong chunk and the model answers confidently from it. This is operationally messier than it sounds: your monitoring has to watch for semantic drift in retrieval quality, not just model accuracy. A change to your embedding model can silently degrade recall. Fine-tuning changes the model's weights. You feed it thousands of input-output pairs, and the model adjusts its parameters to predict those outputs given those inputs. Fine-tuning teaches style. Examples: "Always output valid JSON, never markdown." "Use our internal terminology: 'customer mandate' not 'contract'." "Never refuse; reframe instead." "Your tone is formal, clinical, concise." These patterns — the structural regularities in your training data — get absorbed into the weights. The model learns to emit outputs that match the shape of your examples. Fine-tuning does not teach new facts. This is the misconception that matters. Why? Because facts are not patterns. A fact is a specific piece of information: "Our API rate limit is 1000 requests per minute." When you fine-tune on documents containing that fact, the model doesn't store the fact. It learns correlations: tokens near "rate limit" tend to be followed by numbers in a certain range. Those correlations smear across the weight matrix. The model still hallucinates. It still gets the limit wrong half the time. And when you change the limit to 2000 requests per minute, there is no weight to edit — you have to retrain. Because weights are opaque, the model can't cite the source. It can't distinguish between what it learned during pre-training and what it learned during fine-tuning. Everything is probability. Dimension RAG Fine-tuning What you're teaching New knowledge (facts, data) Consistent behavior (style, format, tone) Does the model's weights change? No Yes Can the model cite sources? Yes (if retrieval includes source metadata) No How do you update when facts change? Reindex (days, sometimes hours) Retrain (weeks, GPU-intensive) Hallucination risk if retrieval fails High (wrong chunk, confident wrong answer) High (weights encode fuzzy patterns, not facts) Cost per inference Higher (chunk tokens in prompt) Lower (no retrieval overhead) Operational complexity Retrieval pipeline, embedding drift, chunk quality Training infrastructure, data labeling, version control If you're running a serving system, RAG and fine-tuning hit your latency budget differently. RAG adds latency to time to first token (TTFT). The retrieval call (embedding the query, vector search, maybe re-ranking) happens before you send the prompt to the model. On a 100ms embedding latency + vector search, you're looking at 150–300ms added to TTFT before the model sees a token. Then chunks in the prompt increase the time per output token (TPOT) because the KV cache is larger. Fine-tuning shifts latency to training time (offline). Inference is faster — shorter prompts, no retrieval. But you pay for retraining whenever behavior needs to change. If you need both low latency and up-to-date facts, RAG is the only option. If you can tolerate retraining cycles, fine-tuning for behavior + a smaller RAG pipeline (for critical facts only) can reduce TTFT. Reach for RAG when: Your knowledge changes (documents, prices, policies, product specs). You need sources. You want to debug failures. You can afford the retrieval pipeline. Reach for fine-tuning when: Your model's behaviour must stay consistent (output format, tone, terminology, refusal strategy). You're not adding facts; you're teaching a style. If you need both: Fine-tune the behaviour first. Then wrap the fine-tuned model in a RAG pipeline that retrieves facts. Fine-tuning should not carry the burden of knowledge management — it will fail at that job, and you'll waste time and GPU budget figuring out why. Watch the 90-second reel for the quick framing.