RAG vs fine-tuning: a decision guide for CIOs (with the cases where each wins)
Every enterprise AI conversation reaches the same fork: do we teach the model our knowledge (fine-tuning) or hand it our knowledge at question time (retrieval-augmented generation)? The honest answer is that most organisations need retrieval first, fine-tuning rarely, and a clear reason before they do either. This is the guide we use in scoping calls.
What each one actually does
Retrieval-augmented generation (RAG) leaves the model untouched. When a user asks a question, the system searches your approved documents and data, retrieves the most relevant passages — respecting the user's permissions — and asks the model to answer only from those passages, citing them. Knowledge lives in your documents; the model is a reader and writer.
Fine-tuning changes the model. You train it on examples so it adopts a style, a format, a vocabulary or a narrow skill. Knowledge is baked into weights at training time, which means it is only as current as the last training run and it cannot show you where an answer came from.
That difference — can it cite its source? — is why regulated organisations almost always start with RAG. It is also why the grounded AI pattern we build on Google Cloud is retrieval-first by default.
When RAG wins
- The knowledge changes. Policies, prices, regulations, case files. Retrieval reads the current version; a fine-tuned model remembers last quarter's.
- You need citations. Compliance, legal, HR, government. "Section 4.2 of the 2026 policy" is a sentence RAG can say and fine-tuning cannot.
- Permissions matter. Retrieval can be permission-aware per user. A fine-tuned model knows what it knows for everyone.
- You want to start this month. A focused RAG assistant is a four-week pilot; our UN agency enterprise search and policy chatbot programmes are both retrieval-based.
When fine-tuning wins
- Style and format, at volume. Thousands of outputs a day that must follow a house style, a fixed schema or a domain vocabulary — fine-tuning makes the behaviour reliable and the prompts shorter.
- Narrow classification and extraction. Triage tickets into 40 categories, extract fields from one document type: small fine-tuned models are cheaper and faster than prompting a large one.
- Latency and cost at scale. A small fine-tuned model can replace a large general one for a single repetitive job.
- Language and tone. Urdu-English code-switching, regional phrasing, or a brand voice that prompting alone fails to hold.
Notice what is not on the list: "teaching the model our company knowledge". Fine-tuning is a poor, expensive way to store facts. Store facts in documents and databases, retrieve them.
Costs and risks, side by side
| Question | RAG | Fine-tuning |
|---|---|---|
| Time to first useful result | Weeks | Weeks to months (data labelling first) |
| Keeps up with changing content | Yes, automatically | No — retrain |
| Can cite sources | Yes | No |
| Respects per-user permissions | Yes | No |
| Main ongoing cost | Retrieval infrastructure, content ownership | Training runs, evaluation, re-training |
| Main risk | Bad retrieval → wrong context | Overfitting, stale facts, hard to audit |
The hybrid most enterprises end up with
Retrieval for knowledge, light tuning (or just careful prompting) for behaviour. A typical production stack: a governed data layer like the one in our BigQuery foundation case study; permission-aware retrieval with citations; an evaluation set that tests accuracy, declines and leakage; and, only where a repetitive narrow job justifies it, a small fine-tuned model for classification or extraction — the pattern in our contract intake work. For the mechanics of grounding on Google Cloud, Google's own grounding overview is the reference.
A five-question checklist
- Does the answer need a source? → RAG.
- Does the knowledge change more than once a quarter? → RAG.
- Do different users have different access rights? → RAG.
- Is the job narrow, repetitive and high-volume with a fixed output? → consider fine-tuning a small model.
- Can you write 100 real test questions with correct answers today? If not, neither approach is ready — start with the evaluation set.
FAQ
Is RAG cheaper than fine-tuning?
Usually, for knowledge use cases. RAG needs retrieval infrastructure and someone to own the content; fine-tuning needs labelled data, training runs and re-training whenever facts change. For narrow high-volume tasks, a small fine-tuned model can be cheaper to run than prompting a large model.
Can we fine-tune a model on our company documents instead of RAG?
You can, but it is a poor way to store facts: the model cannot cite sources, cannot respect per-user permissions, and goes stale as documents change. Use retrieval for facts and fine-tune only for style, format or narrow skills.
Does RAG stop hallucinations?
It reduces them substantially when the retrieval is good and the model is instructed to answer only from retrieved passages and to decline otherwise. An evaluation set that tests declines is what makes that behaviour reliable.
How long does a RAG pilot take?
1 Sol's 4-week AI pilot takes one use case and your documents to a production-shaped assistant with cited answers, fixed price, scoped in one call.
Sources
- Google Cloud — Grounding overview (Vertex AI)
- 1 Sol UK case study — Enterprise search & secure AI for a global UN agency
- 1 Sol UK case study — Governed data layer for AI features
- 1 Sol UK case study — Contract intake with Document AI