Fine-tuning vs RAG vs prompting: which should you use?

When a language model falls short, there are three main levers: change the prompt, give the model better information at answer time or train the model itself. This guide explains what each technique actually changes, offers a decision tree and comparison table, covers fine-tuning methods and managed options as of October 2026 and works through three common scenarios.

What each technique changes

The fastest way to choose is to ask what kind of gap you have: instructions, knowledge or behavior.

• Prompting changes the instructions. The system prompt, examples and output format tell the model what to do on this request. Nothing persists beyond the request, and changes take effect immediately.
• Retrieval-augmented generation changes the knowledge in view. Your application searches a document store and places relevant passages in the prompt, so the model answers from current, private and citable sources.
• Fine-tuning changes the model's behavior. Training on examples adjusts the weights, so the model responds differently by default: a house style, a strict output schema or skill at a narrow task.

An analogy: prompting is a written brief for a contractor, RAG is giving them the company handbook during the job and fine-tuning is months of on-the-job training. Training shapes habits well, but it is a poor way to make anyone memorize a handbook that changes every week.

A decision tree in plain language

Work through these questions in order. Each one assumes you have a small evaluation set, ideally 50 to 200 real inputs with expected outputs, so you can measure progress instead of guessing.

1. Have you tried a strong prompt on a capable model? If not, start there. Write explicit instructions, add three to five few-shot examples, request structured outputs if you need JSON and test a stronger model. Many gaps close at this step.
2. Is the remaining gap about facts? If the model lacks information, such as your product catalog, recent events or internal policies, add RAG. If your corpus is small and stable, simply placing it in the prompt with prompt caching may be enough.
3. Is the gap about behavior that prompting cannot pin down? Examples include a format the model drifts from at volume, a specialized labeling scheme or a tone that requires pages of examples in every prompt. That is a fine-tuning candidate, provided you can assemble hundreds of good examples.
4. Is the problem cost or latency rather than quality? If a large model meets your bar but costs too much, consider distilling it into a smaller fine-tuned model, or route easy requests to a cheaper model.
5. Do you need several of these at once? Combine them, adding one layer at a time and keeping each only if the evaluation scores improve.

Comparison table: cost, data, time and risk

Fine-tuning methods

Fine-tuning covers several techniques with very different costs. Memory figures below are rough, for illustration.

Full fine-tuning

Every weight is updated. This gives the most capacity to change the model but needs the most compute: with a standard optimizer, training takes roughly 16 bytes per parameter before activations, about 130 GB for an 8B model, which means several data-center GPUs. You also store a full copy of the model for every variant, and aggressive training can erase general skills, a problem known as catastrophic forgetting.

LoRA and QLoRA

LoRA (low-rank adaptation) freezes the base model and trains small adapter matrices alongside selected layers. The 2021 paper reported up to 10,000 times fewer trainable parameters than full fine-tuning on GPT-3 175B. Adapters are usually tens to hundreds of megabytes, so one base model can serve many of them.

QLoRA, introduced in 2023, loads the frozen base model in 4-bit precision and trains LoRA adapters on top, bringing an 8B model's weights to roughly 5 GB and putting fine-tuning within reach of a single consumer GPU. Popular tools include Hugging Face TRL and PEFT, Unsloth, Axolotl and, on Macs, MLX LM.

Preference tuning

Supervised fine-tuning teaches by imitation: here is the input, here is the ideal output. Preference tuning teaches by comparison: here are two responses, and this one is better. RLHF trains a reward model and optimizes against it, while direct preference optimization (DPO) learns directly from preference pairs and is much simpler to run.

Preference methods help with qualities that are easier to judge than to write, such as helpfulness, concision or tone. Reinforcement fine-tuning, offered on some platforms, instead scores outputs with a grader you define, which suits tasks with checkable answers.

Distillation

Distillation uses a large teacher model to generate high-quality outputs, often thousands of them, then fine-tunes a smaller student model on that synthetic data. The aim is to get near-teacher quality on one narrow task at a fraction of the cost and latency. Check the teacher's terms of service first, because some providers restrict using outputs to train competing models.

Managed fine-tuning options in 2026

Managed fine-tuning of proprietary models has contracted sharply. Status as of October 2026:

• OpenAI is winding down self-serve fine-tuning. Since May 7, 2026, organizations that never fine-tuned cannot create jobs; organizations without recent fine-tuned inference lost job creation on July 2, 2026; and all customers lose it on January 6, 2027. Existing fine-tuned models keep serving until their base models are deprecated.
• Google no longer offers tuning in the Gemini API or AI Studio. Supervised, preference and reinforcement tuning for Gemini, plus tuning of open models, remain available on Google Cloud, where Vertex AI's tuning documentation now sits under the Gemini Enterprise Agent Platform.
• Microsoft Foundry lists managed fine-tuning for several OpenAI models (supervised, DPO and reinforcement fine-tuning) and open-weight models including gpt-oss, Qwen3.6, Qwen3.8 and Llama 3.3. Availability varies by region, and some options are in preview.
• Amazon Bedrock offers supervised fine-tuning, reinforcement fine-tuning and distillation for selected foundation models; check the console for the current model list in your region.
• Mistral AI marks its fine-tuning feature as deprecated in its documentation.
• Open-weight specialists, including several GPU clouds and inference providers, run LoRA and full fine-tuning jobs on popular open models and host the results.

Worked examples

Support replies in a consistent brand tone

Situation: a support team wants replies that match its voice: warm, brief, no jargon and always ending with a next step. A detailed prompt gets most of the way, but the tone drifts on unusual tickets.

Approach: first build a 30-item style rubric and score 100 replies with a human or an LLM judge. Then try a stronger system prompt with five exemplary replies. If that block grows to 6,000 tokens, it costs about $1,200 a month across 100,000 requests at an example rate of $2 per million input tokens, or roughly $60 with cached reads at $0.10 per million.

Fine-tune only if tone scores still fall short: collect 500 to 1,000 of your best human-written replies, train a LoRA adapter on an open model or use a managed service and compare rubric scores against the prompted baseline. Product facts should still come from RAG, not the fine-tune.

Extracting fields from domain documents

Situation: a logistics firm extracts 25 fields from 50,000 freight invoices a month, about 3,000 input and 400 output tokens each.

Approach: start with a capable model, a JSON schema and a few examples, measuring field-level accuracy on 200 labeled invoices. At example frontier prices of $10 input and $50 output per million tokens, the workload costs $1,500 + $1,000 = $2,500 a month. If a frontier model scores well but a small model at $0.10 and $0.50 per million (about $25 a month) misses edge cases, fine-tune or distill the small model using the frontier model's verified outputs as training data. Keep a regression set of tricky invoices and spot-check production output.

Internal knowledge Q&A

Situation: employees ask questions about HR policies, engineering runbooks and product documentation that change weekly, and answers must cite sources and respect access rights.

Approach: this is a RAG problem. Fine-tuning on the documents would go stale within weeks, could not cite sources and would blur access boundaries. Build retrieval with permission filters, hybrid search and reranking, then prompt the model to answer only from sources.

Fine-tuning becomes worth considering later for narrow behaviors, such as consistent citation formatting or reranking. The RAG guide covers the full pipeline.

Combining techniques

The techniques stack. Common production patterns include:

• Prompted model plus RAG: the default for knowledge assistants.
• Fine-tuned model plus RAG: the model learns your format, tone and refusal rules, and retrieval supplies current facts.
• Distilled specialist plus a frontier fallback: a small tuned model handles routine requests, and low-confidence cases escalate to a larger model.
• Tuned components: a small fine-tuned model classifies intent, rewrites queries or reranks passages, while a general model writes the answer.

Add one layer at a time, and keep it only if your evaluation set shows a gain worth its maintenance cost.

Evaluation before and after

Every technique needs the same discipline: measure, change one thing, measure again.

1. Freeze a test set first. Use real inputs, including edge cases, with expected outputs or scoring rubrics. Never train on it.
2. Record a baseline. Score the best prompted model on the test set, including cost and latency per request.
3. Use task-appropriate metrics. Exact match or field accuracy for extraction, retrieval recall and faithfulness for RAG, rubric scores for tone and pass rates for code.
4. Check for regressions. After fine-tuning, run general-capability and safety checks to catch lost skills or weakened refusals.
5. Compare total cost. Include training, hosting, retraining cadence and engineering time, not just per-token price.
6. Keep monitoring. Sample production outputs, add new failures to the test set and rerun it whenever a model, prompt or index changes.

Common mistakes

• Fine-tuning to teach facts. Use retrieval for knowledge; fine-tune for behavior.
• Skipping the prompt baseline. Teams sometimes spend weeks on training data for a gap a better prompt or model would have closed in an afternoon.
• Training on messy data. Inconsistent examples teach inconsistent behavior. Fifty clean examples beat five hundred noisy ones.
• No held-out test set. Without it, you cannot tell improvement from memorization.
• Ignoring platform risk. Proprietary fine-tunes can vanish with their base model, as OpenAI's 2026 wind-down shows.
• Building RAG for a tiny, static corpus. If everything fits comfortably in the prompt, cache it and skip the pipeline.

Next steps

Draft and test a stronger baseline with the prompt builder before you train anything. Compare prompt-heavy, retrieval and smaller-model options with the LLM cost calculator, and if you plan to host a fine-tuned open model, run the numbers in the self-host vs API calculator. For help designing a training dataset or evaluation plan, you can work with an LLM expert.