Fine-tune, retrieve or prompt? A decision guide

"Should we fine-tune?" is one of the first questions we hear, and the honest answer is usually "it depends on what you need to change." Each technique changes something different.

What each technique changes

Prompting changes instructions. It is the cheapest to try and the fastest to iterate. Modern models follow detailed instructions and examples well. If a better prompt and a few examples clear your quality bar, stop there.

Retrieval changes knowledge. RAG gives the model facts at question time, from your documents. It is the right tool when facts change, when answers must be cited, or when access to information differs by user. Research comparing the two, such as Ovadia and colleagues' 2023 study, found retrieval consistently outperformed unsupervised fine-tuning for injecting new knowledge.

Fine-tuning changes behaviour. Training on examples teaches format, tone, policy, domain phrasing and task skill. It makes outputs more consistent, lets you drop long prompts and can let a smaller model do a big model's job. It is a poor way to teach facts that change, because they are baked into weights you then have to retrain.

Custom models change the task itself. Classical models and models trained from a strong pre-trained base are right for narrow, high-volume predictions on your data, such as classification, ranking and forecasting, where a language model is slow, costly or simply the wrong shape of tool.

A quick guide

• Answers depend on documents that change, or must be cited: retrieval.
• Outputs must follow a strict format or house style every time: fine-tuning, or prompting with validation first.
• Volume is high and the task is narrow: distil a small model, or train a custom model.
• The task is numeric prediction on tables: a custom ML model, not an LLM.
• Data cannot leave your network: any of the above on a private deployment.
• You are not sure yet: prompt first, measure, then decide.

Many production systems combine them: a fine-tuned small model that follows your format, reading from a retrieval layer, with a frontier model on standby for hard cases.

Data and cost, honestly

• Fine-tuning with parameter-efficient methods such as LoRA and QLoRA is cheap in compute. The cost is in collecting a few hundred to a few thousand excellent examples.
• Distillation needs many inputs labelled by a teacher model; check the teacher's terms of use.
• Retrieval costs most in ingestion and evaluation, and almost nothing in training.
• Custom models cost most in data work, then little to run.

The rule that saves the most time

Build the evaluation set first. Fifty to two hundred real inputs with known good outputs let you test a prompt in an hour, a retrieval design in a day and a fine-tune in a week, and compare them on the same scale. Without it, every option looks promising in a demo. With it, the choice usually makes itself.

What to measure

• Score on your evaluation set. For each option, on the same inputs, graded the same way.
• Format and policy adherence. The share of outputs that validate against your schema and rules without repair.
• Cost and latency per request. At your real volume, not in a demo.
• Freshness. For knowledge-heavy tasks, how answers hold up after your documents change, which is where fine-tuning alone falls behind.

How we would start

We run this comparison in the first weeks of custom model, fine-tuning and RAG engagements, and recommend the simplest option that passes. Gaia, our no-code training platform, is built around the same idea: start from a pre-trained model and adapt it, rather than training from nothing.