Signum
Learn

Prompting, RAG or fine-tuning? How to choose the right way to customise an LLM

By Elliot Marsh, Editor · · 4 min read

Out of the box, a large language model knows a lot about the world and nothing about your business. There are three main ways to close that gap: write better prompts, give the model the right information at question time (retrieval-augmented generation, or RAG), or train it further on your own examples (fine-tuning). They solve different problems, and choosing the wrong one wastes time and money.

The one-line answer

  • Prompting changes what you ask.
  • RAG changes what the model can see.
  • Fine-tuning changes how the model behaves.

If the model lacks knowledge, use RAG. If it lacks a skill or a consistent style, consider fine-tuning. Always start with prompting, because it is the cheapest and fastest to try. OpenAI's guide to optimising LLM accuracy frames it the same way: retrieval for context the model is missing, fine-tuning for consistent behaviour.

1. Prompting

Prompting means improving the instructions and examples you send with each request: being specific about the task, giving the model a role and audience, showing two or three examples of good output ("few-shot" prompting), asking it to reason step by step and specifying the output format.

Good for: most tasks, at first. Modern models are strong enough that clear instructions solve a large share of problems.

Costs: your time, plus longer prompts mean more input tokens on every call.

Limits: you cannot fit a whole knowledge base in a prompt, and very long instruction lists become fragile: the model may follow some rules and forget others.

2. Retrieval-augmented generation (RAG)

RAG searches your documents for passages relevant to each question and places them in the prompt, so the model answers from them. The technique was named in a 2020 paper by Lewis et al.

Good for: answering questions from documents that change, are private or are too large to include in full: policies, product manuals, support tickets, research libraries. It also lets you show sources and respect permissions.

Costs: building and maintaining a search index, and extra tokens per request for the retrieved passages.

Limits: the answer is only as good as the retrieval. If the right passage is not found, the model cannot use it. RAG does not teach the model a new style or skill.

3. Fine-tuning

Fine-tuning continues training a model on your own examples (input and the desired output) so its weights change. Common forms:

  • Supervised fine-tuning: show it many examples of the right answer.
  • Preference tuning: show it pairs of better and worse answers so it learns to prefer one kind. The InstructGPT paper (Ouyang et al., 2022) used human preference feedback this way to make models follow instructions.
  • Parameter-efficient fine-tuning such as LoRA (Hu et al., 2021), which trains a small set of added parameters instead of the whole model, making fine-tuning far cheaper and allowing several adaptations of one base model.

Good for: - a consistent output format or house style that prompting cannot hold reliably; - specialised tasks with a clear right answer, such as classification or extracting fields from documents; - getting a smaller, cheaper model to do one task as well as a large one, which cuts cost and latency; - shortening prompts, because behaviour no longer needs long instructions.

Costs: collecting and cleaning training examples (usually the hardest part), training runs, evaluation and redoing the work when the base model is updated.

Limits: fine-tuning is a poor way to add facts. The model may blend them inaccurately, cannot cite where they came from, and needs retraining whenever they change.

A quick decision guide

Problem Try first
Answers are vague or in the wrong format Prompting
Model does not know your products, policies or recent events RAG
Answers must cite sources RAG
Information changes often RAG
Output style is inconsistent despite good prompts Fine-tuning
You need a cheaper or faster model for one narrow task Fine-tuning a smaller model
Prompts have grown huge and brittle Fine-tuning

Combining them

The approaches stack. A common production setup is a fine-tuned model that reliably follows the house format, fed by RAG for up-to-date facts, with a carefully written system prompt on top. Teams usually build it in the opposite order: prompt first, add retrieval, fine-tune last.

Measure before you choose

Whichever route you take, build a small evaluation set first: 50 to 200 real inputs with the outputs you would accept. Run each change against it. Without that, it is impossible to know whether a new prompt, a better retriever or a fine-tune actually helped, or simply changed which cases fail.

Common mistakes

  • Fine-tuning to add knowledge. It rarely works as well as RAG and is harder to update.
  • Skipping prompt work. Many fine-tuning projects are solved by a clearer prompt and two examples.
  • Too few or messy training examples. Fine-tuning copies what it sees, mistakes included.
  • Not re-testing after model updates. A new base model can make a fine-tune unnecessary or break a prompt.

The bottom line

Start with prompting. Add RAG when the model needs information it does not have. Turn to fine-tuning when you need consistent behaviour, a narrow skill or a cheaper model, and you have good examples to train on. In practice the strongest systems use all three, each for what it does best.

Sources