Fine-Tuning vs RAG vs Prompting: Which One Do You Need?

By Ravindra Bagale, trainer · LinkedIn · Last updated: 19 October 2026

Generative AI course · Chapter 40 · Estimated reading time: 12 minutes

1. Suppose a clinic wants "its own ChatGPT"

Suppose Dr. Mehta runs a small clinic. Patients keep calling to ask the same things: clinic timings, consultation fees, which doctor sits on which day, and how to prepare for a blood test. Her nephew says: "We should fine-tune a model on the clinic's data."

Think about what would happen. The clinic's fees change in January. A new doctor joins on Wednesdays. A fine-tuned model would keep giving the old answers until someone trains it again. And it could not say "this is from the fee list, page 1".

What Dr. Mehta actually needs is a bot that reads the current fee list and timetable before it answers. That is RAG, not fine-tuning. Choosing the right method first saves weeks.

2. What each method changes

The order to try them in:

1. Prompting first: a clear role, task, context, format and a few examples (from the prompt engineering chapters).
2. Add RAG when answers must come from your own documents.
3. Consider fine-tuning only when prompting with examples still cannot produce the same style or format reliably, and you have hundreds of good examples.

They also combine: many real systems use a good prompt and RAG; a few add fine-tuning on top.

3. The 5-question decision table

Answer yes or no for your use case:

1. Must answers come from your private documents?
2. Do the facts change (prices, timetables, policies)?
3. Do users need citations ("see page 2")?
4. Must every answer follow a fixed style or format?
5. Do you already have hundreds of good examples of the exact output you want?

Reading the answers:

• Yes to any of 1 to 3 → RAG (plus a good prompt).
• No to 1 to 3, yes to 4 and 5 → fine-tuning may help; try a prompt with examples first.
• Everything else → prompting.

We turned this table into a small Python helper so you can test your own cases:

# Prompting, RAG or fine-tuning? Answer 5 yes/no questions about the job; get a starting point.
def choose(needs_private_docs, facts_change_often, needs_citations, fixed_style_or_format, many_examples_ready):
    reasons = []
    if needs_private_docs or facts_change_often or needs_citations:
        reasons.append("answers must come from your own, changing documents")
        method = "RAG (plus a good prompt)"
    elif fixed_style_or_format and many_examples_ready:
        reasons.append("same style or format every time, and hundreds of good examples")
        method = "Fine-tuning (try prompting with examples first)"
    else:
        reasons.append("the model already knows enough; you need clear instructions")
        method = "Prompting (with a few examples)"
    if fixed_style_or_format and method.startswith("RAG"):
        reasons.append("put the style rules and 2 to 3 examples in the prompt")
    return method, reasons

4. Worked examples: 3 business cases, decided

We ran the helper on 3 cases:

cases = {
    "Clinic reception bot (timings, fees, doctor days)": (True, True, True, False, False),
    "Product descriptions for 2,000 items in one house style": (False, False, False, True, True),
    "Polite replies to customer reviews": (False, False, False, True, False),
}
for name, answers in cases.items():
    method, why = choose(*answers)
    print(f"{name}\n  -> {method}\n     because: {'; '.join(why)}")

Real output (19 October 2026):

Clinic reception bot (timings, fees, doctor days)
  -> RAG (plus a good prompt)
     because: answers must come from your own, changing documents
Product descriptions for 2,000 items in one house style
  -> Fine-tuning (try prompting with examples first)
     because: same style or format every time, and hundreds of good examples
Polite replies to customer reviews
  -> Prompting (with a few examples)
     because: the model already knows enough; you need clear instructions

The helper only gives a starting point. Here is the full reasoning for each case.

Case 1: the clinic bot → RAG

Plan: put the fee list and timetable in a RAG bot (as in the RAG chapter), with a prompt rule: "If the answer is not in the documents, say 'Please call the clinic' ". Never let it give medical advice; it answers only admin questions.

Expected output (yours will be similar, not the same): "Dr. Kulkarni sits on Wednesdays from 5 pm to 8 pm (timetable, page 1)."

Case 2: 2,000 product descriptions in one house style → prompt first, fine-tune only if needed

Suppose Bluehill, an online kitchenware shop, wants every description to have exactly 2 lines: a short benefit, then the key facts. They have 600 approved past descriptions.

Plan: first, a prompt with the style rules and 5 of the best examples. Test it on 50 products. If more than a few break the style, fine-tuning on the 600 examples is worth considering, where your provider still offers it (section 6).

Case 3: replies to customer reviews → prompting

Plan: a saved prompt with the tone, length, a sign-off, and 3 example replies (one for praise, one for a complaint, one for a mixed review). Done in an afternoon.

5. What fine-tuning data looks like (and how to check it)

Fine-tuning data is usually a JSONL file: one example per line, each with the messages and the perfect answer. Suppose Bluehill prepares 4 lines and makes 3 common mistakes. We wrote a checker:

# Check a fine-tuning training file before you upload it anywhere.
import json
rows, problems = 0, []
with open("train.jsonl", encoding="utf-8") as f:
    for n, line in enumerate(f, start=1):
        rows += 1
        try:
            msgs = json.loads(line)["messages"]
        except Exception as e:
            problems.append(f"line {n}: not valid JSON with 'messages' ({e.__class__.__name__})"); continue
        roles = [m.get("role") for m in msgs]
        if roles[-1:] != ["assistant"]:
            problems.append(f"line {n}: last message must be the assistant's answer")
        if any(len(m.get("content", "")) == 0 for m in msgs):
            problems.append(f"line {n}: empty content")
print(f"Rows: {rows}, problems: {len(problems)}")
for p in problems: print(" -", p)

Real output on our sample file (19 October 2026):

Rows: 4, problems: 3
 - line 2: empty content
 - line 3: last message must be the assistant's answer
 - line 4: not valid JSON with 'messages' (JSONDecodeError)

Only 1 of 4 lines was usable. This is the hidden cost of fine-tuning: collecting and cleaning hundreds of perfect examples takes far longer than the training itself. A good line looks like this:

{"messages": [{"role": "system", "content": "Write a 2-line product description in the Bluehill house style."}, {"role": "user", "content": "Steel water bottle, 1 litre, keeps cold 12 hours"}, {"role": "assistant", "content": "Cold water, all day long. Our 1-litre steel bottle keeps drinks cold for 12 hours."}]}

6. Where fine-tuning is available (October 2026)

This changed a lot in 2026. Checked on the official pages on 19 October 2026:

• OpenAI: since 7 May 2026, self-serve fine-tuning is not available to organizations that have not fine-tuned before. Existing active customers can create new fine-tuning jobs only until 6 January 2027. Fine-tuned models keep working until their base model is retired.
• Google Gemini API: no longer supports fine-tuning, and Google says it has no immediate plans to bring it back.
• Google Cloud (Vertex AI): still offers supervised tuning for some Gemini models (for example Gemini 3.5 Flash), with training data in Cloud Storage. This is a cloud project with billing, not a free key.

So for most beginners in 2026, the practical answer is even clearer: prompting and RAG first. Fine-tuning is now mainly a cloud-platform or open-model project.

7. How to know your choice worked

Whatever you choose, test it the same way:

1. Write 30 real questions (or products, or reviews), including 5 hard or tricky ones.
2. Write what a good answer looks like for each.
3. Run them, and score each answer: correct, partly correct, wrong.
4. Change one thing (the prompt, the chunks, the examples) and run all 30 again.
5. Keep the version with the best score, and keep the 30 tests for the future.

This small "test set" is what professionals use to compare prompting, RAG and fine-tuning honestly.

8. Beginner mistakes and quick fixes

9. Try at home

Recap

Next time someone says "let's train the model on our data", ask the 5 questions first. Most of the time, a better prompt and a good RAG setup will get you there faster.

Previous chapter: Vector Databases for Beginners: Chroma, pgvector and OpenSearch

Next chapter: What Is an AI Agent? Agentic AI vs Generative AI Explained with Examples

Ravindra Bagale, trainer: linkedin.com/in/ravindra-bagale. The clinic, the doctors, Bluehill and the products on this page are made up for learning. Fine-tuning availability was checked on OpenAI's API deprecations page, Google's Gemini API fine-tuning page and Google Cloud's Vertex AI tuning pages on 19 October 2026; this changes often. The Python output was produced on 19 October 2026.