What Is RAG? Retrieval-Augmented Generation Explained With a Working Example

Ask a general AI assistant about your company’s refund policy, and it can only guess. It has never seen your documents. RAG, short for retrieval-augmented generation, solves this by looking up the right information and handing it to the model along with the question.

This guide explains each stage in plain language, then builds a tiny version using only Python’s standard library. We ran that code on Python 3.13, and the scores shown come from that run. It uses no language model, so it demonstrates retrieval and prompt building, and we did not test the final answer step.

What is RAG?

Retrieval-augmented generation is a technique that combines a search step with a language model. When a question arrives, the system searches a knowledge source, adds the best results to the prompt, and asks the model to answer based on that material.

Think of an open-book exam. A closed-book student relies on memory and may confidently misremember. An open-book student can look up the right page first. RAG gives the model that book, and also lets it point to which page it used.

The problem RAG solves

A language model on its own has three limits for business use:

• It does not know your private data. Your policies, tickets and contracts were not part of its training.
• Its knowledge has a cutoff. It cannot know about a price you changed this morning.
• It may invent answers. When unsure, it can produce something plausible yet false.

RAG addresses all three, and you do not have to retrain anything. Update a document, re-index it, and the very next answer can use it.

How RAG works, step by step

A RAG system has two pipelines. One prepares your data ahead of time, and the other runs for every question.

Indexing: preparing your data

1. Load: collect content from PDFs, wikis, databases or tickets.
2. Clean and chunk: remove junk, then split documents into small passages.
3. Embed: convert each passage into a list of numbers, called an embedding, that captures its meaning.
4. Store: save the embeddings in a searchable index, often a vector database.

Querying: answering a question

1. Retrieve: embed the question and find the passages closest in meaning.
2. Augment: insert those passages into the prompt.
3. Generate: the model writes an answer using that context.

Retrieval does not have to use vectors. It can be a keyword search or even a SQL query, whichever finds the right information for your data.

A tiny working example

To see the moving parts clearly, here is a small retriever with no libraries. It represents text as word counts, a much simpler stand-in for real embeddings, and it scores similarity using the cosine measure.

import math, re
from collections import Counter

DOCS = {
    "refunds": "Refunds are paid within 7 days to the original payment method.",
    "shipping": "Standard shipping takes 3 to 5 working days across India.",
    "warranty": "All keyboards include a 2 year warranty against defects.",
}

def vectorize(text):
    return Counter(re.findall(r"[a-z0-9]+", text.lower()))

def similarity(a, b):
    dot = sum(a[word] * b[word] for word in a)
    size = math.sqrt(sum(v * v for v in a.values()))
    size *= math.sqrt(sum(v * v for v in b.values()))
    return dot / size if size else 0

def retrieve(question, k=1):
    q = vectorize(question)
    scored = [(similarity(q, vectorize(t)), name) for name, t in DOCS.items()]
    return sorted(scored, reverse=True)[:k]

def build_prompt(question):
    (score, name), = retrieve(question)
    return (
        "Answer using only the context. If it is not there, say you do not know.\n"
        f"Context: {DOCS[name]}\n"
        f"Question: {question}"
    )

Asking about refunds finds the right passage and builds this prompt:

print(retrieve("How many days until refunds are paid?"))
# [(0.4558..., 'refunds')]

print(build_prompt("How many days until refunds are paid?"))
# Answer using only the context. If it is not there, say you do not know.
# Context: Refunds are paid within 7 days to the original payment method.
# Question: How many days until refunds are paid?

In a complete system, you would now send that prompt to a language model. The important detail is the instruction to admit when the answer is missing, which you can learn more about in our guide to prompt engineering.

What our example misses

Now try a question that means the same thing but uses different words:

print(retrieve("When can I get my money back?", k=3))
# [(0.0, 'warranty'), (0.0, 'shipping'), (0.0, 'refunds')]

Every score is zero, because the question shares no words with any document. “Money back” means refund to a human, but word counts cannot tell. This single output explains why real systems use embeddings, which are designed so that text with similar meaning ends up with similar numbers, even with no words in common.

But do not assume embeddings solve everything. They can struggle with exact terms such as product codes and error numbers, which keyword search handles very well. This is why many production systems use hybrid search, combining the two approaches.

Chunking: the step people underestimate

Documents are split into chunks because whole files are too long and imprecise to retrieve. The size you choose involves a real trade-off:

A few hundred words with some overlap between chunks is a common starting point, but treat that as a first guess to test. Split along natural boundaries, such as headings, when you can. And attach metadata, like the document title, date and department, so you can filter results and cite sources.

Improving retrieval

When answers are poor, these are the upgrades most worth trying, roughly in this order:

• Clean the data. Remove outdated and duplicate documents first. No algorithm fixes contradictory sources.
• Use hybrid search. Combine meaning-based and keyword results.
• Add a re-ranker. Fetch more candidates, then use a specialised model to reorder them by relevance.
• Filter by metadata. Restrict results by product, region or date before searching.
• Rewrite the query. Turn a vague follow-up like “what about the second one?” into a complete question.

RAG vs fine-tuning

They are not mutually exclusive. Our comparison of RAG vs fine-tuning vs prompt engineering covers how to choose in more depth.

How to test a RAG system

RAG is software, and it needs tests like any other software. Create a set of real questions along with the passages and answers you expect, then measure the two stages separately:

Include awkward cases too: questions with no answer in your data, questions that use different wording and questions with conflicting sources. Rerun this set whenever you change chunking, models or prompts, just like a regression suite. Our pytest tutorial shows how to run many inputs through a single test.

Why RAG systems fail

• The answer is not in the data. The model may still attempt one, so tell it how to decline.
• The right chunk was not retrieved. This is often a chunking or search problem.
• Too much context. Irrelevant passages can distract the model and raise costs.
• Stale documents. An old policy can be retrieved as confidently as the current one.
• Tables and PDFs parsed badly. If extraction scrambles the text, retrieval never recovers.
• Testing only with demo questions. Real users phrase things very differently.

Security and permissions

RAG brings two risks that are easy to miss. First, access control: if the index holds HR files, a junior employee’s question must not retrieve them. Apply each user’s permissions during retrieval, because instructing the model to keep secrets is not a reliable safeguard.

Second, prompt injection. A retrieved web page or document could contain text such as “ignore your instructions”. Treat retrieved content as untrusted data, limit what actions the model can take and require a human to approve sensitive ones.

Final thoughts

RAG is a simple idea with plenty of practical detail: find the right information, and then ask the model to use it. Start with a small, clean set of documents and around twenty real questions. Get retrieval working well before you add complexity, because a modest pipeline with good data usually beats a sophisticated one with bad data.

Frequently asked questions

Keep learning