Retrieval-augmented generation (RAG) explained

Retrieval-augmented generation (RAG) lets a language model answer from your documents instead of from memory alone. This guide walks through the full pipeline, the choices that matter most at each stage, how to measure quality, how RAG compares with long context and fine-tuning, and the security controls a production system needs.

What RAG is and why it exists

A language model knows only what was in its training data, frozen at a cutoff date, and it knows nothing about your private files. Ask about last week's pricing change or an internal policy and it will either decline or produce a confident guess. That second failure is a hallucination.

RAG fixes this by adding a search step. Before the model answers, your application retrieves the most relevant passages from a knowledge base and places them in the prompt with an instruction to answer from those passages and cite them. The term comes from a 2020 research paper by Patrick Lewis and colleagues, and the pattern is now a standard way to connect LLMs to company knowledge.

RAG addresses four problems at once:

• Freshness. Update a document and the next answer reflects it, with no retraining.
• Private knowledge. The model can use content it never saw in training.
• Verifiability. Answers can cite their sources, which grounds them in text users can check.
• Access control. Retrieval can respect who is allowed to see which document.

RAG does not make a model smarter, and it cannot help if the right passage is never retrieved. Most RAG quality problems are search problems.

The RAG pipeline, step by step

A RAG system has an offline indexing path that prepares documents and an online query path that runs for every question.

1. Ingest. Connectors pull content from file shares, wikis, ticketing systems, databases and websites. Record the source URL, owner, last-modified date and access permissions for every document, and schedule syncs so edits and deletions propagate.
2. Parse. Convert each file to clean text while keeping structure: headings, lists, tables and page numbers. PDFs are the hardest case. Scanned pages need OCR, and complex layouts need a document-conversion tool such as Docling or Apache Tika rather than raw text extraction.
3. Chunk. Split documents into passages small enough to retrieve precisely but large enough to make sense alone.
4. Embed. An embedding model turns each chunk into a vector, a list of numbers positioned so that passages with similar meaning sit close together.
5. Index. Store vectors, chunk text and metadata in a vector database or search engine, usually alongside a keyword index for exact terms.
6. Retrieve. At query time, embed the question, apply metadata and permission filters and fetch the top candidates, commonly 20 to 100.
7. Rerank. A slower, more accurate model rescores the candidates and keeps the best 5 to 10.
8. Generate with citations. Put the passages in the prompt with numbered source labels. Instruct the model to answer only from them, cite each claim and say when the sources do not contain the answer. Before showing the response, check that every cited source ID exists.

Chunking strategies and trade-offs

Chunking decides what the retriever can find. Chunks that are too small lose context ("the limit is 30 days," with no hint of which limit), while chunks that are too large blur the embedding and waste prompt space.

A reasonable starting point is structure-aware chunks of roughly 300-800 tokens with 10-15 percent overlap, each prefixed with its document title and section heading. Then tune against an evaluation set rather than intuition. Anthropic reported that prepending generated context to each chunk before embedding cut top-20 retrieval failures by 35 percent in its tests, and by 49 percent when combined with contextual keyword search. The token counter helps you check chunk sizes.

Retrieval: embeddings, vector stores, hybrid search and reranking

Embedding models and vector databases

Choose the embedding model carefully, because changing it later means re-embedding everything. Compare candidates on a sample of your own queries, and check language coverage, maximum input length and vector size, which drives storage cost. Many current models can output shortened vectors (Matryoshka-style), trading a little accuracy for much smaller indexes. The options below were current as of October 2026.

Most of these use approximate nearest-neighbor indexes such as HNSW, which give up a sliver of recall for large speed gains. Store the embedding model name and version with every vector so incompatible vectors never mix in one index.

Hybrid search: BM25 plus vectors

Vector search matches meaning, so "how do I get my money back" finds a passage titled "Refund policy." It is weaker on exact strings such as product codes, error messages, names and rare acronyms. BM25, the classic keyword-ranking algorithm, has the opposite profile. Hybrid search runs both and merges the results.

The usual merge method is reciprocal rank fusion (RRF). Each document scores the sum of 1 / (k + rank) across the result lists, where k is a smoothing constant that Elasticsearch sets to 60 by default. A chunk ranked first by keywords and third by vectors scores 1/61 + 1/63, about 0.0323, beating a chunk ranked second in only one list (1/62, about 0.0161). Because RRF uses ranks rather than raw scores, you never have to reconcile two incompatible scoring scales.

Reranking

Embedding search compares a precomputed document vector with a query vector. A reranker, usually a cross-encoder, reads the query and each candidate together and scores relevance directly, which is more accurate but too slow to run over a whole corpus.

The standard pattern retrieves 50 to 100 hybrid candidates and reranks them down to the best 5 to 10. Options include Cohere's rerank-v4.0 models (pro and fast), Voyage AI's rerankers and open-weight models such as Qwen3-Reranker and the BGE reranker family. In Anthropic's tests, adding reranking on top of contextual embeddings and keyword search raised the reduction in retrieval failures to 67 percent.

How to evaluate a RAG system

Evaluate retrieval and generation separately, or you will not know which half to fix. Start with 50 to 200 real questions, each labeled with the passages that answer it and a reference answer.

Open-source frameworks automate much of this evaluation. Ragas provides context precision, context recall, faithfulness and response relevancy metrics. TruLens popularized the "RAG triad" of context relevance, groundedness and answer relevance, while DeepEval, Arize Phoenix and promptfoo add test runners, tracing and regression checks. Calibrate any LLM judge against a few dozen human ratings before trusting it, and rerun the suite whenever you change chunking, embeddings, prompts or models.

RAG vs long context vs fine-tuning

Several frontier models now accept around 1 million tokens, so you can sometimes paste a whole knowledge base into the prompt. Fine-tuning is a third option. Each solves a different problem.

Cost makes the trade-off concrete. At an example rate of $2 per million input tokens, sending a 200,000-token corpus with every question costs $0.40 per query, or $12,000 for 30,000 queries a month. Prompt caching at an example cached rate of $0.10 per million cuts that to about $600 plus cache writes. Retrieving 8,000 tokens of context instead costs $0.016 per query, or $480 a month, and answers arrive faster.

Accuracy matters too. The 2023 "Lost in the Middle" study found that models use information at the start and end of a long input more reliably than information in the middle. See context windows explained for more, and fine-tuning vs RAG vs prompting for a full decision guide.

Security: permissions and prompt injection

RAG connects a model to sensitive data. The OWASP Top 10 for LLM Applications (2025 edition) lists the two biggest resulting risks: prompt injection (LLM01) and vector and embedding weaknesses (LLM08).

Document-level permissions

The model must never see a passage the user is not allowed to read. Copy access-control lists into each chunk's metadata during ingestion, filter on the authenticated user's identity and groups inside the retrieval query and sync permission changes as fast as content changes. Never rely on the system prompt to withhold restricted text; once it is in the context window, assume it can leak. Use separate indexes or namespaces per customer in multi-tenant products, and protect stored embeddings as carefully as the text they came from.

Prompt injection in retrieved content

Any retrieved document can contain text that reads like instructions, such as a hidden line in a web page or PDF telling the model to ignore its rules. This is indirect prompt injection, and no filter stops it completely, so limit the damage:

• Wrap retrieved passages in clear delimiters and tell the model they are reference data, not instructions.
• Grant the least privilege possible; a RAG chatbot rarely needs tools that send email or change records.
• Require human approval for consequential actions and validate outputs with deterministic code.
• Strip or flag hidden text during parsing, and control which sources can enter the index.
• Do not auto-render model-generated links or images, a known channel for leaking data.

For broader controls, read the LLM security and privacy guide.

A reference architecture

This design suits most internal knowledge assistants. It has two paths that share one index, plus a feedback loop.

Indexing path. Connectors detect new, changed and deleted documents and place them on a queue. A parsing service converts each file to structured text, and a chunker emits passages with metadata: source URL, title, heading path, timestamps and access groups. An enrichment step adds contextual headers, and an embedding service writes vectors and text to a versioned store that supports both vector and BM25 search, so a bad build can be rolled back.

Query path. The API authenticates the user and resolves their groups. A rewrite step turns follow-ups such as "what about Canada?" into standalone queries. Hybrid retrieval runs with a permission filter, fuses about 50 candidates with RRF and passes them to a reranker that keeps the top 8.

A context builder orders the passages, enforces a token budget and assigns source numbers. The LLM answers with inline citations, a validator confirms the cited IDs exist and guardrails screen the output before it streams to the user with clickable sources.

Feedback loop. Log queries, retrieved IDs, scores, answers and user ratings under your data-retention policy. Add failed questions to the evaluation set, and rerun the suite before every change to chunking, models or prompts.

Common mistakes and how to fix them

Next steps

Estimate per-query cost for your retrieval budget with the LLM cost calculator, and see how much text fits in different prompt sizes with the context window visualizer. Then read fine-tuning vs RAG vs prompting to confirm RAG is the right tool for your problem. If you are building a production knowledge assistant and want an experienced reviewer, you can hire an LLM expert.