Retrieval-augmented generation (RAG) is a technique that retrieves relevant passages from a body of text you control and gives them to a large language model as context, so the model answers from those passages instead of from what it memorized during training.
That is the whole idea. A language model on its own is a closed book: it knows what was in its training data, it cannot tell you where an answer came from, and it has no idea your refund policy changed last month. RAG opens the book.
iShort answer
RAG pairs a search step with a generation step. First it retrieves the passages most relevant to a question from content you supply, then it asks a language model to answer using only those passages. The result is an AI that can answer questions about your business, cite its sources, and stay current when your content changes. It reduces hallucination substantially but does not eliminate it.
17-34%
Hallucination rates measured in leading RAG-based legal research tools by Stanford RegLab and HAI in 2024. Retrieval grounds answers; it does not make them infallible.
Source: Stanford HAI, Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools (2024)Every retrieval-augmented system runs these three steps on every question. Everything else is implementation detail.
1. Retrieve
The question becomes a search. The system finds the passages in your indexed content that are most relevant to it, usually by meaning rather than exact keyword match.
2. Ground
Those passages are placed into the model's prompt as the authorized context for this specific answer, along with an instruction to answer from them.
3. Generate
The language model writes the answer using that context. Because the source passages are known, the system can cite them or refuse when nothing relevant was found.
Key takeaways
| Plain-English answer | |
|---|---|
| What RAG stands for | Retrieval-augmented generation |
| What it does | Fetches relevant text from your content and hands it to a language model before the model answers |
| What problem it solves | Language models cannot be easily updated, cannot show their sources, and invent plausible answers when they do not know |
| Where it came from | A 2020 NeurIPS paper by Patrick Lewis and co-authors at Facebook AI Research, University College London, and New York University (arXiv:2005.11401) |
| RAG vs fine-tuning | Fine-tuning teaches behavior and style; RAG supplies facts. For new or changing facts, research favors retrieval |
| Its main limitation | Answer quality is capped by retrieval quality. A wrong or missing passage produces a wrong answer |
| Who needs to build it | Almost nobody. Most teams get RAG through a platform that already runs the loop |
What is RAG? The longer definition
RAG is an architecture, not a product. It describes any system that answers a question in two stages: first find relevant source material, then generate an answer conditioned on that material.
The 2020 paper that named the technique framed it as combining two kinds of memory. Parametric memory is what a model absorbed into its weights during training: broad, fluent, frozen at a point in time, and impossible to inspect. Non-parametric memory is an external, searchable store of text: specific, updatable, and traceable to a document.
RAG connects the two. The model keeps its language ability, and the facts come from the store.
That distinction explains why "trained on your data" is usually a marketing phrase rather than a technical one. When a chatbot vendor says the bot is trained on your website, the underlying mechanism is almost always retrieval: your content is indexed and looked up at question time. Nothing about the model itself changes. This is a feature, not a shortcut, and it is why answers can update the moment your content does.
Why RAG exists: three problems with plain language models
RAG exists because language models on their own fail in three specific, well-documented ways. The original paper names all three in its opening: such models cannot easily expand or revise their memory, cannot straightforwardly explain their predictions, and may produce hallucinations.
They cannot be updated. A model's knowledge is fixed at training time. Changing one fact means retraining or fine-tuning, which is slow and expensive. A business whose prices, policies, and product line change monthly cannot run on a memory that changes yearly.
They cannot show their work. Ask a plain model why it said something and it produces a plausible explanation, not a citation. For customer service, compliance, or anything a customer might act on, an unverifiable answer is close to useless.
They fill gaps with fluent invention. When a model lacks a fact, it does not reliably say so. It generates the most statistically likely continuation, which reads exactly like a correct answer and is not one.
Retrieval addresses all three at once. The store can be re-indexed in minutes, every answer traces to a passage, and the model is given the facts rather than asked to recall them. The paper's own evaluation found that RAG models generated "more specific, diverse and factual language" than a comparable model without retrieval.
The RAG Loop: retrieve, ground, generate
Every RAG system, from a weekend prototype to an enterprise deployment, runs the same three-step loop on every single question. Naming the steps makes it easy to diagnose which one is failing.
Step 1: Retrieve
The system turns the incoming question into a search over your indexed content and returns the top handful of passages. Retrieval usually works on meaning rather than keywords, so "can I get my money back" finds a passage titled "Refund policy" even with no shared words.
This step decides the ceiling on answer quality. If the right passage is not retrieved, no amount of model quality recovers it.
Step 2: Ground
The retrieved passages are assembled into the prompt as the authorized context, typically with an instruction along the lines of answer using only the following sources. Grounding is what converts a general-purpose model into one that speaks specifically about your business.
The instruction matters as much as the content. A well-grounded system is told what to do when the passages do not contain the answer: say so, or hand off to a human.
Step 3: Generate
The model writes the answer from the grounded context. Because the system knows exactly which passages it supplied, it can attach citations, and it can detect when nothing relevant was retrieved and decline instead of guessing.
How RAG works under the hood
Underneath the three-step loop, a RAG system does the following. Steps 1 to 3 happen once when content is added or refreshed; steps 4 to 7 happen on every question.
- Ingest. Content is pulled from its sources: website pages, help center articles, PDFs, spreadsheets, transcripts, code repositories.
- Chunk. Each document is split into passages small enough to be retrieved precisely and large enough to stay coherent. Chunking is quietly one of the highest-impact decisions in the whole pipeline.
- Embed and index. Each chunk is converted into an embedding, a numeric vector representing its meaning, and stored in a vector index so similar meanings sit near each other.
- Embed the question. The incoming question is converted into a vector the same way.
- Search. The index returns the chunks nearest to the question vector. Many systems combine this with keyword search, an approach known as hybrid retrieval.
- Assemble the prompt. The top chunks, the question, and the system instructions are combined into a single prompt.
- Generate and cite. The model produces the answer, and the system attaches the sources it used.
None of these steps is exotic on its own. The engineering difficulty is in the tuning: chunk sizes, how many passages to retrieve, how to rank them, what to do when confidence is low, and how often to re-index.
RAG vs fine-tuning: which one do you need?
RAG and fine-tuning solve different problems and are often presented as competitors when they are complements.
Fine-tuning changes the model. It continues training on examples so the model absorbs a style, a format, or a task. Fine-tuning is the right tool for teaching a model to always reply in a particular tone, to output structured JSON reliably, or to handle a specialized task format.
RAG changes the input. The model is untouched; it simply receives better context. RAG is the right tool for facts, especially facts that change.
| Dimension | RAG | Fine-tuning |
|---|---|---|
| What it changes | The prompt (context supplied at question time) | The model weights |
| Best at | Facts, policies, documentation, anything that changes | Tone, format, task behavior, style |
| Updating one fact | Re-index the source document, effective immediately | Retrain or fine-tune again |
| Source citations | Native: the system knows which passages it used | Not possible: knowledge is diffused into weights |
| Setup cost | Low: connect content, index it | High: dataset curation, training runs, evaluation |
| Ongoing cost | Storage and retrieval per query | Repeated training as knowledge changes |
| Failure mode | Retrieves the wrong passage | Confidently states outdated training data |
The research points the same way. In a study published at EMNLP 2024, Microsoft researchers compared unsupervised fine-tuning against retrieval for injecting knowledge into large language models and found that RAG outperformed fine-tuning both for knowledge the model had seen in training and for entirely new knowledge (Ovadia et al., arXiv:2312.05934).
The practical reading: if the question is "how do I get my system to know things", the answer is retrieval. If the question is "how do I get my system to behave a certain way", that is where fine-tuning earns its cost.
Does RAG stop hallucination?
No, and any vendor claiming otherwise is overselling. RAG reduces hallucination by giving the model the facts instead of asking it to recall them, but grounding is a strong constraint rather than a guarantee.
The clearest public evidence comes from a 2024 study by Stanford's RegLab and Institute for Human-Centered AI, which benchmarked commercial legal research tools built on retrieval. The researchers measured hallucination rates above 17% for Lexis+ AI and Ask Practical Law AI, and above 34% for Westlaw AI-Assisted Research. These are professionally built RAG systems from established vendors, evaluated on the domain they were designed for.
Three failure modes account for most of it:
- Retrieval misses. The right passage exists but the search does not surface it, so the model answers from a less relevant one.
- Wrong-context retrieval. A passage that looks relevant is actually inapplicable, for example the right policy from the wrong region or the wrong product tier.
- Sycophancy. The model accepts a false premise embedded in the user's question rather than correcting it.
What follows from this is practical, not discouraging. Retrieval is the largest single accuracy lever available, and it should be paired with the things that catch the residual: source citations so answers can be checked, an explicit instruction to decline when context is thin, a clean content library without contradictions, and a route to a human for anything consequential.
The difference is not intelligence. It is whether the system is answering from memory or from your documents.
Model alone
- Answers from training data frozen at a past date
- Cannot cite a source for anything it says
- Invents plausible detail when it lacks a fact
- Knows nothing specific about your business
- Updating knowledge requires retraining
Model plus retrieval (RAG)
- Answers from content you indexed and control
- Can cite the passages it used
- Can be instructed to decline when nothing relevant is found
- Knows your policies, pricing, and documentation
- Updating knowledge means re-indexing a document
RAG in customer service
Customer service is the most common commercial application of RAG, because the job matches the architecture almost exactly. Most support questions have a correct answer that already exists somewhere in a help center, a policy page, or a product doc. The work is finding it and phrasing it, which is precisely retrieve, ground, generate.
A content-grounded support bot is a RAG system with a chat interface. Its accuracy depends far more on the state of the underlying content than on which language model sits behind it. Three properties separate systems that hold up in production from ones that do not.
Source coverage. Retrieval can only find what was indexed. If billing answers live in a PDF nobody imported, the system cannot answer billing questions. Breadth of connectors is therefore an accuracy feature, not a convenience feature. SiteGPT indexes 12+ source types, including websites and sitemaps, help centers (Zendesk, Freshdesk, Intercom, Confluence, Gitbook), cloud storage (Google Drive, Notion, Dropbox, OneDrive, SharePoint, Box), GitHub, YouTube, and direct file uploads.
Freshness. A RAG answer is only as current as the last index refresh. A system that crawled your pricing page a year ago will quote last year's prices with total confidence, which is worse than not answering at all. Automatic re-syncing is what keeps the index honest; on SiteGPT that runs monthly on Growth, and weekly with a daily scan on Scale.
A defined exit. Every RAG system eventually meets a question its content does not answer. The good ones hand off rather than improvise. Escalation to a human with the transcript intact is the safety net under the retrieval layer.
For a walkthrough of setting one up, see how to create an AI chatbot. For how retrieval-based bots compare with the platforms built around live chat, see SiteGPT vs Intercom and SiteGPT vs Chatbase.
RAG and AI agents
Retrieval is also the knowledge layer underneath most AI agents. An agent that takes actions still needs to know things, and it typically gets that knowledge the same way: by retrieving from an indexed store rather than relying on the model's memory. The difference between the two is what happens after the answer is formed, which is covered in AI agent vs chatbot.
This is also where retrieval stops being an internal implementation detail. SiteGPT exposes a remote MCP server that MCP-compatible assistants connect to over OAuth, so an assistant can manage the knowledge base itself: update sources, review leads, adjust personas. Claude is the confirmed, tested client today, and SiteGPT publishes setup guides for Claude Code, Cursor, and Codex. What the protocol changes about service work is covered in MCP for customer support, and SiteGPT for AI agents lays out what an assistant can operate end to end. The practical setup is walked through in how to build and run a chatbot from Claude.
Should you build RAG or buy it?
Building a RAG system from scratch is a well-trodden weekend project and a genuinely hard production system. The prototype is a few hundred lines. What takes the time is everything after it: chunking strategy, retrieval evaluation, handling contradictory sources, re-indexing on a schedule, permissions, latency, cost control, and knowing when the system should decline to answer.
Is retrieval quality itself the product you are selling?
- If yes, retrieval is your differentiator→Build it, and budget for evaluation infrastructure
- If no, you need accurate answers→Buy a platform that runs the loop
Where does your content actually live?
- If scattered across a site, help center, drive, and files→Buy: connector breadth is the hard part
- If one clean, structured internal database→Building is more tractable
How often does the underlying content change?
- If weekly or faster→Prioritize automatic re-syncing over model choice
- If rarely→Freshness matters less; focus on retrieval quality
Who will own answer quality in six months?
- If no dedicated engineer→Buy: unowned RAG systems decay quietly as content drifts
- If a team that owns it→Either path works
| Scenario | Best pick | Why |
|---|---|---|
| Answering customer questions from existing docs and help center articles | Hosted RAG platform | The job is standard, and connector breadth plus auto-syncing are the parts that are tedious to build. |
| Teaching a model to always reply in a specific tone or JSON format | Fine-tuning | Behavior and format live in the weights. Retrieval cannot enforce a style. |
| Facts that change weekly (pricing, stock, policies) | RAG with automatic re-syncing | Re-indexing a page is instant; retraining a model on a new price is not. |
| Internal knowledge search across a company drive | RAG with permission-aware retrieval | The retrieval layer must respect who is allowed to see which document. |
| A regulated domain where a wrong answer has legal consequences | RAG plus mandatory human review | Stanford measured 17-34% hallucination in professional legal RAG tools. Grounding is not a compliance control. |
| Research or highly specialized task formats | Fine-tuning, optionally combined with RAG | Task behavior comes from training; the facts can still be retrieved. |
| Multilingual customers, content written in one language | RAG | Retrieval and generation handle the language gap; SiteGPT supports 95+ languages. |
| A prototype to show stakeholders next week | Hosted platform trial | Minutes to a working demo against real content, versus days to a rough pipeline. |
For most teams the honest answer is that RAG is infrastructure, not strategy. The differentiator is the quality and organization of the content going in, which is work no platform can do for you.
In regulated domains that content question becomes a legal one. Under HIPAA, the retrieval corpus is exactly where protected health information must not live: on a SiteGPT HIPAA-covered workspace, PHI is scoped to end-user conversations while every indexed source (website pages, uploaded files, snippets, Q&A entries) stays PHI-free. Keeping the corpus clean is also what keeps the number of vendors touching PHI small, which is why connected apps are switched off on those workspaces. Handling PHI at all requires a signed BAA, available on Enterprise, with terms published at sitegpt.ai/legal/baa and current status at sitegpt.ai/hipaa.
Frequently asked questions
Basics
What is RAG in AI? RAG stands for retrieval-augmented generation. It searches a body of text you control, retrieves the passages most relevant to a question, and passes them to a language model as context before the model answers. The model supplies the language; the retrieved passages supply the facts.
What does RAG stand for, and where did the term come from? Retrieval-augmented generation. The term was introduced in a 2020 NeurIPS paper by Patrick Lewis and co-authors at Facebook AI Research, University College London, and New York University.
How does RAG work, step by step? Three steps on every question: retrieve the most relevant passages from an index of your content, ground them into the model's prompt as authorized context, and generate an answer from that context. Behind the loop, content is chunked, embedded, and stored in a vector index.
RAG versus the alternatives
What is the difference between RAG and fine-tuning? Fine-tuning changes the model's weights to teach style, format, and behavior. RAG leaves the model alone and supplies facts at question time. For new or changing facts, retrieval wins: Microsoft researchers reported at EMNLP 2024 that RAG outperformed unsupervised fine-tuning for knowledge injection.
Do you still need RAG if models have very long context windows? Yes, for most business use. A context window determines how much text fits in one request; retrieval determines which text deserves to be there. Business content libraries generally exceed any window, and sending everything on every question is slow and costly.
Is RAG the same as a chatbot trained on your data? In commercial products, usually yes. "Trained on your data" almost always describes retrieval over an index of your content, not retraining a model. That is why answers update as soon as content is re-synced.
Accuracy and limits
Does RAG eliminate hallucination? No. It reduces hallucination substantially but does not remove it. A 2024 Stanford RegLab and HAI study found hallucination rates above 17% in Lexis+ AI and Ask Practical Law AI, and above 34% in Westlaw AI-Assisted Research, all retrieval-based tools.
What makes a RAG system give a wrong answer? Most often retrieval, not the model. The right passage may not be surfaced, a superficially relevant but inapplicable passage may be retrieved, or the model may accept a false premise in the question. Diagnose by checking whether the correct passage was retrieved before blaming the model.
How much content do you need for RAG to work well? Quality beats volume. A few dozen accurate pages outperform thousands of contradictory ones, because retrieval cannot tell which of two conflicting passages is current. Deleting stale content is often the highest-return improvement available.
Implementation
What is a vector database, and is one required? A vector database stores embeddings so passages can be found by meaning rather than exact keywords. RAG systems commonly use one, but keyword or hybrid search can serve the same role. The vector index is an implementation choice within the retrieve step, not part of the definition.
How do you keep a RAG system accurate over time? Re-sync sources on a schedule, keep one authoritative version of each fact, and review the questions it answers badly. An index refreshed a year ago will quote year-old prices with full confidence.
Do you have to build RAG yourself? No. Building means owning chunking, embeddings, indexing, retrieval tuning, and re-syncing. Hosted platforms package the loop: SiteGPT connects 12+ content sources and runs retrieval for you, from $39 per month on annual billing with a 7-day free trial. Build when retrieval is your product; buy when answers are.
Sources
- Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., Kiela, D. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems 33 (NeurIPS 2020). Facebook AI Research, University College London, New York University.
- Magesh, V., Surani, F., Dahl, M., Suzgun, M., Manning, C. D., Ho, D. E. (2024). Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools. Stanford RegLab and Stanford Institute for Human-Centered AI.
- Ovadia, O., Brief, M., Mishaeli, M., Elisha, O. (2024). Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs. Proceedings of EMNLP 2024. Microsoft.
- SiteGPT pricing and human handoff documentation, verified 31 July 2026.
Last updated: July 2026. Pricing and product details verified as of July 2026.