What Is RAG? Retrieval-Augmented Generation, Explained for Business

Retrieval-augmented generation (RAG) is a technique that retrieves relevant passages from a body of text you control and gives them to a large language model as context, so the model answers from those passages instead of from what it memorized during training.

A language model on its own only knows what was in its training data. It cannot tell you where an answer came from, and it has no idea your refund policy changed last month. RAG gives it your documents to read before it answers.

What does retrieval-augmented generation mean?

RAG is an architecture that many products use. It describes any system that answers a question in two stages: first find relevant source material, then generate an answer conditioned on that material.

The 2020 paper by Lewis et al. that named the technique framed it as combining two kinds of memory. Parametric memory is what a model absorbed into its weights during training: broad, fluent, frozen at a point in time, and impossible to inspect. Non-parametric memory is an external, searchable store of text: specific, updatable, and traceable to a document.

RAG connects the two. The model keeps its language ability, and the facts come from the store.

"Trained on your data" is usually a marketing phrase for this. When a chatbot vendor says the bot is trained on your website, the mechanism is almost always retrieval: your content is indexed and looked up at question time. The model itself does not change, so answers can update the moment your content does.

How a bot built this way compares with rule-based and general-purpose chatbots is covered in SiteGPT vs generic chatbots.

Why does RAG exist?

Language models on their own fail in three well-documented ways. The original paper names all three in its opening: such models cannot easily expand or revise their memory, cannot straightforwardly explain their predictions, and may produce hallucinations.

First, they cannot be updated. A model's knowledge is fixed at training time. Changing one fact means retraining or fine-tuning, and both are slow and expensive. A business whose prices, policies, and product line change monthly cannot run on a memory that changes yearly.

Second, they cannot show their work. Ask a plain model why it said something and it produces a plausible explanation, not a citation. For customer service, compliance, or anything a customer might act on, an answer nobody can verify has little value.

Third, they fill gaps with fluent invention. When a model lacks a fact, it does not reliably say so. It generates the most likely continuation, and that text reads like a correct answer.

Retrieval addresses all three. The store can be re-indexed in minutes, every answer traces to a passage, and the model is given the facts rather than asked to recall them. The paper's own evaluation found that RAG models generated "more specific, diverse and factual language" than a comparable model without retrieval.

How does the RAG loop work?

Every RAG system runs the same three steps on every question: retrieve, ground, generate. That holds for a weekend prototype and an enterprise deployment alike, and naming the steps makes it easier to find the one that is failing.

Step 1: Retrieve

The system turns the incoming question into a search over your indexed content and returns the top handful of passages. Retrieval usually works on meaning rather than keywords, so "can I get my money back" finds a passage titled "Refund policy" even with no shared words.

This step decides the ceiling on answer quality. If the right passage is not retrieved, no amount of model quality recovers it.

Step 2: Ground

The retrieved passages are assembled into the prompt as the authorized context, typically with an instruction along the lines of answer using only the following sources. Grounding is what converts a general-purpose model into one that speaks specifically about your business.

The instruction matters as much as the content. A well-grounded system is told what to do when the passages do not contain the answer: say so, or hand off to a human.

Step 3: Generate

The model writes the answer from the grounded context. Because the system knows exactly which passages it supplied, it can attach citations, and it can detect when nothing relevant was retrieved and decline instead of guessing.

What happens inside a RAG system?

Underneath the three-step loop, a RAG system runs seven steps. Steps 1 to 3 happen once when content is added or refreshed. Steps 4 to 7 happen on every question.

1. Ingest. Content is pulled from its sources: website pages, help center articles, PDFs, spreadsheets, transcripts, code repositories. Content ingestion explains how this works and why connector breadth decides what the system can answer.
2. Chunk. Each document is split into passages small enough to be retrieved precisely and large enough to stay coherent. Chunk size has a large effect on answer quality.
3. Embed and index. Each chunk is converted into an embedding, a numeric vector representing its meaning, and stored in a vector index so similar meanings sit near each other.
4. Embed the question. The incoming question is converted into a vector the same way.
5. Search. The index returns the chunks nearest to the question vector. Many systems combine this with keyword search, an approach known as hybrid retrieval.
6. Assemble the prompt. The top chunks, the question, and the system instructions are combined into a single prompt.
7. Generate and cite. The model produces the answer, and the system attaches the sources it used.

None of these steps is exotic on its own. The engineering difficulty is in the tuning: chunk sizes, how many passages to retrieve, how to rank them, what to do when confidence is low, and how often to re-index.

Should you use RAG or fine-tuning?

Use RAG for facts and fine-tuning for behavior. The two solve different problems and often work together.

Fine-tuning changes the model. It continues training on examples so the model absorbs a style, a format, or a task. Fine-tuning is the right tool for teaching a model to always reply in a particular tone, to output structured JSON reliably, or to handle a specialized task format.

RAG changes the input. The model is untouched and receives better context. RAG is the right tool for facts, especially facts that change.

The research points the same way. In a study published at EMNLP 2024, Microsoft researchers compared unsupervised fine-tuning with retrieval for adding knowledge to large language models. RAG outperformed fine-tuning for knowledge the model had seen in training and for entirely new knowledge (Ovadia et al., arXiv:2312.05934).

If you want your system to know things, use retrieval. If you want it to behave a certain way, fine-tuning is worth the cost.

Does RAG stop hallucination?

No, and any vendor claiming otherwise is overselling. RAG reduces hallucination by giving the model the facts instead of asking it to recall them, but grounding is a strong constraint and still not a guarantee.

The clearest public evidence comes from a 2024 study by Stanford's RegLab and Institute for Human-Centered AI, which benchmarked commercial legal research tools built on retrieval. The researchers measured hallucination rates above 17% for Lexis+ AI and Ask Practical Law AI, and above 34% for Westlaw AI-Assisted Research. These are professionally built RAG systems from established vendors, evaluated on the domain they were designed for.

Three failure modes account for most of it:

• Retrieval misses. The right passage exists but the search does not surface it, so the model answers from a less relevant one.
• Wrong-context retrieval. A passage that looks relevant does not apply, for example the right policy from the wrong region or the wrong product tier.
• Sycophancy. The model accepts a false premise embedded in the user's question rather than correcting it.

Retrieval is the largest single accuracy lever available. Pair it with four controls that catch what it misses: source citations, an instruction to decline when context is thin, a content library without contradictions, and a route to a human for anything consequential.

How to build those controls, and what public chatbot failures teach about them, is covered in how to stop a customer service chatbot hallucinating.

How is RAG used in customer service?

Customer service is the most common commercial use of RAG, because the job matches the architecture. Most support questions have a correct answer that already exists in a help center, a policy page, or a product doc. The work is finding it and phrasing it, and that is retrieve, ground, generate.

A content-grounded support bot is a RAG system with a chat interface. Its accuracy depends far more on the state of the underlying content than on which language model sits behind it. Three properties decide whether one holds up in production.

The first is source coverage. Retrieval can only find what was indexed. If billing answers live in a PDF nobody imported, the system cannot answer billing questions, so connector breadth directly affects accuracy. SiteGPT indexes 12+ source types, including websites and sitemaps, help centers (Zendesk, Freshdesk, Intercom, Confluence, Gitbook), cloud storage (Google Drive, Notion, Dropbox, OneDrive, SharePoint, Box), GitHub, YouTube, and direct file uploads.

The second is freshness. A RAG answer is only as current as the last index refresh.

A system that crawled your pricing page a year ago will quote last year's prices with total confidence. Automatic re-syncing keeps the index current. On SiteGPT that runs monthly on Growth, and weekly with a daily scan on Scale.

The third is a defined exit. Every RAG system eventually meets a question its content does not answer, and a well-built one hands off to a person instead of improvising. Escalation to a human with the transcript intact is the safety net under the retrieval layer.

For a walkthrough of setting one up, see how to create an AI chatbot. For how retrieval-based bots compare with the platforms built around live chat, see SiteGPT vs Intercom and SiteGPT vs Chatbase.

How does RAG relate to AI agents?

Retrieval is also the knowledge layer underneath most AI agents. An agent that takes actions still needs to know things, and it usually gets that knowledge by retrieving from an indexed store. The two differ in what happens after the answer is formed. AI agent vs chatbot covers that difference, and the AI agents hub collects the rest of the cluster.

Agents can also manage the retrieval layer directly. SiteGPT exposes a remote MCP server that MCP-compatible assistants connect to over OAuth, so an assistant can manage the knowledge base itself: update sources, review leads, adjust personas.

Claude is the confirmed, tested client today, and SiteGPT publishes setup guides for Claude Code, Cursor, and Codex. What the protocol changes about service work is covered in MCP for customer support, and SiteGPT for AI agents lays out what an assistant can operate end to end. The practical setup is walked through in how to build and run a chatbot from Claude.

Should you build RAG or buy it?

Buy it unless retrieval quality is your product. A RAG prototype is a weekend project of a few hundred lines, and a production system is hard. What takes the time is everything after it: chunking strategy, retrieval evaluation, handling contradictory sources, re-indexing on a schedule, permissions, latency, cost control, and knowing when the system should decline to answer.

For most teams RAG is infrastructure. Results depend on the quality and organization of the content going in, and no platform can do that work for you.

In regulated domains the content question becomes a legal one. In financial services, a grounded answer from approved documents explains a product, and an ungrounded one can invent a commitment. AI chatbot for financial services covers that case.

Under HIPAA, protected health information (PHI) must stay out of the retrieval corpus. On a SiteGPT HIPAA-covered workspace, PHI is scoped to end-user conversations while every indexed source (website pages, uploaded files, snippets, Q&A entries) stays PHI-free. A clean corpus also keeps the number of vendors touching PHI small, so connected apps are switched off on those workspaces.

Handling PHI at all requires a signed BAA, available on Enterprise, with terms published at sitegpt.ai/legal/baa and current status at sitegpt.ai/hipaa.

Frequently asked questions

Basics

What is RAG in AI? RAG stands for retrieval-augmented generation. It searches a body of text you control, retrieves the passages most relevant to a question, and passes them to a language model as context before the model answers. The model supplies the language. The retrieved passages supply the facts.

What does RAG stand for, and where did the term come from? Retrieval-augmented generation. The term was introduced in a 2020 NeurIPS paper by Patrick Lewis and co-authors at Facebook AI Research, University College London, and New York University.

How does RAG work, step by step? Three steps on every question: retrieve the most relevant passages from an index of your content, ground them into the model's prompt as authorized context, and generate an answer from that context. Behind the loop, content is chunked, embedded, and stored in a vector index.

RAG versus the alternatives

What is the difference between RAG and fine-tuning? Fine-tuning changes the model's weights to teach style, format, and behavior. RAG leaves the model alone and supplies facts at question time. For new or changing facts, retrieval wins: Microsoft researchers reported at EMNLP 2024 that RAG outperformed unsupervised fine-tuning for knowledge injection.

Do you still need RAG if models have very long context windows? Yes, for most business use. A context window determines how much text fits in one request. Retrieval determines which text deserves to be there. Business content libraries generally exceed any window, and sending everything on every question is slow and costly.

Is RAG the same as a chatbot trained on your data? In commercial products, usually yes. "Trained on your data" almost always describes retrieval over an index of your content, not retraining a model. That is why answers update as soon as content is re-synced.

Accuracy and limits

Does RAG eliminate hallucination? No. It reduces hallucination substantially but does not remove it. A 2024 Stanford RegLab and HAI study found hallucination rates above 17% in Lexis+ AI and Ask Practical Law AI, and above 34% in Westlaw AI-Assisted Research, all retrieval-based tools.

What makes a RAG system give a wrong answer? Most often retrieval, not the model. The right passage may not be surfaced, a superficially relevant but inapplicable passage may be retrieved, or the model may accept a false premise in the question. Diagnose by checking whether the correct passage was retrieved before blaming the model.

How much content do you need for RAG to work well? Quality beats volume. A few dozen accurate pages outperform thousands of contradictory ones, because retrieval cannot tell which of two conflicting passages is current. Deleting stale content is often the highest-return improvement available.

Implementation

What is a vector database, and is one required? A vector database stores embeddings so passages can be found by meaning rather than exact keywords. RAG systems commonly use one, but keyword or hybrid search can serve the same role. The vector index is an implementation choice within the retrieve step, not part of the definition.

Does RAG work in other languages? Yes. Retrieval and generation both work across languages, so a RAG system can retrieve a passage in one language and answer in another. Quality depends on how well the embedding model handles those languages. SiteGPT supports 95+ languages, so content written once can answer questions asked in many.

How do you keep a RAG system accurate over time? Re-sync sources on a schedule, keep one authoritative version of each fact, and review the questions it answers badly. An index refreshed a year ago will quote year-old prices with full confidence.

Do you have to build RAG yourself? No. Building means owning chunking, embeddings, indexing, retrieval tuning, and re-syncing. Hosted platforms package the loop: SiteGPT connects 12+ content sources and runs retrieval for you, from $39/mo billed yearly ($59 billed monthly) with a 7-day free trial. Build only when retrieval quality is the product you sell.

Sources

• Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., Kiela, D. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems 33 (NeurIPS 2020). Facebook AI Research, University College London, New York University.
• Magesh, V., Surani, F., Dahl, M., Suzgun, M., Manning, C. D., Ho, D. E. (2024). Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools. Stanford RegLab and Stanford Institute for Human-Centered AI.
• Ovadia, O., Brief, M., Mishaeli, M., Elisha, O. (2024). Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs. Proceedings of EMNLP 2024. Microsoft.
• SiteGPT pricing and human handoff documentation, verified 31 July 2026.