What Is RAG? How Retrieval-Augmented Generation Works

RAG, or retrieval-augmented generation, is a way to make a large language model answer from your own, current information: before the model responds, the application retrieves relevant passages from a knowledge source and adds them to the prompt. The model then writes its answer from those passages and can cite them, instead of relying only on what it learned in training.

This guide explains what RAG is and why it exists, walks through how it works step by step, and shows where companies use it and what it is built from on AWS, Google Cloud and Azure. It then covers the types of RAG, from classic to graph and agentic, the settings that decide answer quality, and what the AWS, Microsoft, Google and Databricks exams ask about it. The same material is drawn on a whiteboard in the video linked at the end.

What is RAG?

The problem RAG solves

Ask a general-purpose chatbot about your company's holiday policy and it may answer confidently, and wrongly. There are three reasons:

• Training data is static. A model learns from data collected up to a cut-off date and knows nothing that happened after it.
• It never saw your private data. Models are trained on public data, not on your contracts, your wiki or last week's price list.
• It fills gaps convincingly. When a model lacks an answer, it can still produce something that sounds right but is false, or give an out-of-date or generic answer. That is what people call a hallucination.

The definition

RAG fixes this by letting the model look things up first. AWS describes it as having the model consult an authoritative knowledge base outside its training data before it responds. Google calls the same idea grounding: connecting a model's output to verifiable sources of information, which lowers the likelihood of hallucinations.

A useful analogy: a plain model sits a closed-book exam, answering everything from memory. RAG turns it into an open-book exam. The model still has to read and write the answer, but it gets the right pages in front of it first.

The idea comes from a 2020 research paper by Patrick Lewis and colleagues, presented at NeurIPS 2020. It combined a pretrained language model, which the authors called parametric memory, with a dense vector index of Wikipedia, the non-parametric memory, that the model could retrieve from.

Why not just retrain the model?

Because retraining costs time and money, and your documents keep changing. With RAG you update the documents, not the model. AWS calls RAG a more cost-effective way to introduce new data to a model than retraining it. RAG brings other benefits too:

• Current answers. Feed the system the latest policies, research or news, and answers follow.
• Citations. Because each answer is built from specific passages, the application can show which source chunks it used, so users can check them.
• Control. You decide which sources the model can draw on, and you can restrict retrieval by a user's authorization level, so people only get answers from documents they may see.

How RAG works, step by step

A RAG system runs two pipelines: one that prepares your documents ahead of time, and one that runs every time someone asks a question.

The ingestion pipeline

1. Convert documents to text. PDFs, web pages, wiki articles and the like are turned into plain text.
2. Split the text into chunks. Each document is cut into small pieces, a few hundred tokens each, so a search can return just the relevant part.
3. Embed each chunk. An embedding model turns each chunk into a vector, a long list of numbers that captures its meaning.
4. Store the vectors in a vector index. Each vector is kept with its text and a pointer to the document it came from, so answers can cite it later.

Why embeddings?

Numbers can be compared mathematically, and embeddings are built so that text with similar meaning ends up with similar vectors. That means a question about "vacation days" can find a chunk about "annual leave" even though they share no words. Keyword search alone can't do that. Searching by vector similarity is often called semantic search, and the store that holds the vectors is a vector store or vector database.

The query pipeline

When a user asks a question:

1. Embed the question with the same embedding model used for the documents.
2. Retrieve. The vector index returns the chunks closest to the question's vector, typically the top few results (a k-nearest-neighbours search).
3. Optionally rerank. A second model re-scores those results so the most useful come first.
4. Augment. The chosen chunks are added to the prompt next to the question.
5. Generate. The language model writes an answer from that context, with citations to the chunks it used.

Microsoft sums it up in the three words of the name: retrieve the relevant content, augment the prompt with it, and generate the answer.

Keeping it fresh

RAG is only as current as its index. When documents change, they must be re-processed and re-embedded. On AWS, an Amazon Bedrock knowledge base doesn't see a changed file in S3 until you sync its data source; syncs are incremental, so only changed files are processed. Forgetting this is a classic cause of "the bot still quotes the old policy", and the subject of an AIF-C01 practice question on stale knowledge base answers.

Where RAG is used

RAG fits anywhere answers must come from an organisation's own, current information:

• Employee assistants. AWS's own example is an HR chatbot that answers leave questions from the policy documents and the employee's own leave record.
• Customer support chatbots that answer from product documentation.
• Enterprise search across SharePoint, databases and file storage, showing each user only the documents they are allowed to see.

Some published examples, with figures as the companies report them:

• Ring, part of Amazon, built a multi-locale customer support chatbot with RAG on Amazon Bedrock Knowledge Bases, serving 10 international Regions, using Amazon OpenSearch Serverless as the vector store and Titan embeddings. Ring says moving away from per-Region infrastructure cut the cost of adding each new locale by 21 percent.
• JoongAng Ilbo, a Korean newspaper, built RAG over 60 years of its articles, with Gemini and Vertex AI Search (now called Agent Search). It says limiting answers to its own news data greatly raised accuracy.
• PwC's private generative AI assistant, ChatPwC, uses Azure AI Search for RAG and is used by about 200,000 PwC people.
• Microsoft's Licensing Navigator uses RAG with Azure AI Search to answer more than 1,000 questions a week.

The building blocks of a RAG system

Whatever the cloud, a RAG system is built from the same blocks: a place for documents, an embedding model, a vector store, a language model, and often a guardrail. Keep the roles apart, because exams test them: the embedding model turns text into vectors, the vector store searches them, and the language model writes the answer.

On AWS

• Documents: Amazon S3, or connectors such as SharePoint and Confluence for managed knowledge bases.
• Embeddings: Amazon Titan Text Embeddings V2 (up to 8,192 tokens of input, 1,024-dimension vectors) or Cohere Embed.
• Vector stores: Amazon OpenSearch Serverless, Aurora PostgreSQL, Amazon Neptune Analytics, Amazon S3 Vectors, and third-party stores such as Pinecone and MongoDB Atlas. S3 Vectors became generally available in December 2025 and suits lower-cost, infrequently queried vectors.
• Generation: a Bedrock foundation model, called through the RetrieveAndGenerate API.
• Managed pipeline: Amazon Bedrock Knowledge Bases runs ingestion, retrieval and generation for you. AWS now recommends its managed knowledge base option.

On Google Cloud

Google renamed Vertex AI to Gemini Enterprise Agent Platform in April 2026. On it:

• RAG Engine manages a RAG pipeline. Documents come from Cloud Storage or Google Drive, the default embedding model is text-embedding-005, and vectors go into the managed RagManagedDb (the default), Vector Search or Pinecone.
• Agent Search (formerly Vertex AI Search) offers ready-made RAG with its own retrieval and ranking.
• Gemini writes the answer.

On Azure

• Documents: Azure Blob Storage, among other sources.
• Embeddings: Azure OpenAI models such as text-embedding-3-large and text-embedding-3-small.
• Azure AI Search is both the vector store and the retriever: it holds the vectors and runs vector, keyword and hybrid queries with semantic ranking. It also underpins Foundry IQ in the Microsoft Foundry portal (formerly Azure AI Foundry).

Guardrails and frameworks

Retrieval narrows what a model sees, but it doesn't force the model to stick to it. Two managed checks help:

• Amazon Bedrock Guardrails has a contextual grounding check that scores whether a response is grounded in the source and relevant to the question, and blocks it below a threshold you set between 0 and 0.99. AWS says conversational chatbot use cases are not supported. An AIF-C01 question on grounding checks tests when to use it.
• Azure AI Content Safety groundedness detection checks responses against the grounding sources.

To build a pipeline yourself, open-source frameworks such as LangChain and LlamaIndex cover the same steps of indexing, retrieval and generation.

Types of RAG

The "types of RAG" lists on the web vary a lot. A practical way to see them is as layers, each one fixing a weakness of the one before.

1. Classic RAG

One question, one vector similarity search, the top chunks into the prompt. It is simple and fast. Its weakness: vectors match meaning, so they can miss exact things such as a product code, a name or a date.

2. Hybrid search

Run a keyword search (such as BM25) and a vector search on the same question in parallel, then merge the two result lists. Azure AI Search merges them with Reciprocal Rank Fusion (RRF). The keyword side catches the exact part number; the vector side catches the meaning. Amazon Bedrock Knowledge Bases lets you choose hybrid or semantic search where the vector store supports it. Hybrid search is the answer to an AI-300 question on exact-match part numbers and an AI-103 question on hybrid search with the semantic ranker.

3. Reranking

The first search is fast but rough. A reranker model scores the top results again against the question and puts the most useful first:

• Amazon Bedrock: Amazon Rerank 1.0 and Cohere Rerank 3.5.
• Azure: the semantic ranker, which uses models adapted from Bing, re-scores the top 50 results only. So a query must return at least 50 results to give it its full input, which an AI-300 question on the k value tests.
• Google: a ranking API that answers in under 100 milliseconds, or a Gemini model as an LLM reranker, which takes a second or two.

4. GraphRAG

GraphRAG, from Microsoft Research, first has a language model extract the entities in your documents and the relationships between them into a knowledge graph. That helps with questions that connect facts spread across many documents, or ask about a whole collection, such as "what are the top themes in this data?", which chunk-by-chunk similarity search handles poorly. On AWS, Amazon Bedrock Knowledge Bases supports GraphRAG with Amazon Neptune Analytics, generally available since March 2025: after the vector search it follows linked nodes in the graph.

5. Agentic RAG

Instead of one search, a model plans the retrieval. It breaks a complex question into smaller subqueries, runs them, and combines the results. Azure AI Search calls this agentic retrieval and runs the subqueries in parallel; its query planning is still in preview. Amazon Bedrock offers query decomposition, and an agentic retrieval option that retrieves iteratively and judges whether the results are enough before answering.

6. Beyond text: multimodal and structured data

• Multimodal RAG: Amazon Bedrock Knowledge Bases can index images, audio and video. You can search with an image to find similar images, or search video with text and get back timestamped segments, using models such as Amazon Nova Multimodal Embeddings.
• Structured data: a Bedrock knowledge base can turn a natural-language question into SQL and answer from Amazon Redshift (with the AWS Glue Data Catalog), without moving the data.

Settings that decide answer quality

Whichever type you build, a few settings make most of the difference.

Chunk size and overlap

Small chunks give precise matches but can lose context; large chunks are more general and can bury the detail. Defaults differ: Amazon Bedrock's default chunking is about 300 tokens, respecting sentence boundaries, while Google's RAG Engine defaults to 1,024-token chunks with 256 tokens of overlap. Overlap keeps a sentence that straddles a boundary findable from both sides. Two refinements:

• Hierarchical chunking matches small child chunks but returns their larger parent, to get both precision and context.
• Semantic chunking splits text by meaning rather than length; on Bedrock it uses a foundation model, so it costs extra.

The exams ask about this directly: an AIF-C01 question on chunking asks what it is called, and an AI-103 question on chunking strategy and a Databricks question on the embedding model's token limit ask how to tune it.

How many results, and which

Bedrock returns five results by default. Return too few and the answer may miss something; too many and you fill the prompt with noise. Metadata filters narrow retrieval before similarity ranking, for example to only the newest version of each document.

The embedding model

Use the same embedding model for indexing and for questions; vectors from different models can't be compared. Changing the model means embedding the whole collection again. Google's RAG Engine fixes the embedding model for the life of a corpus.

RAG vs fine-tuning

The most common confusion. RAG adds knowledge at question time, which suits private or fast-changing information. Fine-tuning trains a new version of the model to change its behaviour, style or format; Microsoft describes it as a way to change how a model behaves rather than to add fresh knowledge. They also combine: retrieval for the facts, fine-tuning for the house style.

The full comparison, including prompt engineering and continued pre-training, is in the guide to prompt engineering vs RAG vs fine-tuning.

What the exams ask about RAG

RAG appears in several exam guides:

• AWS Certified AI Practitioner (AIF-C01): Domain 3, Task 3.1, asks you to define RAG and its business applications, such as Amazon Bedrock Knowledge Bases, to know which AWS services store vectors, and to weigh the cost of pre-training, fine-tuning, in-context learning and RAG. See the AIF-C01 practice tests and the AIF-C01 question list.
• Microsoft AI-103: implementing RAG, and semantic, hybrid and vector search for grounding. See the AI-103 practice tests and question list.
• Microsoft AI-300: optimizing RAG with thresholds, chunk sizes, embedding models and hybrid search. See the AI-300 practice tests and question list.
• Google Generative AI Leader: grounding techniques, including prebuilt RAG with Agent Search, RAG APIs and grounding with Google Search. See the Generative AI Leader practice tests and question list.
• Databricks Generative AI Engineer Associate: building and deploying RAG applications. See the Databricks practice tests and question list.

The traps they keep setting:

• RAG vs fine-tuning. Private, changing knowledge with citations means RAG, as in the question used in the video, an AIF-C01 question on changing internal documents, and the Generative AI Leader question on grounding internal policies.
• RAG reduces hallucinations; it doesn't eliminate them. If retrieval brings back the wrong chunks, the answer can still be wrong. Evaluate both the retrieval and the generated answer, as in the AIF-C01 question on RAG evaluation.
• Sync after changes. Replaced files don't reach a knowledge base until its data source syncs.
• Managed vs build-your-own. Needing custom chunking, embeddings or vector stores points to lower-level RAG APIs rather than a prebuilt search product, as in the Generative AI Leader question on RAG control.

Frequently asked questions

Does RAG stop hallucinations?

No. Grounding a model in retrieved sources lowers the chance of hallucinations, but it doesn't remove it. Poor retrieval leads to poor answers, which is why grounding checks and evaluation exist.

Is RAG cheaper than fine-tuning?

For adding new or changing information, AWS calls RAG the more cost-effective option, because you update documents instead of retraining. It isn't always cheaper overall: every question also pays for retrieval and a longer prompt. The two solve different problems.

What is a vector database?

A store that holds embeddings and finds the vectors closest to a query vector. In a RAG system it does the searching; it doesn't write answers. Examples include Amazon OpenSearch Serverless, Azure AI Search and Vector Search on Google Cloud.

What is chunking in RAG?

Splitting documents into smaller pieces before they are embedded, so retrieval can return just the relevant passage. Chunk size and overlap are among the most important settings for answer quality.

Do I need to retrain the model to use RAG?

No. The model stays as it is. You index your documents, retrieve from them at question time, and re-index when they change.

Watch the whiteboard video

The video draws both pipelines on one board, from chunking and embedding to retrieval and the grounded answer, compares the types of RAG, and works through the RAG vs fine-tuning trap with a practice question: RAG Explained: How It Works, Types of RAG and Where It's Used on YouTube.