AIF-C01 Practice Question 36: RAG Evaluation
AWS Certified AI Practitioner (AIF-C01) · Test 4 · question 6 of 10
A team built a question-answering assistant with Amazon Bedrock Knowledge Bases. Before launch, it wants to measure two things with a dataset of questions and expected answers: whether the passages retrieved for each question are relevant, and whether the generated answers are correct and faithful to those passages. It also wants to compare two different knowledge base configurations. What should the team use?
Show answer and explanation
Answer:
Amazon Bedrock RAG evaluations measure how effectively a RAG system retrieves relevant information from its data sources and how effective the generated responses are at answering questions. A retrieve-and-generate job evaluates both the retrieved data and the responses from the generator model. It uses metrics such as context relevance, correctness, completeness and faithfulness, and it checks results against the ground truth in your dataset. AWS notes that the results let you compare different knowledge bases and choose the best one (option A). A *retrieve-only* job would cover just the retrieval half.
- B is wrong because BLEU measures word overlap between a generated text and a reference. It says nothing about whether the *retrieved passages* were relevant, or whether an answer is faithful to them. - C is wrong because a model evaluation job scores the foundation model's responses to prompts. It skips the retrieval step, which is half of what the team needs to measure. - D is wrong because operational metrics such as latency and error counts show whether the system is *running*, not whether its retrieved context and answers are *good*.
Reference: Evaluate the performance of RAG sources using Amazon Bedrock evaluations