AI-103 Practice Question 77: Relevance Evaluator Choice
Microsoft Azure AI Apps and Agents Developer Associate (AI-103) · Test 8 · question 7 of 10
A RAG chatbot is evaluated for groundedness and passes consistently; every answer is traceable to the retrieved context with no fabricated content. Manual review turns up a different problem: many answers are technically accurate against the context but wander into tangential detail or only partially address what the user actually asked, so the response feels evasive. The team has no ground-truth answer set to compare against, only the query, the retrieved context, and the model's response.
Which built-in evaluator should they add to catch this failure automatically?
Show answer and explanation
Answer:
Relevance is a *system* evaluator that "measures the accuracy, completeness, and direct relevance of the response to the query," using only query and response as inputs, no ground truth required. That's exactly the gap here: groundedness already confirms the response doesn't fabricate anything, but it says nothing about whether the response actually addresses the question directly rather than drifting into accurate-but-tangential territory, which is what Relevance is built to catch.
- A is wrong because Groundedness Pro is a stricter version of the evaluator the team already has passing; it checks that the response strictly adheres to the retrieved context, not whether the response is on-topic for the query. - B is wrong because Response Completeness requires ground_truth to compare the response against, a labeled expected answer. The team explicitly has no ground-truth set, which rules this evaluator out regardless of how well it fits the symptom. - D is wrong because Retrieval is a *process* evaluator that scores how relevant the retrieved context chunks are to the query; it evaluates the search step, not the final generated response the users are actually complaining about.
Reference: RAG evaluators for generative AI