Databricks Generative AI Engineer Associate Practice Question 4: Hallucination Diagnosis Traces
Databricks Certified Generative AI Engineer Associate · Test 1 · question 4 of 10
Users of an internal benefits RAG assistant report that it confidently gives outdated reimbursement limits. The team runs mlflow.genai.evaluate() on 200 labeled questions:
- Correctness (against reference answers) fails on 38% of questions - RetrievalGroundedness passes on 97% of questions - For the failing questions, the retriever spans in the MLflow traces show chunks from the 2023 benefits handbook, which the 2025 handbook has replaced. Both handbooks are in the index.
Which change is most likely to fix the wrong answers?
Show answer and explanation
Answer:
The metrics separate the two halves of the RAG pipeline. RetrievalGroundedness passing on 97% of questions means the answers faithfully reflect the retrieved context, so the generator is not making things up. Correctness failing, combined with the traces, shows that the retrieved context itself is wrong: the retriever returns chunks from an outdated document. This is a data and retrieval problem. The fix is to curate the source data (drop the superseded handbook) or apply a metadata filter such as a document version or effective date so only current content is retrieved.
Option A is wrong. A stronger model given stale context will still summarize the stale context faithfully. The model is not the component that is failing.
Option B is wrong. Temperature affects sampling randomness, not the facts in the prompt. The answers are already grounded, so the model is not inventing values.
Option C is wrong. The model already answers from the provided context; that is what the high groundedness score measures. A stricter instruction cannot fix context that is outdated.
Reference: Built-in LLM judges