AIF-C01 Practice Question 37: LLM as a Judge
AWS Certified AI Practitioner (AIF-C01) · Test 4 · question 7 of 10
A company wants to compare how helpful and complete the answers from two candidate models are across 800 realistic customer prompts. Hiring human reviewers to grade every response would be too slow and expensive, and word-overlap metrics don't capture helpfulness. The team also wants a short explanation of why each response received its score. Which evaluation approach should it use?
Show answer and explanation
Answer:
In an LLM-as-a-judge evaluation, Amazon Bedrock uses one model (the evaluator) to score another model's responses and provide an explanation of how it scored each prompt and response pair. You supply a dataset of prompts for the generator model, pick built-in metrics or define custom metrics for your business case, and review the scores and explanations in the console, with the full report written to Amazon S3. This covers judgment-based qualities like helpfulness and completeness at a scale that would be slow and costly for human reviewers (option D).
- A is wrong because ROUGE counts overlapping words with a reference text. The stem already says word-overlap metrics don't capture helpfulness, and ROUGE gives no per-response explanation. - B is wrong because a public benchmark measures general capabilities on someone else's tasks, not how well each model answers *this company's* customer prompts. - C is wrong because latency and cost are important selection factors, but they don't measure answer *quality* at all.
Reference: Evaluate model performance using another LLM as a judge