MLA-C01 Practice Question 65: Foundation Model Evaluation
AWS Certified Machine Learning Engineer – Associate (MLA-C01) · Test 7 · question 5 of 10
A software company will use a foundation model on Amazon Bedrock to summarize internal support tickets for engineers. Three candidate models are shortlisted. Before choosing one, the team wants to score each model on 500 of its own tickets with computed quality metrics, and also have senior support engineers rate the summaries for usefulness and tone. No model can be exposed to customers or production traffic during the comparison.
What should the team use?
Show answer and explanation
Answer:
Amazon Bedrock evaluations assess models on your own prompt datasets before you commit to one. *Automatic* evaluations, including ones that use an LLM as a judge, produce computed scores and metrics. Model evaluation jobs that use *human workers* let your own team rate the responses against metrics you define. Running both on the same 500 tickets gives the team computed quality scores and engineers' judgments of usefulness and tone, with no production exposure (option D).
- A is wrong because latency and token counts measure speed and cost. They say nothing about whether a summary is accurate or useful. - B is wrong because a production A/B test exposes the candidates to live traffic, which the team ruled out. It is also a way to validate a choice, not to shortlist before choosing. - C is wrong because Model Monitor watches models deployed on SageMaker AI endpoints, and its model quality checks compare predictions with labels such as classes or numeric values. It is not built to grade free-text summaries from Bedrock models.
Reference: Evaluate the performance of Amazon Bedrock resources