AIF-C01 Practice Question 15: Semantic Summary Evaluation
AWS Certified AI Practitioner (AIF-C01) · Test 2 · question 5 of 10
A team is comparing two foundation models for summarizing support tickets against human-written reference summaries. Reviewers notice that one model often writes accurate summaries in different words from the references, such as "refund issued" where the reference says "money returned to customer", and a word-overlap score penalizes it for this. Which automatic metric better captures whether the meaning matches the reference?
Show answer and explanation
Answer:
BERTScore uses pre-trained contextual embeddings from BERT models and matches words in the candidate and reference text by cosine similarity. Paraphrases that mean the same thing still score well, even without shared words, which is what this team needs (option C). Amazon Bedrock model evaluation uses BERTScore as the accuracy metric for text summarization.
- A is wrong because ROUGE, a common summarization metric, measures overlapping n-grams or the longest common word sequence. "Refund issued" and "money returned to customer" share no words, so ROUGE penalizes an accurate paraphrase. That's the problem the reviewers noticed. - B is wrong because BLEU, designed for machine translation, is also based on n-gram overlap. It has the same blind spot for paraphrases. - D is wrong because classification accuracy compares a discrete predicted label with a ground-truth label. It doesn't apply to free-text summaries.
References: Text summarization for model evaluation in Amazon Bedrock; Review metrics for an automated model evaluation job in Amazon Bedrock (console)