AIF-C01 Practice Question 17: Model Distillation
AWS Certified AI Practitioner (AIF-C01) · Test 2 · question 7 of 10
A retailer uses a large, highly capable foundation model to classify and route millions of customer messages a day. Its answers are excellent, but per-request cost and latency are too high at this volume. A smaller model in the same family is much cheaper and faster, but its accuracy on this task is too low. The retailer has plenty of real customer messages but no human-labeled answers. Which approach best reduces cost and latency while keeping accuracy close to the large model?
Show answer and explanation
Answer:
Model distillation transfers knowledge from a larger, more capable model (the teacher) to a smaller, faster, cheaper one (the student) for a specific use case. In Amazon Bedrock Model Distillation you supply prompts. The teacher generates the responses, which serve as synthetic training data, and the student is fine-tuned on them. That fits the constraints here: plenty of real prompts, no human labels, and a need for small-model cost and latency with accuracy near the large model's (option D). You pay once for the teacher's inference during distillation, then run the cheaper student on millions of daily requests.
- A is wrong because it keeps the large model, so cost and latency don't improve. The extra retrieved passages add input tokens and make each call slower and more expensive. - B is wrong because long few-shot prompts add tokens to every one of millions of daily calls. That eats into the savings, and in-context examples often can't close a large quality gap. - C is wrong because pre-training from scratch is the most expensive customization option by far. It needs huge amounts of data and compute, and a catalog of messages doesn't teach the model the right answers.
Reference: Customize a model with distillation in Amazon Bedrock