MLA-C01 Practice Question 36: Shadow Variant Comparison
AWS Certified Machine Learning Engineer – Associate (MLA-C01) · Test 4 · question 6 of 10
An ML engineer retrained a recommendation model and moved it to a newer inference container. Offline metrics look better, but the team has been burned before by models whose latency and error rate under real production traffic differed from what they saw in testing. The engineer wants to measure the new version's latency and errors on live production requests for a week before deciding whether to promote it. Customers must not receive any response from the new version during that week.
What should the engineer do on the existing real-time endpoint?
Show answer and explanation
Answer:
A SageMaker AI shadow test deploys the candidate as a *shadow variant* on the same endpoint. It routes a copy of each inference request to the shadow variant in real time, but only the production variant's responses go back to the caller. The engineer sees the new model and container handle real traffic, with operational metrics such as latency and error rate on a live dashboard, and no customer is ever served by the new version. Afterwards the engineer can promote the shadow variant or discard it (option B).
- A is wrong because production variants split real traffic. With a 5% weight, 5% of customers get the new version's responses, which the requirement forbids. - C is wrong for the same reason. A canary deployment deliberately sends real customer traffic to the new fleet. It is a safe *rollout* strategy, not a way to evaluate without customer impact. - D is wrong because a batch transform job runs on separate batch infrastructure against replayed data. It says nothing about the latency and error rate of the new container behind a real-time endpoint under live, concurrent traffic, which is what the team needs to measure.
Reference: Shadow tests