AI-300 Practice Question 9: Continuous Evaluation Production
Microsoft Machine Learning Operations Engineer Associate (AI-300) · Test 1 · question 9 of 10
A support agent built in Microsoft Foundry passed its pre-production evaluation against a curated test dataset and is now live. Leadership wants quality and safety scores computed on the traffic real users actually send, shown next to latency and token consumption, and they do not want to pay to evaluate every single request. Which capability should you enable?
Show answer and explanation
Answer:
Microsoft lists continuous evaluation as the post-production capability for *quality and safety evaluation of production traffic at a sampled rate*. That matches all three constraints: real user traffic rather than a curated set, quality and safety metrics, and a sampling rate so you aren't scoring every request. Results land in the Foundry observability dashboard alongside the operational metrics (latency, token consumption, error rates) that Foundry collects through its Azure Monitor Application Insights integration (option B).
- A is wrong because red teaming is adversarial probing with simulated attacks. It hunts for vulnerabilities; it doesn't score what real users are receiving. - C is wrong because scheduled evaluation re-runs a test dataset. It's the right tool for detecting drift against a fixed benchmark, but it never touches production traffic. - D is wrong because tracing captures the execution flow: spans for LLM calls, tool calls, and agent decisions. It is essential for debugging and it feeds evaluation, but a span carries no groundedness or safety score by itself.
Reference: Observability in Generative AI - Microsoft Foundry