AI-300 Practice Question 36: Endpoint Based Custom Evaluator
Microsoft Machine Learning Operations Engineer Associate (AI-300) · Test 4 · question 6 of 10
A support assistant must reference product codes in its answers. The quality team wants a batch evaluation metric that scores 1.0 only when every product code in a response exists in the company's inventory. The inventory can only be queried through an internal HTTPS API. The team already hosts a small scoring service that can reach that API and that authenticates callers with Microsoft Entra ID. The check is deterministic, and they do not want an LLM to guess. Using the Foundry SDK, which type of custom evaluator should they register?
Show answer and explanation
Answer:
Foundry supports three custom evaluator types. An endpoint-based evaluator *"delegates scoring to an external HTTP endpoint that you own and operate"* and is for cases that need *"network access to external services or databases during scoring."* The evaluation service sends the mapped input data to the endpoint, which returns a JSON result with score, reason and status. A project connection stores the endpoint URL and credentials, and Microsoft Entra ID is a supported authentication method (option D).
- A is wrong because code-based evaluators run in a sandbox where *"no network access is available at runtime,"* so the grade() function cannot call the inventory service. - B is wrong because a prompt-based evaluator uses an LLM as a judge. The team wants a deterministic check, and a static list in a prompt would not reflect the live inventory. - C is wrong because the built-in groundedness evaluator checks a response against context that is supplied with each item. It does not query a live system to verify that codes exist.
Reference: Custom Evaluators - Microsoft Foundry