AI-300 Practice Question 66: APIM Token Limit Policy
Microsoft Machine Learning Operations Engineer Associate (AI-300) · Test 7 · question 6 of 10
Several internal apps call one model deployment in Microsoft Foundry through Azure API Management. One app sometimes consumes most of the deployment's tokens, and the other apps then receive 429 errors. The team wants a limit for each app, enforced at the gateway before requests reach the model, and based on tokens rather than the number of calls. What should they use?
Show answer and explanation
Answer:
API Management provides the llm-token-limit policy for language model APIs. It counts prompt and completion tokens per key, such as a subscription or an app identity, and returns 429 when that key exceeds its tokens-per-minute limit. With prompt token estimation on, an over-limit request is rejected before it reaches the backend. This isolates each app on a token basis (option D).
- A is wrong because this counts calls, so a few very large prompts would still consume most of the tokens. The requirement is a token-based limit. - B is wrong because the value is chosen by the client and applies to a single response. It does not enforce a per-minute budget across requests. - C is wrong because more quota raises the shared limit, but one app can still consume most of it and starve the others.
Reference: Limit large language model API token usage