AI-300 Practice Question 60: Streaming Perceived Latency
Microsoft Machine Learning Operations Engineer Associate (AI-300) · Test 6 · question 10 of 10
A chat interface waits for the full model response before showing anything. Long answers take about 12 seconds, and users feel the app has frozen. The answers cannot be shortened. Which change most improves how responsive the app feels?
Show answer and explanation
Answer:
Total generation time depends mostly on the number of output tokens. With streaming, the service sends tokens as they are produced, so the first words appear after the time to first token and users can start reading while the rest is generated (option B).
- A is wrong because a higher limit allows even longer responses, which can only increase the time until the full answer is complete. - C is wrong because batch processing is asynchronous with a target turnaround of up to 24 hours. It is the opposite of interactive. - D is wrong because temperature changes how varied the output is. It does not change how quickly the first token arrives.
Reference: Improve latency in Foundry Models