AI-300 Practice Question 60: Streaming Perceived Latency

Microsoft Machine Learning Operations Engineer Associate (AI-300) · Test 6 · question 10 of 10

Optimize generative AI systems and model performanceeasy

A chat interface waits for the full model response before showing anything. Long answers take about 12 seconds, and users feel the app has frozen. The answers cannot be shortened. Which change most improves how responsive the app feels?

Select one answer
Show answer and explanation

Answer:

  • B. Enable streaming so that tokens are displayed as they are generated

Total generation time depends mostly on the number of output tokens. With streaming, the service sends tokens as they are produced, so the first words appear after the time to first token and users can start reading while the rest is generated (option B).

- A is wrong because a higher limit allows even longer responses, which can only increase the time until the full answer is complete. - C is wrong because batch processing is asynchronous with a target turnaround of up to 24 hours. It is the opposite of interactive. - D is wrong because temperature changes how varied the output is. It does not change how quickly the first token arrives.

Reference: Improve latency in Foundry Models

Practice Test 6 ›

Study guides for AI-300