AI-103 Practice Question 97: Batch API Bulk Evaluation
Microsoft Azure AI Apps and Agents Developer Associate (AI-103) · Test 10 · question 7 of 10
An applied-AI team wants to compare two prompt variants by running each against 50,000 questions from a fixed test set (100,000 requests in total) one time only, to decide which variant to ship. The comparison is not urgent, results by the next morning are fine, and the team wants the lowest possible per-token cost without reserving any ongoing capacity, since the job runs once and will not repeat on a schedule.
What should they use to run the evaluation?
Show answer and explanation
Answer:
Batch processing is defined for exactly this shape of work: "bulk processing workloads without latency requirements. Results are returned asynchronously," at a discounted per-token rate. The mechanics match the scenario directly; each line of a JSONL file is one request with a custom_id, method, url, and body; you upload the file, create a batch job referencing it, and poll until it completes before retrieving the output file. A one-time job that only needs results by morning is precisely what this asynchronous, discounted path is for.
- A is wrong because prompt caching only discounts input tokens that repeat within whatever serving path is already chosen; it doesn't provide a batch-rate discount and does nothing to satisfy "no reserved capacity" for a one-off job. - B is wrong because PTUs bill hourly for capacity reserved continuously, whether or not it's used, the opposite of what a job that runs once, with no recurring schedule, needs. - D is wrong because fine-tuning is a training investment sized for a workload that will run repeatedly; standing one up to answer a one-time "which prompt variant is better" question adds cost and delay the scenario explicitly doesn't call for.
Reference: How to use global batch processing with Azure OpenAI