Claude Certified Architect – Foundations Practice Question 39: Prompt Change Evals
Claude Certified Architect, Foundations · Test 4 · question 9 of 10
A support-summarization feature is in production. The team wants to rewrite the system prompt to improve tone, but has been burned by changes that fixed one case and broke others. The summaries are free-form text, so exact-match comparison is not possible. What is the best way to decide whether to ship the new prompt?
Show answer and explanation
Answer:
Anthropic's guidance on developing tests is to build task-specific evals from realistic inputs including edge cases, and for open-ended output to use LLM-based grading with clear rubrics. Running both versions on the same set shows regressions before users see them, and checking the judge against human grades keeps it trustworthy.
A is too small and subjective. B tests on customers and is slow and noisy. D gives the model no evidence, so its answer is not an evaluation.