Claude Certified Architect – Foundations Practice Question 39: Prompt Change Evals

Claude Certified Architect, Foundations · Test 4 · question 9 of 10

Context Management & Reliabilitymedium

A support-summarization feature is in production. The team wants to rewrite the system prompt to improve tone, but has been burned by changes that fixed one case and broke others. The summaries are free-form text, so exact-match comparison is not possible. What is the best way to decide whether to ship the new prompt?

Select one answer
Show answer and explanation

Answer:

  • C. Run old and new prompts over a fixed golden set of representative and edge-case tickets, and grade the outputs with an LLM judge using a detailed rubric (validated against human ratings), then compare scores.

Anthropic's guidance on developing tests is to build task-specific evals from realistic inputs including edge cases, and for open-ended output to use LLM-based grading with clear rubrics. Running both versions on the same set shows regressions before users see them, and checking the judge against human grades keeps it trustworthy.

A is too small and subjective. B tests on customers and is slow and noisy. D gives the model no evidence, so its answer is not an evaluation.

Reference: Define success criteria and build evaluations

Practice Test 4 ›

Study guides for Claude Certified Architect, Foundations