Prompts are code without tests until you build the harness. Construct a small evaluation system that grades outputs, tracks accuracy per slice, and fails when a change regresses a slice even if the average improves.
What to build
- Build a labelled dataset of at least fifty cases, stratified across the categories you care about.
- Write a grader — exact match where possible, a rubric-based model grade only where it is not.
- Record a baseline per stratum, not just overall.
- Change the prompt and re-run.
- Flag any stratum that regressed even when the aggregate improved.
- Store results so you can diff across versions over time.
Done when
- Per-stratum accuracy is reported, not only the headline number.
- A change that helps overall but breaks one category is caught.
- The grader is deterministic wherever the task allows it.
- You can point at a real regression the harness caught.
The trap this exercise teaches
Reporting a single aggregate accuracy. It hides the case where a change lifts common inputs and destroys a minority category that matters.
Before you start
The theory behind this build is covered in Evaluating prompts: datasets, grading and regression. If any step below is unfamiliar, read that first — the exercise assumes it. Primary source: Claude Docs — Prompt engineering.
Check yourself against the exam
This exercise sits in Domain 3, which is 20% of the CCAR‑F exam. Once you have built it, run a domain drill in the study app and see whether the questions read differently.