This is the single most-tested idea in the domain. A schema guarantees shape. It does not guarantee that the total equals the sum of the lines, that the date is real, or that the extracted number came from the document. Build the proof, then build the layer that actually catches it.
What to build
- Take your extraction from the previous exercise.
- Feed it documents with subtle problems: a total that does not sum, a February 30th, a currency mismatch.
- Confirm the output validates against the schema every time.
- Now write a semantic validation layer in code: arithmetic checks, date realism, cross-field consistency.
- On failure, return the specific violation to the model and retry.
- Cap retries and route persistent failures to human review.
Done when
- You have at least three documents that pass schema validation and fail semantic validation.
- The retry message names the violated constraint rather than saying "invalid".
- Retries are bounded and exhaustion has a defined destination.
- The validator is deterministic code, not a second model call.
If you want to go further
- Log which semantic rule fires most and use it to improve the prompt rather than the retry loop.
The trap this exercise teaches
Answering "how do we guarantee correct extraction?" with "use a strict schema". Schema guarantees parse, not truth.
Before you start
The theory behind this build is covered in Syntax errors vs semantic errors. If any step below is unfamiliar, read that first — the exercise assumes it. Primary source: Claude Docs — Structured outputs.
Check yourself against the exam
This exercise sits in Domain 3, which is 20% of the CCAR‑F exam. Once you have built it, run a domain drill in the study app and see whether the questions read differently.