Batch recovery and accuracy measurement

Handling partial failure, and the statistics trap before you automate.

4 min read · Lesson 7 of 12 in this domain

Two failure modes bracket a batch pipeline. On the way out, partial failure: some requests error, and the instinct to resubmit the whole batch throws away everything that worked. custom_id exists precisely so you can identify and resubmit only the failures. On the way in, measurement: an aggregate accuracy figure is an average across subgroups, and averages hide collapse. A system that is 97% accurate overall can be 99% on clean typed invoices and 61% on handwritten forms, and if you automate on the headline number you have quietly automated the 61%.

• When a subset fails, isolate those items by custom_id, fix the cause (e.g. chunk oversized inputs), and resubmit only the failures as a new batch. Reprocessing successes wastes almost the entire spend.
• Switching failures to the synchronous API costs roughly double per token and is unnecessary for a non-interactive retry.
• Aggregate accuracy hides subgroup collapse. "97% overall at ≥90% confidence" can conceal standard invoices at 99% and handwritten forms at 61%.
• Before automating, segment accuracy by document type and by field. That is what reveals whether automation is safe.
• A uniform random sample review under-represents rare document types — precisely where failures concentrate.
• Raising the confidence threshold does not fix the aggregation illusion; the weak subgroup may still clear the higher bar.

Source: Claude Docs — Batch processing · Independent study aid, not affiliated with or endorsed by Anthropic.