Evaluation Model¶
Last verified: 2026-07-26 against dev at f0388abb.
SDP uses Phoenix/Arize tooling to measure extraction and checklist quality.
Extraction Experiments¶
Extraction experiments use:
- Datasets of document test cases.
- A task that runs extraction and returns structured output.
- Evaluators that compare outputs to ground truth.
Current evaluator categories include type completeness, field completeness, and value correctness.
Checklist Experiments¶
Checklist/report experiments run an end-to-end pipeline for a document package and compare checklist statuses/comments against hand-authored ground truth. This measures the quality of the checklist response, not DOCX rendering.
Why This Matters¶
Extraction systems can regress without obvious API failures. Evaluation docs and experiments give the team a repeatable way to detect drift in:
- Field recall.
- Value correctness.
- Checklist status accuracy.
- Coverage.
- Narrative/comment quality.