How to Run Evaluation Experiments

Last verified: 2026-07-26 against dev at f0388abb.

Use this guide to run Phoenix/Arize experiments for extraction and checklist quality.

Inputs

• Arize account access.
• Client-specific Arize API key and Space ID.
• Document URI and ground-truth URI for extraction experiments.
• Checklist ground truth when running checklist/report experiments.
• Local env hydrated through make env-phoenix.

Extraction Experiment

1. Hydrate Phoenix env.

make env-phoenix

1. Add client-specific Arize credentials to phoenix/.env.local.

2. Set document and ground-truth URIs.

3. Register the dataset.

make register-arize-dataset

1. Run the experiment.

make run-arize-exp

Or run direct extractor mode:

make run-arize-exp-direct

Checklist Experiment

Follow the checklist section in phoenix/README.md. The experiment scores checklist status accuracy, coverage, and comment quality against hand-authored ground truth.

Verification

• Dataset appears in Arize.
• Experiment run completes.
• Evaluator scores are populated.
• Traces do not include secrets or long-lived keys.

Related

• Evaluation model
• Phoenix README