Model migration
Compare providers or model families
Keep cases, external schema, and deterministic correctness fixed while baseline and candidate outputs change.
Use cases
Change the system. Keep the contract. Measure the outcome.
StructTrace is designed for paired migrations where callers expect the same structured contract before and after the change.
Prompt change
Catch semantic drift behind valid JSON
Detect cases where format compliance stays high but extracted values or classifications regress.
Decoder change
Measure constrained-output sensitivity
Compare decoding implementations without assuming that more schema-valid generations are more correct.
Document extraction
Verify financial and identity fields
Use exact decimals, canonical dates, keyed line items, normalized strings, and mapped arithmetic invariants.
Tool calls
Check selection and argument semantics
Score the selected tool, exact argument paths, and deterministic postconditions in retained outputs.
Regression suite
Pin failures across iterations
Track passing, fixed, still-broken, reintroduced, and missing cases before accepting a candidate as the next baseline.
Not the right tool for every problem.
Clear boundaries make the evidence easier to trust.
| Good fit | Not a current promise |
|---|---|
| Stable external contract and paired cases | Schema migration or contract generation |
| Deterministic correctness rules | A default LLM judge for subjective quality |
| Local recorded outputs | Production tracing or multi-tenant observability |
| Reviewable release evidence | A universal schema or prompt optimizer |