StructTrace

Documentation

Your first comparison

Begin with retained outputs. This isolates evaluation semantics from provider behavior and gives the most reproducible first run.

1. Prepare four inputs

  • a JSONL, JSON, or CSV dataset with case identity, input, and expected values;
  • baseline outputs joined by the same case identity;
  • candidate outputs joined by the same case identity;
  • the unchanged caller-facing JSON Schema.

2. Create the project visually

structtrace open

Select Compare a change, add the four files, confirm detected envelope mappings, and choose deterministic correctness rules. StructTrace stages browser-selected sources under the local project, binds their BLAKE3 digests, and generates the reproducible project configuration.

3. Define correctness

Use exact JSON when every detail matters. For extraction, define rules per field: normalized strings, canonical dates, exact integers, exact or tolerant decimals, keyed arrays, and mapped financial invariants. Suggestions do not silently become truth.

4. Choose evidence authority

  • Advisory reports evidence and never authorizes deployment.
  • Regression checks configured relative non-regression.
  • Release adds absolute quality and evidence floors and is the only authorizing mode.

5. Inspect paired results

Begin with the gate state and its reasons. Then review captured rows, independent units, fully evaluated semantic pairs, regressions, improvements, both-pass, both-fail, effect interval, field hotspots, and every discordant case.

6. Replay before relying on it

structtrace --project-root my-comparison replay latest
structtrace --project-root my-comparison gate latest --verify replay

Replay checks manifest hashes, reconstructs built-in evaluator results, verifies retained external-evaluator receipts, and recomputes summaries without re-executing user code.

7. Generate CI after the local run is understood

The UI exports a complete project and a reviewable GitHub Actions starter. Add project-specific installation and data-generation steps before treating it as runnable automation.