Prompt-only
39/49 recoverably correct
The reasoning-first prompt produced valid JSON objects, but zero answers satisfied the stricter numeric-string schema as written.
Research foundation
The failure was not invalid JSON. The failure was valid JSON carrying the wrong answer.
StructTrace grew out of paired experiments on contract sensitivity. The result was not a universal optimization. It was a measurement requirement.
The evidence chain.
Each row is a separate paired study. Effects are not pooled across models or tasks.
| Study | Paired cases | Control | Treatment | Treatment-only | Control-only | Effect and interval |
|---|---|---|---|---|---|---|
| Corrected Qwen representation | 49 | 18/49 | 24/49 | 9 | 3 | +12.2 pp [0.0, 26.5] |
| Canonical Llama replication | 150 | 92/150 | 82/150 | 6 | 16 | -6.7 pp [-12.7, -0.7] |
| Executable tool-call pilot | 30 | 26/30 | 24/30 | 1 | 3 | -6.7 pp [-20.0, 6.7] |
What happened before the replication.
The original 49-item Qwen comparison exposed the validity-versus-correctness gap that shaped the product.
Outlines and XGrammar
30/49 correct, 49/49 schema-valid
Both constrained backends reached complete structural validity while losing nine recoverably correct answers on the paired set.
Observed gap
-18.4 percentage points
The same bounded setup showed that better compliance did not imply better task performance.
Source: canonical machine-readable initial summary. One Qwen model, one cleaned GSM8K subset, one prompt family.
Why this became StructTrace.
The honest conclusion was to measure migrations on the workload that will actually ship.
FINDING / VALIDITYStructure can hide error.
Product response: report parsing, schema validity, semantic correctness, and valid-but-wrong separately.
FINDING / SENSITIVITYRepresentation changes answers.
Product response: use paired case IDs and show every repair and regression instead of only marginal averages.
FINDING / REVERSALOne model did not generalize.
Product response: no universal optimizer claim. Teams evaluate their own model, prompt, contract, and cases.
Provenance is part of the product.
The StructTrace repository pins the research source revision and SHA-256 digest of every accepted summary used to create its normalized demonstration fixtures.