Audit multi-round prompt contamination from degenerate parser outputs

[armaanvgrewal]

Summary

This issue investigates multi-round experiments where a degenerate parser output may have been incorporated into a later-round prompt.

Unlike single-round experiments, these cases cannot necessarily be corrected by rescoring because the downstream prompt itself may have been altered.

Motivation

A declaration-aware rescoring pass can correct measurement artifacts when the final metric depends only on stored outputs.

However, if a malformed parse was injected into a subsequent prompt, then:

round N output
    ↓
degenerate parse
    ↓
later-round prompt
    ↓
later generations

the experiment state may already have diverged.

In those situations, rescoring alone cannot reconstruct what would have happened had the degenerate parse not occurred.

Goals

Identify:

• multi-round experiment families,
• cases where a degenerate parse propagated into a later prompt,
• whether contamination affected only measurement or altered the generated conversation itself.

Deliverables

For each affected experiment:

1. Count potentially contaminated cases.
2. Classify:
   • measurement-only issue,
   • prompt-contamination issue.
3. Determine whether:
   • rescoring is sufficient,
   • rerunning is required,
   • results should be excluded.
4. Document impact on any published or committed metrics.

Constraints

• Preserve existing historical artifacts.
• Prefer analysis from committed artifacts and caches where possible.
• Clearly separate confirmed contamination from theoretical edge cases.

Non-goals

• Re-running experiments as part of the initial audit.
• Changing historical result files during analysis.

Related

Follow-up from review of PR #418.