feat: single-call Likert scoring for deduplicate_levers

[neoneye]

Summary

• Replaces 18 sequential per-lever LLM calls with a single batch call
• Each lever scored on a 5-point Likert relevance scale: how relevant is this lever to this specific plan?
  • 2 = highly relevant, 1 = somewhat relevant, 0 = borderline, -1 = low relevance, -2 = irrelevant
• Levers scoring >= 1 kept; levers scoring <= 0 removed
• Supersedes PR feat: simplify lever classification to primary/secondary/remove #372

Why single-call

The per-lever sequential approach (18 calls) has structural problems:

• Position bias: lever 1 classified with no prior context; lever 18 with 17 prior decisions
• No global view: model can't compare all levers simultaneously
• 18x cost: growing context across 18 calls is expensive and slow

Why relevance scoring instead of category classification

Previous iterations (45, 48, 49) shuffled taxonomy labels (keep/absorb/remove → primary/secondary/remove) without significant improvement. The labels mapped 1:1 to each other — just different names for the same categories.

This PR reframes the task entirely: instead of "classify this lever into a bucket", the model answers "how relevant is this lever to this specific plan?" This is a question the model can answer by reading the project context. Duplicates naturally score lower because a redundant lever is less relevant when a better one already exists — no explicit "absorb" concept needed.

The integer score also eliminates template-lock risk (no text definition to copy verbatim).

Changes

• deduplicate_levers.py: complete rewrite — BatchDeduplicationResult schema, single llm.as_structured_llm() call, Likert relevance prompt, OPTIMIZE_INSTRUCTIONS with pipeline context
• enrich_potential_levers.py: accepts optional classification field (backward-compatible)
• runner.py: calls_succeeded=1 for single batch call
• B1 fix: user_prompt stores project_context, not serialized levers

Test plan

• Run deduplicate_levers step via self-improve runner against snapshot input
• Verify all 7 models produce valid scores for all 18 levers
• Compare score distributions and kept counts against iter 49 (PR feat: simplify lever classification to primary/secondary/remove #372 baseline)
• Check that models produce a spread of scores (not all 1s and 2s)

🤖 Generated with Claude Code

[neoneye]

Iteration 50 Results

Verdict: REVERT (NO)

What works

• Batch architecture proven: 35/35 runs succeeded, all 7 models parse BatchDeduplicationResult in one call
• Speed: 1.5–6.3× faster (silo gpt-oss-20b: 196s → 51s, parasomnia haiku: ~120s → 24s)

What's broken

• Relevance ≠ deduplication: The prompt asks "how relevant is this lever?" but a lever can be highly relevant AND redundant with another. Capable models (gpt-oss-20b, qwen3, haiku, gemini, 4o-mini) score overlapping levers at 1–2 and keep all 18. gpt-oss-20b silo: 2 removals before → 0 after.
• llama3.1 Likert scale inversion: On silo and gta_game, llama3.1 scores 17/18 levers as -2 while justifications say "highly relevant." Only 1 lever survives. Categorical labels can't be inverted; integers can.

Takeaway

The single-call architecture is the real win — keep it. The Likert scoring is the problem — revert to categorical primary/secondary/remove taxonomy (PR #372's schema) inside the batch call.

Full analysis: analysis/50_deduplicate_levers/