Skip to content

fix(evolve): GEPA never executed and evolved output always failed validation - #177

Open
acidsound wants to merge 1 commit into
NousResearch:mainfrom
acidsound:fix/evolved-skill-validation-and-gepa
Open

fix(evolve): GEPA never executed and evolved output always failed validation#177
acidsound wants to merge 1 commit into
NousResearch:mainfrom
acidsound:fix/evolved-skill-validation-and-gepa

Conversation

@acidsound

Copy link
Copy Markdown

Summary

Three bugs in evolve_skill.py meant the Phase 1 pipeline could never produce a valid evolved skill:

1. Validation ran against a body-only artifact

The evolved text (evolved_body) is the skill body with frontmatter stripped, but skill_structure requires YAML frontmatter (---, name:, description:). Every evolved variant failed this check by construction — no improvement could ever deploy.

Fix: validate the reassembled file (reassemble_skill(frontmatter, body)).

2. GEPA never actually ran

dspy.GEPA(metric=..., max_steps=iterations)  # TypeError: unexpected kwarg

Current DSPy's GEPA has no max_steps; every run hit the except block and silently fell back to MIPROv2 — despite GEPA being the project's headline engine. Also:

  • GEPA's metric signature is (gold, pred, trace, pred_name, pred_trace), not DSPy's 3-arg form → adapted via wrapper
  • GEPA raises without a reflection_lm → wired to optimizer_model

3. Evolved text was read from a field GEPA never mutates

optimized_module.skill_text is untouched by optimization; GEPA evolves the predictor's signature instruction. Verified on real runs: baseline/evolved files had identical md5 while reporting +0.006/+0.024 "improvements" (judge variance). Now extracted via named_predictors().

Verification

  • Full test suite: 145 passed
  • Real run (hermes-agent skill, 3 iterations): holdout 0.599 → 0.628, size 12,598 → 3,027 chars, constraints PASS, non-empty diff present
  • Before fix: same run reported fake improvements with md5-identical output

Notes

  • The MIPROv2 fallback remains as a safety net but no longer triggers.
  • Related: computer-use SKILL.md (18.5KB) exceeds the 15KB size constraint at baseline — separate concern, not addressed here.

1. Validation ran against the body-only artifact, which by construction
   lacks YAML frontmatter — so skill_structure failed on every evolved
   variant and no improvement could ever be deployed. Validate the
   reassembled file (frontmatter + evolved body) instead.

2. dspy.GEPA was called with max_steps=..., a kwarg from an older
   standalone gepa API. Current DSPy raises TypeError and the code fell
   back to MIPROv2 on every run — GEPA never actually executed despite
   being the project's headline engine. Use max_metric_calls, adapt the
   metric to GEPA's required (gold, pred, trace, pred_name, pred_trace)
   signature, and pass the required reflection_lm.

3. The evolved text was read from optimized_module.skill_text, which
   GEPA never mutates — it optimizes the predictor's signature
   instruction. Every run reported 'improvements' while writing an
   unchanged skill file (verified: baseline/evolved md5 identical).
   Extract via named_predictors() to get the actual evolved instruction.

Verified locally: hermes-agent skill, 3 iterations → holdout 0.599→0.628,
size 12,598→3,027 chars, constraints PASS, real diff present. Full test
suite (145 tests) passes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant