fix(evolve): GEPA never executed and evolved output always failed validation - #177
Open
acidsound wants to merge 1 commit into
Open
fix(evolve): GEPA never executed and evolved output always failed validation#177acidsound wants to merge 1 commit into
acidsound wants to merge 1 commit into
Conversation
1. Validation ran against the body-only artifact, which by construction lacks YAML frontmatter — so skill_structure failed on every evolved variant and no improvement could ever be deployed. Validate the reassembled file (frontmatter + evolved body) instead. 2. dspy.GEPA was called with max_steps=..., a kwarg from an older standalone gepa API. Current DSPy raises TypeError and the code fell back to MIPROv2 on every run — GEPA never actually executed despite being the project's headline engine. Use max_metric_calls, adapt the metric to GEPA's required (gold, pred, trace, pred_name, pred_trace) signature, and pass the required reflection_lm. 3. The evolved text was read from optimized_module.skill_text, which GEPA never mutates — it optimizes the predictor's signature instruction. Every run reported 'improvements' while writing an unchanged skill file (verified: baseline/evolved md5 identical). Extract via named_predictors() to get the actual evolved instruction. Verified locally: hermes-agent skill, 3 iterations → holdout 0.599→0.628, size 12,598→3,027 chars, constraints PASS, real diff present. Full test suite (145 tests) passes.
19 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three bugs in
evolve_skill.pymeant the Phase 1 pipeline could never produce a valid evolved skill:1. Validation ran against a body-only artifact
The evolved text (
evolved_body) is the skill body with frontmatter stripped, butskill_structurerequires YAML frontmatter (---,name:,description:). Every evolved variant failed this check by construction — no improvement could ever deploy.Fix: validate the reassembled file (
reassemble_skill(frontmatter, body)).2. GEPA never actually ran
Current DSPy's
GEPAhas nomax_steps; every run hit the except block and silently fell back to MIPROv2 — despite GEPA being the project's headline engine. Also:(gold, pred, trace, pred_name, pred_trace), not DSPy's 3-arg form → adapted via wrapperreflection_lm→ wired tooptimizer_model3. Evolved text was read from a field GEPA never mutates
optimized_module.skill_textis untouched by optimization; GEPA evolves the predictor's signature instruction. Verified on real runs: baseline/evolved files had identical md5 while reporting +0.006/+0.024 "improvements" (judge variance). Now extracted vianamed_predictors().Verification
Notes
computer-useSKILL.md (18.5KB) exceeds the 15KB size constraint at baseline — separate concern, not addressed here.