dspy 3.3 compatibility + extract bugs: evolution never writes back real skill changes

[benzboxmail-lgtm]

dspy 3.3 compatibility + extract bugs: skill evolution never writes back real changes

Summary

Running evolve_skill.py against dspy 3.3.0 (current dspy>=3.0.0 pin) fails or produces empty diffs due to four issues:

1. Validator checks the wrong text (always fails skill_structure).
   evolve_skill.py calls validator.validate_all(evolved_body, ...) but evolved_body has no frontmatter (it is split off by load_skill). The validator requires --- + name: + description:, so every evolution run fails constraints with "Skill missing: YAML frontmatter" even though the saved evolved_full file is perfectly valid.
   Fix: validate evolved_full (or skill["raw"] as baseline).

2. GEPA API drift (dspy 3.3).

   • dspy.GEPA(metric=..., max_steps=iterations) → max_steps no longer exists; renamed to max_full_evals.
   • GEPA now requires reflection_lm (AssertionError otherwise).
   • GEPA metric signature changed to 5 args (gold, pred, trace, pred_name, pred_trace); dspy calls the metric with 2 args in some paths (evaluate calls metric(example, prediction)) and 5 in others — the wrapper needs defaults for all optional args.
     Net effect: without fixes, every run silently falls back to MIPROv2 and never actually evolves the skill text.

3. optimized_module.skill_text never changes → evolved file == baseline.
   GEPA mutates the signature instructions, not the module attribute. With SkillModule built as dspy.ChainOfThought(self.TaskWithSkill) (fixed docstring), the evolved instructions are discarded:

   • SkillModule.__init__ must seed the signature: dspy.ChainOfThought(self.TaskWithSkill.with_instructions(skill_text))
   • Extraction must read optimized_module.predictor.predict.signature.instructions (fallback chain for other module shapes).
     Without this, evolved_skill.md is byte-identical to baseline and the reported "+X%" is pure holdout noise.

4. MIPROv2 fallback crashes without optuna (dspy[optuna] not in [project.optional-dependencies]). Installing optuna fixes it.

Extra note

dspy.LM with litellm marks DeepSeek as supporting response_format json_schema, but the DeepSeek API rejects it ("This response_format type is unavailable now"). Workaround used locally: type(lm).supports_response_schema = property(lambda self: False) (forces {"type": "json_object"}). Worth documenting in README for non-OpenAI providers.

Patch

Full diff attached (3 files, +24/-5): evolution/core/dataset_builder.py, evolution/skills/evolve_skill.py, evolution/skills/skill_module.py.

Environment

• dspy 3.3.0, Python 3.11, litellm via dspy
• Verified end-to-end: with fixes, GEPA 10 iterations produces a real evolved SKILL.md (e.g. +5.4% holdout, -3.6% size on arxiv)

Patch (git diff, 3 files)

diff --git a/evolution/core/dataset_builder.py b/evolution/core/dataset_builder.py
index 3a430ce..ca80a24 100644
--- a/evolution/core/dataset_builder.py
+++ b/evolution/core/dataset_builder.py
@@ -124,6 +124,7 @@ class SyntheticDatasetBuilder:
 
         # Configure DSPy to use the judge model for generation
         lm = dspy.LM(self.config.judge_model)
+        type(lm).supports_response_schema = property(lambda self: False)  # DeepSeek et al.: no json_schema support
 
         with dspy.context(lm=lm):
             result = self.generator(
diff --git a/evolution/skills/evolve_skill.py b/evolution/skills/evolve_skill.py
index 2a79a67..9b7dcee 100644
--- a/evolution/skills/evolve_skill.py
+++ b/evolution/skills/evolve_skill.py
@@ -138,6 +138,7 @@ def evolve(
 
     # Configure DSPy
     lm = dspy.LM(eval_model)
+    type(lm).supports_response_schema = property(lambda self: False)  # DeepSeek et al.: no json_schema support
     dspy.configure(lm=lm)
 
     # Create the baseline skill module
@@ -153,9 +154,14 @@ def evolve(
     start_time = time.time()
 
     try:
+        def gepa_metric(gold, pred, trace=None, pred_name=None, pred_trace=None):
+            """Wrap the 2-arg fitness metric for dspy 3.3 GEPA's 5-arg signature."""
+            return skill_fitness_metric(gold, pred)
+
         optimizer = dspy.GEPA(
-            metric=skill_fitness_metric,
-            max_steps=iterations,
+            metric=gepa_metric,
+            max_full_evals=iterations,  # dspy 3.3: max_steps renamed to max_full_evals
+            reflection_lm=lm,  # dspy 3.3: reflection LM is required
         )
 
         optimized_module = optimizer.compile(
@@ -180,12 +186,19 @@ def evolve(
 
     # ── 6. Extract evolved skill text ───────────────────────────────────
     # The optimized module's instructions contain the evolved skill text
-    evolved_body = optimized_module.skill_text
+    # (dspy 3.3 GEPA mutates signature instructions — read them back)
+    try:
+        evolved_body = optimized_module.predictor.predict.signature.instructions
+    except AttributeError:
+        try:
+            evolved_body = optimized_module.predict.signature.instructions
+        except AttributeError:
+            evolved_body = getattr(optimized_module, "skill_text", skill["body"])
     evolved_full = reassemble_skill(skill["frontmatter"], evolved_body)
 
     # ── 7. Validate evolved skill ───────────────────────────────────────
     console.print(f"\n[bold]Validating evolved skill[/bold]")
-    evolved_constraints = validator.validate_all(evolved_body, "skill", baseline_text=skill["body"])
+    evolved_constraints = validator.validate_all(evolved_full, "skill", baseline_text=skill["raw"])
     all_pass = True
     for c in evolved_constraints:
         icon = "✓" if c.passed else "✗"
diff --git a/evolution/skills/skill_module.py b/evolution/skills/skill_module.py
index 6d4d22e..908e574 100644
--- a/evolution/skills/skill_module.py
+++ b/evolution/skills/skill_module.py
@@ -104,7 +104,12 @@ class SkillModule(dspy.Module):
     def __init__(self, skill_text: str):
         super().__init__()
         self.skill_text = skill_text
-        self.predictor = dspy.ChainOfThought(self.TaskWithSkill)
+        # dspy 3.3: seed the signature instructions with the skill body so GEPA
+        # mutates the actual skill text, then read it back from
+        # module.predict.signature.instructions after compilation.
+        self.predictor = dspy.ChainOfThought(
+            self.TaskWithSkill.with_instructions(skill_text)
+        )
 
     def forward(self, task_input: str) -> dspy.Prediction:
         result = self.predictor(