feat: full feature to agent self evolution - #182
Open
numandev1 wants to merge 29 commits into
Open
Conversation
…steps dspy.GEPA expects max_full_evals to control the number of evaluation cycles, not max_steps. The incorrect kwarg caused a TypeError, which was caught and silently fell back to MIPROv2 without notification to the user. This fix allows GEPA to engage as intended for skill text mutation and optimization.
dspy.GEPA requires an explicit reflection_lm to generate program mutations. Without it, DSPy silently falls back to MIPROv2, which only tunes the DSPy instruction wrapper and never mutates the skill body — defeating the purpose of skill evolution. With this fix, GEPA uses the optimizer_model (e.g., Claude Opus) to reflect on failed examples and generate coherent skill text mutations with higher temperature and adequate token budget.
dspy.GEPA passes pred_name and pred_trace to the metric function per the GEPAFeedbackMetric protocol. The original signature did not accept these parameters, causing a TypeError when GEPA tried to call the metric. Add optional pred_name and pred_trace parameters to match the protocol. These are accepted but not used in the basic heuristic metric.
GEPA uses optuna for hyperparameter search. The dependency was missing from pyproject.toml, causing ImportError on fresh installs. Add optuna>=3.0.0 to the main dependencies.
…e attr SkillModule passed skill text as a runtime InputField, but DSPy optimizers (GEPA, MIPROv2) only mutate signature instructions — making the entire optimization pipeline silently return the original baseline. Embed skill text as the signature's instructions parameter so optimizers can actually evolve it, and read back via predictor.predict.signature instead of the unchanged instance attribute.
…lone The constraint validator in evolution/skills/evolve_skill.py:189 was called with evolved_body (the markdown body slice with YAML frontmatter already stripped by load_skill()) instead of evolved_full (the reassembled text produced by reassemble_skill(frontmatter, body)). Result: every successful evolution run was rejected by the skill_structure constraint, the file was written to evolved_FAILED.md, and metrics.json was never produced. The self-evolution pipeline has been silently broken in this way since the constraint was added. The fix passes the reassembled text to validate_all, which is what the function would actually deploy. Adds two regression tests: - test_evolved_full_reassembled_text_passes_skill_structure: pins the contract that the call site depends on (reassembled text passes). - test_evolved_body_alone_fails_skill_structure: confirms the root cause (body alone cannot pass), so a future refactor that reintroduces validate_all(evolved_body, ...) fails loudly. Refs NousResearch#74 (and NousResearch#11, NousResearch#93) — the same bug filed by three other people before this fix.
…re the LLM-scoring cap filter_and_score capped heuristic candidates at max_examples*3 WITHOUT ranking them, so the LLM relevance judge spent its whole budget on the chronologically-first N messages while stronger matches beyond the cap were never scored. Observed on a real session store: 321 heuristic hits -> capped to the first 150 -> only 5 survived LLM scoring -> a 2-example holdout whose verdicts are pure noise. _is_relevant_to_skill is now a thin boolean wrapper over a new _relevance_score (int strength: +10 exact skill-name phrase, +3 per skill-name word >3 chars, +N keyword overlap past the existing 2-keyword floor), and filter_and_score sorts candidates by that score descending before applying the cap. Boolean semantics (score > 0) are unchanged, so existing callers/tests are unaffected. Attributed-to: Diego + Hermes
Two unrelated stock-install bugs that together prevent the optimization loop from completing on a fresh clone: 1. **Missing optuna dependency.** `evolve_skill.py` falls back to MIPROv2 automatically when GEPA fails to initialize (which it currently always does on DSPy >=3.0 — see NousResearch#14, NousResearch#35, NousResearch#39). MIPROv2 imports `optuna` at `_optimize_prompt_parameters` time, so the fallback crashes with `ModuleNotFoundError: No module named 'optuna'` immediately after Step 2 finishes proposing instruction candidates. Switching the declared dependency from `dspy>=3.0.0` to `dspy[optuna]>=3.0.0` lets DSPy itself manage the version pin. 2. **No request timeout on LLM calls.** litellm's default request_timeout is unset, so any silent connection drop from the upstream provider (we hit this on a corporate/proxy gateway that drops long-lived POSTs without a TCP RST) hangs the optimization loop indefinitely with the python process holding an established but dead TCP socket at 0% CPU. We saw the entire 10-iteration loop block for 14+ minutes on a single hung call before manual intervention. Setting `litellm.request_timeout` at module import time gives every DSPy LM call a per-request deadline. Default 90s (generous for sonnet/opus reasoning tokens, short enough to detect a dead socket). Override via `LITELLM_REQUEST_TIMEOUT` env var. Verification: 143 tests pass (139 existing + 4 new tests covering the timeout default, env override, float parsing, and fail-fast on bad input). End-to-end run of `evolve_skill --skill llm-wiki-extract --eval-source synthetic --iterations 3` against a flaky upstream gateway now completes (before this fix it hung indefinitely).
- Add api_base and api_key parameters to EvolutionConfig - Update all DSPy LM initializations to use custom endpoints - Add --api-base and --api-key CLI options to external_importers and evolve_skill - Enable vLLM, Ollama, and other local model endpoints - Update documentation for local model configuration This allows the self-evolution tools to work with: - Local vLLM instances - Ollama endpoints - Any LiteLLM-compatible API - Cost savings on expensive cloud APIs
…uilt filters The local-model support added api_base/api_key as instance attributes set in RelevanceFilter.__init__. Tests construct the filter via __new__ to skip DSPy setup, so filter_and_score hit AttributeError. Declare both as class-level defaults so instances built without __init__ still resolve them.
load_skill() splits YAML frontmatter into skill["frontmatter"], so the baseline pass over skill["body"] could never satisfy skill_structure and printed a false 'baseline has constraint violations' warning on every run. Mirrors the evolved-side fix that validates the reassembled file.
dspy.GEPA() raised on every run, and the surrounding try/except caught it
and dropped to MIPROv2 — so runs reported "Running GEPA optimization" and
completed successfully while never using GEPA at all. Three causes:
1. `max_steps` is not a DSPy >=3.1 argument; the budget is now expressed as
`max_metric_calls` (or `auto`/`max_full_evals`).
2. GEPA requires an explicit `reflection_lm` and none was passed, so it
raised even once the budget argument was correct. `--optimizer-model` was
documented as "Model for GEPA reflections" but never used for anything —
it was printed to the console and then dropped. It now drives the
reflection LM, which is what the flag describes.
3. GEPA requires a five-argument metric returning a score *and* textual
feedback. `skill_fitness_metric` has a three-argument signature and
returns a bare float derived from keyword overlap, leaving GEPA with no
trace-level signal to reflect on.
For (3), route the metric through the existing LLMJudge, whose `feedback`
field is already documented as "Textual feedback for GEPA's reflective
analysis" but was not wired to the optimizer. The heuristic metric remains
as a fallback when a judge call fails, so a flaky judge degrades the score
rather than killing the run.
Verified end-to-end against a local model: GEPA now engages
("Running GEPA for approx 20 metric calls") instead of falling through to
MIPROv2. Existing test suite (145 tests) still passes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Drop the vendor-specific patch note from the cherry-picked validator fix and state the actual invariant: validate the file we would deploy, and compare growth full-file against full-file.
Extracts the Codex slice of upstream PR NousResearch#92 (stephenschoettler) — the hermes_codex helper, the CodexOAuthLM DSPy backend, and its tests — and skips that PR's cron/policy/golden-dataset changes. Any model string prefixed openai-codex/ now resolves through Hermes' own Codex OAuth store and CodexAuxiliaryClient instead of LiteLLM, so evolution can run on a Codex subscription. All four LM construction sites (optimizer, reflection, judge, importer) go through make_dspy_lm, which drops api_base/api_key for Codex models since those credentials come from Hermes. Verified against the local hermes-agent checkout: resolve_codex_runtime_ credentials and CodexAuxiliaryClient signatures match, and source discovery resolves ~/.hermes/hermes-agent automatically.
…me signals Replaces the parts of the evolution loop that never connected to Hermes. Objectives (evolution/core/objectives.py) Size pressure now derives from the candidate. It was computed from the baseline body, so the penalty evaluated to 0.000 for every variant, the search grew the artifact unopposed, and a post-hoc gate rejected the winner. Adds Pareto dominance/selection so GEPA's trade-off structure survives instead of collapsing to one scalar. Hermes paths (evolution/core/hermes_paths.py) Data-directory and profile discovery via HERMES_DATA_DIR/HERMES_HOME instead of Path.home(). Importers resolved to nonexistent paths inside the runtime container and silently mined zero examples. state.db reader (evolution/core/state_db.py) Reads the store Hermes actually keeps sessions in, with FTS5-backed relevance retrieval, tool histograms and system_prompt_hash grouping. Outcome signals (evolution/core/outcome_signals.py) Ground truth from verification_evidence.db exit codes and cron executions, rolled up per skill via jobs.json. Also: corpus-derived size budgets, skill bundles with reference-integrity checks, agent-in-the-loop evaluation harness, candidate-accurate judging, and a single metric shared by search and reporting.
…oyment
Session sourcing now reads state.db through HermesStateImporter and reports
each source individually. A source that is missing and a source that is
present-but-empty are different failures with different fixes; both used to
surface as the same empty dataset, which is how the weekly job stayed broken
without anyone being told.
Adds the deployment half of the product, which did not exist:
- evolution/deploy/pr.py branch, version-bumped commit, gh PR whose
body carries the constraint report and A/B
summary so a reviewer can judge a
machine-authored diff
- evolution/deploy/canary.py rollout tracked against system_prompt_hash
and cron outcomes, with auto-rollback
Adds reporting held to the readtool SUMMARY.md standard: noise bands from
repetitions, deltas inside the band reported as noise, explicit SHIP/HOLD,
and caveats emitted automatically for unbalanced arms and excluded errors.
Adds layered notification that always leaves a local record and never lets
delivery status decide the run's exit code.
…gates Rewrites the skill orchestrator so the thing optimized, the thing measured and the thing reported are the same thing. - make_fitness_metric drives both GEPA and the holdout comparison. They were different functions: GEPA optimized a judge composite while the reported delta came from keyword overlap, so the headline number was not the quantity being improved. - Size budget is derived from the installed corpus and floored at the skill's own size, replacing a constant that disqualified 27 of 201 shipped skills at their own baseline. - Skill lookup searches every tree (profiles, user, repo, optional-skills) instead of only <repo>/skills. - Bundles are loaded and gated: dropping a link to a supporting file, or inventing one, now fails validation. Nearly half the library ships reference files the optimizer previously could not see. - The pytest gate is actually invoked; --run-tests was accepted and ignored. - Evolved skills get a version bump, so they are distinguishable from what they replaced. - Verdicts are stated against a noise band and a run that cannot evaluate exits non-zero with a reason. Replaces the source-grep test on the orchestrator with behavioural tests, and isolates the suite from the developer's live ~/.hermes install.
Phase 2 — tool descriptions (evolution/tools/) Optimizes against tool-selection accuracy rather than a rubric: ground truth is which tool the agent actually reached for after each real user message, so scoring is exact and needs no judge. Targets a measured cost — tool_search and tool_describe account for 1,594 calls in one profile, turns spent finding tools instead of using them. Catalog size is an objective, since every description is paid for on every request. Phase 3 — system prompt sections (evolution/prompts/) Reads the prompt Hermes actually ran from sessions.system_prompt, ranked by how many sessions each variant carried, and optimizes one heading- delimited section at a time. Growth allowance is 5% rather than 20%: a skill loads when relevant, the system prompt is paid on every request. Splices by offset so an unchanged section round-trips byte-identical. Phase 5 — continuous loop (evolution/monitor/) Moves the rotation driver in-tree, where it is tested and reviewed with the code it drives. Replaces alphabetical round-robin with evidence-based priority: skills failing in production first, then never-evolved, then stalest; recently-passed and repeatedly-held skills sink. Round-robin at the deployed cadence implied a 13.4-month cycle over 117 skills, which the tool now states outright. Adds a time budget and reports what was deferred, so a bounded sweep never reads as full coverage. Phase 4 is left as a documented placeholder: darwinian_evolver is AGPL v3 and must stay an external CLI so its licence does not reach this MIT tree.
Adds regression tests for every defect the audit named, against fixtures that mirror the production schema (real sessions/messages DDL, a real FTS5 index, verification and cron stores) — a convenient invented schema would have let the original bug pass just as it passed in production. Two real bugs surfaced while writing them: - state_db conflated 'the FTS index cannot be joined' with 'the query matched nothing'. Both yield zero rows and need opposite responses, so an irrelevant skill would have silently mined the entire corpus. The join is now probed before the query runs. - skill_bundle compared reference paths as raw strings, so rewriting references/api.md to api.md read as deleting the file. Both sides now resolve to the supporting file they identify. Also: a SHIP verdict backed by one observation per arm now says its noise band was assumed rather than measured.
Test coverage was inverted relative to risk: 116 of 163 tests sat on the importer module that pointed at the wrong data store, while the 409-line orchestrator holding three of the audited defects had two, one of which asserted on the module's source text rather than its behaviour. Now 475 tests, with the weight on the code that can actually be wrong: objectives and fitness (the size and judging defects), state.db and paths (the data-layer defects), deployment and canary (new, previously absent), rotation priority, and both new phases. Test fixtures are hermetic: git fixtures neutralize developer global config (commit.gpgsign and core.hooksPath would otherwise fail commits for reasons unrelated to the code), and the orchestrator suite is isolated from a real ~/.hermes so results do not depend on whose machine runs them.
The cron driver is now a thin wrapper over evolution.monitor.run_rotation instead of ~120 lines of policy in bash outside the repository. It resolves the environment, takes a lock, and hands off; rotation policy, failure handling and notification live with the code they drive and are tested. Verified end to end: a sweep that cannot find a Hermes data directory now names the variable to set, writes a local status record, and exits 1. The previous driver reported success and told no one. Adds console entry points (evolve-skill, evolve-tool, evolve-prompt, evolve-rotation), an integration suite covering the full chain and every CLI, and rewrites README/PLAN to describe what the system does rather than what it was intended to do.
…ssions Verified against the live install, the first cut of FTS narrowing was not narrowing. An OR of a dozen terms matched 1,804 of 2,127 pairs for one skill, and three unrelated skills all came back within 1% of each other — the signature of a filter that is not filtering. Capping to the top-ranked sessions did not help, because sessions on a real install are long. Three fixes: - Rank with bm25 rather than taking anything that matched a term. bm25() is only legal in a query directly against the FTS table; SQLite flattens a ranking subquery back into the join and rejects it, and the previous except-clause was quietly turning that rejection into a full scan. - Filter at message level, not session level, so the cap bounds what it is meant to bound. The rest of each matched session is still read, since the answer and the tools used are what give the matched ask its context. - Cap the ranked pool near what the LLM relevance filter will actually score, so ranking chooses the candidates rather than arrival order. Live result: 1,804 pairs (85% of corpus) -> 100 (5%) for the SEO skill, 73 for an unrelated one, 33% overlap between them.
Adopted from upstream review of NousResearch#178 and NousResearch#179, which independently found the same core defects this fork already fixed but also caught data-quality problems it had not. Hermes writes machine text into the `user` role, and mining it verbatim turns agent plumbing into evaluation tasks. Four kinds found on the live install: - [CONTEXT COMPACTION - REFERENCE ONLY] summary blocks - [IMPORTANT: ...scheduled cron job...] delivery preambles wrapping the real instruction - [IMPORTANT: ...invoked the "x" skill] preambles followed by the entire SKILL.md - the worst case, since the optimizer would be trained on the artifact it is meant to improve, with the skill file as the request - [ASYNC DELEGATION COMPLETE] subagent notices Peeling is iterative, because these layer: a preamble wraps a skill file, so the checks have to re-run on whatever a strip exposes. Bracket matching is nesting-aware, because the cron preamble quotes "[SILENT]" inside itself and a scan for the first ] closed on that, leaving boilerplate behind as the task. Deliberately NOT adopting NousResearch#178's filter on the `compacted` flag. That flag means "rolled out of the active context window", not "machine-written": 900 of 1,290 user rows in one profile are compacted and nearly all are genuine asks ("audit this website"), so filtering on it discards most of the corpus while still admitting the injected blocks. Content-based detection does the job the flag was being used to approximate. Also adopted: - UnicodeDecodeError handling in the legacy JSON importer (NousResearch#179) - it is neither JSONDecodeError nor OSError, so one non-UTF8 file aborted the whole import - platform-aware install discovery: %LOCALAPPDATA%, XDG_DATA_HOME, macOS Application Support (NousResearch#178) - USERPROFILE alongside HOME in path tests; Path.expanduser() reads the former on Windows, so these failed there regardless of the code (NousResearch#178) Live result: 2,127 -> 1,989 mined pairs, with zero scaffolding remaining.
The three container failures were not environmental. collect_external_messages probed each importer's default path *before* calling it, which made a guess about where data lives authoritative over the importer itself. Two consequences: a source that could actually produce messages was skipped whenever the guess was wrong, and any test substituting an importer was skipped with it. That is why the suite read 504/0 on a laptop with ~/.claude/history.jsonl and 501/3 in the container without it. Reproduced locally by emptying HOME. Extraction now runs first and the path is consulted only to explain an empty result, so 'the source is not here' and 'it is here and idle' stay distinguishable without the probe being able to veto real data. An importer that raises is reported rather than ending the run. 508 tests, identical under both host conditions.
chore: merged useful prs
Implements Phase 4, and corrects the plan it was written against.
The plan said darwinian_evolver would be driven as an external CLI so its
AGPL-3.0 licence never touched this MIT tree. It cannot be: problems/
registry.py is a hardcoded dict and __main__.py restricts --problem to its
keys. There is no plugin path or entry point, so defining a Hermes problem
means subclassing its classes and importing AGPL code.
The AGPL-linked code now lives in a separate package, hermes-evolver-problems,
which imports both the engine and this package. This package reaches it only
over a subprocess boundary (evolution/code/sidecar.py), and a test asserts
that nothing under evolution/ imports darwinian_evolver — a stray import would
relicense the project and would not otherwise fail anything.
What is MIT and engine-independent:
admission.py Sandboxed gate a candidate must pass before it is scored.
Visible failures feed the mutator; the full suite and replayed
recorded commands are held out, gating admission while leaking
neither names nor output — otherwise an optimizer learns to
satisfy the checks it can see. Credentials are stripped from
check environments; pytest exit 5 is a failure, so deleting the
tests cannot pass.
targets.py Target resolution that refuses tests, packaging and __init__
files, refuses paths outside the repo, and caps target size.
Turns verification_evidence.db rows into replayable checks,
skipping anything state-changing or networked.
evolve_code.py Orchestrator. Verifies the baseline passes its own gate
before evolving — otherwise every candidate scores against a
broken reference. Deploys to a draft PR and nothing else: code
cannot be canaried, because a bad tool implementation is
already executing before any outcome signal exists.
Also repoints the [darwinian] extra at its git source. It named a PyPI package
that does not exist, so the extra had never been installable.
591 tests.
Two reproduced correctness bugs, two subsystems reported as shipped that nothing could reach, and the smaller items behind them. Correctness evolve_code published multi-file changes one file at a time, and each call cut its own branch. A two-file target produced two branches holding one file each, both titled '2 file(s)', only the first pushed, and the branch reported back was the one that never left the machine. PRPublisher gains publish_many; publish() delegates to it. Verified: one branch, one commit, both files. The sidecar's size objective was inert. baseline_chars was stamped on the seed organism and never propagated, so every mutator-produced child divided its size by its own size and scored exactly 0.500 no matter how much code it added — the same defect as the original finding 01. It is a declared field now, carried by the mutator, and the arithmetic moved to hermes_problems.scoring so it can be tested without the AGPL engine. Reachability The agent-in-the-loop harness (~500 lines, 37 tests) was imported by nothing, and its three config fields were never read. Now behind --agent-eval / --agent-eval-reps, failing loudly when the agent cannot be reached rather than quietly scoring completions instead. Canary deployment had no caller and no CLI. Now --canary on evolve_skill, plus evolution.deploy.canary_cli to list, evaluate, promote and roll back. tests/test_wiring.py guards the whole class: every config field must be read, named subsystems must have a caller, flags must exist, console scripts must resolve, and every click CLI must be declared as one. It immediately found two more — external_importers had no console script, and holdout_ratio was a knob that silently did nothing, since the split takes the holdout as the remainder. Also Phase 4's A/B scored a hardcoded 0.0 baseline, making SHIP automatic; both arms now go through the same function. One sandbox per run instead of one per candidate (164 MB x 9,019 files was being re-copied ~20 times); hardlinking would be faster and is unsafe here, since checks execute the candidate's code. The sandbox is documented as a safety boundary, not a security one — verified that candidate code keeps $HOME and network access. The sweep can cover phases 2 and 3 via --phases; code is never swept. Lint 28 -> 5 (the rest are style preferences). Stale docstring in hermes_paths corrected. 621 tests, identical under both host conditions; 27 in the sidecar.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
If someone wanna use it as an experiment, then use it