Skip to content

skill_structure validator false positive — rejects all evolved candidates #110

Description

@solidstatecc

skill_structure validator false positive — rejects all evolved candidates

Problem

The skill_structure constraint validator checks if the evolved skill body contains YAML frontmatter. However, by design, skill["body"] has the frontmatter already stripped. This causes the validator to reject every evolved candidate, including an unmodified baseline, blocking the holdout evaluation step.

Root Cause

load_skill() separates frontmatter from body. The validator then checks the body (with frontmatter stripped) for frontmatter markers — which will always fail.

Evidence

Both Run 01 & Run 02 report:

✗ skill_structure: Skill missing: YAML frontmatter (---), name field, description field
✗ Evolved skill FAILED constraints — not deploying

This blocks holdout eval even for valid, unmodified baselines.

Solution

Check the reassembled skill (with frontmatter re-attached) in the validator instead of the bare body.

Suggested Action

  • Pass reassembled skill to validator's skill_structure check, or
  • Document that skill_structure only applies to reassembled artifacts, or
  • Remove the frontmatter check and verify only during reassembly

Hit this while preparing #109.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions