Skip to content

Benchmark one Agent Skill across Claude Code, Codex, Copilot, Cursor, and Gemini CLI #9

Description

@alsoleg89

The portability map found a precise missing edge: major clients document Agent Skills support, but documentation alone does not prove equivalent behavior across them.

Client work queue

Each remaining child issue is independently contributable:

Comment on a child issue before starting so two people do not duplicate the same client/version.

Frozen protocol

Use the v0.8.0 benchmark kit. Each submitted result must pin the exact tested commit and canonical skill digest; historical runs keep their original pins when the skill changes.

  1. Copy the unchanged synthetic fixture into a fresh repository.
  2. Install the unmodified skill payload using the client's documented repository-local path.
  3. Start a fresh session for each frozen prompt:
    • direct invocation;
    • implicit invocation;
    • negative/non-trigger prompt.
  4. Capture discovery, activation, consent, tool use, resulting artifact, and the real Doubt receipt.
  5. Publish the exact client version, model, configuration, permission mode, install path, and sanitized raw output.
  6. Do not claim behavioral equivalence until all five clients have comparable evidence.

The v0.8.0 skill bundles its zero-dependency validator, so a network-disabled sandbox does not need npm registry access to produce a receipt.

Acceptance criteria

  • Re-runnable protocol
  • Machine-readable result file
  • Exact versions and relevant configuration
  • Full tested Git commit and canonical skill digest
  • Sanitized raw outputs
  • Generated artifact and matching receipt when one exists
  • npm run benchmark:portability passes
  • Skill, fixture, and prompts remain unchanged
  • No credentials, private repository data, personal paths, or user names

Positive, neutral, negative, and blocked results are all accepted when the limitation is explicit.

Current evidence

  • Live evidence map
  • Machine-readable map
  • Evidence receipt: 92489baf27f3cf3ccfa351721e1fc8fcdfcb2d70a5135c86bf799bbd233f39dc
  • Canonical v0.8.0 skill digest: 257e6d00695be382fb5169e405e03ef64bedba634ca1d138082c47382b95f5c2

Current baseline: 2 / 5 submitted clients. Codex passes all three prompts with the bundled validator; its earlier network-dependent failures remain public. Copilot passes direct and negative prompts but fails implicit validation. One client result cannot establish cross-client equivalence.

Metadata

Metadata

Assignees

No one assigned

    Labels

    help wantedExtra attention is neededneeds-evidenceClaim or fixture needs stronger source support

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions