Multi-file skills with executable scripts: do agents need extra guidance?

[marc0olo]

Context

PR #89 (canhelp) introduces a skill that ships shell scripts in scripts/. Unlike icp-cli's reference markdown files (which agents just read into context), these scripts need to be saved to disk and executed.

The serving infrastructure handles this already — index.json lists all files, the catch-all route serves them as text/plain, and the zip bundles everything. But llms.txt only describes the "fetch index → fetch SKILL.md" flow and doesn't mention additional files or execution.

Open question

Is this a real problem, or do capable agents infer the download-and-execute pattern from context? If canhelp (or similar skills) get evals, that would answer it empirically.

If it does turn out to be a problem, lightweight fixes could include:

• A brief note in llms.txt about the files array and how to fetch additional files
• A convention for skill authors to include setup instructions when their skill has executable files

Worth validating before adding guidance that may be unnecessary.

[marc0olo]

An eval for canhelp could directly validate this. The interesting case isn't "does the skill work when pre-loaded" — it's whether the agent can go from the discovery flow to actually executing the scripts.

A focused eval prompt like:

{
  "name": "Resolve canister by name",
  "prompt": "What methods does the NNS Governance canister expose? Use the canhelp skill to find out.",
  "expected_behaviors": [
    "Fetches and executes scripts/resolve-canister-id.sh to resolve the canister name",
    "Fetches and executes scripts/fetch-candid.sh with the resolved canister ID",
    "Reads the resulting .did file",
    "Presents a grouped summary of query and update methods"
  ]
}

This would exercise the full script discovery/execution chain. If the agent fails at the "fetch and execute" step, that's a clear signal that llms.txt or the SKILL.md needs extra guidance.

Note: this eval requires icp CLI and network access, so it can't run in every CI environment. But even running it manually once would answer the question.

[marc0olo]

Related: #89