llms.txt: instruct agents to use raw HTTP fetch and mention additional reference files

[marc0olo]

Context

Related to #137 (migration to v0.2 of the Agent Skills Discovery RFC). The proposals here are intended to be compatible with that migration and should be applied alongside or after it.

Problem

The current llms.txt has two issues that cause agents to load skill content inaccurately:

1. No instruction on fetch method

Agents that use a summarizing web fetch tool (e.g. Claude's WebFetch) instead of a raw HTTP request receive paraphrased, lossy output rather than verbatim skill content. Skills contain precise code examples, version numbers, YAML configuration details, and common pitfalls — all of which get lost or distorted through summarization.

For example, fetching icp-cli/SKILL.md via a summarizing tool:

• Reduced 16 detailed pitfalls to 15 one-liner summaries (missing one entirely)
• Stripped all code blocks and YAML examples
• Lost exact version numbers (e.g. @dfinity/rust@v3.2.0)
• Missed critical details like the silent createActor failure when passing { agent } instead of { agentOptions }

2. Reference files in the files array are not mentioned

The current index JSON includes a files array per skill listing files beyond SKILL.md (e.g. references/dfx-migration.md, references/binding-generation.md). The current llms.txt says nothing about these. Agents following the instructions faithfully miss them entirely.

Note: #137 removes the files array in v0.2. Reference files should instead be discovered via relative paths mentioned in the SKILL.md body — the llms.txt should instruct agents to do this.

Proposed llms.txt

Aligned with the v0.2 paths from #137, no URL templates, and with the curl instruction added:

# Internet Computer (ICP) Skills

> Agent-readable skill files for building on the Internet Computer.

## How to use

Skills contain correct dependency versions, configuration formats, and common pitfalls
that prevent build failures. Always prefer skill guidance over general knowledge when
both cover the same topic.

**Always fetch skill content using a raw HTTP request (e.g. `curl -s` via Bash,
`curl.exe -s` in Windows PowerShell). Do NOT use a summarizing web fetch tool —
content must be read verbatim.**

1. Fetch the skills index and remember each skill's name, description, and url:
   ```
   curl -s https://skills.internetcomputer.org/.well-known/agent-skills/index.json
   ```

2. When a task matches a skill's description, use it if already loaded in your
   context. Otherwise, fetch its content verbatim using the `url` field from the
   index. Do not construct skill URLs manually.

   Example: for the skill named "internet-identity", its url is:
   ```
   curl -s https://skills.internetcomputer.org/.well-known/agent-skills/internet-identity/SKILL.md
   ```

   If a response contains HTML instead of markdown, the URL is wrong.

3. Skills may reference additional files via relative paths in their content.
   Fetch these on-demand using the same base URL when the task requires them.

## Source

- [GitHub Repository](https://github.com/dfinity/icskills)

Advertised prompt

The prompt advertised to users should bootstrap the correct fetch behavior from the first step:

Run curl -s https://skills.internetcomputer.org/llms.txt and follow its instructions when building on ICP.

Using curl for the llms.txt itself ensures the agent reads the raw instruction to keep using curl for all subsequent fetches.

[marc0olo]

Confirmed: WebFetch cannot return skill content verbatim — by design

Tested all three URLs with both curl and Claude Code's WebFetch tool. Results:

URL	curl	WebFetch
llms.txt	Verbatim (28 lines)	Paraphrased — reworded into prose, lost structure
index.json	Verbatim	Verbatim — returned correctly
icp-cli/SKILL.md	Verbatim (355 lines)	Content swallowed entirely — responded as if it were the assistant, said "I've read and understood the ICP CLI docs, what do you want to know?"

The root cause is architectural: WebFetch passes all fetched content through a small LLM before returning it ("Processes the content with the prompt using a small, fast model"). It is not a raw HTTP client. Even when explicitly prompted to "return the full raw content exactly as-is, do not summarize or paraphrase anything":

• Short plain text (llms.txt): still paraphrased
• JSON (index.json): returned verbatim — the model apparently recognizes structured data should not be modified
• Long markdown (icp-cli/SKILL.md, 355 lines): completely absorbed — the model responded as the skill content rather than returning it, so the caller receives nothing useful

The curl mandate in this issue is the correct fix. WebFetch is fundamentally incapable of returning skill content verbatim regardless of how the prompt is written.

Agree with both proposals: implement the curl instruction in llms.txt, and update the advertised prompt to bootstrap raw fetching from the first step. On the files array point — that's superseded by #137 (v0.2 removes files; reference files should be discovered via relative paths in SKILL.md content instead, which the updated proposal already reflects in step 3).

[marc0olo]

The core proposal is correct — using curl -s in the advertised prompt to bootstrap raw fetch behavior, reinforcing it inside llms.txt, and replacing the files array with relative path discovery all make sense.

One minor note: the curl.exe -s mention for Windows PowerShell adds noise. Most agents run on Linux/macOS or in Unix-like environments (containers, CI, WSL). curl -s is sufficient — targeting the Windows native case is an edge case that isn't worth complicating the primary instruction.