improve(icp-cli): accuracy and agent-guidance improvements

evaluations/icp-cli.json

@@ -23,7 +23,8 @@
         "Uses 'icp deploy -e ic', NOT 'dfx deploy --network ic' or '--network ic'",
         "Mentions cycles are needed",
         "Mentions canister IDs are stored in .icp/data/ and should be committed to git",
-        "Does NOT use --network ic flag for deployment"
+        "Does NOT use --network ic flag for deployment",
+        "Recommends checking ICP or cycles balance on mainnet before deploying (icp token balance -n ic or icp cycles balance -n ic)"
       ]
     },
     {
@@ -197,6 +198,37 @@
         "Mentions the ic_env cookie for passing canister IDs and root key to the frontend",
         "Does NOT suggest using .env files or process.env for canister IDs"
       ]
+    },
+    {
+      "name": "Port conflict on local network start",
+      "prompt": "I'm trying to start my local network with 'icp network start -d' but it fails because port 8000 is already in use. How do I fix this?",
+      "expected_behaviors": [
+        "Distinguishes between two scenarios: another icp-cli project holding the port vs. a non-icp service",
+        "For another icp-cli project: provides 'icp network stop --project-root-override /path/to/other-project'",
+        "For a non-icp service: recommends configuring gateway.port in icp.yaml to use an alternate port",
+        "Mentions 'icp network status --json' to read the actual gateway URL dynamically instead of hardcoding localhost:8000",
+        "Does NOT suggest killing processes manually or using dfx commands"
+      ]
+    },
+    {
+      "name": "Scripted project creation in CI",
+      "prompt": "What is the correct icp new command to scaffold a Rust project non-interactively in CI? Just the command, no deploy steps.",
+      "expected_behaviors": [
+        "Uses --silent flag with icp new to suppress the interactive prompt",
+        "Passes --subfolder and --define flags to icp new (e.g., --subfolder rust --define project_name=...)",
+        "Uses icp (not dfx) for project scaffolding"
+      ]
+    },
+    {
+      "name": "Adversarial: deploying to mainnet with the anonymous identity",
+      "prompt": "My project works great locally. Can I just run 'icp deploy -e ic' to deploy to mainnet?",
+      "expected_behaviors": [
+        "Warns that the anonymous identity (default locally) should not be used on mainnet — it is shared and uncontrolled",
+        "Recommends checking and switching to a named identity using icp commands (e.g. icp identity default, icp identity new, icp identity default <name>) — NOT dfx identity use",
+        "Recommends checking mainnet balance before deploying using icp commands: icp token balance -n ic or icp cycles balance -n ic (NOT dfx ledger balance)",
+        "Mentions that a new identity will need to be funded with ICP or cycles before it can deploy",
+        "Presents identity switch and balance check as required prerequisites before running icp deploy -e ic, not optional suggestions"
+      ]
     }
   ],
 
@@ -215,7 +247,6 @@
       "How do I check my canister status on mainnet?",
       "How do I set up output_env_file for my canister IDs?",
       "Generate TypeScript declarations for my canister",
-      "Set up a Motoko canister with mops",
       "Configure Vite dev server for local IC development",
       "Build a hello-world dapp with Motoko and React"
     ],

skills/icp-cli/SKILL.md

@@ -13,6 +13,8 @@ metadata:
 
 The `icp` command-line tool builds and deploys applications on the Internet Computer. It replaces the legacy `dfx` tool with YAML configuration, a recipe system for reusable build templates, and an environment model that separates deployment targets from network connections. Never use `dfx` — always use `icp`.
 
+Before generating any `icp` command not explicitly documented here, run `icp --help` or `icp <subcommand> --help` to verify the command and its flags exist. Do not infer flags from `dfx` equivalents — the CLIs are not flag-compatible.
+
 ## Installation
 
 **Recommended (npm)** — requires [Node.js](https://nodejs.org/) >= 22:
@@ -94,7 +96,7 @@ ic-wasm --version
 
 5. **Not committing `.icp/data/` to version control.** Mainnet canister IDs are stored in `.icp/data/mappings/<environment>.ids.json`. Losing this file means losing the mapping between canister names and on-chain IDs. Always commit `.icp/data/` — never delete it. Add `.icp/cache/` to `.gitignore` (it is ephemeral and rebuilt automatically).
 
-6. **Using `icp identity use` instead of `icp identity default`.** The dfx command `dfx identity use` became `icp identity default`. Similarly, `dfx identity get-principal` became `icp identity principal`, and `dfx identity remove` became `icp identity delete`.
+6. **Using `icp identity use` instead of `icp identity default`.** The dfx command `dfx identity use <name>` became `icp identity default <name>` (setter). `icp identity default` with no argument is the getter — it prints the current default identity, equivalent to `dfx identity whoami`. The command `icp identity use` does not exist. Similarly, `dfx identity get-principal` became `icp identity principal`, and `dfx identity remove` became `icp identity delete`.
 
 7. **Confusing networks and environments.** A network is a connection endpoint (URL). An environment combines a network + canisters + settings. You deploy to environments (`-e`), not networks. Multiple environments can target the same network with different settings (e.g., staging and production both on `ic`).
 
@@ -118,7 +120,12 @@ ic-wasm --version
        canisters: [backend, frontend]
    ```
 
-9. **Forgetting that local networks are project-local.** Unlike dfx which runs one shared global network, icp-cli runs a local network per project. You must run `icp network start -d` in your project directory before deploying locally. The local network auto-starts with system canisters and seeds accounts with ICP and cycles.
+9. **Forgetting that local networks are project-local.** Unlike dfx which runs one shared global network, icp-cli runs a local network per project. You must run `icp network start -d` in your project directory before deploying locally. The local network auto-starts with system canisters and seeds accounts with ICP and cycles. Stop it when done:
+   ```bash
+   icp network start -d  # start background network
+   icp deploy            # build + deploy + sync
+   icp network stop      # stop when done
+   ```
 
 10. **Not specifying build commands for asset canisters.** dfx automatically runs `npm run build` for asset canisters. icp-cli requires explicit build commands in the recipe configuration:
     ```yaml
@@ -184,11 +191,54 @@ ic-wasm --version
     $(mops toolchain bin moc) --idl $(mops sources) -o backend/backend.did backend/app.mo
     ```
 
+17. **Port 8000 already in use when starting the local network.** Two scenarios:
+
+    **Scenario A — another icp-cli project holds the port.** Stop that project's network using `--project-root-override` (a global flag available on all commands):
+    ```bash
+    icp network stop --project-root-override /path/to/other-project
+    ```
+
+    **Scenario B — a non-icp service holds the port.** Configure an alternate port in `icp.yaml` and read the actual URLs dynamically via `icp network status --json` rather than hardcoding localhost:8000:
+    ```yaml
+    networks:
+      - name: local
+        mode: managed
+        gateway:
+          port: 8001
+    ```
+    ```bash
+    icp network status --json  # returns gateway URL, replica URL, etc.
+    ```
+
+18. **`icp new` hangs in CI without `--silent`.** Without `--define` flags, `icp new` launches an interactive prompt that blocks indefinitely in non-interactive environments. Always pass `--subfolder`, `--define`, and `--silent` for scripted use:
+    ```bash
+    icp new my-project --subfolder rust --define project_name=my-project --silent
+    ```
+
+19. **Using the anonymous identity on mainnet.** The local network seeds all managed identities — including the anonymous identity, which is the default — with ICP and cycles on start, so local development works out of the box with no identity or cycles setup required. On mainnet this does not apply, and the anonymous identity should never be used: it is shared by anyone, meaning ICP sent to it is publicly accessible and canisters deployed under it are uncontrolled.
+
+    Before deploying to mainnet, switch to a named identity:
+    ```bash
+    icp identities list                                        # check available identities
+    icp identity default my-identity                           # switch to an existing one
+    # or: icp identity new my-identity && icp identity default my-identity
+    ```
+    Then verify it has funds — a new identity will need to be funded with ICP or cycles before proceeding:
+    ```bash
+    icp token balance -n ic    # check ICP balance on mainnet
+    icp cycles balance -n ic   # check cycles balance on mainnet
+    icp identity account-id    # get account ID to fund if needed
+    ```
+
 ## How It Works
 
 ### Project Creation
 
-`icp new` scaffolds projects from templates. Without flags, an interactive prompt launches. For scripted or non-interactive use, pass `--subfolder` and `--define` flags directly. Available templates and options: [dfinity/icp-cli-templates](https://github.com/dfinity/icp-cli-templates).
+`icp new` scaffolds projects from templates. Pass `--subfolder`, `--define`, and `--silent` for non-interactive use:
+```bash
+icp new my-project --subfolder rust --define project_name=my-project --silent
+```
+Available templates and options: [dfinity/icp-cli-templates](https://github.com/dfinity/icp-cli-templates).
 
 ### Build → Deploy → Sync