Skip to content

Commit f7d68b2

Browse files
Anyeshclaude
andcommitted
docs: rewrite README in natural prose, remove dashes
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 725c5df commit f7d68b2

1 file changed

Lines changed: 12 additions & 19 deletions

File tree

README.md

Lines changed: 12 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
11
# skillprobe
22

3-
AI coding tools like Claude Code, Cursor, Copilot etc all inject instructions into the LLM context behind the scenes -- skills, rules, system prompts, whatever you want to call them. Theres no good way to test whether those instructions are actually being followed though. You write a skill that says "never add docstrings" and half the time the model adds them anyway.
3+
AI coding tools like Claude Code, Cursor, Copilot etc all inject instructions into the LLM context behind the scenes (skills, rules, system prompts, whatever you want to call them). Theres no good way to test whether those instructions are actually being followed though. You write a skill that says "never add docstrings" and half the time the model adds them anyway.
44

55
skillprobe is a local proxy that sits between your tool and the LLM API, captures the full request and response, and lets you run assertions against them. It works with subscriptions (Claude Pro, Cursor Pro, etc) since your tool handles authentication normally and skillprobe just observes the traffic going through.
66

7-
It also has a **harness** that automates the entire flow -- no more manually opening Claude Code, typing prompts, and checking results. One command spins up the proxy, launches `claude -p` or Cursor's `agent -p` as subprocesses, runs your test scenarios, evaluates assertions, and tears everything down. Works like a test suite.
7+
It also has a **harness** that automates the entire flow so you dont have to manually open Claude Code, type prompts, and check results yourself. One command spins up the proxy, launches `claude -p` or Cursor's `agent -p` as subprocesses, runs your test scenarios, evaluates assertions, and tears everything down.
88

99
## Quick start
1010

@@ -60,7 +60,7 @@ Supported assertion types are `contains`, `not_contains`, `regex`, `skill_presen
6060

6161
## Automated harness testing
6262

63-
Instead of manually running prompts through Claude Code or Cursor, the harness automates the full lifecycle. Write scenario YAML, run one command:
63+
The proxy workflow is great for observing what happens in real sessions, but it still requires you to manually type prompts and check results. The harness automates all of that. You write scenario YAML describing what to test and it handles the rest:
6464

6565
```bash
6666
uv run skillprobe harness tests/my-skill.yaml --harness claude-code --model claude-haiku-4-5-20251001
@@ -84,7 +84,7 @@ Running: tests/my-skill.yaml
8484
8585
### Scenario format
8686
87-
Scenarios support multi-step conversations, workspace fixtures, setup commands, and post-run assertions:
87+
Each scenario can have multiple conversational steps, a workspace fixture that gets copied fresh for every run, setup commands that tweak the fixture before the test starts, and post-run assertions that check the state of the workspace after everything finishes:
8888
8989
```yaml
9090
harness: claude-code
@@ -116,28 +116,21 @@ scenarios:
116116
value: "commit"
117117
```
118118

119-
The harness supports `contains`, `not_contains`, `regex`, `skill_present`, `skill_loaded`, `tool_called`, `file_exists`, and `file_contains` assertions. Any assertion can be inverted with `negate: true`.
119+
The harness supports `contains`, `not_contains`, `regex`, `skill_present`, `skill_loaded`, `tool_called`, `file_exists`, and `file_contains` assertions, and any of them can be inverted with `negate: true`.
120120

121-
### Harness support
122-
123-
| Harness | Command | Proxy capture | Behavioral testing |
124-
|---|---|---|---|
125-
| Claude Code | `claude -p` | yes (full system prompt, skill detection) | yes |
126-
| Cursor | `agent -p` | no (Cursor routes through its own API) | yes |
121+
For Claude Code, the harness runs a local proxy so you get full system prompt visibility and skill detection on top of the behavioral checks. Cursor routes its API calls through its own servers so theres no proxy capture there, but behavioral testing (checking what the model actually said and did) works the same way on both.
127122

128123
### Generating tests
129124

130-
Point `init` at a skill directory and it uses the LLM to generate a starter test suite:
125+
You dont have to write scenario YAML from scratch. Point `init` at a skill directory and it reads the SKILL.md, uses the LLM to figure out what should be tested (positive activation, negative activation, behavioral correctness, edge cases), and writes a starter YAML file you can review and tweak:
131126

132127
```bash
133128
uv run skillprobe init ./skills/commit --harness claude-code
134129
```
135130

136-
This reads the SKILL.md, generates scenarios covering positive activation, negative activation, behavioral correctness, and edge cases, then writes a YAML file you can review and tweak.
137-
138131
## Optimizing skills
139132

140-
This part borrows from Karpathy's autoresearch idea -- you tag your captures with session names, look at whats failing, apply mutations to the skill, and then compare before/after.
133+
This part borrows from Karpathy's autoresearch idea. You tag your captures with session names, look at whats failing, apply mutations to the skill, and then compare before and after.
141134

142135
```bash
143136
# run with your current skill and tag captures as v1
@@ -168,7 +161,7 @@ There are six mutation operators (add_constraint, add_negative_example, restruct
168161

169162
## Activation testing
170163

171-
Separate from whether a skill is being followed, theres also the question of whether it gets loaded at the right time. Skills arent always in context -- tools like Claude Code and Cursor load them dynamically based on relevance. If your skill's description or keywords are off, it might not load when it should, or load when it shouldnt.
164+
Separate from whether a skill is being followed, theres also the question of whether it gets loaded at the right time. Skills arent always in context because tools like Claude Code and Cursor load them dynamically based on relevance. If your skill's description or keywords are off, it might not load when it should, or load when it shouldnt.
172165

173166
Activation tests let you define when a skill should and shouldnt be present:
174167

@@ -197,16 +190,16 @@ skillprobe activation tests/test-activation.yaml --last 50
197190
[!!] "what is recursion" -- expected not loaded, was loaded
198191
```
199192

200-
This isnt about testing Claude Code or Cursor's loading logic -- its about making sure your skill file has the right description and content so the tool picks it up when it should.
193+
This isnt about testing Claude Code or Cursor's loading logic, its about making sure your skill file has the right description and content so the tool picks it up when it should.
201194

202195
## Commands
203196

204197
- `start` - run the proxy (`--watch`, `--session`, `--skills`)
205198
- `captures` - list whats been captured
206199
- `inspect <id>` - look at a specific capture in detail
207200
- `assert <test.yaml>` - check captures against assertions
208-
- `harness <test.yaml>` - automated end-to-end skill testing (`--harness`, `--parallel`, `--model`)
209-
- `init <skill-dir>` - generate test YAML from a skill using LLM
201+
- `harness <test.yaml>` - run automated end-to-end skill tests (`--harness`, `--parallel`, `--model`)
202+
- `init <skill-dir>` - generate starter test YAML from a skill
210203
- `analyze <test.yaml>` - find failure patterns, suggest mutations
211204
- `optimize <skill.md>` - apply a mutation (backs up the original)
212205
- `diff <test.yaml>` - compare sessions

0 commit comments

Comments
 (0)