You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+12-19Lines changed: 12 additions & 19 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,10 +1,10 @@
1
1
# skillprobe
2
2
3
-
AI coding tools like Claude Code, Cursor, Copilot etc all inject instructions into the LLM context behind the scenes -- skills, rules, system prompts, whatever you want to call them. Theres no good way to test whether those instructions are actually being followed though. You write a skill that says "never add docstrings" and half the time the model adds them anyway.
3
+
AI coding tools like Claude Code, Cursor, Copilot etc all inject instructions into the LLM context behind the scenes (skills, rules, system prompts, whatever you want to call them). Theres no good way to test whether those instructions are actually being followed though. You write a skill that says "never add docstrings" and half the time the model adds them anyway.
4
4
5
5
skillprobe is a local proxy that sits between your tool and the LLM API, captures the full request and response, and lets you run assertions against them. It works with subscriptions (Claude Pro, Cursor Pro, etc) since your tool handles authentication normally and skillprobe just observes the traffic going through.
6
6
7
-
It also has a **harness** that automates the entire flow -- no more manually opening Claude Code, typing prompts, and checking results. One command spins up the proxy, launches `claude -p` or Cursor's `agent -p` as subprocesses, runs your test scenarios, evaluates assertions, and tears everything down. Works like a test suite.
7
+
It also has a **harness** that automates the entire flow so you dont have to manually open Claude Code, type prompts, and check results yourself. One command spins up the proxy, launches `claude -p` or Cursor's `agent -p` as subprocesses, runs your test scenarios, evaluates assertions, and tears everything down.
Instead of manually running prompts through Claude Code or Cursor, the harness automates the full lifecycle. Write scenario YAML, run one command:
63
+
The proxy workflow is great for observing what happens in real sessions, but it still requires you to manually type prompts and check results. The harness automates all of that. You write scenario YAML describing what to test and it handles the rest:
64
64
65
65
```bash
66
66
uv run skillprobe harness tests/my-skill.yaml --harness claude-code --model claude-haiku-4-5-20251001
@@ -84,7 +84,7 @@ Running: tests/my-skill.yaml
84
84
85
85
### Scenario format
86
86
87
-
Scenarios support multi-step conversations, workspace fixtures, setup commands, and post-run assertions:
87
+
Each scenario can have multiple conversational steps, a workspace fixture that gets copied fresh for every run, setup commands that tweak the fixture before the test starts, and post-run assertions that check the state of the workspace after everything finishes:
88
88
89
89
```yaml
90
90
harness: claude-code
@@ -116,28 +116,21 @@ scenarios:
116
116
value: "commit"
117
117
```
118
118
119
-
The harness supports `contains`, `not_contains`, `regex`, `skill_present`, `skill_loaded`, `tool_called`, `file_exists`, and `file_contains` assertions. Any assertion can be inverted with `negate: true`.
119
+
The harness supports `contains`, `not_contains`, `regex`, `skill_present`, `skill_loaded`, `tool_called`, `file_exists`, and `file_contains` assertions, and any of them can be inverted with `negate: true`.
| Claude Code |`claude -p`| yes (full system prompt, skill detection) | yes |
126
-
| Cursor |`agent -p`| no (Cursor routes through its own API) | yes |
121
+
For Claude Code, the harness runs a local proxy so you get full system prompt visibility and skill detection on top of the behavioral checks. Cursor routes its API calls through its own servers so theres no proxy capture there, but behavioral testing (checking what the model actually said and did) works the same way on both.
127
122
128
123
### Generating tests
129
124
130
-
Point `init` at a skill directory and it uses the LLM to generate a starter test suite:
125
+
You dont have to write scenario YAML from scratch. Point `init` at a skill directory and it reads the SKILL.md, uses the LLM to figure out what should be tested (positive activation, negative activation, behavioral correctness, edge cases), and writes a starter YAML file you can review and tweak:
131
126
132
127
```bash
133
128
uv run skillprobe init ./skills/commit --harness claude-code
134
129
```
135
130
136
-
This reads the SKILL.md, generates scenarios covering positive activation, negative activation, behavioral correctness, and edge cases, then writes a YAML file you can review and tweak.
137
-
138
131
## Optimizing skills
139
132
140
-
This part borrows from Karpathy's autoresearch idea -- you tag your captures with session names, look at whats failing, apply mutations to the skill, and then compare before/after.
133
+
This part borrows from Karpathy's autoresearch idea. You tag your captures with session names, look at whats failing, apply mutations to the skill, and then compare before and after.
141
134
142
135
```bash
143
136
# run with your current skill and tag captures as v1
@@ -168,7 +161,7 @@ There are six mutation operators (add_constraint, add_negative_example, restruct
168
161
169
162
## Activation testing
170
163
171
-
Separate from whether a skill is being followed, theres also the question of whether it gets loaded at the right time. Skills arent always in context -- tools like Claude Code and Cursor load them dynamically based on relevance. If your skill's description or keywords are off, it might not load when it should, or load when it shouldnt.
164
+
Separate from whether a skill is being followed, theres also the question of whether it gets loaded at the right time. Skills arent always in context because tools like Claude Code and Cursor load them dynamically based on relevance. If your skill's description or keywords are off, it might not load when it should, or load when it shouldnt.
172
165
173
166
Activation tests let you define when a skill should and shouldnt be present:
[!!] "what is recursion" -- expected not loaded, was loaded
198
191
```
199
192
200
-
This isnt about testing Claude Code or Cursor's loading logic -- its about making sure your skill file has the right description and content so the tool picks it up when it should.
193
+
This isnt about testing Claude Code or Cursor's loading logic, its about making sure your skill file has the right description and content so the tool picks it up when it should.
201
194
202
195
## Commands
203
196
204
197
-`start` - run the proxy (`--watch`, `--session`, `--skills`)
205
198
-`captures` - list whats been captured
206
199
-`inspect <id>` - look at a specific capture in detail
207
200
-`assert <test.yaml>` - check captures against assertions
0 commit comments