diff --git a/.claude/settings.json b/.claude/settings.json index 77faf37..024c66e 100644 --- a/.claude/settings.json +++ b/.claude/settings.json @@ -54,6 +54,7 @@ "Skill(sentry-skills:gh-review-requests)", "Skill(sentry-skills:gha-security-review)", "Skill(sentry-skills:iterate-pr)", + "Skill(sentry-skills:mcp-usage-research)", "Skill(sentry-skills:pr-link-issue)", "Skill(sentry-skills:pr-writer)", "Skill(sentry-skills:presentation-creator)", diff --git a/README.md b/README.md index 3cfbf26..83eee46 100644 --- a/README.md +++ b/README.md @@ -57,6 +57,7 @@ Works with Claude Code, Cursor, Cline, GitHub Copilot, and other compatible agen | [gh-review-requests](skills/gh-review-requests/SKILL.md) | Fetch unread GitHub notifications for open PRs where review is requested from a specified team or opened by a team member. | | [gha-security-review](skills/gha-security-review/SKILL.md) | GitHub Actions security review for workflow exploitation vulnerabilities. | | [iterate-pr](skills/iterate-pr/SKILL.md) | Iterate on a PR until CI passes and actionable review feedback is addressed. | +| [mcp-usage-research](skills/mcp-usage-research/SKILL.md) | Analyze how users and AI agents use Sentry's MCP server (tools, clients, journeys, error/latency friction) from `mcp-server` project spans. | | [presentation-creator](skills/presentation-creator/SKILL.md) | Create data-driven presentation slides using React, Vite, and Recharts with Sentry branding. | | [pr-link-issue](skills/pr-link-issue/SKILL.md) | Append a GitHub issue link and its Linear ticket to the current PR's description. | | [pr-writer](skills/pr-writer/SKILL.md) | Create, refresh, and rewrite pull request titles and descriptions following Sentry conventions. | diff --git a/skills/claude-settings-audit/SKILL.md b/skills/claude-settings-audit/SKILL.md index 5da9bd2..00bd501 100644 --- a/skills/claude-settings-audit/SKILL.md +++ b/skills/claude-settings-audit/SKILL.md @@ -158,6 +158,7 @@ If this is a Sentry project (or sentry-skills plugin is installed), include: "Skill(sentry-skills:gh-review-requests)", "Skill(sentry-skills:gha-security-review)", "Skill(sentry-skills:iterate-pr)", + "Skill(sentry-skills:mcp-usage-research)", "Skill(sentry-skills:pr-link-issue)", "Skill(sentry-skills:pr-writer)", "Skill(sentry-skills:presentation-creator)", diff --git a/skills/mcp-usage-research/SKILL.md b/skills/mcp-usage-research/SKILL.md new file mode 100644 index 0000000..e8f9cf3 --- /dev/null +++ b/skills/mcp-usage-research/SKILL.md @@ -0,0 +1,85 @@ +--- +name: mcp-usage-research +description: Analyze how real users and AI agents use Sentry's MCP server to surface usage patterns, popular tools, client mix, friction, and failure points. Use when asked "how is the MCP server used", "MCP usage research", "MCP tool usage", "which MCP clients", "how do agents use our MCP server", "MCP server adoption", "what tools are called most", "MCP error rates", "tool call analysis", or "API/MCP usage research". Server-side analog of replay-ux-research for programmatic (non-browser) usage. +argument-hint: '' +--- + +# MCP Usage Research + +Analyze server-side telemetry from Sentry's MCP server to understand how external users and AI agents actually use it — which tools they call, in what sequences, from which clients, and where they hit errors or friction. This is the API/MCP analog of `replay-ux-research`: there are no browser replays, so the evidence is spans in the `mcp-server` project rather than recorded sessions. + +## Inputs + +`$ARGUMENTS` is the research scope. It can be: + +- A **single tool** (e.g., `search_events`, `search_issues`) — deep-dive one tool's usage and reliability. +- A **client** (e.g., `claude-code`, `cursor-vscode`, `codex-mcp-client`) — how one client/agent uses the server. +- A **theme** (e.g., "errors", "adoption", "onboarding funnel"). +- `all` or empty — a broad usage overview across the whole server. + +If `$ARGUMENTS` is empty, ask the user what scope to research, offering the options above. + +## Prerequisites + +This skill requires the Sentry MCP server to be connected and authenticated. Tools used: + +- `search_events` — query spans in the `mcp-server` project (the core data source) +- `search_issues` / `get_sentry_resource` — investigate errors behind failing tool calls +- `find_projects` — confirm the `mcp-server` project slug if needed + +If these tools are unavailable, ask the user to connect/authenticate the Sentry MCP server before proceeding. Read `references/mcp-data-model.md` for the exact org, project, dataset, span schema, and query templates — **load it before running any query**. + +## Step 1: Frame the scope and pull the usage baseline + +Read `references/mcp-data-model.md`. Establish the baseline before drilling in: + +1. **Tool/method distribution** — group `span.op:mcp.server` by `span.description` to see which methods and tools dominate (the description holds method + tool, e.g. `tools/call search_events`; `mcp.tool.name` is null, do not use it). +2. **Client mix** — group by `mcp.client.name` / `mcp.client.version` to see which agents/clients drive usage (claude-code, Cursor, codex, Replit, CodeRabbit, etc.). +3. **Transport mix** — group by `mcp.transport` to split **remote/hosted** (`WorkerTransport`) from **local install** (`stdio`). These are different user populations; report them separately. + +Start with the last 24 hours. Extend to 7d or 30d if volume is low or you want trend context. Use aggregate queries (`count()`) for distributions — they are cheap and cover 100% of traffic, unlike the sampled detail rows. + +## Step 2: Scope-specific drill-down + +Narrow to `$ARGUMENTS`: + +- **Single tool**: filter `span.description:"tools/call "`. Pull call volume, the ok vs `internal_error` split (`span.status`), duration percentiles, and which clients call it. +- **Client**: filter `mcp.client.name:`. Pull that client's tool distribution, method mix, error rate, and version spread. +- **Theme / all**: keep the cross-cutting baseline from Step 1 and prioritize the lenses in Step 4. + +## Step 3: Reconstruct journeys (sequences) + +There is **no session attribute** — each MCP request is its own trace. Reconstruct journeys by grouping a user's calls over time: + +1. Pick active users via `user.id` (the authenticated Sentry account id; it is the closest thing to a "who"). +2. For a sample of high-activity `user.id`s, pull their `span.op:mcp.server` rows ordered by `timestamp` and read the `span.description` sequence. +3. Identify common patterns: + - **Onboarding funnel**: `initialize` → `tools/list` → first `tools/call`. Where do users stop? Many `initialize` with no follow-up `tools/call` = clients that connect but never invoke a tool. + - **Tool chains**: which tools are called back-to-back (e.g., `find_organizations` → `find_projects` → `search_issues`). These reveal real workflows. + - **Retry loops**: the same tool called repeatedly in quick succession by one user, especially after an `internal_error` — a friction signal. + +Always note this is a per-`user.id` reconstruction, not a true session, and that local `stdio` traffic may lack a stable `user.id`. + +## Step 4: Analyze usage through these lenses + +### Adoption & population +1. **Client/agent mix**: which clients dominate, and is usage concentrated or broad? Note long-tail/unknown clients. +2. **Local vs remote**: `stdio` (developers running it locally) vs `WorkerTransport` (hosted). Different needs; call out the split. +3. **Version spread**: are users on current client versions, or stuck on old ones (a sign of stale installs)? + +### What users do +1. **Tool popularity**: the head of the `tools/call` distribution is the product. What are the top 5 tools and what share of calls do they own? +2. **Method mix**: ratio of `tools/call` to discovery calls (`tools/list`, `resources/list`, `prompts/list`) and housekeeping (`ping`, `initialize`). A high discovery-to-call ratio can mean clients poll without acting. +3. **Workflows**: the tool chains from Step 3 — what jobs are users getting done? + +### Friction & failure +1. **Error rates**: per tool and per method, the `span.status:internal_error` share. Rank tools by both error *rate* and absolute error *count*. Some methods (e.g. `resources/list`, `prompts/list`) may be near-100% error — flag these as broken capabilities, not user friction. +2. **Latency**: duration percentiles (p50/p95) per tool. Slow tools hurt agent loops. Note that a few extreme outliers (multi-minute durations) exist — prefer percentiles over averages. +3. **Retry/abandonment**: retry loops and `initialize`-without-`tools/call` from Step 3. +4. **Investigate top failures**: for the highest-impact failing tools, use `search_issues` (project `mcp-server`) or `get_sentry_resource` on an issue/trace to find the underlying exception, how many users/events it affects, and whether it's being worked on. Classify each as **broken** (high error rate, likely always fails), **flaky** (intermittent), or **user-driven** (bad input / expected validation errors). + +## Step 5: Write the report + +Use the template in `references/output-template.md`. Be specific — cite real numbers (call counts, error rates, percentiles) and link to the Sentry query/trace URLs returned by `search_events` so the reader can verify. Separate remote and local populations where they differ. Distinguish **broken capabilities** from **user friction** — they have different owners and fixes. + +**Privacy**: `user.id` is a numeric Sentry account identifier. Do not enumerate raw `user.id`s in the report — anonymize to "User A/B/C" or describe by client/behavior. Never include user emails. diff --git a/skills/mcp-usage-research/SPEC.md b/skills/mcp-usage-research/SPEC.md new file mode 100644 index 0000000..51ab7fe --- /dev/null +++ b/skills/mcp-usage-research/SPEC.md @@ -0,0 +1,62 @@ +# MCP Usage Research Specification + +## Intent + +Give Sentry teams a UX-research-style view of how their MCP server is *actually* used, for usage that is purely programmatic (AI agents and API clients) and therefore invisible to browser session replays. It is the server-side counterpart to `replay-ux-research`: same questions (who, what jobs, what friction, what to fix), different evidence (spans in the `mcp-server` project instead of recorded sessions). + +## Scope + +In scope: +- MCP server usage recorded as `mcp.server` spans in the `sentry` org's `mcp-server` project. +- Tool/method distributions, client and transport mix, version spread. +- Per-`user.id` journey reconstruction, error/failure analysis, latency, retry/abandonment. + +Out of scope: +- Browser/UI usage — that is `replay-ux-research`. +- Non-MCP public REST API usage. **Extension point**: the same skill shape (baseline distribution → drill-down → journeys → friction → report) applies to REST-API telemetry once its data source is identified (likely `http.server` spans in the relevant API project). When adding it, add a sibling data-model reference and a scope branch; keep MCP and REST sections of any report separate. + +## Users And Trigger Context + +- Primary users: Sentry engineers/PMs who own the MCP server or its tools. +- Common user requests: "how is the MCP server used", "which tools/clients are most popular", "MCP error rates", "how do agents use our MCP server". +- Should not trigger for: browser UX questions (use `replay-ux-research`), generic Sentry issue triage, or non-Sentry MCP servers. + +## Runtime Contract + +- Required first actions: confirm Sentry MCP is authenticated; load `references/mcp-data-model.md` before querying. +- Required outputs: a report following `references/output-template.md`, citing real numbers and Sentry query/trace links. +- Non-negotiable constraints: never enumerate raw `user.id`s or user emails; separate remote (`WorkerTransport`) from local (`stdio`) populations; distinguish broken capabilities from user friction. +- Expected bundled files loaded at runtime: `references/mcp-data-model.md`, `references/output-template.md`. + +## Source And Evidence Model + +Authoritative sources: +- `mcp.server` spans (`mcp-server` project, `spans` dataset) via `search_events`. +- Underlying exceptions via `search_issues` / `get_sentry_resource` for failure root-cause. + +Data that must not be stored: +- Raw `user.id` lists, user emails, secrets, or any customer data. Anonymize to "User A/B/C" in outputs. + +## Reference Architecture + +- `SKILL.md` contains: the runtime workflow (scope → baseline → drill-down → journeys → lenses → report). +- `references/` contains: `mcp-data-model.md` (span schema, attributes, query templates, gotchas) and `output-template.md` (report structure). + +## Validation + +- Lightweight: queries return non-empty aggregates for `span.op:mcp.server`; report sections are populated with real numbers. +- Deeper: at least one failing tool is root-caused via an issue/trace link; remote vs local split is reported. +- Acceptance gates: no raw identifiers in output; broken-capability vs user-friction distinction made. + +## Known Limitations + +- No native MCP session attribute; journeys are approximate (`user.id` + timestamp). +- `internal_error` conflates server bugs, client cancellations, auth, and validation errors. +- Mixed audience: spans include Sentry-internal/dogfood usage with no clean employee filter. +- Span attribute names are point-in-time; re-verify if queries return empty. + +## Maintenance Notes + +- Update `SKILL.md`: when the workflow or lenses change, or when REST-API scope is added. +- Update `references/mcp-data-model.md`: when span attribute names, `span.op` values, or connection facts change (the MCP instrumentation evolves). +- Update `references/output-template.md`: when the report's required sections change. diff --git a/skills/mcp-usage-research/references/mcp-data-model.md b/skills/mcp-usage-research/references/mcp-data-model.md new file mode 100644 index 0000000..fa7b03f --- /dev/null +++ b/skills/mcp-usage-research/references/mcp-data-model.md @@ -0,0 +1,120 @@ +# MCP Server Span Data Model + +How MCP usage is recorded in Sentry, and the queries to read it. Verified 2026-06-12 against the `sentry` org's `mcp-server` project. These are point-in-time facts — re-verify attribute names if a query returns unexpectedly empty. + +## Connection facts + +| Field | Value | +| ----- | ----- | +| Org slug | `sentry` | +| Region URL | `https://us.sentry.io` | +| Project slug | `mcp-server` | +| Dataset | `spans` | +| Unit of analysis | `span.op:mcp.server` (one span per MCP JSON-RPC request) | + +There are sibling projects (`mcp-server-test`, `mcp-server-connect-demo`, `spotlight-mcp-server-instrumentation`) — use **`mcp-server`** for production usage. + +## Span operations (`span.op`) + +| `span.op` | Meaning | +| --------- | ------- | +| `mcp.server` | **Primary unit.** One per inbound MCP request (initialize, tools/list, tools/call, ping, etc.). | +| `mcp.notification.client_to_server` | Client→server notifications. | +| `mcp.execute_tool` | Inner tool-execution span. Note: `mcp.tool.name` is **null** here — get the tool name from the parent `mcp.server` span's `span.description`. | +| `http.server` / `http.client` | Underlying HTTP transport spans (not MCP-semantic). | +| `gen_ai.*` | LLM/agent spans from server-side AI tooling (e.g. Seer). Not user-facing MCP calls. | + +## Key attributes on `mcp.server` spans + +| Attribute | Example values | Use | +| --------- | -------------- | --- | +| `span.description` | `tools/call search_events`, `tools/list`, `initialize`, `ping` | **Method + tool name.** Best grouping key for "what was called". | +| `mcp.method.name` | `tools/call`, `tools/list`, `initialize`, `ping`, `resources/list`, `prompts/list`, `notifications/initialized` | The JSON-RPC method. May be null on some rows. | +| `mcp.transport` | `WorkerTransport` (remote/hosted), `stdio` (local install), `null` | Splits the user population. | +| `mcp.client.name` | `claude-code`, `cursor-vscode`, `Cursor`, `codex-mcp-client`, `Anthropic/ClaudeAI`, `Replit-Agent-MCP-Client`, `CodeRabbit` | Which agent/client. **Only set on `initialize` spans — null on `tools/call` and most other methods.** To attribute a tool call to a client, join via `user.id` to that user's `initialize` span. | +| `mcp.client.version` | `2.1.175`, `1.0.0` | Version spread / stale installs. | +| `span.status` | `ok`, `internal_error`, `unknown` | Friction signal. `internal_error` = failed call. `unknown` often = in-flight/streaming. | +| `span.duration` | milliseconds | Latency. Has extreme outliers (multi-minute) — use percentiles, not averages. | +| `user.id` / `user` | `1234567`, `id:1234567` | Authenticated Sentry account id — the "who" and the journey key. | +| `timestamp` | ISO 8601 | Order calls to reconstruct sequences. | + +**Attributes that do NOT exist / are unreliable:** +- `mcp.tool.name` — null; use `span.description`. +- `mcp.session.id` — does not exist. Each call is its own trace; group by `user.id` + `timestamp` for journeys. + +## Tool-call arguments (`mcp.request.argument.*`) + +The arguments a client passed to a `tools/call` are captured as per-parameter attributes named `mcp.request.argument.`, where the attribute *value* is the JSON-encoded argument (strings arrive quoted, e.g. `"resolved"`). This lets you analyze **what** agents do, not just which tool they called. Attribute names mirror the tool's input schema, so they vary per tool — e.g. for `update_issue`: `mcp.request.argument.status`, `mcp.request.argument.issueId`, `mcp.request.argument.organizationSlug`; for `search_events`: `mcp.request.argument.query`, `mcp.request.argument.dataset`. + +Use this to characterize usage and find bulk/loop opportunities: +``` +query: span.op:mcp.server span.description:"tools/call update_issue" +fields: ["mcp.request.argument.status", "count()"] +sort: -count() +``` + +Caveats: +- Only arguments the client actually sent are present; omitted optional params are absent (use `has:mcp.request.argument.` to filter). A grouping will show `null` for rows where that param wasn't passed. +- Values are JSON-encoded (quoted strings). Don't assume the exact attribute name — confirm it by first fetching a few individual rows with candidate fields, since names depend on the tool's schema and may change as tools evolve. +- Treat argument values as potentially sensitive (they can contain user queries, IDs, free text). Do not reproduce raw values verbatim in reports beyond what's needed to make the point. + +## Query templates + +All via `search_events` with `organizationSlug:'sentry'`, `projectSlug:'mcp-server'`, `dataset:'spans'`, `regionUrl:'https://us.sentry.io'`. Pass explicit `fields` to avoid the agent guessing the wrong column. + +**Tool/method distribution (baseline):** +``` +query: span.op:mcp.server +fields: ["span.description", "count()"] +sort: -count() +``` + +**Client mix:** +``` +query: span.op:mcp.server has:mcp.client.name +fields: ["mcp.client.name", "mcp.client.version", "count()"] +sort: -count() +``` + +**Transport split:** +``` +query: span.op:mcp.server +fields: ["mcp.transport", "count()"] +sort: -count() +``` + +**Per-tool reliability (one tool):** +``` +query: span.op:mcp.server span.description:"tools/call search_events" +fields: ["span.status", "count()"] +sort: -count() +``` + +**Per-tool latency:** +``` +query: span.op:mcp.server span.description:"tools/call search_events" +fields: ["p50(span.duration)", "p95(span.duration)", "count()"] +``` + +**One user's journey (sequence):** +``` +query: span.op:mcp.server user.id: +fields: ["timestamp", "span.description", "span.status", "span.duration"] +sort: timestamp +``` + +**Active users to sample:** +``` +query: span.op:mcp.server mcp.method.name:tools/call +fields: ["user.id", "count()"] +sort: -count() +``` + +## Gotchas + +- **The `search_events` agent rewrites natural-language queries** and may substitute the wrong field (it guessed `gen_ai.tool.name` and `has:gen_ai.tool.name` during development, returning zero rows). Prefer explicit Sentry search syntax in `query` plus explicit `fields`. If a result is surprisingly empty, check whether the agent's "Executed Search" line changed your filter. +- **Aggregates cover all traffic; detail rows are sampled.** Use `count()` groupings for distributions and reliability; use individual-row fetches only for journey reconstruction and examples. +- **`internal_error` is not always a server bug.** It can include client cancellations, auth failures, and validation errors. Confirm root cause via `search_issues`/`get_sentry_resource` before calling a tool "broken". +- **Some methods are near-100% error** (e.g. `resources/list`, `prompts/list`, `resources/templates/list` showed high `internal_error`). Treat persistent full-method failure as a broken capability to flag, distinct from per-tool user friction. +- **Local (`stdio`) traffic** may lack a stable `user.id` and is a different population from hosted (`WorkerTransport`). Report them separately. +- **Employee/dogfood traffic is mixed in.** Unlike replays, there is no clean email-domain filter on spans. If the user wants external-only usage, check at runtime whether a `user.email` attribute is available to filter; otherwise state that the sample includes Sentry-internal usage. diff --git a/skills/mcp-usage-research/references/output-template.md b/skills/mcp-usage-research/references/output-template.md new file mode 100644 index 0000000..214df19 --- /dev/null +++ b/skills/mcp-usage-research/references/output-template.md @@ -0,0 +1,133 @@ +# MCP Usage Research: [Scope] + +## Overview + +**Objective**: Understand how [scope — all usage / tool X / client Y] of Sentry's MCP server is used: what's called, by whom, in what sequences, and where it fails. + +**Scope**: [scope], [time range] +**Date**: [today's date] +**Method**: Analyzed `mcp.server` spans in the `mcp-server` project (org `sentry`). Distributions are from aggregate counts (100% of traffic); journeys are reconstructed per `user.id` from sampled rows. See [Methodology & Data](#methodology--data). + +## Key Findings + +1. [Most important finding — 1-2 sentences, with a number] +2. ... +(aim for 4-6) + +## Recommendations + +1. **[Recommendation]** — [which finding it addresses, expected impact, likely owner] +2. ... + +--- + +## Adoption & Population + +### Client / Agent Mix + +| Client | Calls | % | Notes | +| ------ | ----- | - | ----- | +| claude-code | | | | +| cursor-vscode / Cursor | | | | +| codex-mcp-client | | | | +| ... | | | | + +[Is usage concentrated in a few clients or broad? Note any large unknown/long-tail clients.] + +### Local vs Remote + +| Transport | Calls | % | Population | +| --------- | ----- | - | ---------- | +| `WorkerTransport` | | | Hosted / remote | +| `stdio` | | | Local install (developers) | + +[Call out behavioral differences between the two populations.] + +### Version Spread + +[Are clients on current versions? Any notable stale-version tails?] + +## What Users Do + +### Top Tools + +| Tool | Calls | % of `tools/call` | Error rate | +| ---- | ----- | ----------------- | ---------- | +| | | | | + +[The head of this list is effectively the product. What share do the top 5 own?] + +### Method Mix + +[Ratio of `tools/call` to discovery (`tools/list`, `resources/list`, `prompts/list`) and housekeeping (`ping`, `initialize`). High discovery-without-calls = clients polling but not acting.] + +### Workflows (Tool Chains) + +Common back-to-back tool sequences reconstructed from user journeys: + +- `find_organizations` → `find_projects` → `search_issues` — [N users, what job this accomplishes] +- ... + +## Friction & Failure + +### Broken Capabilities (likely always failing) + +Methods/tools with persistent high error rates — these are bugs, not user friction. + +- `[method/tool]` — [error rate], [error count] — [underlying issue + link if investigated] + +### Failing Tools (user-impacting) + +| Tool | Error rate | Error count | Classification | Underlying issue | +| ---- | ---------- | ----------- | -------------- | ---------------- | +| | | | broken / flaky / user-driven | [issue ID + link] | + +[For each significant failure, note: what the error is, how many users/events it affects, whether it's assigned. Link the Sentry issue/trace.] + +### Latency + +| Tool | p50 | p95 | Notes | +| ---- | --- | --- | ----- | +| | | | | + +[Slow tools hurt agent loops. Flag p95 outliers. Use percentiles — extreme multi-minute outliers exist.] + +### Retry Loops & Abandonment + +- **Retry loops**: [tools repeatedly called by one user in quick succession, often after errors] +- **Abandonment**: [share of `initialize` sessions with no following `tools/call` — clients that connect but never invoke] + +## Notable Journeys + +3-5 illustrative `user.id` sequences (anonymized): + +1. **User A** ([client]) — [sequence summary + what it reveals] — [trace/query link] +2. ... + +--- + +## Methodology & Data + +### Sample & Coverage + +| Item | Value | +| ---- | ----- | +| Time range | | +| Total `mcp.server` spans | | +| Distinct `user.id`s | | +| Distinct clients | | +| Transport split (remote / local) | | + +### Limitations + +- **No native session**: each request is its own trace; journeys are reconstructed by grouping `user.id` over time and are approximate. +- **`internal_error` ≠ server bug**: may include client cancellations, auth failures, and validation errors. Root cause confirmed only where noted. +- **Mixed audience**: the sample includes Sentry-internal/dogfood usage; there is no clean employee filter on spans. [State whether any filtering was applied.] +- **Local traffic identity**: `stdio` traffic may lack a stable `user.id`, under-representing local-user journeys. +- **Snapshot, not trend**: a single time window. Re-run at other times to compare. + +### Appendix: Tool Call Distribution + +| Tool / Method | Calls | OK | Error | Error % | +| ------------- | ----- | -- | ----- | ------- | +| | | | | |