feat(analytics): parse-time enrichment from flight-recorder survey (NAS-1467) - #20
Merged
drewburchfield merged 6 commits intoJul 28, 2026
Merged
Conversation
…AS-1467)
Port the highest-value parse-time ideas from the flight-recorder survey:
- ToolTaxonomy module: toolKind() buckets tools into shell/file_edit/
file_read/search/task/web/mcp/other and parses the MCP server name;
fileChangeFromTool() extracts {path, change_kind, added/removed lines}
from Write/Edit/MultiEdit/NotebookEdit inputs.
- tool_usage gains tool_kind, mcp_server, error_count. Errors correlate
tool_result.is_error to the call via tool_use_id, never array order.
- New file_changes table (indexed by path and conversation) makes "which
conversations touched this file" a first-class query.
- messages gains src_line: 1-based transcript line provenance, recorded
at parse time because it cannot be backfilled.
- CANONICAL_CONVERSATIONS predicate: rollup queries exclude subagent
transcripts and empty sessions in one shared fragment.
- New read-only routes: GET /api/analytics/tools and
GET /api/analytics/file-changes?path= (input-validated).
- CONTENT_VERSION 4 -> 5: existing stores mark all files stale and
repopulate enrichment via the resumable background reindex.
Tests: 388 passing (was 366). New ToolTaxonomy unit suite, v5 enrichment
DB suite, parse-path enrichment suite, and an end-to-end integration
suite covering the REST surface and out-of-order error correlation.
…s no artifact Dogfood findings on the v5 build: - The analytics modal reported "Tool Calls 0" for every conversation in database-backend mode: _transformConversation serves a hardcoded toolUsage placeholder and the analytics route echoed it. The route now reads the tool_usage table (per-conversation counts, unique tools, correlated error totals) and falls back to the analyzer shape only in file-backed mode. Regression test pins the DB-backed path. - quality-gate-local.sh wrote its synthetic benchmark transcript into ~/.claude/projects/codex_quality_gate/ and never removed it, so it rendered in the UI as a real project full of "quality gate payload" messages (NAS-1473). The script now cleans up its bench file on exit. Tests: 389 passing.
…class populateAnalyticsModal is a standalone function, so the transcript-value escaping added in the v0.9.0 hardening threw "this.escapeHtml is not a function" the moment toolUsage.breakdown was non-empty. It had never executed with data before: the DB-backed list served a zeroed toolUsage placeholder, so the section always rendered empty and the earlier verification (grepping the served HTML for the escape calls) could not catch it. One shared escapeHtmlText() implementation now backs both the class method and the standalone modal code, so the escaper cannot fork again.
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| } | ||
| } | ||
| } | ||
| } else if (item.type === 'system' && typeof item.content === 'string') { |
Contributor
There was a problem hiding this comment.
🟡 Transcript line numbers missing for system, summary, queued, and attachment entries
The transcript line number is attached to per-message records only in the assistant/user branch (messageRecords.push({ seq: msgSeq++, src_line: lineCount, ...rec }) at src/analytics/data/Indexer.js:311), while the system/summary/queue-operation/attachment branches push records without it, so those rows are stored with no line-of-origin.
Impact: Some indexed entries can never be traced back to their exact place in the original transcript, contrary to the stated goal that every per-message record carries its line number.
Where src_line is dropped for non-message entries
Inside the rl.on('line', ...) handler lineCount is available for every branch. The assistant/user path stamps it at src/analytics/data/Indexer.js:310-312, but the later else if branches (src/analytics/data/Indexer.js:352-375 for system, summary, queue-operation, attachment) build records as { seq: msgSeq++, role, tool_name, text } with no src_line. upsertConversation then stores m.src_line ?? null, so these rows get NULL. Adding src_line: lineCount to those pushes would make provenance uniform.
(Refers to lines 352-375)
Prompt for agents
In src/analytics/data/Indexer.js, the assistant/user branch stamps src_line onto per-message records (line 311), but the four top-level branches that also push into messageRecords (system at ~line 352-358, summary, queue-operation, and attachment) omit src_line, so those rows are stored with NULL line provenance. lineCount is in scope in all these branches. Add `src_line: lineCount` to each of those messageRecords.push(...) calls so provenance is uniform across all indexed record types, matching the stated intent that every per-message record carries its 1-based transcript line.
Was this helpful? React with 👍 or 👎 to provide feedback.
…ojects Agent sessions run in isolated git worktrees (Agent tool worktree isolation, EnterWorktree, Cyrus) get transcript directories keyed by the worktree cwd and carry no parent linkage (isSidechain false, parentUuid null), so each one masqueraded as its own project in the main list: 45 of the ~/.claude/projects directories on the reference machine were worktree checkouts, not projects. WorktreeClassifier recognizes the three cwd conventions: <repo>/.worktrees/<name> (Agent tool / workflows) <repo>/.claude/worktrees/<name> (EnterWorktree) .../.cyrus/worktrees/<issue> (Cyrus; groups under "cyrus") Classified sessions are marked is_subagent (every default list, search, and canonical rollup already excludes those) plus a distinct is_worktree_agent flag, and project is re-attributed to the owning repo. resolveEncodedProjectNames uses the same classifier so it cannot reinstate the fake names. REST rows and MCP search results expose isWorktreeAgent so clients can label them; parent linkage stays null because the child transcript genuinely does not record it. Also: the four explicit column projections in the list/search read paths now include is_worktree_agent; without that the flag surfaced only on single-conversation fetches (caught by the new round-trip test). Tests: 405 passing (12 classifier unit tests, integration coverage for default-hidden / toggle-visible / rollup-excluded, DB round-trip). Companion flight-recorder ticket: NAS-1474.
…ce badges Claude Code message records carry an explicit `entrypoint`: "cli" and "claude-desktop" are interactive; "sdk-cli" (`claude -p` / SDK CLI, e.g. braintrust peer consults) and "sdk-py" (Python SDK harnesses) are headless. Sampled across 400 transcripts the values are cleanly enumerable, so no heuristic is needed (NAS-1473). - Indexer captures the first-seen entrypoint; conversations gains an `entrypoint TEXT` column (with migration); all read projections carry it; rows expose the raw value plus derived isHeadless (sdk prefix). - Headless sessions stay VISIBLE in lists: they are real sessions, unlike worktree agents. The distinction is labeling, not hiding. - Session list rows now render a source badge: "headless", "desktop", or "worktree agent"; interactive CLI (the norm) gets no badge. Labels are a fixed allowlist keyed on server-derived flags; the raw transcript-controlled entrypoint string never reaches the DOM. - MCP search results and the summary projection expose the new fields so API consumers can organize sessions by source. Tests: 411 passing (entrypoint round-trip incl. first-seen-wins and pre-field transcripts; coverage was mutation-checked). Companion flight-recorder tickets: NAS-1474 (worktrees), NAS-1475 (entrypoint).
…uster 748 of 1,839 default-visible sessions are headless SDK runs with no Claude parent in the corpus; as flat siblings they drowned the interactive sessions inside busy project folders. Product decision: keep them visible, but grouped. Each project folder now renders its interactive sessions as the spine, followed by one collapsed "Automated runs (N)" sub-header holding the headless rows (same row markup, source badges intact, deeper indent). The project meta line reads "N sessions · M automated" with N counting interactive sessions only. Cluster expand state is per-project and independent of the project toggle; search/filter clears it. Subagent children follow their parent's stream so a headless parent's children stay attached to it; interactive rendering is unchanged. Behaviorally verified by driving the real renderConversations under happy-dom (collapsed default, counts, toggle isolation, badge survival, click-through) and visually against the production store.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR A of the flight-recorder port series (NAS-1467), plus two session-classification features found while dogfooding it (NAS-1473). Everything is parse-time and rides one CONTENT_VERSION bump (4 to 5), so existing stores re-index once.
Parse-time enrichment (flight-recorder ports)
src/analytics/core/ToolTaxonomy.js):toolKind()buckets every tool into shell / file_edit / file_read / search / task / web / mcp / other and parses the MCP server name frommcp__<server>__<tool>.tool_usagegainstool_kind,mcp_server, anderror_count; errors correlatetool_result.is_errorto the call viatool_use_id, never array order.fileChangeFromTool()turns Write/Edit/MultiEdit/NotebookEdit inputs intofile_changesrows (path, change kind, added/removed lines), indexed by path, so "which conversations touched this file" is a first-class query.src_lineprovenance: every per-message record carries its 1-based transcript line. Recorded at parse time because it cannot be backfilled.GET /api/analytics/toolsandGET /api/analytics/file-changes?path=(input-validated).Agent-worktree session classification
Agent sessions run in isolated git worktrees (Agent tool worktree isolation, EnterWorktree, Cyrus) get transcript directories keyed by the worktree cwd and carry no parent linkage, so each one masqueraded as its own top-level project: 45 of the projects directories on the reference machine were worktree checkouts.
WorktreeClassifierrecognizes the three cwd conventions, marks the sessions as subagents (default lists, search, and canonical rollups already exclude those), adds a distinctis_worktree_agentflag, and re-attributesprojectto the owning repo. The project-name resolver uses the same classifier so it cannot reinstate the fake names. 222 sessions reclassified on the reference store; the five roger-* fake projects collapsed intoroger-marx-collection.Headless-session classification + source badges
Claude Code message records carry an explicit
entrypoint:cliandclaude-desktopare interactive;sdk-cli(claude -p/ SDK CLI, including braintrust peer consults) andsdk-py(Python SDK) are headless. The indexer stores it, rows expose the raw value plus derivedisHeadless, and session list rows now render a fixed-allowlist source badge ("headless", "desktop", "worktree agent"; interactive CLI gets no badge). Headless sessions stay visible: the distinction is labeling, not hiding. MCP search results carryisSubagent/isWorktreeAgent/isHeadlessso API consumers can organize by source.Dogfood findings fixed in this PR
toolUsageplaceholder and the analytics route echoed it. The route now reads thetool_usagetable. Regression test added.this.escapeHtmlinsidepopulateAnalyticsModal, a standalone function. It had never executed with a non-empty breakdown because of the placeholder bug above. One sharedescapeHtmlText()now backs both call sites.is_worktree_agent,entrypoint), which would have pinned the flags to false everywhere except single-conversation fetches. Caught by round-trip tests; all four projections carry them.Verification
--jobs 1) over the changed features: zero findings against this diff; raised findings are pre-existing and tracked for PR B/backlog.Not in scope
Pre-existing clawpatch findings (ConversationAnalyzer first-tool_use-only correlation, SessionAnalyzer token schema mismatch, ProcessDetector path bug, notification lifecycle) tracked for PR B/backlog. UI filter for headless sessions lands with the PR C layout pass. Companion flight-recorder tickets: NAS-1474 (worktrees), NAS-1475 (entrypoint).