feat(search): per-message role/tool granularity (Phase 5) - #14
Conversation
…able Found via real stdio dogfood: reading a conversation resource when the JSONL file isn't accessible (e.g. server running without ~/.claude mounted) returned a misleading 'Conversation not found'. Distinguish the two cases and document the transcript-access requirement. Adds a regression test.
Adds a message_fts table (one row per message, tagged role/tool_name/seq) alongside conversation_fts, so search can filter to a message role (user/assistant/system/tool) or a specific tool. Default search still uses conversation_fts (no blackout); role/tool search uses message_fts, which the background reindex (user_version 3) populates. - Indexer emits per-message records (text/thinking under the message role; tool_use/tool_result as 'tool' rows with tool_name). - DatabaseManager.searchConversationsByRole (best message per conversation, role/tool filters); message_fts cleaned on removeFile; roles facet enum. - SearchService routes role/tool queries to the per-message path. - REST /api/search + GET /api/facets, MCP search_conversations, and the web UI all expose role (and tool via API/MCP). - Tests: DB role/tool search, REST role e2e, MCP role filter, facets roles. Cost: larger index (message_fts duplicates content); flagged as the tradeoff.
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
🟡 removeConversation() does not delete from message_fts, leaving orphaned rows
removeFile() at src/analytics/data/DatabaseManager.js:809 was correctly updated to delete from the new message_fts table, but the parallel removeConversation() method at lines 781-788 was not. When a conversation is removed via removeConversation(), its rows in message_fts are left behind as orphans. While the JOIN in searchConversationsByRole prevents these orphaned rows from appearing in search results, they still leak storage and violate the cleanup contract that removeFile correctly implements.
(Refers to lines 782-786)
Was this helpful? React with 👍 or 👎 to provide feedback.
removeFile already cleaned message_fts; removeConversation didn't, leaving orphan rows. Mirror the cleanup and add a regression test.
|
Fixed: |
1 participant
Phase 5: per-message role granularity
Adds the ability to filter search to a message role (user / assistant / system / tool) or a specific tool — the piece deferred in Phase 2. This is also the foundation for the conversation "state" model behind the observability direction.
How it works
message_ftstable: one row per message, taggedrole/tool_name/seq, alongside the existingconversation_fts. Default search stays onconversation_fts(so there's no search blackout on upgrade); role/tool search usesmessage_fts, which the background reindex (user_version→ 3) populates.tool_use/tool_resultas atoolrow withtool_name).DatabaseManager.searchConversationsByRolereturns the best-matching message per conversation withrole/toolfilters (flat FTS MATCH + JS dedupe — bm25 can't run inside a windowed subquery).message_ftsis cleaned onremoveFile;rolesadded to the facets enum.SearchServiceroutes role/tool queries to the per-message path./api/search+/api/facets), MCP (search_conversationsrole/toolparams,list_facets.roles), and the Web UI (Message-role dropdown).Dogfood (live, against the real corpus, mid-reindex)
Role filtering discriminates correctly across all four surfaces for
cleanupPeriodDays:facets.rolespresent[user,assistant,system,tool], request path user=2/assistant=6/tool=36/none=90message_ftsbuilt in the backgroundAlso includes a dogfood-found MCP fix: reading a conversation resource whose transcript file is unreadable now returns a clear "Transcript file unavailable" error (not "Conversation not found"), with a docs note on transcript access.
Tests: 329 passing (DB role/tool search, REST role e2e, MCP role filter, facets roles, transcript-error case).
Tradeoff
message_ftsduplicates message content, so the index roughly doubles. Accepted to keep default search blackout-free; a future option is to consolidate onto per-message FTS as the single store.