Feature: Auto-reindex when source files change (3 strategies)

[camhoccode]

Pain Point

When files are indexed via ctx_index(path), the index becomes stale silently if the file changes afterward. Users get outdated search results with no warning, leading to incorrect decisions based on old code/data.

Painful scenarios:

• Long coding sessions where source files change frequently
• Config files evolving during refactoring
• git pull/rebase changing many files at once between sessions
• Current 14-day pruning is too coarse — a file indexed 1 hour ago can be completely wrong after a rebase

---

Strategy 1: File Watcher (real-time, mid-session)

Opt-in fs.watch on indexed file paths.

• On ctx_index(path, source) → register path in watch list
• On file change (debounced 2-5s) → auto-reindex that source
• Config: "autoReindex": true | false (default off)
• Ignore patterns: node_modules, dist, .git
• Clean up watchers on session end

Pro: Zero-effort freshness — search always current
Con: Memory overhead from watchers (cap max watch count)

---

Strategy 2: Hash-based Stale Detection (lightweight, at search time)

SHA-256 hash check when returning search results.

• On ctx_index(path) → store file hash + path in sources table
• On ctx_search() → recompute hash for file-based results
• If hash differs → mark as ⚠️ stale or auto-refresh

ALTER TABLE sources ADD COLUMN content_hash TEXT;
ALTER TABLE sources ADD COLUMN file_path TEXT;

Example output:

⚠️ [stale] Title of chunk
   Indexed: 2h ago | File modified: 5m ago

Pro: Near-zero overhead when unchanged (~1-5ms hash per file)
Con: Reactive — discover staleness only at search time

---

Strategy 3: Git-aware Batch Reindex (between sessions)

git diff-driven reindex on session start.

• Store last_reindex_commit SHA in content DB
• On session start → git diff --name-only <last>..HEAD
• Cross-reference with indexed source paths → batch reindex changed files only
• Handle deletes (remove from index) and renames (git diff --name-status)
• Non-git fallback: mtime comparison

ctx_reindex              # reindex git-changed files
ctx_reindex --all        # force reindex everything
ctx_reindex --dry-run    # preview what would reindex

Pro: Smart — only reindexes what changed
Con: Git-only without fallback, doesn't help mid-session

---

Synergy

Strategy	When it helps	Cost
File watcher	Real-time, mid-session	Medium (watchers)
Hash check	On-demand at search time	Low (hash compare)
Git reindex	Between sessions, after pull/rebase	Low (one-time batch)

All three together = complete freshness guarantee. Each can be implemented independently.

Suggested Implementation Order

1. Hash check (smallest change, biggest trust improvement)
2. Git reindex (covers between-session gap)
3. File watcher (full real-time, opt-in)

[mksglu]

Closing as wontfix — the three proposed strategies solve a problem that context-mode's existing architecture already addresses through different mechanisms. Evidence from source:

Why file watchers (Strategy 1) are unnecessary

context-mode is an MCP server — it runs as a child process of the AI coding tool. When the session ends, the process dies. File watchers would need lifecycle management (register on index, clean up on exit, handle max watcher limits) for a process that may live 5 minutes or 5 hours. The complexity is high relative to the benefit.

Why hash-based stale detection (Strategy 2) is already covered

Same-label dedup (store.ts:449-458): When ctx_index(path, source) is called with a label that already exists, the store atomically deletes all old chunks and re-indexes:

DELETE FROM chunks WHERE source_id IN (SELECT id FROM sources WHERE label = ?)
DELETE FROM sources WHERE label = ?

The LLM naturally re-indexes files it's working with. The stale scenario requires the LLM to search content it indexed hours ago without ever re-reading the source — rare in practice because LLMs re-read files they actively work on.

Why git-aware batch reindex (Strategy 3) is unnecessary

ctx_purge already exists as a nuclear option. For git pull/rebase, the user can run ctx purge or simply start a new session (which gets a fresh DB). Adding git diff integration introduces a dependency on git being available, the project being a git repo, and the indexed content being git-tracked — none of which are guaranteed.

What already exists

1. 24h TTL cache on ctx_fetch_and_index with force: true bypass (server.ts:1427-1448)
2. Same-label atomic dedup on every ctx_index call (store.ts:449-458)
3. 14-day auto-pruning of old content DBs (server.ts:191, store.ts:667-673)
4. ctx_purge tool for full knowledge base reset
5. Per-project isolated DBs via hashProjectDir() — no cross-project staleness

When we'd revisit

If we see real-world user reports of "ctx_search returned stale results that caused a bug" — not theoretical scenarios. The current architecture optimizes for the common case (LLM indexes → searches → re-indexes as needed) rather than the edge case (hours-old index searched without re-read).

Thanks @camhoccode for the thorough analysis and the three-strategy breakdown — it was well-researched. We'll keep this as reference if the need arises.

[mksglu]

Strategy 2 (hash-based stale detection) is implemented on next (472634a) — with an agentic twist: instead of flagging stale results, search auto-refreshes changed files transparently.

What shipped

• file_path + content_hash (SHA-256) columns in sources table
• searchWithFallback() runs #refreshStaleSources() before every search
• mtime fast-gate → hash compare → auto re-index changed files
• ctx_search response includes "Auto-refreshed N stale sources" note
• MCP tool descriptions updated

What didn't ship (and why)

• Strategy 1 (file watchers): Unnecessary — auto-refresh on search is simpler, no lifecycle management, no memory overhead
• Strategy 3 (git-aware reindex): Unnecessary — auto-refresh catches all file changes regardless of source (git, editor, build tool)

The auto-refresh approach is strictly better than all three strategies because it's zero-config, zero-overhead for unchanged files, and fully agentic.

4 TDD tests, 111 store tests pass. Thanks @camhoccode for the thorough analysis — Strategy 2 was the right call.