Feature: Daemon Mode for Incremental File Watching and Automatic Reindexing

[marco0560]

Feature: Daemon Mode for Incremental File Watching and Automatic Reindexing

Status

• Type: feature / architecture extension
• Priority: medium (developer UX, non-core)
• Version target: > 2.0.0
• Design status: needs-design
• Depends on: Decouple indexer from SQLite and make backend layer truly plugin-agnostic #8 (backend decoupling), Enable immediate support for multiple production-grade backends #10 (multi-backend support)
• Related to: Design richer symbol modeling for overload metadata and named declarations #21 (Vector Database Backend)

Summary

Introduce an optional daemon mode for Codira that:

• monitors repository files for changes
• performs incremental reindexing automatically
• optionally updates embeddings (if vector backend is enabled)

This mode is intended as a developer productivity feature, not part of the deterministic core.

Motivation

Current workflow:

edit files → manually run `codira index`

This introduces friction:

• stale indexes between edits
• repeated manual commands
• suboptimal feedback loop

Daemon mode enables:

edit files → automatic reindex → immediate query freshness

Key Principle

Daemon mode MUST NOT compromise Codira’s deterministic guarantees.

Scope

Included

• filesystem monitoring (watch mode)
• incremental indexing on change
• batching / debouncing
• optional embedding updates
• status inspection

Excluded

• mandatory background operation
• replacing explicit indexing
• introducing heuristic behavior

---

Architectural Position

Mode A — CLI (authoritative)
    codira index
    codira ctx

Mode B — Daemon (optional)
    codira daemon

Hard Constraints

C1 — Optionality

Codira MUST function fully without daemon mode.

C2 — Deterministic isolation

• CLI commands MUST NOT depend on daemon state
• daemon updates MUST be observable and reproducible

C3 — Explicit state visibility

Users MUST be able to inspect:

• last indexed state
• pending changes
• daemon activity

C4 — No hidden mutation

All daemon actions MUST be:

• logged
• traceable
• reconstructable

High-Level Design

File watching pipeline

file change
   ↓
event queue
   ↓
debounce / batch
   ↓
incremental index
   ↓
(update deterministic backend)
   ↓
(optional) update embeddings (#21)

Backend interaction

• deterministic backend:
  • updated synchronously
• vector backend (Design richer symbol modeling for overload metadata and named declarations #21):
  • updated asynchronously (optional)

Relationship with Vector Backend (#21)

Daemon mode is a natural companion to #21:

• embeddings are expensive to compute
• background updates improve usability
• async pipeline becomes meaningful

However:

daemon mode MUST NOT require a vector backend

Relationship with Docker

• daemon alone → low Docker value
• daemon + vector backend → high Docker value (multi-service setup)

Open Design Questions

1. Watch scope

• entire repository?
• only indexed paths?
• configurable include/exclude?

2. Event model

• file-level vs directory-level events?
• how to handle:
  • renames
  • deletes
  • mass changes (git checkout)?

3. Debounce strategy

• fixed delay?
• adaptive batching?
• max batch size?

4. Consistency model

• what happens if files change during indexing?
• snapshot-at-start vs rolling update?

5. Identity and correctness

• how to ensure index consistency with:
  • file hashes
  • analyzer versions
• how to detect drift?

6. Embedding pipeline (#21)

• synchronous vs asynchronous?
• queue-based?
• failure handling?

7. Status and observability

Define:

codira status

Must include:

• last indexed revision / hash
• pending changes
• daemon running state
• embedding sync status (if applicable)

8. Failure handling

• partial indexing failures
• backend errors
• watcher failures

Policy:

• fail-fast vs retry?
• degradation behavior?

9. CLI surface

Proposed commands:

codira daemon start
codira daemon stop
codira daemon status

Open questions:

• foreground vs background mode?
• PID management?
• integration with system services?

10. Logging

• structured logs required
• reproducible trace of actions
• integration with --explain?

11. Cross-platform behavior

• Linux (inotify)
• macOS (FSEvents)
• Windows (ReadDirectoryChangesW)

Questions:

• abstraction layer?
• fallback polling?

12. Interaction with manual indexing

• what happens when:
  • user runs codira index while daemon is active?
• locking strategy?
• override semantics?

13. Performance limits

• large repos
• rapid change bursts
• editor save patterns

Risks

R1 — Hidden state

Users may not know what is indexed.

R2 — Non-reproducibility

Time-based updates break deterministic workflows.

R3 — Race conditions

Concurrent changes during indexing.

R4 — Debug complexity

Harder to reproduce bugs compared to explicit CLI runs.

Mitigations

• explicit status command
• strict logging
• deterministic identity model
• optional disablement
• manual override (codira index --force)

Suggested Implementation Plan

Phase 1 — Design

• define watcher abstraction
• define consistency model
• define CLI surface

Phase 2 — Minimal implementation

• basic file watching
• debounced incremental indexing
• no embedding integration

Phase 3 — Observability

• status command
• logging
• diagnostics

Phase 4 — Integration with #21

• async embedding updates
• queue-based pipeline

Phase 5 — Hardening

• cross-platform validation
• stress testing
• failure scenarios

Acceptance Criteria

• daemon mode is fully optional
• CLI mode remains deterministic and unchanged
• file changes trigger correct incremental indexing
• system remains consistent under rapid changes
• status and logs provide full visibility
• daemon can be stopped/restarted without corruption
• works without vector backend
• integrates cleanly with vector backend (Design richer symbol modeling for overload metadata and named declarations #21)

Key Principle

Daemon mode is a usability layer, not a correctness mechanism.

[marco0560]

Additional design notes from the measurement campaign and hotspot analysis.

Summary judgment:

• daemon mode is worth exploring, but in a separate feature branch
• expected implementation cost is medium to high
• expected gain is high for interactive command latency and repeated query sessions
• expected gain is limited for the main cold-index bottleneck unless the daemon also changes embedding execution strategy

Why it is worth a branch:

1. The current measurements show a large repeated fixed-cost component on interactive commands:
   • plugin discovery / registry snapshot
   • Python process startup
   • torch / sentence-transformers import and model initialization
2. A long-lived process can keep that state warm.
3. This helps most for:
   • ctx
   • emb
   • plugins
   • caps
   • repeated incremental index operations
4. It does not directly solve the biggest cold-index cost center, which is embedding inference itself.

The main limits of daemon mode from the current data:

• it will not by itself remove the dominant cold-index hotspot in embed_texts() and downstream torch forward passes
• it will not by itself fix _find_references() scanning cost
• it will not by itself fix graph expansion cost
• it will not by itself fix JSON-heavy supports_path() admission cost

So the daemon should be treated as a usability and warm-state architecture feature, not as the first-line fix for all measured performance issues.

Recommended branch scope:

• make it an opt-in feature branch
• start with query-oriented warm-state wins first
• keep CLI mode authoritative and unchanged

Recommended architecture direction:

1. Keep explicit CLI mode authoritative.
2. Add an optional daemon process as a separate operational mode.
3. Use the updated in-memory backend as the daemon’s hot working state.
4. Support optional hydration at startup from a persistent backend snapshot.
5. Support optional flush/dump at shutdown back to a persistent backend (SQLite first; other backends later if useful).
6. Keep repository-root and output-dir isolation explicit so one daemon instance cannot silently cross-contaminate another repository.

Why the in-memory backend matters:

• it is the natural place to hold a warm index for low-latency read/query commands
• it can reduce repeated backend reopen and reload costs in the daemon path
• it provides a clean boundary between warm operational state and durable persisted state

Suggested persistence model:

• startup:
  • optionally read a durable snapshot into the in-memory backend
• steady state:
  • apply watched-file incremental updates to in-memory state first
• shutdown:
  • optionally dump the in-memory state back to a persistent backend
• recovery:
  • if durable load fails, the daemon must still be able to rebuild deterministically from an explicit index operation

Implementation plan summary:

1. Branch and isolate the daemon experiment.
2. Define the daemon process contract:
   • repository root binding
   • output-dir binding
   • backend mode
   • status surface
3. Integrate the updated in-memory backend as the primary live state for daemon mode.
4. Add optional startup hydration from SQLite.
5. Add optional shutdown dump back to SQLite.
6. Restrict first functional scope to:
   • emb
   • ctx
   • plugins
   • caps
   • later incremental index refresh
7. Add explicit status and observability:
   • last applied index state
   • pending changes
   • warm backend mode
   • durable snapshot state
8. Keep indexing and daemon coordination explicit with locking and override semantics.
9. Only after that, evaluate whether daemon-managed incremental indexing should become the default recommended developer workflow.

Recommended rollout order:

• v1 branch prototype: warm query daemon only
• v2: add watched incremental index refresh into in-memory state
• v3: add durable load/dump integration
• v4: harden locking, observability, and failure recovery

My recommendation from the current measurements:

• yes, explore daemon mode in a dedicated branch
• no, do not treat it as the first optimization to land on main
• continue first with direct fixes for the measured hot paths in embeddings, reference scanning, graph expansion, plugin discovery, and JSON analyzer admission cost