fix: queue-based working memory capture (#179) (#180)

## Summary

- Replaces broken transcript extraction with queue-based turn capture
from hook inputs
- UserPromptSubmit (`preamble`) captures user prompts to
`.memory/.pending-turns.jsonl`
- Stop hook captures `assistant_message` (on `end_turn` only) to same
queue, decouples throttle from capture
- Background updater uses `mv`-based atomic handoff with crash recovery
via `.pending-turns.processing`

## What was wrong

The previous implementation extracted user/assistant messages from
session transcript JSONL, but most entries are `tool_result`/`tool_use`
with no text content. In a typical session (89 "user" messages, only 8
with actual text), the `tail -3` windowing almost always missed real
content.

## Design

Hook inputs already provide both data points directly — no transcript
needed:
- **UserPromptSubmit** → `prompt` (user's text)  
- **Stop** → `assistant_message` (full response) + `stop_reason`

Queue accumulates turns; throttle only gates background processing (not
capture). 10-turn batch cap, overflow safety, crash recovery.

## Test plan

- [x] 5 new queue behavior tests in `shell-hooks.test.ts`
- [x] 618/618 tests passing
- [x] `bash -n` syntax validation on all 3 modified hooks
- [x] 16 acceptance scenarios (stop_reason filtering, dual format,
capture before throttle, DEVFLOW_BG_UPDATER guard, crash recovery,
missing .memory/)
- [x] Evaluator: 5/5 plan requirements aligned

Closes #179

---------

Co-authored-by: Dean Sharon <deanshrn@gmain.com>
Co-authored-by: Claude <noreply@anthropic.com>

Author: 3 people

CLAUDE.md

@@ -38,7 +38,7 @@ Commands with Teams Variant ship as `{name}.md` (parallel subagents) and `{name}
 
 **Build-time asset distribution**: Skills and agents are stored once in `shared/skills/` and `shared/agents/`, then copied to each plugin at build time based on `plugin.json` manifests. This eliminates duplication in git.
 
-**Working Memory**: Three shell-script hooks (`scripts/hooks/`) provide automatic session continuity. Toggleable via `devflow memory --enable/--disable/--status` or `devflow init --memory/--no-memory`. Stop hook → reads last turn from session transcript (`~/.claude/projects/{encoded-cwd}/{session_id}.jsonl`), spawns background `claude -p --model haiku` to update `.memory/WORKING-MEMORY.md` with structured sections (`## Now`, `## Progress`, `## Decisions`, `## Modified Files`, `## Context`, `## Session Log`; throttled: skips if triggered <2min ago; concurrent sessions serialize via mkdir-based lock). SessionStart hook → injects previous memory + git state as `additionalContext` on `/clear`, startup, or compact (warns if >1h stale; injects pre-compact memory snapshot when compaction happened mid-session). PreCompact hook → saves git state + WORKING-MEMORY.md snapshot + bootstraps minimal WORKING-MEMORY.md if none exists. Zero-ceremony context preservation.
+**Working Memory**: Four shell-script hooks (`scripts/hooks/`) provide automatic session continuity. Toggleable via `devflow memory --enable/--disable/--status` or `devflow init --memory/--no-memory`. UserPromptSubmit (`prompt-capture-memory`) captures user prompt to `.memory/.pending-turns.jsonl` queue. Stop hook captures `assistant_message` (on `end_turn` only) to same queue, then spawns throttled background `claude -p --model haiku` updater (skips if triggered <2min ago; concurrent sessions serialize via mkdir-based lock). Background updater uses `mv`-based atomic handoff to process all pending turns in batch (capped at 10 most recent), with crash recovery via `.pending-turns.processing` file. Updates `.memory/WORKING-MEMORY.md` with structured sections (`## Now`, `## Progress`, `## Decisions`, `## Modified Files`, `## Context`, `## Session Log`). SessionStart hook → injects previous memory + git state as `additionalContext` on `/clear`, startup, or compact (warns if >1h stale; injects pre-compact memory snapshot when compaction happened mid-session). PreCompact hook → saves git state + WORKING-MEMORY.md snapshot + bootstraps minimal WORKING-MEMORY.md if none exists. Disabling memory removes all four hooks. Use `devflow memory --clear` to clean up pending queue files across projects. Zero-ceremony context preservation.
 
 **Ambient Mode**: Three-layer architecture for always-on intent classification. SessionStart hook (`session-start-classification`) reads lean classification rules (`~/.claude/skills/devflow:router/references/classification-rules.md`, ~30 lines) and injects as `additionalContext` — once per session, deterministic, zero model overhead. UserPromptSubmit hook (`preamble`) injects a one-sentence prompt per message triggering classification + router loading via Skill tool. Router SKILL.md is a pure skill lookup table (~50 lines) loaded on-demand only for GUIDED/ORCHESTRATED depth — maps intent×depth to domain and orchestration skills. Toggleable via `devflow ambient --enable/--disable/--status` or `devflow init`.
 
@@ -57,7 +57,7 @@ devflow/
 ├── plugins/devflow-*/      # 17 plugins (8 core + 9 optional language/ecosystem)
 ├── docs/reference/         # Detailed reference documentation
 ├── scripts/                # Helper scripts (statusline, docs-helpers)
-│   └── hooks/              # Working Memory + ambient + learning hooks (stop, session-start-memory, session-start-classification, pre-compact, preamble, session-end-learning, stop-update-learning [deprecated], background-learning)
+│   └── hooks/              # Working Memory + ambient + learning hooks (prompt-capture-memory, stop-update-memory, background-memory-update, session-start-memory, session-start-classification, pre-compact-memory, preamble, session-end-learning, stop-update-learning [deprecated], background-learning, get-mtime)
 ├── src/cli/                # TypeScript CLI (init, list, uninstall, ambient, learn, flags)
 ├── .claude-plugin/         # Marketplace registry
 ├── .docs/                  # Project docs (reviews, design) — per-project
@@ -105,14 +105,16 @@ Working memory files live in a dedicated `.memory/` directory:
 
 ```
 .memory/
-├── WORKING-MEMORY.md         # Auto-maintained by Stop hook (overwritten each session)
+├── WORKING-MEMORY.md         # Auto-maintained by background updater (queue-based, updated in batch)
 ├── backup.json               # Pre-compact git state snapshot
 ├── learning-log.jsonl        # Learning observations (JSONL, one entry per line)
 ├── learning.json             # Project-level learning config (max runs, throttle, model, debug — no enabled field)
 ├── .learning-runs-today      # Daily run counter (date + count)
 ├── .learning-session-count   # Session IDs pending batch (one per line)
 ├── .learning-batch-ids       # Session IDs for current batch run
 ├── .learning-notified-at     # New artifact notification marker (epoch timestamp)
+├── .pending-turns.jsonl      # Queue of captured user/assistant turns (JSONL, ephemeral)
+├── .pending-turns.processing  # Atomic handoff during background processing (transient)
 └── knowledge/
     ├── decisions.md           # Architectural decisions (ADR-NNN, append-only)
     └── pitfalls.md            # Known pitfalls (PF-NNN, area-specific gotchas)

docs/reference/file-organization.md

@@ -46,10 +46,13 @@ devflow/
 │       ├── stop-update-memory       # Stop hook: writes WORKING-MEMORY.md
 │       ├── session-start-memory     # SessionStart hook: injects memory + git state
 │       ├── pre-compact-memory       # PreCompact hook: saves git state backup
-│       ├── preamble                # UserPromptSubmit hook: ambient skill injection
+│       ├── prompt-capture-memory   # UserPromptSubmit hook: captures prompts to queue
+│       ├── background-memory-update # Background: queue-based WORKING-MEMORY.md updater
+│       ├── preamble                # UserPromptSubmit hook: ambient skill injection (zero file I/O)
 │       ├── session-end-learning      # SessionEnd hook: batched learning trigger
 │       ├── stop-update-learning     # Stop hook: deprecated stub (upgrade via devflow learn)
 │       ├── background-learning      # Background: pattern detection via Sonnet
+│       ├── get-mtime                # Shared helper: portable mtime (BSD/GNU stat)
 │       ├── json-helper.cjs           # Node.js jq-equivalent operations
 │       └── json-parse               # Shell wrapper: jq with node fallback
 └── src/
@@ -144,7 +147,7 @@ Skills and agents are **not duplicated** in git. Instead:
 
 Included settings:
 - `statusLine` - Configurable HUD with presets (replaces legacy statusline.sh)
-- `hooks` - Working Memory hooks (Stop, SessionStart, PreCompact) + Learning Stop hook
+- `hooks` - Working Memory hooks (UserPromptSubmit, Stop, SessionStart, PreCompact) + Learning Stop hook
 - `env.ENABLE_TOOL_SEARCH` - Deferred MCP tool loading (~85% token savings)
 - `env.ENABLE_LSP_TOOL` - Language Server Protocol support
 - `env.CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS` - Agent Teams for peer-to-peer collaboration
@@ -153,17 +156,21 @@ Included settings:
 
 ## Working Memory Hooks
 
-Three hooks in `scripts/hooks/` provide automatic session continuity. Toggleable via `devflow memory --enable/--disable/--status` or `devflow init --memory/--no-memory`.
+Four hooks in `scripts/hooks/` provide automatic session continuity. Toggleable via `devflow memory --enable/--disable/--status` or `devflow init --memory/--no-memory`.
 
-A fourth hook (`session-end-learning`) provides self-learning. Toggleable via `devflow learn --enable/--disable/--status` or `devflow init --learn/--no-learn`:
+A fifth hook (`session-end-learning`) provides self-learning. Toggleable via `devflow learn --enable/--disable/--status` or `devflow init --learn/--no-learn`:
 
 | Hook | Event | File | Purpose |
 |------|-------|------|---------|
-| `stop-update-memory` | Stop | `.memory/WORKING-MEMORY.md` | Throttled (skips if <2min fresh). Slim instruction after first write. |
+| `prompt-capture-memory` | UserPromptSubmit | `.memory/.pending-turns.jsonl` | Captures user prompts to queue. Zero classification overhead. |
+| `stop-update-memory` | Stop | `.memory/.pending-turns.jsonl` | Captures assistant turns to queue. Throttled (skips if <2min fresh). Spawns background updater. |
+| `background-memory-update` | (background) | `.memory/WORKING-MEMORY.md` | Queue-based updater spawned by stop-update-memory. Reads queued turns + git state, writes WORKING-MEMORY.md via `claude -p --model haiku`. |
 | `session-start-memory` | SessionStart | reads WORKING-MEMORY.md | Injects previous memory + git state as `additionalContext`. Warns if >1h stale. Injects pre-compact snapshot when compaction occurred mid-session. |
 | `pre-compact-memory` | PreCompact | `.memory/backup.json` | Saves git state + WORKING-MEMORY.md snapshot. Bootstraps minimal WORKING-MEMORY.md if none exists. |
 
-**Flow**: Session ends → Stop hook checks throttle (skips if <2min fresh) → spawns background updater → background updater reads session transcript + git state → fresh `claude -p --model haiku` writes WORKING-MEMORY.md. On `/clear` or new session → SessionStart injects memory as `additionalContext` (system context, not user-visible) with staleness warning if >1h old.
+**Flow**: User sends prompt → UserPromptSubmit hook (prompt-capture-memory) appends user turn to `.memory/.pending-turns.jsonl`. Session ends → Stop hook appends assistant turn to queue, checks throttle (skips if <2min fresh), spawns background updater → background updater reads queued turns + git state → fresh `claude -p --model haiku` writes WORKING-MEMORY.md. On `/clear` or new session → SessionStart injects memory as `additionalContext` (system context, not user-visible) with staleness warning if >1h old.
+
+`devflow memory --disable` removes all four hooks. Use `devflow memory --clear` to clean up pending queue files (`.pending-turns.jsonl`, `.pending-turns.processing`) across all projects.
 
 Hooks auto-create `.memory/` on first run — no manual setup needed per project.