KnowledgeHarvester is single-PAI-instance — per-project memory dirs (~/.claude/projects/<slug>/memory/) invisible without PAI_DIR override

[nbost130]

Summary

Tools/KnowledgeHarvester.ts is hardcoded single-PAI-instance. KNOWLEDGE_DIR = path.join(PAI_DIR, "MEMORY", "KNOWLEDGE") and WORK_DIR = path.join(PAI_DIR, "MEMORY", "WORK") mean it only sees one tree at a time. Real PAI users accumulate session work across many per-project Claude Code memory directories at ~/.claude/projects/<project-slug>/memory/[MEMORY/]WORK/, and that work is invisible to the central knowledge dashboard.

scanAutoMemory (line 113) reads AUTO_MEMORY_DIR = ~/.claude/projects/-Users-${USER}--claude/memory — but that's only the global home-dir claude session, not the dozens of per-project sessions that contain substantive work.

Repro

1. Use Claude Code from multiple project directories over months (e.g., ~/dev/myproject1, ~/dev/myproject2).
2. Each project accumulates session content at ~/.claude/projects/-Users-myuser-dev-myproject1/memory/MEMORY/WORK/<session>/PRD.md (or ISA.md post-v4.1.0).
3. Run bun KnowledgeHarvester.ts harvest.
4. Result: 0 candidates from your per-project work — only the central ~/.claude/PAI/MEMORY/WORK/ is scanned (and only if it contains ISAs, which NATIVE-mode sessions don't produce).
5. Pulse /knowledge dashboard shows 0/0/0/0 across People/Companies/Ideas/Research despite real work having occurred.

In my case: 40 PRD.md files exist across 7 projects spanning months of substantive work. None are reachable by the harvester.

Workaround (current state)

Per-project run with PAI_DIR override + manual cp bridge:

PROJ=~/.claude/projects/<proj-slug>/memory
# rename PRD → ISA so harvester sees them
/usr/bin/find "$PROJ" -name "PRD.md" -exec bash -c 'mv "$1" "${1%PRD.md}ISA.md"' _ {} \;
# harvest into per-project silo
cd ~/.claude/PAI && PAI_DIR="$PROJ" bun Tools/KnowledgeHarvester.ts harvest
# bridge silo to central
for d in Ideas Companies People Research; do
  [ -d "$PROJ/MEMORY/KNOWLEDGE/$d" ] && \
    mkdir -p ~/.claude/PAI/MEMORY/KNOWLEDGE/$d && \
    /bin/cp "$PROJ/MEMORY/KNOWLEDGE/$d"/*.md ~/.claude/PAI/MEMORY/KNOWLEDGE/$d/ 2>/dev/null
done

Tedious, requires per-project iteration, leaves duplicate KNOWLEDGE silos in each project's memory dir.

Suggested fix

Extend scanWorkISAs (line 145ish) to also discover per-project memory dirs via glob. New behavior:

function scanWorkISAs(state: HarvestState, isBackfill: boolean): HarvestCandidate[] {
  const candidates: HarvestCandidate[] = [];

  // 1. Existing: scan central PAI WORK
  candidates.push(...scanWorkDir(WORK_DIR, state, isBackfill));

  // 2. NEW: scan per-project memory dirs
  const PROJECTS_DIR = path.join(HOME, ".claude", "projects");
  if (fs.existsSync(PROJECTS_DIR)) {
    for (const proj of fs.readdirSync(PROJECTS_DIR)) {
      // Per-project layout has BOTH memory/WORK and memory/MEMORY/WORK historically
      for (const subpath of ["memory/WORK", "memory/MEMORY/WORK"]) {
        const projWorkDir = path.join(PROJECTS_DIR, proj, subpath);
        if (fs.existsSync(projWorkDir)) {
          candidates.push(...scanWorkDir(projWorkDir, state, isBackfill, { sourceProject: proj }));
        }
      }
    }
  }

  return candidates;
}

Where scanWorkDir is the extracted version of the current per-dir loop, and sourceProject becomes a tag/frontmatter field on the resulting harvest candidate so provenance carries.

File-name compatibility

The companion fix to ISASync.hook.ts (commit history shows v4.1.0 added the ISA.md ?? PRD.md read-fallback) should be mirrored in the harvester:

const isaPath = path.join(WORK_DIR, dir, "ISA.md");
const prdPath = path.join(WORK_DIR, dir, "PRD.md");
const filePath = fs.existsSync(isaPath) ? isaPath : (fs.existsSync(prdPath) ? prdPath : null);
if (!filePath) continue;

This avoids forcing every user to mass-rename historical PRDs (most users have months of pre-v4.1.0 sessions).

Verified locally

Patched the workflow manually with PAI_DIR override + bridge cp. End-to-end works: 10 MemberOS PRDs → renamed to ISAs → harvested → 5 entries surfaced into central KNOWLEDGE/. Pulse /knowledge flipped from 0/0/0/0 to 1/1/3/0. Same recipe applies to remaining 30 PRDs across 6 other projects, but the per-project iteration + cp bridge is friction this issue would eliminate.

Related

• Companion issue: KnowledgeHarvester.ts writes directly to domain dirs, bypassing the documented _harvest-queue/ curation step (filed separately).

[hilyfux]

This is a good catch, and I think the per-project aspect is the important part to preserve rather than flatten away.

One implementation pattern that has worked well for Claude Code-style memory is to treat each project memory directory as a separate source lane, then merge at read time with explicit provenance. For this issue that could mean:

1. Discover candidate roots under ~/.claude/projects/*/memory/**, plus the central PAI_DIR/MEMORY/** tree.
2. Normalize each candidate into a record like {sourceProject, sourcePath, kind, mtime, contentHash} before harvesting.
3. Deduplicate by content hash, not just file name, because the same PRD/ISA content can be copied between sessions.
4. Keep the project slug visible in Pulse / dashboard output so central knowledge does not lose where the work came from.
5. Add a PAI_MEMORY_ROOTS override for people who keep Claude workspaces outside the default home path.

The main trap is making the harvester “global” but then losing project boundaries. For agent memory, source provenance is almost as important as retrieval itself; otherwise old context from one repo can quietly contaminate another.

We ran into the same design pressure while building knowledge-graph (https://github.com/hilyfux/knowledge-graph): it keeps memory git-native and zero-dependency, but still has to preserve source/path metadata so Claude Code and Codex can share durable memory without turning everything into one anonymous blob. That might be a useful reference point for the source-lane / provenance side of this fix.

[stratofax]

Heads-up on an overlap: I just filed #1366 for the Linux-platform dimension of the same AUTO_MEMORY_DIR line you reference here.

This issue (#1170) is about scope — per-project memory dirs being invisible — and quotes -Users-${USER}--claude as the current behavior. On Linux, though, that hardcoded macOS slug resolves to a directory that does not exist (-home-${USER}--claude is the real one), so the auto-memory scan silently returns nothing and zero notes graduate.

The two are entangled on the same line: the glob-over-real-projects-dir fix you propose here would incidentally fix the Linux break too, and conversely the cross-platform slug-derivation in #1366 is a prerequisite for your per-project glob to land on the right paths on Linux. Filing separately so the Linux-platform symptom is tracked, but they likely want to be resolved together. Linking for visibility.

[danielmiessler]

Good catch, and the workaround you documented is exactly right for now. The harvester still only scans the central tree in the current release — the memory system has since grown an autonomic capture loop that reduces how much this matters, but per-project discovery is a real gap and it's on the list. Thanks for the detailed repro and proposed patch.