Skip to content

Search finds the note you can't name — and the model ships inside olai - #165

Draft
srid wants to merge 5 commits into
masterfrom
semantic-recall
Draft

Search finds the note you can't name — and the model ships inside olai#165
srid wants to merge 5 commits into
masterfrom
semantic-recall

Conversation

@srid

@srid srid commented Aug 14, 2026

Copy link
Copy Markdown
Member

Search finds a note you can't name — and the model that does it ships inside olai.

Type "protecting young plants from snails" and you get the slugs got the seedlings last year. Not one of those words is in that note.

This is the half of the search roadmap item that was parked, coming back on the exact terms the parking verdict set.

Why it was parked, and what changed

It was built once, inside PR #149, and taken out again before that PR landed. The reason was not the feature — it was the dependency. It embedded through a running Ollama, which nobody's machine has and which HACKING.md forbids outright: olai continues to require NO dependencies outside of Nix itself. The verdict parked it rather than killing it and named the condition for its return: the embedder and the model nix-native — in olai's own closure, reproducible, nothing fetched at run time, no daemon anybody is expected to install.

That condition is met. nix run github:juspay/olai -- web notes/ gets you search-by-meaning with nothing to install, nothing to download at run time, and nothing sent anywhere.

GO/NO-GO was part of the job here, and this is a GO on measured numbers, not on enthusiasm. The full design pass — candidates surveyed, what was rejected and why, every number below — is docs/brainstorming/semantic-recall.md.

What you actually see

Exact matches come first, always. They're evidence: the words are in the node and you can check. Semantic hits only fill what's left of the answer — a query whose exact matches already fill it never asks the index at all — and a hit that arrived that way wears in front of where it lives, because the difference between "this matched" and "this resembles" is yours to have rather than to guess at.

With recall off, search is byte-for-byte what it was. That's a test, not a claim.

The numbers

Measured on x86_64-linux, 16 cores, idle. Every one of these is reproducible from the design doc.

Closure

on disk
olai today 1,095 MB
+ pkgs.llama-cpp +159 MB
+ bge-small-en-v1.5 q8_0 weights +37 MB
after 1,291 MB — +196 MB, +17.9 %

There's a cheaper llama-cpp — blasSupport = false drops BLAS/OpenBLAS/gfortran and leaves +41 MB instead of +159. It is deliberately not taken: that derivation is in no binary cache, so every user and every CI lane would compile llama.cpp locally. Timed: 3 m 27 s on 16 cores. Trading 118 MB of disk for three and a half minutes of build on every fresh machine is the wrong way round for a feature that should arrive without ceremony. It's a one-word lever in nix/embed.nix if that ever inverts.

Speed, CPU only

measured
embedder cold start (spawn → /health) 68–98 ms (~490 ms first ever, weights not in page cache)
query embed, warm, n=100 p50 3.6 ms · p95 4.4 ms · max 6.5 ms
scan 148 vectors 0.40 ms
resident memory, shortly after start 30 – 66 MB
…after indexing 148 essay-length notes ~158 MB

The palette debounces at 200 ms; the whole semantic leg is under 5 ms of that.

Index

Corpus: this repo's own docs/roadmap.jsonl — 148 nodes, 188 KB of prose.

measured
full cold build 4.48 s (30 ms/node)
titles only 0.32 s (2.2 ms/node)
on disk 222 KiB

Paid once, behind the boot, incrementally after — an unchanged node is never embedded twice, across serves.

A correction I owe, because it's the difference between an honest number and a scary one. My first pass at these timings was taken while a 16-core llama.cpp compile was saturating the same box, and reported 490 ms cold start, 10 ms p50 and a 34.8 s index build. Those were contention numbers, not the feature's. They were re-taken on an idle machine and are the tables above; the compile that polluted them is the 3 m 27 s figure in the closure section.

Quality, and the floor that had to move

Q: "the first page load is too heavy"          substring: 0 hits
  ≈ 0.685  load-perf        Roadmap initial load feels very slow
Q: "how do I undo something"                   substring: 0 hits
  ≈ 0.724  undo             Undo
Q: "talking to an AI inside the app"           substring: 0 hits
  ≈ 0.692  form-elicitation the agent's questions render as forms

The retuning that matters: bge-small's cosine scale is compressed, and the parked version's floor of 0.5 is wrong for it. Four deliberately off-topic queries (a bread recipe, the treaty of Westphalia, changing a tyre, APAC revenue) drew a best score of 0.599 from this corpus; real rank-one hits score 0.685–0.724. So the floor is 0.62 — above every junk ceiling observed, below every genuine top hit. Biased conservative on purpose: dropping a weak true positive costs a row nobody was owed, keeping a strong false positive costs trust in the . Caveat stated rather than buried: it's tuned on one corpus, and the measurement is written beside the constant.

What was kept, and what was consciously revised

Kept from the prior art — none of it was what broke: the index as a derived reading off the store's own change stream; the XDG cache (never the served directory, which commit would sweep into everybody's pending panel and the watcher would wake on); the required embedder seam (Query.Recall, an interface ops declares and implements nowhere, which cannot fail by type); exact-first ranking; the marker; and the byte-for-byte degradation pin.

Revised, and each one is a consequence of the embedder no longer being somebody else's daemon:

  1. Detection is a path, not a probe. The wrapper bakes two store paths in — OLAI_EMBED_SERVER, OLAI_EMBED_MODEL — exactly as it already bakes OLAI_ACP_AGENT. Nothing asks the network whether a model server happens to exist. Unset (a bun run out of a source checkout) is the same null the old probe returned, and means the same thing.
  2. olai owns the process, and says so. This llama.cpp ships no batch embedding binary, so embedding is an HTTP call to a server olai spawns itself: lazily (a serve whose index is warm and whose reader never searches by meaning never starts one), on a unix socket in the runtime dir (no port, nothing the network can reach), killed with the scope — and swept at the next boot if a kill -9 left one behind, which a finalizer cannot cover. Declared in docs/running.md. This is the deliberate inverse of the incident this item carries.
  3. Task prefixes follow the model — BGE's query instruction replaces nomic's, through the same EmbedKind seam.
  4. The floor, above.

Evidence

The shot, the tables and the pin in one place: https://claude.ai/code/artifact/877e4ce5-6240-4751-92a1-e8ec02ec8695

The demo, verbatim from the palette drawn by the packaged binary — query protecting young plants from snails, substring hits zero:

the slugs got the seedlings last year
≈ the cold frames · garden #outdoors

One row, because the floor is doing its job. The degradation pin, sabotage-verified — the equality broken, then restored:

$ # sabotage: with no recall standing, quietly drop the last hit
(fail) searchWith — the semantic merge > PIN: with no recall standing, the answer IS `search`s, exactly
(fail) searchWith — the semantic merge > an answer the exact matches already fill never asks the index
 14 pass
  2 fail

$ # sabotage reverted
 16 pass
  0 fail
Ran 16 tests across 1 file. [78.00ms]

Proof, and where each kind lives

  • Unit — the seam, driven by a deterministic fake; nothing starts a model server. The degradation pin is searchWith over a null recall being deep-equal to search across a spread of queries and limits. OLAI_RECALL=off reaches the same null with an embedder standing right there. Plus the reaping guards: a living olai's embedder is never touched, and the sweep reads only its own leavings.
  • e2efeatures/semantic_recall.feature, the one place in the suite where the real model runs, against the packaged binary, because the claim ("nothing outside Nix") is only checkable there. One scenario finds the slugs note by meaning and asserts the ; the other, with recall off, gets nothing for the same words and the node with no marker for "seedlings". Both pass in 3.5 s.
  • Every other e2e scenario runs OLAI_RECALL=off — the product's own switch, for the product's own reason: hundreds of servers × 66 MB is silly, and none of those scenarios is about recall.

Background processes, declared

Per the rule this item carries, everything this branch started and stopped:

  • benchmark llama-server instances (cold start, latency, index build, floor tuning, the fixture-corpus queries) — each started, measured, and killed in the same shell invocation;
  • one backgrounded nix build --rebuild to time the no-BLAS llama.cpp compile — the 3 m 27 s figure — which exited on its own;
  • two hand-run olai web servers to verify SIGTERM teardown and the SIGKILL sweep, both killed.

Nothing is running now, and the SIGKILL-orphan case was found because of this check rather than left for somebody to find in ps: an e2e run leaked a model server, which is why the sweep and the harness's process-group kill are in this PR at all.

Review round — grok + opencode

Two independent reviews, 0 blocking, 9 nits. The union is addressed in b3721c2 (one commit, because four of them are one story: the code was claiming more than it did). Four became behaviour; five became prose that stopped over-claiming.

Behaviour

total counted hits, not matches (both reviewers). The merge set total = exact.total + <what fitted on screen> while Search.total is documented as the uncapped number of nodes that matched — so one field meant two things depending on which half answered. The index now returns every neighbour above its floor (Recall.nearest loses its limit: a bound belongs where the scale is known, and a count cut off at a display limit is not a count), and the merge counts what could have been shown while drawing what fits. Two new tests: one for the count, one for what is not counted — a ghost id, a placement, a node already answered exactly. The dedup also now states why it is complete rather than approximate: room > 0 is only reachable when the substring pass truncated nothing, so the hits in hand are every exact match.

opencode's separate note — that a limit-capped exact answer displaces hits, pre-existing and deliberate — is recorded on Search.total itself, so the next reader finds it as shape rather than as a regression.

A dead child stayed dead (grok). Effect.cached on the start meant a llama-server that died mid-serve — or one start that failed — turned recall off for the life of the serve, with a single log line. That cannot be what "olai owns this process" means. The memo is invalidated on child exit and on a failed start, behind a 15-second cooldown so a crash loop is not respawned per keystroke; a finalizer marks the shutdown so teardown is never mistaken for a death. Tested without a model server: the fake llama-server is two lines of shell that exit, and the second embed answering "cooldown" rather than replaying the first failure is the proof the memo re-armed.

A comment named a retry that does not exist (grok). A failed embed batch claimed the store's backstop probe would come back for it. It does not — the backstop publishes nothing when the listing is unchanged, so on a corpus nobody edits a transient failure stranded the index until a restart. The batch now retries twice on its own fiber (1 s, 4 s), and the comment says what actually happens when those are spent.

An e2e absence proved less than it claimed (grok). The recall-off scenario read an empty result list two animation frames (~32 ms) after typing — under the 200 ms debounce. It now fills the list with a matching query first, so emptiness is something the server did, and outlives the debounce before reading.

Chasing that finding turned up the harder half, and I could not close it — so the scenario says so instead of implying otherwise: no assertion made through the browser can separate recall OFF from recall on-but-not-yet-indexed. The index fills in behind the boot, and an empty answer is honest in both states. Verified the only way it can be: I tagged the off scenario @recall and watched it still pass, twice, including with a 4 s settle. The byte-for-byte contract is the unit pin, which is sabotage-verified. What the scenario holds shut is the thing only the real product can show.

Numbers and prose

The memory number was wrong, in the direction nobody guessed (opencode). The doc claimed a flat 66 MB. opencode measured ~30 MB and asked me to qualify it downward. Re-measuring found the opposite tail: ~30–66 MB shortly after start, and ~158 MB once a real corpus has gone through it, where it stays — about 54 MB of that the mmap'd weights. I looked for a lever and there is not one: --ctx-size 512 / 2048 / 8192 all measure the same, and --sleep-idle-seconds reclaims none of it. Stated as a range now, in the design doc, docs/running.md and the tests README.

olai mcp starts one too (both reviewers). The process list was written under an olai web framing while mcp/serve.ts opens the same index. docs/running.md names the child on that face, says several can be resident at once (including two over one directory when web and mcp are both up), and says why it is being said at all.

is the screen's spelling (opencode). docs/search.md presented the marker as the consequence of resemblance; it now names both — on the header box, the palette and the (( widget, matched: "meaning" over MCP, no field on the screen and no marker in a tool result.

A warm p50 quoted as a guarantee (opencode). The palette's debounce comment said embedding is "about 4 ms" in a file read by browser-client maintainers. It now says warm-p50-on-an-idle-box, names the ~490 ms first query of a serve, and points at the 3 s budget and substring-first boot as what make the slow case a late paraphrase row rather than a late search.

All of it is folded into docs/brainstorming/semantic-recall.md under What the review changed, so the next reader gets the findings and not just the fixes.

CI

Full pipeline green at b3721c2 on x86_64-linux (kolu-ci lane), first run, 2m19s wall. Master merged (already up to date).

leg
_ci-setup pass 17s
bun-nix-fresh pass 8s
fmt-check pass 7s
hm-module pass 0s
install pass 8s
kolu-deps pass 7s
nix pass 23s
test pass 16s
typecheck pass 16s
e2e pass 1m51s

10 ok · 0 failed · 0 errored{"settled":true,"passed":true,"failed":[],"errored":[]}. macOS skipped per the CI rule. Every earlier sha was green too: 5e734d9, cb004fb.

The nix leg is not just a build — it asserts that the packaged wrapper bakes OLAI_EMBED_SERVER and OLAI_EMBED_MODEL and that both paths exist, which is the "nothing outside Nix" claim as a checked fact rather than a sentence in a doc.

Three refactor commits, isolated

  • 5ac6c36 — architecture-first-principles. One confirmed finding, P5: the similarity floor sat in the index, which cannot see which model is behind the seam. A cosine scale is a fact about a vector space; Embedder grows floor, the test fake declares its own, and the graduation candidate (olai now owns two child processes, each with its own copy of spawn/readiness/teardown/orphan) is named as a recorded opportunity, not extracted.
  • 10fcd7c — hickey + lowy, together. A nix export whose comment claimed to unify a spelling that nothing read; a generic palette assertion living in the recall-specific step file rather than beside its siblings; three bare env-var strings where the repo already has a receptacle for that axis (DIST_ENV_VAR, ALLOWED_ORIGINS_ENV_VAR, AGENT_ENV).
  • cb004fb — /simplify. A parameter that carried two questions, a liveness probe that needed no asking when the socket file is already gone, two spellings of one temp directory. Skips named rather than argued.

Deferred, named

  • Chunking long notes. A note is embedded by its first 512 tokens; a paragraph buried in a long one is findable by the note's opening, not by itself.
  • docs/roadmap.jsonl is untouched — the search node still reads PARKED, and moving it is the human's call on approval.
  • macOS is unverified. The derivation is platform-generic and pkgs.llama-cpp evaluates on aarch64-darwin, but nothing here was built or timed there (Linux-only per the CI rule).
  • Approximate nearest neighbour (a linear scan of 10,000 vectors is ~27 ms; an index structure earns its complexity somewhere past that).

srid added 4 commits August 14, 2026 11:05
The second part of the `search` roadmap item, on the terms its parking
verdict set. The first attempt (PR #149's history, 8932336) embedded
through a running Ollama, which no user's machine has and which
HACKING.md forbids — olai requires NO dependency outside Nix itself. It
was removed rather than hidden, and the condition for its return was
named: the embedder AND the model nix-native, in the closure,
reproducible, nothing fetched at run time, no daemon anybody is expected
to install.

DESIGN FIRST, and the numbers are in docs/brainstorming/semantic-recall.md
rather than asserted here. Measured on x86_64-linux, idle:

  closure 1,095 MB -> 1,291 MB (+196 MB, +17.9%): llama-cpp +159, the
  bge-small-en-v1.5 q8_0 weights +37; cold start 68-98 ms; query embed
  p50 3.6 ms / p95 4.4 ms; full index of this repo's own 148-node
  roadmap 4.48 s; index on disk 222 KiB; the running server 66 MB RSS.

The doc also records what was NOT taken and why: `blasSupport = false`
would cost 118 MB less but is in no binary cache (3m27s of llama.cpp on
every machine); onnxruntime has no packaged Node binding; static word
vectors are the standing cheaper alternative. And a correction — the
first timings were taken while a compile was saturating the box, and are
re-taken here.

KEPT from the prior art, because none of it was what broke: the index as
a derived reading off the store's change stream, the XDG cache (never
the served directory, which the commit tool would sweep and the watcher
would wake on), the required `Query.Recall` seam that cannot fail by
type, exact-first ranking (substring is evidence, similarity is
resemblance, never interleaved), the `≈` marker, and the byte-for-byte
degradation pin.

REVISED, consciously: detection is two store paths the wrapper bakes in
(`OLAI_EMBED_SERVER` / `OLAI_EMBED_MODEL`, exactly as `OLAI_ACP_AGENT`
already is) rather than a probe of the network; olai owns the
`llama-server` child, started lazily, on a unix socket, killed with the
scope, swept at the next boot if a `kill -9` left one behind, and
declared in docs/running.md; BGE's query instruction replaces nomic's
prefixes through the same `EmbedKind` seam; and the similarity floor
moves 0.5 -> 0.62, because bge-small's scale is compressed — junk
queries topped out at 0.599 against this corpus and real rank-one hits
scored 0.685-0.724.

Proved twice over. The unit lane drives a deterministic fake through the
seam and never starts a model server; `features/semantic_recall.feature`
drives the REAL one against the packaged binary, where "protecting young
plants from snails" finds "the slugs got the seedlings last year" — zero
substring hits — and the row wears `≈`. `OLAI_RECALL=off` is the
reader's own switch and what every other e2e scenario runs with.

1534 unit tests pass.
…t a space

Architecture-first-principles pass over the diff. One confirmed finding,
P5 (guarantees at the knowing endpoint): the similarity floor sat in
`recall/recall.ts` as a module constant, and `recall.ts` cannot see which
model is behind the seam — it only ever holds an `Embedder`. A cosine
scale is a property of the VECTOR SPACE: bge-small's is compressed where
nomic's is not, which is why this number had to move from 0.5 to 0.62 in
the first place. An index holding one constant is a layer promising
something it does not know enough to promise, and the second embedder
anybody puts behind the seam would inherit a floor tuned for a different
model with nothing anywhere saying so.

So `Embedder` grows `floor`, the packaged one declares the measured
0.62 beside the measurement, and the index reads `embedder.floor`. The
test fake declares its own (its vectors are concept counts, so its scale
is not bge's either) — which is the point demonstrated rather than
argued.

Also recorded, not built: the graduation half of the boundary question.
olai now owns TWO child processes it started itself, and each carries its
own copy of the same volatility — spawn, wait for readiness, drain
stderr, kill with the scope, decide what a `SIGKILL`ed parent leaves
behind. Population two is when an axis becomes nameable, not when it must
be extracted; the volatility and its wanted home are named in the design
doc and nothing is moved.

Checks run over the diff and its one hop down: C1 ecosystem-duplicate
(nothing hand-rolled that the deps or the runtime already ship — no
vector library is in the set, `process.kill(pid, 0)` is the POSIX ask),
C2 consumer-ergonomics, C3 graduation (above), C4 depends-on, C5
fresh-eyes, C6 state-and-time (this finding), C7 conventions.
…re strings

The hickey and lowy pass over the diff, run together. Three findings, and
each is the same shape from a different side: a concept that had drifted
away from where it belongs.

ONE THING SPLIT IN TWO WITH A COMMENT HOLDING IT (hickey, layer 2).
`nix/embed.nix` exported `serverBin = "bin/llama-server"` claiming to be
"one spelling, because the wrapper and scripts/embedder.sh must not be
free to disagree" — and `scripts/embedder.sh` never read it. It cannot:
it is shell, and the whole reason it exists is to answer before nix is
asked. So the export unified nothing and the comment was false, which is
worse than the duplication it pretended to fix. Collapsed to `server`,
the path the wrapper actually bakes, with the second spelling named for
what it is and the `just nix` assertion pointed at as what catches it.

AN ASSERTION IN THE WRONG HOUSE (hickey, layer 3). `the palette lists no
node at all` is a fact about THE PALETTE, and it had been written into
`recall_steps.ts` because recall happened to need it first. Its siblings
— `lists the node`, `does not offer` — live in `palette_steps.ts`, and a
second home for palette assertions is how two of them start disagreeing
about what a node row is. Moved; `recall_steps.ts` keeps only the retry
step, which really is about recall (the index fills in behind the boot).

A RECEPTACLE THE REPO ALREADY HAD (lowy). Every other env var olai reads
is a named constant in the module that owns the decision —
`clientDist.ts`'s `DIST_ENV_VAR`, `allowedOrigins.ts`'s
`ALLOWED_ORIGINS_ENV_VAR`, `@olai/chat`'s `AGENT_ENV`. Recall had three
bare string literals. Not a new boundary — the axis is already
encapsulated, and a parallel spelling of an encapsulated axis is the
finding. `SERVER_ENV_VAR` / `MODEL_ENV_VAR` on the embedder,
`RECALL_ENV_VAR` on the index (it turns off the INDEX, and is answered
before an embedder is even looked for), and the tests name the symbols
rather than re-typing the strings.

Also stated where it was only implied: `pathsFromEnv` parses both paths
or neither AT THE BOUNDARY, because "where the embedder is" is one fact
and half of it is not a state anything below should be able to hold.

1534 unit tests pass.
…asking

The /simplify pass over the diff — quality, not correctness. Four
angles run; three fixes, and the rest reported clean below.

SIMPLIFICATION. `packagedEmbedder` took `root`, the served directory,
purely to turn it into a socket path on its first line — so a function
that has nothing else to do with the served directory named it in its
signature. It takes the socket now, and `detectPackaged` (which already
owns the environment) computes it.

EFFICIENCY. The boot sweep asked every leftover's socket whether anything
was answering, with a timeout in front of the ask. A leftover whose
socket FILE is already gone has nothing listening by construction; the
probe is now skipped for those, which is the ordinary case after a clean
shutdown that raced its own unlink.

REUSE. `embedder.test.ts` had two spellings of "a temp directory of this
test's own" — one inline, one named. One `scratch(prefix)` now.

Reported clean: no ecosystem duplicate to adopt (no vector library is in
the dependency set, and `process.kill(pid, 0)` is the POSIX ask rather
than a hand-roll); the `≈` marker sits in the one function all three
search doors draw through rather than in each door; the e2e recall switch
is the harness's default for every scenario rather than a special case
bolted onto one; and the orphan sweep runs at every boot rather than only
where the leak was noticed — all four are already at the depth the
altitude angle asks for.

SKIPPED, and named rather than argued: the five `open({root, snapshot,
embedder, cacheDir})` blocks in `recall.test.ts` could share a helper,
but that file is the prior art restored, each test reads as its own
statement of a pin, and collapsing them would trade a reviewed shape for
forty fewer lines. The `llama-server` stderr tail keeps accumulating
after start, when only start reads it — bounded at 2 KB, written to
almost never under `--log-disable`, and the comment already says what it
is for.

1534 unit tests pass.

@srid srid left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review, verified locally at cb004fb (nix dev shell): unit tests 33/33 pass across the recall seam, typecheck green all workspaces, fmt-check green, closure measured at 1,291,439,648 B vs master 1,095,429,752 B (+196 MB, +17.9% — the claimed table is exact). Packaged binary bakes both store paths and both exist. Live probe: olai mcp with the packaged bge-small answers "protecting young plants from snails" with exactly one hit, id: "slugs", matched: "meaning"; served directory stays byte-clean (only garden.jsonl); SIGTERM sweeps the llama-server and the socket cleanly.

  1. [nit] olai mcp also lazily spawns the 66 MB embedder, and I measured 30 MB resident on x86_64-linux — both halves worth a sentence in docs. docs/running.md's process list is introduced under an olai web framing and implies a singular child ("stops when olai stops"), but a machine with olai mcp wired into an agent's MCP config may host several concurrent olai processes — one e2e leak in this PR's own history showed how. Nothing is wrong here (the sweep bounds aggregable orphans by one-per-unclean-boot, and the lazy spawn means a MCP serve that never searches never pays); the doc just under-describes "how many of these can be resident at once" for the one process class that earned its own docs section. Alongside it: measured RSS of the running llama-server on this machine is ~30 MB (VSZ is disk-backed by the mmap'd 37 MB GGUF), under the claimed 66 MB — the design doc's number is honest about which machine it came from, but it will read as inflated to anyone who checks.

  2. [nit] search_nodes returns matched: "meaning" hits with no — the marker is web-only, by design, and the tool description says why — but docs/search.md presents as the reader-facing consequence of resemblance without narrowing to surface. The tool description's prose ("Hits with matched: "meaning" follow … worth checking rather than trusting") is the agent-side equivalent and is correctly there. One clause in docs/search.md ("the palette and the search box wear ; search_nodes says matched: \"meaning\" instead") closes the loop for a reader who goes looking for the marker's absence; today the doc answers "all doors wear " and the tool description has to correct it on its own.

  3. [nit] The palette's 200 ms debounce comment says query embedding is "measured at about 4 ms" — that is the warm, n=100 p50 on an idle 16-core x86_64-linux figure, and the design doc itself records ~490 ms first-ever and 10 ms p50 under contention. The comment in nodes.ts risks reading as a guarantee in a file whose readers are browser-client maintainers. One qualifier ("measured warm at about 4 ms; first-ever is ~490 ms") keeps it honest at its point of use. The 3 s QUERY_TIMEOUT and the index's substring-first boot are the right safety nets either way.

  4. [nit] Pre-existing, not introduced here, but newly load-bearing: Query.search caps hits at limit before searchWith consults recall. I verified locally: 13 title hits at the default limit 12 displace a 14th exact hit ranked below them, and the displaced hit is silently absent from total as well as from hits. This was already true of Query.search alone — total is the pre-cap count and hits is the capped window — but searchWith inherits the shape into a world where a semantic hit can appear above a truncated exact-matching desc/title hit the reader will never see. The design doc's "evidence first" claim is satisfied at query time for the window the reader actually sees, which is what the claim needs; nothing to fix in this PR. Recording it here so the next person who reads total: 14, hits: 12 knows the truncation is pre-existing and deliberate-shaped, not a regression the semantic merge introduced.

Verification notes, for whoever reads this review next: unit 33/33 pass (query pin · recall.test 8/8 · embedder.test 8/8 · place.test); typecheck green across all workspaces; fmt-check green. Closure bytes above are byte-exact to the PR's table. Live end-to-end against the packaged binary: "protecting young plants from snails" → one hit, id slugs, matched: "meaning"; "seedlings" exact-search matches without ≈ as advertised; served dir contains only garden.jsonl at the end; llama-server gone on SIGTERM alongside the socket and owner file; XDG cache file lands under ~/.cache/olai/recall/ only. Byte-for-byte degradation pin: query.test.ts pins deep equality of searchWith(recall: null) and search across a 4×2 spread of queries and limits, and the same pin is sabotage-verified in the PR description — the pin is test-backed and the test is in-tree, not aspirational. The 512-token deferral is acceptable: design doc records it as a named second design, the content hash is computed over the truncated window (so no edit-that-changed-only-the-tail re-embeds), and the first-512-token note is findable by its opening which is how people describe long notes anyway. e2e coverage: semantic_recall.feature is @scratch-gated with retry-until-≈ honest to the indexing lag, recall off via the product's own OLAI_RECALL=off for all other scenarios, and killChild process-group SIGKILL keeps a CI lane from leaking the embedder the way the author says it once did. HACKING.md rule compliance: nothing fetches at runtime; the weights are a fetchurl FOD pinned by revision against a 2024-02-frozen CompendiumLabs repo with an explicit sha256; both baked paths exist in the nix store and resolve to cache.nixos.org-backed llama-cpp stock (no local compile); unix socket only, loopback never bound.

@srid srid left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review of PR #165 against HACKING.md (no dependencies outside Nix), docs/brainstorming/semantic-recall.md, and the full diff. I did not read the parallel review.

Judgment

The Nix-native claim holds. I checked the bits the parking verdict named:

  • Weights are a fetchurl FOD in nix/embed.nix (sha256-7Djo2hQllrqpExJK5QVQ3ihLaRa/WVd+8vDLlmDC9RQ=), pinned by revision, not main. Already realized here as /nix/store/j54zzgjpv7zd8c15zkn2lwzwd8vhcmns-bge-small-en-v1.5-q8_0.gguf (36.8 MB). Nothing in the runtime path talks to HuggingFace.
  • Embedder is stock pkgs.llama-cpp 10273 (/nix/store/vc3bha4vhf0szn55m7bwgssqzgjkdsfv-llama-cpp-10273). nix path-info --store https://cache.nixos.org resolves it — the binary cache claim is real.
  • The wrapper bakes both store paths with --set-default; just nix asserts they exist. Detection is those two paths or null, never a network probe.
  • llama-server --help documents --host binding a UNIX socket when the address ends in .sock (the code’s olai-embed-*.sock matches), and --embedding. No --*-default flag that “can download weights from the internet” is passed.
  • No ambient daemon is expected. A bun checkout without the wrapper is substring-only, by type.

A first nix build on a machine that has never seen this FOD still fetches the 37 MB GGUF from HuggingFace as a Nix build, because neither cache.nixos.org nor cache.nixos.asia/oss has that path. That is still Nix, not a runtime dep. Not a HACKING miss.

GO vs the numbers. +196 MB / +17.9% is a lot of disk for a search feature, but it is the feature that was parked specifically to wait for this closure shape, and the alternative (no-BLAS, +78 MB) is the one that would compile llama.cpp on every fresh machine. Rejecting that lever is right: I confirmed the stock derivation is substitutable; the override would not be. Query cost (~4 ms) sits inside the 200 ms debounce. I would not flip this to NO-GO on the numbers in the design doc.

Degradation pin. The unit pin is real. I ran it: searchWith over recall: null is toEqual search across header / two git / purchase food / git × {undefined, 1}. open() returns null for a missing embedder and for OLAI_RECALL=off with a fake standing. Together that is the contract. The e2e OFF scenario is weaker than the PR body claims — see finding 1.

Exact-first ranking. A semantic hit cannot displace an exact one. searchWith takes substring first, computes room = limit - exact.hits.length, and returns exact unchanged when room <= 0 without calling nearest. Fills are appended, never interleaved, and ids already in exact.hits are skipped. When exact matches exceed the limit, room is 0, so a high-cosine neighbour that also happened to be a lower-ranked exact match cannot sneak back in wearing .

Embedder lifecycle. Spawn is lazy; /health is polled; stderr is drained; the kill is a scope finalizer (SIGTERM, then SIGKILL after 2 s); runMain unwinds that scope on SIGINT/SIGTERM. Orphans from kill -9 are swept at the next boot, with a living parent left alone. I do not see a zombie on the clean path. Mid-serve child death is not recovered — finding 2.

Index freshness. The index is a SubscriptionRef.changes consumer (current value first, then revisions), incremental by content hash. The unit test re-embeds exactly the edited node and does not touch the others. A failed reconcile does not get another look from the store backstop — finding 3.

512-token deferral. Named in the design and in docs/search.md. WINDOW = 1400 chars is a hash-honesty cut, not a second ranker. Acceptable for this pass.

e2e / docs. The ON scenario (@scratch:good @recall, retry until ) is the right place to run the real model against the packaged binary. MCP and web share Query.searchWith over one Reading.recall. docs/search.md, docs/running.md, docs/architecture.md, docs/index.md, the website comparison row, and the package READMEs were updated. README.md is a stub that points at the site; that is enough. olai mcp starting the same child is under-documented — finding 4.

Locally: 33 pass / 0 fail across query.test.ts, recall.test.ts, embedder.test.ts, place.test.ts.

Findings

  1. [nit] packages/tests/step_definitions/palette_steps.ts:172 / packages/web/src/client/search/nodes.ts:46 — The e2e OFF scenario’s “snails → nothing” step is not a settled search. the palette lists no node at all waits two requestAnimationFrames (~32 ms). The palette debounces 200 ms before it asks the server. So the empty assert wins by racing the debounce, not by observing a substring answer. The following seedlings step does wait for a row, and an exact hit never wears , so that half is real — but the scenario would still pass if that shared @corpus:good server had recall on. The byte-for-byte pin is the unit test, not this step. Wait for the search to settle (or poll “still empty after the debounce”) before claiming the packaged OFF path.

  2. [nit] packages/server/src/recall/embedder.ts:332Effect.cached(start) means a child that dies after a successful start, or a start that fails once, is not tried again for the life of the serve. nearest then returns [] forever (logged once). That is the typed degradation, not crash recovery. Clean shutdown and the next-boot sweep are solid; a mid-serve llama-server exit is a restart of olai. If you want the process you own to come back, cached has to give way to a restart on EmbedFailure / child exit.

  3. [nit] packages/server/src/recall/recall.ts:203 — The comment says a failed reconcile is retried by “the next revision (or the store’s backstop probe)”. The store’s backstop does not publish when the listing is unchanged (packages/store/src/store.test.ts: “a probe that finds nothing changed publishes nothing”). A transient embed failure on a static corpus therefore leaves that snapshot un-absorbed until a file changes or the process restarts. Drop the backstop claim, and if boot-time indexing is allowed to fail transiently, retry on this fiber (backoff) rather than waiting for a revision that may never come.

  4. [nit] docs/running.md:46 — “The processes it starts” is written only under olai web. packages/server/src/mcp/serve.ts:168 opens the same index. olai mcp will spawn its own llama-server (and a second one if web is already up on that directory). Name the child on that face too — this PR exists because a previous lane started a model server and did not say so.

  5. [nit] packages/ops/src/query.ts:456 / packages/surface/src/search.ts:51Search.total is documented as uncapped (“twelve of ninety”). The merge sets total: exact.total + filled.length, and filled is already sliced to room. nearest scored every neighbour above the floor and then threw the pre-slice count away. Exact hits still report a true total; paraphrase hits do not. Either count the unique above-floor neighbours, or stop telling agents the field is uncapped.

… not exist

The union of the grok and opencode reviews — nine notes, none blocking,
four of them behaviour.

**`total` counted hits, not matches (both reviewers).** The merge set
`total = exact.total + <what fitted on screen>` while the field is
documented as the uncapped number of nodes that MATCHED, so the
paraphrase half quietly meant something else from the exact half. The
index now answers with every neighbour above its floor — `nearest` loses
its `limit`, because a bound belongs where the scale is known and a count
cut off at a display limit is not a count — and the merge counts what
could have been shown while drawing what fits. The dedup is complete
rather than approximate, and now says why: `room > 0` is only reachable
when the substring pass truncated nothing, so the hits in hand ARE every
exact match and no node is counted twice. Two tests, one for the count
and one for what is NOT counted (a ghost id, a placement, a node already
answered exactly). opencode's separate note — that a limit-capped exact
answer displaces hits, pre-existing and deliberate — is now recorded on
`Search.total` itself so the next reader does not re-find it as a
regression.

**A dead child stayed dead (grok).** `Effect.cached` on the start meant a
`llama-server` that died mid-serve, or one start that failed, turned
recall off for the life of the serve with a single log line. That is not
what "olai owns this process" can mean. The memo is invalidated on child
exit and on a failed start, behind a 15-second cooldown so a crash loop
is not respawned per keystroke; a finalizer marks the shutdown so
teardown is not mistaken for a death. Tested without a model server: the
fake `llama-server` is two lines of shell that exit, and the SECOND embed
saying "cooldown" rather than repeating the first failure is the proof
that the memo was re-armed.

**A comment named a retry that does not exist (grok).** A failed embed
batch claimed the store's backstop probe would come back for it. The
backstop publishes nothing when the listing is unchanged, so on a corpus
nobody edits a transient failure stranded the index until a restart. The
batch retries twice on its own fiber (1 s, 4 s) and the comment now says
what actually happens when those are spent.

**An e2e absence proved less than it claimed (grok).** The recall-off
scenario read an empty list two animation frames after typing — under the
200 ms debounce. It now fills the list with a matching query first, so
emptiness is something the server DID, and outlives the debounce before
reading. Chasing it turned up the harder half, and the scenario states it
rather than implying otherwise: no browser assertion can separate recall
OFF from recall on-but-not-yet-indexed, checked by tagging the scenario
`@recall` and watching it still pass. The byte-for-byte contract is the
unit pin, which is sabotage-verified.

**The memory number was wrong in the direction nobody guessed
(opencode).** The doc claimed a flat 66 MB. opencode measured ~30 MB;
re-measuring here found ~30-66 MB shortly after start and **~158 MB**
once a real corpus has gone through it, ~54 MB of that the mmap'd
weights. `--ctx-size` is not the lever (512 / 2048 / 8192 all measure the
same) and `--sleep-idle-seconds` reclaims none of it. Stated as the range
it is, in the design doc, running.md and the tests README.

**`olai mcp` starts one too (both reviewers).** The process list was
written under an `olai web` framing; `mcp/serve.ts` opens the same index.
running.md now names the child on that face, says several can be resident
at once (including two over one directory), and says why it is being said
at all.

Also: docs/search.md narrows `≈` to the screen and names `matched:
"meaning"` as MCP's spelling of the same fact (opencode); the palette's
debounce comment stops quoting a warm p50 as though it covered the first
query of a serve (opencode).

1537 unit tests pass; the recall and palette e2e features pass against
the packaged binary.
srid added a commit that referenced this pull request Aug 14, 2026
…b3721c2 (the count tells the truth, the child comes back, the retry exists), PR #165 CI-green and fielded for the human's word

X-Olai-Writer: mcp
srid added a commit that referenced this pull request Aug 14, 2026
…rgent panel answered at 9cbb779 with five sabotages red on their pins, fielded beside #165 for the human's approval

X-Olai-Writer: mcp
srid added a commit that referenced this pull request Aug 14, 2026
… quote the reader verbatim, and the replace-vs-compose argument is written where it was owed; PR #168 joins #165 and #167 before the human

X-Olai-Writer: mcp
srid added a commit that referenced this pull request Aug 14, 2026
#168)

The third part of the `search` roadmap item, and the one the tags have
been
decorative for since title-markdown: a query does not open a list
somewhere
else — it takes rows away from the outline in front of you, keeping
every match
and the ancestors that lead to it.

Design pass first, as `#needs-brainstorm` asked:

**[docs/brainstorming/filter-in-place.md](../blob/filter-in-place/docs/brainstorming/filter-in-place.md)**
— filter state ownership, the URL representation, how a filter interacts
with
zoom and done-hidden, the operator grammar, the MCP face, and a stated
position
on composing with semantic recall (#165). Then the implementation.

## The load-bearing decision: the matcher moved DOWN

The filter cannot be a fourth caller of `Query.search`. It runs on every
keystroke over rows the browser already holds, it wants *every* match
rather
than twelve, and it wants them as a set of ids to test rows against. And
it
must not be a second implementation of the grammar — that is exactly the
drift
`docs/search.md` was written to forbid.

So `@olai/format` grew `filter.ts`: what a query MEANS (the words, the
operators, which nodes they select) and `keeping`, the row transform
that
narrows a tree to the matches with their ancestors — `withoutDone`'s
sibling.
`Query.search` calls it as its GATE and keeps what is about showing a
stranger a
SHORTLIST: the done penalty, the cap, the uncapped total, and carrying
the
refusal back.

**One matcher, four callers**: the filter, the ⌘K palette, the header
box, and
an agent's `search_nodes`. `is:done` cannot come to mean two things.

## The grammar

| written | selects |
|---|---|
| `is:done` `is:doing` `is:todo` | the mark a node STORES — never a
derived one |
| `is:marked` | any of the three, so `is:marked -is:done` is "work,
unfinished" |
| `is:archived` | what was put away — otherwise out of every reading |
| `has:desc` `has:see` `has:after` `has:doc` | a field the record
carries (the writer's own absence rule, so `desc: ""` is no note) |
| `has:date` | on any day at all — the unbounded `date:`, so the two
cannot disagree |
| `date:2026-08-10` `date:2026-08` `date:2026` | a day, a month, a year
|
| `date:a..b` `date:..b` `date:a..` | an inclusive span, either end
optional |
| `-anything` | takes that word or operator back out |

`date:` reads the two dates a journal reads — scheduled, and finished —
so the
filter, the day page and the calendar cannot disagree about what a date
means.

**A known operator with an unknown value is REFUSED**, in the grammar's
own
words, on every door: `is:blocked` says which values `is:` takes rather
than
quietly searching for that text and finding none. A colon anywhere else
(`TODO:`, a URL) is an ordinary word.

## On the page

- **Matches keep their ancestors**, and their own subtree. `data-match`
says
which rows the query selected and which are the context that leads to
one.
- **Pressing a `#tag` filters by it** — one delegated listener on the
pane,
because a tag pill arrives through `innerHTML` and belongs to no
component.
- **The filter is in the address** (`?q=`), so a narrowed page is a link
and
Back works. Typing REPLACES the history entry rather than pushing one
per
  keystroke; zooming is a navigation and starts unfiltered.
- **Folds are suspended while a filter is on** — and in ONE reading,
which the
tree, the editor, the selection and the drag all walk. Nothing is
written.
- **Hiding finished work goes first** (a standing claim about the
reader), and
the bar says how many matches that held back rather than leaving
`is:done`
  looking mysteriously empty.

## MCP keeps up rather than falling behind

`search_nodes` gains every operator by construction, plus `under` (a
node and
everything beneath it) and `file` — the two scopes a tree page can BE,
so an
agent can ask exactly the question a person asks by filtering one. Both
faces
took one schema. `matched` is now absent for a query that named no
words, since
no field carried it.

## Refactor passes, as isolated commits

- `8fabf826` **architecture-first-principles** — `Filter` was a flat
product of
terms/clauses/refusals where the third made the other two ungroundable
(C6's
flat-facts-product). It is a union: `nothing | refused | asking`. Nine
exports
  with no consumer outside the module went private (C2).
- `2d0bb027` **hickey + lowy** — the click predicate `ours` had quietly
become
two spellings; it moved to `press.ts`, which neither the router nor the
filter
owns. `clauseOf`'s fallthrough would have parsed a fourth operator as a
  `date:`; it is an exhaustive switch.
- `690ae5ca` **/simplify** — four reviewers, union applied. Two were
defects:
the refusal reached only one of the four doors, and fold suspension
lived
where the tree draws while three other walkers still saw the collapses
(a
  filtered page's arrow keys walked rows nobody could see).

## Tests

- `packages/format/src/filter.test.ts` — the grammar, the archive rule,
the tree
  transform, the counts.
- `packages/ops/src/query.test.ts` — the operators from the ops side,
the scope,
  the refusal, `matched` absent for an operator-only query.
- `packages/web/src/client/filter/narrowing.test.ts` — the order of the
two
  prunings and the two numbers.
- `packages/web/src/client/routes.test.ts` — the filter through the
address
  bijection, and `samePage`.
- `packages/tests/features/filter_in_place.feature` — 15 scenarios:
ancestors
kept, the address, reload, clearing, a collapsed ancestor not hiding a
match,
each operator, negation, the refusal on the bar AND on the header box,
tag
click (and that it does not also open an editor), zoom clearing the
filter,
Back restoring it, a zoomed page filtering its own children, no bar
where the
  address cannot keep one.
- `packages/tests/features/an_external_agent.feature` — the same grammar
and the
  same scope over MCP.
- `packages/tests/evidence.ts` — five new sections a person can look at.

## Deferrals, named

- Relative dates (`date:today`, `changed:7d`) — `parseFilter` is pure
and has no
  clock, which is deliberate.
- Quoted phrases, `OR`, and the `>` ancestry operator (`>` is the
palette's ask
  prefix).
- `is:blocked` — blockedness is DERIVED, and every clause today is a
test of the
RECORD; it is the first operator that would need the whole set. It
belongs
  with `edges-ui`'s blocked-derivation.
- Filtering the day, agenda and trash pages.
- Starred/saved searches, and a chord that focuses the filter box.
- `docs/roadmap.jsonl` is untouched: the `search` item's stamp is the
  orchestrator's at merge.

## Composing with semantic recall (#165)

Stated in the design ahead of the merge: **a filter constrains the
candidate
set; recall ranks within it.** The clause half of a query gates BOTH
kinds of
hit — a paraphrase neighbour that is archived, or `done` under
`-is:done`, is
not a hit. Recall replaces the TERMS; it never relaxes the CLAUSES. If
#165
merges first, this branch merges master and re-fits.

Not merged, and not to be merged: held for your review.

## Evidence


**https://claude.ai/code/artifact/a2150f0d-2c9f-45cf-8ead-f625469186f3**
—
screenshots and transcripts from driving the real app one gesture at a
time
(`packages/tests/evidence.ts`, five new sections): the filter with
ancestors
kept, each operator with what it drew, the refusal, the tag click, and a
collapsed parent that does not hide a match.

## CI

`odu run --platform x86_64-linux` at `5b84536` — **10 ok · 0 failed · 0
errored**
(nine required contexts plus `_ci-setup`), e2e 2m10s. Linux only; this
change
does not touch macOS. Locally: 555/555 e2e scenarios, 1557/1557 unit
tests.

---

## Review round (grok + opencode, independent)

**2 blocking + 9 nits, deduped; union addressed at `98b81457`.** Both
blocking
findings were the same shape as each other and as the thing this grammar
exists
to prevent — a query answering with silence.

- **[blocking] An impossible date was answered with an empty tree.**
`date:2026-13` is shape-clean and can never contain a validated day, so
it
parsed, selected nothing, and said nothing — while `date:soon`, the same
mistake one character further from the shape, was refused with the
reason. It
is also the worse of the two to swallow: month 13 *sorts* between
December and
  January, so an empty answer reads as a window rather than as nonsense.
`datePart` bounds the digits (1–12, 1–31) at both ends of a range. The
line is
stated rather than fudged: `2026-02-30` is accepted and finds nothing,
because
telling it from `2026-01-30` needs a calendar and nothing here parses a
date
  into one.
- **[blocking] The refusal misquoted the reader.** The input was folded
before
parsing and the folded token reported, so `is:BLOCKED` was refused as
"you
typed `is:blocked`". The fold is per token and for matching only now; a
refusal carries the token as written. No door had a mixed-case scenario,
which
  is why it slipped — the filter bar and MCP both have one now.

Nits, in order of what they cost: palette + MCP refusal e2e (two of four
doors
were unit-tested only); `Query.search`'s tests now assert `refusals`;
`docs/architecture.md` and `packages/web/README.md` still told the
pre-PR
three-doors/no-matcher story; tag pills looked pressable on pages whose
address
cannot keep a filter (the pane publishes `data-narrowable`, the cursor
is drawn
from it, the listener declines on it, and a scenario pins both halves);
`--is:done` documented as a literal rather than refused (a second
refusal rule
would refuse `--force`, a word people write); `date: 2026` now says a
space
split one word into two rather than reciting the date shapes; the design
doc
argues tag-click **replace vs compose** and records it as reversible;
the
refusal example that contradicted its own note is now `is:open`.

Both reviewers independently verified the `/simplify` fixes as real —
the
four-door refusal parity and the single fold reading (one could not
re-break the
arrow-keys-walk-hidden-rows bug).

**CI at the final sha `98b81457`: 10 ok · 0 failed · 0 errored**, e2e
1m50s.
Locally 560/560 e2e scenarios, 1564/1564 unit tests.
srid added a commit that referenced this pull request Aug 14, 2026
…, it ratifies @olai/child with the wall conceded at closing, binds the PR's re-fit to the floor last week built, pays nothing upstream, and leaves the human one command to run

X-Olai-Writer: mcp
@srid
srid marked this pull request as draft August 14, 2026 23:45
srid added a commit that referenced this pull request Aug 14, 2026
…165 drafted, the resume checklist written where it will be found, the debate write-up deleted at the same word

X-Olai-Writer: mcp
srid added a commit that referenced this pull request Aug 15, 2026
)

"A place in the loaded set" was spelled four times in `@olai/format`. It
is spelled once now, in `node.ts` beside `Located`, and the other three
spread it.

Roadmap node `four-way-site-spelling`. Found during #170's refactor
round and **declined there on the record** — both reviewers concurring
the decline — because folding it touches `OutlineError`, `Located` and
the public index, which is more than a drive-by. Dispatched as its own
PR for exactly that reason.

**Held for review — do not merge.**

## The four, re-located

The node's line numbers are from 2026-08-14 and three of the four have
moved since. All four re-located at `aa23635e`, master's head when this
branch forked:

| spelling | at | was, on the node |
| --- | --- | --- |
| `Site` | `format/src/errors.ts:144` | `errors.ts:144` ✔ |
| `Located` | `format/src/node.ts:143` | `node.ts:98` |
| `Found` | `format/src/reading.ts:88` | `reading.ts:92` |
| `Placement` | `format/src/reading.ts:197` | `reading.ts:207` |

**Two more turned up that the node does not name**, and they are the
same defect one layer out:

| spelling | at | what it was |
| --- | --- | --- |
| `siteOf` | `format/src/validate.ts:327` | `({file, line}: Located): {
file: string; line: number }` — an inline fifth, on the function every
finding in that file gets its `file:line` from |
| `Site` (a component) | `web/src/client/errors/Report.tsx:117` |
`function Site(props: { readonly file: string; readonly line: number })`
— a sixth, in a component named for the very thing the format declares,
handed the pair a field at a time |

**PR #179 (`props-floor`) adds no fifth spelling**, checked rather than
assumed: its new structs are `PropRequest` and the `prop` verb (`{op,
id, key, value}`), neither of which carries a place. #186 and #165
likewise. Nothing absorbed from any of them — all three are unmerged.

## What this is not

It is not a fold of the four into one type. `OutlineError`, `Located`,
`Found` and `Placement` are genuinely four things — an error, a record
in the set, a read's flattened answer, a mirror's location — and #170's
decline was right that folding them would be a type redesign. They stay
four.

What was never four is the PLACE. `{file, line}` is one fact, and it was
written down four times, so it could be extended in one of the four and
compile clean past the other three. The four remain; they now spread one
`Site`.

`node.ts` and not `errors.ts`, for two reasons. An error is one of the
four carriers rather than the atom's home, and `node.ts` is the module
that says what a line of an outline *is* — one node per line is why
`line` is the whole address. It also imports nothing else in the
package, so every carrier can reach it; the reverse arrangement, with
`node.ts` reaching up into the error catalogue, could not have promised
that.

## The property: add a field to a place, watch every carrier fail

The acceptance test, run both ways on a verified-clean tree and reverted
after each. `just typecheck` is twelve packages.

**BEFORE** — a required `drifted` added to `errors.ts`'s `Site`, on
master's arrangement. Seven packages go red, which reads like success
and is not:

```
@olai/format  Exited with code 2      @olai/chat   Exited with code 2
@olai/surface Exited with code 2      @olai/web    Exited with code 2
@olai/ops     Exited with code 2      @olai/server Exited with code 2
@olai/tests   Exited with code 2
```

Every one of the 40 errors names an `OutlineError` or `Related` producer
— `format/src/parse.ts`, `format/src/validate.ts`, `ops/src/codec.ts`,
and three test files. **Not one names a `Found`, a `Located` or a
`Placement`**, because on master those three had never heard of this
declaration:

```
$ grep -icE "foundOf|Placement|Located|query\.ts|reading\.ts|node\.ts"  <before-drift>
0
```

**AFTER** — the same required `drifted` added to `node.ts`'s `Site`. The
same seven packages go red, and now the failures land on all four
carriers at once:

```
@olai/ops  src/query.ts(125,3): error TS2741: Property 'drifted' is missing in type
           '{ see?…; after?…; path: string[]; status?…; id: string; title: string; file: string …'
                                                                   ← foundOf, the Found producer
@olai/ops  src/query.ts(273,3): error TS2322: … not assignable to type 'readonly { readonly id: string;
           readonly file: string; readonly line: number; readonly drifted: string; readonly shows: …'
                                                                   ← Placed
@olai/ops  src/query.ts(300,3): error TS2322: Type '{ parent?: string | undefined; id: string;
           file: string; line: number; }[]' is not assignable to type 'readonly { … drifted: string …'
                                                                   ← Placement
@olai/web  src/client/page.test.ts(9,82): error TS2741: Property 'drifted' is missing in type
           '{ file: string; line: number; node: { readonly title: string; readonly id: string; …'
                                                                   ← Located
@olai/ops  src/codec.ts(52,29): error TS2741: Property 'drifted' is missing in type
           '{ file: string; line: number; code: "unreadable-directory"; message: string; }'
                                                                   ← OutlineError
@olai/web  src/client/palette/items.test.ts(18,25): error TS2345: Argument of type '{ id: string;
           title: string; file: string; line: number; path: string[]; matched: "tit…'
                                                                   ← SearchHit, built on Found
```

67 errors across 7 packages, against 40 across the same 7. **The count
is not the property — the coverage is.** Before, a place could grow a
field and three of its four carriers would not notice. After, there is
no carrier that can be left behind, because there is no second
declaration to leave it in.

Tree verified clean after both; `git status --short` empty, `Site` back
to two fields.

## No behaviour change

Every error message, every search hit and every read answers
byte-identically. The claim rests on two things rather than on reading
the diff.

**Field order is unchanged on every carrier**, checked by evaluating the
exports on master and on this branch and printing the keys — a spread
placed where the pair sat, not appended:

| | master | branch |
| --- | --- | --- |
| `Located` | `["file","line","node"]` | identical |
| `OutlineError` | `["file","line","code","message","related"]` |
identical |
| `Found` | `["id","title","file","line","status","path","see","after"]`
| identical |
| `Placement` | `["id","file","line","parent"]` | identical |
| `Placed` | `["id","file","line","parent","shows"]` | identical |
| `Detail` | 18 keys | identical |
| `Subtree` | 12 keys | identical |

**The e2e suite is unchanged** — no scenario added, removed or edited;
the diff against master touches no file under `packages/tests`. Seven
files change, all of them source or docs.

The one behaviour-adjacent edit is `Report.tsx`, and it is
markup-identical: `<At site={props.error} />` in place of `<Site
file={…} line={…} />`, same `<code>`, same classes, same `hasLine` rule,
same text. The component is renamed because the name `Site` now belongs
to the type it draws.

## Public-surface delta, measured

`@olai/format`, computed by evaluating the export keys on both sides
rather than by reading `index.ts`.

**150 → 151. Nothing removed. One thing added: `Site`.**

```
$ diff exports-MASTER exports-BRANCH
60a61
> Site
```

`Site` was `export const` in `errors.ts` and reachable by nobody —
`index.ts` did not re-export it, and no package outside `@olai/format`
named it. It is on the index now because a consumer that wants to say
where something is should not have to re-derive the pair, and
`@olai/web` is the first to take it.

One signature narrowed, no caller affected: `hasLine(error:
Pick<OutlineError, "line">)` → `hasLine(site: Pick<Site, "line">)`. It
said the field was the error's; the field is the place's, and the
function reads nothing else.

## Refactor round — architecture-first-principles, hickey, lowy,
/simplify

Each pass its own commit:

| commit | what |
| --- | --- |
| `13868446` | `Site` moves to `node.ts` beside `Located`; `Located`
spreads it; on the public index |
| `69e6a393` | `Found` and `Placement` spread it — the two in
`reading.ts`, two declarations apart |
| `1777a23d` | `validate.ts`'s `siteOf` returns a `Site` — the inline
fifth |
| `0738bfcf` | `@olai/web`'s error rows take a `Site` whole — the sixth
|
| `9d4291b1` | `hasLine` asks about a place, not about an error; docs |
| `5f92af7e` | the review round's two doc nits — counted claims lose
their counts |

**Declined, on the record.** `compareErrors` (`errors.ts:210`,
`Order.Struct({file, line, code})`) and `validate.ts:318`'s cycle
canonicaliser both order by file-then-line, which is a second thing a
`Site` could own. Two consumers, and they are different shapes — a
presentation `Order` and a min-scan inside a reduce — so an
`Order.Order<Site>` here would be a receptacle built for one-and-a-half
users. Left, and named here so the next reader does not rediscover it as
an oversight.

## Review round — Grok and opencode

**Two reviews, zero blocking.** Both went looking for a seventh spelling
and **neither found one**, which is the check this PR most needed
independent eyes on: it set out after four and closed six.

Two doc nits from opencode, both taken in `5f92af7e` — docs only, no
source, no schema, no exported symbol:

| finding | done |
| --- | --- |
| `index.ts:8` "Nine things are exported" fights a 151-symbol surface |
**reworded, not recounted.** Recounting rearms the trap — the header was
already wrong once (#170 fixed "Eight" over nine bullets). It now says
the list is the claim and never its length, and says why. The nine were
*bullets* and 151 is symbols; that the two readings were ever confusable
is the argument for carrying no number |
| `format/README.md` hard-codes the drift counts | **taken**, with one
correction that does not change the outcome: the README carried "failed
seven packages", not 40/67 — those are in this body. Same defect, so it
goes the same way. The carrier-coverage claim stays, now stated over the
four carriers by name, which survives a package being added or split.
The snapshot numbers keep their home here, in a document that is dated
by construction |

## Evidence

Transcripts, not screenshots — this is a types PR.

**CI green at head.** odu strict run at `5f92af7` — this PR's head,
master merged — Linux, macOS skipped (this PR does not touch it), fully
settled:

```
{"settled":true,"passed":true,"failed":[],"errored":[],"cancelled_nodes":[],
 "fail_fast_tripped":false,"timed_out":false,"cancelled":false,"sha7":"5f92af7","seq":1,"unposted":[]}
```

```
── ci run summary @ 5f92af7 ──
  ✔ _ci-setup  ✔ bun-nix-fresh  ✔ fmt-check  ✔ hm-module  ✔ install
  ✔ kolu-deps  ✔ nix  ✔ test  ✔ typecheck  ✔ e2e (2m3s)
  10 ok · 0 failed · 0 errored · 0 skipped · 0 cancelled — OK
```

`gh pr checks 188 --required` — **9/9 pass, exit 0.**

Local, on a verified-clean tree:

| what | result | head |
| --- | --- | --- |
| `just typecheck` | **12/12 clean** | every commit, and again after the
merge from master |
| `just test` | **1669 pass / 0 fail**, 139 files | merged head |
| `just e2e` | **604 scenarios / 4545 steps / 0 fail** | branch head,
before the merge — CI's e2e leg covers the merged head |
| drift experiments | both run, both reverted, tree clean after each | |
| export keys, field orders | diffed against master by evaluation, not
by reading | |

**One e2e run before that one had four failures, and all four were
flakes.** Recorded rather than dropped, because a PR claiming "the suite
is unchanged" owes the deviation:

| scenario | why it is not this diff |
| --- | --- |
| `input_widgets.feature:209` — Escape does not close the draft | **a
filed, known flake with a lane of its own**: roadmap node
`input-widgets-escape-flake`, caught the same way in #171's night run
(2026-08-14) — same symptom, same "the identical re-run, the targeted
re-run and CI's e2e leg all passed" |
| `edge_editing.feature:117` — ⌘Z puts the target back | 15 s timeout
waiting on disk |
| `palette_actions.feature:285` — a second capture into the inbox | 15 s
timeout waiting on disk |
| `the_agent.feature:584` — the transcript follows the newest line | 15
s timeout waiting on a scroll |

All four are 15 s timeouts from a parallel run whose steps burned 32 m
of CPU in 8 m of wall clock, and not one of them touches an error, a
site, a search hit or a read. All four pass targeted and serial (`4
scenarios (4 passed)`), and the very next full run was **604/604, zero
failures**.

Docs: `packages/format/README.md` gains the `Site` paragraph beside
`Status` (the same argument about a different field) plus the
before/after of the experiment, and the reads table gains its row.
`src/index.ts`'s contract header names `Site` where it names the
records. **`docs/*.md` needs no edit** — it speaks of `file:line` as a
promise the format makes, which is exactly as true as it was; nothing
there named any of the four declarations.

`docs/roadmap.olai` is untouched by this branch. The merge from master
carries the orchestrator's own edits to it.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
@srid srid added the PARKED label Aug 19, 2026
srid added a commit that referenced this pull request Aug 25, 2026
…sted — the Recall seam and src/recall/ live only in the unmerged semantic-recall branch (#165); the parked story stays told at the search section
srid added a commit that referenced this pull request Sep 5, 2026
…516)

Phase 13 of [the Cordis
plan](https://github.com/juspay/oss.olai/blob/main/projects/olai/brainstorming/cordis-for-olai.md),
and the first of the promotion track. `@olai/index` and `Query.search`
become `packages/plugins/search/`. Core defines a `Search` door in
`@olai/ops` and does not stand behind it; `search.nodes` — one member on
the browser, MCP and agent faces, with `search_nodes` landing through it
— calls through the door and **refuses in words** when nobody is
mounted.

## What moved

| was | is |
| --- | --- |
| `@olai/index` (the trigram table) | `olai-plugin-search`'s
`src/table.ts`, opened on the plugin's fiber scope and closed with it |
| `@olai/ops`' `Query.search` (the walk) | `src/matcher.ts` |
| `@olai/web`'s `client/search/HeaderSearch.tsx` |
`src/browser/HeaderSearch.tsx`, in `app.header`'s new `lead` seat |
| `@olai/ops`' `search.index.test.ts` | `src/matcher.index.test.ts`,
standing the row's door up the way a serve does |

`@olai/ops` no longer depends on `@olai/index` at all — the package is
gone, and `make` opens no table. What is left there is the door,
`NO_SEARCH`, and `foundOf`, which the row **imports rather than
copies**: a second answer to *where does this node live* would be free
to disagree with `read_node`'s in the same turn.

## The door

```ts
export interface Search {
  readonly nodes: (ask: {
    readonly at: Reading          // the very reading the answer is about
    readonly query: SearchRequest
    readonly now: string          // the ops layer's clock
    readonly kinds: KindVocabulary
  }) => Effect.Effect<SearchAnswer>
}
```

The reading comes IN, which is the correctness argument rather than a
convention: a door that read the vault for itself would answer a
revision of its own choosing, and the candidates a table hands back are
only ever right about the snapshot they were levelled with. So the row
has no `Vault` in its `needs`.

Typed at both ends, the way `Ledger` is: `@olai/plugin-api` may not
import the floor, so its tag carries `unknown` and `@olai/ops` spells
the same door with the floor's own types. `serve.ts` is the one file
holding both, which makes a drift a type error in one place.

## `--plugins=` minus `search`

Every one of the five doors answers with no hits **and the reason** —
carried on `refusals`, the field each of them already draws for a query
the grammar could not read, and which an agent reads in its tool result.
The header loses its box at both widths. The filter over the page in
front of you is `Query.narrowing`, a reading of one page, and it goes on
working.

## The bar's `lead` seat

`app.header`'s face is `{ place, body }` now, `sidebar.entry`'s shape
one slot over. `cluster` is the standing row of pills (desktop, after
the connection state); `lead` is the seat ahead of them that may shrink
to nothing and that a phone draws — which is what the search box has
always been and why it has a magnifier arm. Four registrations changed
one line each. `useMaybeGo` is the shell furniture the box needed to
replace `AppHeader`'s optional `go`: two screens draw the bar with no
router under them, and a slot face has no props to be handed that fact
through.

## What stayed core, and why that is the whole scope ruling

The shortlist kit under every search box — `Shortlist`, `Result`,
`nodes`, `cursor`, `props`, `place`, `Count` — is furniture **four core
doors already shared** before this row existed: the ⌘K palette, the
composer's `@` list, the edges panel and the move picker. Moving it
would have made core import a plugin, or moved those four doors into
slots, which is phase 18. Scoped by the reviewer on the agent's
question, 2026-09-04, and recorded in the brief.

## Tests

Every assertion travelled; none was dropped.

- `query.test.ts`'s 15 search-touching blocks are `olai-plugin-search`'s
`matcher.test.ts`. Two are deliberately double-sided and are there
because only this package can now reach both doors: *a key holding
nothing is carried by neither the hit nor `prop:`* (a hit and a node
read agree) and *the order is the PAGE's* (this door and
`Query.narrowing` deliberately disagree).
- `@olai/ops`' tool walk is a harness (`@olai/ops/testlib/tools`) run
twice over **one** fixture: with `NO_SEARCH` in `@olai/ops`, and with
the matcher in the row. Two copies of a maximal set would be two things
to keep maximal.
- The MCP face's search cases split the same way — the walk's half
through that encoder stays, the hit's half is the row's, and a new case
pins what an agent is told when nobody is behind the matcher (the `push`
test's arrangement one door over).

## Green

`typecheck`, `test` (5,572 pass), `e2e` (**1,113 scenarios, 1,113
passed** — the search scenarios unchanged), `nix`, `bun-nix-fresh`,
`fmt-check`, `kolu-deps`, `odu-deps`, `cordis-deps`, `hm-module`.

`packages/format/src/splice.test.ts`'s *the patched view is the derived
view over the real vault* times out at 5s on this machine on
`origin/master` too — pre-existing and unrelated, confirmed against a
clean tree.

## Not in this PR

The parked semantic recall
([#165](#165)) stays parked, by the
human's scope ruling of 2026-09-04. It is a second provider row behind
the same door, with no core PR and no ruling to reopen.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant