Speed up search 2-3x, fix CJK layout and stale-result selection (0.2.0) - #6
Merged
Conversation
The workspace test suite is green at every step and still cannot answer three questions that matter when changing the matcher, the ranking, the renderer's width accounting, or the TUI event loop: did the output change, did it get slower, and does the interface behave when keystrokes arrive faster than searches finish. scripts/qa/ answers them by comparing against a second binary built from a git ref: diffout.py 257 invocations through both binaries, byte compare classify.py splits differences into pure reorder vs content change bench.py absolute timings, min of 5 ab.py interleaved A/B so both binaries see the same machine pty/ drives the real interface through a pty gen_corpus.py five seeded corpora, including mixed-case and a race corpus build-baseline git archive <ref> into its own target dir Nothing runs in CI; it needs a baseline binary and takes minutes.
… case in the matcher These changes are interdependent - they touch the same scoring and ranking paths and the same tests - so they land together. Performance, all verified output-identical except where noted: * Extended queries parsed the query, re-expanded every term's variants, and re-normalized every candidate key once PER CANDIDATE. All of that is now done once per search, exact terms reuse the candidate's existing normalized key, and the extended path parallelizes with Rayon like the standard path. 500k lines: `--filter 'ab cd'` 0.79s -> 0.25s, now the same cost as `--no-extended`. Four terms cost about what two did. * The bounded top-results buffer did two linear scans per scored candidate. It is now a binary heap keyed on the full rank, O(log limit) instead of O(limit). 500k lines, unfiltered `--limit 1000`: 0.78s -> 0.32s. That path used to be 2.4x SLOWER than not bounding results at all. Correctness: * `--ignore-case` was implemented only through a lowercased key, so `--literal` silently defeated it. Case folding is now the matcher's job. This changes ranking for case-insensitive queries: the original text keeps its word-boundary and camel-case bonuses, so a 75-point exact-case bonus is added to keep the literal spelling winning ties. * `--algo nucleo` and `--algo fzf-v2` never applied the case policy at all, and panicked on an uppercase query. Both fixed. Case-sensitive nucleo output therefore changes; case-insensitive output does not. * Folding truncated multi-character lowercase mappings, so `--filter ia --exact` matched `İa`. Folding is now 1:1 and such characters are compared as written, with a retry that writes the mapping out so `--literal` still folds them. * Three case-folded substring scans were O(text x pattern). A repetitive query against long records took 3.5s; now 0.007s.
…olumns Selection was a bare index into a result list that is replaced wholesale whenever a search lands. Nothing kept the index meaning the same row, which is the single design fact behind every stale-result bug here. Selection is now bound to the id of the candidate it is on. That fixes a defect present in 0.1.11: pressing Enter before the search for what you just typed had finished returned a line the query did not match. On 500k lines, typing `ab` then `%` and Enter in one burst accepted `aba/bhs/iphgc/file8329.rs` for the query `ab%`, which matches nothing. Marks are likewise kept by id, so refining a query no longer silently drops items marked earlier. Layout now budgets terminal display columns per grapheme cluster instead of counting characters. A 20-column viewport was emitting 28 columns for a CJK row, wrapping and corrupting the frame - the project's core use case. A cluster is drawn whole or not at all, so combining marks and emoji modifiers survive clipping, including across interleaved SGR sequences, and the gutter is clipped to the viewport so a wide `--marker` cannot overflow. `--delimiter` compiled its regex once per input line per field expression. It is compiled once at startup: 500k lines with `--nth 2 -d /`, 0.95s to 0.34s. An invalid pattern is now reported at startup. Adds `--live-smart-case` as an opt-in preview, off by default. Re-evaluating smart case per keystroke means a result set computed under the previous case policy can still be on screen, and that has produced real defects; the default keeps 0.1.x behaviour of deciding once at startup. Adds unicode-segmentation for grapheme cluster boundaries.
… flag CHANGELOG gains a Breaking section: yuru-core changed the signatures of score_text / score_exact_text / score_key, GreedyMatcher and ExactMatcher are no longer unit structs, SearchKey and TuiOptions gained public fields, MatcherBackend gained folds_case, and TuiState's selection API now takes the rows rather than a row count. Under 0.x semantics that is a minor bump. internals.md corrects a stale complexity claim - the top-results buffer was documented as O(M * R) and is now a heap - and describes where case folding happens, why the fold is one character in and one character out, and why extended queries prepare once per search rather than once per candidate. fzf-compat.md gains a Preview features section stating plainly that --live-smart-case may be changed or removed, and listing what is known to be wrong with it rather than only that it is experimental.
Contributor
Greptile SummaryThe PR substantially revises search matching, ranking, interactive selection, and terminal rendering while adding focused performance and regression coverage.
Confidence Score: 5/5The PR appears safe to merge based on the reviewed changes, with no concrete changed-code defect identified. The revised search, selection, rendering, and delimiter paths preserve their relevant identities and boundaries, and the dependency advisories not fixed here were unchanged from the base revision.
|
| Filename | Overview |
|---|---|
| crates/yuru-core/src/matcher.rs | Moves case handling into matcher implementations, adds exact-case scoring, handles Unicode lowercase expansion, and introduces bounded linear substring search. |
| crates/yuru-core/src/fzf_query.rs | Prepares extended-query terms once per run and reuses normalized candidate keys while preserving exact-term ranking behavior. |
| crates/yuru-core/src/rank.rs | Replaces linear bounded-result maintenance with heap-based top-k selection using the full ranking order. |
| crates/yuru-tui/src/run.rs | Associates rows with query and case-policy identity and delays acceptance until matching results are available. |
| crates/yuru-tui/src/state.rs | Replaces positional selection and unordered marks with candidate-identity selection and mark-order preservation. |
| crates/yuru-tui/src/render/layout.rs | Introduces grapheme-aware terminal-column measurement, clipping, and ellipsis handling. |
| crates/yuru-tui/src/render/results.rs | Applies display-column accounting to gutters, result text, cursor placement, and selected-row padding. |
| crates/yuru/src/fields.rs | Compiles delimiter regular expressions once and reuses them across field transformations. |
| Cargo.lock | Updates rkyv and rkyv_derive to 0.8.17 and records the Unicode segmentation dependency. |
Sequence Diagram
sequenceDiagram
participant U as User
participant T as TUI loop
participant W as Search worker
participant S as Search core
U->>T: Edit query
T->>W: Search(query, case policy, sequence)
W->>S: Execute prepared search
S-->>W: Ranked candidates
W-->>T: Result identity + rows
T->>T: Apply only matching identity
U->>T: Accept
alt Current result set
T-->>U: Selected candidate IDs
else Stale result set
T->>T: Capture candidate target
T->>W: Wait for live search result
W-->>T: Current rows
T-->>U: Resolve captured target
end
Reviews (1): Last reviewed commit: "Document the 0.2.0 changes, the breaking..." | Re-trigger Greptile
There was a problem hiding this comment.
Pull request overview
This PR prepares the 0.2.0 release by refactoring matching/ranking to substantially speed up extended queries and top‑k selection, fixing several long-standing correctness issues (case policy with nucleo, stale-result acceptance, CJK/wide-text layout), and adding extensive tests, docs, and opt-in QA harnesses to guard the new behavior.
Changes:
- Refactors
yuru-corematching/ranking: matcher-driven case folding, extended-query preparation hoisted per-search, bounded top‑k via heap, delimiter regex compiled once, plus new stats/tests. - Fixes TUI correctness: stable selection across result replacement, accept gating on live search identity (incl. live smart case), and width accounting by grapheme cluster/display columns.
- Expands coverage and tooling: new CLI/TUI/core tests, QA scripts, updated docs/changelog, adds
unicode-segmentation, and bumpsrkyvfor a security advisory.
Reviewed changes
Copilot reviewed 51 out of 53 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/golden_ranking.rs | Adds mixed-case golden ranking guards to pin intended reorderings. |
| scripts/qa/README.md | Documents new opt-in QA harnesses (diff, A/B bench, pty drivers). |
| scripts/qa/pty/race_accept.py | Adds PTY reproducer for stale-result acceptance race. |
| scripts/qa/pty/pty_frame.py | Adds fixed-size PTY frame capture helper. |
| scripts/qa/pty/drive_tui.py | Adds PTY driver to exercise smart-case + resize repaint behavior. |
| scripts/qa/gen_corpus.py | Adds deterministic corpus generator for perf/race/case scenarios. |
| scripts/qa/diffout.py | Adds baseline-vs-new stdout differential runner across many cases. |
| scripts/qa/common.py | Centralizes QA path/env resolution for harness scripts. |
| scripts/qa/classify.py | Classifies diffout deltas into reorder-only vs content changes. |
| scripts/qa/build-baseline | Builds a baseline binary via git archive for comparisons. |
| scripts/qa/bench.py | Adds absolute benchmark harness (min of N runs). |
| scripts/qa/ab.py | Adds interleaved A/B benchmark harness for stable comparisons. |
| scripts/check | Adds shellcheck (bash -n) coverage for the new baseline builder. |
| docs/internals.md | Documents new case-folding model, extended-query prep, selection identity, and top‑k heap. |
| docs/fzf-compat.md | Documents delimiter behavior and preview --live-smart-case. |
| crates/yuru/tests/cli_fields.rs | Updates explain output expectation for “matched key: Original”. |
| crates/yuru/tests/cli_case.rs | Adds CLI case policy regression tests, incl. combining-tail and extended exact bonuses. |
| crates/yuru/src/options.rs | Wires compiled delimiter and adds smart_case_active() logic/tests. |
| crates/yuru/src/main.rs | Passes smart_case option into TUI options for interactive modes. |
| crates/yuru/src/fields.rs | Introduces Delimiter (precompiled regex) and threads it through field transforms. |
| crates/yuru/src/cli.rs | Adds preview flag --live-smart-case. |
| crates/yuru-tui/src/tests/state.rs | Updates/expands state tests for SelectionTarget + mark/accept semantics. |
| crates/yuru-tui/src/tests/search_worker.rs | Ensures search responses carry identity (query + case policy). |
| crates/yuru-tui/src/tests/run.rs | Adds tests for resize redraw, live smart case, accept gating, and outcome resolution. |
| crates/yuru-tui/src/tests/render.rs | Adds extensive wide-text/cluster/ellipsis/gutter render guards. |
| crates/yuru-tui/src/tests/mod.rs | Registers the new run test module. |
| crates/yuru-tui/src/tests/highlight.rs | Adds guards for CJK highlight positions and stale phonetic highlight gating. |
| crates/yuru-tui/src/state.rs | Implements selection-by-identity (SelectionTarget) and ordered marks. |
| crates/yuru-tui/src/search_worker.rs | Tags requests/responses with SearchIdentity and threads through APIs. |
| crates/yuru-tui/src/run.rs | Adds terminal-event classification, live search identity, pending accept, and smart-case resolution. |
| crates/yuru-tui/src/render/results.rs | Switches rendering/padding to display-column accounting and per-row gutter width. |
| crates/yuru-tui/src/render/mod.rs | Re-exports layout helpers for tests. |
| crates/yuru-tui/src/render/layout.rs | Adds grapheme-cluster aware truncation and display-width budgeting. |
| crates/yuru-tui/src/render/highlight.rs | Makes highlighting column-budget safe and prevents stale phonetic rows painting as matches. |
| crates/yuru-tui/src/lib.rs | Re-exports SelectionTarget publicly. |
| crates/yuru-tui/src/api.rs | Adds smart_case to TuiOptions (default off). |
| crates/yuru-tui/Cargo.toml | Adds unicode-segmentation dependency to support grapheme-aware rendering. |
| crates/yuru-core/src/rank/tests.rs | Adds many regression tests for case policy, nucleo behavior, extended parallelism, and heap top‑k. |
| crates/yuru-core/src/rank.rs | Refactors search paths, case-aware matcher construction, extended-query parallelism, and heap-based top‑k. |
| crates/yuru-core/src/query/tests.rs | Adds tests for case_fold_only key blocking behavior under different scorers. |
| crates/yuru-core/src/query.rs | Updates key_blocked_by_config to account for case-fold-only keys and scorer folding. |
| crates/yuru-core/src/lib.rs | Documents new expectations for backend normalize_candidate overrides. |
| crates/yuru-core/src/fzf_query/tests.rs | Adds tests for extended exact-term case bonus and term-mode consistency. |
| crates/yuru-core/src/fzf_query.rs | Hoists extended query prep, reuses normalized haystacks, and restores exact-case bonus for exact terms. |
| crates/yuru-core/src/candidate/tests.rs | Adds tests pinning case_fold_only detection for normalized keys. |
| crates/yuru-core/src/candidate.rs | Adds case_fold_only to SearchKey and computes it for normalized base keys. |
| CHANGELOG.md | Documents breaking changes, perf wins, fixes, preview flag, and security update for 0.2.0. |
| Cargo.toml | Adds workspace unicode-segmentation dependency. |
| Cargo.lock | Bumps rkyv and adds unicode-segmentation lock entries. |
| benches/search.rs | Adds criterion benches for extended queries and Japanese extended search. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+24
to
+40
| def drain(master, timeout): | ||
| out = b"" | ||
| deadline = time.time() + timeout | ||
| while True: | ||
| remaining = deadline - time.time() | ||
| if remaining <= 0: | ||
| return out | ||
| ready, _, _ = select.select([master], [], [], remaining) | ||
| if not ready: | ||
| continue | ||
| try: | ||
| chunk = os.read(master, 1 << 16) | ||
| except OSError: | ||
| return out | ||
| if not chunk: | ||
| return out | ||
| out += chunk |
Comment on lines
+27
to
+32
| fd = os.open(cands, os.O_RDONLY) | ||
| os.dup2(fd, 0) | ||
| os.close(master) | ||
| os.environ["TERM"] = "xterm-256color" | ||
| os.execv(binary, [binary, *args]) | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Audit of the workspace at v0.1.11, then the fixes it produced. Four commits,
each building and passing tests on its own.
Performance
Measured with an interleaved A/B harness: the v0.1.11 binary and this branch
run alternately within each case, 7 runs each, so both see identical machine
conditions. 500k-line path corpus,
--lang plain.--filter 'ab cd'--filter 'ab cd ef gh'--limit 1000--nth 2 -d /--filter ab--filter abc --limit 1000--filter 'ab cd' --no-extended--limit 2000The controls landing at exactly 1.00x is what makes the rest trustworthy. On
the criterion benches, plain search over 100k is 0.64x, Japanese 0.76x,
Chinese 0.71x, and both
build_indexbenches are flat - the wins are notpaid for at indexing time.
Three causes: extended queries re-parsed the query and re-normalized every
candidate key per candidate; the bounded top-k buffer did two linear scans
per scored candidate and was 2.4x slower than not bounding at all; and the
--delimiterregex was compiled once per input line per field expression.Bugs fixed that are in v0.1.11 today
--algo nucleopanics on an uppercase query, and has never applied thecase policy at all, so
--algo nucleo --no-ignore-casewas notcase-sensitive.
ab, then%and Enter in one burst on 500k lines accepts
aba/bhs/iphgc/file8329.rsfor the query
ab%, which matches nothing. Selection was a bare index intoa list replaced wholesale when a search landed.
for
日本語検索テスト用の候補行, wrapping the frame - the project's coreuse case.
--ignore-caseis silently defeated by--literal, becausecase-insensitivity was implemented only through a lowercased key that
--literaldisables.--filter ia --exactmatchesİa, because folding truncatedmulti-character lowercase mappings.
against long records took 3.5s, now 0.007s.
The one intended behaviour change
Case folding moved into the matcher, so a case-insensitive query is scored
against the candidate's original text, which still carries its word-boundary
and camel-case bonuses. A 75-point exact-case bonus keeps the literal
spelling winning ties.
Verified by running 257 command invocations through both binaries and
comparing stdout byte for byte: the matching set and exit code are identical
in every case, and the differences are pure reorderings on a mixed-case
corpus. Case-sensitive matching, case-insensitive
--algo fzf-v2/nucleo,and all Japanese, Korean and Chinese phonetic matching are byte-identical.
Case-sensitive nucleo does change, because it never applied the case
policy before.
Breaking - this is 0.2.0, not 0.1.12
yuru-core:score_text/score_exact_text/score_keytake acase_sensitiveargument;GreedyMatcherandExactMatcherare no longerunit structs;
SearchKeygainscase_fold_only;MatcherBackendgainsfolds_case(defaulted).yuru-tui:TuiOptionsgainssmart_case,TuiState::applytakes the rows rather than a count,marked()returns aslice, and
SelectionTargetis new. Theyurucommand line is unchanged.Adds
unicode-segmentationfor grapheme cluster boundaries.--live-smart-caseis a preview flagOff by default, documented in
docs/fzf-compat.mdas changeable orremovable, with its known defects written out. Re-evaluating smart case per
keystroke means a result set computed under the previous policy can still be
on screen; the default keeps 0.1.x behaviour.
Testing
304 -> 426 tests.
golden_ranking.rsgained mixed-case coverage: its fivetests passed byte-identical through a change to default-path ranking order,
so the ranking guard was blind to case folding.
scripts/qa/adds the harnesses that found what the unit suite did not - adifferential output comparison against a baseline binary, an interleaved A/B
benchmark, and pty drivers that race keystrokes against in-flight searches.
Three rounds of independent review by Codex (gpt-5.6-sol) across ranking,
Unicode and concurrency lenses fed into this branch.
Not done here
Not soaked. A rewritten selection model and a changed ranking default want a
week of real use before tagging. The remaining audit backlog - match counter,
ANSI-capable preview, inline
--height, the serial walker - is recorded inthe working notes and not attempted here.