Skip to content

Add Eytzinger layout option for key digests - #33

Merged
hadashiA merged 1 commit into
mainfrom
claude/eytzinger-digests
Aug 16, 2026
Merged

Add Eytzinger layout option for key digests#33
hadashiA merged 1 commit into
mainfrom
claude/eytzinger-digests

Conversation

@hadashiA

@hadashiA hadashiA commented Aug 8, 2026

Copy link
Copy Markdown
Owner

Note

Rebased onto main (#31 and #32 are both merged). The readers now carry three digest search paths: Eytzinger descent (this PR, per-page flag) → SIMD hybrid lower bound from #32 (sorted digests) → classic mixed binary search (netstandard / non-SIMD fallback).

Summary

Adds DatabaseBuilder.EytzingerDigests (default off): each node's key digest array is stored as a complete binary tree in Eytzinger (BFS) order, padded with ulong.MaxValue to 2^k - 1 slots, instead of sorted order. The digest search becomes a fully branch-free descent —

i = 1;
while (i <= size) i = 2 * i + (digest[i] < keyDigest ? 1 : 0);
rank = i - size - 1;   // closed-form lower bound, thanks to the complete-tree padding

— with the top levels of the tree clustered in the first cache line. The rank is then resolved to an entry through the run of equal digests with full key comparisons (entry metadata and keys stay in sorted order; only the digest array is permuted).

Format notes:

  • New per-page flag NodeFlags.EytzingerDigests (bit 9); files using it are marked 1.2. Older readers cannot parse such pages. Files built without the option are byte-identical to before.
  • The MaxValue padding costs up to 2x the digest area at the worst entry count (just past a power of two); ~+3% page space in the 4KB/Int64 benchmark shape.
  • The padding value is also a legal digest; correctness doesn't depend on distinguishing them (a real MaxValue digest sorts last, and the rank walk is bounded by the entry count).

Benchmark — a workload-dependent trade

M4 / .NET 10, ShortRun. RandomKeys repeats the same 1000-key sequence every op (the branch predictor memorizes it); RandomKeys_NoRepeat (added in #32) never repeats — the realistic model of random access.

Benchmark sorted digests Eytzinger
FindByKey (fixed key) 16.4 µs 22.2 µs (+35%)
FindByKey_RandomKeys (repeating) 30.0–31.5 µs 42.6 µs (+38%)
FindByKey_RandomKeys_NoRepeat 86.0 µs 44.1 µs (-49%)

The mechanism cuts both ways: the descent has no mispredictable branches (halves the cost when the predictor is useless) but its loads form a serial dependency chain (a correctly-predicted binary search speculates loads ahead and wins when the access pattern is learnable). Hence opt-in, default off: workloads dominated by unpredictable point lookups can halve them; predictable/looping access should keep the sorted layout.

Tests: 85/85 pass after the rebase (including #32's KeyDigestSearchTest against the merged readers), with new EytzingerDigestsTest coverage for padded-tree boundaries (1..8 entries), digest-collision runs, range/count bounds, non-unique secondary indexes, and iterators.

🤖 Generated with Claude Code

Base automatically changed from claude/readonly-db-optimization-19c03f to main August 16, 2026 05:34
@hadashiA
hadashiA force-pushed the claude/eytzinger-digests branch from 2af2454 to 02ca03d Compare August 16, 2026 05:39
@hadashiA
hadashiA merged commit da23980 into main Aug 16, 2026
1 check passed
@hadashiA
hadashiA deleted the claude/eytzinger-digests branch August 16, 2026 07:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant