Skip to content

Store page pointers as dense ordinals (format 1.3) - #35

Open
hadashiA wants to merge 3 commits into
mainfrom
claude/page-ordinals
Open

Store page pointers as dense ordinals (format 1.3)#35
hadashiA wants to merge 3 commits into
mainfrom
claude/page-ordinals

Conversation

@hadashiA

@hadashiA hadashiA commented Aug 26, 2026

Copy link
Copy Markdown
Owner

Every on-disk page pointer (index roots, node siblings, internal-node children, overflow blob refs, secondary-index PageRefs) now stores a dense page ordinal assigned in flush order instead of a byte offset. A page directory section at the end of the file maps ordinal -> offset (~0.2% of file size at 4 KB pages), back-patched into the header.

The reader's page cache is indexed directly by ordinal:

  • the ConcurrentDictionary map becomes a plain Entry?[] — a cache hit is one volatile array read + a refcount increment; no hashing, no key comparison, no per-entry node objects
  • publish/evict become single CAS operations on the slot
  • the ghost set becomes an eviction-epoch int[] instead of a dictionary (allocation-free, exact FIFO window)

IPageLoader now takes the byte offset directly (translated through the directory only on cache misses), so loader implementations are unchanged in behaviour.

Breaking format change: the builder always writes 1.3 and the reader accepts 1.3 only — older files fail with an explicit StorageFormatException asking for a rebuild. Key digests (1.1) and Eytzinger layout (1.2) remain per-page flags, orthogonal to this.

Benchmarks (same session, Apple M5 / .NET 10, per 1000 lookups)

Method before after delta
FindByKey (10k) 19.99 us 14.89 us -25.5%
FindByKey_RandomKeys 27.27 us 22.71 us -16.7%
FindByKey_RandomKeys_NoRepeat 58.87 us 50.92 us -13.5%
FindByKey_ParallelSpread (8T) 132.12 us 108.14 us -18.2%
FindByKey_1M 31.59 us 23.96 us -24.2%
FindByKey_1M_RandomKeys 80.57 us 56.01 us -30.5%

Open + first read also drops from 117.6 us / 652 KB allocated to 33.2 us / 19 KB, since the cache no longer preallocates dictionary buckets.

All 85 tests pass. The README benchmark table/charts are intentionally left as-is in this PR (the numbers there come from a different machine); they can be refreshed in one sweep later.

🤖 Generated with Claude Code

hadashiA and others added 3 commits August 25, 2026 11:46
Every on-disk page pointer — index roots, node siblings, internal-node
children, overflow blob refs, secondary-index PageRefs — now stores a
dense page ordinal assigned in flush order instead of a byte offset. A
page directory section at the end of the file maps ordinal -> offset
(page_count x 8 bytes, ~0.2% of file size with 4 KB pages), with its
position and the page count back-patched into the header.

The reader's page cache is now indexed directly by ordinal: the
ConcurrentDictionary map becomes a plain Entry?[] (a cache hit is one
volatile array read plus a refcount increment — no hashing, no key
comparison, no per-entry node objects), publish/evict become single CAS
operations on the slot, and the ghost set becomes an eviction-epoch
array instead of a dictionary.

IPageLoader now receives the byte offset directly (translated through
the directory only on cache misses), so loader implementations stay
offset-based and unchanged in behaviour.

The builder always writes format 1.3, and the reader accepts 1.3 only:
older files fail with an explicit StorageFormatException asking for a
rebuild. Key digests (1.1) and the Eytzinger digest layout (1.2) remain
per-page flags, orthogonal to the pointer format.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Re-measures every benchmark class in one session on the ordinal-indexed
page cache. Point lookup 15.5 ns/query, count 0.8 µs; open + first read
drops from 117.6 us / 652 KB allocated to 33.2 us / 19 KB because the
cache no longer preallocates dictionary buckets.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant