Store page pointers as dense ordinals (format 1.3) - #35
Open
hadashiA wants to merge 3 commits into
Open
Conversation
Every on-disk page pointer — index roots, node siblings, internal-node children, overflow blob refs, secondary-index PageRefs — now stores a dense page ordinal assigned in flush order instead of a byte offset. A page directory section at the end of the file maps ordinal -> offset (page_count x 8 bytes, ~0.2% of file size with 4 KB pages), with its position and the page count back-patched into the header. The reader's page cache is now indexed directly by ordinal: the ConcurrentDictionary map becomes a plain Entry?[] (a cache hit is one volatile array read plus a refcount increment — no hashing, no key comparison, no per-entry node objects), publish/evict become single CAS operations on the slot, and the ghost set becomes an eviction-epoch array instead of a dictionary. IPageLoader now receives the byte offset directly (translated through the directory only on cache misses), so loader implementations stay offset-based and unchanged in behaviour. The builder always writes format 1.3, and the reader accepts 1.3 only: older files fail with an explicit StorageFormatException asking for a rebuild. Key digests (1.1) and the Eytzinger digest layout (1.2) remain per-page flags, orthogonal to the pointer format. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Re-measures every benchmark class in one session on the ordinal-indexed page cache. Point lookup 15.5 ns/query, count 0.8 µs; open + first read drops from 117.6 us / 652 KB allocated to 33.2 us / 19 KB because the cache no longer preallocates dictionary buckets. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This reverts commit 9d3e662.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every on-disk page pointer (index roots, node siblings, internal-node children, overflow blob refs, secondary-index PageRefs) now stores a dense page ordinal assigned in flush order instead of a byte offset. A page directory section at the end of the file maps ordinal -> offset (~0.2% of file size at 4 KB pages), back-patched into the header.
The reader's page cache is indexed directly by ordinal:
ConcurrentDictionarymap becomes a plainEntry?[]— a cache hit is one volatile array read + a refcount increment; no hashing, no key comparison, no per-entry node objectsint[]instead of a dictionary (allocation-free, exact FIFO window)IPageLoadernow takes the byte offset directly (translated through the directory only on cache misses), so loader implementations are unchanged in behaviour.Breaking format change: the builder always writes 1.3 and the reader accepts 1.3 only — older files fail with an explicit
StorageFormatExceptionasking for a rebuild. Key digests (1.1) and Eytzinger layout (1.2) remain per-page flags, orthogonal to this.Benchmarks (same session, Apple M5 / .NET 10, per 1000 lookups)
Open + first read also drops from 117.6 us / 652 KB allocated to 33.2 us / 19 KB, since the cache no longer preallocates dictionary buckets.
All 85 tests pass. The README benchmark table/charts are intentionally left as-is in this PR (the numbers there come from a different machine); they can be refreshed in one sweep later.
🤖 Generated with Claude Code