enhance: reduce block inverted index build memory

[chasingegg]

What this PR does

similar to #1810

• Replace the per-dimension encoded posting vectors used by sealed block indexes with reusable per-worker encoding workspaces.
• Encode posting lists once to compute exact byte offsets, allocate the final contiguous container, then encode again directly into disjoint final ranges.
• Reuse document-ID and BM25 scratch buffers and copy IP values directly from the quantized input span.
• Schedule dimensions in dynamic batches to retain load balancing for skewed posting-list distributions.

Why

The previous block-index build materialized one vector per active dimension and retained the complete encoded payload while allocating and filling the final contiguous container. High-dimensional sparse data therefore paid for millions of vector objects and allocations, duplicate encoded payload, allocator fragmentation, and a final copy.

The new path keeps only one scratch workspace per build worker. It trades a second encoding pass for substantially lower peak memory and fewer allocations.

Benchmark results

All builds used DAAT_MAXSCORE and 8 build threads. Benchmark sources are intentionally not included in this PR.

Synthetic high-dimensional IP data

10.56M rows, 24.8M active dimensions, 771.936M postings, 73.1 postings per row, and fp16 values.

Codec	Build time before	Build time after	Peak RSS before	Peak RSS after	Peak reduction	Index size
block_streamvbyte	52.971 s	49.292 s	23.543 GiB	16.479 GiB	7.064 GiB (30.0%)	4.983 GiB
block_adaptive	51.537 s	48.978 s	19.642 GiB	15.066 GiB	4.576 GiB (23.3%)	3.670 GiB

The build-only memory increment decreased by 40.3% for block_streamvbyte and 33.6% for block_adaptive.

Real BM25 corpora

Results are averages of two runs per version. The four corpora are MS MARCO, NQ, FEVER, and HotpotQA.

• block_streamvbyte: peak RSS decreased by 12.2%-17.1%, build-only memory decreased by 22.0%-28.4%, and build time changed by -3.1% to +0.7%.
• block_adaptive: peak RSS decreased by 11.2%-16.0%, build-only memory decreased by 20.7%-27.1%, and build time increased by 3.6%-6.3% because the second adaptive encoding pass is more CPU intensive.

Final serialized index sizes were identical before and after for every dataset and codec.

Compatibility

• The posting-list wire format and section layout are unchanged.
• Search and deserialization paths are unchanged.
• This only changes the sealed block-index build path; growable/incremental indexes are unaffected.

Validation

• pre-commit run --files src/index/sparse/block_inverted_index.h
• Test Sparse Index Codec and Algo Combinations: 752 assertions passed, covering IP and BM25, all block codecs, search, serialization, and deserialization.

Related to #1730.

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: chasingegg

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [chasingegg]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mergify]

@chasingegg 🔍 Important: PR Classification Needed!

For efficient project management and a seamless review process, it's essential to classify your PR correctly. Here's how:

1. If you're fixing a bug, label it as kind/bug.
2. For small tweaks (less than 20 lines without altering any functionality), please use kind/improvement.
3. Significant changes that don't modify existing functionalities should be tagged as kind/enhancement.
4. Adjusting APIs or changing functionality? Go with kind/feature.

For any PR outside the kind/improvement category, ensure you link to the associated issue using the format: “issue: #”.

Thanks for your efforts and contribution to the community!.

[chasingegg]

#1730

[chasingegg]

/kind improvement

[alexanderguzhva]

makes sense
/lgtm