fix(retriever): stream embeddings to the memmap during encode instead of buffering the whole corpus

[AmirF194]

Fixes #233.

Root cause

Encoder.encode() (flashrag/retriever/encoder.py:70-75) appends every batch's embedding array to a python list and only calls np.concatenate(query_emb, axis=0) once all batches are done, so the list of batches and the freshly concatenated array are simultaneously resident: a transient ~2x peak over the full corpus.

Index_Builder.encode_all() (index_builder.py:484-493) then holds that single fully materialized array as the only in-memory copy of the corpus embeddings for the rest of the run. _save_embedding() (index_builder.py:472-482), which writes to a memmap, only runs after encode_all() returns, so it gives no memory relief on the run that actually OOMs: the process is killed inside encode_all() itself, before --save_embedding ever gets a chance to run. That workaround only helps a future re-run skip re-encoding, which the reporter never reaches.

Fix

For the single-GPU, non-SentenceTransformer path (the reporter's case: bge-base, no multi-GPU), encode_all() now pre-allocates the on-disk memmap (self.embedding_save_path) and writes each batch into it directly as it's produced, so peak RAM is O(one batch) instead of O(corpus), independent of --save_embedding. build_dense_index() skips the redundant post-hoc _save_embedding() call for this path and removes the memmap file afterward when --save_embedding wasn't requested, keeping the on-disk footprint unchanged from before.

The multi-GPU (DataParallel) path and the CLIP dual-modal path (encode_all_clip) still call the same accumulate-then-concatenate pattern and were left as follow-up: the reporter's repro (and every OOM report on this issue) is the single-GPU dense-retriever path, and streaming those two would need separate handling (multi-GPU splits work across processes; CLIP builds two modalities into one array).

Verification

• Reproduced the actual failure mode in a memory-constrained container: a synthetic encoder standing in for the model (no GPU/weights needed) drives Index_Builder.encode_all() through a ~500MB corpus under a 700MB hard memory ceiling. On main this gets OOM-killed (exit 137) inside encode_all(); with this patch, the same corpus and ceiling complete successfully.
• Added tests/test_index_builder_streaming.py locally (a synthetic-encoder unit test asserting the memmap receives every batch in order and is readable on disk afterward) and confirmed it fails on main and passes on this branch; it isn't included in this diff because tests/* is in this repo's .gitignore, so I'm stating that plainly rather than force-adding a file the repo excludes. Happy to add it a different way if there's a preferred location.
• This repo has no CI test/lint job (only a PyPI-publish workflow), so there's no existing gate to run beyond the above.
• Not verified: the actual reporter's 21M-passage/126GB-RAM scenario, or the multi-GPU and CLIP paths, which are unchanged.