Kind of improvement
perf
Affected area
sindi
Summary
SINDI's build memory footprint is hard to satisfy at scale. Building ~4.3M rows (average length 268) currently peaks around 29 GB of memory. For datasets in the 10M+ range, it is difficult to find a machine with enough memory to build the index at all.
This is not meant to replace the current implementation, but to provide a configurable option that lets users choose a lower-memory build path when needed.
Some degradation in build time is an acceptable trade-off for a lower peak memory footprint.
Expected benefit
Lower peak RSS during SINDI build, making it feasible to build indexes for 10M+ row datasets on commonly available machines. Acceptable trade-off: increased build wall-clock time. Existing build behavior remains available as the default.
Benchmark evidence
- Dataset: ~4.3M rows, average vector length 268
- Current build peak memory: ~29 GB
- Target scale: 10M+ rows (blocked today by insufficient machine memory)
Additional context
n/a
Drafted with AI assistant: Claude Code:GLM-5.2
Kind of improvement
perf
Affected area
sindi
Summary
SINDI's build memory footprint is hard to satisfy at scale. Building ~4.3M rows (average length 268) currently peaks around 29 GB of memory. For datasets in the 10M+ range, it is difficult to find a machine with enough memory to build the index at all.
This is not meant to replace the current implementation, but to provide a configurable option that lets users choose a lower-memory build path when needed.
Some degradation in build time is an acceptable trade-off for a lower peak memory footprint.
Expected benefit
Lower peak RSS during SINDI build, making it feasible to build indexes for 10M+ row datasets on commonly available machines. Acceptable trade-off: increased build wall-clock time. Existing build behavior remains available as the default.
Benchmark evidence
Additional context
n/a
Drafted with AI assistant: Claude Code:GLM-5.2