Summary
Track and improve train performance of IVF_FLAT_CC and SCANN_DVR.
The train-only path (building the coarse quantizer + index model, excluding
Add) is a significant portion of index build cost for large datasets. This
issue tracks benchmarking the train path and the optimizations applied to it.
Scope
- Benchmark train-only cost for
IVF_FLAT_CC and SCANN_DVR.
- Provide a reproducible benchmark setup supporting multiple local datasets
(start with the SIFT fbin dataset under ~/data/sift).
- Measure train/build-model cost only; do not include Add time.
- Record timing and memory signals for later performance analysis.
Progress
Summary
Track and improve train performance of
IVF_FLAT_CCandSCANN_DVR.The train-only path (building the coarse quantizer + index model, excluding
Add) is a significant portion of index build cost for large datasets. This
issue tracks benchmarking the train path and the optimizations applied to it.
Scope
IVF_FLAT_CCandSCANN_DVR.(start with the SIFT fbin dataset under
~/data/sift).Progress
benchmark/.SCANN_DVRcoarse quantizer training optimized: SuperKMeans(spherical inner-product) enabled by default, recall-equivalent to
Clustering but ~60% faster full build on qwen 4096-dim.
See PR enhance: enable SuperKMeans by default for CPU IVF training #1783.