SOLR-17780: Add support for scalar quantized dense vectors #3385

liangkaiwen · 2025-06-11T21:13:04Z

https://issues.apache.org/jira/browse/SOLR-17780

Description

Lucene 9.9 codec includes an implementation of scalar quantized dense vectors. This is not yet supported in Solr.

Solution

Add a new schema type ScalarQuantizedDenseVectorField
Add usage of Lucene99ScalarQuantizedVectorsFormat / Lucene99HnswScalarQuantizedVectorsFormat in SchemaCodecFactory. This incorporates the respective index reader/writers

Tests

TODO

Checklist

Please review the following and check all that apply:

I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
I have created a Jira issue and added the issue ID to my pull request title.
I have given Solr maintainers access to contribute to my PR branch. (optional but recommended, not available for branches on forks living under an organisation)
I have developed this patch against the main branch.
I have run ./gradlew check.
I have added tests for my changes.
I have added documentation for the Reference Guide

liangkaiwen · 2025-06-11T21:43:15Z

I still need to add tests and update documentation

solr/core/src/java/org/apache/solr/schema/ScalarQuantizedDenseVectorField.java

dsmiley · 2025-06-15T03:38:15Z

solr/core/src/java/org/apache/solr/schema/ScalarQuantizedDenseVectorField.java

+            this.confidenceInterval = Lucene99ScalarQuantizedVectorsFormat.DYNAMIC_CONFIDENCE_INTERVAL;
+        }
+
+        this.confidenceInterval = ofNullable(args.get(CONFIDENCE_INTERVAL))


this overwrites the confidenceInterval set when useDynamicConfidenceInterval==true above

good catch. I was hoping to make any explicit confidence interval override the dynamic behavior, however with both absence of arg and the default being null I think I will flip the behavior (use dynamic if specified otherwise confidence interval if specified otherwise default confidence interval if none provided)

Okay but then let's check if both are set and throw an exception.

If neither are set, it should default to the default behavior (scaled confidence interval) - which according to lucene spec is passing a null confidence interval

dsmiley · 2025-06-15T03:51:33Z

solr/core/src/java/org/apache/solr/core/SchemaCodecFactory.java

@@ -126,16 +131,41 @@ public DocValuesFormat getDocValuesFormatForField(String field) {
          public KnnVectorsFormat getKnnVectorsFormatForField(String field) {
            final SchemaField schemaField = core.getLatestSchema().getFieldOrNull(field);
            FieldType fieldType = (schemaField == null ? null : schemaField.getType());
-            if (fieldType instanceof DenseVectorField vectorType) {
+            KnnVectorsFormat delegate;


The instanceof checks can be a code smell. I look at all this and wonder: wouldn't it make sense for a method DenseVectorField.buildKnnVectorsFormat() to exist? Or just a getter if it's built in the field type's init()... which would also mean no need to have each of those fields (e.g. no bits, ...) with their getters either.

reworked it. Let me know if this is along the lines of what you were thinking

Much better. Did you consider init() also creating the codec to thus avoid the need corresponding fields, getters, setters?

I'm not quite sure I follow. You mean the ScalarQuantizedDenseVectorField init() or the SchemaCodecFactory init()?

ScalarQuantizedDenseVectorField.init() (and its subclass, DenseVectorField) creating a KnnVectorsFormat that later can simply be returned.

That's probably true. However it does look like the getters are used in the DenseVectorField tests as KnnVectorsFormat doesn't expose the internal setting values. Personally I think it's ok to leave patterned as-is and having getters affords some level of flexibility and transparency.

But I can make this change if you feel strongly about it

al params and use map remove

alessandrobenedetti

I believe the overall idea to be ok, having the quantised format under a dedicated new field type, given the amount of parameters make sense from a Solr code perspective (the user on the schema won't see much difference).

I left some comments, the 'vectorField.buildKnnVectorsFormat' doesn't look all right to me, for the quantised one.
I think it's just a misunderstanding of how the formats work, so it should be easily solvable.
Once tests are added I think this Pull Request would be quickly good to go!

alessandrobenedetti · 2025-07-02T10:53:12Z

solr/core/src/java/org/apache/solr/schema/DenseVectorField.java

@@ -64,6 +65,7 @@
 public class DenseVectorField extends FloatPointField {
  private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
  public static final String HNSW_ALGORITHM = "hnsw";
+  public static final String FLAT_ALGORITHM = "flat";


'VECTOR_STORAGE_ALGORITHM' maybe?
Is it used somewhere?

reading it more we are mixing up the 'knn' algorithm (only HNSW supported right now), with the 'vector storage' (flat, scalarQuantised and binaryQuantised

alessandrobenedetti · 2025-07-02T10:56:43Z

solr/core/src/java/org/apache/solr/schema/ScalarQuantizedDenseVectorField.java

+    public static final String BITS = "bits"; //
+    public static final String CONFIDENCE_INTERVAL = "confidenceInterval";
+    public static final String DYNAMIC_CONFIDENCE_INTERVAL = "dynamicConfidenceInterval";
+    public static final String COMPRESS = "compress"; // can only be enabled when bits = 4 per Lucene codec spec


these first four lines are param names if I'm not mistaken.

Maybe we can call them '_PARAM' e.g. 'BITS_PARAM'

Or add a comment line at the beginning that clearly group them as param names,
it's a minor though, but can increase code readability

to be fair, checking the original DenseVectorField, the same renaming could help there as well I suspect

alessandrobenedetti · 2025-07-02T11:10:56Z

solr/core/src/java/org/apache/solr/schema/ScalarQuantizedDenseVectorField.java

+    @Override
+    public KnnVectorsFormat buildKnnVectorsFormat() {
+        final String knnAlgorithm = getKnnAlgorithm();
+        if (KNN_ALGORITHM.equals(knnAlgorithm)) {


mmmm not sure about this part:

The only knn algorithm supported right now is 'HNSW'.
While for vector storage, there's flat (what currently supported), scalar quantised and binary quantised.
Effectively in Lucene HNSW vector formats define how to build and read the HNSW graph (where each vector ID is a node).
Vector storage formats define how to store the vectors themselves.

The vector storage format is directly (and only) used by a the graph format parent ( I had in my to-do list to re-organise the package to easily differentiate graph from storage formats)

Lucene99HnswVectorsFormat uses Lucene99FlatVectorsFormat.
• Store for each field the vector metadata (.vemf) and vector data (vec).
• It has a dedicated writer and reader, dependent on the vector encoding (BYTE or
FLOAT32).
• It has its scorer (that just uses the supplied similarity).

Lucene99HnswScalarQuantizedVectorsFormat uses Lucene99ScalarQuantizedVectorsFormat (for scalar quantisation).
• Lossy compression from float32 to a number of bits (4 or 7)
• Raw disk increases (raw + quantized vectors).
• Off-heap memory decreases as quantised vectors are loaded.
• Search is faster.
• It has a dedicated writer and reader, dependent on the bits to use for quantisation.

Given that, I suspect with this contribution we want to differentiate between:
Lucene99HnswVectorsFormat and Lucene99HnswScalarQuantizedVectorsFormat (without manipulating directly the vector storage).

-> the if then else should be changed I suspect, I don't understand it as it is right now as we are mixing a graph format (Lucene99HnswScalarQuantizedVectorsFormat already using the scalar quantised storage format) and a vectore storage format (Lucene99ScalarQuantizedVectorsFormat)

From an initial review the ScalarQuantizedDenseVectorField should always return the Lucene99HnswScalarQuantizedVectorsFormat.

The traditional DenseVectorField, should always return Lucene99HnswVectorsFormat

Add schema type and codec usage for scalar quantized dense vectors

a9d1f2a

github-actions bot added the cat:schema label Jun 11, 2025

add licenese header to new class

974d4b2

dsmiley self-requested a review June 15, 2025 04:13

dsmiley reviewed Jun 15, 2025

View reviewed changes

kliang78 added 2 commits June 23, 2025 11:15

rework confidence interv

69ce5f7

al params and use map remove

refactor constrtuction of KnnVectorsFormat by type

4f6d163

alessandrobenedetti requested changes Jul 2, 2025

View reviewed changes

SOLR-17780: Add support for scalar quantized dense vectors #3385

Are you sure you want to change the base?

SOLR-17780: Add support for scalar quantized dense vectors #3385

Uh oh!

Conversation

liangkaiwen commented Jun 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Solution

Tests

Checklist

Uh oh!

liangkaiwen commented Jun 11, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alessandrobenedetti left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

liangkaiwen commented Jun 11, 2025 •

edited

Loading