SOLR-5707: Lucene Expressions in Solr #1244

risdenk · 2022-12-16T15:07:27Z

https://issues.apache.org/jira/browse/SOLR-5707

This is me taking Hoss's VSP patch from https://issues.apache.org/jira/browse/SOLR-5707 and trying to look at it again.

./gradlew check -x test -Pvalidation.errorprone=true -Pvalidation.sourcePatterns.failOnError=false passes :)
nocommit are still in there
do we NEED to deal w/ score? can we do score different now that its been years later
does ./gradlew -p solr/core test --tests ExpressionValueSourceParserTest pass?
do all tests pass?
how does this perform - this is my real goal here compared to other say boostfunctions

risdenk · 2024-10-30T19:07:48Z

Not looking at this anymore

dsmiley · 2025-05-01T20:43:56Z

I commented in JIRA about my interest in furthering this. I'm sure there are some nice-to-have's (score function access, documentation), but simply working code & tested is enough to merge. I can brush the dust off get this PR mergeable; what do you say @risdenk ? This functionality plugs in nicely, showcasing Solr's great abstractions without needing to hack on any Solr plumbing. I would later expand the usage to support access to text fields via a DocTransformer use case but I think another PR can do that on top of the fine work already done here.

risdenk · 2025-05-01T20:46:06Z

Go for it @dsmiley I ended up not needing it but it did work when I was trying it

# Conflicts: # solr/core/src/java/org/apache/solr/response/transform/ValueSourceAugmenter.java

dsmiley · 2025-05-06T17:34:51Z

Updating the PR to main was easy. Note that there is another PR #3340 that will overlap with this one which makes it possible for a DocTransformer to access the score. The failing tests relate to sorting by an expression that makes use of the score. If we mark those tests with an Ignore and add a follow-up to improve that matter, then I think we have something mergeable. Progress not perfection; right?

Addressing the score issue is complicated by the fact that Lucene's ValueSource API is legacy, has no concept of needsScores, but the replacement DoubleValuesSource does have this concept. Solr is still ValueSource based. I could see doing a switch for Solr 10 but that's for another issue and dev list discussion.

dsmiley · 2025-05-13T04:43:18Z

The ability to reference scores in a custom ValueSource based on DoubleValuesSource won't work with sorting. I filed this PR apache/lucene#14654 to Lucene to trivially fix that issue. But since Lucene 9.x is EOL; I don't know when we might expect that if ever in Solr 9.

# Conflicts: # solr/core/src/java/org/apache/solr/response/transform/ValueSourceAugmenter.java

dsmiley · 2025-07-07T12:42:49Z

@hossman as the original author, maybe you'd like to review this?

dsmiley · 2025-07-08T02:32:54Z

I cherry-picked the Lucene 9.12.2 upgrade (thanks @HoustonPutman ), removed the Ignore annotation from the 2 tests, fixed a minor test bug that one Ignored test was hiding, and confirmed that this Lucene upgrade addressed the lingering concern :-)
I didn't push the Lucene upgrade commit so as to not confuse this PR, and thus those 2 tests will fail.

@houston not sure if you have concerns on this getting into 9.9.

dsmiley · 2025-07-09T05:10:59Z

@AndreyBozhko you might be a good reviewer here

bruno-roustant

Very nice!

bruno-roustant · 2025-07-09T17:22:46Z

solr/solr-ref-guide/modules/query-guide/pages/expression-value-source-parser.adoc

+
+The `ExpressionValueSourceParser` allows you to implement a custom valueSource merely by adding a concise JavaScript expression to your `solrconfig.xml`.
+The expression is precompiled, and offers competitive performance to those written in Java.
+The syntax is a limited subset of JavaScript that is purely numerical oriented, and only certain built-in fuctions can be called.


Typo "fuctions"

AndreyBozhko

This is great - and the Lucene expressions definitely cover the use case I was working on in #3340.

I have a few tests in my PR - mostly around accessing score in function queries in various contexts, and I'm happy to adapt the tests and contribute them here.

A couple of other things that I worked on in #3340 and that would be good to consolidate:

making sure functions can access scores in distributed queries,
exposing score to functions that act as post-filters.

AndreyBozhko · 2025-07-10T18:43:27Z

solr/core/src/java/org/apache/solr/response/transform/ValueSourceAugmenter.java

+      final MutableScorable scorable; // stored in fcontext (when not null)
+      final IntFloatHashMap docToScoreMap;
+      if (context.wantsScores()) { // TODO switch to ValueSource.needsScores once it exists
+        docToScoreMap = new IntFloatHashMap(prefetchSize);


This makes me wish for something like

DocList sortedDocList = docList .subset(docList.offset(), prefetchSize) .sortedByDocId();

and then wrap the sortedDocList.iterator() as a Scorable. This could remove the need for a custom mapping between docids and scores in ValueSourceAugmenter.

The sorting & mapping has to happen somewhere. It's inelegant. Is your point to add a convenience API/method? If we do this in multiple places, then that'd make sense, but not otherwise.

It looked to me that the mapping between docIds and scores is already established in the DocList (so why do the mapping again?), and only the sorting piece is missing. That's why I was thinking of a way to handle just the sorting.

But I agree - if this is the only place, then I'm OK with the logic as long as it works.

dsmiley · 2025-07-10T21:43:58Z

making sure functions can access scores in distributed queries,

What would go wrong if this functionality were used in such? An aside: I have a wish for our tests to be able to run in both modes easily. I've used that strategy at work. The test here is based on our oldest test infrastructure, and that which wouldn't easily support that. I'd like to see further use of the new SolrClientTestRule such as using SolrCloud; there's a JIRA for that.

exposing score to functions that act as post-filters.

I saw that one-liner you added for frange specifically (which I wish wasn't a PostFilter, but I digress). Is there a more general adaptation?

Do you have concerns with the merge-ability of this PR as it stands for 9.9?

AndreyBozhko

What would go wrong if this functionality were used in such?

I think if the expression tried to use the score, the distributed query would fail - see this comment #3340 (comment).

But overall, I'm OK with merging this - since the goal here is to support Lucene Expressions, and not necessarily to support accessing scores in function queries in every possible context.

AndreyBozhko · 2025-07-11T22:53:03Z

solr/core/src/java/org/apache/solr/response/transform/ValueSourceAugmenter.java

+      final MutableScorable scorable; // stored in fcontext (when not null)
+      final IntFloatHashMap docToScoreMap;
+      if (context.wantsScores()) { // TODO switch to ValueSource.needsScores once it exists
+        docToScoreMap = new IntFloatHashMap(prefetchSize);


It looked to me that the mapping between docIds and scores is already established in the DocList (so why do the mapping again?), and only the sorting piece is missing. That's why I was thinking of a way to handle just the sorting.

But I agree - if this is the only place, then I'm OK with the logic as long as it works.

dsmiley · 2025-07-12T02:14:14Z

DocList / DocIterator / DocIterationInfo != a Lucene Scorable (which is an abstract class, BTW; limits creative class hierarchy possibilities). Furthermore the former is an iterator but the latter returns the current state. These things aren't compatible.

@HoustonPutman I don't want to push too last minute for this in 9.9 unless you are comfortable with this as the RM. I think this is a great feature that has waited over 11 years too long.

HoustonPutman · 2025-07-12T02:40:19Z

@dsmiley I just started producing the 9.9.0 release, so let's leave this for 9.10. We don't have to wait months for 9.10 either, we can do it in a month or two and get this out after baking for a little bit.

HoustonPutman · 2025-07-13T05:52:11Z

Actually I’m having issues with the smoketester erroring in the tests, so if you can get this in by Monday and are sure it wont break the tests, go for it. It seems like a new feature, so im not too worried about it breaking anything other than the tests.

New ExpressionValueSourceParser that allows custom function queries / VSPs to be defined in a subset of JavaScript, pre-compiled, and that which can access the score and fields. It's powered by the Lucene Expressions module. ValueSourceAugmenter: score propagation --------- Co-authored-by: Chris Hostetter <[email protected]> Co-authored-by: David Smiley <[email protected]> Co-authored-by: Ryan Ernst <[email protected]> (cherry picked from commit c3b5f57)

dsmiley · 2025-07-14T03:24:34Z

Yep I'm confident in it. It's all merged.

risdenk self-assigned this Dec 16, 2022

SOLR-5707: Lucene Expressions in Solr

443e84b

risdenk force-pushed the SOLR-5707 branch from eb23a16 to 443e84b Compare December 16, 2022 15:19

risdenk closed this Oct 30, 2024

Merge branch 'main' into fork/risdenk/SOLR-5707

ab38d9c

# Conflicts: # solr/core/src/java/org/apache/solr/response/transform/ValueSourceAugmenter.java

dsmiley added 2 commits May 6, 2025 20:05

ValueSourceAugmenter from SOLR-15030

2dbfbde

Convert nocommit to TODO

fecb3cd

dsmiley reopened this May 7, 2025

github-actions bot added tests cat:search labels May 7, 2025

dsmiley added 7 commits June 12, 2025 21:16

Merge branch 'main' into fork/risdenk/SOLR-5707

a8c1e44

# Conflicts: # solr/core/src/java/org/apache/solr/response/transform/ValueSourceAugmenter.java

VSA: fix prefetch to consider need for Scorable

e05ebe7

ignore another test that can't be supported yet

00f8435

optimization (avoid needless subIndex)

38d1eb5

Ref Guide page

3270f77

Single expression, and support positional args

f47cfa3

refer to it in function-queries.adoc

164603d

github-actions bot added the documentation Improvements or additions to documentation label Jun 25, 2025

CHANGES.txt

c634796

dsmiley marked this pull request as ready for review July 8, 2025 02:20

un-ignore tests

d6a388e

dsmiley added 2 commits July 8, 2025 23:58

Merge branch 'main' into fork/risdenk/SOLR-5707

dc25024

fix lucene "expressions" javadoc reference in ref guide

8234a81

github-actions bot added dependencies Dependency upgrades client:solrj tool:build labels Jul 9, 2025

dsmiley mentioned this pull request Jul 9, 2025

[SOLR-15030] Add score() function that returns the score of the current hit #3340

Open

7 tasks

bruno-roustant approved these changes Jul 9, 2025

View reviewed changes

AndreyBozhko reviewed Jul 10, 2025

View reviewed changes

code review feedback

992ba1a

AndreyBozhko reviewed Jul 11, 2025

View reviewed changes

dsmiley added 2 commits July 13, 2025 16:34

typo/formatting

b53e50f

Merge branch 'main' into fork/risdenk/SOLR-5707

2c8967e

dsmiley merged commit c3b5f57 into apache:main Jul 13, 2025
4 checks passed

SOLR-5707: Lucene Expressions in Solr #1244

SOLR-5707: Lucene Expressions in Solr #1244

Uh oh!

Conversation

risdenk commented Dec 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

risdenk commented Oct 30, 2024

Uh oh!

dsmiley commented May 1, 2025

Uh oh!

risdenk commented May 1, 2025

Uh oh!

dsmiley commented May 6, 2025

Uh oh!

dsmiley commented May 13, 2025

Uh oh!

dsmiley commented Jul 7, 2025

Uh oh!

dsmiley commented Jul 8, 2025

Uh oh!

dsmiley commented Jul 9, 2025

Uh oh!

bruno-roustant left a comment

Choose a reason for hiding this comment

Uh oh!

bruno-roustant Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

AndreyBozhko left a comment

Choose a reason for hiding this comment

Uh oh!

AndreyBozhko Jul 10, 2025

Choose a reason for hiding this comment

Uh oh!

dsmiley Jul 10, 2025

Choose a reason for hiding this comment

Uh oh!

AndreyBozhko Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

dsmiley commented Jul 10, 2025

Uh oh!

AndreyBozhko left a comment

Choose a reason for hiding this comment

Uh oh!

AndreyBozhko Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

dsmiley commented Jul 12, 2025

Uh oh!

HoustonPutman commented Jul 12, 2025

Uh oh!

HoustonPutman commented Jul 13, 2025

Uh oh!

Uh oh!

dsmiley commented Jul 14, 2025

Uh oh!

Uh oh!

risdenk commented Dec 16, 2022 •

edited

Loading