feat: support JSON subpaths in programmatic term-query constructors

[stumpylog]

Summary

Related to #434 — that issue asks about fuzzy matching over an entire JSON document the way you would a text field; this PR delivers fuzzy (and other term-based) matching on a specific, named JSON subpath, which is narrower.

Query.term_query(schema, "notes.user", "alice") currently raises ValueError: Field 'notes.user' is not defined in the schema. even when notes is a JSON field containing a user key — the only way to query a JSON subpath today is the string query parser (index.parse_query("notes.user:alice")). This PR brings the programmatic Query constructors closer to what the parser can already do for this case, though it isn't full parity — see the two gaps called out below.

• Query.term_query, term_set_query, phrase_query, phrase_prefix_query, fuzzy_term_query, and Searcher.doc_freq now accept a JSON subpath as field_name (e.g. "attrs.user").
• make_term resolves field_name via Schema::find_field instead of an exact-name-only lookup. A field literally named "attrs.user" still takes precedence over the subpath interpretation (this is find_field's own resolution order).
• For a JSON subpath, the Python value is typed the way tantivy's own JSON indexing path types a JSON leaf: a string is tried as a fast numeric/bool/date value before falling back to text, floats and out-of-i64-range ints are normalized/widened to match what the JSON indexer actually stores, and dates are truncated to search precision. Phrase words and fuzzy-query values on a JSON subpath are always treated as text, never as typed fast values, matching how the parser only ever tokenizes phrase content.
• Also exposes JsonObjectOptions::set_expand_dots_enabled() on SchemaBuilder.add_json_field (not previously reachable from Python at all) — needed to exercise the expand_dots-enabled path in tests.
• Two known, documented gaps versus the string query parser:
  • A JSON string leaf whose text looks numeric/bool/date (e.g. "5") is not reachable via term_query. The query parser builds an OR of the fast-value term and the text term for this case; a single Term can't represent that union. Documented on term_query's docstring and in the tutorial, with index.parse_query() given as the workaround.
  • term_query does not tokenize its input the way the parser does — it takes the value as-is, consistent with term_query's existing "not tokenized, pre-tokenize yourself" contract for non-JSON fields. This is intentional, not a bug, but it means a term_query value and a parse_query string aren't always interchangeable even when both resolve to the same field.

Motivation: paperless-ngx builds tantivy queries programmatically, constructing an AST of Query objects rather than a query string — this keeps parity with how its search layer worked under the older Whoosh library, which it's migrating off of. Whoosh's query builder gave fine-grained, predictable control over how each search term was constructed, and reproducing that as closely as possible minimizes the risk of subtly changing what users' existing searches match as part of the migration — a difference in how a query string gets parsed is a much easier way to accidentally break someone's saved search than an explicitly constructed term. JSON-subpath terms were the one leaf type that still forced a fallback to the string parser, reintroducing exactly that risk.

AI disclosure

Per CONTRIBUTING.md's AI policy: this PR was built with AI assistance (Claude, via Claude Code), including a multi-pass review step that caught and fixed a few real bugs (date-truncation precision, phrase queries mistyping numeric-looking words, a missing NaN/inf guard on the float-normalization path) before and during review. I've read through the diff and understand the changes — happy to discuss any part of it.

Test plan

• cargo fmt --check passes

• Full pytest suite passes

• New tests cover: basic and nested JSON subpath term queries, typed values (int/str-fallback/bool/date/u64-range-int/whole-number float folding/fractional float/non-finite float rejection), phrase queries (including a numeric-looking phrase word), phrase-prefix queries, term-set queries, exact-name-over-subpath precedence, unknown-root and non-JSON-field-path error cases, the documented numeric-looking-string-leaf limitation (0 hits via term_query vs. 1 via parse_query), parity against index.parse_query() for equivalent queries, Searcher.doc_freq, fuzzy_term_query on a JSON subpath (including a numeric-looking value), and expand_dots_enabled exercised through both parse_query and term_query directly

• Manually verified the previously-broken scenario (Query.term_query(schema, "notes.user", "alice") on a JSON field) now works as expected

• I have read the contributing guidelines which also contains information about our AI policy.

[cjrh]

How does the upstream tantivy crate deal with this kind of query? I'm trying to figure out whether this feature belongs in tantivy-py or tantivy.

[wallies]

@cjrh To answer your question directly, since you asked the same thing on #434 — I went and read the tantivy 0.26 source we pin.

Upstream has no public API for this. What it has are the primitives, all pub in 0.26:

• Schema::find_field(full_path) -> Option<(Field, &str)> — src/schema/schema.rs:326
• Term::from_field_json_path(field, json_path, expand_dots) — src/schema/term.rs:78
• Term::append_type_and_str / append_type_and_fast_value — src/schema/term.rs:230 / :219
• json_utils::convert_to_fast_value_and_append_to_json_term(&term, text, truncate_date) — src/core/json_utils.rs:263

The logic that composes those into "user-supplied value → correct term" is generate_literals_for_json_object at src/query/query_parser/query_parser.rs:998, and it is private.

So: expressible in Rust, but only by reassembling the parser's private logic out of public parts — which is exactly what this PR does. It isn't duplicating an upstream API; there is no upstream API to use.

The detail that decides the design question. Look at query_parser.rs:1023-1042: the parser pushes both the fast-value term and the tokenized text term into logical_literals, i.e. it builds an OR. That is the structural reason for the limitation @stumpylog documented around "5" — a single Term cannot express that union. It isn't an implementation shortcut, and no amount of work in tantivy-py fixes it at the Term level.

That splits the PR cleanly, and I think the two halves have different answers:

• Binding ergonomics — accepting "attrs.user" as field_name, and exposing expand_dots_enabled on add_json_field. Squarely tantivy-py's business. JsonObjectOptions::set_expand_dots_enabled is public upstream and simply was never wired into the Python API; that's a plain binding gap worth closing on its own merits.
• Value-typing logic — the string→fast-value coercion, f64 normalization, u64 widening, date truncation. This is a hand-copy of upstream private logic, and it's the real maintenance liability: if tantivy changes JSON typing in 0.27, this copy drifts silently and starts producing terms that don't match what the indexer stored, with no test here that would catch it.

My suggestion: land it, because there's a concrete need and no upstream API to wait for, but open an issue upstream asking for a public helper that returns the term set (Vec<Term>, or a ready Query) for a JSON path + value. Then tantivy-py deletes its copy when that ships. The PR already names generate_literals_for_json_object in its doc comment, which is the right instinct — a link to the upstream issue next to it would make the deletion trigger obvious.

---

Findings, all minor — I found no correctness bugs:

**1. This is about locking the behavior in, not about bugs:

• term_set_query on a JSON subpath — the docstring and PR summary both claim support, no test.
• phrase_prefix_query on a JSON subpath — same.
• The entire Value::F64 → NumericalValue::normalize() arm in make_json_path_term, including all three of its I64/U64/F64 outcomes. Nothing in the suite passes a float. The whole-number-folding case (term_query(schema, "attrs.count", 5.0) matching an integer leaf 5) is the interesting one and is exactly the kind of thing an upstream change would break silently.
• is_expand_dots_enabled() as consumed by make_json_path_term. test_json_field_expand_dots_enabled only exercises expand_dots through parse_query, never through term_query — so the line in the new code that reads the option is untested, even though the PR summary gives "needed to exercise the expand_dots-enabled path in tests" as the reason for adding the option.

2. make_term_for_phrase_word is also used by fuzzy_term_query, but its doc comment says it's "for a single word inside a phrase (Query.phrase_query / Query.phrase_prefix_query)". The behavior is right — you clearly intended fuzzy to be text-only, and there's a test for it — but the name and comment will mislead the next reader. Something like make_term_text_only with the rationale kept would age better.

3. "parity with the parser" overstates it, in the summary rather than the code. Two ways it isn't parity: the union thing above, and the parser tokenizes via the field's tokenizer (query_parser.rs:1031-1036) where this appends the raw string. The latter is correct and consistent with term_query's existing "not tokenized, pre-tokenize yourself" contract — but test_term_query_json_subpath_matches_parse_query only uses values that tokenize to themselves, so it demonstrates something narrower than the name suggests. The docstrings are accurate; it's just the framing.

4. "Closes #434" may be a stretch. That issue asks about fuzzy matching over an entire JSON doc the way you would a text field; this delivers fuzzy on a specific subpath. Worth confirming with the reporter rather than auto-closing.

The fast: bool parameter added to the add_json_field stub is a pre-existing .pyi gap, unrelated to this feature but correct — no objection, just noting it for whoever reads the diff.

[cjrh]

So: expressible in Rust, but only by reassembling the parser's private logic out of public parts — which is exactly what this PR does. It isn't duplicating an upstream API; there is no upstream API to use.

I know that upstream doesn't have it yet, nor do we. My question is: should this new functionality go into upstream or us? Why us? I would expect this functionality would be more widely applicable than just the python wrapper. Why must it go into the python wrapper specifically?

[cjrh]

Query.term_query(schema, "notes.user", "alice")

I am unsure about whether to use the existing Query.term_query for this. To what degree are we confident that upstream would extend their term_query in this way? If upstream is more likely to use a different method, or a new method, then we'd find ourselves in a pickle because we made a release with the new functionality in the term_query method. This is a good example of why I would want to aligned with upstream.

[stumpylog]

Honestly I made the code and pr here because we had a need for it and this repo had been much more responsive. I didn't really consider it some code would be better in the main crate

But in happy to try and help align with the main project however I can. Is there a process? Create a discussion thread or issue?

[cjrh]

Their discord might be most direct way https://discord.gg/MT27AG5EVE

[stumpylog]

Just for posterity: https://discord.com/channels/908281611840282624/1536788223316205708

It doesn't seem like the most active community sadly.

[cjrh]

No worries, I added a comment and tagged people. They're usually pretty responsive.

[cjrh]

@stumpylog there is a response on the discord

[stumpylog]

Thanks, a little afk, but I'll have a look soon.

[stumpylog]

Coming back to this after a bit, sorry for the gap. Went back through the review, agreed on all four points. I will:

• Add the missing test cases: term_set_query and phrase_prefix_query on a JSON subpath, the Value::F64 normalize path (including the whole-number-folding case), and expand_dots_enabled exercised through term_query itself, not just parse_query.
• Rename make_term_for_phrase_word (it's also used by fuzzy_term_query, the current name/comment implies phrase-only) and fix the doc comment.
• Reword the summary's "parity with the parser" claim to name the two real gaps instead: the OR-of-terms limitation ("5"-as-text) and that term_query doesn't tokenize where the parser does.
• Unlink "Closes Fuzzy queries with JSON field #434", it's old and this only covers fuzzy on a named subpath, not fuzzy over a whole JSON doc.

Also found one more while going back over this: the Value::F64 branch doesn't guard against non-finite values (NaN/inf) the way the actual indexing path does (json_utils.rs's JSON-leaf indexing silently drops a non-finite float, never writing a term for it). Right now term_query would build a term for float('nan') that can never match anything, since no such term was ever indexed. Not a crash, just a query that quietly matches nothing. I will add the same guard and a test for it.

Separately, re: whether any of this belongs in tantivy itself — I raised generate_literals_for_json_object on Discord as the thing to make public, and that was the wrong target. My mistake was reasoning from the docstring I wrote here ("mirrors tantivy's query parser"), not from what this PR's code actually needs. generate_literals_for_json_object is the parser's own logic, and it tokenizes the value, which doesn't fit term_query's contract here (pre-tokenize yourself, one term, not tokenized). Once I actually checked what calls this PR downstream (a query layer that pre-tokenizes itself and just needs JSON field resolution), it was clear that function was never going to help this PR either way, and nothing downstream is waiting on it.

There's a small, real overlap with a different private tantivy function though: index_json_value in json_utils.rs (still pub(crate)). The Value::F64 normalize and Value::Date truncate branches here mirror two of its match arms (the F64 one being the arm I'd missed a guard on above). Both sides call the same already-public primitives (NumericalValue::normalize(), .truncate()), so I don't think this is a big deal, just wanted to point at the right function this time instead of the parser one.