Retrieve /match candidates per comparable name part

[pudo]

Rebuilds how /match retrieves name candidates from Elasticsearch: one rigour normaliser on both sides, one unit of evidence per name part, and four explicitly named ways a query part may reach an indexed one. Closes #1252, closes #1253, closes #1254.

Index

build_indexable_entity_doc now writes a third derived name field, name_joined: for every matchable name, the comparable forms of its parts concatenated without spaces (alqaeda, vladimirputin). It is a plain keyword set excluded from _source, like name_parts and name_symbols. The name_phonetic field and the names.ngrams trigram sub-field (with its osa-ngram-filter/osa-ngram-analyzer) are removed; the indexer no longer computes metaphones. The names text field and its weak_length_norm similarity stay for /search; name_parts keeps its normalizer because weak aliases share the field. Field sizes on the full default collection: name_joined 89 MB added, name_phonetic 28 MB and names.ngrams 117 MB removed; total store size 2,645 MB vs 2,657 MB.

Query

names_query builds one dis_max (tie_breaker 0) per unique comparable name part across all of the query's names, taken from the same cached entity_names(entity, is_query=True) the scorer uses, so twenty transliterations of one name yield about as many clauses as one name has parts. Inside each dis_max are up to three channels; a document scores each query part once, whichever channel matched and however many aliases carry it, and the outer should sums over parts, so a document ranks by how many query parts it covers, weighted by part rarity. One terms clause on name_joined covers all space-less forms of the query names; weak aliases are queried exactly as before.

The four channels, in plain words:

• exact: the indexed part is the same comparable string (casefolded, latinised where the script allows it, diacritics and punctuation stripped). IDF-scored term.
• fuzzy: the indexed part is within Damerau-Levenshtein AUTO of the query part (0 edits up to 2 chars, 1 edit at 3–5, 2 edits from 6) with the first letter fixed; only when MATCH_FUZZY is on and the part has at least three characters. Run as a constant_score filter so a record with ten spellings of one part scores the hit once, worth a fixed FUZZY_BOOST = 6.0 (about the exact score of a part shared by 25,000 records, so an exact hit on any but the most common tokens outranks an approximate one).
• symbol: rigour tagged both parts with the same known-name identity, nickname, org class etc. IDF-scored dis_max of terms on name_symbols, boost 0.9. The only bridge for scripts that are not latinised (Arabic, Han) and for variants beyond two edits (alexander/aleksandr).
• joined: a query name with its spaces removed equals an indexed name with its spaces removed. terms on name_joined.

Caps: MAX_PARTS = 100, MAX_SYMBOLS_PER_PART = 30 bound the query at roughly 3,200 clauses (observed maxima: 22 parts, 29 symbols on one part). Primary names are visited before aliases so a truncated query keeps its primary name whole.

Two calibrations of clauses outside the name channels were needed once the ×3 match names clause was gone: TYPE_BOOSTS[date] goes from 3.0 to 1.0, because a year-only birth date at 3.0 scored more than an exact two-part name match and over a thousand records share any given year, so they filled the candidate window on their own (US Congress fixture: 98.6% → 86.2% before this change, 98.7% after). The fuzzy clause is wrapped in constant_score because a bare fuzzy clause is rewritten into an OR of its expansions and sums the ones a document carries, which handed heavily aliased records the very alias bonus this query removes ("Andrea Salinas" lost to a record with ten transliterations of Andrey scoring 40 on "andrea" alone). Using the constant_score rewrite instead would have been 3–5× slower on many-part queries because it enumerates every term in the edit budget rather than the top 200.

Recall@50, before/after

Presence of the expected id(s) in a 50-candidate window; router-identical query construction (entity_query, filter_op OR, DEFAULT_SORTS, track_total_hits: false). "Before" is origin/main against the 018 index of default, "after" is this branch against a 019 index of the same collection built with this code (a slightly newer data release: 4,049,275 vs 4,049,113 documents). Fixtures from contrib/validation_report/build/fixtures.

fixture	n	recall@50 before	after	median rank before	after
positives_merged_aliases	200	85.0% (170)	99.5% (199)	2	1
positives_name_bench	246	95.9% (236)	97.6% (240)	1	1
positives_un_treated	732	96.6% (707)	96.6% (707)	1	1
positives_un_untreated	732	96.6% (707)	96.6% (707)	1	1
positives_us_congress_untreated	1321	98.6% (1302)	98.7% (1304)	1	1

Treated UN fixture by whether the name was mangled: name changed 238, 96.2% → 96.2%; name unchanged 494, 96.8% → 96.8%. The 25 UN misses are shared by both queries (entities outside the window for both).

Misses that changed:

• merged aliases: 29 gained, 0 lost. Charles III, Elizabeth II, Dmitry Medvedev, Pope Francis, the Federal Penitentiary Service, Donald Trump ("Don"), Abu Nidal Organisation ("ANO") and other heavily merged records now rank first or near first for their own most-shared alias; previously the window was filled by single-alias records via BM25 on the names text field.
• name bench: 4 gained (Emanuel Macron, Emmanuel Macrone, Emmanuel Marcon → rank 1/1/3; Mark Cooney → Mark Carney rank 26), 5 lost: rimaldiV nituP and Vladimir nitPu (reversed strings, only trigrams could match these), Kim Yong-chol → Kim Yong-chun twice (two edits on a four-letter syllable, outside AUTO), عبد الفتاح السري → السيسي (Arabic single-part edit, first letter fixed and no NAME symbol).
• US Congress: 2 gained (Randy (Duke) Cunningham, Harry E. Mitchell, both rank 1), 0 lost.

Latency

contrib/perf_bench/perf_bench.py run --output … from both trees, same index builds as above, 100 queries, medians of 3 warm runs.

stage	p50 before	after	p95 before	after	max before	after
search (ms)	8.4	8.4	41.9	35.0	99.5	71.7
ES took (ms)	5.0	5.5	38.1	31.1	93.0	67.0
score (ms)	7.5	6.5	70.7	48.2	423.0	543.7

query	tags	took before	after	hits before	after
sparse-ali	sparse	7	4	50	50
sparse-hamas	sparse	3	2	50	36
sparse-hezbollah	sparse	3	2	50	12
sparse-wagner	sparse	5	3	50	50
sparse-putin	sparse	7	5	50	50
sparse-ivan-petrov	sparse	6	5	50	50
sparse-rosneft	sparse	3	3	50	45
sparse-gazprom	sparse	5	3	50	50
sparse-legalentity-bank	sparse	4	2	50	50
names-zakharov-7	many-names	34	20	50	50
names-ermakov-20	many-names	28	58	50	50
names-putin-10	many-names	34	31	50	50
names-rotenberg-5	many-names	38	22	50	50
names-deripaska-6	many-names	17	15	50	50
names-kim-12	many-names	46	25	50	50
names-rosneft-15	many-names	37	38	50	50
names-stroygaz-40	many-names	40	40	50	50
co-sberbank-capital-aliases	many-names	40	13	50	50
dense-putin	dense	30	13	50	50
dense-rotenberg	dense	53	21	50	50
dense-deripaska	dense	19	17	50	50
dense-tupikin	dense	27	21	50	50
dense-rosneft	dense	16	13	50	50
dense-huawei	dense	13	11	50	50
dense-stroygaz	dense	8	21	50	50
dense-negative-person	dense	14	10	50	50
long-name-string	long	93	67	50	50
long-address	long	8	10	50	50
degenerate-diacritics	long	4	6	50	50
degenerate-numeric-name	long	1	0	50	0
degenerate-repeated-words	long	6	2	50	50

names-ermakov-20 (22 unique parts, one fuzzy clause each) is the one query that got materially slower; dense-stroygaz is the other. Hamas and Hezbollah now return 36 and 12 candidates rather than a window padded with trigram matches; the numeric-only company name 12345 returns nothing instead of 50 trigram neighbours.

Known limits

• Common single-token queries ("Ali", "Muhammad", "Maria") have thousands of exact matches for 50 slots; they all tie on score and entity_id order decides. Previously BM25 on alias count decided. Whether entity_values_count should break ties for /match is a separate decision.
• Edits at the first letter (wagner/vagner, gaddafi/qadhafi) are not retrieved unless a NAME symbol bridges them; two-edit first-letter changes need one. A second one-edit clause without the prefix constraint is the documented upgrade if this class matters.
• A merged query whose expansion is only a sub-sequence of a longer indexed name (vladimirputin against an entity that only has "Vladimir Vladimirovich Putin") is not reached; name_joined is exact.
• Two edits on a short part (chol/chun) and single-part Arabic edits are outside AUTO with a fixed first letter.
• The names.ngrams removal drops the reversed-string and split-token cases that trigrams happened to catch.

Removed, and why

representative_names (clause count no longer depends on alias count), SYMBOL_BOOSTS (IDF already ranks org-class and location parts near zero), the is_short rule (short parts get exact and symbol channels; AUTO gives 0 edits under 3 chars anyway), Name.consolidate_names at query time (the one place query and index analysis differed), the match names clause and its ×3 boost (the alias penalty of #1252), the n-gram clauses and analyzer (#1254; trigrams at 70% have 0% overlap on john/jon and admit 20–1000× the documents), the phonetic channel (never the only bridge for a variant pair on the fixtures, unavailable for the scripts where a cross-script bridge is needed), NAME_PHONETIC_FIELD and NAME_NGRAMS_FIELD.

Operational

INDEX_VERSION is bumped to 019: the mapping changed, so deploying this triggers a full reindex of every dataset, and the old query code cannot serve from a 019 index nor the new code from an 018 one. YENTE_MATCH_FUZZY now toggles only the fuzzy sub-clause; exact, symbol and joined matching are always on, so merged names ("alexandervyacheslavovichzakharov") are retrieved regardless of the setting.

Tests

tests/test_queries.py is new: one dis_max per unique part across seven transliterations, fuzzy sub-clause shape and absence when MATCH_FUZZY is off or the part is under three characters, symbol sub-clause presence and cap, joined clause contents, weak alias clause, the 100-part cap, and primary-before-alias ordering. test_mappings.py asserts name_joined is indexed (and name_phonetic is gone); test_match_name_without_spaces now asserts the merged name works with fuzzy on and off. ruff, ruff format and mypy --strict pass; the full suite passes except the four pre-existing httpx_mock fixture errors in test_loader.py/test_updater.py (pytest-httpx not installed in the local environment).

🤖 Generated with Claude Code

[pudo]

Underlying prompt:

I feel like the entity_names function and it's indexing counterpart have gone through a lot of evolution, and some of the contained decisions (representative_names, metaphone use) may not longer be ideal. I would like to come up with a simple alternative that makes ideal use of the available primitives (maybe (!) ngrams, symbols, name objects more generally) from first principles. Some thoughts on this:

• we want to retrieve based on all matchable names, which can be just one (most common) or many. We don't want to ever exceed to clause limit of ES cloud (4096?).
• if a person has N aliases including the name part John, that usually doesn't mean they're a better (more relevant) "john" than a sparser record with just one "john" name part. this is true of both queries and results. the exception here is a person or organisation where the token is actually mentioned many times ("John Mc John"), but this seems exceedingly rare and maybe not a thing the candidate generation query needs to consider?
• the goal is to generate candidates for the symbol-based and fuzzy string matching process. a big ambition is the ability to retrieve candidates with spelling mistakes because then the fuzzy matcher can still make a positive decision. Term searches alone won't quite do this.
• we want to match names across scripts and languages. transliteration of comparable, name symbols, metaphone and ES-side asciifolding are all theories about how we get this done, but some of them are duplicative, and metaphone is badly implemented - can we drop asciifolding and/or metaphone without loss of recall? I am particularly questioning the continued use of asciifolding in this context.
• ngrams and fuzzy matching using a match query serve the same goal, do we ever need both?

Other failure modes:

• Multiple mis-spelled name parts (jon smyth ~ john smith)
• Merged names (alqaeda ~ al qaeda, vladimirputin ~ vladimir putin)

For context, see existing work in the name matching benchmarks, and these issues:

• Candidate retrieval penalises entities with many aliases via the names text clause #1252
• Two normalisation families in the match index: names/ngrams (ES analyzer) vs name_parts (rigour) #1253
• Rework n-gram candidate retrieval: per-token trigrams on comparable parts #1254

The goal here is not to come up with a more complex, but with a really simple, explainable, error-reistant and reliable logic.

Other rules:

• Reindexing is cheap, we can always change the index format for free.
• We can add stuff to rigour, but we want to keep the core of the indexing logic in yente.

Let's leave the weak alias query logic untouched, that's not a goal for this.