Add part-pair blocking tokens

[pudo]

Builds on the tokenizer fixes from #345.

Emits within-name part 2-combinations as a new pp blocking token field (boost 7.0): the sorted, deduplicated non-STOP/LEGAL parts of each name, capped at the 6 longest parts (≤15 pairs per name).

Why

Dynamic cost-based stopwords remove exactly the tokens common names are made of: on the 2023 consolidated sanctions corpus, 28.2% of np (single name part) rows are stopworded at the default bucket cap. Pair buckets are structurally smaller — df(pair) ≤ min(df(part)) — so only 1.6% of pp rows get stopworded, and pairs stay alive deep into the common-name band. Unlike the full-name fingerprint, pairs are robust to an extra or missing part:

• John Smith ↔ John Quincy Smith share pp:john smith
• Acme Holdings LLC ↔ Acme Holdings GmbH share pp:acme holdings (legal forms excluded)
• Li Yu gets a lexical token (pp:li yu) despite both parts falling under the 3-char np floor

The full-name fingerprint stays: it remains the last resort for names whose every pair is also common (fleet vessels, research institutes), effectively-mononym orgs ("OJSC Gazprom"), and the exact-name ranking signal.

Cost

pp roughly doubles name-derived index rows (+357k on 360k np+fingerprint rows on the measurement corpus). Production-scale row growth/peak RSS still needs the blocker-spec scale test; see the labelled-pair ablation results in the comments for measured recall contribution.

🤖 Generated with Claude Code

[pudo]

Labelled-pair ablation results

Ran the full matcher_training/pairs.jsonl (2026-07-09 generation) through the tokenizer from this branch: all 309,754 positive pairs, with a stopword df context built from one entity per distinct final cluster (251,787 clusters, first replay occurrence ≈ the unmerged records a fresh xref would index), df cap 60 matching the default max_bucket_size.

For each positive pair: tokenize both sides, intersect, drop tokens whose df exceeds the cap, classify by which fields still connect the pair.

metric (all positives)	share
reachable before stopwords	97.99%
reachable after stopwords	97.76%
via np (name parts)	94.12%
via pp (part pairs)	90.46%
via name (fingerprint)	89.47%
via sy (symbols)	45.96%
np+sy dark — only pp and/or fingerprint carry the name link	2.54% (7,866 pairs)
pp is the only surviving link in any field	0.15% (463 pairs)
fingerprint is the only surviving link in any field	0.13% (399 pairs)
blocker-blind (no shared surviving token at all)	2.24%

Person/Org split: persons are nearly saturated (99.21% reachable; pp almost ties np at 96.55% vs 96.69%, symbols cover 72.7%). The hard cases are orgs: 4.27% blocker-blind, and both the pp-only and fingerprint-only rescues concentrate there (0.33% / 0.27%).

Reading

• Stopwording costs only 0.23pp of labelled recall at this df scale — the common-name dark band is thin in this population.
• pp and the fingerprint have nearly equal, small, non-overlapping lifeline value — they are complementary, as designed, but at ~0.3% combined.

Caveats

1. Survivor bias: these pairs were surfaced for judgement by the previous blocker+matcher pipeline, so pairs the old blocker couldn't reach are systematically missing. Any new token class's unique contribution is a floor, not an estimate.
2. df scale: the context is 252k entities. In production xref against registries (millions of entities), part dfs inflate, np coverage degrades, and the np+sy-dark band (already 2.5% here) grows — that regime is what pp is for. The blocker-spec scale test remains the decisive measurement.

🤖 Generated with Claude Code