feat(profiles): promote DIV-level LCA to nearest MAIN/ARTICLE (ratio-capped)#89
Merged
Merged
Conversation
…capped) Fixes the pypi-class narrow-selector failure: the VLM's deep anchors plus the mapper's depth-outlier filter collapsed pypi.org profiles to div[class*="project"] (the description card, 1,818 of 10,903 main chars). Root cause chain: 8/10 anchors inside one nested code block, the 2 shallow legit anchors (tagline, install command) dropped as depth outliers, LCA computed over the deep cluster only. Fix: after LCA computation, if the LCA is a DIV and a MAIN/ARTICLE ancestor exists with innerText ratio <= PROMOTION_MAX_RATIO (8.0; pypi measured 6.0), promote to it. Only fires on DIV LCAs, so semantic-tag LCAs (TABLE/ARTICLE/SECTION/TBODY) are never touched. A prompt-side fix (anchor-allocation diversity rules) was tried first and REJECTED by same-day A/B: it did not move pypi (the outlier drop, not allocation, is decisive) and it regressed reddit_sub + imdb_movie from IDEAL to OK. prompts.py is unchanged. Gates: - Profile eval 37 cases (pypi_project case added): IDEAL 16/36 -> 26/37, 0 too-wide, success 92%. 12 promotions, ratios 1.0-5.1, no degenerate whole-page grabs. pypi_project: IDEAL, selector #content (MAIN/3). At-risk IDEAL sites (tailwind_flex, reddit_sub, gh_fastapi, imdb_movie) unchanged with non-DIV LCAs. 3 fails are live-site bot blocks / error pages (NYT, Guardian, OpenAI), not rule-caused. - Parity 15/15; mapper unit tests 22 pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the pypi-class narrow-selector failure (recorded 2026-05-21:
div[class*="project"], reproduced across forced VLM reruns). Root cause chain, established this session: the VLM put 8/10 anchors inside one nested code block; the mapper's depth-outlier filter then dropped the 2 shallow legitimate anchors (tagline + install command, depth 3 vs median 9) that would have widened the LCA toward<main>; the LCA collapsed to the description card (1,818 of 10,903 main chars).Fix: after LCA computation, a DIV-level LCA is promoted to the nearest enclosing MAIN/ARTICLE when the innerText size ratio is <=
PROMOTION_MAX_RATIO = 8.0(pypi measured 6.0). Only fires on DIV LCAs — semantic-tag LCAs (TABLE/ARTICLE/SECTION/TBODY) are structurally untouchable, which protects the previously-identified at-risk IDEAL sites.A prompt-side fix (anchor-allocation diversity rules) was tried first and rejected by same-day A/B: it did not move pypi and it regressed reddit_sub + imdb_movie from IDEAL to OK.
prompts.pyis unchanged in this PR.Gates
pypi_projectcase added)pypi_project→ IDEAL, selector#content(MAIN/3)3 eval fails are live-site bot blocks / error pages (NYT, Guardian, OpenAI blog), not rule-caused.
🤖 Generated with Claude Code