feat(extractor): extract default HTML metadata + cache XPath expressions#164
Open
feat(extractor): extract default HTML metadata + cache XPath expressions#164
Conversation
Populate ExtractData with standard HTML metadata by default (title, description, OpenGraph, Twitter Card, canonical, keywords, author), parse <script type="application/ld+json"> blocks into jsonld.type and jsonld.raw, and cache compiled XPathExpression objects per thread to eliminate per-call recompilation under high crawl rates. The default-field rule map is fully overridable via setDefaultFieldRules and both subsystems can be disabled independently with setExtractDefaultMetadata / setExtractJsonLd. Malformed JSON-LD blocks are logged and skipped without aborting extraction.
…anup, warn on metadata collisions
Three regressions / gaps were uncovered in the HtmlExtractor PR #164 review: 1. Malformed XPath expressions (in contentXpath or metadataXpathMap) used to log a warning and yield empty values — XPathAPI.eval threw XPathException for both compile and evaluate failures and the catch handled them uniformly. The compile cache split that into a separate getXPathExpression path that throws CrawlerSystemException, which was not caught downstream and therefore propagated out of createExtractData, aborting the whole extraction. Catch CrawlerSystemException in getStringsByXPath (and in extractJsonLd, for symmetry) and restore the warn+empty contract. 2. extractJsonLd unconditionally putValues for jsonld.raw / jsonld.type, silently overwriting any value that an operator-supplied addMetadata("jsonld.raw"/"jsonld.type", ...) rule had already populated. Mirror the precedence rule used by applyDefaultFieldRules: only auto- populate when the key is absent. 3. collectTypeNodes only inspected @type on the immediate object (or array elements). Schema.org markup commonly nests typed entities under @graph, mainEntity, author, publisher, etc.; those @type values were therefore never exposed via jsonld.type. Walk every object child recursively (skipping @type / @context to avoid double-collection and vocabulary leakage). Recursion is bounded by the parser's existing JSONLD_MAX_NESTING_DEPTH guard. Five regression tests added: malformed metadata XPath, malformed contentXpath, custom jsonld metadata key precedence, @graph type collection, and the @context-object negative case.
…s blank, match JSON-LD type case-insensitively Two regressions surfaced in code review of PR #164: 1. extractor.xml in fess-crawler-lasta registers addMetadata("title", "//TITLE"), so the metadataXpathMap loop unconditionally calls putValues("title", []) on pages without a <title>. The default-rule existence check (getValues != null) then sees the empty array and skips the og:title fallback, silently disabling the PR's "extract default HTML metadata" intent in real deployments. Switch the predicate to "has a non-blank value" so default rules backfill when the custom rule produced nothing. 2. JSONLD_XPATH matched only the literal lowercase 'application/ld+json'. Per RFC 6838 / HTML5 the type attribute is case-insensitive and may carry surrounding whitespace; NekoHTML uppercases element names but preserves attribute values verbatim, so 'Application/LD+JSON' or ' application/ld+json ' was missed. Use translate(normalize-space(@type), ...) so common real-world variants are picked up.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
<script type="application/ld+json">blocks; exposejsonld.typeandjsonld.raw(multivalue). Malformed JSON is skipped with a warn log.XPathExpressionper thread (viaThreadLocal<Map>andThreadLocal<XPath>) to eliminate per-call recompilation under high crawl rate.setDefaultFieldRules(Map),setExtractDefaultMetadata(boolean),setExtractJsonLd(boolean)for full opt-out / customization. NewclearXPathCache()for dynamic rule changes.Why
Most search use cases want title and description for snippet rendering; users currently have to wire each XPath manually. JSON-LD provides high-quality structured data signals. XPath compilation in a hot loop was wasted work.
Threat model
HTML content is untrusted. JSON-LD parsing uses Jackson with default settings; malformed input is caught and logged, never fails extraction. XPath cache key is the expression string (admin-configured), bounded by configured rules — no unbounded growth from untrusted input.
Tests
@type, malformed JSON resilience.clearXPathCache()empties cache.Verification
mvn -pl fess-crawler test -Dtest=HtmlExtractorTest→ 18/18 pass.mvn -pl fess-crawler test→ 1706 run, 0 failures, 55 pre-existing env-dependent errors (Docker/LibreOffice).mvn formatter:format && mvn license:formatclean.Test plan