Context
Non-Wikipedia sources (user-submitted URLs like government portals) currently get their language from the <html lang> attribute during archiving (_detect_source_language in poliloom/archiving.py, added in ee4077e). This is brittle: many pages omit the attribute, declare it wrongly, or serve mixed-language content.
Language matters because review serving is language-scoped: a property is only visible to a user if one of its sources has a SourceLanguage row in the user's selected languages. Sources where detection fails fall back to "unknown language = visible to everyone", which dilutes the per-language review model.
Proposal
Replace (or augment) the <html lang> heuristic with an LLM call during source processing:
- Feed the extracted page text (truncated) to the model and ask for the content's language.
- Constrain the output to the current user's language filters — the user who submitted the source (or triggered the enrichment) already told us which languages they review; the page is almost certainly in one of them. Pass their language QIDs as the allowed set (structured output / enum), plus an explicit "other/unknown" option so the model isn't forced into a wrong bucket.
- Map the result to a
Language entity and link via SourceLanguage, as today.
- Keep "unknown" as a real outcome: no
SourceLanguage rows, preserving the designed fallback instead of guessing.
Open questions
- Which model/reasoning effort (extraction currently uses OPENAI_MODEL with configurable effort)?
- Where exactly in
process_source this belongs relative to archiving and extraction.
- Whether Wikipedia sources should ever fall back to this when project relations are missing.
Context
Non-Wikipedia sources (user-submitted URLs like government portals) currently get their language from the
<html lang>attribute during archiving (_detect_source_languageinpoliloom/archiving.py, added in ee4077e). This is brittle: many pages omit the attribute, declare it wrongly, or serve mixed-language content.Language matters because review serving is language-scoped: a property is only visible to a user if one of its sources has a
SourceLanguagerow in the user's selected languages. Sources where detection fails fall back to "unknown language = visible to everyone", which dilutes the per-language review model.Proposal
Replace (or augment) the
<html lang>heuristic with an LLM call during source processing:Languageentity and link viaSourceLanguage, as today.SourceLanguagerows, preserving the designed fallback instead of guessing.Open questions
process_sourcethis belongs relative to archiving and extraction.