feat: infer stream language from TMDB translations #482
+193
−13
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
WalkthroughThis PR extends the metadata pipeline to carry language-tagged titles from TMDB and introduces language inference logic that uses these titles to populate missing language information for streams whose filenames match known translations. Changes
Sequence DiagramsequenceDiagram
participant Filterer as Stream Filterer
participant TMDB as TMDB Metadata
participant Infer as Language Inferrer
participant Stream as Stream Object
Filterer->>TMDB: getMetadata() with titlesWithLanguages
TMDB-->>Filterer: Metadata with titlesWithLanguages[]
loop For each stream with Unknown/empty language
Filterer->>Stream: Check parsedFile.languages
alt Language is Unknown or empty
Filterer->>Infer: inferLanguageFromTitle(filename, titlesWithLanguages)
Infer->>Infer: Normalize filename and titles
Infer->>Infer: Match filename against titles
alt Single language match
Infer-->>Filterer: Language name (mapped from ISO code)
else Multiple language matches
Infer-->>Filterer: "Multi"
else No match
Infer-->>Filterer: Existing language or Unknown
end
Filterer->>Stream: Update parsedFile.languages with inferred value
Filterer->>Filterer: Increment inferred count
end
end
Filterer->>Filterer: Log aggregated inference results
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes
Poem
Pre-merge checks and finishing touches✅ Passed checks (5 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
packages/core/src/metadata/tmdb.ts(6 hunks)packages/core/src/metadata/utils.ts(1 hunks)packages/core/src/streams/filterer.ts(1 hunks)packages/core/src/utils/languages.ts(1 hunks)
🧰 Additional context used
🧬 Code graph analysis (2)
packages/core/src/streams/filterer.ts (1)
packages/core/src/utils/languages.ts (1)
inferLanguageFromTitle(2018-2103)
packages/core/src/metadata/tmdb.ts (1)
packages/core/src/metadata/utils.ts (1)
TitleWithLanguage(1-6)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: test
Comment on lines
+2003
to
+2007
| return title | ||
| .toLowerCase() | ||
| .replace(/[^\p{L}\p{N}]/gu, ' ') | ||
| .replace(/\s+/g, ' ') | ||
| .trim(); |
Contributor
There was a problem hiding this comment.
Strip accents when normalising titles
normalizeTitle lowercases and removes punctuation, but it keeps diacritic marks. As a result a filename such as Amelie.2001.mkv normalises to amelie, while the TMDB translation Amélie normalises to amélie, so the subsequent includes test never matches and we fail to infer the language. This hits a large share of non-English titles (French, Spanish, Portuguese, etc.), undermining the feature. Please strip combining marks before the comparison.
Apply this diff to fix it:
function normalizeTitle(title: string): string {
- return title
- .toLowerCase()
+ return title
+ .normalize('NFD')
+ .replace(/[\u0300-\u036f]/g, '')
+ .toLowerCase()
.replace(/[^\p{L}\p{N}]/gu, ' ')
.replace(/\s+/g, ' ')
.trim();
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| return title | |
| .toLowerCase() | |
| .replace(/[^\p{L}\p{N}]/gu, ' ') | |
| .replace(/\s+/g, ' ') | |
| .trim(); | |
| return title | |
| .normalize('NFD') | |
| .replace(/[\u0300-\u036f]/g, '') | |
| .toLowerCase() | |
| .replace(/[^\p{L}\p{N}]/gu, ' ') | |
| .replace(/\s+/g, ' ') | |
| .trim(); |
🤖 Prompt for AI Agents
In packages/core/src/utils/languages.ts around lines 2003 to 2007,
normalizeTitle currently lowercases and removes punctuation but leaves diacritic
marks; update the normalization chain to first apply Unicode NFD decomposition
and then strip all combining marks (remove \p{M} characters) before continuing
with the existing punctuation/whitespace replaces so accents are removed (e.g.
use .normalize('NFD') followed by .replace(/\p{M}/gu, '') then the current
.replace(/[^\p{L}\p{N}]/gu, ' ').replace(/\s+/g, ' ').trim()).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary by CodeRabbit
New Features
Improvements