Skip to content

ilib-localedata: Fix race condition in LocaleData.ensureLocale()#148

Open
ehoogerbeets wants to merge 557 commits intocvtLocaleDataToJestfrom
fixEnsureLocale
Open

ilib-localedata: Fix race condition in LocaleData.ensureLocale()#148
ehoogerbeets wants to merge 557 commits intocvtLocaleDataToJestfrom
fixEnsureLocale

Conversation

@ehoogerbeets
Copy link
Copy Markdown
Contributor

@ehoogerbeets ehoogerbeets commented Jul 21, 2025

  • If too much data was loaded at once, ensureLocale would sometimes have a Promise that would resolve to "true" when the data was not yet parsed and cached, leading to a situation where code would continue assuming the locale data was there.
  • Had to call ensureLocale with many locales hundreds of times to consistently repro the situation and show that it was fixed
  • Tests would sometimes fail in one browser, and sometimes in another, and it would be different tests that failed, which was the clue that it was somewhere here in the async data loading. Node loads the files synchronously, so never a race condition. It was only within webpack on the headless browsers that we would run into this.

@ehoogerbeets ehoogerbeets self-assigned this Jul 21, 2025
@changeset-bot
Copy link
Copy Markdown

changeset-bot bot commented Jul 21, 2025

🦋 Changeset detected

Latest commit: 6819925

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 24 packages
Name Type
ilib-loader Minor
ilib-assemble Minor
ilib-localedata Patch
ilib-address Patch
ilib-istring Patch
ilib-lint-react Patch
ilib-loctool-regex Patch
ilib-tools-common Patch
ilib-lint-apple Patch
ilib-lint-java Patch
ilib-lint-javascript Patch
ilib-lint-python-gnu Patch
ilib-lint-python Patch
ilib-lint Patch
ilib-loctool-json Patch
ilib-loctool-po Patch
ilib-loctool-tap-i18n Patch
ilib-po Patch
ilib-tmx Patch
ilib-yaml Patch
loctool Patch
tmxtool Patch
ilib-loctool-openapi Patch
ilib-loctool-pendo-md Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 22, 2025

Jest Code Coverage

Code Coverage for changed files • (0%) 
File% Stmts% Branch% Funcs% LinesUncovered Line #s
report-only-changed-files is enabled. No files were changed in this commit :)
Title Lines Statements Branches Functions
ilib-lint Coverage: 88%
86.5% (2339/2704) 76.81% (1368/1781) 88.72% (425/479)
ilib-lint-apple Coverage: 95%
92.39% (85/92) 84.05% (58/69) 100% (8/8)
ilib-lint-java Coverage: 90%
89.33% (134/150) 79.8% (83/104) 93.1% (27/29)
ilib-lint-javascript Coverage: 81%
80.48% (132/164) 71.66% (86/120) 84.21% (32/38)
ilib-lint-python Coverage: 73%
73.43% (141/192) 60.46% (52/86) 62.06% (18/29)
ilib-lint-python-gnu Coverage: 76%
75.15% (118/157) 71.18% (42/59) 60.97% (25/41)
ilib-lint-react Coverage: 96%
95.88% (303/316) 84.91% (152/179) 91.04% (61/67)
ilib-loctool-android-layout Coverage: 73%
72.17% (166/230) 67.25% (76/113) 54.83% (17/31)
ilib-loctool-android-resource Coverage: 61%
61.86% (292/472) 60.85% (171/281) 57.14% (24/42)
ilib-loctool-csv Coverage: 80%
79.38% (208/262) 73.33% (132/180) 65.21% (30/46)
ilib-loctool-ghfm Coverage: 85%
85.84% (637/742) 78.01% (408/523) 86.2% (50/58)
ilib-loctool-haml Coverage: 92%
92.56% (436/471) 86.48% (224/259) 72.97% (27/37)
ilib-loctool-html Coverage: 91%
91.17% (279/306) 82.84% (140/169) 73.68% (28/38)
ilib-loctool-java-android-ilib Coverage: 71%
70.28% (123/175) 60% (45/75) 48% (12/25)
ilib-loctool-javascript Coverage: 74%
73.25% (178/243) 61.15% (85/139) 45.16% (14/31)
ilib-loctool-javascript-resource Coverage: 68%
69.32% (113/163) 56.41% (44/78) 37.5% (12/32)
ilib-loctool-json Coverage: 79%
79.06% (442/559) 72.08% (266/369) 73.07% (57/78)
ilib-loctool-jst Coverage: 89%
89.16% (288/323) 81.34% (157/193) 73.68% (28/38)
ilib-loctool-jsx Coverage: 72%
70.8% (114/161) 68.75% (55/80) 39.13% (9/23)
ilib-loctool-mrkdwn Coverage: 88%
87.5% (357/408) 80.35% (229/285) 74.41% (32/43)
ilib-loctool-objectivec Coverage: 70%
68.57% (96/140) 69.56% (32/46) 38.46% (10/26)
ilib-loctool-php-resource Coverage: 65%
65.8% (102/155) 45.76% (27/59) 37.14% (13/35)
ilib-loctool-po Coverage: 88%
87.87% (232/264) 76.92% (130/169) 76.92% (30/39)
ilib-loctool-properties Coverage: 67%
67.67% (157/232) 57.54% (61/106) 34.21% (13/38)
ilib-loctool-regex Coverage: 74%
73.72% (202/274) 65.06% (95/146) 56.81% (25/44)
ilib-loctool-ruby-ilib Coverage: 67%
66.53% (173/260) 73.63% (81/110) 40% (12/30)
ilib-loctool-salesforce-metaxml Coverage: 88%
87.08% (209/240) 70.49% (86/122) 68.62% (35/51)
ilib-loctool-strings Coverage: 75%
74.74% (222/297) 78.91% (131/166) 48.71% (19/39)
ilib-loctool-swift Coverage: 68%
66.88% (101/151) 62.74% (32/51) 40.74% (11/27)
ilib-loctool-tap-i18n Coverage: 87%
87.58% (254/290) 85.07% (114/134) 71.42% (35/49)
ilib-loctool-xml Coverage: 87%
86.53% (437/505) 78.1% (289/370) 78.26% (54/69)
ilib-loctool-yaml Coverage: 85%
85.71% (348/406) 82.16% (235/286) 71.42% (40/56)
ilib-loctool-yaml-resource Coverage: 65%
65.31% (194/297) 60.86% (84/138) 47.61% (20/42)
ilib-tmx Coverage: 93%
92.73% (217/234) 80.24% (130/162) 90.9% (50/55)
ilib-tools-common Coverage: 89%
89.01% (1199/1347) 81.74% (721/882) 91.39% (223/244)
ilib-yaml Coverage: 94%
93.96% (109/116) 91.3% (105/115) 100% (19/19)
loctool Coverage: 81%
81.44% (6838/8396) 74.07% (3500/4725) 73.09% (785/1074)
tmxtool Coverage: 75%
76.47% (13/17) 66.66% (6/9) 100% (3/3)
Title Tests Skipped Failures Errors Time
ilib-lint 1271 0 💤 0 ❌ 0 🔥 42.071s ⏱️
ilib-lint-apple 31 0 💤 0 ❌ 0 🔥 6.722s ⏱️
ilib-lint-java 29 0 💤 0 ❌ 0 🔥 8.049s ⏱️
ilib-lint-javascript 57 0 💤 0 ❌ 0 🔥 10.66s ⏱️
ilib-lint-python 52 0 💤 0 ❌ 0 🔥 13.521s ⏱️
ilib-lint-python-gnu 35 0 💤 0 ❌ 0 🔥 14.187s ⏱️
ilib-lint-react 111 12 💤 0 ❌ 0 🔥 12.035s ⏱️
ilib-loctool-android-layout 44 0 💤 0 ❌ 0 🔥 18.04s ⏱️
ilib-loctool-android-resource 61 0 💤 0 ❌ 0 🔥 19.543s ⏱️
ilib-loctool-csv 57 0 💤 0 ❌ 0 🔥 20.717s ⏱️
ilib-loctool-ghfm 202 0 💤 0 ❌ 0 🔥 7.733s ⏱️
ilib-loctool-haml 192 0 💤 0 ❌ 0 🔥 19.989s ⏱️
ilib-loctool-html 89 0 💤 0 ❌ 0 🔥 19.519s ⏱️
ilib-loctool-java-android-ilib 62 0 💤 0 ❌ 0 🔥 19.189s ⏱️
ilib-loctool-javascript 76 0 💤 0 ❌ 0 🔥 18.418s ⏱️
ilib-loctool-javascript-resource 39 0 💤 0 ❌ 0 🔥 19.003s ⏱️
ilib-loctool-json 103 0 💤 0 ❌ 0 🔥 20.504s ⏱️
ilib-loctool-jst 115 0 💤 0 ❌ 0 🔥 19.497s ⏱️
ilib-loctool-jsx 48 0 💤 0 ❌ 0 🔥 8.525s ⏱️
ilib-loctool-mrkdwn 76 0 💤 0 ❌ 0 🔥 20.5s ⏱️
ilib-loctool-objectivec 34 0 💤 0 ❌ 0 🔥 18.639s ⏱️
ilib-loctool-php-resource 22 0 💤 0 ❌ 0 🔥 18.278s ⏱️
ilib-loctool-po 107 0 💤 0 ❌ 0 🔥 18.643s ⏱️
ilib-loctool-properties 49 0 💤 0 ❌ 0 🔥 19.123s ⏱️
ilib-loctool-regex 46 0 💤 0 ❌ 0 🔥 16.266s ⏱️
ilib-loctool-ruby-ilib 90 0 💤 0 ❌ 0 🔥 19.774s ⏱️
ilib-loctool-salesforce-metaxml 101 0 💤 0 ❌ 0 🔥 20.026s ⏱️
ilib-loctool-strings 55 0 💤 0 ❌ 0 🔥 18.271s ⏱️
ilib-loctool-swift 35 0 💤 0 ❌ 0 🔥 19.401s ⏱️
ilib-loctool-tap-i18n 64 0 💤 0 ❌ 0 🔥 19.417s ⏱️
ilib-loctool-xml 85 0 💤 0 ❌ 0 🔥 19.717s ⏱️
ilib-loctool-yaml 102 0 💤 0 ❌ 0 🔥 19.105s ⏱️
ilib-loctool-yaml-resource 47 0 💤 0 ❌ 0 🔥 20.169s ⏱️
ilib-po 75 0 💤 0 ❌ 0 🔥 23.731s ⏱️
ilib-tmx 90 0 💤 0 ❌ 0 🔥 14.415s ⏱️
ilib-tools-common 704 0 💤 0 ❌ 0 🔥 10.596s ⏱️
ilib-yaml 37 0 💤 0 ❌ 0 🔥 14.836s ⏱️
loctool 2365 82 💤 0 ❌ 0 🔥 2m 31s ⏱️
tmxtool 6 0 💤 0 ❌ 0 🔥 2.866s ⏱️

@ehoogerbeets ehoogerbeets changed the title Fix race condition in LocaleData.ensureLocale() ilib-localedata: Fix race condition in LocaleData.ensureLocale() Nov 4, 2025
wadimw and others added 19 commits November 14, 2025 14:17
(ilib-loctool-pendo-md) Rework how markdown syntax is escaped
ilib-lint: Fix a bug in the state checker rule
- it is not used very much in Japanese in translations of
  English sentences that end in an exclamation point, so
  we don't need to check it
…izeOnly is false or pseudo is enabled

`translations of Project' is not used when localizeOnly is true or
pseudo is disabled.
It is referenced by PseudoHant and not used in `write` step.
- if the config includes mappings like "**/*.mdx", then the
  file name extensions in that mapping should be added to the
  list of valid extensions that this plugin handles on the
  assumption that the user put those mappings there on purpose
  to pick up that type of file.
- loctool now warns when the mappings mention an extension that is
  not mentioned in extensions
gouniLee and others added 30 commits February 25, 2026 12:16
…rings set

When a mapping uses a frontmatter allowlist (array of key names), the
extraction phase correctly filters to only extract those keys. However,
during localization the yaml plugin processes all frontmatter keys and
reports any without a translation as 'new'. Keys outside the allowlist
(e.g. 'type', 'mode', 'related_pages') were never sent to the TMS, so
they will never have translations — yet they appeared in the new-strings
XLIFF as false positives.

Filter yaml new-resources in MdxFileType.getNew() using the same
frontmatter allowlist that governs extraction. This ensures symmetry
between what is extracted and what is reported as untranslated.

When frontmatter is boolean true (localize all), no filtering is applied.
fix(ilib-loctool-mdx): filter non-allowlisted frontmatter
ilib-loctool-mdx: Add missing changeset for frontmatter filter fix
ilib-loctool-openapi: Move open api plugin from ilib-js/ilib-loctool-openapi to here
- regex was not anchored to the beginning of the string, so when
  you have a template like "[language]/[dir]/[filename]", it would
  match against "guides/ai-zone/index.mdx" where the language
  would end up being "des". (Last 3 letters of "guides".)
- Now with it anchored properly, it returns "" for the language
  because "guides" does not match an ISO language code.
Fix a problem with getLocaleFromPath
ilib-assemble: Fix the issue of duplicated code inclusion of legacyilibassemble
Add key filtering to prevent prototype pollution attacks (CWE-1321) in
JSUtils.extend, extend2, merge, deepCopy, and shallowCopy functions.

The fix blocks __proto__, constructor, and prototype keys from being
processed during object merging/extending operations, preventing
attackers from polluting Object.prototype via malicious input.

Fixes #266
If there was anyOf or oneOf in the schema, it would sometimes
localize the json object as a string instead of as an object.
fix(ilib-common): prevent prototype pollution
ilib-loctool-json: Fix localized output path and fix broken output
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants