Hindi ITN: Telephone, Quarterly Measures, Fraction Exceptions, Changes to Date #301

tarushi2k2 · 2025-07-08T08:38:22Z

Adds Hindi ITN support for the Telephone semiotic class, mixed/exception fractions, quarterly measures, and century ordinals in the Date class. Includes all updates, test cases, and rebase with the latest upstream changes.

Before your PR is "Ready for review"

Pre checks:

PR Type:

New Feature
Bugfix
Documentation
Test

If you haven't finished some of the above items you can still open "Draft" PR.

Signed-off-by: Tarushi V <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: Tarushi V <[email protected]>

mgrafu · 2025-07-11T12:50:59Z

nemo_text_processing/inverse_text_normalization/hi/data/date/date_days.tsv

 ३०	तीस
 ३१	इकतीस
+३१	इकतिस
+३१	इकत्तीस
+३१	इकत्तिस


do we have the same term multiple times in this tsv? is this necessary?

also, are these mappings any different than cardinals?

We have the same terms in the TSV file because of different spellings and character differences. I kept all the other versions on purpose because inverse text normalization allows many-to-one mapping. Having all the versions makes it work better and more accurately.

I added the numbers used for dates in a separate file because the date semiotic class only needs numbers from 1 to 31. For cardinal numbers, we already have two separate files: one for single digits and another called teens and ties for numbers from 10 to 99. So it was easier and cleaner to create a new TSV file just for dates instead of using the existing cardinal number files.

is there no way to optimize (1) with rules instead of one long tsv file?

let's use the cardinal graph and restrict inputs to 1-31 for (2), that will be cleaner and easier to maintain in the future

Numbers 0-99 have unique words in Hindi that cannot be represented by grammars.

I’ve deleted the date_days.tsv file and updated it to use the cardinal graph instead. The inputs are now restricted to 1–31 as suggested. Will push it as soon as all comments on the PR are resolved.

mgrafu · 2025-07-11T12:54:42Z

nemo_text_processing/inverse_text_normalization/hi/data/telephone/STD_codes_eng.tsv

@@ -0,0 +1,596 @@
+११	one one


let's implement graphs to process digits instead of having a long tsv file

is this completed? if so, let's remove this file

mgrafu · 2025-07-11T12:55:04Z

nemo_text_processing/inverse_text_normalization/hi/data/telephone/STD_codes_hin.tsv

@@ -0,0 +1,2750 @@
+११	एक एक


let's either try to use the cardinals graph or create a graph for this one as well

is this completed? if so, let's remove this file

mgrafu · 2025-07-11T12:55:35Z

nemo_text_processing/inverse_text_normalization/hi/data/telephone/country_codes_eng.tsv

@@ -0,0 +1 @@
+९१	nine one


same comment as for std codes

mgrafu · 2025-07-11T12:55:42Z

nemo_text_processing/inverse_text_normalization/hi/data/telephone/country_codes_hin.tsv

@@ -0,0 +1 @@
+९१	नौ एक


same comment as for std codes

mgrafu · 2025-07-11T12:56:12Z

...ext_processing/inverse_text_normalization/hi/data/telephone/landline_operator_digits_hin.tsv

@@ -0,0 +1,7 @@
+२	दो


are these any different than cardinals?

Yes, these are also different from cardinals because landline operator digits are specifically only 2, 3, 4, and 6. In India, landline numbers must start with one of these digits to be valid. The cardinal numbers have a single file, digits.tsv, which contains all numbers from 1 to 9. Since we do not need all these digits for landline operators, it made sense to create a separate TSV file specifically for the landline operator digits.

you can import the carinal graph and restrict inputs -- this is necessary to optimize upkeep in the future

only create new files when the mapping is different

mgrafu · 2025-07-11T12:56:50Z

nemo_text_processing/inverse_text_normalization/hi/data/time/hour.tsv

@@ -23,5 +23,6 @@
 २०	बीस


are these any different than cardinals?

Yes, this TSV file for time is separate because time only requires numbers from 1 to 24. For cardinal numbers, we already have two separate files: one for single digits and another called teens and ties for numbers from 10 to 99. Since the cardinal numbers are split across two files, using one and then extracting only the numbers from 10 to 24 from the other seemed more complex. So it was simpler and cleaner to create a dedicated TSV file for time instead of reusing and modifying the existing cardinal files.

same comment as for dates

I’ve deleted the hour.tsv file and updated it to use the cardinal graph instead. The inputs are now restricted to 1–24 as suggested. Will push it as soon as all comments on the PR are resolved.

nemo_text_processing/inverse_text_normalization/hi/taggers/cardinal.py

mgrafu · 2025-07-11T13:05:31Z

nemo_text_processing/inverse_text_normalization/hi/utils.py

+
+
+def load_column_from_tsv(filepath, column_index=1):
+    """


is this necessary? doesn't pynini have a function to get the inputs or outputs only?

I used load_column_from_tsv because it reads just one column from the TSV file and gives a list of STD codes. This list is needed to build the telephone FST with those codes and landline numbers.
Pynini has functions like string_file that load the whole TSV file as one big FST with all pairs together. You can get inputs or outputs by inverting the FST, but these work on FSTs that are already made, not directly on the file.
When I tried replacing load_column_from_tsv with string_file, I got an error. It was harder to get a list of codes from the big FST than just using load_column_from_tsv, which gives the list straight away.
So, using load_column_from_tsv was easier and simpler, especially with a big database where I needed the full list of STD codes. That’s why I decided to use load_column_from_tsv.

please reimplement with pynini.project as used here

NeMo-text-processing/nemo_text_processing/text_normalization/sv/taggers/date.py

Line 54 in ac07488

era_names = pynini.project(era_words, "output")

mgrafu · 2025-07-23T16:15:02Z

nemo_text_processing/inverse_text_normalization/hi/data/numbers/paune.tsv

+९८	निन्यान्बे
+९८	निन्यानबे
+९८	निन्यानवे
+९८	निन्यान्वे


let's also either leverage cardinal graph or optimize with rules

We can't use the cardinal graph here because the number mapping is completely different for this particular TSV file. Also, numbers 0-99 have unique words in Hindi that cannot be represented by grammars.

are you saying that the "9" in 93 is a different word than the "9" in 94? what about the "4" in 34 vs the "4" in 74?

mgrafu · 2025-07-23T16:15:42Z

nemo_text_processing/inverse_text_normalization/hi/data/numbers/teens_and_ties.tsv

@@ -9,6 +9,7 @@
 १७	सत्रह
 १७	सतरह


let's also optimize with rules

Numbers 0-99 have unique words in Hindi that cannot be represented by grammars.

Signed-off-by: Tarushi V <[email protected]>

github-actions · 2025-08-20T02:10:26Z

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

github-actions · 2025-08-28T02:07:37Z

This PR was closed because it has been inactive for 7 days since being marked as stale.

tarushi2k2 and others added 9 commits November 27, 2024 15:17

Addition of whitelist and word classes

513fff5

Signed-off-by: Tarushi V <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

535af69

for more information, see https://pre-commit.ci

Updation of Jenkins date

d4e380f

Signed-off-by: Tarushi V <[email protected]>

Cleanup

60f8757

Signed-off-by: Tarushi V <[email protected]>

Updation

9aa85c0

Signed-off-by: Tarushi V <[email protected]>

Updation

bf6ebe3

Signed-off-by: Tarushi V <[email protected]>

Merge branch 'NVIDIA:main' into main

28c2cd7

Merge branch 'NVIDIA:main' into main

bcf6b28

All updates - Hindi ITN 2.0

0837161

Signed-off-by: Tarushi V <[email protected]>

tarushi2k2 marked this pull request as ready for review July 8, 2025 08:43

mgrafu reviewed Jul 11, 2025

View reviewed changes

nemo_text_processing/inverse_text_normalization/hi/taggers/cardinal.py Show resolved Hide resolved

mgrafu reviewed Jul 11, 2025

View reviewed changes

mgrafu reviewed Jul 23, 2025

View reviewed changes

tarushi2k2 added 2 commits August 5, 2025 15:44

WIP: Changes made on hindi_2.0

c9bb5fd

Telephone changes

05f6237

Signed-off-by: Tarushi V <[email protected]>

tarushi2k2 force-pushed the hindi_itn_2.0 branch from a328412 to 05f6237 Compare August 5, 2025 10:19

github-actions bot added the Stale label Aug 20, 2025

github-actions bot closed this Aug 28, 2025

Hindi ITN: Telephone, Quarterly Measures, Fraction Exceptions, Changes to Date #301

Hindi ITN: Telephone, Quarterly Measures, Fraction Exceptions, Changes to Date #301

Uh oh!

Conversation

tarushi2k2 commented Jul 8, 2025

Before your PR is "Ready for review"

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tarushi2k2 Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tarushi2k2 Jul 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tarushi2k2 Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mgrafu Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Aug 20, 2025

Uh oh!

github-actions bot commented Aug 28, 2025

Uh oh!

Uh oh!

tarushi2k2 Jul 29, 2025 •

edited

Loading

tarushi2k2 Jul 15, 2025 •

edited

Loading

tarushi2k2 Jul 29, 2025 •

edited

Loading

mgrafu Jul 29, 2025 •

edited

Loading