Invalid translations when using the pre-trained NLLB model

[mmartin9684-sil]

Using the pre-trained NLLB model to translate a USFM file is producing invalid translations - i.e., a apparently random output of either Latin text or Devanagari text instead of the expected 100% Devanagari output. The problem is happening with both the experiment script and with the translate script.

An experiment demonstrating this problem can be found here: Nepal\WesternTamang\NLLB.1.3B.npi-WTNBT.en-WTEBT. This configuration is intended to translate a Nepali USFM file into English. Instead, the generated USFM file is mostly Devangari text, with some Latin text - but no English output. The infer subfolder has a draft created using the translate script and a draft created using the experiment script. Both drafts exhibit similar problems.

[ddaspit]

Are you fine tuning a model or just inferencing on NLLB as-is?

[mmartin9684-sil]

This run was intended to use the NLLB 1.3B model as-is (no fine-tuning).

[ddaspit]

I'm guessing there is a bug when inferencing on NLLB as-is. Have you successfully inferenced using NLLB recently?

[mmartin9684-sil]

It does seem to be specific to using NLLB as-is. When the model is fine-tuned, I'm getting reasonable drafts.

[benjaminking]

This sounds similar to a previously-fixed issue: #656

[benjaminking]

I've identified (at least part of) why this issue is occurring. SILNLP uses the source iso code of the validation set to initialize the tokenizer. The NLLB tokenizer adds a token corresponding to the source language code to the beginning of everything it tokenizes. Without a validation set, this language code is never set.

I'm not sure yet what the right way to fix this is, but I do have a workaround in the meantime. If I change the type under corpus_pairs to val, I'm able to get the expected output from the translate command.

[mmartin9684-sil]

Thanks Ben ... this workaround really does get past the problem! Thanks!