inference with new zero-shot language: phoneme frequencies for ASP

[imiles2021]

Hi there,

I am working on including the Alsatian dialect (with its variants) as a zero shot language. To do so, I am updating the files for distance lookups so that the language embeddings can be updated.

I inspected the 'asp_dict.pkl' file and I assume that for each language, the associated values are the ASP values between that language and every other language.

As I want to include Alsatian, I therefore need to calculate the same values. To do so however, based on the ASPF formula in Do et al. (2023):

where PFA is the vector of frequencies of language A

I need the phoneme frequencies for all the other languages the model can perform inference on. Is there a file somewhere where these frequencies are stored? For instance, one that was used to create the 'asp_dict' file?’

Or if this metric was calculated in another way, please let me know!

Thanks in advance,
Isobelle Miles

[Flux9665]

Hi Isobelle!

The asp_dict.pkl file was provided as it is by the authors of Do et al. (2023), so the code for this is not part of this toolkit. The good news is, that in our evaluation, the ASP did not perform particularly good when it comes to predicting which language embeddings should be averaged. Both the distance on the world map, as well as the distance to the lowest common ancestor in the language family tree performed much better, so you might be better off just using those. It is a bit complicated though, I'm sorry, I didn't expect someone to try to extend this beyond the ISO 639-3 list.

You can include the Alsatian dialect into the list of coordinates

https://github.com/DigitalPhonetics/IMS-Toucan/blob/MassiveScaleToucan/Preprocessing/multilinguality/iso_to_long_lat.json

and the list of language family memberships

https://github.com/DigitalPhonetics/IMS-Toucan/blob/MassiveScaleToucan/Preprocessing/multilinguality/iso_to_memberships.json

Then you can create the distance lookup files for them using the CacheCreator class:

IMS-Toucan/Preprocessing/multilinguality/create_distance_lookups.py

Line 15 in 3b0340f

class CacheCreator:

But you need to adapt and overwrite two default files here that are downloaded by default:

IMS-Toucan/Preprocessing/multilinguality/create_distance_lookups.py

Line 17 in 3b0340f

self.iso_codes = list(load_json_from_path(hf_hub_download(cache_dir=MODEL_DIR, repo_id="Flux9665/ToucanTTS", filename="iso_to_fullname.json")).keys())

and

IMS-Toucan/Preprocessing/multilinguality/create_distance_lookups.py

Line 18 in 3b0340f

self.iso_lookup = load_json_from_path(hf_hub_download(cache_dir=MODEL_DIR, repo_id="Flux9665/ToucanTTS", filename="iso_lookup.json"))

You may need to do this in even more places I'm afraid, like

IMS-Toucan/Preprocessing/multilinguality/SimilaritySolver.py

Line 35 in 3b0340f

iso_to_fullname_path = hf_hub_download(cache_dir=MODEL_DIR, repo_id="Flux9665/ToucanTTS", filename="iso_to_fullname.json") if not iso_to_fullname_path else iso_to_fullname_path

Then you can call the CacheCreator's methods create_tree_cache and create_map_cache, but you need to overwrite the path to the iso_to_memberships.json and the iso_to_long_lat.json in their respective functions, otherwise a default will be downloaded that doesn't contain your changes once more.

Next, you can create another cache file using the LangDistDatasetCreator and its method called create_dataset

IMS-Toucan/Preprocessing/multilinguality/create_lang_dist_dataset.py

Line 73 in fa64ca6

def create_dataset(self,

As the arguments, I recommend the distance_type "combined" while putting ["asp"] into excluded_distances.

Then all decisions will be based on the average of map and tree distance.

Finally you can use the following file to load you distance dataset and save the new model with the averaged embeddings for Alsatian:

https://github.com/DigitalPhonetics/IMS-Toucan/blob/MassiveScaleToucan/Preprocessing/multilinguality/generate_zero_shot_lang_embs.py

I tried to make it really easy to reproduce our work with all of those defaults, but now for your use-case this is really annoying, I'm sorry and I hope you can make it work.

Best, Florian

[imiles2021]

Hi Florian,

Thank you so much for your response! The steps for adding Alsatian are very helpful and I have managed to get it to work.

Incase it interests or helps someone else, in order for transphone to be able to predict phonemes for the new language, we also need to make some edits. As transphone relies on phonepiece’s read_tree functionality, the file we need to edit is here.

• phonepiece/phonepiece/data/tree.txt

I cloned the repo (https://github.com/xinjli/phonepiece), edited the file and installed this version rather than the pip package in the IMSToucan requirements.
The pipeline still works if you don’t do this, but it uses the english model instead of computing the closest langs.

In terms of the ASP dictionary, thank you for your suggestion of using the combination of map and tree distances, I will give this a go! As well as this, I have experimented with updating the embeddings with the learned distance - using a copy of Swiss German's asp values for Alsatian (as I know that Swiss German is similar to one of the Alsatian variants).

I also have a question regarding your use of iso-codes. I assume the reason you used these codes instead of something else (like glottocodes) is so that Toucan could work with other models (like transphone which also maps to iso-codes)? But I would be interested to know if there are any other reasons? As, although the Alsatian variants don’t have iso-codes, they do have glottocodes and are in the family language tree (although they don’t have longitude and latitude information). Perhaps, by using glottocodes for instance, it may be easier to extend to languages which aren’t officially recognised with an iso-code.

Thanks again and I look forward to hearing what you think!
Isobelle