About the selection of glove

[futian00]

Hello, I would like to ask why the choice of glove embeddings is Common Crawl and the choice of agwe embeddings is librispeech in the code. Shouldn't the choice of glove embeddings also be librispeech

[ankitapasad]

Hi @futian00

When we compare representations from self-supervised speech models (S3Ms) to any external embedding space (AGWE or GloVE), we are mainly checking for similarities between the S3M's word embedding space and the specific property offered by the external embedding space (pronunciation for AGWEs and semantic for GloVe). So, we don't necessarily have to match the pre-training data.

It is nice to have a matched domain for AGWEs, as these are jointly trained with acoustic word embeddings on speech data, and speech data has much more variability than text. So, the domain of the pre-training data could impact the learned embeddings. However, it remains to be seen whether using AGWEs trained on a different dataset still offers insights similar to those trained on LibriSpeech.

Related to your question: In a different experiment, we compare S3Ms representation with syntactic properties extracted from a text corpus. Our findings remain consistent irrespective of the corpus used to extract the syntactic properties.

I hope that offers some clarity!

[futian00]

Thank you for your answer.

I have another question. If I need to conduct cca-agwe experiment on other languages, I will need agwe embeddings in other languages, but I haven't found the code about agwe embeddings. I also didn't get a reply when I sent an email to Shane Settle. May I ask how to get agwe embeddings in other languages? Can you give me some help.

Looking forward to your reply!

[futian00]

There is another problem, in the cca_phone experiment, what you described in the README is $exp_name = cca_phone, $span = phone, which means that when executing extract_rep.sh, the fbank feature is generated. Then fbank features are input into the model to get a phoneme-level representation. I hope I understand this correctly.
However, in get_scores.py, the cca_phone method calculates the CCA score of the model's internal representation and embedded vector checks.
Whether the two are contradictory.
My understanding from your paper is that the word-level acoustic-speech information used to evaluate model features is to calculate cca scores using agwe and the word-level features represented within the model.
This makes me a little confused. Can you tell me which two features of cca_phone, cca_word, cca_agwe and cca_glove are used to calculate the cca score?