About the dim of representations and labels

[LYPinASR]

Hello,

Thanks for your works! What I want to ask is: When the representation's size is (T, F), and it's corresponding labels' size is (L,), how to ensure that the first dimension is consistent before calculating their mutual information.

Have a nice day!

[ankitapasad]

Hi @LYPinASR

Let's take a concrete example:

Say we have $N=10000$ word instances of $L=500$ distinct words, where each word instance is represented as $D=768$ dimensional vector.

For CCA, we convert each word label into a $500$-dim one-hot vector and then calculate the CCA correlation between the representation matrix, $X \in \mathbb{R}^{10000 \times 768}$, and the label matrix, $Y \in \mathbb{R}^{10000 \times 500}$.

For discrete MI, we first run a clustering algorithm with $k$ clusters (suppose $k=1000$ in this example) on $X$ and obtain a discrete cluster ID for each of the $10000$ samples. Then, we calculate discrete MI between the distribution of cluster IDs (with a support size of $1000$) and the distribution of ground-truth word labels (with a support size of $500$).

Does that answer your question?

[LYPinASR]

Thanks for your reply!

In your example, each speech utterance is transfered to a representation matrix X for each layer of the model, which is easy to understand. However, for the label matrix Y, how do I ensure that its length is equal to the length of the representation matrix?

In my case, the size of representation matrix X is (1500, 1024), but the number of word instances of corresponding label is only 36.

Based on your experience, what should I do in order to calculate the mutual information between X and label?

Look forward for your reply again! And have a nice day!