1x1 invertible convolutions logdet

[p0p4k]

While reading GlowTTS and WaveGlow papers, they refer to Glow (Generative Flow with Invertible 1x1 Convolutions) paper.
In the following Table 1 [Glow paper], we can see the log-determinant (logdet) of the invertible 1x1 convolution (highlighted in yellow).

In the 🐸 TTS repo, we use this logdet as follows:

• make det(W) positive, we take the absolute value (*EDITED : thanks @erogol)
  

  TTS/TTS/tts/layers/glow_tts/glow.py

  Lines 99 to 100 in 5094499

  	if torch.det(w_init) < 0:
  	w_init[:, 0] = -1 * w_init[:, 0]

• logdet |W| x h x c
  

  TTS/TTS/tts/layers/glow_tts/glow.py

  Line 131 in 5094499

  logdet = torch.logdet(self.weight) * (c / self.num_splits) * x_len # [b]

I was not sure about the derivation for that formula and no Google search gave me satisfactory answer (especially about the h x w term), so I came up with my own proof as show below. I hope someone who wants to learn the math behind these things can benefit from these. I would advice at least reading the Glow paper as a prerequisite.

These days, I am exploring the code in 🐸 TTS and comparing with the math in the paper to understand the implementation line by line. Doing things like these once a while makes you appreciate the genius behind the deep learning architectures both mathematically and in the form of code. I will post about that as soon as I get some more free time.
(Previous paper discussions : #1494, #1501, #1506)

[erogol]

Thanks for the detailed post. I love the notes :)

It is not actually making W positive, more like making its determinant positive.

About h and w, they are the height and width of the input. However, in our case, tensors are sequences so we use only length x_len.

I hope it helps with the confusion if there is any.

[p0p4k]

Yes, I overlooked the |det (W)| . Thank you for the clarification. Will read VITS and transformer stuff next. The "aha!" moment after understanding the code along with the paper is so addictive!
Although, the 🐸 TTS-papers repo is a more appropriate for posting my notes, I think this repo is much more popular and can increase visibility, hope you don't mind.

[erogol]

No worries, post it anywhere you like. Happy to discuss.

One note about VITS before you start. VITS assumes non-volume changing transformations a.k.a det(w) == 0. Therefore there is no logdet computation there. It is explained in the paper but easy to overlook.