Add and test quantization from Kokoro

[mmwillet]

Add basic quantization support for Kokoro.

[ecyht2]

I am not that expert in ML and AI and also GGML, so I am curious how do you decide which weights to quantize and which not to quantize?

[danielzgtg]

@ecyht2 ggml-cpu.c only supports FP32 for most non-matmul operations. Our convert_dac_to_f16 skips the weight suffixed "alpha" because our dac_model.cpp:build_residual_unit uses util.cpp:snake_1d which uses ggml.c:ggml_mul which uses ggml-cpu.c:ggml_compute_forward_mul which requires FP32. This is 1/3 of the weights in dac_model.h. Convolutions and biases are the other 2/3 and support quantization. This can't be the real reason because there's always ggml_cast.

Then there's benefit-cost ratio. GPT3's weights for examples are 66.2% MLP and 33.1% attention. As attention has less weights than MLP, we have quantize_output_heads and quantize_cross_attn_kv opt-in only. Although we have quantize_text_embeddings, its 0.7% is negligible. This is analogous to dia_model.h and parler_model.h.

I'm still looking through kokor_model.cpp, which is more complicated than the other two. A quick skim shows that most weights should be quantizable. I'd need percentages of each weight type before recommending which are actually worth quantizing.

[mmwillet]

@danielzgtg excellent summary! @ecyht2 In addition to what has already been said I'd add a couple things:

First, it is helpful to be specific. The quantization technique that we are using (i.e. that GGML has implemented as its baseline) is a form of dynamic quantization called the k-quant system (for a breakdown on how this works see here or here). Different forms of quantization, e.g. AWQ, static + QAT, etc, have different advantages and disadvantages and work better with different architectures, operations and topologies.

For the purposes of understanding how k-quant quantization will impact model performance, it is important to keep in mind that there is a relationship between model architecture, size, training procedure and the impact that quantization (k-quant in this context) will have on model performance. From my experience here are a few examples (please treat these examples as purely anecdotal):

• decoder-only transformer architectures trained with higher dropout tend to display less perplexity degradation when quantized.
• Visual classification models that use a transformer architecture as opposed to a purely residual convolutional architecture tend to display less AUC under ROC degradation when quantized.
• Finetuned LLMs display less task specific perplexity degradation when quantized compared to purely pretrained LLMs.

As such, I have found that with novel architectures this process necessarily involves some experimentation. For example, for Kokoro I really don't know how well the LSTMs will handle quantization and the literature is not clear on the subject. While the literature generally implies that the k-quant system works fine for LSTMs, I haven't adequately thought through the impact that quantization will have along side the non-linear activation functions present in LSTMs or along side the general rnn architecture.

[mmwillet]

addressed by #77