Knowledge distillation from a large zipformer2 to a smaller one

[joazoa]

Hello,

Can i use the knowledge distillation ( as in distillation_with_hubert.sh ) to train a smaller model based on an already trained larger zipformer b2 model or this can only be used for a fairseq model ?

And if so, can i use a large non streaming model to distill to a smaller streaming zipformer v2 ?

Best regards,

Joachim

[pkufool]

Yes, both of them. You can train a smaller model by distillating a trained larger model, the smaller model can be streaming or non-streaming even if the larger model is a non-streaming one.

[joazoa]

thank you! i will give it a ry!

[mark95]

Hello,

What knowledge distillation code should i use to train a smaller model based on an already trained larger zipformer?

[marcoyang1998]

Hi Mark,

You can have a look at https://k2-fsa.github.io/icefall/recipes/Non-streaming-ASR/librispeech/distillation.html.

You need to extract the distillation target from the large teacher Zipformer. To do so, you need to follow and modify the code here: https://github.com/k2-fsa/icefall/blob/master/egs/librispeech/ASR/pruned_transducer_stateless6/extract_codebook_index.py.