[DSV3] GroupedExperts weights conversion optimization

[wwwjn]

Algorithm summary:

Numerical comparison using 16B model:

• Note: the final results are not exactly same because the numerics are small

[wwwjn]

Need to clean up the test part a little bit before ready to review :)

Updated: Ready for review now

[fegin]

I want to return instead of continue, but I'm not sure if that is true. Also see the comment below. I want to understand if we can avoid looping layers.

Suggested change

	if len(experts) == expected_n_experts:
	if len(experts) < expected_n_experts:
	continue

[wwwjn]

When a new fqn is processed, the _concatenate_local_expert_weights function is called to check if we can concatenate the individual experts weights into GroupedExpert weights. If the fqn has layer_num, the only possible layer can be merged is layer with id=layer_num. In this way, we could remove the loop of layers as you suggested.

So we could use return here instead of continue after remove the loop

[tianyu-l]

Sounds good to me. Please address any remaining concerns @fegin has.