Using official HuggingFace script to convert DCP weights to HF format，the outputs are not human-readable

[guang11644331]

DCP -> torch (in PyTorch, see https://github.com/pytorch/torchtitan/blob/main/docs/checkpoint.md)
torch -> HF (from HF, although missing params.json if saved from DCP)

Does anyone have a working convert script or know how to fix this issue?

[fegin]

@guang11644331 Can I confirm the script is from PyTorch or Torchtitan? cc., @ankitageorge

[guang11644331]

@guang11644331 Can I confirm the script is from PyTorch or Torchtitan? cc., @ankitageorge

Thanks,
torch -> HF script:(https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/convert_llama_weights_to_hf.py)

[guang11644331]

@fegin

1. The DCP -> PyTorch script is the method recommended in the Torchtitan doc, utilizing PyTorch for the transformation. (https://github.com/pytorch/torchtitan/blob/main/docs/checkpoint.md)

2. The PyTorch -> Hugging Face (HF) script is sourced from the Hugging Face project. (https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/convert_llama_weights_to_hf.py)

3. Are there any other recommended methods for converting from DCP to HF ?
   Thank you.

[fegin]

Are you using both tools? Like from DCP -> PyTorch -> HF and load the HF checkpoint to verify? It is very possible that both tools are correct. The issue may be the model definitions are different (for example, in TorchTitan, the definition may be layer1.weight while it is layer1.w in HF). Are you able to check the state_dict keys difference?

[vwxyzjn]

Hey I recently figured this out. https://gist.github.com/vwxyzjn/81338803f612d3bef5c0cbfe4b5982b6

It turns out HF's llama implementation and torchtitan's implementation are slightly different: HF's llama impl would do a permutation of the attention weights.

I trained model using the Tulu 3 dataset. After the conversion, the model's output is reasonable as expected.

Will follow up with a PR once #1300 is merged (since format is slightly differ), but want to give a comment on this issue in case anyone is working on similar things.

[fegin]

Thanks @vwxyzjn for confirming it. This is what I meant "The issue may be the model definitions are different (for example, in TorchTitan, the definition may be layer1.weight while it is layer1.w in HF). Are you able to check the state_dict keys difference?", @guang11644331

[tianyu-l]

@vwxyzjn This issue was discovered long time ago #335
Solution of DCP -> HF is here #305 (comment)
(We should land this or something equivalent.)

What I don't understand is if we follow the DCP -> torch -> HF path, who should do the permutation? I think the torch->HF script should, but it doesn't?