[QUESTION] What's the internal difference for training when setting only "fp8-format" or setting "fp8-format"+"bf16"

[dong-liuliu]

Your question
I'm trying to train GPT/LLAMA on top of Megatron-LM, but confused on fp8 performance.

Setting fp8 format parameters together with "--bf16" is much better than the situation without "--bf16". So what's difference between them inside Megatron-LM?

When setting fp8 + bf16, will Megatron-LM try to split some computation to bf16 if more efficient, or to fp8 for high throughtput?

[yaox12]

Only "fp8-format": FP32 + FP8
"fp8-format"+"bf16": BF16 + FP8

You could consider FP8 as an additional feature on top of your current (BF16 or no BF16) training recipe.

[yanchenmochen]

when I use this fp8 paramaters , same parameter fp16 can run successfully, but fp8 will fail becaue OOM.

[yanchenmochen]

do you know what parameter I can try. torch.OutOfMemoryError。

[github-actions[bot]]

Marking as stale. No activity in 60 days.