[QUESTION] How to set --rotary-seq-len-interpolation-factor for rope scaling?

[eagle705]

Hi,

I'm curious if it's possible to set --rotary-seq-len-interpolation-factor in Megatron-LM to match huggingface's rope_scaling setting ({"type": "dynamic", "factor": 2.0}).

Is there any information you can share on how the --rotary-seq-len-interpolation-factor option is used?

[ethanhe42]

Megatron-LM/megatron/core/models/common/embeddings/rotary_pos_embedding.py

Line 52 in 4bd4e74

seq_len_interpolation_factor (float, optional): scale of linearly interpolating RoPE for longer sequences. The value must be a float larger than 1.0. Defaults to None

[github-actions[bot]]

Marking as stale. No activity in 60 days.