[QUESTION] Will the data get re-shuffled if the sequence length is modified during training?

[SefaZeng]

Your question
If I change the seq-length during training, like 4096 -> 8192, or 4096 -> 1024, will the data loader reset the order of the dataset or continue with the old order?
I want to change the seq-length after training maybe 100k steps, and I need the model to continue train with the left data, will Megatron-LM do this by default?