Refactor script to use 'overwrites' variable for command-line arguments in training scripts #1473
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ts in training scripts
facebook-github-bot
added
the
CLA Signed
tianyu-l
reviewed
Jul 28, 2025
| @@ -34,7 +34,7 @@ export LOGLEVEL=INFO | |||
| export FI_PROVIDER="efa" | |||
| # Ensure that P2P is available | |||
| # export NCCL_P2P_DISABLE=1 | |||
| export NCCL_IB_DISABLE=1 | |||
| # export NCCL_IB_DISABLE=1 | |||
There was a problem hiding this comment.
why disabling this? cc @lessw2020 if you have more context
There was a problem hiding this comment.
In my experience, I always had to comment this out as new severs typically support IB, which is probably similar experience to others who use torchtitan for pre-training.
tianyu-l
approved these changes
Jul 31, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The goal of this PR is to add support for command line arguments to the bash training scripts. The
run_train.shhad support foroverrides, however, themultinode_trainer.slurmscript did not. Thisoverridesflag add supports for commands like:sbatch multinode_trainer.slurm --job.description="TEST_RUN"However, there is a problem with the current
overridesimplementation, when passing arguments with space such as"TEST RUN"instead of"TEST_RUN"then the variablejob.descriptionwould only getTESTas input and the training script throws an error for unrecognizing the argumentRUNwhich is passed in a different line. To address this I simplify the code and directly pass the additional overrides through$@. This solves the issue for commands such as:sbatch multinode_trainer.slurm --job.description="TEST RUN"