Skip to content

Remove the fixed eot_token mechanism for SFT #927

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
loadams merged 4 commits into from
Oct 30, 2024
Merged

Remove the fixed eot_token mechanism for SFT #927

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
7697726
Remove the fixed `eot_token` mechanism for SFT
Xingfu-Yi Sep 15, 2024
f370d3c
Merge branch 'master' into eot_token_bugfix
tjruwase Sep 20, 2024
44da9cd
Merge branch 'master' into eot_token_bugfix
loadams Oct 29, 2024
73b814e
Merge branch 'master' into eot_token_bugfix
loadams Oct 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions applications/DeepSpeed-Chat/training/step1_supervised_finetuning/main.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -191,7 +191,13 @@ def parse_args():
parser.add_argument(
"--add_eot_token",
action='store_true',
help="Add <|endoftext|> as additional special token to tokenizer")
help="Add `eot_token` as additional special token to tokenizer")
parser.add_argument(
"--eot_token",
type=str,
default="<|endoftext|>",
help="Specify the format of the `eot_token`",
)
## Print loss
parser.add_argument('--print_loss',
action='store_true',
Expand Down Expand Up @@ -234,8 +240,7 @@ def main():
torch.distributed.barrier()

# load_hf_tokenizer will get the correct tokenizer and set padding tokens based on the model family
args.end_of_conversation_token = "<|endoftext|>"
additional_special_tokens = args.end_of_conversation_token if args.add_eot_token else None
additional_special_tokens = args.eot_token if args.add_eot_token else None
tokenizer = load_hf_tokenizer(args.model_name_or_path,
fast_tokenizer=True,
add_special_tokens=additional_special_tokens)
Expand Down
Loading