Skip to content

Word Embedding init std adjustment#85

Open
hatanp wants to merge 3 commits intomainfrom
emb_init
Open

Word Embedding init std adjustment#85
hatanp wants to merge 3 commits intomainfrom
emb_init

Conversation

@hatanp
Copy link

@hatanp hatanp commented May 6, 2025

This pull request adds two new arguments to adjust word embedding initialization:
--adjust-word-embedding-init which enables reading a std value from a new different variable,
and
--word-embedding-init-std which controls the init std if above flag is set.
I chose to have the enable flag so that we do not break any existing functionality in a case where a script is changing normal init std and thus the word embedding init std too. This way we retain that functionality as --adjust-word-embedding-init defaults to False if not set.

This does not adjust positional embeddings and they are currently initialized with the default initialization method.

@hatanp hatanp requested review from mngom2 and saforem2 May 6, 2025 22:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant