Concatenated dim #336
Open
Concatenated dim #336
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
oleksost
reviewed
Aug 11, 2025
| @property | ||
| def global_expanded_shape(self) -> tuple[int, ...]: | ||
| return sum((dim.global_expanded_shape for dim in self._dims), ()) | ||
| def local_to_global_partial( |
There was a problem hiding this comment.
Is it possible to add function signatures at least to newly introduced functions including short description of what new functions do? This would simplify reviewing a lot.
There was a problem hiding this comment.
I'll add some doc. Where is the signature missing? See also their descriptions in the associated TensorMeta methods. https://github.com/ServiceNow/Fast-LLM/pull/336/files#diff-3517f1a1541e286fce5d2a79d1eb3efab238c23f888696fc44e37d85a1692671R195
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
✨ Description
Another round of prequisites for TP Mamba and misc improvements.
ConcatenatedTensorDimto support complex parallel dimensions (ex. SSM inner projection). Clean up theTensorDiminterface and move some functionality from TensortoTensorDim` for smoother integration.Initializerclass to allow for more detailed initialization strategies. Addrequires_global_initializationfor initializers that won't work on local slices (ex. init_A).init_uniform_centered