Skip to content

[Dev] Hybrid dev branch #347

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 176 commits into
base: main
Choose a base branch
from
Open

[Dev] Hybrid dev branch #347

wants to merge 176 commits into from

Conversation

RaymondLi0
Copy link
Contributor

@RaymondLi0 RaymondLi0 commented Aug 7, 2025 •
edited
Loading

✨ Description

For tracking: Hybrid-SSM dev branch

Outstanding issues

  • Missing preprocessing when flash-attn is disabled for vision-encoder. -> KeyError: 'image_encoder_attention_mask' (is an issue to run the tests)
  • when vision_encoder.image_break_token is not set, shape-mismatch error in the Multimodal-embedding layer.
  • debug-layer-outputs/gradients hangs.

jlamypoirier and others added 24 commits July 25, 2025 17:14
@jlamypoirier
Mamba 2, misc
c14b764
@RaymondLi0
flexible import
fa21174
@RaymondLi0
Merge branch 'tp_mamba' into raymond/debug_mm_tp_mamba
a3e5bde
@RaymondLi0
update import
d3cc158
@RaymondLi0
fix automodel export
6d245c0
@RaymondLi0
try: remove assert for TP and distillation
61ecb5d
@RaymondLi0
more verbose config
2565dac
@RaymondLi0
use local token_ids instead of modifying batch
7a7f12c
@RaymondLi0
fix allreduce
743b42c
@RaymondLi0
fix
c7247dc
@RaymondLi0
revert images_sizes conversion to np array
3074ec9
@RaymondLi0
debug logs
c4cdd86
@RaymondLi0
rm debug logs
24d7a05
@RaymondLi0
changes for stp reverse-kl
37ddef4
@RaymondLi0
reverse kl: add clamping
a0d7a09
@RaymondLi0
add loss mask for vision. should also handle padded sequences
72945bc
@RaymondLi0
simplify loss-masking
26541c6
@RaymondLi0
data time debug
d66942b
@RaymondLi0
fix mm embedding indices
f00509d
@RaymondLi0
cleanup
8a5e8f0
@oleksost @jlamypoirier
@bigximik @RaymondLi0
Mamba2 to be merged (#349)
01783de 
Co-authored-by: Joel Lamy-Poirier <[email protected]>
Co-authored-by: Denis Kochetkov <[email protected]>
Co-authored-by: Toolkit User <[email protected]>
@oleksost
checkpoint format for llava
ad6c0c0
@RaymondLi0
fixes
cbc94e0
@oleksost
[Dev Hybrid] Distill with loss_mask (SFT dataset) and sequence-TP (#350)
5263996
@jlamypoirier jlamypoirier mentioned this pull request Aug 15, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@RaymondLi0 @sohamparikh @shruthan @tscholak @jlamypoirier @oleksost