Skip to content

Fix nan issue for certain shape#10

Open
hebiao064 wants to merge 1 commit intothinking-machines-lab:mainfrom
hebiao064:main
Open

Fix nan issue for certain shape#10
hebiao064 wants to merge 1 commit intothinking-machines-lab:mainfrom
hebiao064:main

Conversation

@hebiao064
Copy link
Copy Markdown

I found that for certain shape such as (32, 128) * (128, 1024), if we replace aten::mm with matmul_persistent, the output would consist NaN hence generate gibberish output from LLMs.

I tried to fixed in by removing the dual pipeline logic(?) and it should work.

I've also tested with more shapes combination here: https://github.com/sgl-project/sglang/pull/11368/files#diff-fc88db1a639b98e0e4cceb66f72842a1612290664ba6753ad6f54d8276e0cd6aR88-R131

@Jah-yee
Copy link
Copy Markdown

Jah-yee commented Apr 15, 2026

Good day,

Thank you for digging into this tricky NaN issue! The root cause analysis with the shape combinations is very helpful.

I particularly appreciate the connection to the sglang PR - that's exactly the kind of cross-project collaboration that helps move the ecosystem forward.

Hope to see this merged soon!

Thank you for your work on this project...

Warmly, RoomWithOutRoof

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants