Skip to content

Moe bf16 ep #4144

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 17 commits into from
Nov 28, 2025
Merged

Moe bf16 ep #4144

Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
d767bca
refactor pytorch.nn.moe
grimoire Nov 20, 2025
d0865d4
add ep support
grimoire Nov 21, 2025
f663b34
fix tp
grimoire Nov 21, 2025
30fbac1
support blocked fp8 moe with split_size<world_size
grimoire Nov 24, 2025
2e64bb5
unit test allow both fa3 and fa
grimoire Nov 24, 2025
1107fe2
add singleton
grimoire Nov 24, 2025
89e7050
singleton and ctxmgrbase
grimoire Nov 24, 2025
bd53fb5
comment
grimoire Nov 25, 2025
6a09849
Merge branch 'main' into moe-bf16-ep
grimoire Nov 25, 2025
6d581eb
add static
grimoire Nov 25, 2025
32e0699
remove chunk
grimoire Nov 25, 2025
eeaafef
merge main
grimoire Nov 26, 2025
640bc25
remove forward dptp
grimoire Nov 26, 2025
260fba7
bound check
grimoire Nov 26, 2025
755a987
remove monkey patch
grimoire Nov 27, 2025
1e1d669
merge main
grimoire Nov 27, 2025
00a096c
rename kernel
grimoire Nov 27, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions lmdeploy/pytorch/backends/cuda/graph_runner.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
import torch
from torch.profiler import record_function

from lmdeploy.pytorch.backends.deepep_moe_checker import moe_backend
from lmdeploy.pytorch.backends.deepep_moe_checker import get_moe_backend
from lmdeploy.pytorch.backends.selector import get_backend
from lmdeploy.pytorch.config import BackendConfig, CacheConfig, ModelConfig
from lmdeploy.pytorch.model_inputs import StepContext, get_step_ctx_manager
Expand Down Expand Up @@ -252,7 +252,7 @@ def prepare_inputs_for_generation(
):
"""Prepare inputs."""

if moe_backend.use_deepep_moe_backend():
if get_moe_backend().use_deepep_moe_backend():
from dlblas.layers.moe.token_dispatcher import DeepEPBuffer, DeepEPMode
deepep_mode = DeepEPMode.LOW_LATENCY if context.is_decoding else DeepEPMode.NORMAL
DeepEPBuffer.set_deepep_mode(deepep_mode)
Expand Down
Loading