Moe bf16 ep #4144
Moe bf16 ep #4144
Conversation
| # we don't need to read this, it would be passed to ray workers | ||
| # If Ray is launched from outside, it may fail to access the environment variables. | ||
| os.getenv('DEEPEP_MAX_BATCH_SIZE', None) | ||
| os.getenv('DEEPEP_MAX_TOKENS_PER_RANK', None) |
There was a problem hiding this comment.
Do we need to set those envs manually?
There was a problem hiding this comment.
DLBlas read these vars to build buffer.
https://github.com/DeepLink-org/DLBlas/blob/1710a860f654ddf50907251ec51670910368ee45/dlblas/layers/moe/token_dispatcher.py#L43
|
Qwen3_30b_A3B_Thinking_2507 dp2 ep4
|
|
#4084 has been merged. |
There was a problem hiding this comment.
https://github.com/InternLM/lmdeploy/blob/main/lmdeploy/metrics/metrics_processor.py#L20
May consider use singleton to simply this as well, or leave it to me just as a reminder here.
| num_warps = 8 | ||
| num_experts = num_recv_tokens_per_expert.shape[0] | ||
| hidden_size = recv_x.shape[1] | ||
| # grid = (triton.cdiv(hidden_size, BLOCK_D), num_experts) |
|
|
||
|
|
||
| @triton.jit | ||
| def _fwd_kernel_ep_scatter_1( |
There was a problem hiding this comment.
Though from dlblas, should be their problem, _fwd_kernel_ep_scatter_1, _fwd_kernel_ep_scatter_2 seems confusing function names.
| class MoeType(Enum): | ||
| """Batch ecex type.""" | ||
| Default = auto() | ||
| DSAsyncDecode = auto() |
There was a problem hiding this comment.
What's the meaning of DS?
There was a problem hiding this comment.
Deepseek, I guess?
3 participants
backends/moe.py and nn/moe.py has been refactored.
Reuse token dispatcher in DLBlas