Skip to content

Conversation

@mxuax
Copy link

@mxuax mxuax commented Dec 11, 2025

This PR follows Diffusion Ulysses-Sequence-Parallelism support PR#189, allowing users to set Ring Attention for diffusion model, e.g., qwen-image.

Purpose

  1. Create Ring Attention Module (vllm_omni/diffusion/ring_attn)

  2. Refactor Distributed State (vllm_omni/diffusion/distributed/parallel_state.py)

  3. Implement Ring Attention Logic (vllm_omni/diffusion/attention/layer.py)

  4. Modify the generic Attention layer to support Ring Attention and Hybrid (Ulysses + Ring) modes.

  5. Verification and Testing
    Create a new test file to verify the correctness of the Ring Attention implementation on the Qwen model components.

Test Plan

  • UT: Test File: tests/distributed/test_ring_attention.py

Test Result


Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft.

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

Signed-off-by: Didan Deng <[email protected]>
Signed-off-by: Didan Deng <[email protected]>
Signed-off-by: Didan Deng <[email protected]>
Signed-off-by: Didan Deng <[email protected]>
Signed-off-by: Didan Deng <[email protected]>
Signed-off-by: Didan Deng <[email protected]>
Signed-off-by: Didan Deng <[email protected]>
Signed-off-by: Didan Deng <[email protected]>
Signed-off-by: Didan Deng <[email protected]>
Signed-off-by: Didan Deng <[email protected]>
Signed-off-by: Didan Deng <[email protected]>
Signed-off-by: Didan Deng <[email protected]>
Signed-off-by: Didan Deng <[email protected]>
Signed-off-by: Didan Deng <[email protected]>
Signed-off-by: Didan Deng <[email protected]>
wtomin and others added 30 commits December 22, 2025 12:59
Signed-off-by: Didan Deng <[email protected]>
Signed-off-by: Didan Deng <[email protected]>
Signed-off-by: Didan Deng <[email protected]>
Signed-off-by: Didan Deng <[email protected]>
… remove Ulysses support. Implement ring attention logic in forward pass. Update build_parallel_attention_strategy to support pure Ring Attention.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants