model: support ModernBERT embedding model

[0xtoward]

Motivation

#7429 Support ModernBert

Modifications

create modernbert.py and its unit test

Accuracy Tests

similarity diff tensor(1.1921e-07)
similarity diff tensor(5.9605e-08)
...
PASSED

Benchmarking and Profiling

for torch_native backend : sgl_time/st_time 0.306
for triton backend : sgl_time/st_time 0.184
both on my local rtx4070

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.

[gemini-code-assist]

Summary of Changes

Hello @0xtoward, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly expands the system's capabilities by integrating the ModernBERT embedding model. This allows the platform to process and generate embeddings using this specific architecture, broadening the range of supported models for various natural language processing tasks. The changes involve adding the model's core implementation, updating system configurations to recognize it, and incorporating it into the testing framework.

Highlights

• New Model Support: Added support for the ModernBERT embedding model, enabling its use within the system for embedding-related tasks.
• Model Definition: Introduced a new Python module (modernbert.py) that defines the ModernBERT model architecture, including its embedding, attention, MLP, and encoder components.
• Configuration Update: Updated the model configuration to recognize ModernBertModel as a valid embedding model.
• Testing Integration: Integrated ModernBERT into the existing test suite for encoder embedding models, specifically adding answerdotai/ModernBERT-base.
• Backend Limitation: Currently, ModernBERT support is limited to the torch_native backend, with flashinfer and triton backends explicitly skipped in tests.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request adds support for the ModernBERT embedding model. The implementation is well-structured and follows the existing patterns in the codebase. I have identified a potential precision issue related to float32 handling in the rotary embeddings, which could impact model accuracy, and have provided a high-severity comment with a suggested fix. Additionally, I've included a few medium-severity comments to improve code clarity and maintainability by removing redundant code and simplifying conditional logic.

[0xtoward]

/gemini review

[gemini-code-assist]

Code Review

This pull request adds support for the ModernBERT embedding model. The implementation is well-structured and follows the existing patterns in the codebase. I've identified two high-severity issues in the new modernbert.py file that should be addressed. One is related to missing weight mappings for the MLP layers, which will prevent the model from loading correctly. The other is a potential out-of-bounds access in the rotary embedding calculation due to missing position clamping. The rest of the changes, including the test additions, look good.

[0xtoward]

/gemini review

[gemini-code-assist]

Code Review

Thank you for your contribution! This pull request adds support for the ModernBERT embedding model, which is a great addition. The implementation looks solid and follows the existing patterns in the codebase. I've added a few suggestions to improve code clarity and maintainability in the new modernbert.py file.

[0xtoward]

Hi @hnyls2002 , can you check this pr and add run-ci label

[0xtoward]

Hi @mickqian I saw unit-test-backend-1-gpu-amd (linux-mi300-gpu-1, 10) failed, which is sensitive because I edited triton_backend by 2 lines for encoder-only.
After checking test_radix_attention.py, however, I thought that test is only for decoder model, and shouldn't caused by my edit. Is that true? Could you help to merge this PR?
Thanks a lot!