[None][feat] Add NIXL-LIBFABRIC support #9225
Conversation
zackyoray
changed the title
📝 WalkthroughWalkthroughThe changes add runtime selection of NIXL KV cache backends via environment variables. A new environment variable getter is introduced in envUtils, the transfer agent is updated to dynamically select and validate backends instead of using hardcoded values, Docker build scripts are enhanced to install libfabric, and NIXL is upgraded from 0.5.0 to 0.7.1. A new unit test validates the LIBFABRIC backend path. Changes
Sequence Diagram(s)sequenceDiagram
participant App as Application
participant TransferAgent as TransferAgent
participant Env as getEnvNixlBackend()
participant BackendFactory as Backend Factory
App->>TransferAgent: Initialize
TransferAgent->>Env: Read TRTLLM_NIXL_KVCACHE_BACKEND
Env->>Env: Check cache / Initialize static
Env-->>TransferAgent: Backend name (e.g., "UCX" or "LIBFABRIC")
TransferAgent->>BackendFactory: Create backend with selected name
BackendFactory-->>TransferAgent: Backend instance
TransferAgent-->>App: Ready
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes
Pre-merge checks and finishing touches❌ Failed checks (2 warnings)
✅ Passed checks (1 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (7)
cpp/tensorrt_llm/common/envUtils.cpp(1 hunks)cpp/tensorrt_llm/common/envUtils.h(1 hunks)cpp/tensorrt_llm/executor/cache_transmission/nixl_utils/transferAgent.cpp(2 hunks)cpp/tests/unit_tests/executor/transferAgentTest.cpp(1 hunks)docker/Dockerfile.multi(2 hunks)docker/common/install_libfabric.sh(1 hunks)docker/common/install_nixl.sh(2 hunks)
🧰 Additional context used
🧠 Learnings (4)
📚 Learning: 2025-09-23T15:12:38.312Z
Learnt from: nv-lschneider
Repo: NVIDIA/TensorRT-LLM PR: 7910
File: cpp/tensorrt_llm/thop/allreduceOp.cpp:352-446
Timestamp: 2025-09-23T15:12:38.312Z
Learning: In TensorRT-LLM NCCL device implementation, NCCL version 2.28+ requirements are handled at runtime in the nccl_device/config layer rather than with compile-time guards. This allows the allreduceOp to remain version-agnostic and delegates version compatibility validation to the appropriate lower-level components that can gracefully handle unsupported configurations.
Applied to files:
cpp/tensorrt_llm/executor/cache_transmission/nixl_utils/transferAgent.cpp
📚 Learning: 2025-09-23T15:01:00.070Z
Learnt from: nv-lschneider
Repo: NVIDIA/TensorRT-LLM PR: 7910
File: cpp/tensorrt_llm/kernels/nccl_device/config.cu:15-17
Timestamp: 2025-09-23T15:01:00.070Z
Learning: In TensorRT-LLM NCCL device kernels, the <sstream> header is not needed as an explicit include in config.cu because it's provided transitively through other headers. Local compilation testing confirms this works without the explicit include.
Applied to files:
cpp/tensorrt_llm/executor/cache_transmission/nixl_utils/transferAgent.cpp
📚 Learning: 2025-11-14T11:22:03.729Z
Learnt from: nzmora-nvidia
Repo: NVIDIA/TensorRT-LLM PR: 9163
File: tensorrt_llm/_torch/auto_deploy/custom_ops/quant.py:107-113
Timestamp: 2025-11-14T11:22:03.729Z
Learning: In TensorRT-LLM AutoDeploy custom ops, when adding hardware capability checks to select between kernel implementations (e.g., cuBLAS vs. CUDA kernel), use descriptive variable names that identify the specific GPU architectures or families being targeted (e.g., `is_blackwell_geforce_or_ada`) rather than generic names like `enable_cuda_core`. This makes it clear that the code is selecting an implementation path based on hardware capabilities, not enabling/disabling hardware features.
Applied to files:
cpp/tensorrt_llm/executor/cache_transmission/nixl_utils/transferAgent.cpp
📚 Learning: 2025-08-20T15:04:42.885Z
Learnt from: dbari
Repo: NVIDIA/TensorRT-LLM PR: 7095
File: docker/Dockerfile.multi:168-168
Timestamp: 2025-08-20T15:04:42.885Z
Learning: In docker/Dockerfile.multi, wildcard COPY for benchmarks (${CPP_BUILD_DIR}/benchmarks/*Benchmark) is intentionally used instead of directory copy because the benchmarks directory contains various other build artifacts during C++ builds, and only specific benchmark executables should be copied to the final image.
Applied to files:
docker/Dockerfile.multi
🧬 Code graph analysis (2)
cpp/tensorrt_llm/common/envUtils.h (1)
cpp/tensorrt_llm/common/envUtils.cpp (2)
getEnvNixlBackend(321-341)getEnvNixlBackend(321-321)
cpp/tensorrt_llm/executor/cache_transmission/nixl_utils/transferAgent.cpp (1)
cpp/tensorrt_llm/common/envUtils.cpp (2)
getEnvNixlBackend(321-341)getEnvNixlBackend(321-321)
🪛 Shellcheck (0.11.0)
docker/common/install_libfabric.sh
[warning] 22-22: Quote this to prevent word splitting.
(SC2046)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Pre-commit Check
🔇 Additional comments (7)
docker/Dockerfile.multi (1)
44-44: LGTM! Proper integration of libfabric installation.The libfabric installation script is correctly added to the COPY list and executed in the appropriate sequence (after UCX, before NIXL), which aligns with the dependency chain.
Also applies to: 84-86
cpp/tensorrt_llm/common/envUtils.h (1)
91-92: LGTM! Function declaration follows established patterns.The new
getEnvNixlBackend()function is properly placed and consistent with other environment accessors in this header.cpp/tensorrt_llm/executor/cache_transmission/nixl_utils/transferAgent.cpp (1)
31-31: LGTM! Required header for backend validation.The
<set>header is correctly added to support thekSUPPORTED_BACKENDSlookup.cpp/tensorrt_llm/common/envUtils.cpp (1)
321-341: LGTM! Implementation follows established patterns.The
getEnvNixlBackend()function correctly implements thread-safe lazy initialization withstd::call_once, consistent with other environment getters in this file. The default value of "UCX" ensures backward compatibility when the environment variable is not set.docker/common/install_libfabric.sh (1)
1-27: LGTM! Libfabric installation script is well-structured.The script properly installs dependencies, builds libfabric v1.21.0 with the necessary providers (verbs, tcp, rxm), and cleans up afterward. The LD_LIBRARY_PATH update ensures the library is discoverable at runtime.
docker/common/install_nixl.sh (2)
7-7: LGTM! Libfabric path correctly propagated to NIXL build.The
LIBFABRIC_INSTALL_PATHvariable is properly defined and passed to Meson via-Dlibfabric_path, enabling NIXL to link against the newly installed libfabric library. This aligns with the installation performed byinstall_libfabric.sh.Also applies to: 42-42
8-8: NIXL version 0.7.1 compatibility verified—no action needed.NIXL 0.7.1 is a maintenance and polish release focused on Libfabric stability improvements, and the codebase is already fully compatible. The codebase includes:
- Multiple Libfabric fixes (offsets, asymmetrical rails, metadata/partial multi-load transfers), which are supported by the existing backend selection logic (
kSUPPORTED_BACKENDS = {"UCX", "LIBFABRIC"})- Dedicated tests for the Libfabric backend (
LibfabricBackendBasictest)- Proper handling of the async Device API change where
postXferReqreturnsNIXL_IN_PROG
cpp/tensorrt_llm/executor/cache_transmission/nixl_utils/transferAgent.cpp
Show resolved
Hide resolved
zackyoray
changed the title
zackyoray
force-pushed
the
yorayz/nixl_libfabric
branch
2 times, most recently
from
02fe264 to
1837830
Compare
|
I have no issues with the code modifications, but as mentioned above, we need to confirm the legal compliance of the third-party library. |
zackyoray
force-pushed
the
yorayz/nixl_libfabric
branch
2 times, most recently
from
2290069 to
9588ab8
Compare
| std::string nixlBackend = common::getEnvNixlBackend(); | ||
| // List of supported backends - extend this list as new backends are added | ||
| static const std::set<std::string> kSUPPORTED_BACKENDS = {"UCX"}; | ||
| static const std::set<std::string> kSUPPORTED_BACKENDS = {"UCX", "LIBFABRIC"}; |
There was a problem hiding this comment.
Is there any other config that is necessary for libfabric or it autoconfigures everything once selected?
There was a problem hiding this comment.
Currently there is no other config needed beside changing backend name from UCX to LIBFABRIC.
zackyoray
force-pushed
the
yorayz/nixl_libfabric
branch
2 times, most recently
from
a689963 to
605c087
Compare
Allows users to use LIBFABRIC as NIXL backend for KV cache transmission. Usage: TRTLLM_NIXL_KVCACHE_BACKEND=LIBFABRIC # (or UCX) Currently supports only UCX and LIBFABRIC as NIXL KV cahce backends. Unsupported backend types will fallback to Default (UCX). Signed-off-by: Yoray Zack <[email protected]>
zackyoray
force-pushed
the
yorayz/nixl_libfabric
branch
from
aa27880 to
cef73bf
Compare
|
/bot run --stage-list "Build-Docker-Images" |
|
PR_Github #25185 [ run ] triggered by Bot. Commit: |
zackyoray
force-pushed
the
yorayz/nixl_libfabric
branch
from
cef73bf to
b49d5ef
Compare
|
/bot run --stage-list "Build-Docker-Images" |
|
PR_Github #25189 [ run ] triggered by Bot. Commit: |
|
PR_Github #25185 [ run ] completed with state |
Signed-off-by: Yoray Zack <[email protected]>
zackyoray
force-pushed
the
yorayz/nixl_libfabric
branch
from
b49d5ef to
aa2fe9a
Compare
|
/bot run --stage-list "Build-Docker-Images" |
|
PR_Github #25209 [ run ] triggered by Bot. Commit: |
|
PR_Github #25189 [ run ] completed with state |
|
PR_Github #25209 [ run ] completed with state |
5 participants
Summary by CodeRabbit
Release Notes
New Features
Tests
Chores
Description
Test Coverage
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...Provide a user friendly way for developers to interact with a Jenkins server.
Run
/bot [-h|--help]to print this help message.See details below for each supported subcommand.
run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental)]Launch build/test pipelines. All previously running jobs will be killed.
--reuse-test (optional)pipeline-id(OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.--disable-reuse-test(OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.--disable-fail-fast(OPTIONAL) : Disable fail fast on build/tests/infra failures.--skip-test(OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.--stage-list "A10-PyTorch-1, xxx"(OPTIONAL) : Only run the specified test stages. Examples: "A10-PyTorch-1, xxx". Note: Does NOT update GitHub check status.--gpu-type "A30, H100_PCIe"(OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.--test-backend "pytorch, cpp"(OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.--only-multi-gpu-test(OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.--disable-multi-gpu-test(OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.--add-multi-gpu-test(OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.--post-merge(OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx"(OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx".--detailed-log(OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.--debug(OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in thestage-listparameter to access the appropriate container environment. Note: Does NOT update GitHub check status.For guidance on mapping tests to stage names, see
docs/source/reference/ci-overview.mdand the
scripts/test_to_stage_mapping.pyhelper.kill
killKill all running builds associated with pull request.
skip
skip --comment COMMENTSkip testing for latest commit on pull request.
--comment "Reason for skipping build/test"is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.reuse-pipeline
reuse-pipelineReuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.