[TRTLLM-9000][feat] Add multi-node Perf Tests into CI #8800

chenfeiz0326 · 2025-10-30T14:54:05Z

Summary by CodeRabbit

Chores
- Add DSR1 FP4 multinode perf cases.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run /bot [-h|--help] to print this help message.

See details below for each supported subcommand.

run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental)]

Launch build/test pipelines. All previously running jobs will be killed.

--reuse-test (optional)pipeline-id (OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.

--disable-reuse-test (OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.

--disable-fail-fast (OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test (OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.

--stage-list "A10-PyTorch-1, xxx" (OPTIONAL) : Only run the specified test stages. Examples: "A10-PyTorch-1, xxx". Note: Does NOT update GitHub check status.

--gpu-type "A30, H100_PCIe" (OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.

--test-backend "pytorch, cpp" (OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.

--only-multi-gpu-test (OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.

--disable-multi-gpu-test (OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.

--add-multi-gpu-test (OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.

--post-merge (OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.

--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" (OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx".

--detailed-log (OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.

--debug (OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in the stage-list parameter to access the appropriate container environment. Note: Does NOT update GitHub check status.

For guidance on mapping tests to stage names, see docs/source/reference/ci-overview.md
and the scripts/test_to_stage_mapping.py helper.

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request. --comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

chenfeiz0326 · 2025-10-30T14:55:55Z

/bot run --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Post-Merge-1"

coderabbitai · 2025-10-30T14:56:24Z

📝 Walkthrough

Walkthrough

Added two runtime logging statements to jenkins/L0_Test.groovy that print the contents of scriptContent and scriptExec variables before they are written to local and remote locations, respectively.

Changes

Cohort / File(s)	Summary
Logging additions `jenkins/L0_Test.groovy`	Added two runtime log statements that output `scriptContent` and `scriptExec` before writing to local and remote destinations.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~2 minutes

Pre-merge checks and finishing touches

❌ Failed checks (3 warnings)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	You can run `@coderabbitai generate docstrings` to improve docstring coverage.
Description Check	⚠️ Warning	The pull request description is largely incomplete and does not follow the required template structure. The PR title is a placeholder ("tmp"), the Description section is entirely empty with only the template comment remaining, the Test Coverage section contains no actual test information, and the PR Checklist items are not properly addressed despite being mostly unchecked. While the author checked one checkbox at the end, none of the critical descriptive sections have been filled out. The template clearly requires formatted sections for title, description explaining the changes, and test coverage information, all of which are absent from this submission.	The author must provide a complete PR description by: filling in the PR title using the required format (e.g., [JIRA-1234][chore] Add logging to jenkins script), adding a clear explanation in the Description section of what changes are being made and why, listing the relevant tests in the Test Coverage section that validate the changes, and properly reviewing and checking the relevant items in the PR Checklist. The current submission appears to be the empty template without substantive content.
Title check	⚠️ Warning	The pull request title describes adding multi-node performance tests to CI, but the actual changes only add logging statements to an existing test file with no functional behavior changes.	Update the title to reflect the actual changes, such as 'Add runtime logging to L0_Test.groovy for script content debugging' or similar, which accurately describes the logging additions made.

✨ Finishing touches

📝 Generate docstrings

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

tensorrt-cicd · 2025-10-30T15:02:46Z

PR_Github #23060 [ run ] triggered by Bot. Commit: d31d4b1

tensorrt-cicd · 2025-10-30T16:55:26Z

PR_Github #23060 [ run ] completed with state SUCCESS. Commit: d31d4b1
/LLM/main/L0_MergeRequest_PR pipeline #17387 (Partly Tested) completed with status: 'SUCCESS'

chenfeiz0326 · 2025-11-05T08:02:17Z

/bot run --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Perf-Sanity-Post-Merge-1"

tensorrt-cicd · 2025-11-05T08:07:55Z

PR_Github #23614 [ run ] triggered by Bot. Commit: 2d9ecc1

tensorrt-cicd · 2025-11-05T08:45:41Z

PR_Github #23614 [ run ] completed with state FAILURE. Commit: 2d9ecc1
/LLM/main/L0_MergeRequest_PR pipeline #17768 (Partly Tested) completed with status: 'FAILURE'

chenfeiz0326 · 2025-11-05T10:09:49Z

/bot run --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Perf-Sanity-Post-Merge-1"

tensorrt-cicd · 2025-11-05T10:15:31Z

PR_Github #23637 [ run ] triggered by Bot. Commit: 6cb155c

tensorrt-cicd · 2025-11-05T12:20:37Z

PR_Github #23637 [ run ] completed with state SUCCESS. Commit: 6cb155c
/LLM/main/L0_MergeRequest_PR pipeline #17784 (Partly Tested) completed with status: 'FAILURE'

chenfeiz0326 · 2025-11-06T03:32:45Z

/bot run --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Perf-Sanity-Post-Merge-1"

tensorrt-cicd · 2025-11-06T03:48:38Z

PR_Github #23699 [ run ] triggered by Bot. Commit: c0943af

tensorrt-cicd · 2025-11-06T06:20:00Z

PR_Github #23699 [ run ] completed with state FAILURE. Commit: c0943af
/LLM/main/L0_MergeRequest_PR pipeline #17833 (Partly Tested) completed with status: 'FAILURE'

chenfeiz0326 · 2025-11-11T07:29:22Z

/bot run --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Perf-Sanity-Post-Merge-1"

chenfeiz0326 · 2025-11-11T07:36:47Z

/bot run --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Perf-Sanity-Post-Merge-1"

tensorrt-cicd · 2025-11-11T07:43:35Z

PR_Github #24134 [ run ] triggered by Bot. Commit: 853b08d

chenfeiz0326 · 2025-11-27T03:26:37Z

/bot run --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-PerfSanity-Post-Merge-1"

tensorrt-cicd · 2025-11-27T03:33:57Z

PR_Github #25953 [ run ] triggered by Bot. Commit: 016fd25

tensorrt-cicd · 2025-11-27T04:43:46Z

PR_Github #25953 [ run ] completed with state FAILURE. Commit: 016fd25
/LLM/main/L0_MergeRequest_PR pipeline #19683 (Partly Tested) completed with status: 'FAILURE'

chenfeiz0326 · 2025-11-27T06:17:41Z

/bot run --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-PerfSanity-Post-Merge-1"

tensorrt-cicd · 2025-11-27T06:23:14Z

PR_Github #25972 [ run ] triggered by Bot. Commit: 3057f8c

tensorrt-cicd · 2025-11-27T07:07:59Z

PR_Github #25972 [ run ] completed with state SUCCESS. Commit: 3057f8c
/LLM/main/L0_MergeRequest_PR pipeline #19697 (Partly Tested) completed with status: 'FAILURE'

chenfeiz0326 · 2025-11-27T08:53:50Z

/bot run --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-PerfSanity-Post-Merge-1"

tensorrt-cicd · 2025-11-27T08:59:45Z

PR_Github #26007 [ run ] triggered by Bot. Commit: da0d471

tensorrt-cicd · 2025-11-27T10:27:59Z

PR_Github #26007 [ run ] completed with state SUCCESS. Commit: da0d471
/LLM/main/L0_MergeRequest_PR pipeline #19730 (Partly Tested) completed with status: 'FAILURE'

Signed-off-by: Chenfei Zhang <[email protected]>

chenfeiz0326 · 2025-11-27T11:15:05Z

/bot run --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Perf-Sanity-Post-Merge-1"

tensorrt-cicd · 2025-11-27T11:21:17Z

PR_Github #26028 [ run ] triggered by Bot. Commit: 0235dde

tensorrt-cicd · 2025-11-27T12:21:24Z

PR_Github #26028 [ run ] completed with state SUCCESS. Commit: 0235dde
/LLM/main/L0_MergeRequest_PR pipeline #19754 (Partly Tested) completed with status: 'FAILURE'

Signed-off-by: Chenfei Zhang <[email protected]>

chenfeiz0326 · 2025-11-27T14:07:11Z

/bot run --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Perf-Sanity-Post-Merge-1"

tensorrt-cicd · 2025-11-27T14:12:37Z

PR_Github #26042 [ run ] triggered by Bot. Commit: d0d6f59

jenkins/scripts/open_search_db.py

chzblych · 2025-11-27T15:02:52Z

jenkins/L0_Test.groovy

        // Perf sanity post merge test
-        // Disable perf stages due to https://nvbugs/5643646
-        // "DGX_B200-4_GPUs-PyTorch-Perf-Sanity-Post-Merge-1": ["b200-x4", "perf_sanity_l0_dgx_b200", 1, 1, 4],
+        "DGX_B200-4_GPUs-PyTorch-Perf-Sanity-Post-Merge-1": ["b200-x4", "perf_sanity_l0_dgx_b200", 1, 2, 4],


Could you check if the test splitting really happens? I vaguely remember that the test splitting doesn't work for the existing Perf test stages.

CC @EmmaQiaoCh

tests/integration/test_lists/test-db/perf_sanity_l0_dgx_b200.yml

tensorrt-cicd · 2025-11-27T15:24:49Z

PR_Github #26042 [ run ] completed with state SUCCESS. Commit: d0d6f59
/LLM/main/L0_MergeRequest_PR pipeline #19766 (Partly Tested) completed with status: 'SUCCESS'

Signed-off-by: Chenfei Zhang <[email protected]>

chenfeiz0326 · 2025-11-28T00:28:44Z

/bot run --stage-list "DGX_B200-4_GPUs-PyTorch-Perf-Sanity-Post-Merge-1"

tensorrt-cicd · 2025-11-28T00:34:18Z

PR_Github #26076 [ run ] triggered by Bot. Commit: 64907c5

tensorrt-cicd · 2025-11-28T02:11:13Z

PR_Github #26076 [ run ] completed with state SUCCESS. Commit: 64907c5
/LLM/main/L0_MergeRequest_PR pipeline #19800 (Partly Tested) completed with status: 'FAILURE'

Signed-off-by: Chenfei Zhang <[email protected]>

chenfeiz0326 · 2025-11-28T02:33:06Z

/bot run --stage-list "DGX_B200-4_GPUs-PyTorch-Perf-Sanity-Post-Merge-1,DGX_B200-4_GPUs-PyTorch-Perf-Sanity-Post-Merge-2"

tensorrt-cicd · 2025-11-28T02:38:47Z

PR_Github #26094 [ run ] triggered by Bot. Commit: e448781

chenfeiz0326 requested review from a team as code owners October 30, 2025 14:54

chenfeiz0326 requested review from poweiw and yuanjingx87 October 30, 2025 14:54

chenfeiz0326 changed the title ~~tmp~~ [TRTLLM-8825][feat] Support Pytest Perf Results uploading to Database Oct 30, 2025

chenfeiz0326 changed the title ~~[TRTLLM-8825][feat] Support Pytest Perf Results uploading to Database~~ [TRTLLM-9000][feat] tmp Oct 30, 2025

chenfeiz0326 removed request for poweiw and yuanjingx87 October 31, 2025 02:21

chenfeiz0326 force-pushed the chenfeiz/support-multi-node-perf-test branch from d31d4b1 to c746c07 Compare November 5, 2025 07:48

chenfeiz0326 changed the title ~~[TRTLLM-9000][feat] tmp~~ [TRTLLM-9000][feat] Add multi-node Perf Tests into CI Nov 5, 2025

chenfeiz0326 force-pushed the chenfeiz/support-multi-node-perf-test branch from 2d9ecc1 to 6cb155c Compare November 5, 2025 10:08

chenfeiz0326 requested review from chzblych and litaotju November 6, 2025 08:52

chenfeiz0326 force-pushed the chenfeiz/support-multi-node-perf-test branch from f140b9c to 5809fc7 Compare November 11, 2025 07:36

chenfeiz0326 force-pushed the chenfeiz/support-multi-node-perf-test branch from 3057f8c to da0d471 Compare November 27, 2025 08:53

chenfeiz0326 added 3 commits November 27, 2025 03:10

update

5380312

Signed-off-by: Chenfei Zhang <[email protected]>

update

86b2342

Signed-off-by: Chenfei Zhang <[email protected]>

update

0235dde

Signed-off-by: Chenfei Zhang <[email protected]>

chenfeiz0326 force-pushed the chenfeiz/support-multi-node-perf-test branch from da0d471 to 0235dde Compare November 27, 2025 11:11

update

d0d6f59

Signed-off-by: Chenfei Zhang <[email protected]>

chzblych requested changes Nov 27, 2025

View reviewed changes

update

64907c5

Signed-off-by: Chenfei Zhang <[email protected]>

chenfeiz0326 added 2 commits November 27, 2025 18:31

update

63baaf4

Signed-off-by: Chenfei Zhang <[email protected]>

update

e448781

Signed-off-by: Chenfei Zhang <[email protected]>

[TRTLLM-9000][feat] Add multi-node Perf Tests into CI #8800

Are you sure you want to change the base?

[TRTLLM-9000][feat] Add multi-node Perf Tests into CI #8800

Conversation

chenfeiz0326 commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Description

Test Coverage

PR Checklist

GitHub Bot Help

kill

skip

reuse-pipeline

Uh oh!

chenfeiz0326 commented Oct 30, 2025

Uh oh!

coderabbitai bot commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Pre-merge checks and finishing touches

Uh oh!

tensorrt-cicd commented Oct 30, 2025

Uh oh!

tensorrt-cicd commented Oct 30, 2025

Uh oh!

chenfeiz0326 commented Nov 5, 2025

Uh oh!

tensorrt-cicd commented Nov 5, 2025

Uh oh!

tensorrt-cicd commented Nov 5, 2025

Uh oh!

chenfeiz0326 commented Nov 5, 2025

Uh oh!

tensorrt-cicd commented Nov 5, 2025

Uh oh!

tensorrt-cicd commented Nov 5, 2025

Uh oh!

chenfeiz0326 commented Nov 6, 2025

Uh oh!

tensorrt-cicd commented Nov 6, 2025

Uh oh!

tensorrt-cicd commented Nov 6, 2025

Uh oh!

chenfeiz0326 commented Nov 11, 2025

Uh oh!

chenfeiz0326 commented Nov 11, 2025

Uh oh!

tensorrt-cicd commented Nov 11, 2025

Uh oh!

chenfeiz0326 commented Nov 27, 2025

Uh oh!

tensorrt-cicd commented Nov 27, 2025

Uh oh!

tensorrt-cicd commented Nov 27, 2025

Uh oh!

chenfeiz0326 commented Nov 27, 2025

Uh oh!

tensorrt-cicd commented Nov 27, 2025

Uh oh!

tensorrt-cicd commented Nov 27, 2025

Uh oh!

chenfeiz0326 commented Nov 27, 2025

Uh oh!

tensorrt-cicd commented Nov 27, 2025

Uh oh!

tensorrt-cicd commented Nov 27, 2025

Uh oh!

chenfeiz0326 commented Nov 27, 2025

Uh oh!

tensorrt-cicd commented Nov 27, 2025

Uh oh!

tensorrt-cicd commented Nov 27, 2025

Uh oh!

chenfeiz0326 commented Nov 27, 2025

Uh oh!

tensorrt-cicd commented Nov 27, 2025

Uh oh!

chenfeiz0326 commented Oct 30, 2025 •

edited

Loading

coderabbitai bot commented Oct 30, 2025 •

edited

Loading