Skip to content

Improve linux parallelism in CI. #6730

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

Improve linux parallelism in CI. #6730

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
31627e9
Improve linux parallelism in CI.
alliepiper Nov 21, 2025
31ae96c
Split up thrust transform test.
alliepiper Nov 21, 2025
e33bcb5
WIP Need to port %PARAM% to Thrust.
alliepiper Nov 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/actions/workflow-run-job-linux/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -191,6 +191,7 @@ runs:
--env "GITHUB_REPOSITORY=$GITHUB_REPOSITORY" \
--env "HOST_WORKSPACE=${{github.workspace}}" \
--env "JOB_ID=$JOB_ID" \
--env "NVCC_APPEND_FLAGS=-t=100" \
Copy link
Contributor

@oleksandr-pavlyk oleksandr-pavlyk Nov 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The nvcc will not utilize more than the number of architectures it is compiling for. Perhaps in addition to -t 16 we could also use --split-compile 16 and --split-compile-extended 16.

--threads <number>                                  (-t)
        Specify the maximum number of threads to be created in parallel when compiling
        for multiple architectures. If <number> is 1 or if compiling for one architecture,
        this option is ignored. If <number> is 0, the number of threads will be the
        number of CPUs on the machine.

--split-compile <number>                            (-split-compile)
        Specify the maximum amount of concurrent threads to to be utilized when running
        compiler optimizations. If <number> is 1, this option is ignored. If <number>
        is 0, the number of threads will be the number of CPUs on the machine.
        This option will have minimal (if any) impact on performance of the compiled
        binary.

--split-compile-extended <number>                   (-split-compile-extended)
        Specify the maximum amount of concurrent threads to be utilized when running
        compiler optimizations in LTO mode. If <number> is 1, this option is ignored.
        If <number> is 0, the number of threads will be the number of CPUs on the
        machine.
        This option is a more aggressive form of split compilation, and can potentially
        impact performance of the compiled binary. It is available in LTO mode only.

Copy link
Contributor Author

@alliepiper alliepiper Nov 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question.

This would only work if those commands spawn jobs that sccache knows about.

Judging from the job server docs: https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html#jobserver-jobserver

When using -split-compile or --threads inside of a build controlled by GNU Make, require that job slots are acquired Make’s jobserver for each of the threads used, helping prevent oversubscription. This option does not restrict -split-compile-extended (the number of threads created by it will not be controlled).

It sounds like --split-compile is spawning new processes, and --split-compile-extended is threading within processes. So almost certainly not safe to use --split-compile-extended.

@trxcllnt Have you looked into these for the distributed build PR?

Copy link
Member

@trxcllnt trxcllnt Nov 21, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sccache disables --split-compile and forces cicc/ptxas to use one core. It is fine to pass -t=100 because sccache will intercept that flag and use it to determine the number of parallel device compiles to kick off (all scheduled on the jobserver of course).

Copy link
Member

@trxcllnt trxcllnt Nov 21, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you looked into these for the distributed build PR?

Yes, I have looked into supporting --split-compile, but decided against it for now.

I could make one job reserve multiple permits from the jobserver when building locally. That'd block progress on other tasks (like preprocessing other files, etc.), so we'd need to measure whether it improves or degrades throughput.

It's more difficult for distributed compilation, since build servers would need to know how many threads a job will use before it fetches it. AFAIK that's not baked into AMQP or Celery, and I can't think of an easy way to accomplish that.

If a build server pulled a Big Job that needed 4 cores when only 1 was available (the default behavior), it would need to wait for 3 others to finish before starting Big Job. It could've possibly finished other jobs in the meantime, so again, we'd have to measure whether it actually improved the throughput.

Copy link
Contributor

@davebayer davebayer Nov 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Q: does it make sense to run a single compilation in parallel, when the build already runs in parallel? Other cores/threads will be already occupied by other builds

Copy link
Member

@trxcllnt trxcllnt Nov 22, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, running device compiles in parallel improves build times quite a bit.

For example, you compile two TUs for five archs, one that takes 1min per arch and the other 20min per arch. With the default -t=1, the device compiles are serialized, so the first object builds in 5min, and the second builds in 100min. If you increase -t=5, the first object builds in 1min and the second object builds in 20min.

The sccache client ensures no more than $(nproc) number of compilations are run concurrently, but even with thousands of TUs, compiling the device code for each arch in parallel is still significantly faster than serializing them.

Copy link
Contributor Author

@alliepiper alliepiper Nov 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is sccache magic -- sccache internally throttles jobs to nproc, so we can take advantage of both nvcc -t and ninja -j simultaneously without manual load balancing.

Copy link
Contributor

@davebayer davebayer Dec 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, thanks for the explanation!

--env "NVIDIA_VISIBLE_DEVICES=${NVIDIA_VISIBLE_DEVICES:-}" \
--env "RUNNER_TEMP=$RUNNER_TEMP" \
--volume "${ARTIFACT_ARCHIVES}:${ARTIFACT_ARCHIVES}" \
Expand Down
Loading
Loading