Improve linux parallelism in CI.

[alliepiper]

sccache internally throttles jobs and we've been doing this on windows. This will improve build util.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[oleksandr-pavlyk]

The nvcc will not utilize more than the number of architectures it is compiling for. Perhaps in addition to -t 16 we could also use --split-compile 16 and --split-compile-extended 16.

--threads <number>                                  (-t)
        Specify the maximum number of threads to be created in parallel when compiling
        for multiple architectures. If <number> is 1 or if compiling for one architecture,
        this option is ignored. If <number> is 0, the number of threads will be the
        number of CPUs on the machine.

--split-compile <number>                            (-split-compile)
        Specify the maximum amount of concurrent threads to to be utilized when running
        compiler optimizations. If <number> is 1, this option is ignored. If <number>
        is 0, the number of threads will be the number of CPUs on the machine.
        This option will have minimal (if any) impact on performance of the compiled
        binary.

--split-compile-extended <number>                   (-split-compile-extended)
        Specify the maximum amount of concurrent threads to be utilized when running
        compiler optimizations in LTO mode. If <number> is 1, this option is ignored.
        If <number> is 0, the number of threads will be the number of CPUs on the
        machine.
        This option is a more aggressive form of split compilation, and can potentially
        impact performance of the compiled binary. It is available in LTO mode only.

[alliepiper]

Good question.

This would only work if those commands spawn jobs that sccache knows about.

Judging from the job server docs: https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html#jobserver-jobserver

When using -split-compile or --threads inside of a build controlled by GNU Make, require that job slots are acquired Make’s jobserver for each of the threads used, helping prevent oversubscription. This option does not restrict -split-compile-extended (the number of threads created by it will not be controlled).

It sounds like --split-compile is spawning new processes, and --split-compile-extended is threading within processes. So almost certainly not safe to use --split-compile-extended.

@trxcllnt Have you looked into these for the distributed build PR?

[trxcllnt]

sccache disables --split-compile and forces cicc/ptxas to use one core. It is fine to pass -t=100 because sccache will intercept that flag and use it to determine the number of parallel device compiles to kick off (all scheduled on the jobserver of course).

[trxcllnt]

Have you looked into these for the distributed build PR?

Yes, I have looked into supporting --split-compile, but decided against it for now.

I could make one job reserve multiple permits from the jobserver when building locally. That'd block progress on other tasks (like preprocessing other files, etc.), so we'd need to measure whether it improves or degrades throughput.

It's more difficult for distributed compilation, since build servers would need to know how many threads a job will use before it fetches it. AFAIK that's not baked into AMQP or Celery, and I can't think of an easy way to accomplish that.

If a build server pulled a Big Job that needed 4 cores when only 1 was available (the default behavior), it would need to wait for 3 others to finish before starting Big Job. It could've possibly finished other jobs in the meantime, so again, we'd have to measure whether it actually improved the throughput.

[alliepiper]

This option has the unfortunate side effect that all of our worst TUs are now more likely to launch in parallel 😄

Some Thrust tests are triggering OOMs, gonna put this on draft while I port some infra from CUB to help split these up.

[github-actions]

😬 CI Workflow Results

🟥 Finished in 3h 04m: Pass: 97%/267 | Total: 3d 06h | Max: 3h 01m | Hits: 98%/361824

See results here.

[davebayer]

Q: does it make sense to run a single compilation in parallel, when the build already runs in parallel? Other cores/threads will be already occupied by other builds

[trxcllnt]

Yes, running device compiles in parallel improves build times quite a bit.

For example, you compile two TUs for five archs, one that takes 1min per arch and the other 20min per arch. With the default -t=1, the device compiles are serialized, so the first object builds in 5min, and the second builds in 100min. If you increase -t=5, the first object builds in 1min and the second object builds in 20min.

The sccache client ensures no more than $(nproc) number of compilations are run concurrently, but even with thousands of TUs, compiling the device code for each arch in parallel is still significantly faster than serializing them.

[alliepiper]

This is sccache magic -- sccache internally throttles jobs to nproc, so we can take advantage of both nvcc -t and ninja -j simultaneously without manual load balancing.