Add new conda environments

[pentschev]

UCXX is not necessarily CUDA dependent. This change adds two new environments: cpp.yaml for C++-only builds and all.yaml for C++ and Python builds.

[bdice]

This should match the name from -n ucxx above.

Suggested change

	$ conda activate all_cuda-118_arch-x86_64
	$ conda activate ucxx

[bdice]

All environment files need to be generated by dependencies.yaml.

The failure in pre-commit is because rapids-dependency-file-generator looks for files that it generated and removes them before re-generating the new files. You have the header here saying it's a generated file, but there's no declaration of this environment file in dependencies.yaml.

This is how you tell it to generate a given environment file:

ucxx/dependencies.yaml

Lines 3 to 17 in 4464ca9

	all:
	output: conda
	matrix:
	cuda: ["11.8", "12.0"]
	arch: [x86_64]
	includes:
	- build_cpp
	- build_python
	- checks
	- cudatoolkit
	- dev
	- py_version
	- run_python
	- test_cpp
	- test_python

For example, you might want to add a section like this for the C++ dependency list.

files:
  cpp:     # This is used as part of the environment's filename
    output: conda
    matrix:
      arch: [x86_64]
    includes:
      - build_cpp
      - checks
      - dev
      - test_cpp

[pentschev]

Thanks Bradley, this is helpful. I'd still like to trim that more though, for example the above will result in:

# This file is generated by `rapids-dependency-file-generator`.
# To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`.
channels:
- rapidsai
- rapidsai-nightly
- dask/label/dev
- conda-forge
- nvidia
dependencies:
- autoconf
- automake
- c-compiler
- cmake>=3.26.4
- cxx-compiler
- dask-cuda==23.12.*
- dask-cudf==23.12.*
- fmt>=9.1.0,<10
- gmock>=1.13.0
- gtest>=1.13.0
- librmm==23.12.*
- libtool
- ninja
- pip
- pkg-config
- pre-commit
- spdlog>=1.11.0,<1.12
name: cpp_arch-x86_64

That include a bunch of things we don't need: rapidsai/rapidsai-nightly/dask/label/dev/nvidia channels and dask-cuda/dask-cudf/librmm/pip/pkg-config. Is there a way to minimize that?

Additionally, this is supposed to be valid for aarch64 too, can we make it architecture agnostic?

[bdice]

There's currently no way to drop extra channels from the generated files. You can file an issue to rapids-dependency-file-generator if you have a proposal for how the dependencies.yaml schema should accommodate that idea, but I'm not sure if it's worth changing. https://github.com/rapidsai/dependency-file-generator

If you depend only on architecture-agnostic packages, you can just drop the arch from the files: spec.

files:
  cpp:     # This is used as part of the environment's filename
    output: conda
    includes:
      - build_cpp
      - checks
      - dev
      - test_cpp

[jameslamb]

Hey @pentschev it's been a while since this was last updated ... could it be closed?

(context: I am looking through https://github.com/pulls/review-requested now that I'm part of ops-codeowners and found this)

[pentschev]

Hi @jameslamb , thanks for the ping here. This is something that we'll need to get done at some point, so I would prefer to keep it open until we have time/solution for that. Do you by chance happen to know whether there's been any progress to the requirements to tackle this issue, based on the previous conversation with Bradley in this PR?

[bdice]

If this is important, I would recommend that we update this PR to the latest branch, use rapids-dependency-file-generator, and merge it. We don't have a solution for dropping channels from the rapids-dependency-file-generator output but we have already removed our dependencies on several of those channels as part of dropping CUDA 11 support.

Let me know if you'd like me to take a stab at that, I have a pretty clear idea of what I would like to see as a "minimal but functional" solution.

[jameslamb]

Ok, thanks for confirming it's intentionally still open.

Do you by chance happen to know whether there's been any progress to the requirements to tackle this issue, based on the previous conversation with Bradley in this PR?

I agree with @bdice , hopefully the lack of support for excluding conda channels (or including them selectively) in rapids-dependency-file-generator is less of a concern here in 2025 given that:

• the nvidia and dask/label/dev channels are not included by default
• ucxx has a build-time dependency on rmm, so probably wants the rapidsai / rapidsai-nightly channels anyway if using conda to set up a build environment

If you really really really do not want to have rapidsai / rapidsai-nightly in these conda environments either, it could be possible through some combination of removing those from channels: in dependencies.yaml and then adding them via CLI arguments --prepend-channel wherever they're needed.

A separate pre-commit hook configuration could then manage these other files that don't need these channels, similar to this:

      - repo: https://github.com/rapidsai/dependency-file-generator
        rev: v1.19.0
        hooks:
            - id: rapids-dependency-file-generator
              args: ["--clean", "--prepend-channel=rapidsai", "--prepend-channel=rapidsai-nightly"]
            - id: rapids-dependency-file-generator-no-rapids-channels
              args: ["--output=conda", "--file-key=cpp-no-rapids"]

[pentschev]

• ucxx has a build-time dependency on rmm, so probably wants the rapidsai / rapidsai-nightly channels anyway if using conda to set up a build environment

RMM is currently a build dependency because we (I) never made the effort to support CPU-only, but ideally we should do that. I see two ways in which we could do it:

1. Have runtime checks for existing CUDA, or more recently dlopen RMM since now RMM is switching to being a shared library; or
2. Build separate CPU and CUDA packages.

The above is probably something we would need to do to support the change being made here, but is doable.

If you really really really do not want to have rapidsai / rapidsai-nightly in these conda environments either, it could be possible through some combination of removing those from channels: in dependencies.yaml and then adding them via CLI arguments --prepend-channel wherever they're needed.

A separate pre-commit hook configuration could then manage these other files that don't need these channels, similar to this:

      - repo: https://github.com/rapidsai/dependency-file-generator
        rev: v1.19.0
        hooks:
            - id: rapids-dependency-file-generator
              args: ["--clean", "--prepend-channel=rapidsai", "--prepend-channel=rapidsai-nightly"]
            - id: rapids-dependency-file-generator-no-rapids-channels
              args: ["--output=conda", "--file-key=cpp-no-rapids"]

This looks like a potential good solution, thanks James. I'll see if I can find some time to try it out some time in the near(ish) future.