[CI] rapids-xgboost 26.2 has conflicting CUDA version constraints, breaking ALL tests

[csadorf]

Summary

CRITICAL: All CI test jobs on the main branch (26.02) are failing during conda environment setup due to contradictory CUDA version constraints in the rapids-xgboost==26.2.* dependency chain. The package has conflicting requirements that make it unsolvable for any CUDA version:

• CUDA 12.x: Direct constraint requires CUDA ≥13
• CUDA 13.x: Transitive dependencies require CUDA <13

This is a critical packaging/build error that blocks all testing on the main branch.

Failing test/component: conda environment creation (ALL Python tests on ALL CUDA versions)

Failure observed in:

• CUDA 12.2.2: https://github.com/rapidsai/cuml/actions/runs/19538382314/job/55938025612
• CUDA 13.0.2: https://github.com/rapidsai/cuml/actions/runs/19538382314/job/55938023313
• All other CUDA versions affected

Environment

• OS: All (rockylinux8, ubuntu, etc.)
• Python version: All (3.10, 3.11, 3.12, 3.13)
• CUDA version: ALL (both 12.x and 13.x confirmed broken)
• GPU: All
• Dependencies: All dependency configurations (oldest-deps, latest-deps)
• Branch: main (26.02)

Test Details

• Test file: N/A (failure occurs during environment setup, before tests run)
• Test name: All tests on ALL CUDA versions

Scenario 1: CUDA 12.x Failure

Error message:

error    libmamba Could not solve for environment specs
    The following packages are incompatible
    ├─ cuda-version =12.2 * is requested and can be installed;
    └─ rapids-xgboost =26.2,>=0.0.0a0 * is not installable because it requires
       └─ cuda-version >=13,<14.0a0 *, which conflicts with any installable versions previously reported.

Full error context (job 55938025612, log lines 1204-1211):

2025-11-20T13:27:32.8283476Z error    libmamba Could not solve for environment specs
2025-11-20T13:27:32.8285222Z     The following packages are incompatible
2025-11-20T13:27:32.8287242Z     ├─ cuda-version =12.2 * is requested and can be installed;
2025-11-20T13:27:32.8288803Z     └─ rapids-xgboost =26.2,>=0.0.0a0 * is not installable because it requires
2025-11-20T13:27:32.8290535Z        └─ cuda-version >=13,<14.0a0 *, which conflicts with any installable versions previously reported.
2025-11-20T13:27:32.8825107Z critical libmamba Could not solve for environment specs
2025-11-20T13:27:32.9079141Z [rapids-conda-retry] conda returned exit code: 1

Scenario 2: CUDA 13.x Failure

Error message:

error    libmamba Could not solve for environment specs
    The following packages are incompatible
    ├─ cuda-version =13.0 * is requested and can be installed;
    └─ rapids-xgboost =26.2,>=0.0.0a0 * is not installable because it requires
       └─ libxgboost =3.0.5 rapidsai_h* but there are no viable options
          └─ libxgboost 3.0.5 would require
             └─ librmm >=25.12.0a8,<25.13.0a0 * but there are no viable options
                └─ librmm [versions] would require
                   └─ cuda-version >=12,<13.0a0 *, which conflicts with any installable versions previously reported.

Full error context (job 55938023313, log lines 1220-1268):

2025-11-20T13:27:19.6040868Z error    libmamba Could not solve for environment specs
2025-11-20T13:27:19.6050065Z     The following packages are incompatible
2025-11-20T13:27:19.6055643Z     ├─ cuda-version =13.0 * is requested and can be installed;
[... dependency chain showing libxgboost → librmm → cuda-version <13 conflict ...]
2025-11-20T13:27:19.6118187Z              └─ cuda-version >=12,<13.0a0 *, which conflicts with any installable versions previously reported.
2025-11-20T13:27:19.7099577Z critical libmamba Could not solve for environment specs

Root Cause Analysis

The rapids-xgboost package version 26.2 has contradictory CUDA version constraints in its dependency chain that make it unsolvable for any CUDA version:

The Contradiction

1. Top-level constraint (rapids-xgboost → cuda-version):

   • rapids-xgboost =26.2 directly requires cuda-version >=13,<14.0a0
   • This blocks all CUDA 12.x installations

2. Transitive constraint (rapids-xgboost → libxgboost → librmm → cuda-version):

   • rapids-xgboost =26.2 requires libxgboost =3.0.5
   • libxgboost =3.0.5 requires librmm >=25.12.0a8,<25.13.0a0
   • Compatible librmm versions require cuda-version >=12,<13.0a0
   • This blocks all CUDA 13.x installations

Dependency Chain Visualization

rapids-xgboost 26.2
├─ DIRECT: cuda-version >=13,<14.0a0  ❌ Blocks CUDA 12.x
└─ libxgboost 3.0.5
   └─ librmm 25.12.x
      └─ cuda-version >=12,<13.0a0    ❌ Blocks CUDA 13.x

Result: No CUDA version (12.x or 13.x) can satisfy both constraints simultaneously.

[csadorf]

Expected to be fixed by rapidsai/integration#816 .

[csadorf]

Expected to be fixed by rapidsai/integration#817

[csadorf]

Introducing temporary mitigation in #7523 .

[jcrist]

I see that rapidsai/integration#817 was merged, but I'm still not able to create a local env on cuda 12. Here's what mamba reports

$ mamba env create -n test -f conda/environments/all_cuda-129_arch-x86_64.yaml -y
...
error    libmamba Could not solve for environment specs
    The following packages are incompatible
    ├─ cuda-version =12.9 * is requested and can be installed;
    ├─ librmm =26.2,>=0.0.0a0 * is installable with the potential options
    │  ├─ librmm [25.10.00|25.10.00a39|...|26.02.00a8] would require
    │  │  └─ cuda-version >=13,<14.0a0 *, which conflicts with any installable versions previously reported;
    │  └─ librmm [26.02.00a10|26.02.00a12|...|26.02.00a8], which can be installed;
    └─ rapids-xgboost =26.2,>=0.0.0a0 * is not installable because there are no viable options
       ├─ rapids-xgboost [26.02.00a3|26.02.00a5] would require
       │  └─ cuda-version >=13,<14.0a0 *, which conflicts with any installable versions previously reported;
       └─ rapids-xgboost 26.02.00a5 would require
          └─ libxgboost =3.0.5 rapidsai_h* but there are no viable options
             ├─ libxgboost 3.0.5 would require
             │  └─ cuda-version >=13,<14.0a0 *, which conflicts with any installable versions previously reported;
             ├─ libxgboost 3.0.5 would require
             │  └─ librmm [>=25.10.0a33,<25.11.0a0 *|>=25.10.0a36,<25.11.0a0 *|>=25.10.0a38,<25.11.0a0 *] but there are no viable options
             │     ├─ librmm [25.10.00|25.10.00a39|25.10.00a40] conflicts with any installable versions previously reported;
             │     └─ librmm [25.10.00|25.10.00a39|...|26.02.00a8], which cannot be installed (as previously explained);
             └─ libxgboost 3.0.5 would require
                └─ librmm >=25.12.0a8,<25.13.0a0 * but there are no viable options
                   ├─ librmm [25.10.00|25.10.00a39|...|26.02.00a8], which cannot be installed (as previously explained);
                   └─ librmm [25.12.00a59|25.12.00a60|25.12.00a62] conflicts with any installable versions previously reported.

cc @gforsyth for some insights - should we have expected that PR to have resolved this issue? I see there were build failures in upstreaming to main (rapidsai/integration#818), perhaps packages weren't pushed? I confess I don't fully understand our build infra.

[gforsyth]

I think what's still missing is a build of xgboost that accounts for the changes in rmm for 25.12 -- that's being worked on here: rapidsai/xgboost-feedstock#110