Require CUDA 12.2+ #860

jakirkham · 2025-10-28T21:41:28Z

Make CUDA 12.2 the new minimum. Drops logic related to CUDA 12.0 & 12.1. Thus simplifying the CUDA build and dependency logic.

Part of issue:

Require CUDA >=12.2 build-planning#223

java/pom.xml

bdice · 2025-10-28T21:46:53Z

conda/recipes/libkvikio/recipe.yaml

-  # Each case has different cuda-version constraints as expressed below
-  should_use_cufile: ${{ x86_64 or (aarch64 and cuda_version >= "12.2") }}
  # When reverting, instances of cuda_key_string can be replaced with cuda_major
  cuda_key_string: ${{ cuda_version | replace(".", "_") }}


I think we want this to go back to cuda_major as the comment suggests. Delete cuda_key_string, uncomment cuda_version / cuda_major, and replace cuda_key_string with cuda_major.

Am ambivalent here. On the one hand can see that point. However on the other it could be useful to call out CUDA 12.2 is a minimum here

I don't understand how this helps. We already set a lower bound on cuda-version. Let's stop using special-case behavior here and revert as the comment recommends.

The same argument could be made for the explicit inclusion of the CUDA version in our build string currently

It is also already handled via cuda-version and the build string hash

Think it is worth discussing this point

The same argument could be made for the explicit inclusion of the CUDA version in our build string currently

How? conda doesn't allow you to upload two builds that have identical versions and build strings but different dependencies. You have to include some bit of information that makes the build unique. I guess we could use PKG_HASH to distinguish the builds in the build string, but if you have to include something it might as well be more meaningful than that.

That being said, I don't look at the build string as a useful way of conveying the minimum supported CUDA version given that the cuda-version dependency is part of the package metadata as well, so my vote would be to go back to the previous status. 12 vs 13 tells me enough about the purpose of the two packages. I'm not looking for anything more than that.

So that argument goes like this

If cuda-version is in the variants (like conda_build_config.yaml or variants.yaml), then it is included in the string hash. So there is no need to manually add it to the string

The challenge with including it manually (as we have to date) is it means writing the build string from scratch each time. As a result we end up missing details that would have otherwise been captured for us if it was added as a variant and included in the string hash

Right so there are essentially two viewpoints we encounter. One view is the as you have articulated that the build string is not a reliable or consistent way to find this information so it shouldn't be relied on and package dependencies (like cuda-version) should be used to do that. The other view is that it can be useful to have the relevant info in one place on the relevant package to simplify the end-user workflow

Given today we already recommend users specify cuda-version, there is no real simplification offered by adding this info to the build string. So would tend to agree with the first view for our case (similar to you). And would go further, that it would be better to minimize string customization and rely on variants and the string hash

The reason we have never included the CUDA version in the variants directly is that we need the flexibility of building different variants in different CI jobs on different runners, whereas conda-build and rattler-build will build all the variants at once (the slowdown in end-to-end CI time that would result from serializing the builds has too big a negative impact on developers). With the switch to rattler-build we have a bit of additional flexibility because rattler-build supports providing the variant config on the CLI, but ultimately that means we'd be constructing the variants on the fly in CI jobs which leads to more divergence between local dev and CI where we have been aiming to close the gap over time. If you have a solution that allows us to specify all the information in the variants file and avoid environment and recipe context variables, I'm all for it. Otherwise I think we're stuck with some degree of manual string customization.

Both conda-build and rattler-build can specify variants by CLI

Think we are conflating two things:

How the matrix is specified

How CUDA version is specified

We need not change 1 to change 2. Am focused here on 2

If you have an idea of how to implement a proposed improved, feel free to push a commit with the change and I'm happy to review.

I do. Though will approach this in a follow up PR so we can discuss. We can then decide if it merits broader rollout

For now have simply reverted the relevant bits here

conda/recipes/libkvikio/recipe.yaml

jakirkham · 2025-10-28T22:07:04Z

cpp/src/shim/cufile.cpp

+  // Note: `version` is v1.7 (in CUDA 12.2) for cuFile versions prior to v1.8 (CUDA 12.3)
+  // because `cuFileGetVersion` did not exist until CUDA 12.3 and CUDA 12.2 is the oldest
+  // version we support. This trade-off is made for improved robustness.
  if (version >= 1060) {


CUDA 12.2.0 shipped with cuFile 1.7.0.49

In other words it had the batch and async features all supported. Given this all of the symbols checked for below are defined and the checks could be dropped. For now have just bumped the version value to represent 1.7.0

Refs:

https://docs.nvidia.com/cuda/archive/12.2.0/cuda-toolkit-release-notes/index.html#cuda-toolkit-major-component-versions

https://docs.nvidia.com/gpudirect-storage/release-notes/index.html#new-features-and-changes

Thanks for the clean-up! This will help to simplify the logic in C++ compatibility mode manager as well.

Thanks for the encouragement Tianyu! 🙏

Went ahead and dropped all of this logic

As of CUDA 12.2.0 all of these functions are available. So drop the version checks around them. Though still get the version (if possible) as it is defined here and used throughout the codebase. If it is not possible to get the version, then this is CUDA 12.2 (as CUDA 12.3 defines the function). So set the version to cuFile 1.7.0, which was included in CUDA 12.2.

As of CUDA 12.2+, these functions are always true as the relevant APIs are always included. So go ahead drop them. Also equate their calls to `true` in relevant logic.

jakirkham · 2025-11-06T18:48:53Z

Have been seeing some unrelated flaky CI behavior in a few jobs. Here is one example

Download action repository 'aws-actions/configure-aws-credentials@00943011d9042930efac3dcd3a170e4273319bc8' (SHA:00943011d9042930efac3dcd3a170e4273319bc8)
Warning: Failed to download action 'https://api.github.com/repos/aws-actions/configure-aws-credentials/tarball/00943011d9042930efac3dcd3a170e4273319bc8'. Error: Resource temporarily unavailable (codeload.github.com:443) 
Warning: Back off 25.333 seconds before retry.
Warning: Failed to download action 'https://api.github.com/repos/aws-actions/configure-aws-credentials/tarball/00943011d9042930efac3dcd3a170e4273319bc8'. Error: Resource temporarily unavailable (codeload.github.com:443) 
Warning: Back off 18.664 seconds before retry.
Error: Resource temporarily unavailable (codeload.github.com:443)

Have restarted CI a few times to get this to clear up. Will also follow up offline

Edit: Looks like this was do to a networking issue that has since been resolved

jakirkham · 2025-11-06T18:49:26Z

That being said, think this is otherwise ready for another review. So would appreciate if folks could take another look

kingcrimsontianyu · 2025-11-06T20:18:27Z

cpp/CMakeLists.txt


  # Check API support
  try_compile(
    cuFile_BATCH_API_FOUND SOURCE_FROM_CONTENT


Should this be removed too?

Was thinking about these. I think we may still want to check, but instead error if it doesn't work

What do you think?

Since the batch/stream APIs are guaranteed to exist in cuFile under CTK 12.2+, I think we can remove the check here for simplicity.

kingcrimsontianyu · 2025-11-06T20:18:42Z

cpp/CMakeLists.txt

  )
  message(STATUS "Found cuFile Batch API: ${cuFile_BATCH_API_FOUND}")
  try_compile(
    cuFile_STREAM_API_FOUND SOURCE_FROM_CONTENT


... and this one as well

Let's discuss in the above thread: #860 (comment)

We can apply any changes needed to both after

kingcrimsontianyu · 2025-11-06T20:36:07Z

cpp/src/compat_mode_manager.cpp

 void CompatModeManager::validate_compat_mode_for_async() const
 {
  KVIKIO_NVTX_FUNC_RANGE();
  if (_is_compat_mode_preferred_for_async && _compat_mode_requested == CompatMode::OFF) {
    std::string err_msg;
-    if (!is_stream_api_available()) { err_msg += "Missing the cuFile stream api."; }
-
-    // When checking for availability, we also check if cuFile's config file exists. This is
+    // When checking for availability, we check if cuFile's config file exists. This is
    // because even when the stream API is available, it doesn't work if no config file exists.
-    if (config_path().empty()) { err_msg += " Missing cuFile configuration file."; }
+    if (config_path().empty()) { err_msg += "Missing cuFile configuration file."; }

    KVIKIO_FAIL(err_msg, std::runtime_error);
  }


Suggested change

void CompatModeManager::validate_compat_mode_for_async() const

{

KVIKIO_NVTX_FUNC_RANGE();

if (_is_compat_mode_preferred_for_async && _compat_mode_requested == CompatMode::OFF) {

std::string err_msg;

if (!is_stream_api_available()) { err_msg += "Missing the cuFile stream api."; }

// When checking for availability, we also check if cuFile's config file exists. This is

// When checking for availability, we check if cuFile's config file exists. This is

// because even when the stream API is available, it doesn't work if no config file exists.

if (config_path().empty()) { err_msg += " Missing cuFile configuration file."; }

if (config_path().empty()) { err_msg += "Missing cuFile configuration file."; }

KVIKIO_FAIL(err_msg, std::runtime_error);

}

void CompatModeManager::validate_compat_mode_for_async() const

{

KVIKIO_NVTX_FUNC_RANGE();

// cuFile stream API will not work if the config file does not exist.

if (_is_compat_mode_preferred_for_async && _compat_mode_requested == CompatMode::OFF &&

config_path().empty()) {

KVIKIO_FAIL("Missing cuFile configuration file.", std::runtime_error);

}

}

On a side note, I didn't know the presence of config file is required for the stream API to function properly. This is quite interesting. I'll run some tests and see if this behavior has changed in recent cuFile API.

kingcrimsontianyu · 2025-11-06T20:51:00Z

cpp/src/shim/cufile.cpp

 #ifdef KVIKIO_CUFILE_VERSION_API_FOUND
  try {
    get_symbol(GetVersion, lib, KVIKIO_STRINGIFY(cuFileGetVersion));
    int ver;
    CUfileError_t const error = GetVersion(&ver);
    if (error.err == CU_FILE_SUCCESS) { version = ver; }
  } catch (std::runtime_error const&) {
+    version = 1070;
  }
+#else
+  version = 1070;
 #endif


To be consistent with previous behavior, we don't need to set the version number here in the else branch. It will take the default value of 0 (specified in cuFileAPI's constructor). I think we could add back some of the comments related to the version checking API. So I'd suggest this change:

Suggested change

#endif

// Note: `version` is 0 for cuFile versions prior to v1.8 because `cuFileGetVersion`

// did not exist.

#ifdef KVIKIO_CUFILE_VERSION_API_FOUND

try {

get_symbol(GetVersion, lib, KVIKIO_STRINGIFY(cuFileGetVersion));

int ver;

CUfileError_t const error = GetVersion(&ver);

if (error.err == CU_FILE_SUCCESS) { version = ver; }

} catch (std::runtime_error const&) {

}

#endif

bdice

Approving packaging/CI changes. I'll let @kingcrimsontianyu finalize his feedback and approve the C++ side, but it looks good to me too

kingcrimsontianyu

Suggested a couple of changes. Thanks!

jakirkham requested a review from a team as a code owner October 28, 2025 21:41

jakirkham added the breaking Introduces a breaking change label Oct 28, 2025

jakirkham requested review from a team as code owners October 28, 2025 21:41

jakirkham added the improvement Improves an existing functionality label Oct 28, 2025

jakirkham requested a review from gforsyth October 28, 2025 21:41

jakirkham commented Oct 28, 2025

View reviewed changes

java/pom.xml Outdated Show resolved Hide resolved

bdice reviewed Oct 28, 2025

View reviewed changes

jakirkham force-pushed the req_cuda_12.2+ branch from c23914f to cff534c Compare October 28, 2025 21:50

jakirkham requested a review from a team as a code owner October 28, 2025 21:50

jakirkham force-pushed the req_cuda_12.2+ branch 2 times, most recently from 9eebbfa to 82937d0 Compare October 28, 2025 22:02

jakirkham commented Oct 28, 2025

View reviewed changes

jakirkham force-pushed the req_cuda_12.2+ branch from 82937d0 to 05f1d88 Compare October 28, 2025 22:28

jakirkham mentioned this pull request Oct 28, 2025

Require CUDA >=12.2 rapidsai/build-planning#223

Open

17 tasks

madsbk mentioned this pull request Oct 29, 2025

Should we remove the Java bindings? #861

Open

jakirkham force-pushed the req_cuda_12.2+ branch from 6ab3851 to d1bfd7e Compare November 5, 2025 22:31

jakirkham added 5 commits November 5, 2025 14:38

Require CUDA 12.2+

b654093

Drop unused minor_version from recipe context

e4dabce

Revert jcuda version change

4278d8e

Revert from cuda_key_string back to cuda_major

facd783

Drop extra /

3c2f34a

jakirkham force-pushed the req_cuda_12.2+ branch from d1bfd7e to c1f0d37 Compare November 5, 2025 22:41

jakirkham added 2 commits November 5, 2025 14:44

Drop is_batch_api_available & is_stream_api_available

5405ae4

As of CUDA 12.2+, these functions are always true as the relevant APIs are always included. So go ahead drop them. Also equate their calls to `true` in relevant logic.

jakirkham force-pushed the req_cuda_12.2+ branch from c1f0d37 to 5405ae4 Compare November 5, 2025 22:44

jakirkham added 4 commits November 5, 2025 14:55

Fix comment for style check

6c6030e

Fix code wrapping for style check

ea6f1be

Treat missing cuFileGetVersion as CUDA 12.2

c5e46ae

Align assignments to fix lint

6e5c958

jakirkham requested review from bdice, kingcrimsontianyu, madsbk and vyasr November 6, 2025 18:49

kingcrimsontianyu reviewed Nov 6, 2025

View reviewed changes

bdice approved these changes Nov 7, 2025

View reviewed changes

kingcrimsontianyu requested changes Nov 9, 2025

View reviewed changes

Require CUDA 12.2+ #860

Are you sure you want to change the base?

Require CUDA 12.2+ #860

Uh oh!

Conversation

jakirkham commented Oct 28, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vyasr Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jakirkham Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jakirkham commented Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jakirkham commented Nov 6, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bdice left a comment

Choose a reason for hiding this comment

Uh oh!

kingcrimsontianyu left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

vyasr Oct 31, 2025 •

edited

Loading

jakirkham Oct 28, 2025 •

edited

Loading

jakirkham commented Nov 6, 2025 •

edited

Loading