Skip to content

Conversation

@Micket
Copy link
Contributor

@Micket Micket commented Nov 24, 2025

(created using eb --new-pr)

numba.cuda is dead, long live numba-cuda!

(this is required if you update your nvidia drivers to much, the "builtin" cuda backend will just segfault)

Needs --include-easyblocks-from-pr 3975

@Micket Micket added the update label Nov 24, 2025
@github-actions github-actions bot added the 2025a issues & PRs related to 2025a common toolchains label Nov 24, 2025
@github-actions
Copy link

github-actions bot commented Nov 24, 2025

Updated software filecheck-python-1.0.3-GCCcore-14.2.0.eb

Diff against filecheck-python-1.0.3-GCCcore-14.3.0.eb

easybuild/easyconfigs/f/filecheck-python/filecheck-python-1.0.3-GCCcore-14.3.0.eb

diff --git a/easybuild/easyconfigs/f/filecheck-python/filecheck-python-1.0.3-GCCcore-14.3.0.eb b/easybuild/easyconfigs/f/filecheck-python/filecheck-python-1.0.3-GCCcore-14.2.0.eb
index 942b5e50ac..2e2f9dd467 100644
--- a/easybuild/easyconfigs/f/filecheck-python/filecheck-python-1.0.3-GCCcore-14.3.0.eb
+++ b/easybuild/easyconfigs/f/filecheck-python/filecheck-python-1.0.3-GCCcore-14.2.0.eb
@@ -6,15 +6,15 @@ version = '1.0.3'
 homepage = 'https://github.com/AntonLydike/filecheck'
 description = """This tries to be as close a clone of LLVMs FileCheck as possible, without going crazy."""
 
-toolchain = {'name': 'GCCcore', 'version': '14.3.0'}
+toolchain = {'name': 'GCCcore', 'version': '14.2.0'}
 
 builddependencies = [
-    ('binutils', '2.44'),
-    ('poetry', '2.1.3'),
+    ('binutils', '2.42'),
+    ('poetry', '2.1.2'),
 ]
 
 dependencies = [
-    ('Python', '3.13.5'),
+    ('Python', '3.13.1'),
 ]
 
 exts_list = [

Updated software numba-cuda-0.20.0-foss-2025a-CUDA-12.8.0.eb

Diff against numba-cuda-0.20.0-foss-2025b-CUDA-12.9.1.eb

easybuild/easyconfigs/n/numba-cuda/numba-cuda-0.20.0-foss-2025b-CUDA-12.9.1.eb

diff --git a/easybuild/easyconfigs/n/numba-cuda/numba-cuda-0.20.0-foss-2025b-CUDA-12.9.1.eb b/easybuild/easyconfigs/n/numba-cuda/numba-cuda-0.20.0-foss-2025a-CUDA-12.8.0.eb
index 7ae890011c..42383a7778 100644
--- a/easybuild/easyconfigs/n/numba-cuda/numba-cuda-0.20.0-foss-2025b-CUDA-12.9.1.eb
+++ b/easybuild/easyconfigs/n/numba-cuda/numba-cuda-0.20.0-foss-2025a-CUDA-12.8.0.eb
@@ -10,28 +10,33 @@ Compiler. It is used for writing SIMT kernels in Python, for providing
 Python bindings for accelerated device libraries, and as a compiler for
 user-defined functions in accelerated libraries like RAPIDS."""
 
-toolchain = {'name': 'foss', 'version': '2025b'}
+toolchain = {'name': 'foss', 'version': '2025a'}
 toolchainopts = {'pic': True}
 
 builddependencies = [
-    ('poetry', '2.1.3'),  # required for tests
-    ('ml_dtypes', '0.5.3'),  # required for tests
+    ('poetry', '2.1.2'),  # required for tests
+    ('ml_dtypes', '0.5.1'),  # required for tests
     ('filecheck-python', '1.0.3'),  # required for tests
 ]
 
 dependencies = [
-    ('Python', '3.13.5'),
-    ('Python-bundle-PyPI', '2025.07'),  # for pytest
-    ('CUDA', '12.9.1', '', SYSTEM),
-    ('CUDA-Python', '12.9.2', versionsuffix),
+    ('Python', '3.13.1'),
+    ('Python-bundle-PyPI', '2025.04'),  # for pytest
+    ('CUDA', '12.8.0', '', SYSTEM),
+    ('CUDA-Python', '12.8.0', versionsuffix),
     ('numba', '0.62.0'),
 ]
 
 exts_list = [
-    ('numba_cuda', '0.20.0', {
+    ('numba_cuda', version, {
         'source_urls': ['https://github.com/NVIDIA/numba-cuda/archive/refs/tags/'],
         'sources': ['v%(version)s.tar.gz'],
-        'checksums': ['17e6803ed07589c37b5fa0757f9c0fc850146dbc24302fcea93497676f3a21fb'],
+        'patches': ['numba-cuda-%(version)s_old_license_syntax.patch'],
+        'checksums': [
+            {'v0.20.0.tar.gz': '17e6803ed07589c37b5fa0757f9c0fc850146dbc24302fcea93497676f3a21fb'},
+            {'numba-cuda-0.20.0_old_license_syntax.patch':
+             '3924e9ea64e7ac6281a883de896b4d1f8c281b72e82cb8521a233f26854b1854'},
+        ],
         'testinstall': True,
         'runtest': 'pytest --pyargs numba.cuda.tests -v',
     }),

@Micket Micket added this to the next release (5.2.0?) milestone Nov 24, 2025
@Micket
Copy link
Contributor Author

Micket commented Nov 24, 2025

Test report by @Micket
FAILED
Build succeeded for 4 out of 5 (2 easyconfigs in total)
vera-r07-06 - Linux Rocky Linux 9.6, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 580.95.05, Python 3.9.21
See https://gist.github.com/Micket/4d6a97343a93f490a76590843e72dc35 for a full test report.

@Thyre
Copy link
Collaborator

Thyre commented Nov 24, 2025

Test report by @Thyre
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3975
FAILED
Build succeeded for 2 out of 3 (total: 22 mins 25 secs) (2 easyconfigs in total)
jrc0900.jureca - Linux Rocky Linux 9.6, AArch64, ARM UNKNOWN (neoverse_v2), 1 x NVIDIA NVIDIA GH200 480GB, 580.95.05, Python 3.9.21
See https://gist.github.com/Thyre/be55bf94611ad8b1ae05fc7cbec9de87 for a full test report.

@Micket
Copy link
Contributor Author

Micket commented Nov 24, 2025

Test report by @Micket
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
vera-r07-06 - Linux Rocky Linux 9.6, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 580.95.05, Python 3.9.21
See https://gist.github.com/Micket/e6f6fe15164af226d024a6d88f0af11e for a full test report.

@Micket
Copy link
Contributor Author

Micket commented Nov 24, 2025

Test report by @Micket
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3975
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
vera-r07-06 - Linux Rocky Linux 9.6, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 580.95.05, Python 3.9.21
See https://gist.github.com/Micket/a92bd832335bcd831517bb602353ee33 for a full test report.

@Thyre
Copy link
Collaborator

Thyre commented Nov 24, 2025

Test report by @Thyre
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3975
FAILED
Build succeeded for 1 out of 2 (total: 42 secs) (2 easyconfigs in total)
jrc0900.jureca - Linux Rocky Linux 9.6, AArch64, ARM UNKNOWN (neoverse_v2), 1 x NVIDIA NVIDIA GH200 480GB, 580.95.05, Python 3.9.21
See https://gist.github.com/Thyre/2e29f3a4152a92c4e15d2fb58d50304a for a full test report.

@Thyre
Copy link
Collaborator

Thyre commented Nov 24, 2025

Test report by @Thyre
FAILED
Build succeeded for 1 out of 3 (total: 15 mins 23 secs) (3 easyconfigs in total)
jrc0900.jureca - Linux Rocky Linux 9.6, AArch64, ARM UNKNOWN (neoverse_v2), 1 x NVIDIA NVIDIA GH200 480GB, 580.95.05, Python 3.9.21
See https://gist.github.com/Thyre/d13985dddd38a59c54578952ebc1d8c7 for a full test report.

@Micket
Copy link
Contributor Author

Micket commented Nov 24, 2025

Test report by @Micket
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3975
FAILED
Build succeeded for 0 out of 2 (2 easyconfigs in total)
vera-r07-06 - Linux Rocky Linux 9.6, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 580.95.05, Python 3.9.21
See https://gist.github.com/Micket/9df604ea7caae2fb389c8d56087cd636 for a full test report.

@Thyre
Copy link
Collaborator

Thyre commented Nov 24, 2025

Test report by @Micket Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3975 FAILED Build succeeded for 0 out of 2 (2 easyconfigs in total) vera-r07-06 - Linux Rocky Linux 9.6, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 580.95.05, Python 3.9.21 See https://gist.github.com/Micket/9df604ea7caae2fb389c8d56087cd636 for a full test report.

License once again...

@Micket
Copy link
Contributor Author

Micket commented Nov 24, 2025

Test report by @Micket
FAILED
Build succeeded for 2 out of 3 (3 easyconfigs in total)
vera-r07-06 - Linux Rocky Linux 9.6, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 580.95.05, Python 3.9.21
See https://gist.github.com/Micket/c001f076bb7b446f463c1c65d96426bb for a full test report.

@Thyre
Copy link
Collaborator

Thyre commented Nov 24, 2025

Test report by @Thyre
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3975
FAILED
Build succeeded for 2 out of 3 (total: 33 mins 9 secs) (3 easyconfigs in total)
jrc0900.jureca - Linux Rocky Linux 9.6, AArch64, ARM UNKNOWN (neoverse_v2), 1 x NVIDIA NVIDIA GH200 480GB, 580.95.05, Python 3.9.21
See https://gist.github.com/Thyre/3c8686afc77b5f33f59083da4bdba6a3 for a full test report.

@Thyre
Copy link
Collaborator

Thyre commented Nov 24, 2025

Test report by @Thyre
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3975
FAILED
Build succeeded for 2 out of 3 (total: 33 mins 9 secs) (3 easyconfigs in total) jrc0900.jureca - Linux Rocky Linux 9.6, AArch64, ARM UNKNOWN (neoverse_v2), 1 x NVIDIA NVIDIA GH200 480GB, 580.95.05, Python 3.9.21
See https://gist.github.com/Thyre/3c8686afc77b5f33f59083da4bdba6a3 for a full test report.

Most of the tests are failing with:

>   ???
E   cuda.bindings._internal.utils.NotSupportedError: only CUDA 12 driver is supported

cuda/bindings/_internal/utils.pyx:136: NotSupportedError

What a nightmare...

@Micket
Copy link
Contributor Author

Micket commented Nov 25, 2025

Test report by @Micket
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3975
FAILED
Build succeeded for 2 out of 3 (3 easyconfigs in total)
vera-r07-05 - Linux Rocky Linux 9.6, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 580.95.05, Python 3.9.21
See https://gist.github.com/Micket/dc20ef7dae111101bb650c1a6d856b21 for a full test report.

@Thyre
Copy link
Collaborator

Thyre commented Nov 25, 2025

Test report by @Thyre
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3975
FAILED
Build succeeded for 2 out of 3 (total: 33 mins 9 secs) (3 easyconfigs in total) jrc0900.jureca - Linux Rocky Linux 9.6, AArch64, ARM UNKNOWN (neoverse_v2), 1 x NVIDIA NVIDIA GH200 480GB, 580.95.05, Python 3.9.21
See https://gist.github.com/Thyre/3c8686afc77b5f33f59083da4bdba6a3 for a full test report.

Most of the tests are failing with:

>   ???
E   cuda.bindings._internal.utils.NotSupportedError: only CUDA 12 driver is supported

cuda/bindings/_internal/utils.pyx:136: NotSupportedError

What a nightmare...

CUDA-Python 12.9.1 god rid of this error message. CUDA-Python 12.9.0 still has it. 12.9.1 added cuda-pathfinder, which is used instead of the old approach IIUC.

@Micket
Copy link
Contributor Author

Micket commented Nov 25, 2025

side note "12.9.1 added cuda-pathfinder": Should i not have included in this bundle then (i intepret this as cuda-pathfinder not being used here).

So, the problem code is

cdef tuple get_nvjitlink_dso_version_suffix(int driver_ver):
    if 12000 <= driver_ver < 13000:
        return ('12', '')
    else:
        raise NotSupportedError('only CUDA 12 driver is supported')

and
NVIDIA/cuda-python@dd138dc
which removed it with pathfinder, which evidently isn't as strict anymore.

Looking at cuda/core/experimental/_linker.py i see

_nvjitlink = None  # populated if nvJitLink can be used
_nvjitlink_input_types = None  # populated if nvJitLink cannot be used

so, clearly, this code is meant ti work if nvJitLink can't be used, which, seems to be a strict requirement here not to tie the older version up to this. I'm thinking I would just shortcut

def _decide_nvjitlink_or_driver() -> bool:

to never consider nvjitlink.

@Micket
Copy link
Contributor Author

Micket commented Nov 25, 2025

I added a patch to disable nvjitlink as it's to picky about os drivers (for this version of CUDA-Python at least)

@Micket
Copy link
Contributor Author

Micket commented Nov 25, 2025

Test report by @Micket
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3975
SUCCESS
Build succeeded for 3 out of 3 (3 easyconfigs in total)
vera-r07-04 - Linux Rocky Linux 9.6, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A40, 580.95.05, Python 3.9.21
See https://gist.github.com/Micket/d88df5d19184f250365e92ade6314833 for a full test report.

@Thyre
Copy link
Collaborator

Thyre commented Nov 26, 2025

Test report by @Thyre
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3975
SUCCESS
Build succeeded for 3 out of 3 (total: 36 mins 29 secs) (3 easyconfigs in total)
jrc0900.jureca - Linux Rocky Linux 9.6, AArch64, ARM UNKNOWN (neoverse_v2), 1 x NVIDIA NVIDIA GH200 480GB, 580.95.05, Python 3.9.21
See https://gist.github.com/Thyre/cb55beb07e905e0ba1fd3ce31670928f for a full test report.

@boegel boegel changed the title {lang,tools}[GCCcore/14.2.0,foss/2025a] numba-cuda v0.20.0, filecheck-python v1.0.3 w/ CUDA 12.8.0 add cuda-binding, cuda-core, cuda-pathfinder extensions to CUDA-Python v12.8.0 + add easyconfigs for numba-cuda v0.20.0, filecheck-python v1.0.3 w/ foss/2025a + CUDA 12.8.0 Nov 27, 2025
Copy link
Member

@boegel boegel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Copy link
Member

@boegel boegel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@boegel boegel added enhancement and removed change labels Nov 27, 2025
@boegel
Copy link
Member

boegel commented Nov 27, 2025

@boegelbot please test @ jsc-zen3-a100
CORE_CNT=16

@boegelbot
Copy link
Collaborator

@boegel: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=24665 EB_ARGS= EB_CONTAINER= EB_REPO=easybuild-easyconfigs EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_24665 --ntasks="16" --partition=jsczen3g --gres=gpu:1 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 8932

Test results coming soon (I hope)...

- notification for comment with ID 3584778192 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link
Collaborator

Test report by @boegelbot
SUCCESS
Build succeeded for 3 out of 3 (total: 42 mins 12 secs) (3 easyconfigs in total)
jsczen3g1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.6, x86_64, AMD EPYC-Milan Processor (zen3), 1 x NVIDIA NVIDIA A100 80GB PCIe, 580.95.05, Python 3.9.21
See https://gist.github.com/boegelbot/1d0cb4c00aa86341fed7b7d2406b51be for a full test report.

@boegel
Copy link
Member

boegel commented Nov 27, 2025

Test report by @boegel
SUCCESS
Build succeeded for 3 out of 3 (total: 34 mins 19 secs) (3 easyconfigs in total)
node3905.accelgor.os - Linux RHEL 9.6, x86_64, AMD EPYC 7413 24-Core Processor (zen3), 1 x NVIDIA NVIDIA A100-SXM4-80GB, 580.95.05, Python 3.9.21
See https://gist.github.com/boegel/70c2f235b5dc197d11326d8ec216414b for a full test report.

@boegel
Copy link
Member

boegel commented Nov 27, 2025

Going in, thanks @Micket!

@boegel boegel merged commit 4428107 into easybuilders:develop Nov 27, 2025
8 checks passed
@Micket Micket deleted the 20251124153945_new_pr_numba-cuda0200 branch November 27, 2025 15:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

2025a issues & PRs related to 2025a common toolchains enhancement update

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants