prgenv-gnu with ROCm 7 #273

msimberg · 2025-11-20T15:15:53Z

Just for testing builds, I don't know if this will work.

…tests

msimberg · 2025-11-20T15:16:34Z

cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:25.12

msimberg · 2025-11-21T12:47:56Z

hsa-amd-aqlprofile had missing compiler dependencies: spack/spack-packages#2532.

msimberg · 2025-11-24T12:00:05Z

cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:25.12

msimberg · 2025-11-24T20:09:08Z

cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:25.12

msimberg · 2025-11-25T08:33:01Z

ROCm is a menace: https://gitlab.com/cscs-ci/ci-testing/webhook-ci/mirrors/551234120955960/1440398897047560/-/jobs/12201564988#L3724. hipblaslt seems to be picking up amdclang++ from the system... needs further investigation.

msimberg · 2025-11-26T09:05:12Z

cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:25.12

msimberg · 2025-11-26T09:09:05Z

cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:25.12

msimberg · 2025-11-26T13:59:40Z

cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:25.12

jgphpc · 2025-11-28T14:48:35Z

Hi,
Could you add rocprofiler_sdk/package.py to the recipe ? needed for linaro...

recipes/prgenv-gnu/25.12/amdgpu/environments.yaml

msimberg · 2025-11-28T15:51:47Z

Hi, Could you add rocprofiler_sdk/package.py to the recipe ? needed for linaro...

I'll try it out, hopefully no bigger issues (though note that I have issues with other packages before this is usable unfortunately).

msimberg · 2025-12-03T09:20:18Z

spack/spack-packages#2287 to add rocm 7.1.0 is also currently open. It may help, it may make things worse... The PR description does mention a change to hipblaslt, which may change something.

afzpatel · 2025-12-03T14:58:32Z

ROCm is a menace: https://gitlab.com/cscs-ci/ci-testing/webhook-ci/mirrors/551234120955960/1440398897047560/-/jobs/12201564988#L3724. hipblaslt seems to be picking up amdclang++ from the system... needs further investigation.

Yes, it looks like an older version of ROCm is installed on the system and it's choosing the incorrect version of amdclang++. I'll see if I can reproduce the issue and put in a fix.

iomaganaris · 2025-12-15T10:28:47Z

cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:25.12

iomaganaris · 2025-12-16T13:58:34Z

cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:25.12

afzpatel · 2025-12-16T14:02:16Z

ROCm is a menace: https://gitlab.com/cscs-ci/ci-testing/webhook-ci/mirrors/551234120955960/1440398897047560/-/jobs/12201564988#L3724. hipblaslt seems to be picking up amdclang++ from the system... needs further investigation.

Yes, it looks like an older version of ROCm is installed on the system and it's choosing the incorrect version of amdclang++. I'll see if I can reproduce the issue and put in a fix.

I was able to reproduce and fix the previous error with hipblaslt by setting Tensile_COMPILER in the recipe:

diff --git a/repos/spack_repo/builtin/packages/hipblaslt/package.py b/repos/spack_repo/builtin/packages/hipblaslt/package.py
index eba3fbc25b..e2337352ee 100644
--- a/repos/spack_repo/builtin/packages/hipblaslt/package.py
+++ b/repos/spack_repo/builtin/packages/hipblaslt/package.py
@@ -278,6 +278,11 @@ class Hipblaslt(CMakePackage):
                     "ROCROLLER_ASSEMBLER_PATH", f"{self.spec['llvm-amdgpu'].prefix}/bin/amdclang++"
                 )
             )
+            args.append(
+                self.define(
+                    "Tensile_COMPILER", f"{self.spec['llvm-amdgpu'].prefix}/bin/amdclang++"
+                )
+            )
         if self.spec.satisfies("@7.1:"):
             args.append(self.define("HIPBLASLT_ENABLE_CLIENT", self.run_tests))
         else:

iomaganaris · 2025-12-16T14:07:38Z

ROCm is a menace: https://gitlab.com/cscs-ci/ci-testing/webhook-ci/mirrors/551234120955960/1440398897047560/-/jobs/12201564988#L3724. hipblaslt seems to be picking up amdclang++ from the system... needs further investigation.

Yes, it looks like an older version of ROCm is installed on the system and it's choosing the incorrect version of amdclang++. I'll see if I can reproduce the issue and put in a fix.

I was able to reproduce and fix the previous error with hipblaslt by setting Tensile_COMPILER in the recipe:
diff --git a/repos/spack_repo/builtin/packages/hipblaslt/package.py b/repos/spack_repo/builtin/packages/hipblaslt/package.py
index eba3fbc25b..e2337352ee 100644
--- a/repos/spack_repo/builtin/packages/hipblaslt/package.py
+++ b/repos/spack_repo/builtin/packages/hipblaslt/package.py
@@ -278,6 +278,11 @@ class Hipblaslt(CMakePackage):
                     "ROCROLLER_ASSEMBLER_PATH", f"{self.spec['llvm-amdgpu'].prefix}/bin/amdclang++"
                 )
             )
+            args.append(
+                self.define(
+                    "Tensile_COMPILER", f"{self.spec['llvm-amdgpu'].prefix}/bin/amdclang++"
+                )
+            )
         if self.spec.satisfies("@7.1:"):
             args.append(self.define("HIPBLASLT_ENABLE_CLIENT", self.run_tests))
         else:

Good suggestion, thank you 👍 Any reason why this was not merged in develop?

afzpatel · 2025-12-16T14:10:36Z

ROCm is a menace: https://gitlab.com/cscs-ci/ci-testing/webhook-ci/mirrors/551234120955960/1440398897047560/-/jobs/12201564988#L3724. hipblaslt seems to be picking up amdclang++ from the system... needs further investigation.

Yes, it looks like an older version of ROCm is installed on the system and it's choosing the incorrect version of amdclang++. I'll see if I can reproduce the issue and put in a fix.

I was able to reproduce and fix the previous error with hipblaslt by setting Tensile_COMPILER in the recipe:
diff --git a/repos/spack_repo/builtin/packages/hipblaslt/package.py b/repos/spack_repo/builtin/packages/hipblaslt/package.py
index eba3fbc25b..e2337352ee 100644
--- a/repos/spack_repo/builtin/packages/hipblaslt/package.py
+++ b/repos/spack_repo/builtin/packages/hipblaslt/package.py
@@ -278,6 +278,11 @@ class Hipblaslt(CMakePackage):
                     "ROCROLLER_ASSEMBLER_PATH", f"{self.spec['llvm-amdgpu'].prefix}/bin/amdclang++"
                 )
             )
+            args.append(
+                self.define(
+                    "Tensile_COMPILER", f"{self.spec['llvm-amdgpu'].prefix}/bin/amdclang++"
+                )
+            )
         if self.spec.satisfies("@7.1:"):
             args.append(self.define("HIPBLASLT_ENABLE_CLIENT", self.run_tests))
         else:
Good suggestion, thank you 👍 Any reason why this was not merged in develop?

I was trying to get my 7.1.0 PR merged for the past couple weeks, I didn't want to add any additional changes because I thought it might delay it getting merged. We'll add the change with 7.1.1: spack/spack-packages#2782

iomaganaris · 2025-12-16T14:12:30Z

ROCm is a menace: https://gitlab.com/cscs-ci/ci-testing/webhook-ci/mirrors/551234120955960/1440398897047560/-/jobs/12201564988#L3724. hipblaslt seems to be picking up amdclang++ from the system... needs further investigation.

Yes, it looks like an older version of ROCm is installed on the system and it's choosing the incorrect version of amdclang++. I'll see if I can reproduce the issue and put in a fix.

I was able to reproduce and fix the previous error with hipblaslt by setting Tensile_COMPILER in the recipe:
diff --git a/repos/spack_repo/builtin/packages/hipblaslt/package.py b/repos/spack_repo/builtin/packages/hipblaslt/package.py
index eba3fbc25b..e2337352ee 100644
--- a/repos/spack_repo/builtin/packages/hipblaslt/package.py
+++ b/repos/spack_repo/builtin/packages/hipblaslt/package.py
@@ -278,6 +278,11 @@ class Hipblaslt(CMakePackage):
                     "ROCROLLER_ASSEMBLER_PATH", f"{self.spec['llvm-amdgpu'].prefix}/bin/amdclang++"
                 )
             )
+            args.append(
+                self.define(
+                    "Tensile_COMPILER", f"{self.spec['llvm-amdgpu'].prefix}/bin/amdclang++"
+                )
+            )
         if self.spec.satisfies("@7.1:"):
             args.append(self.define("HIPBLASLT_ENABLE_CLIENT", self.run_tests))
         else:
Good suggestion, thank you 👍 Any reason why this was not merged in develop?
I was trying to get my 7.1.0 PR merged for the past couple weeks, I didn't want to add any additional changes because I thought it might delay it getting merged. We'll add the change with 7.1.1: spack/spack-packages#2782

Great! Thank you very much for the clarification and the updates to the ROCm packages 👍

iomaganaris · 2025-12-16T14:17:51Z

cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:25.12

iomaganaris · 2025-12-16T14:25:45Z

cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:25.12

iomaganaris · 2025-12-19T11:41:57Z

cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:25.12

iomaganaris · 2025-12-19T14:33:51Z

cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:25.12

…with llvm-amdgpu to avoid rebuilding python packages

iomaganaris · 2025-12-19T15:12:36Z

cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:25.12

simonpintarelli · 2025-12-23T15:31:18Z

the gtl library of [email protected] is linked to libamdhip64.so.6. I think we need [email protected], which is now in alps-cluster-config. It will only work on reservation sles15sp6, as 8.1.33 requires the new glibc.

➜  lib git:(main) ✗ readelf -d libmpi_gtl_hsa.so                                                                                                                                           [25/12/23| 4:28PM]

Dynamic section at offset 0xa1da0 contains 28 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libamdhip64.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libhsa-runtime64.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 ...

UPDATE 🤦🏻 cray-mpich 8.1.33 and 9.0.1 are also linked to libamdhip64.so.6 :) We can still try patchelf and replace by so.7, not sure this is going to work..

simonpintarelli · 2025-12-23T15:31:44Z

cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:25.12

simonpintarelli · 2025-12-23T16:22:27Z

cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:25.12

iomaganaris · 2026-01-07T14:49:30Z

the gtl library of [email protected] is linked to libamdhip64.so.6. I think we need [email protected], which is now in alps-cluster-config. It will only work on reservation sles15sp6, as 8.1.33 requires the new glibc.
➜  lib git:(main) ✗ readelf -d libmpi_gtl_hsa.so                                                                                                                                           [25/12/23| 4:28PM]

Dynamic section at offset 0xa1da0 contains 28 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libamdhip64.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libhsa-runtime64.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 ...
UPDATE 🤦🏻 cray-mpich 8.1.33 and 9.0.1 are also linked to libamdhip64.so.6 :) We can still try patchelf and replace by so.7, not sure this is going to work..

I haven't looked into this issue but I saw that there's not only libamdhip64.so.6 coming from /opt/rocm-6.1.0/lib but also libamd_comgr.so.2:

[beverin][ioannmag@beverin-ln001 ICON4PY]$ ldd /user-environment/linux-zen3/cray-gtl-8.1.32-dykogaehen633o4fzsqlr5hhept7i7j3/lib/libmpi_gtl_hsa.so | grep "opt/rocm"
        libamdhip64.so.6 => /opt/rocm-6.1.0/lib/libamdhip64.so.6 (0x00007f98acf47000)
        libamd_comgr.so.2 => /opt/rocm-6.1.0/lib/libamd_comgr.so.2 (0x00007f98a464a000)
[beverin][ioannmag@beverin-ln001 ICON4PY]$ find /user-environment -name "libamdhip64.so*"
/user-environment/env/._default/ogqp57onxwlinpjjcp4rwqcypo5yqqtb/lib/libamdhip64.so
/user-environment/env/._default/ogqp57onxwlinpjjcp4rwqcypo5yqqtb/lib/libamdhip64.so.7
/user-environment/env/._default/ogqp57onxwlinpjjcp4rwqcypo5yqqtb/lib/libamdhip64.so.7.1.25505
/user-environment/linux-zen3/hip-7.1.0-hlbhwy6epcpcbymr3qpsu6x3nxr2fizf/lib/libamdhip64.so
/user-environment/linux-zen3/hip-7.1.0-hlbhwy6epcpcbymr3qpsu6x3nxr2fizf/lib/libamdhip64.so.7
/user-environment/linux-zen3/hip-7.1.0-hlbhwy6epcpcbymr3qpsu6x3nxr2fizf/lib/libamdhip64.so.7.1.25505
[beverin][ioannmag@beverin-ln001 ICON4PY]$ find /user-environment -name "libamd_comgr.so*"
/user-environment/linux-zen3/comgr-7.1.0-d6zrig6m7vdftmyb4g5onyts7p5vrahc/lib/libamd_comgr.so
/user-environment/linux-zen3/comgr-7.1.0-d6zrig6m7vdftmyb4g5onyts7p5vrahc/lib/libamd_comgr.so.3
/user-environment/linux-zen3/comgr-7.1.0-d6zrig6m7vdftmyb4g5onyts7p5vrahc/lib/libamd_comgr.so.3.0

I'm not really familiar with the cray-mpich package but maybe adding %c,cxx,[email protected] helps. I would try this out in a separate PR or locally though 🙈

simonpintarelli · 2026-01-07T16:26:16Z

@iomaganaris The problem is that the binary rpms for cray-mpich are linked against libamdhip.so.6, since there is no libamdhip.so.6 in the spack installation, it finds the libraries in /opt/rocm-6.1.0. I think we have to wait for HPE to release a new cray-mpich, which is built against rocm7.

A workaround would be to build without the gtl/hsa library (cray-mpich~rocm).

msimberg · 2026-01-08T08:59:26Z

BTW, it occurred to me now that if we want to avoid the issues with HPE's precompiled binaries, we could make an OpenMPI uenv instead with ROCm 7. I'd make it a separate uenv, but it might be a faster/smoother option at the moment instead of trying to patch up HPE's binaries. What do you think? If ROCm 7 is otherwise building ok now I'd expect changing to OpenMPI to be relatively simple. I can try to set that up if you think it's useful. I'm anyway about to deploy #263.

simonpintarelli · 2026-01-08T10:22:28Z

@msimberg I built q-e-sirius with openmpi+rocm7:
https://github.com/eth-cscs/alps-uenv/pull/288/files

I'm not sure how to run it correctly:

srun --mpi=pmix --jobid=205659 --overlap -n2 osu_bw D D                                                                                                    
--------------------------------------------------------------------------                                                                                                                                    
A requested component was not found, or was unable to be opened.  This                                                                                                                                        
means that this component is either not installed or is unable to be                                                                                                                                          
used on your system (e.g., sometimes this means that shared libraries                                                                                                                                         
that the component requires are unable to be found/loaded).  Note that                                                                                                                                        
PMIx stopped checking at the first component that it did not find.                                     

Host:      nid002920                                                                                                                                                                                          
Framework: psec                                                                                                                                                                                               
Component: munge

--mpi=pmi2 worked, but still gave a warning:

[beverin][simonpi@nid002920 osu-micro-benchmarks]$ srun --mpi=pmi2 --jobid=205659 --overlap -n2 osu_bw D D
No PMIx server was reachable, but a PMI1/2 was detected.
If srun is being used to launch application,  2 singletons will be started.

msimberg · 2026-01-08T10:35:35Z

@simonpintarelli have a look here: https://docs.tds.cscs.ch/301/software/communication/openmpi/#uenv (not merged yet). The munge warning should be harmless if it ran otherwise (but I'm pretty sure you need to set the other variables mentioned on that page). That said, pmix might be set up differently on beverin and I haven't tested there.

CXI should be the safe choice. LNX is faster if it works, but it may not work...

msimberg · 2026-01-08T10:45:20Z

I just did a quick test with q-e-sirius/v1.0.2-rocm7:2235068710 and osu_bw works with CXI, but it fails immediately with LNX.

iomaganaris · 2026-01-08T18:14:39Z

cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:25.12

iomaganaris · 2026-01-08T19:04:13Z

cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:25.12

iomaganaris · 2026-01-08T22:10:35Z

cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:25.12

msimberg and others added 6 commits November 14, 2025 11:05

Add rccl and rccl-tests to rocm uenv

0d1a2a8

Disable duplicates strategy for now

be76c7f

Verify that we can't concretize with one cray-mpich

c58f3bd

Merge remote-tracking branch 'origin/main' into prgenv-gnu-rocm-rccl-…

fe92e52

…tests

Enable duplicates:strategy:full again

161e5e4

Add prgenv-gnu rocm 7.0.2 uenv

bc2d134

Update spack-packages

9126f90

Temporarily disable rocprofiler-systems

565dda6

msimberg added 2 commits November 26, 2025 10:04

Merge remote-tracking branch 'origin/main' into prgenv-gnu-rocm-7

54f4b3a

Disable hipblaslt for now

4a7989e

Fix linaro forge config

c40da6f

Disable tensile temporarily

47ff00f

msimberg commented Nov 28, 2025

View reviewed changes

recipes/prgenv-gnu/25.12/amdgpu/environments.yaml Outdated Show resolved Hide resolved

Apply suggestion from @msimberg

023d28e

Try ROCm 7.1.0 from spack PR

de8e342

Trying to fix hipblaslt

cb127d1

Fix 7.1.0 hipblaslt

63cb72f

Fix compiler version in yml file

71a0efb

New try with fixes of clang-offload-bunder

ca3cd15

Disable all the rocprofiler related packages

7df04a7

Add unwind dependency to rocprofiler-register and use python 3.12.12 …

01633bc

…with llvm-amdgpu to avoid rebuilding python packages

[email protected]

fccb073

[email protected] in rccl-test dependencies

8182e42

Trying to enable rocprofiler-compute and -systems

069636e

Update py-dash recipe

d98ce89

prgenv-gnu with ROCm 7 #273

Are you sure you want to change the base?

prgenv-gnu with ROCm 7 #273

Uh oh!

Conversation

msimberg commented Nov 20, 2025

Uh oh!

msimberg commented Nov 20, 2025

Uh oh!

msimberg commented Nov 21, 2025

Uh oh!

msimberg commented Nov 24, 2025

Uh oh!

msimberg commented Nov 24, 2025

Uh oh!

msimberg commented Nov 25, 2025

Uh oh!

msimberg commented Nov 26, 2025

Uh oh!

msimberg commented Nov 26, 2025

Uh oh!

msimberg commented Nov 26, 2025

Uh oh!

jgphpc commented Nov 28, 2025

Uh oh!

Uh oh!

msimberg commented Nov 28, 2025

Uh oh!

msimberg commented Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

afzpatel commented Dec 3, 2025

Uh oh!

iomaganaris commented Dec 15, 2025

Uh oh!

iomaganaris commented Dec 16, 2025

Uh oh!

afzpatel commented Dec 16, 2025

Uh oh!

iomaganaris commented Dec 16, 2025

Uh oh!

afzpatel commented Dec 16, 2025

Uh oh!

iomaganaris commented Dec 16, 2025

Uh oh!

iomaganaris commented Dec 16, 2025

Uh oh!

iomaganaris commented Dec 16, 2025

Uh oh!

iomaganaris commented Dec 19, 2025

Uh oh!

iomaganaris commented Dec 19, 2025

Uh oh!

iomaganaris commented Dec 19, 2025

Uh oh!

simonpintarelli commented Dec 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

simonpintarelli commented Dec 23, 2025

Uh oh!

simonpintarelli commented Dec 23, 2025

Uh oh!

iomaganaris commented Jan 7, 2026

Uh oh!

simonpintarelli commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

msimberg commented Jan 8, 2026

Uh oh!

simonpintarelli commented Jan 8, 2026

Uh oh!

msimberg commented Jan 8, 2026

Uh oh!

msimberg commented Jan 8, 2026

Uh oh!

iomaganaris commented Jan 8, 2026

Uh oh!

iomaganaris commented Jan 8, 2026

Uh oh!

iomaganaris commented Jan 8, 2026

msimberg commented Dec 3, 2025 •

edited

Loading

simonpintarelli commented Dec 23, 2025 •

edited

Loading

simonpintarelli commented Jan 7, 2026 •

edited

Loading