Add support to install the PMIX plugin for Slurm

[tomschultz-psivant]

Hello!

I'm trying to deploy a cluster with pmix installed as a slurm plugin using the "image-builder" blueprint ( https://github.com/GoogleCloudPlatform/hpc-toolkit/blob/develop/examples/image-builder.yaml ) and I can seem to get the image to build/deploy correctly.

The communication between the controller and login nodes keeps getting messed up, something similar to this https://groups.google.com/g/slurm-users/c/Q7FVkhx-bOsand this https://bugs.schedmd.com/show_bug.cgi?id=10066.

I'm guessing that I'm injecting my pmix installation stuff at the wrong moment (https://github.com/GoogleCloudPlatform/hpc-toolkit/blob/develop/examples/image-builder.yaml#L42). Also, it requires re-configuring and installing slurm so I would want to run this at the very end of everything getting setup.

My current guess is that I would actually need to make an Ansible task and follow something similar to how the “install_ompi” tasks are setup through Ansible?

https://github.com/GoogleCloudPlatform/slurm-gcp/blob/849107c9f2e8044051877de579c7308e4e655139/packer/main.pkr.hcl#L31

Is this something that is on the roadmap or possibly something that you all have done before?

[cdunbar13]

Hi, just letting you know that we have seen this and are working on it.

We wanted to know if this is an immediate need or is something that will be needed in the future.

[tomschultz-psivant]

So when I have yum install -y pmix in the startup-script section it doesn't seem to load it completely.

yum search slurm doesn't produce and specific pmix plugins. :/

also yum search pmix:

That didn't seem to work :(

[cboneti]

yes, I have verified it is not sufficient to install pmix after slurm, it has to be done before installing slurm and so using the build slurm image blueprint would be much easier, you could add that on line 52 or so.

[cboneti]

I haven't verified this cluster, but in interest of time, the below blueprint should produce an image that you can use for a slurm cluster and that will have pmix setup:

blueprint_name: hpc-build-slurm-image

vars:
  project_id:  ## Set GCP Project ID Here ##
  deployment_name: build-slurm-1
  region: us-central1
  zone: us-central1-a

  image_build_machine_type: n2d-standard-16
  build_from_image_family: hpc-centos-7
  build_from_image_project: cloud-hpc-image-public
  build_from_git_ref: 5.10.4
  built_image_family: my-custom-slurm
  # built_instance_image:
  #   family: $(vars.built_image_family)
  #   project: $(vars.project_id)
  # instance_image_custom: true

deployment_groups:
- group: setup
  modules:
  - id: network
    source: modules/network/vpc

  - id: slurm-build-script
    source: modules/scripts/startup-script
    settings:
      # Do not create Ansible virtual env; Install system wide Ansible below.
      install_ansible: false
      runners:
      - type: shell
        destination: prep-for-slurm-build.sh
        content: |
          #!/bin/bash
          set -e -o pipefail
          # Slurm build on Rocky8 will upgrade to python38 as part of build
          # This is not compatible with ansible-local runner
          yum install -y python3
          yum install -y pmix
          # alternatives --set python3 /usr/bin/python3
          python3 -m pip install pip --upgrade
          python3 -m pip install ansible
          python3 -m pip install selinux
          export PATH=/usr/local/bin:$PATH
          ansible --version
          ansible-galaxy role install googlecloudplatform.google_cloud_ops_agents
      - type: data
        destination: /var/tmp/slurm_vars.json
        content: |
          {
            "reboot": false,
            "install_cuda": false,
            "nvidia_version": "latest",
            "install_ompi": true,
            "install_lustre": false,
            "install_gcsfuse": true
          }
      - type: shell
        destination: install_slurm.sh
        content: |
          #!/bin/bash
          set -e -o pipefail
          ansible-pull \
              -U https://github.com/GoogleCloudPlatform/slurm-gcp -C $(vars.build_from_git_ref) \
              -i localhost, --limit localhost --connection=local \
              -e @/var/tmp/slurm_vars.json \
              ansible/playbook.yml

- group: build-slurm
  modules:
  - id: slurm-custom-image
    source: modules/packer/custom-image
    kind: packer
    settings:
      machine_type: $(vars.image_build_machine_type)
      source_image_family: $(vars.build_from_image_family)
      source_image_project_id: [$(vars.build_from_image_project)]
      image_family: $(vars.built_image_family)
    use:
    - network
    - slurm-build-script

[tomschultz-psivant]

Thank you for putting that together!

I might have communicated the wrong need before, I'm pretty new to slurm. But I need to get the PMIX MPI plugin installed with slurm.

There's a few tests that I do typically to check.

1. I run srun --mpi=list and look for the pmix plugin type
2. Then if that's present in the login or controller node, then I test to make sure that the worker nodes that are spun up also have it by running something like srun --mpi=pmix -n 2 -N2 srun --mpi=list or srun --mpi=pmix -n2 -N2 hostname etc...

When I run this script on the slurm controller or login nodes, it seems to work. But obviously, when I try to spin up new nodes they don't have the pmix plugin installed since they're being started up from the base images.

         #/bin/bash
          sudo yum install -y wget

          mkdir -p /usr/local/src/pmix
          cd /usr/local/src/pmix

          # Download and install pmix
          wget https://github.com/openpmix/openpmix/releases/download/v4.2.4/pmix-4.2.4.tar.gz
          tar -xzvf pmix-4.2.4.tar.gz
          cd pmix-4.2.4

          # Slurm defaultly looks for pmix in /usr/local/pmix
          mkdir /usr/local/pmix

          ./configure --prefix=/usr/local/pmix |& tee config.out
          make -j 8 |& tee make.out
          make install |& tee install.out

          # Install slurm with pmix support
          cd /usr/local/src/slurm

          ./configure --with-pmix=/usr/local/pmix

          make -j install

Also, when I add this to the image-builder blueprint, it installs successfully (no errors) but then it starts the controller instance, it seems that slurm configless is confused and unable to receive connections from other nodes and slurmdbd doesn't seem to want to start back up.

Let me know if that does or doesn't make sense?

Thanks again, we really appreciate your help on this!

References

• https://slurm.schedmd.com/mpi_guide.html#pmix

[tpdownes]

@tomschultz-roivant I'm going to move this discussion to the #2274 (an issue I created to track this in part because it is mirrored to a tool we use internally for tracking work).