rootless podman sees all GPUs despite cgroups setup

[arogozhnikov]

There is a discussion related to docker #211
But for docker it is expected that root-daemon has access to all gpus.

In my case, I run podman within SLURM, which uses cgroups to control acccess to devices.
CPU virtualization works correcctly, but not GPU virtualization, e.g. compare:

srun --label --nodes=2 --ntasks-per-node=1 --gpus-per-task=2 bash -c \
  'nvidia-smi --query-gpu=name,utilization.gpu,memory.used --format=csv'
0: name, utilization.gpu [%], memory.used [MiB]
0: NVIDIA A100-SXM4-80GB, 0 %, 0 MiB
0: NVIDIA A100-SXM4-80GB, 0 %, 0 MiB
1: name, utilization.gpu [%], memory.used [MiB]
1: NVIDIA A100-SXM4-80GB, 0 %, 0 MiB
1: NVIDIA A100-SXM4-80GB, 0 %, 0 MiB

vs

srun --label --nodes=2 --ntasks-per-node=1 --gpus-per-task=2 podman run --rm --device nvidia.com/gpu=all docker.io/nvidia/cuda:12.2.2-base-ubuntu22.04 bash -c \
  'nvidia-smi --query-gpu=name,utilization.gpu,memory.used --format=csv'
1: name, utilization.gpu [%], memory.used [MiB]
1: NVIDIA A100-SXM4-80GB, 0 %, 0 MiB
1: NVIDIA A100-SXM4-80GB, 0 %, 0 MiB
1: NVIDIA A100-SXM4-80GB, 0 %, 0 MiB
1: NVIDIA A100-SXM4-80GB, 0 %, 0 MiB
1: NVIDIA A100-SXM4-80GB, 0 %, 0 MiB
1: NVIDIA A100-SXM4-80GB, 0 %, 0 MiB
1: NVIDIA A100-SXM4-80GB, 0 %, 0 MiB
1: NVIDIA A100-SXM4-80GB, 0 %, 0 MiB
0: name, utilization.gpu [%], memory.used [MiB]
0: NVIDIA A100-SXM4-80GB, 0 %, 0 MiB
0: NVIDIA A100-SXM4-80GB, 0 %, 0 MiB
0: NVIDIA A100-SXM4-80GB, 0 %, 0 MiB
0: NVIDIA A100-SXM4-80GB, 0 %, 0 MiB
0: NVIDIA A100-SXM4-80GB, 0 %, 0 MiB
0: NVIDIA A100-SXM4-80GB, 0 %, 0 MiB
0: NVIDIA A100-SXM4-80GB, 0 %, 0 MiB
0: NVIDIA A100-SXM4-80GB, 0 %, 0 MiB

[elezar]

@arogozhnikov it seems as if specifying --device nvidia.com/gpu=all is overriding the devices that are available to the container.

Here it would be interesting to see whether running podman in general allows you to modify the cgroup permissions?

Does running:

podman run --rm --device-cgroup-rule=”c 195:* rwm” docker.io/nvidia/cuda:12.2.2-base-ubuntu22.04 bash

Allow you to access unexpected devices in the container?

[arogozhnikov]

srun --label --nodes=2 --ntasks-per-node=1 --gpus-per-task=2 podman run --rm --device-cgroup-rule="c 195:* rwm" --device nvidia.com/gpu=all docker.io/nvidia/cuda:12.2.2-base-ubuntu22.04 bash -c \
  'nvidia-smi --query-gpu=name,utilization.gpu,memory.used --format=csv'

output:

1: Error: device cgroup rules are not supported in rootless mode or in a user namespace
srun: error: compute-permanent-node-652: task 1: Exited with exit code 125
0: Error: device cgroup rules are not supported in rootless mode or in a user namespace
srun: error: compute-permanent-node-345: task 0: Exited with exit code 125

[arogozhnikov]

(I am not sure what conclusion should I draw, so let me know if that's helpful or if you want me to run something else)

Running on worker node verbatim command you posted provides the same result:

podman run --rm --device-cgroup-rule="c 195:* rwm" docker.io/nvidia/cuda:12.2.2-base-ubuntu22.04 bash
Error: device cgroup rules are not supported in rootless mode or in a user namespace

[elezar]

@arogozhnikov it is my understanding that rootless podman (and runc/crun as the low-level runtime) use bind mounts for device nodes in the rootless case.

What does something like:

srun --label --nodes=2 --ntasks-per-node=1 --gpus-per-task=2 podman run --rm --device nvidia.com/gpu=0 docker.io/nvidia/cuda:12.2.2-base-ubuntu22.04 bash -c \
  'nvidia-smi -L'

yield?

Then what about:

srun --label --nodes=2 --ntasks-per-node=1 --gpus-per-task=2 podman run --rm --device nvidia.com/gpu=0 -v /dev/nvidia3:/dev/nvidia3 docker.io/nvidia/cuda:12.2.2-base-ubuntu22.04 bash -c \
  'nvidia-smi -L'

[nlgranger]

Any updates on this? From my understanding, nvidia ctk bypasses the parent cgroup of podman in this situation.

[nlgranger]

No device reserved, no container:

$ srun nvidia-smi -L
No devices found.

No device reserved, container with device bind request:

$ srun podman run --rm --device nvidia.com/gpu=0 docker.io/nvidia/cuda:12.9.1-cudnn-runtime-ubuntu24.04 nvidia-smi -L
GPU 0: NVIDIA L40S (UUID: GPU-9ed9f8fb-59ab-136d-cfc6-0cf5956ee04a)

[nlgranger]

Ok I think I found the problem (many thanks to the folks at red hat):

• If systemd session lingering is enabled
• If Slurm is not configured to intercept logins
• If podman is configured to create it cgroup via systemd (the default)

Then the cgroup for the container is created in the user session and that is outside of the slurm job cgroup, thus device restrictions are not in effect.

This is neither an nvidia-container-toolkit nor a podman issue, it is a slurm and systemd configuration mistake. I think we can close this.

[arogozhnikov]

@nlgranger I don't mind closing issue if there is a clear next direction (e.g. should I configure slurm to intercept logins? did you check it works?)

[sents]

@nlgranger Do you have more Information on the solution that worked for you?
I've tried to make slurm+podman+nvidia-container-toolkit work for some time and could not make it work.

As I understand it Slurm creates it's on cgroup scope and within this scope cgroups are managed manually and not by systemd. Running podman without --cgroups=disabled will start the container in the user.slice and not in the slurmstepd.scope systemd scope that is managed by slurm.

Running podman with --cgroups=disabled keeps the container running in the correct cgroup but I still observe all gpus within the container.

[nlgranger]

I disabled session lingering which forces podman to fallback from "systemd" to the "cgroupfs" management method. The latter creates its cgroup nested within the slurm job cgroup as desired as far as I can tell.

I assume most headless systems have session lingering disabled by default, but I had it on because I got distracted while following https://slurm.schedmd.com/pam_slurm_adopt.html and had stopped before implementing the pam slurm_adopt part. I'm not entirely sure that is sufficient to prevent all forms of cgroup escape anyway.

Out of topic but if the home storage on your cluster is NFS-based you will face other problems with podman. Also podman really needs sub-uids and sub-gids which is a pain to manage. I think I will stick to enroot for now.

[github-actions]

This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed. To skip these checks, apply the "lifecycle/frozen" label.

[elezar]

Closing. Note that the solution to the issue is documented by @nlgranger in #585 (comment)