GPU recognized inside of podman container, but GPU calculations fail.

[jgforbes]

I built the container-toolkit version 1.16 RPMs for centos8-x86-64 so that they could be installed on a FIPS enabled system.
We were running GPU driver 555.42.02

After installing on a server with m40 GPUs and running the command to get the CDI config file, running podman like so:
podman run - --device nvidia.com/gpu=0 --security-opt=label=disable nvcr.io/nvidia/k8s/cuda-sample:devicequery
retrurns the expected information. nvidia-smi in the container also worked.
However, running,
podman run - --device nvidia.com/gpu=0 --security-opt=label=disable nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubi8

returns
Failed to allocate device vector A (error code CUDA-capable device(s) is/are busy or unavailable)!
[Vector addition of 50000 elements]

Installing the driver version, 550.90.07, gave different results. The devicequery container work and nvidia-smi worked int he container. but the vectoradd container reported:
WARNING: CUDA Minor Version Compatibility mode ENABLED.
Using driver version 550.90.07 which has support for CUDA 12.4. This container
was built with CUDA 12.5 and will be run in Minor Version Compatibility mode.
CUDA Forward Compatibility is preferred over Minor Version Compatibility for use
with this container but was unavailable:
[[Invalid argument (CUDA_ERROR_INVALID_VALUE) cuDevicePrimaryCtxRetain()=1]]
See https://docs.nvidia.com/deploy/cuda-compatibility/ for details.

Installing GPU driver 535 gave the same results as the 555 driver.

Installing GPU driver version 525,147.05 worked for all gpu calculations as expected.

Are these results due to the older M40 GPU even though it is supported in all of the drivers that we tried or something else?
Any ideas?

[github-actions]

This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed. To skip these checks, apply the "lifecycle/frozen" label.

[ArangoGutierrez]

The nvidia-smi and devicequery tests work because they primarily interact with the NVIDIA Driver, which still recognizes the M40 hardware.

The vectoradd calculation fails because the Maxwell architecture has been deprecated and is no longer supported by the CUDA 12.x toolkit family. When the vectoradd application (built with CUDA 12.5) tries to run, it cannot create a valid CUDA context on the unsupported Maxwell hardware.

The error you see with the 550 driver (cuDevicePrimaryCtxRetain()=1) is the low-level symptom of this unsupported combination. The 525 driver likely works because its compatibility layer for older hardware was the last one to fully support this architecture.

To resolve this, you must either use a newer GPU (Pascal architecture or newer) or run containers built with an older CUDA Toolkit (e.g., CUDA 11) that still supports the Maxwell architecture.

As this is a hardware support limitation and not a bug in the toolkit, we are closing this as answered.