`ERROR: driverInitFileInfo 53` when trying to launch a sample load

[liopeer]

Hi, I was following the official instructions for setting up the container toolkit. I have done this several times without any issues, but this time I get a weird error that seems to have something to do with the driver. I initially ignored it, because the output of nvidia-smi still showed the correct output. But when actually trying to then run things with the GPU inside the container it does not seem to pick up the GPU. Unfortunately, no LLM nor search engine were helpful in debugging this, so I am trying it here:

sudo docker run --rm --runtime=nvidia --gpus '"device=7"' ubuntu nvidia-smi
ERROR: driverInitFileInfo 531 result=1ERROR: init 608 result=1ERROR: init 250 result=1ERROR: driverInitFileInfo 531 result=1ERROR: init 608 result=1ERROR: init 250 result=1Thu Oct 30 10:01:56 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.124.06             Driver Version: 570.124.06     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA B200                    On  |   00000000:DF:00.0 Off |                    0 |
| N/A   33C    P0            140W / 1000W |       1MiB / 183359MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

So, I suspect that this is the culprit:

ERROR: driverInitFileInfo 531 result=1ERROR: init 608 result=1ERROR: init 250 result=1ERROR: driverInitFileInfo 531 result=1ERROR: init 608 result=1ERROR: init 250 result=1Thu Oct 30 10:01:56 2025

[elezar]

Question: Which version of the nvidia-container-toolkit is this? Also, doer running:

sudo docker run --rm --runtime=nvidia --gpus=all ubuntu nvidia-smi -L

also show this error?

Does running with --privileged mode change the behaviour?