Running a docker container with different drivers than the host

[Muhammad0312]

I want to use specific nvidia drivers for a docker container which are different from the host. I cannot update the drivers on the host. I have container toolkit and drivers installed on the host and after starting a container nvidia-smi shows the same cuda and driver version as the host. I think this is supposed to be like this but how can i change the drivers after i have started a container.
Thank you

[paulbkeelp]

#530 similar issue .. @Muhammad0312 did you fing any way forward please?

[github-actions]

This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed. To skip these checks, apply the "lifecycle/frozen" label.

[ArangoGutierrez]

Thank you for this question. It touches on the core design of the NVIDIA container stack.

The behavior you see is correct and intentional. You cannot run a different NVIDIA driver version inside a container than on the host.

Here is why:

1. Containers share the host's kernel. The NVIDIA driver has a component loaded directly into the host kernel.

2. The nvidia-container-toolkit's job is to mount the driver libraries from the host into the container so they can communicate with the kernel component.

This means the driver is fundamentally shared, not virtualized. You are likely confusing the NVIDIA Driver (shared from host) with the CUDA Toolkit (which can be different in the container).

For example, you can run a container built with the CUDA 11.8 Toolkit on a host that has a driver from the CUDA 12.x family. This is called "Minor Version Compatibility" and is fully supported.

As this is an explanation of the toolkit's fundamental design, we are closing this as answered.