kubelet plugins: add /opt/bin to binary search paths

[Gacko]

Hello everyone!

This is my first contribution to this repository, so please feel free to point me to any requirement for contributions I might have missed.

This change in particular adds support for finding the nvidia-smi binary on distributions which have it installed to /opt/bin on the host. Flatcar Linux for example offers an NVIDIA sysext which installs the binaries and drivers when run on a GPU instance. This way you do not need to provide them via a DaemonSet or sidecar container.

Sadly the current approach of searching for the nvidia-smi binary does not cover /opt/bin. Actually the Flatcar NVIDIA sysext creates a symlink /usr/bin/nvidia-smi, but this is pointing to /opt/bin/nvidia-smi. So when either the kubelet plugin pre-start script or the kubelet plugin itself look for the nvidia-smi in /driver-root/usr/bin, they only find a symlink pointing to outside the host filesystem mounted below /driver-root and therefore fail to access the actual binary.

My change adds /opt/bin to the search directories of the pre-start script and both kubelet plugins, so they can find the real binary there before looking for it in /usr/bin where they would only find a dead end symlink.

Additionally I fixed a minor bug in the pre-start script: /sbin was duplicated there, while it most likely should be /usr/sbin and /sbin. Also I re-ordered the paths to be aligned with the kubelet plugins.

Thanks for reviewing my PR in advance
Marco

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[Gacko]

The force-push was meant to sign-off the commit and satisfy CI.

[jgehrcke]

This is my first contribution to this repository

Sweet!

they only find a symlink pointing to outside the host filesystem mounted below /driver-root and therefore fail to access the actual binary.

Thanks for the precise description.

My change adds /opt/bin to the search directories of the pre-start script and both kubelet plugins, so they can find the real binary there

My most important question is: did you test that this works -- did this lead to a (from your point of view) functional state of the DRA driver? Which plugin did you try?

If your answer is that "yes, I could allocate GPUs and/or ComputeDomains that way" I am very happy to take your word for it and land this. As you can imagine, testing the different path / placement variants systematically is a bit of a hassle.

Additionally I fixed a minor bug in the pre-start script: /sbin was duplicated there, while it most likely should be /usr/sbin and /sbin

Thanks for noticing.

Also I re-ordered the paths to be aligned with the kubelet plugins.

👍 :)

[jgehrcke]

Three more points, if you allow :)

1. What's your involvement with Flatcar container Linux?

2. A curious question I have: where is libnvidia-ml.so.1 placed in your case?

3. Can you maybe change the commit message to kubelet plugins: add /opt/bin to binary search paths?

[jgehrcke]

Question to us: could this (considering /opt) in any way be a security concern?

[Gacko]

Good morning @jgehrcke!

Thank you for your fast response - and sorry for my delayed one, I wasn't expecting that!

My most important question is: did you test that this works -- did this lead to a (from your point of view) functional state of the DRA driver? Which plugin did you try?

If your answer is that "yes, I could allocate GPUs and/or ComputeDomains that way" I am very happy to take your word for it and land this. As you can imagine, testing the different path / placement variants systematically is a bit of a hassle.

It took me some time to get the GitHub Actions running in my fork since you hard-coded NVIDIA in some places, which isn't a lot useful when trying to publish the image and the chart to your own registry, but finally I got everything up and running. 😅

Since I was initially confronted with the kubelet plugin pre-start script not being able to find the nvidia-smi binary, the scope of this PR is limited to adapting the search behavior. If at any point I am confronted with other problems, I will come up with an issue and, if I am able to fix it, another PR.

This said I can tell: Yes, I tested my change and with it the pre-start script is able to find the nvidia-smi binary and the libnvidia-ml.so.1 library. Also the GPU kubelet plugin I am currently using is coming up fine, even though I had to add a NetworkPolicy to allow access to the API server. I haven't tested anything beyond that, yet, so also cannot assure the plugin itself to be working as expected. Also I haven't tested the Compute Domains kubelet plugin, as this isn't required for our use case, but I will do so later.

Following you can find the container logs of both the pre-start script and the GPU kubelet plugin. This will also already give you some details about my environment:

kubelet plugin pre-start script

create symlink: /driver-root -> /driver-root-parent/
2025-10-30T08:15:49Z  /driver-root (/ on host): nvidia-smi: '/driver-root/opt/bin/nvidia-smi', libnvidia-ml.so.1: '/driver-root/usr/lib64/libnvidia-ml.so.1', current contents: [].
invoke: env -i LD_PRELOAD=/driver-root/usr/lib64/libnvidia-ml.so.1 /driver-root/opt/bin/nvidia-smi
Thu Oct 30 08:15:49 2025       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.230.02             Driver Version: 535.230.02   CUDA Version: N/A      |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       Off | 00000000:00:1E.0 Off |                    0 |
| N/A   29C    P8              14W /  70W |      0MiB / 15360MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+
nvidia-smi returned with code 0: success, leave

GPU kubelet plugin

I1030 08:15:50.484600       1 utils.go:43] 
Feature gates: map[string]bool{"AllAlpha":false, "AllBeta":false, "ContextualLogging":true, "IMEXDaemonsWithDNSNames":true, "LoggingAlphaOptions":false, "LoggingBetaOptions":true, "MPSSupport":false, "TimeSlicingSettings":false}
Verbosity: 4
Flags: (*main.Flags)({
  kubeClientConfig: (flags.KubeClientConfig) {
    KubeConfig: (string) "",
    KubeAPIQPS: (float64) 5,
    KubeAPIBurst: (int) 10
  },
  nodeName: (string) (len=41) "ip-10-0-90-166.eu-west-2.compute.internal",
  namespace: (string) (len=11) "kube-system",
  cdiRoot: (string) (len=12) "/var/run/cdi",
  containerDriverRoot: (string) (len=12) "/driver-root",
  hostDriverRoot: (string) (len=1) "/",
  nvidiaCDIHookPath: (string) "",
  imageName: (string) (len=54) "ghcr.io/gacko/k8s-dra-driver-gpu:v25.12.0-dev-426973ab",
  kubeletRegistrarDirectoryPath: (string) (len=33) "/var/lib/kubelet/plugins_registry",
  kubeletPluginsDirectoryPath: (string) (len=24) "/var/lib/kubelet/plugins",
  healthcheckPort: (int) 51516
})
I1030 08:15:50.485155       1 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I1030 08:15:50.485174       1 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I1030 08:15:50.485178       1 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I1030 08:15:50.485184       1 envvar.go:172] "Feature gate default state" feature="InOrderInformers" enabled=true
I1030 08:15:50.485189       1 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I1030 08:15:50.507678       1 device_state.go:71] using devRoot=/driver-root
I1030 08:15:50.592389       1 draplugin.go:597] "Starting"
I1030 08:15:50.592764       1 nonblockinggrpcserver.go:90] "GRPC server started" logger="dra" endpoint="/var/lib/kubelet/plugins/gpu.nvidia.com/dra.sock"
I1030 08:15:50.593000       1 nonblockinggrpcserver.go:90] "GRPC server started" logger="registrar" endpoint="/var/lib/kubelet/plugins_registry/gpu.nvidia.com-reg.sock"
I1030 08:15:50.595998       1 resourceslicecontroller.go:528] "Starting ResourceSlice informer and waiting for it to sync" logger="ResourceSlice controller"
I1030 08:15:50.596012       1 health.go:100] starting healthcheck service at [::]:51516
I1030 08:15:50.596092       1 reflector.go:358] "Starting reflector" type="*v1.ResourceSlice" resyncPeriod="0s" reflector="k8s.io/dynamic-resource-allocation/resourceslice/resourceslicecontroller.go:534"
I1030 08:15:50.596110       1 reflector.go:404] "Listing and watching" type="*v1.ResourceSlice" reflector="k8s.io/dynamic-resource-allocation/resourceslice/resourceslicecontroller.go:534"
I1030 08:15:50.602545       1 reflector.go:436] "Caches populated" type="*v1.ResourceSlice" reflector="k8s.io/dynamic-resource-allocation/resourceslice/resourceslicecontroller.go:534"
I1030 08:15:51.596801       1 resourceslicecontroller.go:543] "ResourceSlice informer has synced" logger="ResourceSlice controller"
I1030 08:15:51.596851       1 resourceslicecontroller.go:184] "Starting" logger="ResourceSlice controller"

Three more points, if you allow :)

Sure, let's go!

1. What's your involvement with Flatcar container Linux?

I am working for Giant Swarm and our Kubernetes distribution is based on Flatcar Linux. You didn't ask this, but since I'm also a maintainer I feel like I need to clarify this, since it's a thought that often comes to my mind when people are creating pull requests to Ingress NGINX: I think I could have addressed this issue at Flatcar, specifically in their sysext-bakery project where the binaries and drivers we are using most likely come from, but since other Linux distributions could also use /opt for placing binaries and/or libraries, it seemed more sustainable to me to change this on your end.

Overall we are currently trying to get our product compliant to the Kubernetes AI Conformance Tests. In Kubernetes v1.34+ these require the DRA feature to work and currently I am testing the NVIDA DRA Driver on a Kubernetes v1.33 cluster on AWS, specifically g4dn instances with Tesla T4 Tensor GPUs, with the DRA Feature Gate enabled.

2. A curious question I have: where is libnvidia-ml.so.1 placed in your case?

I think you can see this from the kublet plugin pre-start script logs above: /driver-root/usr/lib64/libnvidia-ml.so.1 in the container or /usr/lib64/libnvidia-ml.so.1 on the host.

Interestingly the libraries get provided as real files. When the NVIDIA runtime sysext in Flatcar is enabled, it is mounting them directly to /usr/lib and /opt/nvidia/current/usr/lib, so no symlinks. The only file being provided as a symlink is the nvidia-smi binary and this is then pointing to /opt/bin/nvidia-smi.

3. Can you maybe change the commit message to kubelet plugins: add /opt/bin to binary search paths?

Sure, will do so in a second!

As said above, I will now also try the Compute Domains plugin, but I cannot promise it's working in our use case. Also I will try to get some actual workload allocated to the instances with the DRA Driver plugin enabled.

Marco

[Gacko]

I changed the commit message.

[Gacko]

I just tested the Compute Domains kubelet plugin: It's basically coming up, but fails to find nvidia-caps-imex-channels device in /proc/devices, most likely because the instance type I am using isn't supporting it.

So best I can do for now is this:

I1030 12:58:33.463792       1 utils.go:43] 
Feature gates: map[string]bool{"AllAlpha":false, "AllBeta":false, "ContextualLogging":true, "IMEXDaemonsWithDNSNames":true, "LoggingAlphaOptions":false, "LoggingBetaOptions":true, "MPSSupport":false, "TimeSlicingSettings":false}
Verbosity: 4
Flags: (*main.Flags)({
  kubeClientConfig: (flags.KubeClientConfig) {
    KubeConfig: (string) "",
    KubeAPIQPS: (float64) 5,
    KubeAPIBurst: (int) 10
  },
  nodeName: (string) (len=41) "ip-10-0-103-42.eu-west-2.compute.internal",
  namespace: (string) (len=11) "kube-system",
  cdiRoot: (string) (len=12) "/var/run/cdi",
  containerDriverRoot: (string) (len=12) "/driver-root",
  hostDriverRoot: (string) (len=1) "/",
  nvidiaCDIHookPath: (string) "",
  kubeletRegistrarDirectoryPath: (string) (len=33) "/var/lib/kubelet/plugins_registry",
  kubeletPluginsDirectoryPath: (string) (len=24) "/var/lib/kubelet/plugins",
  healthcheckPort: (int) 51515
})
I1030 12:58:33.464180       1 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I1030 12:58:33.464197       1 envvar.go:172] "Feature gate default state" feature="InOrderInformers" enabled=true
I1030 12:58:33.464201       1 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I1030 12:58:33.464205       1 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I1030 12:58:33.464208       1 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I1030 12:58:33.473269       1 main.go:181] shutdown
Error: error creating driver: failed to create device library: error getting nvcap for IMEX channel '0': error getting device major: error parsing '/proc/devices': unexpected regex match: []

[Gacko]

It's currently failing at this point: https://github.com/NVIDIA/k8s-dra-driver-gpu/blob/main/cmd/compute-domain-kubelet-plugin/nvlib.go#L102.

The Compute Domains kubelet plugin already loaded the driver and found the nvidia-smi binary. So we should be good even if I couldn't test it beyond that.

[Gacko]

I just tested the Compute Domains kubelet plugin: It's basically coming up, but fails to find nvidia-caps-imex-channels device in /proc/devices, most likely because the instance type I am using isn't supporting it.

Ok, after some reading I think I got it. It's just not possible to implement this on AWS with normal instance types. Took me some time to understand the IMEX channels, but from what I understood these require the GPUs to be physically connected and configured, which is clearly not possible on AWS. Or at least only with their specific p6e-gb200 instance type.

[jgehrcke]

Overall we are currently trying to get our product compliant to the Kubernetes AI Conformance Tests. In Kubernetes v1.34+ these require the DRA feature to work and currently I am testing the NVIDA DRA Driver on a Kubernetes v1.33 cluster on AWS, specifically g4dn instances with Tesla T4 Tensor GPUs, with the DRA Feature Gate enabled.

Lovely. Thank you for the background!

Interestingly the libraries get provided as real files. When the NVIDIA runtime sysext in Flatcar is enabled, it is mounting them directly to /usr/lib and /opt/nvidia/current/usr/lib, so no symlinks. The only file being provided as a symlink is the nvidia-smi binary and this is then pointing to /opt/bin/nvidia-smi.

Well, as you said -- this may also be worth changing over there. But whatever is least friction here I think is a great path forward.

It took me some time to get the GitHub Actions running in my fork since you hard-coded NVIDIA in some places,

Also the GPU kubelet plugin I am currently using is coming up fine

I will now also try the Compute Domains plugin, but I cannot promise it's working in our use case.

Oh wow, you really went through a bunch of effort :-). I did not mean to imply that you have to do all these things. I was merely trying to ask what you did in terms of verification / testing, so that I can better understand what we still have to do here. But big thank you for trying to do a bunch of verification work. You seemingly also have learned some things along the way about the funky intrinsics of our DRA driver here, the (somewhat) lacking purpose of our CI, and the fact that (yes) meaningful ComputeDomain plugin testing can only be done in environments that actually support multi-node NVLink :)

I'd say -- given the effort that went into this, I'd just love to a head and merge this now.

I still want to understand if we have potential security challenges down the line by generally (always) looking into /opt. Maybe @elezar has an opinion here.

[jgehrcke]

LGTM. Thanks Marco for your great communication around all those details here.

I hope this can stay in the code base in the longer run. We will have to see if this should maybe be back-ported to the 25.8.x branch.

[Gacko]

Thanks for merging this - and also considering back-porting!

Well, as you said -- this may also be worth changing over there. But whatever is least friction here I think is a great path forward.

Actually I am currently investigating this and trying to get a deeper understanding. For now I learnt that Flatcar is using systemd-sysext to provide additional components via mounting them from images or directories as overlay filesystem into /usr and so on.

Some like Docker or containerd are pre-installed and just need to get mounted. Others can be enabled via config and then get downloaded at boot time. And that's already the issue: If you would enable the officially provided NVIDIA Drivers sysext, they would get downloaded and mounted to /usr on boot - but to each and every instance, since there is no "is this instance having GPUs?" logic.

But: This logic actually exists in their nvidia systemd service. This service is capable of detecting an NVIDIA GPU at runtime and only then downloading the drivers. But sadly it places the nvidia-smi binary in /opt/bin and the drivers under /opt/nvidia/current/usr/lib64, where current is a symlink to actual driver directory (e.g. "535.230.02/6.6.106-flatcar", so /opt/nvidia/535.230.02/6.6.106-flatcar).

At this point neither the drivers nor the nvidia-smi binary is available in /usr. So the dynamic driver installation I described above puts everything to /opt.

But where do the libraries and the symlink I see in /usr come from? Well, that's tricky... we are also using the NVIDIA Runtime sysext. And this comes with this little file. This script is basically responsible for creating the symlink I see and mounting the whole /opt/nvidia/current as an overlay filesystem to /, with which it provides everything in /opt/nvidia/current/usr also to /usr. 🤦

tl;dr:

1. With just the Flatcar NVIDIA Drivers sysext enabled we would not need this change here, because the sysext already mounts everything to /usr. The nvidia-smi would be a real file and not just a symlink. Downside: Each and every instance, independent of it having a GPU or not, always downloads and installs the drivers, meaning increased boot time for downloading and installing the sysext and increased disk space usage (and maybe security risk?) for always having the drivers and binaries installed.
2. With just the dynamically and at boot time installed drivers (if their is a GPU), we can save bandwidth and disk space, but get the binary placed to just /opt/bin (no symlink) and the libraries placed to /opt/nvidia/current/usr/lib64. In this case my change would detect the binary, but fail to detect the library as it only exists in /opt/nvidia/current/usr/lib64 which isn't covered by the current search logic. And changing the driver root to /opt/nvidia/current via Helm values would enable the plugin to find the libraries, but no longer the binaries.
3. So last but not least my change only helps when using the Flatcar NVIDIA Runtime sysext in conjunction with the dynamically downloaded drivers as the former makes sure everything is available either in /opt/bin or /usr/lib64 (see the script I linked).

What does this mean to this PR?

Basically I think it's not a bad idea to have /opt/bin on the search path, but I agree it only fixes an edge case. Also I agree it should not be required, especially for security reasons.

• People using the Flatcar NVIDIA Drivers sysext (no. 1 in the listing) do not have issues anyway - with or without this change - even though they might waste bandwidth and disk.
• People using the dynamically installed drivers do not waste bandwidth or disk, but also this change does not help them. They most likely don't get the DRA plugin up and running, no matter what they set the driver root to.
• People using the dynamically installed drivers AND the Flatcar NVIDIA Runtime sysext are lucky, because my change makes the plugin work for them.

[Gacko]

Ok, wow... I just wasted like two hours or so. The Flatcar NVIDIA Drivers sysext is not yet available on their stable release channel and I was trying to enable it on... stable.

So ultimately the first options of my listings are not applicable and therefore this change here seems to be the only way to make it work. 😅