Ability to add extra volumes and extra volume mounts to daemonsets

[chiragjn]

We would like to manage daemonsets like gpu-feature-discovery and dcgm-exporter on GKE via the gpu-operator where the driver is installed in a non standard location /home/kubernetes/bin/nvidia/

We do not want to use the device-plugin and container-toolkit via the gpu-operator because

• GKE NAP does not allow labelling nodes to disable the GKE's default device plugin
• Just enabling container-toolkit via the gpu-operator can cause race conditions where GKE's device plugin registers devices before gpu-operator's toolkit gets chance to run

Sadly this means we have to play by rules of GKE's runc which calls GKE's own nvidia-container-cli variant which does not inject devices, nvidia-smi and libnvml to gpu-feature-discovery and dcgm-exporter since these pods do not ask for any gpu devices using resources.limits

Ability to add volumes and mounts will allow to bypass this limitation by manually exposing devices, binaries and libraries

      volumes:
        - name: dev
          hostPath:
            path: /dev
            type: ''
        - name: nvidia-install-dir-host
          hostPath:
            path: /home/kubernetes/bin/nvidia
            type: ''
        - name: nvidia-config
          hostPath:
            path: /etc/nvidia
            type: ''
      containers:
        - name: nvidia-dcgm-exporter
          volumeMounts:
            - name: dev
              mountPath: /dev
            - name: nvidia-install-dir-host
              mountPath: /usr/local/nvidia
            - name: nvidia-config
              mountPath: /etc/nvidia
          env:
            - name: DCGM_EXPORTER_LISTEN
              value: ':9400'
            - name: DCGM_EXPORTER_KUBERNETES
              value: 'true'
            - name: DCGM_EXPORTER_COLLECTORS
              value: /etc/dcgm-exporter/dcp-metrics-included.csv
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName
            - name: DCGM_EXPORTER_KUBERNETES_GPU_ID_TYPE
              value: uid
            - name: NVIDIA_INSTALL_DIR_HOST
              value: /home/kubernetes/bin/nvidia
            - name: NVIDIA_INSTALL_DIR_CONTAINER
              value: /usr/local/nvidia
            - name: LD_LIBRARY_PATH
              value: >-
                /usr/local/nvidia/lib64:/usr/local/nvidia/lib:/usr/local/cuda/lib64

[cdesiniotis]

I am not familiar with GKE's node auto-provisioning (NAP). Is it not possible to configure NAP such that node-pools get created with custom labels? FWIW our official install procedure for GKE instructs users to disable GKE's device plugin by applying labels at node-pool creation time: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/google-gke.html

[chiragjn]

Is it not possible to configure NAP such that node-pools get created with custom labels

I have looked at docs and their cli configuration but didn't find any such options. I would happily take that route if possible.

our official install procedure for GKE instructs users to disable GKE's device plugin by applying labels at node-pool creation time: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/google-gke.html

Yes, I know about this, but we are not creating nodepools manually with GKE 😅 and rely mostly on NAP.
Fwiw, I will also open a ticket on GKE side, but my hopes for resolution there are not high

[chiragjn]

I will also open a ticket on GKE side, but my hopes for resolution there are not high

From Google's issue tracker, it is not supported at the moment:

• https://issuetracker.google.com/issues/419367293
• https://issuetracker.google.com/issues/419082683
• https://issuetracker.google.com/issues/392424839

[github-actions]

This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed. To skip these checks, apply the "lifecycle/frozen" label.

[guptaNswati]

@chiragjn i have labeled this as Feature Request. I am not sure when we can prioritize it so if its urgent for you I will encourage you to do a PR. Thanks

[chiragjn]

Thanks, this might not be a priority for us at the moment.

GKE now seems to have a way to add labels via ComputeClass but I haven't given it a shot yet

I am also not sure how this will play with CDI mode which is now the default. I'll get back once I get some time to spend on this.