It is mainly for metering and diagnosis purpose. We'd like to monitor the usage and the health status of each GPU. Having UUIDs in node label can help us to search data in prometheus.

We don't have a use case to select a particular GPU right now. I guess that could be useful if there are multiple GPUs on a node, but models are not exactly the same?

In my case, we need to schedule pod to specific gpu . we has a map record all gpus(uuid), assign gpu to different job.

we run pod with env as below to make sure pod use specific gpu.

         env:
            - name: NVIDIA_VISIBLE_DEVICES
              value: GPU-7dfa7b70-34fb-bcce-e2df-130ebb7fd047

However, if the pod is scheduled to a host that does not own the specified GPU, the pod will fail to run and return an error as follows:

Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running prestart hook #0: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'\nnvidia-container-cli: device error: GPU-7dfa7b70-34fb-bcce-e2df-130ebb7fd047: unknown device: unknown

Therefore, we must record the mapping between node names and GPU UUIDs. When running a pod, both the node name label and the GPU UUID environment variable should be specified. It would be even better if the GPU UUID could be directly provided from the node label.

Add gpu uuids to node labels #1015

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions