Expose per Pod GPU metrics collected by dcgm-exporter to include DRA information

[guptaNswati]

To start adding DRA information. Follow these steps.

Step 1: Update to k8s 1.32 and enable DRA and PodResourcesDRA feature gates. These are necessary to add DRA information to the podresources api. Refer kubernetes/enhancements#3695

# To enable with kubeadm
$ cat kubeadm-init.yaml
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
apiServer:
  extraArgs:
  - name: "feature-gates"
    value: "DynamicResourceAllocation=true"
  - name: "runtime-config"
    value: "resource.k8s.io/v1beta1=true"
controllerManager:
  extraArgs:
  - name: "feature-gates"
    value: "DynamicResourceAllocation=true"
scheduler:
  extraArgs:
  - name: "feature-gates"
    value: "DynamicResourceAllocation=true"
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
featureGates:
  DynamicResourceAllocation: true
  KubeletPodResourcesDynamicResources: true
  KubeletPodResourcesGet: true

Step 2: Disable k8s-device-plugin. And install nvidia-k8s-dra-driver with --set gpuResourcesEnabledOverride=true option to enable gpu allocation. Follow the official guide or this for multinode testing #249

Step 3: Install dcgm-exporter version DCGM-Exporter 4.3.1-4.4.0 with --set kubernetesDRA.enabled=true option.

If running with gpu-operator, manually create the clusterrole and bindings to allow dcgm-exporter to access resourceslices and add this env var to the exporter pod:

env:
 name: KUBERNETES_ENABLE_DRA
      value: "true"

Step 4: Deploy an example gpu pod DRA style

cat gpu-test1.yaml 
---
apiVersion: v1
kind: Namespace
metadata:
name: gpu-test1

---
apiVersion: resource.k8s.io/v1beta1
kind: ResourceClaimTemplate
metadata:
namespace: gpu-test1
name: single-gpu
spec:
spec:
  devices:
    requests:
    - name: gpu
      deviceClassName: gpu.nvidia.com

---
apiVersion: v1
kind: Pod
metadata:
namespace: gpu-test1
name: pod1
labels:
  app: pod
spec:
containers:
- name: ctr
  image: ubuntu:22.04
  command: ["bash", "-c"]
  args: ["nvidia-smi -L; trap 'exit 0' TERM; sleep 9999 & wait"]
  resources:
    claims:
    - name: gpu
resourceClaims:
- name: gpu
  resourceClaimTemplateName: single-gpu
tolerations:
- key: "nvidia.com/gpu"
  operator: "Exists"
  effect: "NoSchedule"

Step 5: Follow dcgm-exporter instructions to see the metrics

$ kubectl port-forward  dcgm-exporter-pod-name 8080:9400 &
$ curl -sL http://127.0.0.1:8080/metrics