DCGM_FI_DEV_GPU_UTIL metric giving empty value from prometheus

[Vijaygawate]

I have deployed GPU operator on EKS cluster using helm chart, also I have deployed prometheus, but when I am querying GPu utilization metric DCGM_FI_DEV_GPU_UTIL it is returning empty even when I have 27% GPU utilization

curl localhost:9090/api/v1/query?query=DCGM_FI_DEV_GPU_UTIL

output:
{"status":"success","data":{"resultType":"vector","result":[]}}

[github-actions]

This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed. To skip these checks, apply the "lifecycle/frozen" label.

[shivamerla]

@Vijaygawate are these GPUs configured with MIG? Please check these issues (#614 (comment), #798) if they apply to your environment.

[rahulait]

@Vijaygawate if prometheus is not configured to scrape metrics from dcgm-exporter, you'll get such empty responses.

Can you share your prometheus scrape_config used to scrape metrics from dcgm-exporter? This looks like prometheus configuration issue where if there is no configuration to scrape dcgm-exporter endpoints, it will show such output.

You can check metrics published by dcgm-exporter by querying its metrics endpoint. For example:

curl -fsSL <dcgm-exporter-endpoint>:9400/metrics | grep -i DCGM_FI_DEV_GPU_UTIL
# HELP DCGM_FI_DEV_GPU_UTIL GPU utilization (in %).
# TYPE DCGM_FI_DEV_GPU_UTIL gauge
DCGM_FI_DEV_GPU_UTIL{gpu="0",UUID="GPU-15744c16-49d5-e70d-196e-be8e538ce214",pci_bus_id="00000000:2D:00.0",device="nvidia0",modelName="NVIDIA RTX A4000",Hostname="abc-def-aaaa",DCGM_FI_DRIVER_VERSION="580.95.05"} 0

Here is sample prometheus scrape config one can use. Make sure namespace is updated in targets if its running in a different namespace:

    ...
    scrape_configs:
      ...

      - job_name: 'dcgm'
        static_configs:
          - targets: ['nvidia-dcgm-exporter.gpu-operator.svc.cluster.local:9400']

DCGM Exporter emits exemplars by default. Prometheus drops metrics that contain exemplars if exemplar storage is disabled. Make sure to start your prometheus server with examplar-storage enabled. You can set the following flag for prometheus-server:

     - --enable-feature=exemplar-storage

Once everything is set correctly, you should be able to see the results in the query you've posted above. Example:

curl <prometheus-endpoint>:9090/api/v1/query?query=DCGM_FI_DEV_GPU_UTIL
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"DCGM_FI_DRIVER_VERSION":"580.95.05","Hostname":"abc-def-aaaa","UUID":"GPU-15744c16-49d5-e70d-196e-be8e538ce214","__name__":"DCGM_FI_DEV_GPU_UTIL","device":"nvidia0","gpu":"0","instance":"nvidia-dcgm-exporter.gpu-operator.svc.cluster.local:9400","job":"dcgm","modelName":"NVIDIA RTX A4000","pci_bus_id":"00000000:2D:00.0"},"value":[1763413015.035,"0"]}]}}

I am going to close this issue for now as its mostly a prometheus configuration issue. If you still see this issue even after configuring correct settings, please feel free to re-open it so that someone from the team can take a look.