istio.io/service/server/response_latencies metric that HPA collected is different to Cloud monitoring

[nguyen-viet-hung]

Dear team,

I have an issue that the metric istio.io/service/server/response_latencies that HPA collected is different to the value I see on cloud monitoring. I have enabled Anthos service mesh on the cluster, and installed custom-metrics-stackdriver-adapter on the gke cluster. Already set enable-distribution-support=true for deployment custom-metrics-stackdriver-adapter to enable getting distribution metrics like istio.io/service/server/response_latencies. but the metric of istio.io/service/server/response_latencies that HPA collected is too high compares to that metric show on the cloud monitoring (for example: 6144s compares to 3.12s). Here is the information:

• GKE version: v1.24.10-gke.2300
• custom-metrics-stackdriver-adapter version: gcr.io/gke-release/custom-metrics-stackdriver-adapter:v0.13.1-gke.0
• custome-metric model: adapter_new_resource_model.yaml

HPA declaration:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  annotations:
  name: face-api-hpa
  namespace: ekyc
spec:
  maxReplicas: 1
  metrics:
  - external:
      metric:
        name: istio.io|service|server|response_latencies
        selector:
          matchLabels:
            metric.labels.destination_service_name: face-api
            reducer: REDUCE_PERCENTILE_50
      target:
        averageValue: "4"
        type: AverageValue
    type: External
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: face-api

The metrics that HPA collected as follow:

NAME                     REFERENCE                       TARGETS                           MINPODS   MAXPODS   REPLICAS   AGE
face-api-hpa             Deployment/face-api             12697600m/4 (avg)                 1         1         1          4d18h

The metric that show on monitoring:

Is there anything wrong here or I mis-understand something?

[CatherineF-dev]

Hi, is this issue resolved or not?