stackdriver adapter fails to filter metrics using resource metadata labels

[juli4n]

The stackdriver adapter supports filtering time series via HPA’s LabelSelectors. Cloud Monitoring metrics have a data model where time series have two source of labels:

• Metric labels: These are labels that are automatically added to each value and augment it across different dimensions. For example, the Cloud Storage metric type storage.googleapis.com/api/request_count has two labels, response_code and method.
• Monitored Resource labels: These are labels which are not associated with the metric time series, but with the monitored resource associated with it (e.g. k8s_pod), and can be broken down into two:
  • Resource labels: These are the labels that unequivocally identify the resource. For example, for k8s_node this will be the tuple (project id, location, cluster name, node name).
  • Metadata labels: Which can be further categorized into system and user labels.
    • System: GKE automatically adds labels to k8s monitored resources
    • User: GKE automatically adds labels on the corresponding k8s resources as user metadata labels. For example, if a k8s Node resource has a label with key beta.kubernetes.io/instance-type, metrics associated with k8s_node resources will have that label defined as a user label.

When querying time series using the Cloud Monitoring API, filters for labels must be provided using a dedicated prefix for each label type: "metric.labels", "resource.labels", "metadata.system_labels", "metadata.user_labels". HPA supports this today by mapping HPA’s label selectors 1:1 to requests to the CM API.

For example, if a user wants HPA to scale based on an external metric, but filtering on a specific user label, they can do:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: External
    external:
      metric:
        name: kubernetes.io|node|memory|allocatable_utilization
        selector:
          matchLabels:
            metadata.user_labels.mylabel: <my-label-value>
      target:
        type: AverageValue
        averageValue: 600m

The issue arises with labels that have a prefix/name format. The prefix is optional and if provided it must be a valid DNS sub-domain name.

If a user metadata label uses this format, for example cloud.google.com/gke-nodepool, the filter would need to be encoded in HPA’s label selector as: metadata.user_labels.mylabel: <my-label-value> which is not a valid label (the prefix is not a valid DNS sub-domain name as it contains an underscore character).

Note that this applies to both system and user labels.

Workaround

As a workaround, users can add their own labels to the k8s resources associated with the metric using label keys that don’t have a prefix (e.g. gke-nodepool instead of cloud.google.com/gke-nodepool. Then the resulting label key in the selector will be valid.

Possible Fix

We could introduce another way to filter by metadata labels by allowing two new prefixes (e.g. metadata.system-labels and metadata.user-labels). The adapter will need to special-case the handling of these labels and translate them into the prefix that Cloud Monitoring API expects.