OpenCost metrics interfere with OpenShift's "degraded control plane" detection?

[kastl-ars]

Dear OpenCost maintainers,

since last week we noticed that our OpenShift cluster show a degradation warning, as only 50% of the apiservers are responding.

Turns out this seems to be related to metrics exposed by OpenCost, scraped by Prometheus and then returned by the query used for this degradation detection.

We have explictly disabled the emission of pod annotations, namespace annotations and ksm V1 metrics and the error vanished.

  opencost:
    metrics:
      serviceMonitor:
        enabled: true
      kubeStateMetrics:
        emitPodAnnotations: false
        emitNamespaceAnnotations: false
        emitKsmV1Metrics: false

The following lines appeared in the deployment:

        - name: EMIT_POD_ANNOTATIONS_METRIC
          value: 'false'
        - name: EMIT_NAMESPACE_ANNOTATIONS_METRIC
          value: 'false'
        - name: EMIT_KSM_V1_METRICS
          value: 'false'

I would like to see this added to the documentation that @mittal-ishaan was working on IIRC.

The query that went wrong was this:

count(kube_pod_labels{label_app="openshift-kube-apiserver", label_apiserver="true", namespace="openshift-kube-apiserver" })

Before we introduced the workaround described above, this returned 6 pods, while only three were really running. Hence the degradation warning as only 50% were working...

Kind Regards,
Johannes

[kastl-ars]

Hmmm, this seems to completely break any cost calculation in OpenCost. After setting this, there are no more metrics visible. I enabled the emitNamespaceAnnotations again, let's see if this changes something...

[kastl-ars]

Hmmm, this seems to completely break any cost calculation in OpenCost. After setting this, there are no more metrics visible. I enabled the emitNamespaceAnnotations again, let's see if this changes something...

Even after re-enabling emitNamespaceAnnotations (by removing setting the attribute to false in the values.yaml) and the emitPodAnnotations a little later I can no longer see any costs in OpenCost for the last couple of hours. Removing the disabling of the emitKsmV1Metrics makes OpenCost show values again almost instantaneously, but also the OpenShift degradation warning is back...

[kastl-ars]

As just stated in #252 I am not sure if this issue should rather go to the opencost repository, as it seems (to me, with the knowledge I have today...) like not just a problem of disabling some things on OpenShift, but a general problem of OpenCost not working on OpenShift without interfering with OpenShift itself?

[mittal-ishaan]

Hi @kastl-ars
I will look into this issue. Can you tell me the opencost version that you are using right now?

[kastl-ars]

Thank you! We are using the latest chart version 1.43.1.

[kastl-ars]

The more pressing issue would be #252 as a wrong CPU count sounds more problematic. But my guess is they are related...

[mittal-ishaan]

Thank you,
Sure. Let me check that too

[kastl-ars]

Any news on this?

[mittal-ishaan]

So sorry, did not get time for it this week. Checking right now

[mittal-ishaan]

I am able to reproduce this at my end.

count(kube_pod_labels{label_app="openshift-kube-apiserver", label_apiserver="true", namespace="openshift-kube-apiserver" })

is giving me a value 6 while there are only 3 with API Servers 50% degradation warning.
The workaround you suggested should be ideal in this case as then opencost job will not be pushing this metrics and we would be getting the correct count 3 in this case. I will look into why the cost is not visible when disabling emitKsmV1Metrics

[mittal-ishaan]

This also leads to number of CPUs being doubled in the overview page as stated in #252

[kastl-ars]

Thanks for looking into this, glad you could reproduce this!

[mittal-ishaan]

Hi @kastl-ars

One thing to point out here as I was looking into this further is that setting

opencost:
  metrics:
    kubecostMetrics:
      emitKsmV1Metrics: false

does not break anything and I am still able to view the cost in the UI. Can you confirm this if possible? I am not able to reproduce what you were experiencing here