[BUG] This Metrics Config Wipes all non gotk_ metrics from the kube-state-metrics exporter

[Routhinator]

When the changes in this PR were marked "Good" - were they marked good because the GOTK metrics were being produced - or were they marked good because the GOTK metrics and all default kube-state-metrics worked?

Because following the current state of the config at the latest HEAD breaks all default kube-state-metrics and leaves only gotk_ metrics exported by kube-state-metrics?

Are we expected to run two copies of kube-state-metrics here? Is this missed testing? Or has some other change hit kube-state-metrics?

With this config this is what comes out of kube-state-metrics:

-> % curl localhost:8080/metrics                                                                                                                  (⎈|kubernetes-admin@red-squadron:default) 
# HELP gotk_resource_info The current state of a Flux HelmRelease resource.
# TYPE gotk_resource_info gauge
gotk_resource_info{chart_app_version="1.12.1",chart_name="ingress-nginx",chart_source_name="ingress-nginx",customresource_group="helm.toolkit.fluxcd.io",customresource_kind="HelmRelease",customresource_version="v2",exported_namespace="ingress-nginx",name="external-ingress-nginx",ready="True",revision="4.12.1"} 1
gotk_resource_info{chart_app_version="1.17.1",chart_name="cilium",chart_source_name="cilium",customresource_group="helm.toolkit.fluxcd.io",customresource_kind="HelmRelease",customresource_version="v2",exported_namespace="kube-system",name="cilium",ready="True",revision="1.17.1"} 1
gotk_resource_info{chart_app_version="3.0.0",chart_name="promtail",chart_source_name="grafana",customresource_group="helm.toolkit.fluxcd.io",customresource_kind="HelmRelease",customresource_version="v2",exported_namespace="monitoring",name="promtail",ready="True",revision="6.16.6"} 1
gotk_resource_info{chart_app_version="v0.17.3",chart_name="node-feature-discovery",chart_source_name="nfd",customresource_group="helm.toolkit.fluxcd.io",customresource_kind="HelmRelease",customresource_version="v2",exported_namespace="node-feature-discovery",name="node-feature-discovery",ready="True",revision="0.17.3"} 1
gotk_resource_info{chart_app_version="1.0",chart_name="democratic-csi",chart_source_name="democratic-csi",customresource_group="helm.toolkit.fluxcd.io",customresource_kind="HelmRelease",customresource_version="v2",exported_namespace="storage-deployments",name="truenas-iscsi",ready="True",revision="0.15.0"} 1
gotk_resource_info{chart_app_version="v1.16.2",chart_name="cert-manager",chart_source_name="jetstack",customresource_group="helm.toolkit.fluxcd.io",customresource_kind="HelmRelease",customresource_version="v2",exported_namespace="cert-manager",name="cert-manager",ready="True",revision="v1.16.2"} 1
gotk_resource_info{chart_name="importer",chart_source_name="firefly-iii",customresource_group="helm.toolkit.fluxcd.io",customresource_kind="HelmRelease",customresource_version="v2",exported_namespace="firefly-iii",name="firefly-iii-importer",ready="True",revision="1.4.0"} 1
gotk_resource_info{chart_app_version="24.04.12.3.1",chart_name="collabora-online",chart_source_name="collabora-online",customresource_group="helm.toolkit.fluxcd.io",customresource_kind="HelmRelease",customresource_version="v2",exported_namespace="collabora",name="collabora-online",ready="True",revision="1.1.33"} 1
gotk_resource_info{chart_app_version="1.0",chart_name="democratic-csi",chart_source_name="democratic-csi",customresource_group="helm.toolkit.fluxcd.io",customresource_kind="HelmRelease",customresource_version="v2",exported_namespace="storage-deployments",name="truenas-manual",ready="True",revision="0.15.0"} 1
gotk_resource_info{chart_app_version="v0.85.0",chart_name="kube-prometheus-stack",chart_source_name="prometheus-community",customresource_group="helm.toolkit.fluxcd.io",customresource_kind="HelmRelease",customresource_version="v2",exported_namespace="monitoring",name="kube-prometheus-stack",ready="True",revision="77.5.0"} 1
gotk_resource_info{chart_app_version="7.0.0",chart_name="forgejo-runner-podman",chart_source_name="routhca-forgejo-oci-helm",customresource_group="helm.toolkit.fluxcd.io",customresource_kind="HelmRelease",customresource_version="v2",exported_namespace="forgejo-act-runners",name="forgejo-runner-podman",ready="True",revision="0.3.70"} 1
gotk_resource_info{chart_app_version="v1.2.0",chart_name="reloader",chart_source_name="stakater-charts",customresource_group="helm.toolkit.fluxcd.io",customresource_kind="HelmRelease",customresource_version="v2",exported_namespace="deployment-services",name="stakater-reloader",ready="True",revision="1.2.0"} 1
gotk_resource_info{chart_app_version="v25.3.1",chart_name="gpu-operator",chart_source_name="nvidia",customresource_group="helm.toolkit.fluxcd.io",customresource_kind="HelmRelease",customresource_version="v2",exported_namespace="device-plugins",name="nvidia-gpu-operator",ready="True",revision="v25.3.1"} 1
gotk_resource_info{chart_app_version="3.4.2",chart_name="loki",chart_source_name="grafana",customresource_group="helm.toolkit.fluxcd.io",customresource_kind="HelmRelease",customresource_version="v2",exported_namespace="monitoring",name="loki",ready="True",revision="6.27.0"} 1
gotk_resource_info{chart_app_version="1.25.0",chart_name="cloudnative-pg",chart_source_name="cloudnative-pg",customresource_group="helm.toolkit.fluxcd.io",customresource_kind="HelmRelease",customresource_version="v2",exported_namespace="databases",name="cloudnative-pg",ready="True",revision="0.23.0"} 1
gotk_resource_info{chart_app_version="7.2.5",chart_name="redis",chart_source_name="bitnami-charts",customresource_group="helm.toolkit.fluxcd.io",customresource_kind="HelmRelease",customresource_version="v2",exported_namespace="nextcloud",name="nextcloud-redis",ready="True",revision="19.6.4"} 1
gotk_resource_info{chart_app_version="0.17.0",chart_name="external-dns",chart_source_name="external-dns",customresource_group="helm.toolkit.fluxcd.io",customresource_kind="HelmRelease",customresource_version="v2",exported_namespace="external-dns",name="external-dns-pi-hole",ready="True",revision="1.17.0"} 1
gotk_resource_info{chart_app_version="0.31.1",chart_name="intel-device-plugins-operator",chart_source_name="intel",customresource_group="helm.toolkit.fluxcd.io",customresource_kind="HelmRelease",customresource_version="v2",exported_namespace="device-plugins",name="intel-device-plugins-operator",ready="True",revision="0.31.1"} 1
gotk_resource_info{chart_app_version="0.31.1",chart_name="intel-device-plugins-gpu",chart_source_name="intel",customresource_group="helm.toolkit.fluxcd.io",customresource_kind="HelmRelease",customresource_version="v2",exported_namespace="device-plugins",name="intel-device-plugins-gpu",ready="True",revision="0.31.1"} 1
gotk_resource_info{chart_app_version="1.7.0",chart_name="kubernetes-event-exporter",chart_source_name="routhca-forgejo-oci-helm",customresource_group="helm.toolkit.fluxcd.io",customresource_kind="HelmRelease",customresource_version="v2",exported_namespace="monitoring",name="kubernetes-event-exporter",ready="True",revision="1.7.1"} 1
gotk_resource_info{chart_app_version="1.0",chart_name="democratic-csi",chart_source_name="democratic-csi",customresource_group="helm.toolkit.fluxcd.io",customresource_kind="HelmRelease",customresource_version="v2",exported_namespace="storage-deployments",name="truenas-nfs",ready="True",revision="0.15.0"} 1
gotk_resource_info{chart_app_version="1.12.1",chart_name="ingress-nginx",chart_source_name="ingress-nginx",customresource_group="helm.toolkit.fluxcd.io",customresource_kind="HelmRelease",customresource_version="v2",exported_namespace="ingress-nginx",name="ingress-nginx",ready="True",revision="4.12.1"} 1
# HELP gotk_resource_info The current state of a Flux Kustomization resource.
# TYPE gotk_resource_info gauge
gotk_resource_info{customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1",exported_namespace="flux-system",name="media-server",ready="True",revision="v0.14.14@sha1:2b7e71aa48df7f6d4fa6e6fbe470825ab2a4cc1d",source_name="media-server"} 1
gotk_resource_info{customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1",exported_namespace="flux-system",name="nextcloud",ready="True",revision="v0.13.1@sha1:13812ca0c4903614b7a7d28c951ffb0304c92aa3",source_name="nextcloud"} 1
gotk_resource_info{customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1",exported_namespace="cert-manager",name="cert-manager",ready="True",revision="main@sha1:4643c701076024bcbe51736d7d0d2d1e4a7dfcbb",source_name="flux-system"} 1
gotk_resource_info{customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1",exported_namespace="flux-system",name="flux-system",ready="True",revision="main@sha1:4643c701076024bcbe51736d7d0d2d1e4a7dfcbb",source_name="flux-system"} 1
gotk_resource_info{customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1",exported_namespace="flux-system",name="firefly-iii",ready="True",revision="v0.1.14@sha1:6a2a236ab5230985547b7948507763663d6e0a3e",source_name="firefly-iii"} 1
gotk_resource_info{customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1",exported_namespace="flux-system",name="infrastructure",ready="True",revision="main@sha1:4643c701076024bcbe51736d7d0d2d1e4a7dfcbb",source_name="flux-system"} 1
gotk_resource_info{customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1",exported_namespace="cluster-conf",name="pull-secrets",ready="True",revision="main@sha1:4643c701076024bcbe51736d7d0d2d1e4a7dfcbb",source_name="flux-system"} 1
gotk_resource_info{customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1",exported_namespace="flux-system",name="luanti",ready="True",revision="v0.2.1@sha1:4ae4edb20d762d82b7190c86a5637f601c36a709",source_name="luanti"} 1
gotk_resource_info{customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1",exported_namespace="cluster-conf",name="certificates",ready="True",revision="main@sha1:4643c701076024bcbe51736d7d0d2d1e4a7dfcbb",source_name="flux-system"} 1
# HELP gotk_resource_info The current state of a Flux GitRepository resource.
# TYPE gotk_resource_info gauge
gotk_resource_info{customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1",exported_namespace="flux-system",name="flux-system",ready="True",revision="main@sha1:4643c701076024bcbe51736d7d0d2d1e4a7dfcbb",url="ssh://git@forgejo.home.routh.ca/routh-ca/cluster-config"} 1
gotk_resource_info{customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1",exported_namespace="flux-system",name="nextcloud",ready="True",revision="v0.13.1@sha1:13812ca0c4903614b7a7d28c951ffb0304c92aa3",url="ssh://git@forgejo.home.routh.ca/routh-ca/nextcloud"} 1
gotk_resource_info{customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1",exported_namespace="flux-system",name="firefly-iii",ready="True",revision="v0.1.14@sha1:6a2a236ab5230985547b7948507763663d6e0a3e",url="ssh://git@forgejo.home.routh.ca/routh-ca/firefly-iii"} 1
gotk_resource_info{customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1",exported_namespace="flux-system",name="media-server",ready="True",revision="v0.14.14@sha1:2b7e71aa48df7f6d4fa6e6fbe470825ab2a4cc1d",url="ssh://git@forgejo.home.routh.ca/routh-ca/media-server"} 1
gotk_resource_info{customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1",exported_namespace="flux-system",name="forgejo-act-runners",ready="True",revision="main@sha1:4643c701076024bcbe51736d7d0d2d1e4a7dfcbb",url="ssh://git@forgejo.home.routh.ca/routh-ca/cluster-config"} 1
gotk_resource_info{customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1",exported_namespace="flux-system",name="luanti",ready="True",revision="v0.2.1@sha1:4ae4edb20d762d82b7190c86a5637f601c36a709",url="ssh://git@forgejo.home.routh.ca/routh-ca/luanti"} 1
# HELP gotk_resource_info The current state of a Flux HelmChart resource.
# TYPE gotk_resource_info gauge
gotk_resource_info{chart_name="ingress-nginx",chart_version="4.12.1",customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmChart",customresource_version="v1",exported_namespace="cluster-conf",name="ingress-nginx-ingress-nginx",ready="True",revision="4.12.1"} 1
gotk_resource_info{chart_name="gpu-operator",chart_version="v25.3.1",customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmChart",customresource_version="v1",exported_namespace="cluster-conf",name="device-plugins-nvidia-gpu-operator",ready="True",revision="v25.3.1"} 1
gotk_resource_info{chart_name="forgejo-runner-podman",chart_version="0.3.70",customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmChart",customresource_version="v1",exported_namespace="cluster-conf",name="forgejo-act-runners-forgejo-runner-podman",ready="True",revision="0.3.70"} 1
gotk_resource_info{chart_name="reloader",chart_version="v1.2.0",customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmChart",customresource_version="v1",exported_namespace="cluster-conf",name="deployment-services-stakater-reloader",ready="True",revision="1.2.0"} 1
gotk_resource_info{chart_name="kube-prometheus-stack",chart_version="77.5.0",customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmChart",customresource_version="v1",exported_namespace="cluster-conf",name="monitoring-kube-prometheus-stack",ready="True",revision="77.5.0"} 1
gotk_resource_info{chart_name="cloudnative-pg",chart_version="v0.23.0",customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmChart",customresource_version="v1",exported_namespace="cluster-conf",name="databases-cloudnative-pg",ready="True",revision="0.23.0"} 1
gotk_resource_info{chart_name="cert-manager",chart_version="1.16.2",customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmChart",customresource_version="v1",exported_namespace="cluster-conf",name="cert-manager-cert-manager",ready="True",revision="v1.16.2"} 1
gotk_resource_info{chart_name="democratic-csi",chart_version="0.15.0",customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmChart",customresource_version="v1",exported_namespace="cluster-conf",name="storage-deployments-truenas-nfs",ready="True",revision="0.15.0"} 1
gotk_resource_info{chart_name="intel-device-plugins-operator",chart_version="v0.31.1",customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmChart",customresource_version="v1",exported_namespace="cluster-conf",name="device-plugins-intel-device-plugins-operator",ready="True",revision="0.31.1"} 1
gotk_resource_info{chart_name="loki",chart_version="6.27.0",customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmChart",customresource_version="v1",exported_namespace="cluster-conf",name="monitoring-loki",ready="True",revision="6.27.0"} 1
gotk_resource_info{chart_name="cilium",chart_version="1.17.1",customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmChart",customresource_version="v1",exported_namespace="cluster-conf",name="kube-system-cilium",ready="True",revision="1.17.1"} 1
gotk_resource_info{chart_name="democratic-csi",chart_version="0.15.0",customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmChart",customresource_version="v1",exported_namespace="cluster-conf",name="storage-deployments-truenas-manual",ready="True",revision="0.15.0"} 1
gotk_resource_info{chart_name="redis",chart_version="19.6.4",customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmChart",customresource_version="v1",exported_namespace="cluster-conf",name="nextcloud-nextcloud-redis",ready="True",revision="19.6.4"} 1
gotk_resource_info{chart_name="promtail",chart_version="6.16.6",customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmChart",customresource_version="v1",exported_namespace="cluster-conf",name="monitoring-promtail",ready="True",revision="6.16.6"} 1
gotk_resource_info{chart_name="importer",chart_version="1.4.0",customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmChart",customresource_version="v1",exported_namespace="cluster-conf",name="firefly-iii-firefly-iii-importer",ready="True",revision="1.4.0"} 1
gotk_resource_info{chart_name="external-dns",chart_version="1.17.0",customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmChart",customresource_version="v1",exported_namespace="cluster-conf",name="external-dns-external-dns-pi-hole",ready="True",revision="1.17.0"} 1
gotk_resource_info{chart_name="intel-device-plugins-gpu",chart_version="v0.31.1",customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmChart",customresource_version="v1",exported_namespace="cluster-conf",name="device-plugins-intel-device-plugins-gpu",ready="True",revision="0.31.1"} 1
gotk_resource_info{chart_name="kubernetes-event-exporter",chart_version="1.7.1",customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmChart",customresource_version="v1",exported_namespace="cluster-conf",name="monitoring-kubernetes-event-exporter",ready="True",revision="1.7.1"} 1
gotk_resource_info{chart_name="democratic-csi",chart_version="0.15.0",customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmChart",customresource_version="v1",exported_namespace="cluster-conf",name="storage-deployments-truenas-iscsi",ready="True",revision="0.15.0"} 1
gotk_resource_info{chart_name="node-feature-discovery",chart_version="0.17.3",customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmChart",customresource_version="v1",exported_namespace="cluster-conf",name="node-feature-discovery-node-feature-discovery",ready="True",revision="0.17.3"} 1
gotk_resource_info{chart_name="ingress-nginx",chart_version="4.12.1",customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmChart",customresource_version="v1",exported_namespace="cluster-conf",name="ingress-nginx-external-ingress-nginx",ready="True",revision="4.12.1"} 1
gotk_resource_info{chart_name="collabora-online",chart_version="1.1.33",customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmChart",customresource_version="v1",exported_namespace="cluster-conf",name="collabora-collabora-online",ready="True",revision="1.1.33"} 1
# HELP gotk_resource_info The current state of a Flux HelmRepository resource.
# TYPE gotk_resource_info gauge
gotk_resource_info{customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmRepository",customresource_version="v1",exported_namespace="cluster-conf",name="bitnami-charts",url="oci://registry-1.docker.io/bitnamicharts"} 1
gotk_resource_info{customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmRepository",customresource_version="v1",exported_namespace="cluster-conf",name="democratic-csi",ready="True",revision="sha256:1005c49084a570af3e76bf3899231ef057f642df013df66308cedc9fb508348a",url="https://democratic-csi.github.io/charts/"} 1
gotk_resource_info{customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmRepository",customresource_version="v1",exported_namespace="cluster-conf",name="routhca-forgejo-oci-helm",ready="True",revision="sha256:86cdbef83e147c28be5775beccd18c4a804dd3c9ebaba7466a1203a7bc19e0cf",url="https://forgejo.home.routh.ca/api/packages/routh-ca/helm"} 1
gotk_resource_info{customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmRepository",customresource_version="v1",exported_namespace="cluster-conf",name="metallb",ready="True",revision="sha256:9e1683809c3473a7229236cc222247b459def97b99f6e87c8cee5fd06ed62081",url="https://metallb.github.io/metallb"} 1
gotk_resource_info{customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmRepository",customresource_version="v1",exported_namespace="cluster-conf",name="external-dns",ready="True",revision="sha256:371fd37598c20fddaea7f1704bd201bb221a165001b91d5a46f4d67ba9b68f25",url="https://kubernetes-sigs.github.io/external-dns/"} 1
gotk_resource_info{customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmRepository",customresource_version="v1",exported_namespace="cluster-conf",name="grafana",ready="True",revision="sha256:83e0b03a34f8caea7ac12514a4d2bc611c1d1e15a02dea5442ab99cfd7b543ca",url="https://grafana.github.io/helm-charts"} 1
gotk_resource_info{customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmRepository",customresource_version="v1",exported_namespace="cluster-conf",name="cloudnative-pg",ready="True",revision="sha256:92fc65679d866c6adb8cfa031e7cada83254e6b9bf1e75d581d9880627a164e2",url="https://cloudnative-pg.github.io/charts"} 1
gotk_resource_info{customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmRepository",customresource_version="v1",exported_namespace="cluster-conf",name="collabora-online",ready="True",revision="sha256:6647cd92cd5300e6136fb4b763b4f1b6e9d44c2258eabdaeb25a21de5d3305d3",url="https://collaboraonline.github.io/online/"} 1
gotk_resource_info{customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmRepository",customresource_version="v1",exported_namespace="cluster-conf",name="nfd",ready="True",revision="sha256:955d72da7733dc8735a5f96714a4fd9dd59ba82eefaa502cee678d11f4be0c40",url="https://kubernetes-sigs.github.io/node-feature-discovery/charts"} 1
gotk_resource_info{customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmRepository",customresource_version="v1",exported_namespace="cluster-conf",name="jetstack",ready="True",revision="sha256:05de998518dae49671f01e665011a34d19725c80dbe8e2ccf05f4833b07bbb95",url="https://charts.jetstack.io"} 1
gotk_resource_info{customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmRepository",customresource_version="v1",exported_namespace="cluster-conf",name="cilium",ready="True",revision="sha256:025e33e6b8c6ca1be94287add6ffc574d34d1f03cebc43b3cb341e64bb0a3c4d",url="https://helm.cilium.io/"} 1
gotk_resource_info{customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmRepository",customresource_version="v1",exported_namespace="cluster-conf",name="ingress-nginx",ready="True",revision="sha256:b2fbc105a9565cd463285c01b6ecbf332bd5bede9ad476fab139b68d66a2fc4a",url="https://kubernetes.github.io/ingress-nginx"} 1
gotk_resource_info{customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmRepository",customresource_version="v1",exported_namespace="cluster-conf",name="nvidia",ready="True",revision="sha256:050e25958ca82a762f44b49333becddf3f2d876bf07d37b07174046e769592d3",url="https://helm.ngc.nvidia.com/nvidia"} 1
gotk_resource_info{customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmRepository",customresource_version="v1",exported_namespace="cluster-conf",name="sealed-secrets",ready="True",revision="sha256:a91246abf47e96214f08f44c1ef3a46daadd8f64dbed85c222d32abb80c3db73",url="https://bitnami-labs.github.io/sealed-secrets"} 1
gotk_resource_info{customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmRepository",customresource_version="v1",exported_namespace="cluster-conf",name="firefly-iii",ready="True",revision="sha256:67d3afbdb5e4d1839f3bfd13ea0d5a9bf6461fa1ec3d6802374a54aa7bd948a5",url="https://firefly-iii.github.io/kubernetes/"} 1
gotk_resource_info{customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmRepository",customresource_version="v1",exported_namespace="cluster-conf",name="nextcloud",ready="True",revision="sha256:5da18863d9a3876b4d5193f09f4c0799321e839185b1c4665e2cb4a1ac40d65f",url="https://nextcloud.github.io/helm/"} 1
gotk_resource_info{customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmRepository",customresource_version="v1",exported_namespace="cluster-conf",name="emberstack",ready="True",revision="sha256:8e27a3dfbc4a63a7fb53165779df26a25db3b7dcee4d76db8a89c208602895fb",url="https://emberstack.github.io/helm-charts"} 1
gotk_resource_info{customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmRepository",customresource_version="v1",exported_namespace="cluster-conf",name="intel",ready="True",revision="sha256:86484cc4f046530733d972cd477202fc5a81ad73265d1b2a8ba7eb6e904da28d",url="https://intel.github.io/helm-charts/"} 1
gotk_resource_info{customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmRepository",customresource_version="v1",exported_namespace="cluster-conf",name="stakater-charts",ready="True",revision="sha256:6034707236d9e7e9a69b9ac78eddf4bdbb193c5c8e2f8b519e442b18a674435c",url="https://stakater.github.io/stakater-charts"} 1
gotk_resource_info{customresource_group="source.toolkit.fluxcd.io",customresource_kind="HelmRepository",customresource_version="v1",exported_namespace="cluster-conf",name="prometheus-community",ready="True",revision="sha256:14703fff1ce0d2868945d7a8f4fec1dca11e0be6300d44fcc7e14ed2811685b2",url="https://prometheus-community.github.io/helm-charts"} 1

Without this config, we get all the important metrics for everything else in the cluster:

kube-state-metrics-default.txt

The differential here is huge and I cannot fathom this being missed - So now I'm trying to determine if kube-state-metrics has changed somehow as this config wipes out everything but custom metrics.

[Routhinator]

Ugh - ok I just noticed what it was:

  # For kube-prometheus-stacks that are already installed and configured with
  # custom collectors, commenting out the collectors and extraArgs below will
  # retain any existing kube-state-metrics configuration.
  collectors: []
  extraArgs:
    - --custom-resource-state-only=true

🤦 I missed this - but also - why is this the default? Should probably comment these out and change that whole section to this:

  # If you only want this deployment to collect Flux metrics, uncomment the below options to 
  # strip all of the default kube-state-metrics from this deployment.
  # collectors: []
  # extraArgs:
  #   - --custom-resource-state-only=true

I can't imagine many people will be wanting or expecting this to be deploying a kube-prometheus-stack for only flux metrics... that would make more sense as a dedicated deployment of just the kube-state-metrics chart.