feat: CR-configurable per-cluster domain metrics on the Prometheus /metrics endpoint

[xrl]

Refs #381

What

Exposes custom per-cluster domain metrics for every reconciled EtcdCluster on the operator's existing Prometheus /metrics endpoint, and makes Prometheus observability configurable via the CR (spec.metrics).

The operator already serves a controller-runtime metrics endpoint backed by a global registry; this PR registers custom collectors against that same registry, so there is no new HTTP server or port. Every series is labelled with the cluster namespace and name, so a single operator managing many clusters yields one time series per cluster.

Metrics (subsystem etcd_operator_cluster_)

metric	type	meaning
member_count_desired	gauge	desired members (spec.size)
member_count	gauge	members from the etcd member-list API (status.memberCount)
member_count_ready	gauge	ready etcd pods (StatefulSet readyReplicas)
member_count_healthy	gauge	members reported healthy
learner_count	gauge	non-voting (learner) members
has_quorum	gauge 1/0	healthy members form a strict majority
has_leader	gauge 1/0	a leader is currently known
tls_enabled	gauge 1/0	cluster configured for TLS
leader_changes_total	counter	observed leader-ID transitions
reconcile_duration_seconds	histogram	reconcile wall-clock
reconcile_errors_total	counter	reconciles returning an error

Gauge values are derived from the freshly-computed EtcdClusterStatus at the end of updateStatus — no extra API calls. Reconcile duration/errors are observed in Reconcile. Leader changes are tracked per cluster (the first observation and leaderless windows do not count, so operator restarts don't inflate the counter). When a cluster is deleted, its series are dropped to bound cardinality.

CR config knob (spec.metrics)

spec:
  metrics:
    enabled: true            # default true; false drops this cluster's series
    podMonitor:
      enabled: true          # emit a prometheus-operator PodMonitor for the etcd member pods
      interval: "30s"
      port: "client"

• spec.metrics.enabled (*bool, default true) toggles/scopes the operator domain metrics per cluster.
• spec.metrics.podMonitor, when enabled, makes the operator reconcile a PodMonitor (owned by the EtcdCluster, so GC'd on delete) selecting the etcd member pods (app/controller labels) so Prometheus scrapes etcd's own /metrics.

Why unstructured for the PodMonitor

The PodMonitor is created via unstructured so the operator takes no hard dependency on the prometheus-operator Go module and does not require its CRDs to be installed unless a user opts in. If the CRD is absent, the operator logs and skips PodMonitor reconciliation without failing the rest of the reconcile (meta.IsNoMatchError).

Files

• NEW internal/metrics/ — collector definitions, registration (MustRegister/RegisterTo), RecordClusterMetrics, ObserveReconcile, DeleteClusterMetrics, per-cluster leader tracker.
• NEW internal/controller/podmonitor.go — PodMonitor reconcile/cleanup.
• api/v1alpha1/etcdcluster_types.go — MetricsSpec / PodMonitorSpec + MetricsEnabled()/PodMonitorEnabled() helpers (CRD/deepcopy regenerated via the pinned controller-gen).
• internal/controller/etcdcluster_controller.go — minimal, localized instrumentation (time the loop, record metrics after status, reconcile PodMonitor, drop series on delete) + PodMonitor RBAC marker.
• cmd/main.go — register collectors at startup.
• NEW test/e2e/metrics_test.go — compile-verified e2e that creates a cluster with spec.metrics.podMonitor and scrapes the operator /metrics.
• config/samples/sample_metrics_etcdcluster.yaml — example CR.

Testing

• Unit tests (internal/metrics/metrics_test.go) register collectors to an isolated registry, drive cluster state, scrape via prometheus/client_golang/testutil, and assert exact values + labels — including quorum/leader edge cases, leader-change counting across leaderless windows, reconcile error counting, and series deletion (and leader-tracker reset) on cluster delete. Spec-helper defaults are covered too.
• go build ./..., go vet ./..., go vet ./test/e2e/..., and the controller envtest suite all pass.
• No live KinD run in this PR; the e2e test is authored to compile in CI and run under make test-e2e.

🤖 Generated with Claude Code

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: xrl
Once this PR has been reviewed and has the lgtm label, please assign justinsb for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

Hi @xrl. Thanks for your PR.

I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.