feat(metrics): add Prometheus /metrics endpoint with DNS, dialer, connection, and runtime stats - #1015
Open
MaurUppi wants to merge 7 commits into
Open
feat(metrics): add Prometheus /metrics endpoint with DNS, dialer, connection, and runtime stats#1015MaurUppi wants to merge 7 commits into
MaurUppi wants to merge 7 commits into
Conversation
added 5 commits
June 16, 2026 10:11
…registry, config)
Author
|
Grafana Dashboard UI 
|
added 2 commits
June 19, 2026 11:25
In the staged same-port reload path, metricsState.SetControlPlane(newC) was never called because the preparation phase hits `continue` before reaching the call site at line 679 (the non-staged path). After the staged handoff completes and clearPendingStagedHandoff() is called, metricsState still pointed to the retired old control plane. Once that plane's outbounds and DNS controller were torn down during retirement, all dialer and DNS collectors emitted no samples, causing every Grafana panel to show "no data" until a full process restart. Mirror the non-staged path pattern: switch metricsState to the new control plane before retiring the old one.
dae_dialer_latency_last_seconds is only emitted when hasLastLatency=true (set only after a successful health probe). New dialers introduced by a routing config change won't emit this metric until their first probe completes (up to check_interval=900s), leaving the \$network variable empty and all \$network-filtered panels showing "no data". Switch to dae_dialer_alive which is emitted immediately at dialer creation, so \$network populates instantly after reload.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
/metricsHTTP endpoint to dae, gated behindendpoint_listen_addressinglobal {}(disabled by default)inFlight,latency,errTotal) indialSendatomic.Pointer[ControlPlane], and pprof co-hosting.plan/metrics/dae_Transparent_Proxy-Grafana_dashboard.json) covering all exposed metric familiesWhat's in this PR
Infrastructure (
pkg/metrics/,pkg/metricshttp/)state.go—metrics.Stateholdingatomic.Pointer[control.ControlPlane]for reload-safe scrapingregistry.go—NewRegistry: registers 4 collectors +process_*+go_*metricshttp/auth.go— BasicAuth middlewaremetricshttp/server.go—NewEndpointServer/StartEndpointServerwith optional TLSCollectors
collector_dialer.go— dialer/health-check metrics (alive, latency, moving avg, health check totals)collector_dns.go— DNS metrics (cache, concurrency, forwarder, upstream counters + histograms)collector_conn.go— connection metrics (TCP active/total, UDP endpoints/queues/total)collector_runtime.go— runtime/node metrics (SnapshotRuntimeStats,SnapshotNodeLatencies), labeled with dialer group/nameDNS write-side wiring (
control/dns_control.go)Three upstream metrics were structurally present but had no write side. Now instrumented in
dialSendaround theforwardWithFallbackcall, withinFlightreleased immediately after network I/O (not deferred) so the gauge stays scoped to actual upstream wait time.Config (
config/config.go,example.dae)New
endpoint_*fields inglobal {}:TLS file permission policy enforced at startup and reload (cert: 0640/0644, key: 0600).
Hot-reload (
cmd/run.go)metricsState.SetControlPlane(newC)(atomic)endpointServer.Shutdown()→ new serverTest plan
go test ./pkg/metrics/... ./pkg/metricshttp/...— descriptor tests + endpoint server tests PASSgo test ./control/... ./cmd/... ./common/...PASSgo vet ./...CLEANcurl http://localhost:5556/metrics→ all metric families presentdae_dns_forwarder_in_flight{upstream=...} > 0under concurrent DNS loaddae_dns_upstream_err_totalincrements on upstream errordae_dns_upstream_latency_secondshas non-zero bucket datacurlwithout credentials → 401; with credentials → 200