API endpoints for health(z)

[sthaha]

Target Version

Current version (0.10.0+) - New Architecture

Feature Description

Applications running in Kubernetes often rely on external probes (liveness/readiness) to determine pod health. Currently, the kepler does not expose any HTTP endpoints that can be used by these probes or by monitoring systems. This makes it difficult to:

1. Verify that the service is fully initialized before accepting traffic.
2. Detect and recover from runtime failures automatically via Kubernetes’ self‑healing mechanisms.

Desired Feature

Problem Statement

Currently, the kepler does not expose any HTTP endpoints that can be used by these probes or by monitoring systems. This makes it difficult to:

1. Verify that the service is fully initialized before accepting traffic.
2. Detect and recover from runtime failures automatically via Kubernetes’ self‑healing mechanisms.

Proposed Solution

Introduce two lightweight RESTful endpoints: (see: https://kubernetes.io/docs/reference/using-api/health-checks/ )

Endpoint	Purpose	Example Response
GET /probe/readyz	Readiness – confirms the application is ready to serve traffic.	{ "status": "ok" }
GET /probe/livez	Liveness – checks that the process is running and not dead‑locked.	{ "status": "alive" }

Both endpoints should:

• Return HTTP 200 when healthy.
• Return a non‑2xx status (e.g., 503) when unhealthy.

[ExplorerRay]

According to the docs you refer, it mentions that healthz is deprecated since Kubernetes v1.16. Therefore, I think this should change healthz to readyz, otherwise we may need to change it again after healthz disappears.

[sthaha]

@ExplorerRay thats only an example .. I think a better example would have been https://kubernetes.io/docs/concepts/configuration/liveness-readiness-startup-probes/

There are readiness, and liveness probes and the actual endpoint really doesn't matter but it is better to follow the convention. so /healthz and /livez are just fine. Perhaps the path could be

• /probe/readyz
• /probe/livez

[ExplorerRay]

Sorry for mistaking that. I originally thought it is something like a standard, but it is just an implementation of Kubernetes API server.

[sthaha]

No worries, I wasn't clear either :(
I shouldn't have pasted what "looked" like the right reference to these probe endpoint.

[ExplorerRay]

About the readyz, is it good to use whether exporter provides metrics to check readiness? (Maybe similar to current check of /metrics)

For the livez, I think if the API server runs well (no err when createServices), this will mean Kepler is alive (not in deadlock). So just needs to implement another endpoint for responding 200.

Are these ideas enough for what this feature wants to achieve?

[sthaha]

About the readyz, is it good to use whether exporter provides metrics to check readiness? (Maybe similar to current check of /metrics)

Since all access to /metrics results in computation, we shouldn't use /metrics and reserve it solely for proemetheus to scrape.

For the livez, I think if the API server runs well (no err when createServices), this will mean Kepler is alive (not in deadlock). So just needs to implement another endpoint for responding 200.

Yes, livez and readyz should be just minimal 200 OK endpoints for the k8s probes.

[ExplorerRay]

I originally consider that readyz should check Kepler's functions works well (like that computation are available to work and export).
If we are afraid that the computation may hurt performance, should we mock something to let readyz know exporter works well?

[robinlem]

Hi!

Proposal: add two ultra-light endpoints on the same server/port as /metrics

• GET /probe/livez → 200 if the sampling loop ticked recently, 503 if stalled.
  -GET /probe/readyz → 200 after critical init (eBPF/sensors) and first successful sample/export; else 503.

OK to proceed with these paths and defaults? If yes, I can try to open a draft PR with code , unit tests, Helm and docs

[robinlem]

Hi All! I’ve implemented the endpoints requested in this issue:

This is my first PR to Kepler—feedback is very welcome, and I’m happy to adjust anything I missed

Endpoints created

/probe/readyz: Readiness probe (200 after the first sample has been collected)

/probe/livez: Liveness probe (200 if sampling has been active within the last 30s)

Performance

< 1 ms response time (vs 50–200 ms for /metrics)

Minimal CPU/memory usage

JSON responses as specified

Files changed

internal/server/probe.go — endpoint implementation

internal/server/probe_test.go — comprehensive tests

internal/monitor/monitor.go — added LastCollectionTime()

manifests/helm/kepler/ — Helm configuration with legacy fallback

HEALTH_PROBES.md — full documentation

Helm configuration

daemonset:
  useOptimizedProbes: true  # default
  livenessProbe:
    httpGet:
      path: /probe/livez
  readinessProbe:
    httpGet:
      path: /probe/readyz

Backward compatibility: set useOptimizedProbes: false to fall back to /metrics.

Ready for review!