IsAlive on /metrics

[ahjohannessen]

Could be useful to have IsAlive on /metrics for alerting.

[linear]

DB-1666 IsAlive on /metrics

[alexeyzimarev]

I am wondering why it would be needed. If the server is not alive, the metrics endpoint won't respond. So the value will always be positive or there will be no value because there's no metric.

[ahjohannessen]

I suppose the determination of IsAlive in gossip is probably more involved than metrics endpoint responding - eventstore_exporter that I used prior to moving to 25.x used gossip info to set metrics like:

clusterMemberAlive             *prometheus.Desc
clusterMemberIsClone           *prometheus.Desc
clusterMemberIsLeader          *prometheus.Desc
clusterMemberIsFollower        *prometheus.Desc
clusterMemberIsReadonlyReplica *prometheus.Desc

I have some alerting rules in grafana.net:

KdbUp

last_over_time(up{job="kdb"}[90s]) != 1

KdbLeader

count(last_over_time(kurrentdb_statuses{name = "Node", status = "Leader"}[90s])) != 1

I had this:

EsdbClusterUp

eventstore_cluster_member_alive{} != 1

I guess I do not need that one when I have defined KdbUp.

In essence, just want some alerts that trigger when the cluster is in trouble and when cluster members with important status (Follower/Leader, not so much ReadOnlyReplica) are down.