Describe the bug
The built-in icingadb health check may continue to report OK even though configuration and state synchronization between Icinga 2, Redis, Icinga DB and the database has effectively stopped.
We encountered this while being affected by:
#10941
While the underlying race condition is tracked separately, the absence of health-check detection is an independent problem. A failure in the data pipeline can leave Icinga DB Web without current data and eventually mark the entire environment as overdue, while the official health check still indicates that the Icinga DB stack is healthy.
In our multi-master environment, the following metrics clearly expose the broken state:
Primary master
icinga2_state_dump_items_1min shows that Icinga 2 performs the state dump.icingadb_state_sync_items_1min shows no corresponding state synchronization.
Secondary master
- No
icinga2_state_dump_items_1min activity is visible.
icingadb_state_sync_items_1min shows synchronization activity.icingadb_overdue_sync_items_1min increases continuously.- Synchronization nevertheless does not complete successfully.
icinga2_pending_config_and_state_updates_count continues to increase.icinga2_pending_config_and_state_updates_backlog continues to increase.
As a result, Icinga DB Web no longer receives new data and eventually marks all objects as overdue.
Despite this clearly degraded and continuously worsening state, the icingadb check remains OK.
This is a false negative in the official self-monitoring check. It is not merely a request for additional performance data or optional observability.
To Reproduce
The currently known trigger is the race condition described in:
#10941
The detection should not necessarily be limited to the exact race condition in #10941. The metrics describe the health and progress of the configuration/state pipeline and could identify an entire class of currently unknown failure modes where queues accumulate or synchronization stops making progress.
Expected behavior
The icingadb check should return a non-OK state when configuration or state synchronization is stalled or when the associated queues continuously accumulate without making progress.
At minimum, the check should evaluate sustained conditions such as:
- A continuously increasing
icinga2_pending_config_and_state_updates_count.
- A continuously increasing or excessively old
icinga2_pending_config_and_state_updates_backlog.
- State/config updates being produced by Icinga 2 but not processed by Icinga DB.
- A continuously increasing
icingadb_overdue_sync_items_1min.
- A persistent mismatch between dump activity and synchronization activity.
- No measurable synchronization progress over a configurable time window.
A short transient backlog during a reload should not immediately trigger an alert. However, a queue that does not drain or continues growing for several minutes must result in at least WARNING, and eventually CRITICAL.
The exact thresholds can remain configurable, but the check should provide safe defaults that detect a stalled pipeline without requiring every user to implement their own external logic.
Screenshots
Grafana screenshots showing the diverging metrics and continuously increasing backlogs can not be provided.
Your Environment
Detailed version output and topology information can be added when required.
Additional context
Using Grafana to create a custom alert from these metrics is technically possible, but it is not an adequate solution.
The built-in check is the component that is supposed to determine whether the Icinga DB data path is operational. Requiring a separate monitoring system and custom alert logic to detect that the official health check is returning a false OK defeats the purpose of having the check.
This problem also substantially increases the operational impact of #10941. We have disabled automated reloads outside business hours to reduce the risk of entering this undetected degraded state without an operator being present. This negatively affects customers that rely on our automation and timely configuration deployments.
We are therefore forced to choose between:
- Allowing automated configuration changes and risking an undetected platform-wide loss of current monitoring data.
- Restricting reloads and accepting delayed customer configuration changes.
Neither is an acceptable long-term workaround.
Please treat this as a high-priority reliability issue and consider backporting the detection improvements to the current stable release line. A core data pipeline being stalled while the official self-monitoring check reports OK is a severe operational failure mode.
This class of problem should also be covered by regression tests. If configuration/state queues stop draining, synchronization rates diverge persistently, or no forward progress is made for a defined interval, the health check must not remain green.
For operators of large environments, repeatedly discovering stable-release regressions only after production data flow has silently stopped makes safe upgrades increasingly difficult. Reliable self-detection would at least prevent such failures from remaining unnoticed and significantly reduce their operational impact.
Describe the bug
The built-in
icingadbhealth check may continue to reportOKeven though configuration and state synchronization between Icinga 2, Redis, Icinga DB and the database has effectively stopped.We encountered this while being affected by:
#10941
While the underlying race condition is tracked separately, the absence of health-check detection is an independent problem. A failure in the data pipeline can leave Icinga DB Web without current data and eventually mark the entire environment as overdue, while the official health check still indicates that the Icinga DB stack is healthy.
In our multi-master environment, the following metrics clearly expose the broken state:
Primary master
icinga2_state_dump_items_1minshows that Icinga 2 performs the state dump.icingadb_state_sync_items_1minshows no corresponding state synchronization.Secondary master
icinga2_state_dump_items_1minactivity is visible.icingadb_state_sync_items_1minshows synchronization activity.icingadb_overdue_sync_items_1minincreases continuously.icinga2_pending_config_and_state_updates_countcontinues to increase.icinga2_pending_config_and_state_updates_backlogcontinues to increase.As a result, Icinga DB Web no longer receives new data and eventually marks all objects as overdue.
Despite this clearly degraded and continuously worsening state, the
icingadbcheck remainsOK.This is a false negative in the official self-monitoring check. It is not merely a request for additional performance data or optional observability.
To Reproduce
The currently known trigger is the race condition described in:
#10941
The detection should not necessarily be limited to the exact race condition in #10941. The metrics describe the health and progress of the configuration/state pipeline and could identify an entire class of currently unknown failure modes where queues accumulate or synchronization stops making progress.
Expected behavior
The
icingadbcheck should return a non-OK state when configuration or state synchronization is stalled or when the associated queues continuously accumulate without making progress.At minimum, the check should evaluate sustained conditions such as:
icinga2_pending_config_and_state_updates_count.icinga2_pending_config_and_state_updates_backlog.icingadb_overdue_sync_items_1min.A short transient backlog during a reload should not immediately trigger an alert. However, a queue that does not drain or continues growing for several minutes must result in at least
WARNING, and eventuallyCRITICAL.The exact thresholds can remain configurable, but the check should provide safe defaults that detect a stalled pipeline without requiring every user to implement their own external logic.
Screenshots
Grafana screenshots showing the diverging metrics and continuously increasing backlogs can not be provided.
Your Environment
icinga2 --version): Icinga 2.16.4-1icinga2 daemon -C): SuccessfulDetailed version output and topology information can be added when required.
Additional context
Using Grafana to create a custom alert from these metrics is technically possible, but it is not an adequate solution.
The built-in check is the component that is supposed to determine whether the Icinga DB data path is operational. Requiring a separate monitoring system and custom alert logic to detect that the official health check is returning a false
OKdefeats the purpose of having the check.This problem also substantially increases the operational impact of #10941. We have disabled automated reloads outside business hours to reduce the risk of entering this undetected degraded state without an operator being present. This negatively affects customers that rely on our automation and timely configuration deployments.
We are therefore forced to choose between:
Neither is an acceptable long-term workaround.
Please treat this as a high-priority reliability issue and consider backporting the detection improvements to the current stable release line. A core data pipeline being stalled while the official self-monitoring check reports
OKis a severe operational failure mode.This class of problem should also be covered by regression tests. If configuration/state queues stop draining, synchronization rates diverge persistently, or no forward progress is made for a defined interval, the health check must not remain green.
For operators of large environments, repeatedly discovering stable-release regressions only after production data flow has silently stopped makes safe upgrades increasingly difficult. Reliable self-detection would at least prevent such failures from remaining unnoticed and significantly reduce their operational impact.