add runbooks for new alerts by yati1998 · Pull Request #363 · openshift/runbooks

yati1998 · 2025-12-12T07:31:07Z

there are new alerts introduced for odf
health score calculation. This commit adds
runbooks for each of them

yati1998 · 2025-12-12T07:31:40Z

@aruniiird @weirdwiz please do have a look.

weirdwiz

needs some changes

alerts/openshift-container-storage-operator/ODFCorePodRestarted.md

weirdwiz · 2025-12-15T06:06:32Z

alerts/openshift-container-storage-operator/ODFDiskUtilizationHigh.md

no mitigation section? please add mitigation steps,

I am not sure of what mitigation steps should be added here, so I left it empty for now!!
@weirdwiz if you have any suggestions, we can discuss offline.

Mitigation for this is to either move workloads to other storage systems or (preferred) add more disks.
Ceph is one of the few storage systems that grows IO performance linearly with capacity... so more disks = more performance

weirdwiz · 2025-12-15T06:07:38Z

alerts/openshift-container-storage-operator/ODFNodeMTULessThan9000.md

same here, add mitigation steps

The MTU runbook should mention how to verify jumbo frames work end-to-end

I am not sure about this, maybe we can work on it once you are back.

You can find many "Jumbo Frame" test instructions on the internet - for example this one:
https://blah.cloud/networks/test-jumbo-frames-working/

In the end you use ping with a certain icmp size (which different for the different OSs) and you tell the network stack not to fragment the package (but send it whole).

As a mitigation, customers need to ensure the node network interfaces are configured for 9000 bytes AND that all switches in between the nodes also support 9000 bytes on their ports.

weirdwiz · 2025-12-15T06:08:55Z

alerts/openshift-container-storage-operator/ODFCorePodRestarted.md

existing runbooks reference shared helper documents like:

helpers/podDebug.md

helpers/troubleshootCeph.md

helpers/gatherLogs.md

helpers/networkConnectivity.md

the new runbooks embed all commands inline instead of referencing these. consider using helper links for consistency and maintainability.

weirdwiz · 2025-12-15T06:09:47Z

alerts/openshift-container-storage-operator/ODFNodeLatencyHighOnOSDNodes.md

+ping <node-internal-ip>
+```
+4. Use mtr or traceroute to analyze path and hops.
+5. Verify if the node is under high CPU or network load:


Suggested change

5. Verify if the node is under high CPU or network load:

5. Verify if the node is under high CPU or network load:

oc debug node/<node>

top -b -n 1 | head -20

sar -u 1 5

weirdwiz · 2025-12-15T06:10:13Z

alerts/openshift-container-storage-operator/ODFNodeNICBandwidthSaturation.md

+sar -n DEV 1 5
+```
+3. Use Prometheus to graph:
+```prompql


Suggested change

```prompql

```promql

yati1998 · 2025-12-31T06:11:23Z

@weirdwiz updated the PR except for the 2 comments, we can work on them once you are back.

mulbc · 2026-01-13T13:45:53Z

alerts/openshift-container-storage-operator/ODFCorePodRestarted.md

+
+## Impact
+
+* Brief service interruption (e.g., MON restart may cause quorum re-election).


service interruption sounds worse than it is... unless the MONs cannot agree to a quorum any more, there is no "downtime".
Instead let's put all of the Impact points in relative points... Since Ceph is very resilient, Pod restarts should only have an affect if they happen frequently (more than 10 times in a 5min window).

mulbc · 2026-01-13T13:48:50Z

alerts/openshift-container-storage-operator/ODFCorePodRestarted.md

+## Impact
+
+* Brief service interruption (e.g., MON restart may cause quorum re-election).
+* OSD restart triggers PG peering and potential recovery.


Someone who doesn't know Ceph will not understand this :) (even though it is factually correct)

How do you like my proposal:

If OSDs are restarted frequently or do not start up within 5 minutes, the cluster might decide to rebalance the data onto other more reliable disks. If this happens, the cluster will temporarily be slightly less performant.

mulbc · 2026-01-13T13:49:59Z

alerts/openshift-container-storage-operator/ODFDiskUtilizationHigh.md

+
+## Impact
+
+* Increased I/O latency for RBD/CephFS clients.


RBD and CephFS are Ceph terms. Let's keep it simple and just call them Block, Object and File (all of these would be affected)

mulbc · 2026-01-13T13:50:28Z

alerts/openshift-container-storage-operator/ODFDiskUtilizationHigh.md

+## Impact
+
+* Increased I/O latency for RBD/CephFS clients.
+* Slower OSD response times, risking heartbeat timeouts.


I don't think that's true. If the underlying storage is busy, the process should still be able to send heartbeats?!

mulbc · 2026-01-13T13:51:45Z

alerts/openshift-container-storage-operator/ODFDiskUtilizationHigh.md

Mitigation for this is to either move workloads to other storage systems or (preferred) add more disks.
Ceph is one of the few storage systems that grows IO performance linearly with capacity... so more disks = more performance

mulbc · 2026-01-13T13:55:30Z

alerts/openshift-container-storage-operator/ODFNodeLatencyHighOnNONOSDNodes.md

+5. Review Ceph monitor logs if the node hosts MONs:
+```bash
+oc logs -l app=rook-ceph-mon -n openshift-storage
+```


Another step could be to check switch / networking monitoring to see if any ports are too busy

mulbc · 2026-01-13T13:57:47Z

alerts/openshift-container-storage-operator/ODFNodeLatencyHighOnOSDNodes.md

+## Diagnosis
+
+
+1. Identify affected node(s):


Why do we have this step if we get the node name and IP in step #2 from the alert?

mulbc · 2026-01-13T13:58:27Z

alerts/openshift-container-storage-operator/ODFNodeLatencyHighOnOSDNodes.md

+
+## Mitigation
+
+1. Network tuning: Ensure jumbo frames (MTU ≥ 9000) are enabled end-to-end 


Are you sure Jumbo Frames will help with latency? Why?

not exactly sure.

mulbc · 2026-01-13T14:01:28Z

alerts/openshift-container-storage-operator/ODFNodeMTULessThan9000.md

You can find many "Jumbo Frame" test instructions on the internet - for example this one:
https://blah.cloud/networks/test-jumbo-frames-working/

In the end you use ping with a certain icmp size (which different for the different OSs) and you tell the network stack not to fragment the package (but send it whole).

As a mitigation, customers need to ensure the node network interfaces are configured for 9000 bytes AND that all switches in between the nodes also support 9000 bytes on their ports.

mulbc · 2026-01-13T14:02:09Z

alerts/openshift-container-storage-operator/ODFNodeNICBandwidthSaturation.md

+
+## Mitigation
+
+1. Short term: Throttle non-essential traffic on the node.


mulbc

lgtm - small error corrections where I added recommendations. Aside from these I'm good with the PR

mulbc · 2026-01-21T15:08:56Z

alerts/openshift-container-storage-operator/ODFCorePodRestarted.md

+
+* If OSDs are restarted frequently or do not start up within 5 minutes,
+  the cluster might decide to rebalance the data onto other more reliable
+  disks.If this happens, the cluster will temporarily be slightly less


Suggested change

disks.If this happens, the cluster will temporarily be slightly less

disks. If this happens, the cluster will temporarily be slightly less

mulbc · 2026-01-21T15:10:01Z

alerts/openshift-container-storage-operator/ODFDiskUtilizationHigh.md

+
+## Mitigation
+
+* Increase more disks to enhance the performance.


Suggested change

* Increase more disks to enhance the performance.

* Add more disks to the cluster to enhance the performance.

mulbc · 2026-01-21T15:11:32Z

alerts/openshift-container-storage-operator/ODFNodeLatencyHighOnNONOSDNodes.md

+## Diagnosis
+
+1. From the alert, note the instance (node IP).
+2. Confirm the node does not run OSDs:


If this is an OSD node and triggers the >100ms alert, we're in trouble :P
So I think this check does not provide any value (we're not doing anything with the data we gather with this)

there are new alerts introduced for odf health score calculation. This commit adds runbooks for each of them Signed-off-by: yati1998 <ypadia@redhat.com>

yati1998 · 2026-01-22T04:48:07Z

@weirdwiz please review the PR, I have addressed all the comments.

yati1998 · 2026-01-27T12:13:18Z

@agarwal-mudit can you please review the PR, it has been approved by @mulbc and @weirdwiz and all other comments are addressed

agarwal-mudit

/lgtm

openshift-ci · 2026-01-28T06:45:59Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: agarwal-mudit, malayparida2000, mulbc, yati1998

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~alerts/openshift-container-storage-operator/OWNERS~~ [agarwal-mudit,malayparida2000]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci · 2026-01-28T06:53:09Z

@yati1998: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Madhu-1 · 2026-01-28T06:55:36Z

alerts/openshift-container-storage-operator/ODFCorePodRestarted.md

+
+## Meaning
+
+A core ODF pod (OSD, MON, MGR, ODF operator, or metrics exporter) has


Why ODF operator and metrics exporter are considered as the core pods?

Madhu-1 · 2026-01-28T06:56:50Z

alerts/openshift-container-storage-operator/ODFDiskUtilizationHigh.md

+```bash
+iostat -x 2 5
+```
+4. Correlate with Ceph:


Are they supposed to run these commands on the toolbox pod?

Madhu-1 · 2026-01-28T06:57:18Z

alerts/openshift-container-storage-operator/ODFNodeLatencyHighOnNONOSDNodes.md

+
+## Meaning
+
+ICMP RTT latency to non-OSD ODF nodes (e.g., MON, MGR, MDS, or client nodes)


What are client nodes here? csi client?

Madhu-1 · 2026-01-28T06:57:54Z

alerts/openshift-container-storage-operator/ODFNodeLatencyHighOnNONOSDNodes.md

+
+* Delayed Ceph monitor elections or quorum instability.
+* Slower metadata operations in CephFS.
+* Increased latency for CSI controller operations.


what about the CSI node operation and csi-addons operations?

Madhu-1 · 2026-01-28T06:58:27Z

alerts/openshift-container-storage-operator/ODFNodeLatencyHighOnNONOSDNodes.md

+
+1. From the alert, note the instance (node IP).
+2. Test connectivity:
+```bash


from there they are suppose to run these commands?

Madhu-1 · 2026-01-28T06:58:50Z

alerts/openshift-container-storage-operator/ODFNodeLatencyHighOnNONOSDNodes.md

+3. Check system load and network interface stats on the node:
+```bash
+oc debug node/<node-name>
+sar -n DEV 1 5


please add details what is 1 and 5 here and how to get it

Madhu-1 · 2026-01-28T06:59:41Z

alerts/openshift-container-storage-operator/ODFNodeLatencyHighOnOSDNodes.md

+## Diagnosis
+
+1. Check the alert’s instance label to get the node IP.
+2. From a monitoring or debug pod, test connectivity:


can we provide a example command to get the monitoring or debug pod

Madhu-1 · 2026-01-28T07:00:14Z

alerts/openshift-container-storage-operator/ODFNodeLatencyHighOnOSDNodes.md

+top -b -n 1 | head -20
+sar -u 1 5
+```
+5. Check Ceph health and OSD status:


Is toolbox already enabled or they need to enable it?

malayparida2000 · 2026-01-29T05:56:30Z

@yati1998 Please take a look at the concerns Madhu has raised & address them in a follow up PR if possible

openshift-ci bot requested review from agarwal-mudit and malayparida2000 December 12, 2025 07:31

yati1998 force-pushed the odfhealthscore branch from 9a36696 to 9773226 Compare December 12, 2025 07:38

weirdwiz suggested changes Dec 15, 2025

View reviewed changes

yati1998 force-pushed the odfhealthscore branch from 9773226 to 2deb710 Compare December 31, 2025 06:09

yati1998 force-pushed the odfhealthscore branch from 2deb710 to 4efb275 Compare December 31, 2025 06:16

mulbc suggested changes Jan 13, 2026

View reviewed changes

yati1998 force-pushed the odfhealthscore branch from 4efb275 to 0f8b761 Compare January 20, 2026 07:00

yati1998 requested review from mulbc and weirdwiz January 20, 2026 07:01

yati1998 force-pushed the odfhealthscore branch from 0f8b761 to 5ca12a0 Compare January 21, 2026 14:42

mulbc approved these changes Jan 21, 2026

View reviewed changes

add runbooks for new alerts

f770989

there are new alerts introduced for odf health score calculation. This commit adds runbooks for each of them Signed-off-by: yati1998 <ypadia@redhat.com>

yati1998 force-pushed the odfhealthscore branch from 5ca12a0 to f770989 Compare January 22, 2026 04:47

malayparida2000 approved these changes Jan 27, 2026

View reviewed changes

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 27, 2026

agarwal-mudit approved these changes Jan 28, 2026

View reviewed changes

openshift-ci bot assigned agarwal-mudit Jan 28, 2026

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jan 28, 2026

openshift-merge-bot bot merged commit 9f906b3 into openshift:master Jan 28, 2026
2 checks passed

Madhu-1 reviewed Jan 28, 2026

View reviewed changes


		## Impact

		* Brief service interruption (e.g., MON restart may cause quorum re-election).


		## Mitigation

		1. Network tuning: Ensure jumbo frames (MTU ≥ 9000) are enabled end-to-end


		## Mitigation

		1. Short term: Throttle non-essential traffic on the node.

	disks.If this happens, the cluster will temporarily be slightly less
	disks. If this happens, the cluster will temporarily be slightly less


		## Mitigation

		* Increase more disks to enhance the performance.

	* Increase more disks to enhance the performance.
	* Add more disks to the cluster to enhance the performance.


		## Meaning

		A core ODF pod (OSD, MON, MGR, ODF operator, or metrics exporter) has


		## Meaning

		ICMP RTT latency to non-OSD ODF nodes (e.g., MON, MGR, MDS, or client nodes)

Conversation

yati1998 commented Dec 12, 2025

Uh oh!

yati1998 commented Dec 12, 2025

Uh oh!

weirdwiz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yati1998 commented Dec 31, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mulbc left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yati1998 commented Jan 22, 2026

Uh oh!

yati1998 commented Jan 27, 2026

Uh oh!

agarwal-mudit left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented Jan 28, 2026

Uh oh!

openshift-ci bot commented Jan 28, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment