[Feature]: Support for rack "management"

[lalitadithya]

Prerequisites

• I searched existing issues

Feature Summary

In GB200 or other rack type deployment, we need to be able to drain an entire rack when sufficient number of nodes go bad. Currently NVSentinel does a good job of identifying bad nodes in a rack, but it doesn't "escalate" it to a rack level issue when there are persistent bad nodes in a rack.

Problem/Use Case

As an operator I want the ability to remediate an entire rack when 50+% of the nodes in a rack are bad due to switch/other errors.

Proposed Solution

TBD

Component

Fault Management

[mchmarny]

Agree, historically, we thought of break-fix as replacement of VM node when it was emitting GPU-related errors, with 0 impact on the surrounding nodes. With GB200, single GPU failure observed on a single node can actually be symptoms of a larger issue on NVL level.

With most CSPs offering only rack-level remediation (no individual node repairs) we may need adjust how we think of break-fix in context of that kind of architecture:

• NVSwitch mesh
• NVLink fabric
• IMEX channels
• PCIe bridges
• Power tuning, GPU watchdog logic
• FW/fabric-level software

At minimum, NVS should be aware of the broader topology context for a given node and in case when error is indicating fault at the rack level have the option to potentially cordon all nodes when number of nodes in that in that UlstraServer is above X%.