Summary
On a host with multiple active RDMA NICs, RDMABuffer provides no public way to select a device/rail or request striping. Transfers therefore use the backend's implicit choice, which prevents applications from controlling placement or using aggregate multi-rail bandwidth.
Minimal API demonstration
import inspect
from monarch.rdma import RDMABuffer
print(inspect.signature(RDMABuffer))
# (data: torch.Tensor | memoryview) -> None
RDMABuffer.read_into() likewise accepts only the destination and timeout. On a system with multiple active devices (for example mlx5_0 and mlx5_1), there is no public parameter or policy object for:
- selecting the local/remote RDMA device;
- binding a transfer to one rail; or
- striping a large transfer across rails.
Requested behavior
Please expose a stable selection surface, such as a device/rail policy at buffer construction or connection setup, and document how the selected local and remote devices are paired. An explicit single-rail selector would already make placement deterministic; optional striping could follow separately.
Why this is separate
#3376 and #3759 report transfer failures on EFA/ibverbs devices. This request is not a failure on a selected device; it is the absence of a user-facing selection/striping API when several working rails exist.
Observed API surface in torchmonarch 0.5.0.
Summary
On a host with multiple active RDMA NICs,
RDMABufferprovides no public way to select a device/rail or request striping. Transfers therefore use the backend's implicit choice, which prevents applications from controlling placement or using aggregate multi-rail bandwidth.Minimal API demonstration
RDMABuffer.read_into()likewise accepts only the destination and timeout. On a system with multiple active devices (for examplemlx5_0andmlx5_1), there is no public parameter or policy object for:Requested behavior
Please expose a stable selection surface, such as a device/rail policy at buffer construction or connection setup, and document how the selected local and remote devices are paired. An explicit single-rail selector would already make placement deterministic; optional striping could follow separately.
Why this is separate
#3376 and #3759 report transfer failures on EFA/ibverbs devices. This request is not a failure on a selected device; it is the absence of a user-facing selection/striping API when several working rails exist.
Observed API surface in torchmonarch 0.5.0.