Skip to content

Expose RDMA device/rail selection for RDMABuffer transfers #4396

Description

@Balaxxe

Summary

On a host with multiple active RDMA NICs, RDMABuffer provides no public way to select a device/rail or request striping. Transfers therefore use the backend's implicit choice, which prevents applications from controlling placement or using aggregate multi-rail bandwidth.

Minimal API demonstration

import inspect

from monarch.rdma import RDMABuffer

print(inspect.signature(RDMABuffer))
# (data: torch.Tensor | memoryview) -> None

RDMABuffer.read_into() likewise accepts only the destination and timeout. On a system with multiple active devices (for example mlx5_0 and mlx5_1), there is no public parameter or policy object for:

  • selecting the local/remote RDMA device;
  • binding a transfer to one rail; or
  • striping a large transfer across rails.

Requested behavior

Please expose a stable selection surface, such as a device/rail policy at buffer construction or connection setup, and document how the selected local and remote devices are paired. An explicit single-rail selector would already make placement deterministic; optional striping could follow separately.

Why this is separate

#3376 and #3759 report transfer failures on EFA/ibverbs devices. This request is not a failure on a selected device; it is the absence of a user-facing selection/striping API when several working rails exist.

Observed API surface in torchmonarch 0.5.0.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions