Skip to content

[Distributed][HCCL] Validate all_to_all_single split sums against tensor numel#2758

Merged
lvyufeng merged 1 commit into
candle-org:masterfrom
lvyufeng:feat/distributed-split-group-parent-seq
Mar 8, 2026
Merged

[Distributed][HCCL] Validate all_to_all_single split sums against tensor numel#2758
lvyufeng merged 1 commit into
candle-org:masterfrom
lvyufeng:feat/distributed-split-group-parent-seq

Conversation

@lvyufeng

@lvyufeng lvyufeng commented Mar 8, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Add explicit all_to_all_single validation to ensure split-size sums match input/output tensor numel before collective dispatch.
  • Keep existing HCCL matrix split-compatibility checks, but fail earlier with clearer user-facing errors for malformed split/numel combinations.
  • Add HCCL multicard tests covering 2/4/8 cards for both input-sum mismatch and output-sum mismatch paths.

Key Changes

  • src/mindtorch_v2/distributed/__init__.py
  • tests/mindtorch_v2/test_hccl_all_to_all_single_split_numel_validation_multicard.py

Test Plan

  • PYTHONPATH=/usr/local/Ascend/ascend-toolkit/latest/python/site-packages:src pytest -q tests/mindtorch_v2/test_hccl_all_to_all_single_split_numel_validation_multicard.py
  • PYTHONPATH=/usr/local/Ascend/ascend-toolkit/latest/python/site-packages:src pytest -q tests/mindtorch_v2/test_hccl_all_to_all_single_invalid_splits_multicard.py tests/mindtorch_v2/test_hccl_all_to_all_single_async_unequal_multicard.py
  • PYTHONPATH=src pytest -q tests/mindtorch_v2/contract/test_schema_registration_order.py tests/mindtorch_v2/contract/test_schema_coverage.py

Notes

  • HCCL multicard checks were validated on local Ascend hardware; CI without Ascend devices may not exercise these paths.

@lvyufeng lvyufeng merged commit 6e859f2 into candle-org:master Mar 8, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant