Fixed a bug in the channel that was not detecting out of seq messages #663
+5
−36
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D79044526 |
vidhyav
added a commit
to vidhyav/monarch
that referenced
this pull request
Jul 28, 2025
…pytorch-labs#663) Summary: This was because we need to check for out of seq messages only if it is not the first message of the connection (even if it is for subsequent sessions). Differential Revision: D79044526
vidhyav
force-pushed
the
export-D79044526
branch
from
9717486 to
bbc9258
Compare
vidhyav
added a commit
to vidhyav/monarch
that referenced
this pull request
Jul 28, 2025
…pytorch-labs#663) Summary: This was because we need to check for out of seq messages only if it is not the first message of the connection (even if it is for subsequent sessions). Differential Revision: D79044526
vidhyav
force-pushed
the
export-D79044526
branch
from
bbc9258 to
f0b8b45
Compare
|
This pull request was exported from Phabricator. Differential Revision: D79044526 |
vidhyav
added a commit
to vidhyav/monarch
that referenced
this pull request
Jul 28, 2025
…pytorch-labs#663) Summary: Pull Request resolved: pytorch-labs#663 This was because we need to check for out of seq messages only if it is not the first message of the connection (even if it is for subsequent sessions). Differential Revision: D79044526
vidhyav
force-pushed
the
export-D79044526
branch
from
f0b8b45 to
ab59d0e
Compare
|
This pull request was exported from Phabricator. Differential Revision: D79044526 |
vidhyav
added a commit
to vidhyav/monarch
that referenced
this pull request
Jul 28, 2025
…pytorch-labs#663) Summary: Pull Request resolved: pytorch-labs#663 1) This was because we need to check for out of seq messages only if it is not the first message of the connection (even if it is for subsequent sessions). 2) There was another bug in which we don't exactly mutate session state into the session manager correctly. We clone the session state and don't mutate. This particular case is NOT taken care of the tcp_reconnect test: in this test, we shut down the entire server and reenter the listen loop thereby resetting the entire session manager. The only way we can test (2) is if we can disconnect the underlying connection and reconnect altogether while the listen stays alive. Differential Revision: D79044526
vidhyav
force-pushed
the
export-D79044526
branch
from
ab59d0e to
6682e28
Compare
vidhyav
added a commit
to vidhyav/monarch
that referenced
this pull request
Jul 28, 2025
…pytorch-labs#663) Summary: 1) This was because we need to check for out of seq messages only if it is not the first message of the connection (even if it is for subsequent sessions). 2) There was another bug in which we don't exactly mutate session state into the session manager correctly. We clone the session state and don't mutate. This particular case is NOT taken care of the tcp_reconnect test: in this test, we shut down the entire server and reenter the listen loop thereby resetting the entire session manager. The only way we can test (2) is if we can disconnect the underlying connection and reconnect altogether while the listen stays alive. Differential Revision: D79044526
vidhyav
force-pushed
the
export-D79044526
branch
from
6682e28 to
2a97f6d
Compare
|
This pull request was exported from Phabricator. Differential Revision: D79044526 |
vidhyav
force-pushed
the
export-D79044526
branch
from
2a97f6d to
fd2696c
Compare
vidhyav
added a commit
to vidhyav/monarch
that referenced
this pull request
Jul 29, 2025
…pytorch-labs#663) Summary: 1) This was because we need to check for out of seq messages only if it is not the first message of the connection (even if it is for subsequent sessions). 2) There was another bug in which we don't exactly mutate session state into the session manager correctly. We clone the session state and don't mutate. This particular case is NOT taken care of the tcp_reconnect test: in this test, we shut down the entire server and reenter the listen loop thereby resetting the entire session manager. The only way we can test (2) is if we can disconnect the underlying connection and reconnect altogether while the listen stays alive. Differential Revision: D79044526
vidhyav
force-pushed
the
export-D79044526
branch
from
fd2696c to
d6d8df2
Compare
vidhyav
added a commit
to vidhyav/monarch
that referenced
this pull request
Jul 29, 2025
…pytorch-labs#663) Summary: 1) This was because we need to check for out of seq messages only if it is not the first message of the connection (even if it is for subsequent sessions). 2) There was another bug in which we don't exactly mutate session state into the session manager correctly. We clone the session state and don't mutate. This particular case is NOT taken care of the tcp_reconnect test: in this test, we shut down the entire server and reenter the listen loop thereby resetting the entire session manager. The only way we can test (2) is if we can disconnect the underlying connection and reconnect altogether while the listen stays alive. Differential Revision: D79044526
|
This pull request was exported from Phabricator. Differential Revision: D79044526 |
vidhyav
added a commit
to vidhyav/monarch
that referenced
this pull request
Jul 29, 2025
…pytorch-labs#663) Summary: Pull Request resolved: pytorch-labs#663 1) This was because we need to check for out of seq messages only if it is not the first message of the connection (even if it is for subsequent sessions). 2) There was another bug in which we don't exactly mutate session state into the session manager correctly. We clone the session state and don't mutate. This particular case is NOT taken care of the tcp_reconnect test: in this test, we shut down the entire server and reenter the listen loop thereby resetting the entire session manager. The only way we can test (2) is if we can disconnect the underlying connection and reconnect altogether while the listen stays alive. Differential Revision: D79044526
vidhyav
force-pushed
the
export-D79044526
branch
from
d6d8df2 to
b0a21c7
Compare
|
This pull request was exported from Phabricator. Differential Revision: D79044526 |
vidhyav
added a commit
to vidhyav/monarch
that referenced
this pull request
Jul 29, 2025
…pytorch-labs#663) Summary: Pull Request resolved: pytorch-labs#663 1) This was because we need to check for out of seq messages only if it is not the first message of the connection (even if it is for subsequent sessions). 2) There was another bug in which we don't exactly mutate session state into the session manager correctly. We clone the session state and don't mutate. This particular case is NOT taken care of the tcp_reconnect test: in this test, we shut down the entire server and reenter the listen loop thereby resetting the entire session manager. The only way we can test (2) is if we can disconnect the underlying connection and reconnect altogether while the listen stays alive. Differential Revision: D79044526
vidhyav
force-pushed
the
export-D79044526
branch
2 times, most recently
from
ed44d80 to
9576e50
Compare
vidhyav
added a commit
to vidhyav/monarch
that referenced
this pull request
Jul 29, 2025
…pytorch-labs#663) Summary: 1) This was because we need to check for out of seq messages only if it is not the first message of the connection (even if it is for subsequent sessions). 2) There was another bug in which we don't exactly mutate session state into the session manager correctly. We clone the session state and don't mutate. This particular case is NOT taken care of the tcp_reconnect test: in this test, we shut down the entire server and reenter the listen loop thereby resetting the entire session manager. The only way we can test (2) is if we can disconnect the underlying connection and reconnect altogether while the listen stays alive. Differential Revision: D79044526
vidhyav
added a commit
to vidhyav/monarch
that referenced
this pull request
Jul 29, 2025
…pytorch-labs#663) Summary: 1) This was because we need to check for out of seq messages only if it is not the first message of the connection (even if it is for subsequent sessions). 2) There was another bug in which we don't exactly mutate session state into the session manager correctly. We clone the session state and don't mutate. This particular case is NOT taken care of the tcp_reconnect test: in this test, we shut down the entire server and reenter the listen loop thereby resetting the entire session manager. The only way we can test (2) is if we can disconnect the underlying connection and reconnect altogether while the listen stays alive. Differential Revision: D79044526
vidhyav
force-pushed
the
export-D79044526
branch
from
9576e50 to
2162ba4
Compare
|
This pull request was exported from Phabricator. Differential Revision: D79044526 |
vidhyav
added a commit
to vidhyav/monarch
that referenced
this pull request
Jul 29, 2025
…pytorch-labs#663) Summary: Pull Request resolved: pytorch-labs#663 Now, we break the session if we detect an out of seq message. This was disabled because of a failing test. But it turns out the test is wrong. The test stops NetRx but that should kill NetTx because netrx failure results in the entire session state nullified on the receiver side. Differential Revision: D79044526
vidhyav
force-pushed
the
export-D79044526
branch
from
2162ba4 to
7ad4261
Compare
|
This pull request was exported from Phabricator. Differential Revision: D79044526 |
vidhyav
added a commit
to vidhyav/monarch
that referenced
this pull request
Jul 29, 2025
…pytorch-labs#663) Summary: Pull Request resolved: pytorch-labs#663 Now, we break the session if we detect an out of seq message. This was disabled because of a failing test. But it turns out the test is wrong. The test stops NetRx but that should kill NetTx because netrx failure results in the entire session state nullified on the receiver side. Differential Revision: D79044526
vidhyav
force-pushed
the
export-D79044526
branch
2 times, most recently
from
4c50da7 to
b79a68d
Compare
vidhyav
added a commit
to vidhyav/monarch
that referenced
this pull request
Jul 31, 2025
…pytorch-labs#663) Summary: Now, we break the session if we detect an out of seq message. This was disabled because of a failing test. But it turns out the test is wrong. The test stops NetRx but that should kill NetTx because netrx failure results in the entire session state nullified on the receiver side. Reviewed By: pzhan9, mariusae Differential Revision: D79044526
vidhyav
force-pushed
the
export-D79044526
branch
from
b79a68d to
e0e7fe2
Compare
vidhyav
added a commit
to vidhyav/monarch
that referenced
this pull request
Jul 31, 2025
…pytorch-labs#663) Summary: Now, we break the session if we detect an out of seq message. This was disabled because of a failing test. But it turns out the test is wrong. The test stops NetRx but that should kill NetTx because netrx failure results in the entire session state nullified on the receiver side. Reviewed By: pzhan9, mariusae Differential Revision: D79044526
|
This pull request was exported from Phabricator. Differential Revision: D79044526 |
vidhyav
force-pushed
the
export-D79044526
branch
from
e0e7fe2 to
9297e9e
Compare
vidhyav
added a commit
to vidhyav/monarch
that referenced
this pull request
Jul 31, 2025
…pytorch-labs#663) Summary: Pull Request resolved: pytorch-labs#663 Now, we break the session if we detect an out of seq message. This was disabled because of a failing test. But it turns out the test is wrong. The test stops NetRx but that should kill NetTx because netrx failure results in the entire session state nullified on the receiver side. Reviewed By: pzhan9, mariusae Differential Revision: D79044526
…pytorch-labs#663) Summary: Pull Request resolved: pytorch-labs#663 Now, we break the session if we detect an out of seq message. This was disabled because of a failing test. But it turns out the test is wrong. The test stops NetRx but that should kill NetTx because netrx failure results in the entire session state nullified on the receiver side. Reviewed By: pzhan9, mariusae Differential Revision: D79044526
|
This pull request was exported from Phabricator. Differential Revision: D79044526 |
vidhyav
force-pushed
the
export-D79044526
branch
from
9297e9e to
0dd142d
Compare
|
This pull request has been merged in 434137a. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
This was because we need to check for out of seq messages
only if it is not the first message of the connection (even if it is
for subsequent sessions).
Differential Revision: D79044526