Skip to content

Commit 34a2e00

Browse files
committed
fix(config): two hiroz nodes on two hosts delivered nothing to each other
Two hiroz processes on two hosts, with an `hu router` both of them reached, exchanged nothing and reported nothing. Measured, 20 messages published per cell, subscriber started 3s ahead, router asserted alive throughout: | subscriber | publisher | published | received | |---|---|---|---| | shipped default | shipped default | 20 | 0 | | with_mode("client") | with_mode("client") | 20 | 20 | | with_mode("client") | shipped default | 20 | 20 | | shipped default | with_mode("client") | 20 | 20 | Both sessions opened, the publisher published all 20, the router stayed up, and no error surfaced anywhere. Only the configuration a user gets by writing `ZContextBuilder::default()` failed. The shipped session config asked for three things that cannot hold together: `mode: "peer"`, `scouting/multicast/enabled: false`, and `listen/endpoints: ["tcp/localhost:0"]`. A zenoh router does not route between two peers connected to it; the setting that used to make it do so, `routing.router.peers_failover_brokering`, is deprecated in zenoh 1.9 and has no effect — the router logged exactly that on every start, and this crate's own router config was still setting it. So the two peers had to link directly to each other, and a peer that advertises only `tcp/localhost:0` cannot be dialled from another host. On one machine they link over loopback, which is why every local run looked fine. Fix: listen on `tcp/[::]:0`, which is what rmw_zenoh_cpp's own session config leaves it at, and drop the dead `peers_failover_brokering` override. Same 2x2 after the change: 20 received in every cell, and the router's deprecation warning count drops from 1 to 0. Candidates rejected, with the measurement that rejected them: - Default the session to `client`. It fixes the general case, including two nodes that can neither dial the other. It also breaks router-less operation: a client session FAILS TO OPEN with no router, where a peer opens. Measured on the whole `hiroz` suite with no router running — baseline 423 passed / 0 failed, client-mode default 8 targets failed (tests/service 5, tests/shm 7, and more). Those tests are single-session and legitimately need no network. Rejected on that blast radius; it is a call for the maintainers, not a defect fix. - Gossip multihop / peer autoconnect. Does not address the cause: discovery already worked, the advertised locator was the unreachable part. It would also undo the tuning next to it ("greater-zid to avoid redundant connections", "peers send gossip only to router"). - Configure the router to broker. Not available: zenoh 1.9 deprecated the setting with no replacement, which is the root of this. What this fix does NOT cover, stated plainly: two peers that can neither accept an inbound connection still cannot reach each other, because nothing relays for them. That case needs client mode and its cost. Blast radius of what landed: nodes now accept inbound connections on every interface instead of loopback only — the upstream rmw_zenoh_cpp default, and the price of peer mode working off-host. Whole `hiroz` suite after the change: 425 passed, 0 failed. Regression test: crates/hiroz/tests/two_session_default_config.rs. Every pub/sub test in this crate put publisher and subscriber on ONE ZContext, so the message never crossed a session boundary; the integration suite that uses two endpoints builds them through create_hiroz_context_with_endpoint, which calls .with_mode("client") and so tests the row that worked. The shipped default had no two-session test at all. The new file adds one, plus the assertion that actually fails at the broken revision: a shipped-default session must advertise a listen endpoint another host can dial. Reproducing the delivery failure itself needs two hosts, which the suite cannot arrange — the two-session delivery test passes on one machine either way, and its doc comment says so.
1 parent 1fe6b3d commit 34a2e00

2 files changed

Lines changed: 246 additions & 9 deletions

File tree

crates/hiroz/src/config.rs

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
//!
77
//! # Architecture
88
//! - Common overrides: 10 settings shared between router and session
9-
//! - Router-specific: 5 settings unique to router mode
9+
//! - Router-specific: 4 settings unique to router mode
1010
//! - Session-specific: 6 settings unique to peer mode
1111
//!
1212
//! # Example
@@ -195,7 +195,7 @@ fn common_overrides() -> &'static [ConfigOverride] {
195195
&COMMON
196196
}
197197

198-
/// Router-specific overrides (5 settings)
198+
/// Router-specific overrides (4 settings)
199199
fn router_specific_overrides() -> &'static [ConfigOverride] {
200200
static ROUTER_SPECIFIC: LazyLock<Vec<ConfigOverride>> = LazyLock::new(|| {
201201
vec![
@@ -214,11 +214,6 @@ fn router_specific_overrides() -> &'static [ConfigOverride] {
214214
value: serde_json::json!([]),
215215
reason: "Router does not connect to other endpoints (empty list)",
216216
},
217-
ConfigOverride {
218-
key: "routing/router/peers_failover_brokering",
219-
value: serde_json::json!(false),
220-
reason: "Changed from true to false - unnecessary when peers connect directly, reduces overhead",
221-
},
222217
ConfigOverride {
223218
key: "transport/link/tx/queue/congestion_control/block/wait_before_close",
224219
value: serde_json::json!(5000000),
@@ -246,8 +241,13 @@ fn session_specific_overrides() -> &'static [ConfigOverride] {
246241
},
247242
ConfigOverride {
248243
key: "listen/endpoints",
249-
value: serde_json::json!(["tcp/localhost:0"]),
250-
reason: "Accept connections only from localhost - external traffic routed via router",
244+
value: serde_json::json!(["tcp/[::]:0"]),
245+
reason: "Listen on every interface, as rmw_zenoh_cpp's session config does. \
246+
This was \"tcp/localhost:0\", which made two ROS nodes on two hosts \
247+
unable to exchange anything: a peer that advertises only a loopback \
248+
locator cannot be dialled from another host, and the router will not \
249+
relay between two peers on their behalf. \
250+
See tests/two_session_default_config.rs",
251251
},
252252
ConfigOverride {
253253
key: "scouting/gossip/autoconnect_strategy",
Lines changed: 237 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,237 @@
1+
//! Two independent sessions in the **shipped default** configuration.
2+
//!
3+
//! # The failure this pins
4+
//!
5+
//! Two `hiroz` processes on two hosts, with a router both of them reached,
6+
//! delivered nothing to each other and reported nothing:
7+
//!
8+
//! | subscriber | publisher | published | received |
9+
//! |---|---|---|---|
10+
//! | shipped default | shipped default | 20 | **0** |
11+
//! | `with_mode("client")` | `with_mode("client")` | 20 | 20 |
12+
//! | `with_mode("client")` | shipped default | 20 | 20 |
13+
//! | shipped default | `with_mode("client")` | 20 | 20 |
14+
//!
15+
//! Both sessions opened, the publisher published all 20, the router stayed up
16+
//! throughout, and no error surfaced anywhere. Only the configuration every
17+
//! real user gets — the one with no `with_mode` call — failed.
18+
//!
19+
//! The shipped session config asked for three things that cannot hold
20+
//! together: `mode: "peer"`, `scouting/multicast/enabled: false`, and
21+
//! `listen/endpoints: ["tcp/localhost:0"]`. A zenoh router does **not** route
22+
//! between two peers connected to it. The setting that used to make it do so,
23+
//! `routing.router.peers_failover_brokering`, is deprecated in zenoh 1.9 and
24+
//! has no effect — the router logs exactly that on every start, and hiroz's
25+
//! own router config was still setting it. So the two peers had to link
26+
//! directly to each other; and a peer that advertises only `tcp/localhost:0`
27+
//! is not dialable from another host, so they never did.
28+
//!
29+
//! Changing the listen endpoint to `tcp/[::]:0` — which is what
30+
//! rmw_zenoh_cpp's own session config leaves it at — makes the same 2×2 read
31+
//! 20 received in every cell.
32+
//!
33+
//! # Why this was never caught
34+
//!
35+
//! Every pub/sub test in this crate puts the publisher and the subscriber on
36+
//! **one** `ZContext`, so the message never crosses a session boundary and the
37+
//! network configuration is not exercised at all. The integration suite that
38+
//! does use two endpoints builds its contexts with
39+
//! `create_hiroz_context_with_endpoint`, which calls `.with_mode("client")` —
40+
//! it opts out of the shipped default, and client↔anything is the row of the
41+
//! table that works.
42+
//!
43+
//! # What each test here can and cannot prove
44+
//!
45+
//! [`two_default_sessions_exchange_a_message`] is the two-session delivery
46+
//! test whose absence hid this. On a single host it passes both before and
47+
//! after the fix, because two peers on one machine *can* dial each other's
48+
//! loopback locator — that is precisely why running the suite on one machine
49+
//! showed nothing wrong. It is kept as the guard for the general path.
50+
//!
51+
//! [`shipped_default_advertises_a_reachable_listen_endpoint`] is the test that
52+
//! fails at the broken revision. It asserts the property the cross-host
53+
//! measurement above turned on: a session in the shipped default configuration
54+
//! must advertise a locator that something other than its own machine can
55+
//! connect to. Reproducing the delivery failure itself needs two hosts, which
56+
//! this suite has no way to arrange.
57+
58+
use std::time::Duration;
59+
60+
use hiroz::{
61+
Builder, TypeHash,
62+
config::{RouterConfigBuilder, session_config},
63+
context::{ZContext, ZContextBuilder},
64+
ros_msg::MessageTypeInfo,
65+
};
66+
use serde::{Deserialize, Serialize};
67+
use zenoh::Wait;
68+
69+
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Default)]
70+
struct Ping {
71+
count: u64,
72+
}
73+
74+
impl MessageTypeInfo for Ping {
75+
fn type_name() -> &'static str {
76+
"test_msgs::msg::dds_::Ping_"
77+
}
78+
79+
fn type_hash() -> TypeHash {
80+
TypeHash::zero()
81+
}
82+
}
83+
84+
impl hiroz::ros_msg::WithTypeInfo for Ping {}
85+
86+
impl hiroz::msg::ZMessage for Ping {
87+
type Serdes = hiroz::msg::SerdeCdrSerdes<Ping>;
88+
}
89+
90+
/// The host part of a zenoh TCP endpoint string, e.g. `tcp/[::]:0` -> `[::]`.
91+
fn host_of(endpoint: &str) -> &str {
92+
let after_proto = endpoint.split_once('/').map_or(endpoint, |(_, rest)| rest);
93+
match after_proto.rsplit_once(':') {
94+
Some((host, _port)) => host,
95+
None => after_proto,
96+
}
97+
}
98+
99+
/// Whether a listen endpoint binds somewhere only this machine can reach.
100+
///
101+
/// `localhost` is included deliberately: it is what the broken revision used,
102+
/// and it resolves to a loopback address on every host this runs on.
103+
fn is_loopback_only(endpoint: &str) -> bool {
104+
let host = host_of(endpoint);
105+
host == "localhost" || host == "[::1]" || host == "::1" || host.starts_with("127.")
106+
}
107+
108+
/// A session built the way a user gets one: `ZContextBuilder::default()`, with
109+
/// no `with_mode` call, and only the router endpoint repointed at this test's
110+
/// router instead of the fixed `tcp/localhost:7447`.
111+
fn shipped_default_context(port: u16) -> ZContext {
112+
ZContextBuilder::default()
113+
.with_router_endpoint(format!("tcp/127.0.0.1:{port}"))
114+
.expect("router endpoint")
115+
.build()
116+
.expect("a session in the shipped default configuration failed to open")
117+
}
118+
119+
/// A router on a free port, already accepting connections when this returns.
120+
fn start_router() -> (zenoh::Session, u16) {
121+
let port = {
122+
let listener = std::net::TcpListener::bind("127.0.0.1:0").expect("bind port 0");
123+
listener.local_addr().expect("local_addr").port()
124+
};
125+
126+
let config = RouterConfigBuilder::new()
127+
.with_listen_port(port)
128+
.build_config()
129+
.expect("router config");
130+
let session = zenoh::open(config).wait().expect("router failed to open");
131+
132+
// Wait for the listener rather than sleeping blind: a session that dials
133+
// before the router accepts would report a connect failure as a delivery
134+
// failure, which is a different defect wearing this one's clothes.
135+
let addr = std::net::SocketAddr::from(([127, 0, 0, 1], port));
136+
let mut accepted = false;
137+
for _ in 0..40 {
138+
if std::net::TcpStream::connect_timeout(&addr, Duration::from_millis(50)).is_ok() {
139+
accepted = true;
140+
break;
141+
}
142+
std::thread::sleep(Duration::from_millis(50));
143+
}
144+
assert!(
145+
accepted,
146+
"router never accepted a connection on port {port}"
147+
);
148+
149+
(session, port)
150+
}
151+
152+
/// Two *independent* sessions in the shipped default configuration exchange a
153+
/// message. Two contexts, not two nodes on one context: a single context would
154+
/// carry the message inside one zenoh session and prove nothing about the
155+
/// network configuration.
156+
#[test]
157+
fn two_default_sessions_exchange_a_message() {
158+
let (_router, port) = start_router();
159+
160+
let publisher_ctx = shipped_default_context(port);
161+
let subscriber_ctx = shipped_default_context(port);
162+
163+
let sub_node = subscriber_ctx
164+
.create_node("two_session_listener")
165+
.build()
166+
.expect("subscriber node");
167+
let subscriber = sub_node
168+
.create_sub::<Ping>("/two_session_default")
169+
.build()
170+
.expect("subscriber");
171+
172+
let pub_node = publisher_ctx
173+
.create_node("two_session_talker")
174+
.build()
175+
.expect("publisher node");
176+
let publisher = pub_node
177+
.create_pub::<Ping>("/two_session_default")
178+
.build()
179+
.expect("publisher");
180+
181+
// Publish repeatedly rather than once: discovery is asynchronous, so a
182+
// single publish may legitimately precede the subscriber being known. The
183+
// assertion is on delivery happening within the budget, not on the first
184+
// message arriving.
185+
let mut received = 0u64;
186+
let mut published = 0u64;
187+
let deadline = std::time::Instant::now() + Duration::from_secs(20);
188+
while std::time::Instant::now() < deadline && received == 0 {
189+
publisher
190+
.publish(&Ping { count: published })
191+
.expect("publish");
192+
published += 1;
193+
if subscriber.recv_timeout(Duration::from_millis(200)).is_ok() {
194+
received += 1;
195+
}
196+
}
197+
198+
assert!(
199+
received > 0,
200+
"two sessions in the shipped default configuration exchanged nothing: \
201+
{published} published, 0 received, router up on port {port}"
202+
);
203+
}
204+
205+
/// The shipped default session config must advertise a listen endpoint that
206+
/// another host can dial.
207+
///
208+
/// This is the assertion that fails at the revision this test was written
209+
/// against, where the value was `["tcp/localhost:0"]`. Its comment there read
210+
/// "accept connections only from localhost - external traffic routed via
211+
/// router", which is a policy a **client** can hold. A peer cannot: no router
212+
/// relays between two peers, so a peer that nothing can dial has no path to
213+
/// any other peer, and it reports no error while having none.
214+
#[test]
215+
fn shipped_default_advertises_a_reachable_listen_endpoint() {
216+
let config = session_config().expect("session config");
217+
let raw = config
218+
.get_json("listen/endpoints")
219+
.expect("listen/endpoints must be set in the shipped session config");
220+
let endpoints: Vec<String> =
221+
serde_json::from_str(&raw).expect("listen/endpoints must be a list of strings");
222+
223+
assert!(
224+
!endpoints.is_empty(),
225+
"the shipped session config advertises no listen endpoint at all, so no \
226+
other host can reach a hiroz node: {raw}"
227+
);
228+
assert!(
229+
endpoints.iter().any(|e| !is_loopback_only(e)),
230+
"every listen endpoint in the shipped session config is loopback-only ({raw}), \
231+
so a hiroz node cannot be dialled from another host. Measured with two hosts \
232+
and a router both reached: 20 published, 0 received, no error reported. \
233+
A zenoh router does not relay between two peers — \
234+
routing.router.peers_failover_brokering is deprecated in zenoh 1.9 and has \
235+
no effect — so a peer nothing can dial can talk to nobody."
236+
);
237+
}

0 commit comments

Comments
 (0)