Skip to content

fix(core): fall back to non-multicast zenoh discovery when scouting bind fails - #2850

Draft
bobdingAI wants to merge 1 commit into
mainfrom
fix/zenoh-multicast-fallback
Draft

fix(core): fall back to non-multicast zenoh discovery when scouting bind fails#2850
bobdingAI wants to merge 1 commit into
mainfrom
fix/zenoh-multicast-fallback

Conversation

@bobdingAI

@bobdingAI bobdingAI commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Summary

open_zenoh_session_with_listen (libraries/core/src/topics.rs, called from the
daemon at binaries/daemon/src/lib.rs:1225) opens the tuned zenoh session and,
on failure, retried with a plain zenoh::Config::default(). The default keeps
multicast scouting enabled, so when the failure's cause is the scouting
socket failing to bind its multicast group (224.0.0.224:7446), the retry
re-attempts the identical bind, fails again, and the dataflow aborts at startup.

This bites when dora shares one network namespace with a busy DDS/ROS 2 multicast
graph — i.e. dev containers / CI / any co-located docker-netns setup (the
existing comment in topics.rs already notes multicast scouting is "broken in
dev containers and many CI environments"). Reproduced in docker/ros2dev (dora +
ROS 2 in one netns) alongside a ~25-participant DDS graph.

Fix

Disable multicast scouting on the fallback retry only. Gossip + explicit
connect endpoints (DORA_ZENOH_CONNECT) still cover discovery for co-located and
connect-configured setups, so degrading to non-multicast keeps dora running where
it would otherwise fail outright. The primary open is unchanged — normal
(uncontended) environments still get multicast by default and the fallback never
triggers. It's strictly safer than today: the fallback runs only after the
primary already failed to bind multicast, so a multicast-dependent setup wasn't
getting multicast on that attempt anyway.

Scope — what this does and does NOT fix

This is the dora daemon zenoh-transport layer only. It does not touch the
ROS 2 bridge's DDS path (Ros2Transport::Dds / Ros2Context default), which
opens no zenoh session at all.

There is a separate discovery bug with a confusingly similar ~50% rate: the
ROS 2 bridge's RustDDS discovery going silently empty under DDS domain
contention (process healthy, subscription just receives nothing — no bind error,
no daemon required). This PR does not address that; it lives at the
ros2_client/rustdds layer, not in dora's zenoh session. The two share an
environmental cause (multicast contention) but are distinct code paths.

Host deployments where dora and ROS 2 do not share a netns are not exposed to
the failure this PR fixes (measured: 10/10 dataflow starts with and without
ZENOH_CONFIG against a live 32-node graph, zero bind errors) — for those, the
DDS-layer bug above is the relevant one.

Validation

rust-ros2-dataflow bridge example vs a ~25-participant DDS graph, 12 runs each,
in docker/ros2dev (dora + ROS 2 share one netns):

Condition Dataflow starts
clean graph, multicast on (main) 11/12
busy graph, multicast on (main) 0/12 — all fail on 224.0.0.224:7446 Address already in use
busy graph, this PR 12/12 — primary fails, fallback logs retrying with multicast scouting disabled, dataflow finishes
clean graph, this PR unchanged (fallback never triggers)

cargo fmt / cargo clippy -p dora-core -- -D warnings clean.

A deterministic unit test isn't practical (triggering the fallback needs a
contended-multicast environment where the primary bind fails), so this is
validated by the reproduction above.

Workaround (no rebuild)

Set ZENOH_CONFIG to a config with scouting: { multicast: { enabled: false } },
or set DORA_ZENOH_CONNECT, to opt out of multicast scouting directly.

…ind fails

`open_zenoh_session_with_listen` opens the tuned zenoh session, and on failure
retried with a plain `zenoh::Config::default()`. But the default keeps
multicast scouting enabled, so when the *cause* of the failure is the scouting
socket failing to bind its multicast group (`224.0.0.224:7446`), the retry
re-attempts the identical bind, fails again, and the whole dataflow aborts at
startup.

This happens whenever a busy DDS/ROS 2 multicast graph shares the network with
dora — e.g. running the ROS 2 bridge alongside a full TurtleBot3 / nav2 stack
(dozens of FastDDS participants). Discovery on the DDS side is fine; it's
dora's own zenoh scouting bind that fails, so the dataflow intermittently
never starts.

Disable multicast scouting on the fallback retry: gossip + explicit connect
endpoints (`DORA_ZENOH_CONNECT`) still cover discovery for co-located and
connect-configured setups, so degrading to non-multicast keeps dora running
where it would otherwise fail outright. The primary open is unchanged, so
normal (uncontended) environments still get multicast by default.

Repro (dora ROS2 bridge example vs a 25-participant DDS graph, x12 runs):
  - busy graph, multicast on (before): 0/12 started
  - busy graph, multicast on (after):  12/12 started
  - clean graph (after): unchanged, fallback never triggers

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@trunk-io

trunk-io Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Merging to main in this repository is managed by Trunk.

  • To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

Copy link
Copy Markdown
Collaborator

🤖 Fully automated review by Claude — this review was generated end-to-end by an automated agent with no human vetting. Treat it accordingly.

No issues found. Disabling multicast scouting only on the fallback retry is a strictly-safer degradation: the fallback runs only after the primary tuned open already failed to bind, so a multicast-dependent setup wasn't getting multicast on that attempt anyway, and gossip / explicit DORA_ZENOH_CONNECT endpoints still cover discovery. insert_json5("scouting/multicast/enabled", "false") passes a valid JSON5 boolean, and its error path only warns rather than aborting the retry, so a future zenoh key rename degrades gracefully. The primary open is untouched, so uncontended environments are unaffected. Agreed that a deterministic unit test isn't practical here given it requires a contended-multicast bind failure.


Generated by Claude Code

Copy link
Copy Markdown
Collaborator

Automated review.

The fallback direction is reasonable for the co-located / dev-container multicast-contention case this targets, and the insert_json5("scouting/multicast/enabled", "false") call matches the existing usage at topics.rs:347. One correctness gap is worth tightening, though:

The retry disables multicast unconditionally, but the Err(err) arm (libraries/core/src/topics.rs:446) is reached on any zenoh::open failure — not only a multicast-bind failure. So the "strictly safer / wasn't getting multicast anyway" rationale only holds when the multicast bind was the actual cause. If the primary open fails for an unrelated reason in a multi-daemon deployment that genuinely relies on multicast scouting, the old Config::default() fallback would still recover via multicast, whereas this one disables it. And because the fallback is a bare Config::default(), it also carries no listener and none of the coordinator_addr connect endpoints that the tuned config set — so the result is a silently-partitioned session rather than the loud startup failure it replaces, which is harder to debug.

Gating the multicast-disable on the error actually being the 224.0.0.224:7446 bind failure — or re-applying the connect endpoints on the fallback so gossip has a seed — would close the gap. Not blocking, and the change is still a net improvement for the target scenario.


🤖 This is a fully automated review by Claude (Claude Code). No human has vetted these findings — please confirm before acting on them.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants