fix(ros): make the interop gate count interop tests - #276
Open
YuanYuYuan wants to merge 2 commits into
Open
Conversation
This was referenced Jul 29, 2026
The gate added in #271 asserted a non-zero total and then reported it as "N ROS interop tests ran against rmw_zenoh_cpp". That total is the whole hiroz-tests package. On a healthy run it printed 125, of which only 41 are interop -- the rest are lifecycle, cache, parameter_tests and friends. Deleting every interop test would still have printed a confident number. That is the defect #271 existed to remove, reintroduced in the fix. Three changes: - Count the six interop binaries by name and require each to be present. A run with 84 passing non-interop tests now fails instead of reporting "84 ROS interop tests ran". - Fail when any test self-skipped for a missing ros2 CLI. Those return early, still pass, and are still counted -- a pass with no interop in it. #271 named this failure mode in its own description and did not close it. - Gate the run that decided the outcome. The retry path re-ran nextest but the gate still parsed the first attempt, so a passing retry was validated against the discarded output, and a first attempt that died before producing a summary failed an otherwise-green run. Also drops the `0 tests run` branch: cargo-nextest 0.9.138 defaults --no-tests to fail, so it was unreachable. The repo already knew this -- test.yml passes --no-tests=warn precisely because the default fails. Proven in four directions: healthy output passes; 84 non-interop tests fail; a self-skip line fails; empty output fails.
The suite is `#![cfg(not(ros_humble))]` -- Humble has no type description service to interoperate with -- so it compiles to an empty binary there and contributes no tests. Requiring it on every distro failed a healthy humble run. Every other interop suite stays mandatory everywhere.
YuanYuYuan
force-pushed
the
fix/interop-gate-counts-interop
branch
from
August 14, 2026 18:23
47ddff3 to
c1ebcea
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The interop gate added in #271 does not count interop tests. It asserts a non-zero package total, then reports that total as the interop count. A healthy jazzy run reads
125 tests run: 125 passed, and the gate prints125 ROS interop tests ran against rmw_zenoh_cpp. Only 41 of those are interop tests.hiroz-testsalso holdslifecycle,cache,hu_meter,parameter_testsand other non-interop suites. Deleting every interop test still leaves the gate printing a confident number. #271 removed that defect, and its own fix reintroduced it.Interop test functions per suite, counted at
47ddff36d2d99cc1bb36e7791d70de0cc51b6424:pubsub_interopdemo_nodesservice_interoptype_description_interopaction_interopparameter_interopWhat this PR does
One file changes:
scripts/test-ros.nu.ros2 CLI not availableros2CLI is no longer a pass0 tests runbranch--no-testsdefault is to failThe required suite list is distro-dependent.
type_description_interop.rsis#![cfg(not(ros_humble))], because Humble has no type description service to interoperate with. The gate therefore requires that suite on every distro except Humble, and the other five everywhere. It omitspython_interopdeliberately: thepython-interopfeature gates that suite, and this command does not enable it.The self-skip check is not uniform across suites. The shared message is what makes it detectable.
ros2pubsub_interopcheck_ros2_available, at two call sitesSkipping …: ros2 CLI not availableand returns — passes, and countsdemo_nodes,service_interop,parameter_interop,type_description_interoppanic!with the same substring — already fails the runaction_interopThe gate greps for the shared substring, so it catches both spellings.
Note
Reading, not execution, established the
0 tests runbranch as unreachable. Thehiroz-unioninvocation intest.ymlpasses--no-tests=warnprecisely because the default fails the job.Evidence
8e68984af7abd15e50aea6981085e5cacdc7671cROS Tests (clippy-rmw, interop) (humble)failed withROS interop suites produced no tests: type_description_interop.The old gate had reported that same Humble run as116 ROS interop tests ran47ddff36d2d99cc1bb36e7791d70de0cc51b6424license/clasuccess47ddff36d2d99cc1bb36e7791d70de0cc51b6424Humble legitimately has no
type_description_interop, so the head commit exempts that suite there. That Humble failure is the proof that the detector detects, rather than merely never firing.Gate logic fed recorded nextest output:
ros2 CLI not availablelineBreaking changes
scripts/test-ros.nufails when interop suites are absent or self-skippedros2CLIcargo nextestdirectlyNo library, API or message-format change. The change is CI-only, and the gate becomes strictly stricter.
Coverage this does not have
These gaps do not block the fix. They bound what green CI proves.
Interop tests with ROS 2 <distro>jobs are ungated.github/workflows/test.ymlrunscargo nextest rundirectly and never callsscripts/test-ros.nu. Only theROS Tests (clippy-rmw, interop)jobs inci.ymlinvoke the gated script--no-tests=warnkeeps one path green by designhiroz-unioninvocation intest.ymlwarns instead of failing on a zero-test runinterop_suitesintest-ros.nurmw_zenoh_cppstill passes the availability guardCommand::new("ros2").arg("--version").output().is_ok(), which is true if the process merely spawned. #271 named this failure mode and did not close it. This PR does not close it either