Skip to content

fix(ros): make the interop gate count interop tests - #276

Open
YuanYuYuan wants to merge 2 commits into
mainfrom
fix/interop-gate-counts-interop
Open

fix(ros): make the interop gate count interop tests#276
YuanYuYuan wants to merge 2 commits into
mainfrom
fix/interop-gate-counts-interop

Conversation

@YuanYuYuan

@YuanYuYuan YuanYuYuan commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Summary

The interop gate added in #271 does not count interop tests. It asserts a non-zero package total, then reports that total as the interop count. A healthy jazzy run reads 125 tests run: 125 passed, and the gate prints 125 ROS interop tests ran against rmw_zenoh_cpp. Only 41 of those are interop tests. hiroz-tests also holds lifecycle, cache, hu_meter, parameter_tests and other non-interop suites. Deleting every interop test still leaves the gate printing a confident number. #271 removed that defect, and its own fix reintroduced it.

Interop test functions per suite, counted at 47ddff36d2d99cc1bb36e7791d70de0cc51b6424:

suite tests
pubsub_interop 14
demo_nodes 9
service_interop 7
type_description_interop 4 (absent on Humble)
action_interop 4
parameter_interop 3
total 41 (37 on Humble)

What this PR does

One file changes: scripts/test-ros.nu.

change effect
Count the interop binaries by name; require each to be present A run of only non-interop tests fails and names the missing suites, instead of reporting them as interop
Fail when the output contains ros2 CLI not available A run whose tests skipped or died for a missing ros2 CLI is no longer a pass
Gate the attempt that decided the outcome The retry path re-ran nextest, but the gate parsed the first attempt's output
Drop the 0 tests run branch Unreachable: nextest's --no-tests default is to fail
Print a per-suite breakdown Replaces the single aggregate number

The required suite list is distro-dependent. type_description_interop.rs is #![cfg(not(ros_humble))], because Humble has no type description service to interoperate with. The gate therefore requires that suite on every distro except Humble, and the other five everywhere. It omits python_interop deliberately: the python-interop feature gates that suite, and this command does not enable it.

The self-skip check is not uniform across suites. The shared message is what makes it detectable.

suite guard behaviour with no usable ros2
pubsub_interop check_ros2_available, at two call sites prints Skipping …: ros2 CLI not available and returns — passes, and counts
demo_nodes, service_interop, parameter_interop, type_description_interop same helper panic! with the same substring — already fails the run
action_interop none no guard at all

The gate greps for the shared substring, so it catches both spellings.

Note

Reading, not execution, established the 0 tests run branch as unreachable. The hiroz-union invocation in test.yml passes --no-tests=warn precisely because the default fails the job.

Evidence

check commit result
The new suite-presence check fires on a genuinely empty suite 8e68984af7abd15e50aea6981085e5cacdc7671c ROS Tests (clippy-rmw, interop) (humble) failed with ROS interop suites produced no tests: type_description_interop. The old gate had reported that same Humble run as 116 ROS interop tests ran
Full CI on the head commit 47ddff36d2d99cc1bb36e7791d70de0cc51b6424 28 checks green, 0 failed, 0 pending; license/cla success
Gate logic against recorded nextest output 47ddff36d2d99cc1bb36e7791d70de0cc51b6424 see the table below — a manual check, not an automated test

Humble legitimately has no type_description_interop, so the head commit exempts that suite there. That Humble failure is the proof that the detector detects, rather than merely never firing.

Gate logic fed recorded nextest output:

input result
healthy jazzy output, all six suites pass — prints the 41-test breakdown
healthy humble output, five suites pass
84 passing tests, no interop suites fail — names the missing suites
healthy output plus one ros2 CLI not available line fail
empty output fail — no summary line

Breaking changes

tag what changes who is affected before → after action
BC1 scripts/test-ros.nu fails when interop suites are absent or self-skipped anyone running the script locally without a working ros2 CLI silent green → explicit failure install and source ROS 2 before running, or run cargo nextest directly

No library, API or message-format change. The change is CI-only, and the gate becomes strictly stricter.

Coverage this does not have

These gaps do not block the fix. They bound what green CI proves.

tag gap detail
G1 The four Interop tests with ROS 2 <distro> jobs are ungated .github/workflows/test.yml runs cargo nextest run directly and never calls scripts/test-ros.nu. Only the ROS Tests (clippy-rmw, interop) jobs in ci.yml invoke the gated script
G2 --no-tests=warn keeps one path green by design The hiroz-union invocation in test.yml warns instead of failing on a zero-test run
G3 The suite list is hand-maintained A new interop suite is not required until someone adds its name to interop_suites in test-ros.nu
G4 The gate itself is untested Its behaviour was checked by hand against recorded output, not by a test that runs in CI
G5 Per-suite counts are not asserted Only presence is required, so a suite that shrinks from 14 tests to 1 still passes
G6 A broken rmw_zenoh_cpp still passes the availability guard The helper is Command::new("ros2").arg("--version").output().is_ok(), which is true if the process merely spawned. #271 named this failure mode and did not close it. This PR does not close it either

The gate added in #271 asserted a non-zero total and then reported it as
"N ROS interop tests ran against rmw_zenoh_cpp". That total is the whole
hiroz-tests package. On a healthy run it printed 125, of which only 41 are
interop -- the rest are lifecycle, cache, parameter_tests and friends.
Deleting every interop test would still have printed a confident number.

That is the defect #271 existed to remove, reintroduced in the fix.

Three changes:

- Count the six interop binaries by name and require each to be present.
  A run with 84 passing non-interop tests now fails instead of reporting
  "84 ROS interop tests ran".
- Fail when any test self-skipped for a missing ros2 CLI. Those return
  early, still pass, and are still counted -- a pass with no interop in it.
  #271 named this failure mode in its own description and did not close it.
- Gate the run that decided the outcome. The retry path re-ran nextest but
  the gate still parsed the first attempt, so a passing retry was validated
  against the discarded output, and a first attempt that died before
  producing a summary failed an otherwise-green run.

Also drops the `0 tests run` branch: cargo-nextest 0.9.138 defaults
--no-tests to fail, so it was unreachable. The repo already knew this --
test.yml passes --no-tests=warn precisely because the default fails.

Proven in four directions: healthy output passes; 84 non-interop tests
fail; a self-skip line fails; empty output fails.
The suite is `#![cfg(not(ros_humble))]` -- Humble has no type description
service to interoperate with -- so it compiles to an empty binary there and
contributes no tests. Requiring it on every distro failed a healthy humble
run. Every other interop suite stays mandatory everywhere.
@YuanYuYuan
YuanYuYuan force-pushed the fix/interop-gate-counts-interop branch from 47ddff3 to c1ebcea Compare August 14, 2026 18:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant