Re-Export Generated Messages in rclrs #556

maspe36 · 2025-12-09T04:22:07Z

Summary

Working example of re-exporting of the generated interface crates found on the AMENT_PREFIX_PATH env var. With this change generated interface symbols are available directly from rclrs::, so for example, rclrs::std_msgs::msg::String is how you'd access the String type of the generated rust code for the std_msgs package.

This depends on ros2-rust/rosidl_rust#12 for fully relocatable symbols.

The way we find the generated code is as follows

For all entries in AMENT_PREFIX_PATH, look for a share/ directory.
For all entries in share/ look for <package>/rust/
If there is a Cargo.toml within that entry, with the metadata to re-export symbols, include! all source files within src/ such that all symbols resolve the same.

i.e. std_msgs::msg::String for a standalone crate build, is effectively the same symbol as rclrs::std_msgs::msg::String.

~~There is still some work needed here, namely I do not have tests fully passing, but I wanted to get this out there early for folks to see it as an option.~~ Tests are now passing. For now, I am simply not installing colcon-ros-cargo in CI. I believe we will need to remove the patching logic, but need to re-evaluate.

I believe this to be a viable alternative to patching, and (could) effectively eliminate the need for vendoring some messages.

Open questions

How does AMENT_PREFIX_PATH get populated? If its populated as colcon is building packages (which I believe to be the case), then this will also re-export symbols in your own workspace as well as sourced overlays.
Should this actually go into rclrs or a separate crate (i.e. ros_msgs?) By going into rclrs we eliminate the need for vendoring.
Its important to know that if we re-build rclrs, and the AMENT_PREFIX_PATH env var has changed, then the build.rs will be re-built, which will then re-build all of the generated code.

Honestly, I'm not sure if we can really get around this. Rust just really wants you to recompile the code you're using each time you invoke cargo. We can and should have everything in a colcon workspace share a target directory, but this won't stop all rebuilds. Its possible we expect compiled .rlibs in the share directory and we generate a file which has extern crate calls to force rustc to find the dependencies, but this will likely not work with any intellisense, and its overall pretty fragile IMO.

Can we get better error messages? If a user forgot to source something or expects an interface to exists but one doesn't, they are met with errors like this

  error[E0433]: failed to resolve: could not find `action_msgs` in the crate root
   --> rclrs/src/action/action_client.rs:5:6
    |
  5 |     {action_msgs::srv::CancelGoal_Response, builtin_interfaces::msg::Time},
    |      ^^^^^^^^^^^ could not find `action_msgs` in the crate root

…nd on the AMENT_PREFIX_PATH env var. Able to colcon build the entire workspace.

maspe36 · 2025-12-09T04:28:14Z

rclrs/build.rs

+    let dest_path = PathBuf::from(out_dir).join("interfaces.rs");
+
+    // TODO I would like to run rustfmt on this generated code, similar to how bindgen does it
+    fs::write(dest_path, content.clone()).expect("Failed to write interfaces.rs");


interface.rs ends up looking like this

pub mod builtin_interfaces { ... }; pub mod std_msgs { pub mod msg { use crate::builtin_interfaces; // use crate::...; etc include!("path/to/generated/code"); pub mod rmw { use crate::builtin_interfaces; // use crate::...; etc include!("path/to/generated/rmw/code"); } } }

Where all generated interface code is placed into a module based on the paths found in the AMENT_PREFIX_PATH share directories.

Interestingly enough, I also learned that the #[path = "..."] attribute for modules works with absolute paths. So that also could work, but I didn't have proper intellisense with that approach so I stuck with include!()

Is there a way to change the location of the interfaces.rs to be a bit easier to find, perhaps in the install folder?
I find its current location in an obscure build subdirectory fairly confusing and it makes it hard to inspect what is going on:

lucadv@noble:~/ws_rclrs$ find . -name "interfaces.rs" ./src/ros2_rust/target/debug/build/rclrs-d6fb61407f6fcb8c/out/interfaces.rs ./src/ros2_rust/target/debug/build/rclrs-e547ca5eac3bba00/out/interfaces.rs

This is generally bad practice for build scripts.
https://doc.rust-lang.org/cargo/reference/build-scripts.html#outputs-of-the-build-script

(TIL) In fact, this is disallowed during publishing, so I think we should leave it as is.

cargo won’t allow such scripts when packaging.

https://doc.rust-lang.org/cargo/reference/build-script-examples.html#code-generation

maspe36 · 2025-12-09T04:29:21Z

rclrs/src/lib.rs


 mod rcl_bindings;

+include!(concat!(env!("OUT_DIR"), "/interfaces.rs"));


This generates a lot of warnings right now. Will need to look at how to hygienically do that (or move to another crate and suppress there)

From what I see, the vast majority of the warnings are about missing docstrings. While I do agree that we should suppress warnings I feel this is a symptom that we can still improve our message generation pipeline here.
Specifically, I inspected the output of generating the messages in action_msgs:

#[cfg_attr(feature = "serde", derive(Deserialize, Serialize))] #[derive(Clone, Debug, PartialEq, PartialOrd)] pub struct GoalInfo { pub goal_id: unique_identifier_msgs::msg::UUID, pub stamp: builtin_interfaces::msg::Time, }

Which maps to GoalInfo:

# Goal ID unique_identifier_msgs/UUID goal_id # Time when the goal was accepted builtin_interfaces/Time stamp

Now rustc will complain about missing docstrings, and I do wonder if we should take the comments preceeding idl definitions and treat them as field docstrings.
After all, we already do this for constant declaration. For example in the same package GoalStatus has the following:

# Indicates status has not been properly set. int8 STATUS_UNKNOWN = 0 # The goal has been accepted and is awaiting execution. int8 STATUS_ACCEPTED = 1 [...]

And the rust code looks like this:

impl GoalStatus { /// Indicates status has not been properly set. pub const STATUS_UNKNOWN: i8 = 0; /// The goal has been accepted and is awaiting execution. pub const STATUS_ACCEPTED: i8 = 1; [...]

So we are already doing this in part.
Again this won't fix all the cases (since user packages that don't religiously document all their fields will still result in compile time warnings) but I feel we can do better and, most importantly, we should be consistent and either always generate docstrings or never do so.

I like this idea a lot. I'll see how much effort it is, and if its too much I'll punt to another PR in rosidl_rust and silence warnings somehow.

…using tests to fail

maspe36 · 2025-12-16T16:09:17Z

@esteve I believe this is essentially an implementation of the thought you had back in #394. We could also use feature flags instead of embedding metadata in the generated crates like I did. That may cut down on some rebuilds?

luca-della-vedova

Very interesting approach! And that's the kind of diff that gets me excited 😄
I gave this a try and I have some comments.

First of all, removing colcon-ros-cargo to fix CI is not a fix and actually breaks CI, so it should be reverted and the CI issues should be solved.
Removing colcon-ros-cargo will actually skip all the ros-cargo packages, including rclrs, if you look at the logs from the green CI you will notice that these warnings are printed:

[0.461s] WARNING:colcon.colcon_core.verb:No task extension to 'build' a 'ros.ament_cargo' package

And the build log shows that rclrs was actually not built and not tested, so that change effectively made the CI a noop.
There are two test failures, one is a very underwhelming cargo fmt run, the other is a failure to compile a unit test (seems to me that line was added by accident?):

error[E0423]: expected value, found struct `crate::rcl_interfaces::msg::IntegerRange`
   --> rclrs/src/subscription.rs:537:17
    |
537 |           let _ = crate::rcl_interfaces::msg::IntegerRange;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: use struct literal syntax instead: `crate::rcl_interfaces::msg::IntegerRange { from_value: val, to_value: val, step: val }`
    |
   ::: ~/ws_rclrs/install/rcl_interfaces/share/rcl_interfaces/rust/src/msg.rs:52:1
    |
52  | / pub struct IntegerRange {
53  | |     pub from_value: i64,
54  | |     pub to_value: i64,
55  | |     pub step: u64,
56  | | }
    | |_- `crate::rcl_interfaces::msg::IntegerRange` defined here

For more information about this error, try `rustc --explain E0423`.
error: could not compile `rclrs` (lib test) due to 1 previous error

Removing the line and running cargo fmt makes colcon test succeed on my machine.
Now to the questions

Open questions

How does AMENT_PREFIX_PATH get populated? If its populated as colcon is building packages (which I believe to be the case), then this will also re-export symbols in your own workspace as well as sourced overlays.

It's... Interesting... I gave this a try and it seems that the content on the variable contains (together with the environment variable) the prefixes of the package that the current package depends on. My guess (and what I hope colcon would do) is that behind the scenes it looks for all the packages specified in package.xml and sources them before building the required package, so the prefix contains all the required dependencies but not more than that, which is pretty good behavior.

Should this actually go into rclrs or a separate crate (i.e. ros_msgs?) By going into rclrs we eliminate the need for vendoring.

I'm very partial about red diffs but what is imho even more great about bundling is removing these confusing vendoring issues.
A question still stands though, I thought some of the reason for vendoring was distributing the crate standalone on crates.io. How would we go about that with this approach?

Its important to know that if we re-build rclrs, and the AMENT_PREFIX_PATH env var has changed, then the build.rs will be re-built, which will then re-build all of the generated code.

Yeah this is not ideal, other people might disagree but personally I don't feel it is a definitive blocker, if the behavior is robust and developer friendly we can accept less than ideal compile times for now.
Probably a noob question but how does the script detect that the environment variable changed? It seems to work and I expected to see a rerun-if-env-changed directive but I couldn't find any.

Other general comments

I also tried to build the message_demo example and noticed that this doesn't work. I think the issue is that you are not including feature flags for message packages.

error[E0277]: the trait bound `rclrs::rclrs_example_msgs::msg::rmw::VariousTypes: serde::Deserialize<'de>` is not satisfied
    --> src/message_demo.rs:110:28
     |
110  |     let rmw_deserialized = serde_json::from_str(&rmw_serialized)?;
     |                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `serde_core::de::Deserialize<'_>` is not implemented for `rclrs::rclrs_example_msgs::msg::rmw::VariousTypes`
     |
     = note: for local types consider adding `#[derive(serde::Deserialize)]` to your `rclrs::rclrs_example_msgs::msg::rmw::VariousTypes` type

And finally, I noticed that you implemented parsing of the environment variable into paths / splitting manually, have you considered using ament_rs to cut down on that boilerplate?

luca-della-vedova · 2025-12-17T06:51:12Z

rclrs/build.rs

+            // Find all dependencies for this crate that have a `*` version requirement.
+            // We will assume that these are other exported dependencies that need symbols
+            // exposed in their module.
+            let dependencies: String = Manifest::from_path(package_dir.clone().join("Cargo.toml"))
+                .iter()
+                .map(star_deps_to_use)
+                .collect();


This feels like the main hairy part to me, does this mean that we will require message packages to always have a * version requirement? Would specifying message versions like we do in our examples not work?
On the other hand, how do we make sure we are only re-exporting message packages? What happens if a user has a Rust library installed instead? Is there a risk that if I have a foo library that is:

A colcon-ros-cargo package.

Installed and in the prefix.

Dependency (possibly with a * version).

Then rclrs will end up bundling it and exporting rclrs::foo?

I think this should answer the last question: ros2-rust/rosidl_rust@main...maspe36:rosidl_rust:feature/relative_symbol_resolution#diff-bd0ce76b9774a1e4b569a95ae620dfb3422798d60bcf4c01ed6f501c978e3426
There is a meta tag added to the packages generated by rosidl, that we filter for.

Regarding versioning: I don't think using anything else than a * dependency make sense here.
Since the dependency resolution is managed by colcon and should be specified in the package.xml.

In the context of standalone rclrs. Maybe we could work with feature flags, to only include vendored messages given the standalone feature.

luca-della-vedova · 2025-12-17T07:10:59Z

rclrs/build.rs

+    let dest_path = PathBuf::from(out_dir).join("interfaces.rs");
+
+    // TODO I would like to run rustfmt on this generated code, similar to how bindgen does it
+    fs::write(dest_path, content.clone()).expect("Failed to write interfaces.rs");


Is there a way to change the location of the interfaces.rs to be a bit easier to find, perhaps in the install folder?
I find its current location in an obscure build subdirectory fairly confusing and it makes it hard to inspect what is going on:

lucadv@noble:~/ws_rclrs$ find . -name "interfaces.rs" ./src/ros2_rust/target/debug/build/rclrs-d6fb61407f6fcb8c/out/interfaces.rs ./src/ros2_rust/target/debug/build/rclrs-e547ca5eac3bba00/out/interfaces.rs

luca-della-vedova · 2025-12-17T07:19:55Z

rclrs/src/lib.rs


 mod rcl_bindings;

+include!(concat!(env!("OUT_DIR"), "/interfaces.rs"));


From what I see, the vast majority of the warnings are about missing docstrings. While I do agree that we should suppress warnings I feel this is a symptom that we can still improve our message generation pipeline here.
Specifically, I inspected the output of generating the messages in action_msgs:

#[cfg_attr(feature = "serde", derive(Deserialize, Serialize))] #[derive(Clone, Debug, PartialEq, PartialOrd)] pub struct GoalInfo { pub goal_id: unique_identifier_msgs::msg::UUID, pub stamp: builtin_interfaces::msg::Time, }

Which maps to GoalInfo:

# Goal ID unique_identifier_msgs/UUID goal_id # Time when the goal was accepted builtin_interfaces/Time stamp

Now rustc will complain about missing docstrings, and I do wonder if we should take the comments preceeding idl definitions and treat them as field docstrings.
After all, we already do this for constant declaration. For example in the same package GoalStatus has the following:

# Indicates status has not been properly set. int8 STATUS_UNKNOWN = 0 # The goal has been accepted and is awaiting execution. int8 STATUS_ACCEPTED = 1 [...]

And the rust code looks like this:

impl GoalStatus { /// Indicates status has not been properly set. pub const STATUS_UNKNOWN: i8 = 0; /// The goal has been accepted and is awaiting execution. pub const STATUS_ACCEPTED: i8 = 1; [...]

So we are already doing this in part.
Again this won't fix all the cases (since user packages that don't religiously document all their fields will still result in compile time warnings) but I feel we can do better and, most importantly, we should be consistent and either always generate docstrings or never do so.

Working example of re-exporting of the generated interface crates fou…

709ead8

…nd on the AMENT_PREFIX_PATH env var. Able to colcon build the entire workspace.

maspe36 commented Dec 9, 2025

View reviewed changes

maspe36 added 4 commits December 9, 2025 05:41

Swap humble to custom rosidl branch

3114d96

Swap jazzy to custom rosidl branch

442abcc

Swap kilted to custom rosidl branch

d0d22d7

Swap rolling to custom rosidl branch

7cc282c

maspe36 mentioned this pull request Dec 13, 2025

Install a curated registry of crates with libraries colcon/colcon-cargo#67

Open

maspe36 added 2 commits December 13, 2025 12:21

Removed colcon-ros-cargo from the github action to see if this was ca…

448c44e

…using tests to fail

Removed colcon-ros-cargo for the windows build

8b55304

maspe36 requested review from esteve and mxgrey December 16, 2025 15:54

maspe36 requested a review from jhdcs December 16, 2025 16:30

luca-della-vedova reviewed Dec 17, 2025

View reviewed changes


		mod rcl_bindings;

		include!(concat!(env!("OUT_DIR"), "/interfaces.rs"));

Re-Export Generated Messages in rclrs #556

Are you sure you want to change the base?

Re-Export Generated Messages in rclrs #556

Uh oh!

Conversation

maspe36 commented Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Open questions

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maspe36 commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

luca-della-vedova left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Open questions

Other general comments

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

maspe36 commented Dec 9, 2025 •

edited

Loading

maspe36 commented Dec 16, 2025 •

edited

Loading

luca-della-vedova left a comment •

edited

Loading