Service server exposes by controller manager hangs the control loop

[rafa-martin]

Describe the bug
I am using the UR library to control the arm: https://github.com/UniversalRobots/Universal_Robots_ROS2_Driver.

rqt_joint_trajectory_controller calls the /controller_manager/list_controllers service every second, and this causes several adverse effects:

The following warning appears in the monitor:

Overrun might occur, Total time : 3198.487 us (Expected < 2000.000 us) --> Read time : 30.260 us, Update time : 3161.148 us, Write time : 7.079 us Overrun detected! The controller manager missed its desired rate of 500 Hz. The loop took 3.327106 ms (missed cycles : 2).

If it is called too frequently, UR’s internal buffer fills up and the following appears: Pipeline producer overflowed! <RTDE Data Pipeline>, and the arm drops the connection, etc.

To Reproduce
Steps to reproduce the behavior:

1. Run the driver
2. (Option 1) Open rqt_joint_trajectory controller
3. (Option 2) Call manually /controller_manager/list_controllers

Expected behavior
Do not hang the control loop to calculate all these things

Environment (please complete the following information):

• OS: Ubuntu 24.04 (6.8.1-1037-realtime 38-Ubuntu SMP PREEMPT_RT)
• Version: jazzy debs (4.38.0-1)

[saikishor]

@rafa-martin are you certain that the above logs only appear when you open the rqt_joint_trajectory_controller?. Would you be able to share a PlotJuggler plot of the ~/statistics topic as described in the link: https://control.ros.org/rolling/doc/ros2_control/doc/introspection.html, and please share the plot of the controller_manager.total_time variable before, while and after exiting the rqt_joint_trajectory_controller

[rafa-martin]

The issue occurs primarily when invoking any ROS 2 control service, such as list_controllers, which rqt_joint_trajectory_controller calls once per second.

Later I will attach some videos.

[rafa-martin]

Calling once the service:

Video opening rqt:
https://github.com/user-attachments/assets/8677e57c-2946-4db8-9eec-159b2760e3b3

System load (below is the controller manager for the arm)

The system specs:
6.8.1-1037-realtime #38-Ubuntu SMP PREEMPT_RT
Intel i7-11700 @ 2.50GHz

Hope this helps you.

I was checking the code and there is a mutex for all these services, we need to check a way to avoid blocking main loop.

[saikishor]

@rafa-martin This is the plot on our robot opening rqt_joint_trajectory_controller for more than 20 seconds

[saikishor]

@rafa-martin Looking at the video, it is very evident. I can take a look again and get back to you.

[rafa-martin]

@saikishor Which package version are you using?

[saikishor]

@saikishor Which package version are you using?

@rafa-martin 4.38.0

[saikishor]

@rafa-martin I cannot reproduce it locally :(

[rafa-martin]

I'm using from rosdistro debs (ros-jazzy-controller-manager 4.38.0-1noble.20251008.000820)
Are you using as debs or compiled locally?
Which kernel are you using? uname -a

[saikishor]

This is when I tried to time the part to access the part with mutex, and the max I got on my non-RT PC is a peak of 20 us and during this time I opened the rqt_trajectory_controller

ros2_control/controller_manager/src/controller_manager.cpp

Lines 3020 to 3021 in eba8665

	std::vector<ControllerSpec> & rt_controller_list =
	rt_controllers_wrapper_.update_and_get_used_by_rt_list();

[rafa-martin]

I probably found the issue. It could be related to zenoh. I changed from RMW_IMPLEMENTATION = rmw_zenoh_cpp to RMW_IMPLEMENTATION = rmw_fastrtps_cpp and it works as expected. I will debug more to understand why rmw introduces a small lock in there.

[saikishor]

I probably found the issue. It could be related to zenoh. I changed from RMW_IMPLEMENTATION = rmw_zenoh_cpp to RMW_IMPLEMENTATION = rmw_fastrtps_cpp and it works as expected. I will debug more to understand why rmw introduces a small lock in there.

That's very weird that the RMW is introducing delays in the RT loop. If you find something, please post it here.

Thank you

[rafa-martin]

100% confirmed. Just changing RMW it solve the issue.

I heard an issues related with services and publish topics with zenoh. In the following weeks I will debug more. For now, I will switch to fastdds or cyclonedds

For the future, tested on:

• rmw_zenoh (packages.ros.org - 0.2.8-1noble.20251007.090035)
• rmw_zenoh (source build - 0.2.9 - 858f7b5f9e404c5d759ffb365b338dbb41d5bd1f)

[saikishor]

100% confirmed. Just changing RMW it solve the issue.

I heard an issues related with services and publish topics with zenoh. In the following weeks I will debug more. For now, I will switch to fastdds or cyclonedds

For the future, tested on:

• rmw_zenoh (packages.ros.org - 0.2.8-1noble.20251007.090035)
• rmw_zenoh (source build - 0.2.9 - 858f7b5f9e404c5d759ffb365b338dbb41d5bd1f)

Thanks a lot for the update.