CPU usage component container about rclcpp HOT 10 CLOSED

@albertarla Just in case you are interested in testing, we wrote a ROS2 package to measure the CPU usage of one or a set of processes: https://github.com/ual-arm/robotic-simulators-benchmark/tree/main/measure_process_ros2_pkg

We'll try to release it as an independent package... someday :-)

from rclcpp.

@albertarla out of curiosity, how do you check the CPU usage? do you use specific command?

from rclcpp.

I'd be curious to also know if intra-process comms was enabled within the container? And also know the QoS for the image subscriptions?

If not enabled, the RMW layer (cyclonedds without any SHM) would be responsible for serializing/deserializing multiple copies of the image for the different subs. At this point, cyclonedds send/receive buffer sizes and the UDP buffer limits in the kernel (in combination with QoS) may be throttling transport. I would expect that increasing both those limits + sizes as described here would result in a CPU load increase while improving transport performance.

I'd reckon the CPU increase outside the container might have more to do with increased CPU footprint from 5 separate processes vs 1 component container.

from rclcpp.

Is the publisher node one of the components? Or is it in a separate process?

from rclcpp.

Hi @fujitatomoya , I don't have precise metrics of the CPU usage. Just the general behaviour. I used htop to monitorize it and the values I'm showing as an example emulates what I'm seeing with htop.

Hi @Yadunund I don't have intra-process comms enabled because (answering the question of @alsora) the publisher node is from another process and not a component. The QoS are the default values for the image_transport, Reliable and Volatile. Morover, the depth for the subscriber is of 1. Additionally, I have tried with Best effort and Volatile with the same results.

I'll try to increase both limits you are talking about and see if something changes. Thanks for the support!

from rclcpp.

Ah whoops I missed the part where the publisher is not within the component.

from rclcpp.

We are lookint into our high demanding process inside the callback as it seems the delay may be caused by it and it's not related with ROS2. I'll keep the issue updated.

from rclcpp.

@albertarla you are running multiple processes; can you provide more details about the CPU measurement you reported?
Is that the total CPU usage of the computer? or the CPU usage of the subscriptions process?
Is that normalized with the number of CPU cores?

To get a full understanding, we would need to see the CPU usage of both the publisher and the subscriber process.
The bottleneck may be on the publisher side that it's not able to send messages to that many subscriptions at the desired rate (the publisher is one, so it uses a single thread to serve all publications).

from rclcpp.

@alsora I'll try to record the CPU usage of those single processes by the end of this week and let you know. The data I show is the average CPU usage of the whole computer with all the different process. What I tried to show in this data is that, even the CPU is not at its maximum the fps of the subscription is not 30fps after adding more components. I was expecting for the CPU to increase at its maximum before decreasing performance.

For the moment, the tests I had run are showing me that the ferquency of the publisher is stable at 30fps. To test that I'm computing the frequency the publisher node enters in the pub callback.

from rclcpp.

Hi all, I changed the code to use intra process communications. Now ros2 topic hz has the expected frequency.

The subscriber still receives the images at a lower frequency but this is now caused only because the processing time of one task inside the callback. I'm still not sure why the CPU is not at its maximum before adding this delay in the processing task but it seems is not related with ROS or rclcpp.

I'll close the issue as I'm going to investigate why this task running inside the callback doesn't use the 100% of the CPU available. I'm pretty sure this is not related to ROS anymore, thanks for the support! If at the end is ROS related I'll reopen this issue and let you know.

from rclcpp.