Empty and partially-filled metadata files from trace about ros2_tracing HOT 4 CLOSED

My team is developing a robotic platform that runs as a hybrid ROS 1 (Noetic) and ROS 2 (Foxy) system, and I have instrumented the ROS 2 side of our system with ros2_tracing off of the foxy branch. When everything runs properly, the outputs from ros2_tracing are great, and I've been able to do some insightful analysis from the data with help from the starter scripts in tracetools_analysis.

Happy to hear that!

Since the Trace launch action is added at the beginning of our system's main ROS 2 launch.py launch file, I feel like we should be satisfying this requirement, but I might be wrong.

I've always wondered if the order in the launch file guaranteed the launch order (you can still find some TODOs in the code about this), but I've personally never come across any situation where this was a problem. It could be worth looking into this again, though. Maybe this is an issue in older distros.

I did notice that the metadata (generated from lttng?) file is populated with data when shutting down our system, rather than during the beginning/middle of operation, so perhaps everything is shutting down before lttng has an opportunity to output the data?

I personally couldn't say; I would have to ask the LTTng maintainers. I've personally never had problems with shutting down the system and stopping the tracer, but of course that doesn't really mean much.

Any thoughts from others on this would be appreciated! Please let me know if you'd like any additional information from my side, and thanks for creating this great tool!

Some thoughts/questions:

• What LTTng version(s) are you using? E.g., lttng --version and ideally dpkg -l | grep lttng.
• By partial/empty metadata files, you're referring to ust/uid/$UID/64-bit/metadata under the trace directory, correct? Or are you referring to the trace data/events?
  • If you're referring to the trace data: how do you know that it is totally/half empty or full? Have you tried looking at the data using the babeltrace or babeltrace2 command?
  • Note that the metadata LTTng file and the "metadata" that I refer to in the part of the README you quoted are different things. The "metadata" I mention is the trace data that is generated in the application's initialization phase (e.g., publisher/subscription creation trace events, etc.).
• One common problem is missing events due to generating too many too quickly for the ring buffer to keep up. You can increase the buffer size to fix this:

  ros2_tracing/tracetools_launch/tracetools_launch/action.py

  Line 118 in 0041e07

  subbuffer_size_ust: int = 8 * 4096,

  (if you're using Foxy, you might have to manually backport this feature or try to use tracetools_launch @ Iron with ROS 2 Foxy, but it might be tricky)
• Is there a way you could create and provide a reliable-enough minimal reproducer? Without any confidential information, etc.

from ros2_tracing.

Thanks for the helpful response!

What LTTng version(s) are you using? E.g., lttng --version and ideally dpkg -l | grep lttng.

Output from lttng --version is lttng (LTTng Trace Control) 2.11.2 - Lafontaine

Output from dpkg -l | grep lttng:

ii  liblttng-ctl0:amd64                             2.11.2-1build1                    amd64        LTTng control and utility library
ii  liblttng-ust-ctl4:amd64                         2.11.0-1                          amd64        LTTng 2.0 Userspace Tracer (trace control library)
ii  liblttng-ust-dev:amd64                          2.11.0-1                          amd64        LTTng 2.0 Userspace Tracer (development files)
ii  liblttng-ust-python-agent0:amd64                2.11.0-1                          amd64        LTTng 2.0 Userspace Tracer (Python agent native library)
ii  liblttng-ust0:amd64                             2.11.0-1                          amd64        LTTng 2.0 Userspace Tracer (tracing libraries)
ii  lttng-modules-dkms                              2.12.5-1ubuntu2~20.04.5           all          Linux Trace Toolkit (LTTng) kernel modules (DKMS)
ii  lttng-tools                                     2.11.2-1build1                    amd64        LTTng control and utility programs
ii  python3-lttng                                   2.11.2-1build1                    amd64        LTTng control and utility Python bindings

By partial/empty metadata files, you're referring to ust/uid/$UID/64-bit/metadata under the trace directory, correct? Or are you referring to the trace data/events?
If you're referring to the trace data: how do you know that it is totally/half empty or full? Have you tried looking at the data using the babeltrace or babeltrace2 command?
Note that the metadata LTTng file and the "metadata" that I refer to in the part of the README you quoted are different things. The "metadata" I mention is the trace data that is generated in the application's initialization phase (e.g., publisher/subscription creation trace events, etc.).

Yes, I am referring to the ust/uid/$UID/64-bit/metadata file. And thanks for the clarification on that.

One common problem is missing events due to generating too many too quickly for the ring buffer to keep up. You can increase the buffer size to fix this:

Just to clarify, is this suggestion only applicable to the trace data/events, or is it applicable to the issue with the empty/partially-filled ust/uid/$UID/64-bit/metadata file? I can look into manually backporting the mentioned feature to Foxy if so and see if that helps.

Luckily our system is open source, but it may be difficult for me to put together a minimal reproducible example. If I am able to come up with a way to do so, I'll report back.

from ros2_tracing.

Output from lttng --version is lttng (LTTng Trace Control) 2.11.2 - Lafontaine

I would recommend upgrading to LTTng 2.13 and seeing if it helps. If you're on Ubuntu, you can use this PPA: https://launchpad.net/~lttng/+archive/ubuntu/stable-2.13. Otherwise, you have to build it from source.

Just to clarify, is this suggestion only applicable to the trace data/events, or is it applicable to the issue with the empty/partially-filled ust/uid/$UID/64-bit/metadata file?

I don't think that increasing the buffer size will help with the metadata file unfortunately.

Luckily our system is open source, but it may be difficult for me to put together a minimal reproducible example. If I am able to come up with a way to do so, I'll report back.

I would try to configure LTTng directly and see if that changes anything: https://lttng.org/docs/v2.13/#doc-controlling-tracing. If so, then it's probably an issue somewhere in ROS 2 or ros2_tracing specifically. Otherwise, we could report the issue to the LTTng maintainers.

from ros2_tracing.

I'll close this for now. If you're still facing this issue, let me know.

from ros2_tracing.