Ubuntu 22.04 BUG: Failed to delete datawriter in nav2_util tests about rmw HOT 9 OPEN

We see this in Nav2's CI periodically as well, as reported on the Slack (@clalancette ) It is pretty concerning to us that we again have new RMW errors appearing for really basic tests. This is a new thing in the last few months

from rmw.

@SteveMacenski @Ryanf55

This issue has been there for a long time? Or just stated failing recently? if you happen know which commit causes this error behavior on Nav2, that would be really helpful.

[ERROR] [1702060366.623059261] [fibonacci_server_node]: Error in shutdown of get_type_description service: Fail in delete datawriter

could be related to Type Description Support... but not sure.

CC: @iuhilnehc-ynos @Barry-Xu-2018

from rmw.

It’s not deterministic in our CI, so I can’t pinpoint the exact commit. Its been the last few months though.

from rmw.

@SteveMacenski @Ryanf55

This issue has been there for a long time? Or just stated failing recently? if you happen know which commit causes this error behavior on Nav2, that would be really helpful.

[ERROR] [1702060366.623059261] [fibonacci_server_node]: Error in shutdown of get_type_description service: Fail in delete datawriter

could be related to Type Description Support... but not sure.

CC: @iuhilnehc-ynos @Barry-Xu-2018

I'll try to set up a source build of rmw to run a bisect on when I get home in a few days.

from rmw.

Sorry, I haven't had time to do this yet. It might be easier for me to just debug this with gdb because a bisect involves recompiling ros2, then nav2, which takes a while each iteration (158 packages using --packages-up-to).

from rmw.

Here's the bisect report:

• 6.0.0 - doesn't compile in rmw_dds_common
• 7.0.0 - doesn't compile in rmw_dds_common

/home/ryan/Dev/ros2_rolling/src/ros2/rmw_dds_common/rmw_dds_common/include/rmw_dds_common/qos.hpp:242:3: error: ‘rosidl_type_hash_t’ has not been declared

• 7.2.2 - test fails for the same reason
• 7.1.0 - test fails for same reason

If I want to bisect this further, I'll need to start doing the bisect across multiple repos since they are coupled together due to the public API additions. Are there any tools to roll back state across multiple repos, say with vcs? Or, do I manually need to figure out which hashes of each repo correspond to each other?

from rmw.

@SteveMacenski Does NAV2 CI give you any way to look at test regressions per test to see the first date that test started failing? I know cdash has regression reporting capabilities.

From the CDash blog, it looks like you can look at test failure frequency, but not over time.
https://circleci.com/blog/how-to-output-junit-tests-through-circleci-2-0-for-expanded-insights/

from rmw.

Unless CircleCI produces that, we dont have any unique tools

from rmw.

I encountered very similar error, https://github.com/mavlink/mavros/actions/runs/8129217925/job/22234646274 the same code works on Humble, but fails on Iron.

from rmw.