Possible regression in rcl preshutdown callbacks - context invalid? about rclcpp HOT 11 OPEN

No worries, shit happens. Another good reason Nav2's around using all these bells and whistles as a canary in the coalmine. Only used up like a day on this so not bad at all in the scheme of things. Lovely that there's already a solution! Thanks @fujitatomoya

from rclcpp.

The second comment is definitely the same as this.

The first is perhaps? Its also another 'things failing on shutdown' example and you see similar logs with the pre-shutdown (the third is a summarized version of the same). I didn't assume it was a Fast-DDS specific problem, but 🤷

    [INFO] [1716243365.746525522] [costmap]: Running Nav2 LifecycleNode rcl preshutdown (costmap)
    [INFO] [1716243365.746611543] [costmap]: Destroying bond (costmap) to lifecycle manager.
    Failed to delete datawriter, at ./src/publisher.cpp:45 during '__function__'
    cannot publish data, at ./src/rmw_publish.cpp:66 during '__function__'
    Fail in delete datareader, at ./src/rmw_service.cpp:89 during '__function__'

from rclcpp.

I don't know if its related, but also point out this issue as well discovered which points to issues in spin() exiting cleanly when Control+C is hit so that rclcpp::shutdown or others can be called once the node is kicked out. It seems related, but I can file a separate ticket for it. Similarly to that @tonynajjar has another ticket on spin_some not working the first time its called in some cases.

from rclcpp.

@SteveMacenski

#2545 can fix that graceful shutdown case, and generated by #2528. (sorry...)

from rclcpp.

#2545 fixes LifecycleNode to avoid leaving in unknow state, if the object is destroyed. (originally this is issued #997).

But if the context is gracefully shutdown (e.g deferred signal handler via SIGINT, which is the exactly case here), the context is already shutdown, and that makes the lifecycle node dtor fails to publish the transition (lifecyclenode should be shutdown to avoid unknow state for sensors and devices but fails since context is not valid anymore). #2545 bypass this situation at this moment, shutdown is called only when the context is valid. this is tagged with TODO until we figure out how to address this condition.

#2545 CI is running now, once i have green light, i will merge it to rolling.

from rclcpp.

BTW: this worked fine those pre-24.04 Rolling (22.04 Rolling didn't exhibit this) so there was something that changed that caused this, it was fully functioning for ~a year prior. I think my understanding of your response is that there's a bit of a catch-22, so mentioning this as a "I think there should be a solution, no?"

from rclcpp.

@SteveMacenski i think there are 2 problems here,

[component_container_isolated-8] [ERROR] [1717089793.875015316] [controller_server]: Unable to start transition 5 from current state shuttingdown: Could not publish transition: publisher's context is invalid, at ./src/rcl/publisher.c:423, at ./src/rcl_lifecycle.c:368
[component_container_isolated-8] [WARN] [1717089793.875026497] [rclcpp_lifecycle]: Shutdown error in destruction of LifecycleNode: final state(unconfigured)

I am sure these messages are generated by #2528 as explained #2547 (comment), and #2545 can fix it.

note: #2528 IS NOT merged to jazzy. (#2542 staging but merged), so we should not meet these message with jazzy.

24.04 and seen in both Jazzy and current rolling. There has been some extended discussion wrt the unit tests in the comments:

• ros-navigation/navigation2#4298 (comment)
• ros-navigation/navigation2#4298 (comment)
• ros-navigation/navigation2#4298 (comment)

these 3 issues are with rmw_fastrtps, right? this looks the same problem with ros2/rmw_fastrtps#761

from rclcpp.

@SteveMacenski

#2545 has been merged to rolling, the following waning and error should not be printed even with graceful shutdown.

[component_container_isolated-8] [ERROR] [1717089793.875015316] [controller_server]: Unable to start transition 5 from current state shuttingdown: Could not publish transition: publisher's context is invalid, at ./src/rcl/publisher.c:423, at ./src/rcl_lifecycle.c:368
[component_container_isolated-8] [WARN] [1717089793.875026497] [rclcpp_lifecycle]: Shutdown error in destruction of LifecycleNode: final state(unconfigured)

from rclcpp.

Great! Currently in the middle of other firefighting with getting Nav2 out for Jazzy, but I'll test this out once the nightly docker containers update and I'm between burning buildings. If you're confident that it addresses the issue, feel free to close

from rclcpp.