ros2 / build_farmer Goto Github PK

build_farmer's Introduction

About

The Robot Operating System (ROS) is a set of software libraries and tools that help you build robot applications. From drivers to state-of-the-art algorithms, and with powerful developer tools, ROS has what you need for your next robotics project. And it's all open source. Full project details on ROS.org

Getting Started

Looking to get started with ROS? Our installation guide is here. Once you've installed ROS start by learning some basic concepts and take a look at our beginner tutorials.

Join the ROS Community

Community Resources

ROS Discussion Forum
ROS Discord Server
Robotics Stack Exchange (preferred ROS support forum).
Official ROS Videos
ROSCon, our yearly developer conference.
Cite ROS 2 in academic work using DOI: 10.1126/scirobotics.abm6074

Developer Resources

ROS 2 Documentation
ROS Package API reference
ROS Package Index
ROS on Docker Hub
ROS Resource Status Page
REP-2000: ROS 2 Releases and Target Platforms

Project Resources

ROS is made possible through the generous support of open source contributors and the non-profit Open Source Robotics Foundation (OSRF). Tax deductible donations to the OSRF can be made here.

build_farmer's People

Watchers

build_farmer's Issues

Need to install libpcl-dev in the docker containers

To get ros2/pcl_conversions#1 working, we need to have libpcl-dev installed inside of the Linux docker containers.

Buildcop handoff 2017-05-30

Notes from the previous week: #21

Summary takeaway: We can pare down a few more flakes by resolving some strange FastRTPS errors. I could discern no standout pattern with the other flakes.

Recent issues with work in progress

pip can't run with python debug interpreter due to SSL failures was error generating repository hashes causing jobs to fail.

Recent issues I didn't have time for

Update chocolatey and packages on Windows
Configure eatable as a Windows 10 Machine (deferred until necessary)

Feel free to close this at your leisure @wjwwood. Tag you're it. 💥

Java 8u131 on windows makes jenkins client crash

I tried to update java on windshield. The jenkins client could not connect to the jenkins master after the update. I downgraded java to version 8u121 and it's back online. We need to investigate this before considering updating java on the ci machines

New windows debug exceptions

Lots of new failures on windows debug nightlies as of 8 April.

Impacted tests

This is the list of new failures (there are many other failures on these jobs for other reasons)

projectroot.gtest_client_wait_for_service_shutdown__rmw_fastrtps_cpp
projectroot.gtest_executor__rmw_fastrtps_cpp
projectroot.gtest_intra_process__rmw_fastrtps_cpp
projectroot.gtest_local_parameters__rmw_fastrtps_cpp
projectroot.gtest_multiple_service_calls__rmw_fastrtps_cpp
projectroot.gtest_multithreaded__rmw_fastrtps_cpp
projectroot.gtest_publisher__rmw_fastrtps_cpp
projectroot.gtest_repeated_publisher_subscriber__rmw_fastrtps_cpp
projectroot.gtest_services_in_constructor__rmw_fastrtps_cpp
projectroot.gtest_spin__rmw_fastrtps_cpp
projectroot.gtest_subscription__rmw_fastrtps_cpp
projectroot.gtest_timeout_subscriber__rmw_fastrtps_cpp
projectroot.gtest_timer__rmw_fastrtps_cpp
projectroot.test_callback_exceptions
projectroot.test_client__rmw_fastrtps_cpp
projectroot.test_composition__rmw_fastrtps_cpp
projectroot.test_default_state_machine
projectroot.test_externally_defined_services
projectroot.test_find_weak_nodes
projectroot.test_get_node_names__rmw_fastrtps_cpp
projectroot.test_lifecycle_node
projectroot.test_messages_c__rmw_fastrtps_cpp
projectroot.test_multiple_instances
projectroot.test_node__rmw_fastrtps_cpp
projectroot.test_publisher__rmw_fastrtps_cpp
projectroot.test_publisher_subscriber_cpp__rmw_fastrtps_cpp__BoundedArrayNested
projectroot.test_publisher_subscriber_cpp__rmw_fastrtps_cpp__BoundedArrayPrimitives
projectroot.test_publisher_subscriber_cpp__rmw_fastrtps_cpp__Builtins
projectroot.test_publisher_subscriber_cpp__rmw_fastrtps_cpp__DynamicArrayNested
projectroot.test_publisher_subscriber_cpp__rmw_fastrtps_cpp__DynamicArrayPrimitives
projectroot.test_publisher_subscriber_cpp__rmw_fastrtps_cpp__Empty
projectroot.test_publisher_subscriber_cpp__rmw_fastrtps_cpp__Nested
projectroot.test_publisher_subscriber_cpp__rmw_fastrtps_cpp__Primitives
projectroot.test_publisher_subscriber_cpp__rmw_fastrtps_cpp__StaticArrayNested
projectroot.test_publisher_subscriber_cpp__rmw_fastrtps_cpp__StaticArrayPrimitives
projectroot.test_service__rmw_fastrtps_cpp
projectroot.test_state_machine_info
projectroot.test_subscription__rmw_fastrtps_cpp
projectroot.test_subscription_valid_data_cpp__rmw_fastrtps_cpp
projectroot.test_tutorial_add_two_ints_server_add_two_ints_client__rmw_fastrtps_cpp
projectroot.test_tutorial_add_two_ints_server_add_two_ints_client_async__rmw_fastrtps_cpp
service_client__rmw_fastrtps_cpp.wait_for_service_shutdown
test_add_two_ints_server_add_two_ints_client__rmw_fastrtps_cpp_Debug.test_executable
test_add_two_ints_server_add_two_ints_client_async__rmw_fastrtps_cpp_Debug.test_executable
test_composition__rmw_fastrtps_cpp_Debug.test_api_pubsub_composition
test_composition__rmw_fastrtps_cpp_Debug.test_api_srv_composition
test_executor__rmw_fastrtps_cpp.multiple_executors
test_executor__rmw_fastrtps_cpp.multithreaded_spin_call
test_executor__rmw_fastrtps_cpp.notify
test_executor__rmw_fastrtps_cpp.recursive_spin_call
test_intra_process_within_one_node__rmw_fastrtps_cpp.nominal_usage
test_local_parameters__rmw_fastrtps_cpp.get_from_node_primitive_type
test_local_parameters__rmw_fastrtps_cpp.get_from_node_variant_type
test_local_parameters__rmw_fastrtps_cpp.get_parameter_or
test_local_parameters__rmw_fastrtps_cpp.helpers
test_local_parameters__rmw_fastrtps_cpp.local_asynchronous
test_local_parameters__rmw_fastrtps_cpp.local_synchronous
test_local_parameters__rmw_fastrtps_cpp.local_synchronous_repeated
test_local_parameters__rmw_fastrtps_cpp.set_parameter_if_not_set
test_multiple_service_calls__rmw_fastrtps_cpp.multiple_clients
test_multithreaded__rmw_fastrtps_cpp.multi_access_publisher
test_multithreaded__rmw_fastrtps_cpp.multi_access_publisher_intra_process
test_multithreaded__rmw_fastrtps_cpp.multi_consumer_clients
test_multithreaded__rmw_fastrtps_cpp.multi_consumer_intra_process
test_multithreaded__rmw_fastrtps_cpp.multi_consumer_single_producer
test_publisher__rmw_fastrtps_cpp.publish_with_const_reference

Most only impact the rmw_fastrtps variant of tests, but some are not run for each rmw implementation. The ones that don't have fastrtps in the name are:
projectroot.test_callback_exceptions
projectroot.test_default_state_machine
projectroot.test_externally_defined_services
projectroot.test_find_weak_nodes
projectroot.test_lifecycle_node
projectroot.test_multiple_instances
projectroot.test_state_machine_info

Without digging into the details of these tests I would think it's because they're using the default RMW implementation of rmw_fastrtps, given how rmw_connext_cpp tests seem unaffected. If these tests are actually rmw-agnostic could someone please point it out.

Symptom
They most common error output is an SEH exception (e.g. http://ci.ros2.org/view/nightly/job/nightly_win_deb/435/testReport/(root)/projectroot/test_find_weak_nodes/):

-- run_test.py: invoking following command in 'C:\J\workspace\nightly_win_deb\ws\src\ros2\rclcpp\rclcpp':
 - C:/J/workspace/nightly_win_deb/ws/build/rclcpp/Debug/test_find_weak_nodes.exe --gtest_output=xml:C:/J/workspace/nightly_win_deb/ws/build/rclcpp/test_results/rclcpp/test_find_weak_nodes.gtest.xml
Running main() from gtest_main.cc
[==========] Running 2 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 2 tests from TestFindWeakNodes
[ RUN      ] TestFindWeakNodes.allocator_strategy_with_weak_nodes
unknown file: error: SEH exception with code 0xc0000005 thrown in the test body.
[  FAILED  ] TestFindWeakNodes.allocator_strategy_with_weak_nodes (70 ms)
[ RUN      ] TestFindWeakNodes.allocator_strategy_no_weak_nodes
unknown file: error: SEH exception with code 0xc0000005 thrown in the test body.
[  FAILED  ] TestFindWeakNodes.allocator_strategy_no_weak_nodes (73 ms)
[----------] 2 tests from TestFindWeakNodes (143 ms total)

but sometimes it's just bad return codes e.g. http://ci.ros2.org/view/nightly/job/nightly_win_deb/435/testReport/(root)/projectroot/test_tutorial_add_two_ints_server_add_two_ints_client__rmw_fastrtps_cpp/

launch_testing.UnmatchedOutputError: Example output ([]) does not match expected output ([b'Result of add_two_ints: 5'])
-------------------- >> begin captured stdout << ---------------------
(test_executable_0) pid 8616: ['C:/J/workspace/nightly_win_deb/ws/build/demo_nodes_cpp/Debug/add_two_ints_server.exe', 'test_executable'] (all > console, InMemoryHandler: test_executable_0)
(test_executable_1) pid 12648: ['C:/J/workspace/nightly_win_deb/ws/build/demo_nodes_cpp/Debug/add_two_ints_client.exe', 'test_executable'] (all > console, InMemoryHandler: test_executable_1)
[test_executable_0] Incoming request
[test_executable_0] a: 2 b: 3
(test_executable_1) rc 3221225477
() tear down
(test_executable_0) signal SIGTERM
(test_executable_0) rc 1

Investigations
At the time we were still using a pinned version of fastrtps, so I would rule out any changes on that repo. There also don't seem to have been any work on relevant windows buildfarm machines then (eatable was touch on the 7th but it hasn't run debug nightlies since mach 29). These are the PRs that were merged on our repos around that time: https://github.com/search?p=1&q=user%3Aament+user%3Aros2+merged%3A%222017-04-06T22%3A00%3A00-08%3A00+..+2017-04-08T15%3A30%3A00-08%3A00%22&type=Issues&utf8=%E2%9C%93

It's almost all the namespace implementation. There were two other PRs in ament_tools, but even resetting the head of ament_tools to before those two PRs were merged, the tests still fail: http://ci.ros2.org/job/ci_windows/2679/#showFailuresLink

Conclusion
This seems to be caused by the changes to support namespaces in ros2/rmw#95 or one of the related PRs. Not all tests fail, (e.g. this pub-sub passes http://ci.ros2.org/job/ci_windows/2679/testReport/(root)/test_publisher_subscriber__StaticArrayNested__rclcpp__rmw_connext_cpp__rmw_fastrtps_cpp_Debug/).

My suspicion is that it's only tests that are creating nodes with anything more than the node name passed. This is because only the test_rclcpp tests modified in https://github.com/ros2/system_tests/pull/196/files were affected. @wjwwood does it seem plausible that those PRs introduced this? (I may have missed a possibility and I wouldn't want to declare an incorrect cause)

Tracking flaky tests

To avoid reducing the SNR I'll open a meta ticket for this.

TL;DR we need a good way to track flaky tests over time.
Long term solution: Improve the testing logic to run every single test many times whether or not another test of the same package failed. This is captured in ros2/ci#19, we could also find a way to label them as flaky on a per build basis, @nuclearsandwich mentionned that he did that in other test frameworks in the past and we could look into doing it on our test suites

Short term solution: Aggregate failures on repeated nightlies by poking the jenkins API. Minimal script available here

Result of the script ran today on all jobs since beta2 release:

$ python3 ./aggregate_flaky_tests.py -n 21 --skip

 - [ ] AllocatorTest__rmw_connext_cpp.allocator_unique_ptr, ['nightly_linux_repeated']
 - [ ] TestGetNodeNames__rmw_connext_cpp.test_rcl_get_node_names, ['nightly_osx_repeated']
 - [ ] TestGetNodeNames__rmw_fastrtps_cpp.test_rcl_get_node_names, ['nightly_linux-aarch64_repeated', 'nightly_osx_repeated']
 - [ ] TestServiceFixture__rmw_connext_cpp.test_service_nominal, ['nightly_osx_repeated']
 - [ ] TestServiceFixture__rmw_fastrtps_cpp.test_service_nominal, ['nightly_osx_repeated']
 - [ ] WaitSetTestFixture__rmw_connext_cpp.zero_timeout, ['nightly_win_rep']
 - [ ] WaitSetTestFixture__rmw_fastrtps_cpp.finite_timeout, ['nightly_win_rep']
 - [ ] projectroot.gtest_executor__rmw_connext_cpp, ['nightly_linux_repeated']
 - [ ] projectroot.gtest_executor__rmw_fastrtps_cpp, ['nightly_osx_repeated']
 - [ ] projectroot.gtest_local_parameters__rmw_connext_cpp, ['nightly_linux_repeated', 'nightly_win_rep']
 - [ ] projectroot.gtest_local_parameters__rmw_fastrtps_cpp, ['nightly_linux-aarch64_repeated', 'nightly_linux_repeated']
 - [ ] projectroot.gtest_multiple_service_calls__rmw_connext_cpp, ['nightly_linux_repeated']
 - [ ] projectroot.gtest_multithreaded__rmw_connext_cpp, ['nightly_linux_repeated', 'nightly_osx_repeated', 'nightly_win_rep']
 - [ ] projectroot.gtest_multithreaded__rmw_fastrtps_cpp, ['nightly_linux_repeated', 'nightly_osx_repeated']
 - [ ] projectroot.gtest_services_in_constructor__rmw_connext_cpp, ['nightly_osx_repeated']
 - [ ] projectroot.gtest_services_in_constructor__rmw_fastrtps_cpp, ['nightly_osx_repeated']
 - [ ] projectroot.gtest_timeout_subscriber__rmw_connext_cpp, ['nightly_win_rep']
 - [ ] projectroot.rclpytests, ['nightly_osx_repeated']
 - [ ] projectroot.rosidl_generator_py.test_publisher_subscriber__Nested__rclpy__rmw_fastrtps_cpp, ['nightly_linux_repeated']
 - [ ] projectroot.rosidl_generator_py.test_requester_replier__Empty__rclpy__rmw_fastrtps_cpp, ['nightly_linux-aarch64_repeated', 'nightly_linux_repeated', 'nightly_osx_repeated', 'nightly_win_rep']
 - [ ] projectroot.rosidl_generator_py.test_requester_replier__Primitives__rclpy__rmw_fastrtps_cpp, ['nightly_linux-aarch64_repeated', 'nightly_linux_repeated', 'nightly_osx_repeated', 'nightly_win_rep']
 - [ ] projectroot.test_client_scope_consistency_cpp__rmw_connext_cpp, ['nightly_linux_repeated', 'nightly_osx_repeated', 'nightly_win_rep']
 - [ ] projectroot.test_client_scope_consistency_cpp__rmw_fastrtps_cpp, ['nightly_linux_repeated', 'nightly_osx_repeated']
 - [ ] projectroot.test_client_scope_cpp__rmw_connext_cpp, ['nightly_linux_repeated', 'nightly_osx_repeated']
 - [ ] projectroot.test_client_scope_cpp__rmw_fastrtps_cpp, ['nightly_osx_repeated']
 - [ ] projectroot.test_composition__rmw_connext_cpp, ['nightly_linux_repeated', 'nightly_osx_repeated']
 - [ ] projectroot.test_composition__rmw_fastrtps_cpp, ['nightly_linux-aarch64_repeated', 'nightly_linux_repeated', 'nightly_osx_repeated', 'nightly_win_rep']
 - [ ] projectroot.test_demo_cyclic_pipeline__rmw_connext_cpp, ['nightly_linux_repeated', 'nightly_osx_repeated']
 - [ ] projectroot.test_demo_cyclic_pipeline__rmw_fastrtps_cpp, ['nightly_linux-aarch64_repeated', 'nightly_linux_repeated', 'nightly_osx_repeated']
 - [ ] projectroot.test_externally_defined_services, ['nightly_osx_repeated']
 - [ ] projectroot.test_find_weak_nodes, ['nightly_osx_repeated']
 - [ ] projectroot.test_get_node_names__rmw_connext_cpp, ['nightly_osx_repeated']
 - [ ] projectroot.test_get_node_names__rmw_fastrtps_cpp, ['nightly_linux-aarch64_repeated', 'nightly_osx_repeated']
 - [ ] projectroot.test_graph__rmw_fastrtps_cpp, ['nightly_linux_repeated', 'nightly_osx_repeated', 'nightly_win_rep']
 - [ ] projectroot.test_node, ['nightly_linux-aarch64_repeated']
 - [ ] projectroot.test_pendulum__rmw_connext_cpp.test_pendulum__rmw_connext_cpp, ['nightly_linux_repeated']
 - [ ] projectroot.test_pendulum_teleop__rmw_connext_cpp, ['nightly_linux_repeated']
 - [ ] projectroot.test_secure_publisher_subscriber__DynamicArrayNested__rmw_fastrtps_cpp__secure_comm_0, ['nightly_linux_repeated', 'nightly_osx_repeated', 'nightly_win_rep']
 - [ ] projectroot.test_secure_publisher_subscriber__DynamicArrayNested__rmw_fastrtps_cpp__secure_comm_1, ['nightly_win_rep']
 - [ ] projectroot.test_secure_publisher_subscriber__DynamicArrayNested__rmw_fastrtps_cpp__secure_comm_2, ['nightly_win_rep']
 - [ ] projectroot.test_secure_publisher_subscriber__DynamicArrayNested__rmw_fastrtps_cpp__secure_comm_3, ['nightly_win_rep']
 - [ ] projectroot.test_secure_publisher_subscriber__Empty__rmw_fastrtps_cpp__secure_comm_0, ['nightly_win_rep']
 - [ ] projectroot.test_secure_publisher_subscriber__Empty__rmw_fastrtps_cpp__secure_comm_2, ['nightly_win_rep']
 - [ ] projectroot.test_service__rmw_connext_cpp, ['nightly_osx_repeated']
 - [ ] projectroot.test_service__rmw_fastrtps_cpp, ['nightly_osx_repeated']
 - [ ] projectroot.test_services__rmw_connext_cpp, ['nightly_win_rep']
 - [ ] projectroot.test_services__rmw_fastrtps_cpp, ['nightly_linux_repeated']
 - [ ] projectroot.test_services_cpp__rmw_fastrtps_cpp, ['nightly_linux_repeated']
 - [ ] projectroot.test_showimage_cam2image__rmw_fastrtps_cpp, ['nightly_linux-aarch64_repeated', 'nightly_linux_repeated', 'nightly_win_rep']
 - [ ] projectroot.test_tlsf__rmw_connext_cpp, ['nightly_linux_repeated']
 - [ ] projectroot.test_tutorial_add_two_ints_server_add_two_ints_client__rmw_connext_cpp, ['nightly_osx_repeated']
 - [ ] projectroot.test_tutorial_add_two_ints_server_add_two_ints_client__rmw_fastrtps_cpp, ['nightly_linux-aarch64_repeated', 'nightly_linux_repeated', 'nightly_win_rep']
 - [ ] projectroot.test_tutorial_add_two_ints_server_add_two_ints_client_async__rmw_fastrtps_cpp, ['nightly_linux_repeated', 'nightly_osx_repeated', 'nightly_win_rep']
 - [ ] projectroot.test_tutorial_list_parameters__rmw_connext_cpp, ['nightly_linux_repeated']
 - [ ] projectroot.test_tutorial_list_parameters__rmw_fastrtps_cpp, ['nightly_linux-aarch64_repeated', 'nightly_linux_repeated', 'nightly_osx_repeated']
 - [ ] projectroot.test_tutorial_list_parameters_async__rmw_fastrtps_cpp, ['nightly_linux-aarch64_repeated', 'nightly_linux_repeated', 'nightly_osx_repeated']
 - [ ] projectroot.test_tutorial_parameter_events__rmw_connext_cpp, ['nightly_osx_repeated']
 - [ ] projectroot.test_tutorial_parameter_events__rmw_fastrtps_cpp, ['nightly_osx_repeated']
 - [ ] projectroot.test_tutorial_parameter_events_async__rmw_fastrtps_cpp, ['nightly_linux_repeated']
 - [ ] projectroot.test_tutorial_set_and_get_parameters__rmw_connext_cpp, ['nightly_osx_repeated']
 - [ ] projectroot.test_tutorial_set_and_get_parameters__rmw_fastrtps_cpp, ['nightly_linux-aarch64_repeated', 'nightly_linux_repeated', 'nightly_osx_repeated']
 - [ ] projectroot.test_tutorial_set_and_get_parameters_async__rmw_fastrtps_cpp, ['nightly_linux-aarch64_repeated', 'nightly_linux_repeated', 'nightly_osx_repeated']
 - [ ] projectroot.test_wait__rmw_connext_cpp, ['nightly_win_rep']
 - [ ] projectroot.test_wait__rmw_fastrtps_cpp, ['nightly_win_rep']
 - [ ] test.test_timer.test_timer_zero_callbacks1000hertz, ['nightly_osx_repeated']
 - [ ] test_add_two_ints_server_add_two_ints_client__rmw_connext_cpp_.test_executable, ['nightly_osx_repeated']
 - [ ] test_add_two_ints_server_add_two_ints_client__rmw_fastrtps_cpp_.test_executable, ['nightly_linux-aarch64_repeated']
 - [ ] test_add_two_ints_server_add_two_ints_client__rmw_fastrtps_cpp_Release.test_executable, ['nightly_win_rep']
 - [ ] test_add_two_ints_server_add_two_ints_client_async__rmw_fastrtps_cpp_.test_executable, ['nightly_linux_repeated', 'nightly_osx_repeated']
 - [ ] test_add_two_ints_server_add_two_ints_client_async__rmw_fastrtps_cpp_Release.test_executable, ['nightly_win_rep']
 - [ ] test_client_scope_consistency_cpp__rmw_fastrtps_cpp_.test_client_scope_consistency_cpp, ['nightly_linux_repeated', 'nightly_osx_repeated']
 - [ ] test_client_scope_cpp__rmw_fastrtps_cpp_.test_client_scope_cpp, ['nightly_osx_repeated']
 - [ ] test_composition__rmw_connext_cpp_.test_dlopen_composition, ['nightly_osx_repeated']
 - [ ] test_composition__rmw_connext_cpp_.test_linktime_composition, ['nightly_osx_repeated']
 - [ ] test_composition__rmw_fastrtps_cpp_.test_dlopen_composition, ['nightly_osx_repeated']
 - [ ] test_composition__rmw_fastrtps_cpp_Release.test_api_srv_composition, ['nightly_win_rep']
 - [ ] test_multithreaded__rmw_connext_cpp.multi_access_publisher_intra_process, ['nightly_linux_repeated', 'nightly_osx_repeated', 'nightly_win_rep']
 - [ ] test_multithreaded__rmw_fastrtps_cpp.multi_consumer_intra_process, ['nightly_linux_repeated', 'nightly_osx_repeated']
 - [ ] test_pendulum__rmw_connext_cpp.test_executable, ['nightly_linux_repeated']
 - [ ] test_publisher_subscriber__Nested__rclpy__rmw_fastrtps_cpp_.test_publisher_subscriber, ['nightly_linux_repeated']
 - [ ] test_requester_replier__Empty__rclpy__rmw_fastrtps_cpp_.test_requester_replier, ['nightly_linux-aarch64_repeated', 'nightly_linux_repeated', 'nightly_osx_repeated']
 - [ ] test_requester_replier__Empty__rclpy__rmw_fastrtps_cpp_Release.test_requester_replier, ['nightly_win_rep']
 - [ ] test_requester_replier__Primitives__rclpy__rmw_fastrtps_cpp_.test_requester_replier, ['nightly_linux-aarch64_repeated', 'nightly_linux_repeated', 'nightly_osx_repeated']
 - [ ] test_requester_replier__Primitives__rclpy__rmw_fastrtps_cpp_Release.test_requester_replier, ['nightly_win_rep']
 - [ ] test_secure_publisher_subscriber__DynamicArrayNested__rmw_fastrtps_cpp__secure_comm_0_.test_secure_publisher_subscriber, ['nightly_linux_repeated', 'nightly_osx_repeated']
 - [ ] test_secure_publisher_subscriber__DynamicArrayNested__rmw_fastrtps_cpp__secure_comm_0_Release.test_secure_publisher_subscriber, ['nightly_win_rep']
 - [ ] test_secure_publisher_subscriber__DynamicArrayNested__rmw_fastrtps_cpp__secure_comm_1_Release.test_secure_publisher_subscriber, ['nightly_win_rep']
 - [ ] test_secure_publisher_subscriber__DynamicArrayNested__rmw_fastrtps_cpp__secure_comm_2_Release.test_secure_publisher_subscriber, ['nightly_win_rep']
 - [ ] test_secure_publisher_subscriber__DynamicArrayNested__rmw_fastrtps_cpp__secure_comm_3_Release.test_secure_publisher_subscriber, ['nightly_win_rep']
 - [ ] test_secure_publisher_subscriber__Empty__rmw_fastrtps_cpp__secure_comm_0_Release.test_secure_publisher_subscriber, ['nightly_win_rep']
 - [ ] test_secure_publisher_subscriber__Empty__rmw_fastrtps_cpp__secure_comm_2_Release.test_secure_publisher_subscriber, ['nightly_win_rep']
 - [ ] test_services__rmw_connext_cpp_Release.test_services, ['nightly_win_rep']
 - [ ] test_services_cpp__rmw_fastrtps_cpp_.test_services_cpp, ['nightly_linux_repeated']
 - [ ] test_showimage_cam2image__rmw_fastrtps_cpp_.test_reliable_qos, ['nightly_linux-aarch64_repeated', 'nightly_linux_repeated']
 - [ ] test_showimage_cam2image__rmw_fastrtps_cpp_Release.test_reliable_qos, ['nightly_win_rep']
 - [ ] test_timeout_subscriber__rmw_connext_cpp.timeout_subscriber, ['nightly_win_rep']

nightly windows machine failing to find OpenSSL

It's hard for me to tell if this is an issue with Fast-RTPS's CMake or something else (because of the parallel interleaving).

07:33:35 CMake Error at C:/Program Files (x86)/CMake/share/cmake-3.5/Modules/FindPackageHandleStandardArgs.cmake:148 (message):
07:33:35   Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the
07:33:35   system variable OPENSSL_ROOT_DIR (missing: OPENSSL_LIBRARIES
07:33:35   OPENSSL_INCLUDE_DIR)
07:33:35 Call Stack (most recent call first):
07:33:35   C:/Program Files (x86)/CMake/share/cmake-3.5/Modules/FindPackageHandleStandardArgs.cmake:388 (_FPHSA_FAILURE_MESSAGE)
07:33:35   C:/Program Files (x86)/CMake/share/cmake-3.5/Modules/FindOpenSSL.cmake:370 (find_package_handle_standard_args)
07:33:35   CMakeLists.txt:211 (find_package)

-- http://ci.ros2.org/job/nightly_win_rep/687/consoleFull

@mikaelarguedas do you have any idea?

OSX nightly release build failed a tf2 test

This job: http://ci.ros2.org/view/nightly/job/nightly_osx_release/433/

@dhood and @clalancette both pushed some changes to tf2 yesterday, so either one of those could be at fault. However, in both cases the CI was green before the push, so it is odd that this failed. Also, this particular test did not fail on the other OSs. We need to look into this and figure out what is going on.

Update java on Windows and Mac OS machines

Java8u144 is available and updates pop-ups show up on most machines. Looks like some windows ones updated automatically. We should update a few, test and if all goes well update the remaining ones

WaitSetTestFixture__rmw_connext_cpp/zero_timeout flaky on windows

This has been flaky for a while but I saw it fail 10 times today so I figured I'll report it here.
It's this test: https://github.com/ros2/rcl/blob/90176d9f1a3551deb861e698f36f81534e7763ac/rcl/test/rcl/test_wait.cpp#L131

Failed on this ci; http://ci.ros2.org/job/ci_windows/2612/testReport/junit/(root)/WaitSetTestFixture__rmw_connext_cpp/zero_timeout/

Fails only on windows, it doesn't seem to be machine related AFAIK

mass failures due to setuptools bug

See: pypa/setuptools#1042

Basically all our Linux jobs are failing, here is a sample:

I think the best thing to do is to wait for a new setuptools release. They are aware of it and should fix it asap since it's affecting so many. If it is not fixed in the morning I'll try to find a way to version lock us to setuptools < 36 in all the right places.

Uncrustify unit test failed on windows when repeated

An uncrustify unit test failed on the 13th retry on the windows repeated job last night:
http://ci.ros2.org/job/nightly_win_rep/629/testReport/junit/projectroot/tests/test_uncrustify/

Stacktrace

Tests: ['c-sharp', 'c', 'cpp', 'd', 'java', 'pawn', 'objective-c', 'vala', 'ecma']
Processing c-sharp.test
Processing c.test
FAILED2: 01021
UNSTABLE: 02100
UNSTABLE: 02102
Processing cpp.test
UNSTABLE: 30265
UNSTABLE: 30815
Processing d.test
Processing java.test
Processing pawn.test
Processing objective-c.test
UNSTABLE: 50007
Processing vala.test
Processing ecma.test
Passed 615 / 616 tests
Failed 1 test(s)

I opened ament/uncrustify#16 but won't spend more time fixing this.

Linux packaging job is still unstable

For example: http://ci.ros2.org/view/packaging/job/packaging_linux/704/

I thought this was going to be resolved by:

ros2/rmw_fastrtps#114 (and friends)

But it looks like something else is wrong. I'll look into it tomorrow.

test_communication took six hours on MacOS

This ci_osx job had been running for more than six hours when I came to check on builds this morning. I haven't yet killed it, because I want to know what other tests have missing or malfunctioning timeouts.

It doesn't appear that the job ever locked up. Instead it spent just over six hours on the Test: test_communication step. I have not yet determined if this is due to a misconfigured timeout or a missing timeout or something else entirely.

Earlier tests also appear to have failed but I have not yet dove into that either.

nightly_win_rep has some known flakey tests

There were a few new issues in the Windows build, and some that went away. New issues:

projectroot.test_wait__rmw_connext_cpp
projectroot.test_wait__rmw_fastrtps_cpp
WaitSetTestFixture__rmw_fastrtps_cpp.finite_timeout
WaitSetTestFixture__rmw_connext_cpp.zero_timeout

Update chocolatey version on windows machines

And also chocolatey packages versions as well.

This has proven to be a problem for users installing from "binaries" having a more recent version than us see ros2/ros2#311

Segfault during nightly_osx_release

Build nightly_osx_release #416 failed with a segfault. A long excerpt is in a gist here

Here's a short excerpt to guide searches for future buildcops.

07:35:41 Traceback (most recent call last):
07:35:41   File "/Users/osrf/jenkins/workspace/nightly_osx_release/ws/install/rosidl_typesupport_connext_cpp/lib/rosidl_typesupport_connext_cpp/rosidl_typesupport_connext_cpp", line 61, in <module>
07:35:41     sys.exit(main())
07:35:41   File "/Users/osrf/jenkins/workspace/nightly_osx_release/ws/install/rosidl_typesupport_connext_cpp/lib/rosidl_typesupport_connext_cpp/rosidl_typesupport_connext_cpp", line 56, in main
07:35:41     service_specs,
07:35:41   File "/Users/osrf/jenkins/workspace/nightly_osx_release/ws/install/rosidl_typesupport_connext_cpp/lib/python3.6/site-packages/rosidl_typesupport_connext_cpp/__init__.py", line 89, in generate_dds_connext_cpp
07:35:41     subprocess.check_call(cmd)
07:35:41   File "/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 291, in check_call
07:35:41     raise CalledProcessError(retcode, cmd)
07:35:41 subprocess.CalledProcessError: Command '['/Applications/rti_connext_dds-5.2.3/bin/rtiddsgen_server', '-I', '/Users/osrf/jenkins/workspace/nightly_osx_release/ws/build/pendulum_msgs/rosidl_generator_dds_idl', '-I', '/Users/osrf/jenkins/workspace/nightly_osx_release/ws/install/builtin_interfaces/share', '-d', '/Users/osrf/jenkins/workspace/nightly_osx_release/ws/build/pendulum_msgs/rosidl_typesupport_connext_cpp/pendulum_msgs/msg/dds_connext', '-language', 'C++', '-namespace', '-update', 'typefiles', '-unboundedSupport', '/Users/osrf/jenkins/workspace/nightly_osx_release/ws/build/pendulum_msgs/rosidl_generator_dds_idl/pendulum_msgs/msg/dds_connext/JointState_.idl']' died with <Signals.SIGSEGV: 11>.

I triggered a manual rebuild http://ci.ros2.org/view/nightly/job/nightly_osx_release/417 which succeeded so this is more to start the papertrail for future issues.

packet machine crash on "resource busy"

When running several jobs in parallel on the packet machine sometimes docker crashes throwing
"device or resource busy" or "mkdir /var/run/docker/libcontainerd/containerd/: file exists"

examples:
http://ci.ros2.org/view/nightly/job/nightly_linux-aarch64_repeated/127
http://ci.ros2.org/view/nightly/job/nightly_linux-aarch64_repeated/112
http://ci.ros2.org/view/nightly/job/nightly_linux-aarch64_repeated/109

@nuclearsandwich FYI

Service calls test takes a long time on connext

Job

http://ci.ros2.org/view/nightly/job/nightly_linux_repeated/772/
linux_slave_on_master

Test

test_rclcpp.gtest_multiple_service_calls__rmw_connext_cpp.gtest.missing_result
projectroot.gtest_multiple_service_calls__rmw_connext_cpp

The test timed out after 30 seconds. It's gtest based and has a couple tests inside of the executable. One of them took a long time (23 seconds), and the second one timed out.

Failed output

      Start  8: gtest_multiple_service_calls__rmw_connext_cpp

8: Test command: /home/rosbuild/ci_scripts/venv/bin/python3 "-u" "/home/rosbuild/ci_scripts/ws/install/ament_cmake_test/share/ament_cmake_test/cmake/run_test.py" "/home/rosbuild/ci_scripts/ws/build/test_rclcpp/test_results/test_rclcpp/gtest_multiple_service_calls__rmw_connext_cpp.gtest.xml" "--output-file" "/home/rosbuild/ci_scripts/ws/build/test_rclcpp/ament_cmake_gtest/gtest_multiple_service_calls__rmw_connext_cpp.txt" "--env" "RCL_ASSERT_RMW_ID_MATCHES=rmw_connext_cpp" "RMW_IMPLEMENTATION=rmw_connext_cpp" "--append-env" "LD_LIBRARY_PATH=/home/rosbuild/ci_scripts/ws/build/test_rclcpp" "--command" "/home/rosbuild/ci_scripts/ws/build/test_rclcpp/gtest_multiple_service_calls__rmw_connext_cpp" "--gtest_output=xml:/home/rosbuild/ci_scripts/ws/build/test_rclcpp/test_results/test_rclcpp/gtest_multiple_service_calls__rmw_connext_cpp.gtest.xml"
8: Test timeout computed to be: 30
8: -- run_test.py: extra environment variables:
8:  - RMW_IMPLEMENTATION=rmw_connext_cpp
8:  - RCL_ASSERT_RMW_ID_MATCHES=rmw_connext_cpp
8: -- run_test.py: extra environment variables to append:
8:  - LD_LIBRARY_PATH+=/home/rosbuild/ci_scripts/ws/build/test_rclcpp
8: -- run_test.py: invoking following command in '/home/rosbuild/ci_scripts/ws/src/ros2/system_tests/test_rclcpp':
8:  - /home/rosbuild/ci_scripts/ws/build/test_rclcpp/gtest_multiple_service_calls__rmw_connext_cpp --gtest_output=xml:/home/rosbuild/ci_scripts/ws/build/test_rclcpp/test_results/test_rclcpp/gtest_multiple_service_calls__rmw_connext_cpp.gtest.xml
8: [==========] Running 2 tests from 2 test cases.
8: [----------] Global test environment set-up.
8: [----------] 1 test from test_two_service_calls__rmw_connext_cpp
8: [ RUN      ] test_two_service_calls__rmw_connext_cpp.two_service_calls
8: RTI Data Distribution Service Evaluation License issued to OSRF - OSRF01 [email protected] For non-production use only.
8: Expires on 05-may-2018 See www.rti.com for more information.
8: Sending two requests...
8: Waiting for first reply...
8: Received first reply
8: Waiting for second reply...
8: Received second reply
8: [       OK ] test_two_service_calls__rmw_connext_cpp.two_service_calls (23417 ms)
8: [----------] 1 test from test_two_service_calls__rmw_connext_cpp (23417 ms total)
8: 
8: [----------] 1 test from test_multiple_service_calls__rmw_connext_cpp
8: [ RUN      ] test_multiple_service_calls__rmw_connext_cpp.multiple_clients
8: RTI Data Distribution Service Evaluation License issued to OSRF - OSRF01 [email protected] For non-production use only.
8: Expires on 05-may-2018 See www.rti.com for more information.
8: Sending 5 requests...
 8/40 Test  #8: gtest_multiple_service_calls__rmw_connext_cpp .................***Timeout  30.00 sec

Successful output
On a successful run the test before the failing one took much less time: 3 seconds instead of 23 seconds.

      Start  8: gtest_multiple_service_calls__rmw_connext_cpp

8: Test command: /home/rosbuild/ci_scripts/venv/bin/python3 "-u" "/home/rosbuild/ci_scripts/ws/install/ament_cmake_test/share/ament_cmake_test/cmake/run_test.py" "/home/rosbuild/ci_scripts/ws/build/test_rclcpp/test_results/test_rclcpp/gtest_multiple_service_calls__rmw_connext_cpp.gtest.xml" "--output-file" "/home/rosbuild/ci_scripts/ws/build/test_rclcpp/ament_cmake_gtest/gtest_multiple_service_calls__rmw_connext_cpp.txt" "--env" "RCL_ASSERT_RMW_ID_MATCHES=rmw_connext_cpp" "RMW_IMPLEMENTATION=rmw_connext_cpp" "--append-env" "LD_LIBRARY_PATH=/home/rosbuild/ci_scripts/ws/build/test_rclcpp" "--command" "/home/rosbuild/ci_scripts/ws/build/test_rclcpp/gtest_multiple_service_calls__rmw_connext_cpp" "--gtest_output=xml:/home/rosbuild/ci_scripts/ws/build/test_rclcpp/test_results/test_rclcpp/gtest_multiple_service_calls__rmw_connext_cpp.gtest.xml"
8: Test timeout computed to be: 30
8: -- run_test.py: extra environment variables:
8:  - RMW_IMPLEMENTATION=rmw_connext_cpp
8:  - RCL_ASSERT_RMW_ID_MATCHES=rmw_connext_cpp
8: -- run_test.py: extra environment variables to append:
8:  - LD_LIBRARY_PATH+=/home/rosbuild/ci_scripts/ws/build/test_rclcpp
8: -- run_test.py: invoking following command in '/home/rosbuild/ci_scripts/ws/src/ros2/system_tests/test_rclcpp':
8:  - /home/rosbuild/ci_scripts/ws/build/test_rclcpp/gtest_multiple_service_calls__rmw_connext_cpp --gtest_output=xml:/home/rosbuild/ci_scripts/ws/build/test_rclcpp/test_results/test_rclcpp/gtest_multiple_service_calls__rmw_connext_cpp.gtest.xml
8: [==========] Running 2 tests from 2 test cases.
8: [----------] Global test environment set-up.
8: [----------] 1 test from test_two_service_calls__rmw_connext_cpp
8: [ RUN      ] test_two_service_calls__rmw_connext_cpp.two_service_calls
8: RTI Data Distribution Service Evaluation License issued to OSRF - OSRF01 [email protected] For non-production use only.
8: Expires on 05-may-2018 See www.rti.com for more information.
8: Sending two requests...
8: Waiting for first reply...
8: Received first reply
8: Waiting for second reply...
8: Received second reply
8: [       OK ] test_two_service_calls__rmw_connext_cpp.two_service_calls (3414 ms)
8: [----------] 1 test from test_two_service_calls__rmw_connext_cpp (3415 ms total)
8: 
8: [----------] 1 test from test_multiple_service_calls__rmw_connext_cpp
8: [ RUN      ] test_multiple_service_calls__rmw_connext_cpp.multiple_clients
8: RTI Data Distribution Service Evaluation License issued to OSRF - OSRF01 [email protected] For non-production use only.
8: Expires on 05-may-2018 See www.rti.com for more information.
8: Sending 5 requests...
8: Got response #0 with value 1
8: Got response #1 with value 3
8: Got response #2 with value 5
8: Got response #3 with value 7
8: Got response #4 with value 9
8: [       OK ] test_multiple_service_calls__rmw_connext_cpp.multiple_clients (3444 ms)
8: [----------] 1 test from test_multiple_service_calls__rmw_connext_cpp (3444 ms total)
8: 
8: [----------] Global test environment tear-down
8: [==========] 2 tests from 2 test cases ran. (6859 ms total)
8: [  PASSED  ] 2 tests.
8: -- run_test.py: return code 0
8: -- run_test.py: verify result file '/home/rosbuild/ci_scripts/ws/build/test_rclcpp/test_results/test_rclcpp/gtest_multiple_service_calls__rmw_connext_cpp.gtest.xml'
 8/40 Test  #8: gtest_multiple_service_calls__rmw_connext_cpp .................   Passed    7.00 sec

unpin and upgrade tinyxml2 to version 5.0.0 on macOS

We've currently pinned the version of tinyxml2 to 4.0.1 on our macOS build machines, but we should update them to the latest version once Fast-RTPS integrates my patch (or a similar patch) into their master:

eProsima/Fast-DDS#114

Also the instructions should be updated too:

osx failed with segfault from rti code generator

08:39:26 subprocess.CalledProcessError: Command '['/Applications/rti_connext_dds-5.2.3/bin/rtiddsgen_server', '-I', '/Users/osrf/jenkins/workspace/nightly_osx_repeated/ws/build/geometry_msgs/rosidl_generator_dds_idl', '-I', '/Users/osrf/jenkins/workspace/nightly_osx_repeated/ws/install/std_msgs/share', '-I', '/Users/osrf/jenkins/workspace/nightly_osx_repeated/ws/install/builtin_interfaces/share', '-d', '/Users/osrf/jenkins/workspace/nightly_osx_repeated/ws/build/geometry_msgs/rosidl_typesupport_connext_cpp/geometry_msgs/msg/dds_connext', '-language', 'C++', '-namespace', '-update', 'typefiles', '-unboundedSupport', '/Users/osrf/jenkins/workspace/nightly_osx_repeated/ws/build/geometry_msgs/rosidl_generator_dds_idl/geometry_msgs/msg/dds_connext/QuaternionStamped_.idl']' died with <Signals.SIGSEGV: 11>.

-- http://ci.ros2.org/job/nightly_osx_repeated/723/consoleFull

I restarted it, but keep an eye on it.

upgrade dependencies on macOS ci.ros2.org build nodes

This is process I am using.

Once for the whole upgrade process:

duplicate ci_osx, changing the label to have a _testing suffix

(I'm only testing ci_osx because macOS doesn't do anything different for debug builds or packaging builds)

take the node offline
change the label by adding a _testing suffix
vnc in to see if there are updates (open App Store app), do them if there are
ssh or vnc in to terminal and do brew update (this updated brew's definitions, like apt update)
do brew outdated to see what can be upgraded
pin the version of tinyxml2 for now brew pin tinyxml2
- if you accidentally already installed 5.0.0, you can install 4.0.1 with this: brew uninstall tinyxml2; brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/196dee239038f34b2bbe8ca2f183ec78a5a53ce2/Formula/tinyxml2.rb && brew pin tinyxml2
- see: eProsima/Fast-DDS#113 and eProsima/Fast-DDS#114
either do brew upgrade to upgrade all or do selective ones with brew upgrade [[[pkg1] pkg2] ...]
update to Java 8 update 131 (by downloading from java website via safari via vnc)
reboot
bring the node back online (with the adjusted label in place) and run the ci_osx_testing job (ensuring that it gets on the target machine by disabling the other machines on that label if any)
if the test passes, change the label back (remove the _testing suffix)

nightly_linux_repeated had different failures from the night before

http://ci.ros2.org/view/nightly/job/nightly_linux_repeated/691/

It had both new failures, and didn't have some of the failures from the night before, suggesting that some of the tests were either fixed or flakey. Hoping for the former, here are the new failures:

projectroot.test_demo_cyclic_pipeline__rmw_connext_cpp
projectroot.test_messages_c__rmw_connext_cpp
projectroot.test_tf2_geometry_msgs
projectroot.test_showimage_cam2image__rmw_fastrtps_cpp
TfGeometry.Frame
TfGeometry.Vector
TfGeometry.Point
TestMessagesFixture.test_dynamicarraynested
test_showimage_cam2image__rmw_fastrtps_cpp_.test_reliable_qos
test_pendulum__rmw_connext_cpp.test_executable
intra_process_demo.test_demo_cyclic_pipeline__rmw_connext_cpp.xunit.missing_result
test_services_cpp__rmw_fastrtps_cpp_.test_services_cpp

Connext no space left on device during pendulum demo on linux coverage job

Pendulum demo tests on this linux coverage job on linux3 failed with a lot of connext errors.

Both test_pendulum_teleop__rmw_connext_cpp.test_executable and projectroot.test_pendulum__rmw_connext_cpp.test_pendulum__rmw_connext_cpp output the following warning multiple times:

[test_executable_0] [D0108|ENABLE]NDDS_Transport_Shmem_create_recvresource_rrEA:failed to initialize shared memory resource mutex for key 0xb086c2
[test_executable_0] [D0108|ENABLE]RTIOsapiSharedMemoryMutex_create:OS semget() failure, error 0X1C: No space left on device

before finally:

[test_executable_0] [D0108|ENABLE]DDS_DomainParticipantPresentation_reserve_participant_index_entryports:!enable reserve participant index
[test_executable_0] [D0108|ENABLE]DDS_DomainParticipant_enableI:Automatic participant index failed to initialize. PLEASE VERIFY CONSISTENT TRANSPORT / DISCOVERY CONFIGURATION. 
[test_executable_0] DDSDomainParticipant_impl::createI:ERROR: Failed to auto-enable entity
[test_executable_0] DomainParticipantFactory_impl::create_participant():!create failure creating participant
(test_executable_0) rc -6
[test_executable_1] [D0108|ENABLE]NDDS_Transport_Shmem_create_recvresource_rrEA:failed to initialize shared memory resource mutex for key 0xb08758
[test_executable_1] [D0108|ENABLE]DDS_DomainParticipantPresentation_reserve_participant_index_entryports:!enable reserve participant index
[test_executable_1] [D0108|ENABLE]DDS_DomainParticipant_enableI:Automatic participant index failed to initialize. PLEASE VERIFY CONSISTENT TRANSPORT / DISCOVERY CONFIGURATION. 
[test_executable_1] DDSDomainParticipant_impl::createI:ERROR: Failed to auto-enable entity
[test_executable_1] DomainParticipantFactory_impl::create_participant():!create failure creating participant
(test_executable_1) rc 250

document how to set up and access aarch64 machine

The packet.net machine is down and I can't fine documentation on how to access it to revive it

nightly_linux-aarch64-repeated has flakey tests

http://ci.ros2.org/view/nightly/job/nightly_linux-aarch64_repeated/78/

Listed here:

projectroot.test_tutorial_add_two_ints_server_add_two_ints_client_async__rmw_fastrtps_cpp
test_add_two_ints_server_add_two_ints_client_async__rmw_fastrtps_cpp_.test_executable

installing connext fails with timeout and ifconfig not found

I was surprised by /entry_point.sh: 20: /entry_point.sh: ifconfig: not found which was in this job failure:

http://ci.ros2.org/view/nightly/job/nightly_linux_debug/469/console

I think it is just a timeout while installing or something. Perhaps this is a bug in the retry branch of the logic, but I don't know. I'll look into it tomorrow.

Build cop notes and tracking 2017-05-17 through 2017-05-31ish

Regressions 2017-05-17

Nightly OSX Repeated #688

test_tutorial_add_two_ints_client_async_rmw_fastrtps_cpp 🔗 link
test_add_two_ints_server_add_two_ints_client_async__rmw_fastrtps_cpp_.test_executable🔗 link

The test output what appears to be a successful communication between the two processes. The issue looks like it might be related to differences in expected test output.

Nightly Linux Coverage #383

projectroot.test_tutorial_add_two_ints_server_add_two_ints_client__rmw_fastrtps_cpp 🔗 link
projectroot.test_tutorial_add_two_ints_server_add_two_ints_client_async__rmw_fastrtps_cpp 🔗 link
demo_nodes_cpp.test_tutorial_add_two_ints_server_add_two_ints_client_async__rmw_fastrtps_cpp.xunit.missing_result 🔗 link
demo_nodes_cpp.test_tutorial_add_two_ints_server_add_two_ints_client__rmw_fastrtps_cpp.xunit.missing_result 🔗 link

I can't find anything that looks obviously like the reson for the timeouts in the console logs for job #383 demo_nodes_cpp.

rtiddsgen crashes sporadically on OSX

we knew that was an issue requiring patching on Windows. It showed up on OSX last night:
at elapsed time 00:27:34.721: http://ci.ros2.org/view/nightly/job/nightly_osx_debug/514.

I remember seeing the same happen on Linux recently, not sure if it was my work station or a CI job.

perf test failures due to slave service on OSX

A lot of new test failures on the OS X debug nightly, most seem time-related:

Test Result (32 failures / +32)
projectroot.rclpytests
projectroot.gtest_spin__rmw_connext_cpp
projectroot.gtest_timer__rmw_connext_cpp
projectroot.gtest_timeout_subscriber__rmw_connext_cpp
projectroot.gtest_local_parameters__rmw_connext_cpp
projectroot.gtest_spin__rmw_fastrtps_cpp
projectroot.gtest_timer__rmw_fastrtps_cpp
projectroot.gtest_timeout_subscriber__rmw_fastrtps_cpp
projectroot.test_rate
projectroot.test_wait__rmw_connext_cpp
projectroot.test_wait__rmw_fastrtps_cpp
test_spin__rmw_connext_cpp.spin_until_future_complete_timeout
test_spin__rmw_connext_cpp.spin_until_future_complete_interrupted
test_spin__rmw_fastrtps_cpp.spin_until_future_complete_timeout
test_spin__rmw_fastrtps_cpp.spin_until_future_complete_interrupted
test_timeout_subscriber__rmw_connext_cpp.timeout_subscriber
TestRate.rate_basics
TestRate.wall_rate_basics
test_timeout_subscriber__rmw_fastrtps_cpp.timeout_subscriber
test_rclcpp.gtest_local_parameters__rmw_connext_cpp.gtest.missing_result
test_time__rmw_connext_cpp.timer_fire_regularly
WaitSetTestFixture__rmw_connext_cpp.finite_timeout
WaitSetTestFixture__rmw_connext_cpp.negative_timeout
WaitSetTestFixture__rmw_connext_cpp.canceled_timer
test_time__rmw_fastrtps_cpp.timer_fire_regularly
WaitSetTestFixture__rmw_fastrtps_cpp.finite_timeout
WaitSetTestFixture__rmw_fastrtps_cpp.negative_timeout
WaitSetTestFixture__rmw_fastrtps_cpp.canceled_timer
test.test_timer.test_timer_zero_callbacks1000hertz
test.test_timer.test_timer_number_callbacks100hertz
test.test_timer.test_timer_cancel_reset_10hertz
test.test_timer.test_timer_cancel_reset_1000hertz

there are new failures on the repeated osx job so this might not just be debug-related.
they didn't fail on the nightly osx release job so it might be the configuration of mini1 & mini2 are different to dosa at the moment.

Newly flaky tests 19 april 2017

The following tests failed during the "flakiness finder" repeat-until-fail nightly jobs last night, but passed in the "flakiness ignorer" repeat-until-pass nightly jobs.

It's good that they are passing when repeated, but I don't remember them being flaky before.

http://ci.ros2.org/job/nightly_linux_repeated/624/testReport/junit/(root)/TestStateMachineInfo/available_transitions/ http://ci.ros2.org/view/nightly/job/nightly_linux-aarch64_repeated/15/testReport/junit/(root)/TestStateMachineInfo/available_transitions/

Stacktrace

[TestStateMachineInfo.available_transitions](/home/rosbuild/ci_scripts/ws/src/ros2/rclcpp/rclcpp_lifecycle/test/test_state_machine_info.cpp:66)
Value of: transition.start_state().label().empty()
  Actual: true
Expected: false

ros2/rclcpp#324

projectroot.test_showimage_cam2image__rmw_fastrtps_cpp
timeout (30s)

test.test_timer.test_timer_zero_callbacks1000hertz

Traceback (most recent call last):
  File "/usr/lib/python3.5/unittest/case.py", line 58, in testPartExecutor
    yield
  File "/usr/lib/python3.5/unittest/case.py", line 600, in run
    testMethod()
  File "/home/rosbuild/ci_scripts/venv/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/rosbuild/ci_scripts/ws/src/ros2/rclpy/rclpy/test/test_timer.py", line 150, in test_timer_zero_callbacks1000hertz
    func_launch(func_zero_callback, ['0.001'], 'received callbacks when not expected')
  File "/home/rosbuild/ci_scripts/ws/src/ros2/rclpy/rclpy/test/test_timer.py", line 127, in func_launch
    args=(function, args)
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 253, in apply
    return self.apply_async(func, args, kwds).get()
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 608, in get
    raise self._value
AssertionError: shouldn't have received any callback, received 1

waits for 0.0005s for a timer with period 0.001s, checking it doesn't receive anything, but it does. Underneath it must have taken longer than expected to wake up.

test_pendulum__rmw_connext_cpp.test_executable

Stacktrace

  File "/usr/lib/python3.5/unittest/case.py", line 58, in testPartExecutor
    yield
  File "/usr/lib/python3.5/unittest/case.py", line 600, in run
    testMethod()
  File "/home/rosbuild/ci_scripts/venv/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/rosbuild/ci_scripts/ws/build/pendulum_control/test_pendulum__rmw_connext_cpp.py", line 60, in test_executable
    handler.check()
  File "/home/rosbuild/ci_scripts/ws/install/launch_testing/lib/python3.5/site-packages/launch_testing/__init__.py", line 106, in check
    (output_lines, self.expected_lines))
Example output ([b'Initial major pagefaults: 0', b'Initial minor pagefaults: 8625', b'PRESWriterHistoryDriver_completeBeAsynchPub:!make_sample_reclaimable', b'rttest statistics:', b'  - Minor pagefaults: 36', b'  - Major pagefaults: 0', b'  Latency (time after deadline was missed):', b'    - Min: 17403 ns', b'    - Max: 16213992 ns', b'    - Mean: 284679 ns', b'    - Standard deviation: 685.729', b'', b'', b'PendulumMotor received 242 messages', b'PendulumController received 245 messages']) does not match expected output ([b'Initial major pagefaults: \\d+', b'Initial minor pagefaults: \\d+', b'rttest statistics:', b'  - Minor pagefaults: \\d+', b'  - Major pagefaults: \\d+', b'  Latency \\(time after deadline was missed\\):', b'    - Min: \\d+ ns', b'    - Max: \\d+ ns', b'    - Mean: \\d+.\\d+ ns', b'    - Standard deviation: \\d+(\\.\\d+)?(e[\\+\\-]\\d+)?', b'', b'', b'PendulumMotor received \\d+ messages', b'PendulumController received \\d+ messages'])

ros2/demos#126

projectroot.test_showimage_cam2image__rmw_fastrtps_cpp

----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rosbuild/ci_scripts/venv/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/rosbuild/ci_scripts/ws/build/image_tools/test_showimage_cam2image__rmw_fastrtps_cpp_.py", line 32, in test_reliable_qos
    _test_executables(pub_executable_args, sub_executable_args)
  File "/home/rosbuild/ci_scripts/ws/build/image_tools/test_showimage_cam2image__rmw_fastrtps_cpp_.py", line 95, in _test_executables
    handler.check()
  File "/home/rosbuild/ci_scripts/ws/install/launch_testing/lib/python3.5/site-packages/launch_testing/__init__.py", line 106, in check
    (output_lines, self.expected_lines))
launch_testing.UnmatchedOutputError: Example output ([b"Publishing data on topic 'image'", b'\x1b[31;1m[\x1b[37;1mRTPS_ERROR\x1b[31;1m Error] \x1b[37mWriter 1.f.0.2.8e.d2.0.0.0.0.0.0|0.0.2.3 cannot match reader 1.f.0.2.8d.d2.0.0.0.0.0.0|0.0.2.4\x1b[34;1m -> Function \x1b[36mpairingReaderProxy\x1b[m', b'Publishing image #1', b'Publishing image #2', b'signal SIGINT', b'signal_handler(2)']) does not match expected output ([b"Publishing data on topic 'image'", b'Publishing image #1'])
-------------------- >> begin captured stdout << ---------------------
(test_showimage) pid 53901: ['/home/rosbuild/ci_scripts/ws/build/image_tools/showimage', '-r', '1', '-s', '0', '-b'] (InMemoryHandler: test_showimage, all > console)
(test_cam2image) pid 53902: ['/home/rosbuild/ci_scripts/ws/build/image_tools/cam2image', '-r', '1', '-s', '0', '-b', '-f', '5'] (InMemoryHandler: test_cam2image, all > console)
[test_showimage] Subscribing to topic 'image'
[test_cam2image] Publishing data on topic 'image'
[test_cam2image] [NON-XML-CHAR-0x1B][31;1m[[NON-XML-CHAR-0x1B][37;1mRTPS_ERROR[NON-XML-CHAR-0x1B][31;1m Error] [NON-XML-CHAR-0x1B][37mWriter 1.f.0.2.8e.d2.0.0.0.0.0.0|0.0.2.3 cannot match reader 1.f.0.2.8d.d2.0.0.0.0.0.0|0.0.2.4[NON-XML-CHAR-0x1B][34;1m -> Function [NON-XML-CHAR-0x1B][36mpairingReaderProxy[NON-XML-CHAR-0x1B][m
[test_cam2image] Publishing image #1
[test_cam2image] Publishing image #2
[test_showimage] Received image #1
[test_showimage] signal_handler(2)

Update [15 May 2017]: This did not fail when tested 40 times so might have fixed itself http://ci.ros2.org/job/ci_linux-aarch64/180/

Update [5 June 2017]: More details in this ticket: #21

projectroot.test_demo_cyclic_pipeline__rmw_fastrtps_cpp
timeout 30s

packaging_linux{-aarch64} jobs failed dynamic_bridge tests

http://ci.ros2.org/view/packaging/job/packaging_linux/717/
http://ci.ros2.org/view/packaging/job/packaging_linux-aarch64/82/

The relevant output seems to be:

(test_dynamic_bridge_msg_1to2__roscore) pid 36139: ['/opt/ros/kinetic/env.sh', '/opt/ros/kinetic/bin/roscore'] (stderr > stdout, all > console)
(test_dynamic_bridge_msg_1to2__dynamic_bridge) pid 36141: ['/opt/ros/kinetic/env.sh', '/home/rosbuild/ci_scripts/ws/install/bin/dynamic_bridge'] (stderr > stdout, all > console)
(test_dynamic_bridge_msg_1to2__ros1talker) pid 36144: ['/opt/ros/kinetic/env.sh', 'rosrun', 'roscpp_tutorials', 'talker'] (stderr > stdout, all > console)
(test_dynamic_bridge_msg_1to2__ros2listener) Failed to spawn command ['listener']: [Errno 2] No such file or directory: 'listener'

Similarly, it couldn't find talker to launch either.

rtiddsgen sometimes segfaults on nightlies for macOS

I've seen this at least twice now, where rtiddsgen will segfault:

08:00:44 subprocess.CalledProcessError: Command '['/Applications/rti_connext_dds-5.2.3/bin/rtiddsgen_server', '-I', '/Users/osrf/jenkins/workspace/nightly_osx_release/ws/build/nav_msgs/rosidl_generator_dds_idl', '-I', '/Users/osrf/jenkins/workspace/nightly_osx_release/ws/install/builtin_interfaces/share', '-I', '/Users/osrf/jenkins/workspace/nightly_osx_release/ws/install/geometry_msgs/share', '-I', '/Users/osrf/jenkins/workspace/nightly_osx_release/ws/install/std_msgs/share', '-d', '/Users/osrf/jenkins/workspace/nightly_osx_release/ws/build/nav_msgs/rosidl_typesupport_connext_cpp/nav_msgs/srv/dds_connext', '-language', 'C++', '-namespace', '-update', 'typefiles', '-unboundedSupport', '/Users/osrf/jenkins/workspace/nightly_osx_release/ws/build/nav_msgs/rosidl_generator_dds_idl/nav_msgs/srv/dds_connext/GetMap_Request_.idl']' died with <Signals.SIGSEGV: 11>.

I'm not sure why it's happening, perhaps it is running out of memory due to parallel?

http://ci.ros2.org/job/nightly_osx_release/570/consoleFull

Three builds failed with errors cloning Fast-RTPS

packaging_osx, nightly_osx_repeated, and nightly_osx_debug jobs all failed overnight with the same error cloning fast-rtps. I tried cloning locally this morning and then rebuilt the packaging_osx job and both operations succeeded, I've restarted builds of the two longer running failed jobs expecting them to succeed (or fail for other reasons).

08:10:40 === src/eProsima/Fast-RTPS (git) ===
08:10:40 Could not clone repository 'https://github.com/eProsima/Fast-RTPS.git': Cloning into '.'...
08:10:40 error: RPC failed; curl 56 SSLRead() return error -9806
08:10:40 fatal: The remote end hung up unexpectedly
08:10:40 fatal: early EOF
08:10:40 fatal: index-pack failed

I don't think there's anything to be done to mitigate this, but again wanted to start the papertrail in case this becomes recurring.

Build farmer/cop handoff 2017-09-05

I'm basing this off of the previous handoff (#44). As with that one, @sloretz you can close this or leave it open to iterate on, up to you.

Farmer:

Pending tasks:
- update java on ~~OSX~~ and Windows nodes (#43)
- update packages on master and linux2:
  - may fix #30 now that docker 17.05 available on trusty, I updated linux3 and havent seen the docker failure since
- investigate packet.net node failure (#42)
Nice to have: Try VS2017 and assess if we can migrate Windows nodes to it
Recurrent tasks:
- make system updates on MacOS and Windows machines
  - I just updated the macOS version, homebrew packages, and java versions on all the macOS machines
- upgrade eatable to windows 10
- keep chocolatey packages up to date on Windows nodes

Cop:

Outstanding issues:
- rcl_get_node_names became flakier than before on all platforms (example here), mainly tracked here. Still confused of why it started failing much more often recently
- rcl_test_wait / test_timeout became flakier on Windows (has always been flaky but now fail on most CI not only repeated nightlies)
Job status:
- Packaging jobs: green
- Nightlies: green
- Repeated: all yellow, <15 failing tests per night per platform 🎉. Still 58 different tests failed in the last 5 days (script here)

nightly_linux_coverage build failed

The console output looks to be:

+++ Building 'ament_cmake_core'
07:31:30 Running cmake because arguments have changed.
07:31:30 ==> '. /home/rosbuild/ci_scripts/ws/build/ament_cmake_core/cmake__build.sh && /usr/bin/cmake /home/rosbuild/ci_scripts/ws/src/ament/ament_cmake/ament_cmake_core -DBUILD_TESTING=1 -DCMAKE_BUILD_TYPE=Debug -DSECURITY=ON --ament-cmake-args -DCMAKE_CXX_FLAGS=  -fprofile-arcs -ftest-coverage   -DCMAKE_C_FLAGS=  -fprofile-arcs -ftest-coverage  -DCMAKE_INSTALL_PREFIX=/home/rosbuild/ci_scripts/ws/install/ament_cmake_core' in '/home/rosbuild/ci_scripts/ws/build/ament_cmake_core'
07:31:30 CMake Error: The source directory "/home/rosbuild/ci_scripts/ws/build/ament_cmake_core/--ament-cmake-args" does not exist.
07:31:30 Specify --help for usage, or press the help button on the CMake GUI.

Not quite sure what is going on there, looking into it now.

Buildcop handoff 2017-04-18

@nuclearsandwich is passing the build 👮 👩‍🌾 hat to @dhood

Open issues on handoff day

⛏ #3 Image demo not working on eatable (unstarted)
🔧 #6 Need libpcl on MacOS (in progress)
🔧 #7 Need libpcl-dev in Docker containers (in progress)
🐛 ros2/ci#56 Daily invalidation of Dockerfiles isn't working

Notable recent changes

Jenkins agent on MacOS nodes mini1 and mini2 is supervised by launchd.
NTP installed on Linux machines.

Add/change turtlebot demo to build on aarch64

The current turtlebot_demo CI job runs on labels marked as "linux", which in our build farm is implicitly x86_64 machines. However, the real turtlebot demo is almost certainly going to run on aarch64, so we should either change the label on the turtlebot-demo job to be "linux_aarch64", or we should add an additional job to build turtlebot-demo on both architectures. @nuclearsandwich @gerkey does that make sense?

configure eatable as windows 10 machine

see discussion about dropping support for windows 8.1 and focusing on Windows 10

Catastrophic test failures on nightly_linux_*

The nightly linux jobs for 11/12 April had a huge jump in failed tests.

Two things happened yesterday before these jobs ran which may point to the failures:

A PR to rmw_fastrtps designed to work against Fast-RTPS master (but tested briefly on our targeted revision) ros2/rmw_fastrtps#101
A Jenkins upgrade that altered plugin configuration

While arbitrarily sampling tests I came across an out of memory error in test_publisher_subscriber__BoundedArrayPrimitives__rclpy__rmw_connext_cpp__rmw_fastrtps_cpp

Traceback (most recent call last):
  File "/home/rosbuild/ci_scripts/venv/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/rosbuild/ci_scripts/ws/build/test_communication/test_publisher_subscriber__BoundedArrayPrimitives__rclpy__rmw_connext_cpp__rmw_fastrtps_cpp_Debug.py", line 43, in test_publisher_subscriber
    "The launch file failed with exit code '" + str(rc) + "'. " \
AssertionError: The launch file failed with exit code '-6'. May be the subscriber did not receive any messages?
-------------------- >> begin captured stdout << ---------------------
(test_publisher) pid 23105: ['/home/rosbuild/ci_scripts/venv/bin/python3', '/home/rosbuild/ci_scripts/ws/src/ros2/system_tests/test_communication/test/publisher_py.py', 'BoundedArrayPrimitives'] (stderr > stdout, all > console)
(test_subscriber) pid 23106: ['/home/rosbuild/ci_scripts/venv/bin/python3', '/home/rosbuild/ci_scripts/ws/src/ros2/system_tests/test_communication/test/subscriber_py.py', 'BoundedArrayPrimitives'] (stderr > stdout, all > console)
[test_subscriber] terminate called after throwing an instance of 'eprosima::fastcdr::exception::NotEnoughMemoryException'
[test_subscriber]   what():  Not enough memory in the buffer stream
(test_subscriber) rc -6
() tear down

cc @wjwwood who wanted a link to this issue.

nodes fail to delete workspace if previous job was aborted

http://ci.ros2.org/job/ci_linux/3083/console failed after I aborted a job on the same node http://ci.ros2.org/job/ci_linux/3082/

17:29:02 + rm -rf ws workspace work space
17:29:03 rm: cannot remove ‘ws/src/ros2’: Directory not empty
17:29:03 Build step 'Execute shell' marked build as failure

SSH'ing to the directory showed it had many files. I did not check the permissions before deleting the workspace using sudo.

ubuntu@ip-10-234-134-39:/home/jenkins/workspace/ci_linux$ ls -R ws
ws:
src

ws/src:
ros2

ws/src/ros2:
common_interfaces  example_interfaces  poco_vendor  rcl_interfaces  rmw_fastrtps  system_tests

ws/src/ros2/common_interfaces:
actionlib_msgs      CONTRIBUTING.md  geometry_msgs  nav_msgs   sensor_msgs  std_msgs  stereo_msgs      visualization_msgs
builtin_interfaces  diagnostic_msgs  LICENSE        README.md  shape_msgs   std_srvs  trajectory_msgs

ws/src/ros2/common_interfaces/actionlib_msgs:
CMakeLists.txt  msg  package.xml

ws/src/ros2/common_interfaces/actionlib_msgs/msg:
GoalID.msg  GoalStatusArray.msg  GoalStatus.msg

ws/src/ros2/common_interfaces/builtin_interfaces:
CMakeLists.txt  msg  package.xml

ws/src/ros2/common_interfaces/builtin_interfaces/msg:
Duration.msg  Time.msg

ws/src/ros2/common_interfaces/diagnostic_msgs:
CMakeLists.txt  msg  package.xml  srv

ws/src/ros2/common_interfaces/diagnostic_msgs/msg:
DiagnosticArray.msg  DiagnosticStatus.msg  KeyValue.msg

ws/src/ros2/common_interfaces/diagnostic_msgs/srv:
AddDiagnostics.srv  SelfTest.srv

ws/src/ros2/common_interfaces/geometry_msgs:
CMakeLists.txt  msg  package.xml

ws/src/ros2/common_interfaces/geometry_msgs/msg:
Accel.msg                       PointStamped.msg        PoseWithCovarianceStamped.msg  TwistWithCovarianceStamped.msg
AccelStamped.msg                Polygon.msg             Quaternion.msg                 Vector3.msg
AccelWithCovariance.msg         PolygonStamped.msg      QuaternionStamped.msg          Vector3Stamped.msg
AccelWithCovarianceStamped.msg  Pose2D.msg              Transform.msg                  Wrench.msg
Inertia.msg                     PoseArray.msg           TransformStamped.msg           WrenchStamped.msg
InertiaStamped.msg              Pose.msg                Twist.msg
Point32.msg                     PoseStamped.msg         TwistStamped.msg
Point.msg                       PoseWithCovariance.msg  TwistWithCovariance.msg

ws/src/ros2/common_interfaces/nav_msgs:
CMakeLists.txt  msg  package.xml  srv

ws/src/ros2/common_interfaces/nav_msgs/msg:
GridCells.msg  MapMetaData.msg  OccupancyGrid.msg  Odometry.msg  Path.msg

ws/src/ros2/common_interfaces/nav_msgs/srv:
GetMap.srv  GetPlan.srv  SetMap.srv

ws/src/ros2/common_interfaces/sensor_msgs:
CMakeLists.txt  include  msg  package.xml  srv

ws/src/ros2/common_interfaces/sensor_msgs/include:
sensor_msgs

ws/src/ros2/common_interfaces/sensor_msgs/include/sensor_msgs:
distortion_models.hpp  image_encodings.hpp  impl  point_cloud2_iterator.hpp

ws/src/ros2/common_interfaces/sensor_msgs/include/sensor_msgs/impl:
point_cloud2_iterator.hpp

ws/src/ros2/common_interfaces/sensor_msgs/msg:
BatteryState.msg     Illuminance.msg       JoyFeedback.msg    MultiDOFJointState.msg  PointCloud.msg        Temperature.msg
CameraInfo.msg       Image.msg             Joy.msg            MultiEchoLaserScan.msg  PointField.msg        TimeReference.msg
ChannelFloat32.msg   Imu.msg               LaserEcho.msg      NavSatFix.msg           Range.msg
CompressedImage.msg  JointState.msg        LaserScan.msg      NavSatStatus.msg        RegionOfInterest.msg
FluidPressure.msg    JoyFeedbackArray.msg  MagneticField.msg  PointCloud2.msg         RelativeHumidity.msg

ws/src/ros2/common_interfaces/sensor_msgs/srv:
SetCameraInfo.srv

ws/src/ros2/common_interfaces/shape_msgs:
CMakeLists.txt  msg  package.xml

ws/src/ros2/common_interfaces/shape_msgs/msg:
Mesh.msg  MeshTriangle.msg  Plane.msg  SolidPrimitive.msg

ws/src/ros2/common_interfaces/std_msgs:
CMakeLists.txt  msg  package.xml

ws/src/ros2/common_interfaces/std_msgs/msg:
Bool.msg            Empty.msg              Header.msg           Int64.msg                MultiArrayLayout.msg  UInt32MultiArray.msg
Byte.msg            Float32.msg            Int16.msg            Int64MultiArray.msg      String.msg            UInt64.msg
ByteMultiArray.msg  Float32MultiArray.msg  Int16MultiArray.msg  Int8.msg                 UInt16.msg            UInt64MultiArray.msg
Char.msg            Float64.msg            Int32.msg            Int8MultiArray.msg       UInt16MultiArray.msg  UInt8.msg
ColorRGBA.msg       Float64MultiArray.msg  Int32MultiArray.msg  MultiArrayDimension.msg  UInt32.msg            UInt8MultiArray.msg

ws/src/ros2/common_interfaces/std_srvs:
CMakeLists.txt  package.xml  srv

ws/src/ros2/common_interfaces/std_srvs/srv:
Empty.srv  SetBool.srv  Trigger.srv

ws/src/ros2/common_interfaces/stereo_msgs:
CMakeLists.txt  msg  package.xml

ws/src/ros2/common_interfaces/stereo_msgs/msg:
DisparityImage.msg

ws/src/ros2/common_interfaces/trajectory_msgs:
CMakeLists.txt  msg  package.xml

ws/src/ros2/common_interfaces/trajectory_msgs/msg:
JointTrajectory.msg  JointTrajectoryPoint.msg  MultiDOFJointTrajectory.msg  MultiDOFJointTrajectoryPoint.msg

ws/src/ros2/common_interfaces/visualization_msgs:
CMakeLists.txt  msg  package.xml

ws/src/ros2/common_interfaces/visualization_msgs/msg:
ImageMarker.msg               InteractiveMarkerFeedback.msg  InteractiveMarker.msg      InteractiveMarkerUpdate.msg  Marker.msg
InteractiveMarkerControl.msg  InteractiveMarkerInit.msg      InteractiveMarkerPose.msg  MarkerArray.msg              MenuEntry.msg

ws/src/ros2/example_interfaces:
CMakeLists.txt  CONTRIBUTING.md  LICENSE  mapping_rules.yaml  package.xml  README.md  srv

ws/src/ros2/example_interfaces/srv:
AddTwoInts.srv

ws/src/ros2/poco_vendor:
cmake  CMakeLists.txt  LICENSE  poco_vendorConfig.cmake.in  poco_vendorConfig-version.cmake.in  README.md

ws/src/ros2/poco_vendor/cmake:
Modules

ws/src/ros2/poco_vendor/cmake/Modules:
FindPoco.cmake

ws/src/ros2/rcl_interfaces:
CONTRIBUTING.md  LICENSE  lifecycle_msgs  rcl_interfaces  README.md

ws/src/ros2/rcl_interfaces/lifecycle_msgs:
CMakeLists.txt  msg  package.xml  srv

ws/src/ros2/rcl_interfaces/lifecycle_msgs/msg:
State.msg  TransitionDescription.msg  TransitionEvent.msg  Transition.msg

ws/src/ros2/rcl_interfaces/lifecycle_msgs/srv:
ChangeState.srv  GetAvailableStates.srv  GetAvailableTransitions.srv  GetState.srv

ws/src/ros2/rcl_interfaces/rcl_interfaces:
CMakeLists.txt  msg  package.xml  srv

ws/src/ros2/rcl_interfaces/rcl_interfaces/msg:
IntraProcessMessage.msg   ParameterDescriptor.msg        ParameterEvent.msg  ParameterType.msg   SetParametersResult.msg
ListParametersResult.msg  ParameterEventDescriptors.msg  Parameter.msg       ParameterValue.msg

ws/src/ros2/rcl_interfaces/rcl_interfaces/srv:
DescribeParameters.srv  GetParameters.srv  GetParameterTypes.srv  ListParameters.srv  SetParametersAtomically.srv  SetParameters.srv

ws/src/ros2/rmw_fastrtps:
CONTRIBUTING.md  fastrtps_cmake_module  LICENSE  rmw_fastrtps_cpp

ws/src/ros2/rmw_fastrtps/fastrtps_cmake_module:
cmake  CMakeLists.txt  fastrtps_cmake_module-extras.cmake  package.xml

ws/src/ros2/rmw_fastrtps/fastrtps_cmake_module/cmake:
Modules

ws/src/ros2/rmw_fastrtps/fastrtps_cmake_module/cmake/Modules:
FindFastRTPS.cmake

ws/src/ros2/rmw_fastrtps/rmw_fastrtps_cpp:
CMakeLists.txt  include  package.xml  rmw_fastrtps_cpp-extras.cmake  src

ws/src/ros2/rmw_fastrtps/rmw_fastrtps_cpp/include:
rmw_fastrtps_cpp

ws/src/ros2/rmw_fastrtps/rmw_fastrtps_cpp/include/rmw_fastrtps_cpp:
custom_client_info.hpp       get_client.hpp       identifier.hpp               ServiceTypeSupport.hpp       writer_info.hpp
custom_participant_info.hpp  get_participant.hpp  macros.hpp                   ServiceTypeSupport_impl.hpp
custom_publisher_info.hpp    get_publisher.hpp    MessageTypeSupport.hpp       TypeSupport.hpp
custom_service_info.hpp      get_service.hpp      MessageTypeSupport_impl.hpp  TypeSupport_impl.hpp
custom_subscriber_info.hpp   get_subscriber.hpp   reader_info.hpp              visibility_control.h

ws/src/ros2/rmw_fastrtps/rmw_fastrtps_cpp/src:
assign_partitions.hpp      namespace_prefix.cpp                   rmw_node.cpp                         rmw_topic_names_and_types.cpp
client_service_common.cpp  namespace_prefix.hpp                   rmw_node_names.cpp                   rmw_trigger_guard_condition.cpp
client_service_common.hpp  qos.cpp                                rmw_publish.cpp                      rmw_wait.cpp
demangle.cpp               qos.hpp                                rmw_publisher.cpp                    rmw_waitset.cpp
demangle.hpp               rmw_client.cpp                         rmw_request.cpp                      ros_message_serialization.cpp
get_client.cpp             rmw_compare_gids_equal.cpp             rmw_response.cpp                     ros_message_serialization.hpp
get_participant.cpp        rmw_count.cpp                          rmw_service.cpp                      types
get_publisher.cpp          rmw_get_gid_for_publisher.cpp          rmw_service_names_and_types.cpp      type_support_common.cpp
get_service.cpp            rmw_get_implementation_identifier.cpp  rmw_service_server_is_available.cpp  type_support_common.hpp
get_subscriber.cpp         rmw_guard_condition.cpp                rmw_subscription.cpp
identifier.cpp             rmw_init.cpp                           rmw_take.cpp

ws/src/ros2/rmw_fastrtps/rmw_fastrtps_cpp/src/types:
custom_waitset_info.hpp  guard_condition.hpp

ws/src/ros2/system_tests:
CONTRIBUTING.md  LICENSE  test_communication  test_rclcpp

ws/src/ros2/system_tests/test_communication:
CMakeLists.txt  msg  package.xml  srv  test

ws/src/ros2/system_tests/test_communication/msg:
BoundedArrayNested.msg      DynamicArrayNested.msg            Empty.msg               Primitives.msg             UInt32.msg
BoundedArrayPrimitives.msg  DynamicArrayPrimitives.msg        FieldsWithSameType.msg  StaticArrayNested.msg
Builtins.msg                DynamicArrayPrimitivesNested.msg  Nested.msg              StaticArrayPrimitives.msg

ws/src/ros2/system_tests/test_communication/srv:
Empty.srv  Primitives.srv

ws/src/ros2/system_tests/test_communication/test:
__init__.py           service_fixtures.hpp                     test_publisher_subscriber.cpp    test_secure_publisher_subscriber.py.in
message_fixtures.hpp  service_fixtures.py                      test_publisher_subscriber.py.in  test_secure_subscriber.cpp
message_fixtures.py   subscriber_py.py                         test_replier.cpp                 test_security_files
publisher_py.py       test_invalid_secure_node_creation_c.cpp  test_requester.cpp               test_subscriber.cpp
replier_py.py         test_messages_c.cpp                      test_requester_replier.py.in     test_subscription_valid_data.cpp
requester_py.py       test_publisher.cpp                       test_secure_publisher.cpp

ws/src/ros2/system_tests/test_communication/test/test_security_files:
1000.pem  1004.pem     ca.key.pem      index.txt           publisher               serial.old
1001.pem  1005.pem     ecdsaparam      index.txt.attr      publisher_invalid_cert  subscriber
1002.pem  ca.cert.pem  governance.p7s  index.txt.attr.old  publisher_missing_key   test_security_policies.yaml
1003.pem  ca_conf.cnf  governance.xml  index.txt.old       serial

ws/src/ros2/system_tests/test_communication/test/test_security_files/publisher:
ca.cert.pem  cert.pem  ecdsaparam  governance.p7s  key.pem  permissions.p7s  permissions.xml  req.pem  request.cnf

ws/src/ros2/system_tests/test_communication/test/test_security_files/publisher_invalid_cert:
ca.cert.pem  cert.pem  ecdsaparam  governance.p7s  key.pem  permissions.p7s  permissions.xml  req.pem  request.cnf

ws/src/ros2/system_tests/test_communication/test/test_security_files/publisher_missing_key:
ca.cert.pem  cert.pem  ecdsaparam  governance.p7s  permissions.p7s  permissions.xml  req.pem  request.cnf

ws/src/ros2/system_tests/test_communication/test/test_security_files/subscriber:
ca.cert.pem  cert.pem  ecdsaparam  governance.p7s  key.pem  permissions.p7s  permissions.xml  req.pem  request.cnf

ws/src/ros2/system_tests/test_rclcpp:
CMakeLists.txt  include  msg  package.xml  srv  test

ws/src/ros2/system_tests/test_rclcpp/include:
test_rclcpp

ws/src/ros2/system_tests/test_rclcpp/include/test_rclcpp:
utils.hpp

ws/src/ros2/system_tests/test_rclcpp/msg:
UInt32.msg

ws/src/ros2/system_tests/test_rclcpp/srv:
AddTwoInts.srv

ws/src/ros2/system_tests/test_rclcpp/test:
parameter_fixtures.hpp                        test_remote_parameters.cpp
pub_sub_fixtures.hpp                          test_repeated_publisher_subscriber.cpp
test_avoid_ros_namespace_conventions_qos.cpp  test_services_client.cpp
test_client_scope_client.cpp                  test_services_in_constructor.cpp
test_client_scope_consistency_client.cpp      test_services_server.cpp
test_client_scope_consistency_server.cpp      test_signal_handler_after_shutdown__expected_output.txt
test_client_scope_server.cpp                  test_signal_handler_after_shutdown__trigger_shutdown.txt
test_client_wait_for_service_shutdown.cpp     test_signal_handler_before_shutdown__expected_output.txt
test_executable_output.py.in                  test_signal_handler_before_shutdown__trigger_shutdown.txt
test_executor.cpp                             test_signal_handler.cpp
test_intra_process.cpp                        test_spin.cpp
test_local_parameters.cpp                     test_subscription.cpp
test_multiple_service_calls.cpp               test_timeout_subscriber.cpp
test_multithreaded.cpp                        test_timer.cpp
test_parameters_server.cpp                    test_two_executables.py.in
test_publisher.cpp

Talker/Listener output test failed despite having correct output

Possibly a bug belonging on https://github.com/ros2/launch . Making ticket here because I can't investigate it fully right now.

Job

http://ci.ros2.org/view/nightly/job/nightly_linux_repeated/772/
linux_slave_on_master

Test

projectroot.test_tutorial_talker_listener__rmw_fastrtps_cpp
demo_nodes_cpp.test_tutorial_talker_listener__rmw_fastrtps_cpp.xunit.missing_result

The tests use launch_testing to check that the output matches expectations.

Listener regex: I heard: \[Hello World: \d+\]
Talker text: Publishing: 'Hello World: 1'

Test output

      Start 16: test_tutorial_talker_listener__rmw_fastrtps_cpp

16: Test command: /home/rosbuild/ci_scripts/venv/bin/python3 "-u" "/home/rosbuild/ci_scripts/ws/install/ament_cmake_test/share/ament_cmake_test/cmake/run_test.py" "/home/rosbuild/ci_scripts/ws/build/demo_nodes_cpp/test_results/demo_nodes_cpp/test_tutorial_talker_listener__rmw_fastrtps_cpp.xunit.xml" "--output-file" "/home/rosbuild/ci_scripts/ws/build/demo_nodes_cpp/ament_cmake_nose/test_tutorial_talker_listener__rmw_fastrtps_cpp.txt" "--env" "RCL_ASSERT_RMW_ID_MATCHES=rmw_fastrtps_cpp" "RMW_IMPLEMENTATION=rmw_fastrtps_cpp" "--command" "/home/rosbuild/ci_scripts/venv/bin/python3" "-u" "/usr/bin/nosetests3" "/home/rosbuild/ci_scripts/ws/build/demo_nodes_cpp/test_talker_listener__rmw_fastrtps_cpp_.py" "--nocapture" "--with-xunit" "--xunit-file=/home/rosbuild/ci_scripts/ws/build/demo_nodes_cpp/test_results/demo_nodes_cpp/test_tutorial_talker_listener__rmw_fastrtps_cpp.xunit.xml" "--xunit-testsuite-name=demo_nodes_cpp.nosetests"
16: Test timeout computed to be: 30
16: -- run_test.py: extra environment variables:
16:  - RCL_ASSERT_RMW_ID_MATCHES=rmw_fastrtps_cpp
16:  - RMW_IMPLEMENTATION=rmw_fastrtps_cpp
16: -- run_test.py: invoking following command in '/home/rosbuild/ci_scripts/ws/src/ros2/demos/demo_nodes_cpp':
16:  - /home/rosbuild/ci_scripts/venv/bin/python3 -u /usr/bin/nosetests3 /home/rosbuild/ci_scripts/ws/build/demo_nodes_cpp/test_talker_listener__rmw_fastrtps_cpp_.py --nocapture --with-xunit --xunit-file=/home/rosbuild/ci_scripts/ws/build/demo_nodes_cpp/test_results/demo_nodes_cpp/test_tutorial_talker_listener__rmw_fastrtps_cpp.xunit.xml --xunit-testsuite-name=demo_nodes_cpp.nosetests
16: (test_executable_0) pid 2824: ['/home/rosbuild/ci_scripts/ws/build/demo_nodes_cpp/talker'] (all > console, InMemoryHandler: test_executable_0)
16: (test_executable_1) pid 2825: ['/home/rosbuild/ci_scripts/ws/build/demo_nodes_cpp/listener'] (all > console, InMemoryHandler: test_executable_1)
16: [test_executable_0] Publishing: 'Hello World: 1'
16: [test_executable_0] Publishing: 'Hello World: 2'
16: [test_executable_0] Publishing: 'Hello World: 3'
16: [test_executable_0] Publishing: 'Hello World: 4'
16: [test_executable_0] Publishing: 'Hello World: 5'
16: [test_executable_0] Publishing: 'Hello World: 6'
16: [test_executable_0] Publishing: 'Hello World: 7'
16: [test_executable_1] I heard: [Hello World: 2]
16: [test_executable_1] I heard: [Hello World: 3]
16: [test_executable_1] I heard: [Hello World: 4]
16: [test_executable_1] I heard: [Hello World: 5]
16: [test_executable_1] I heard: [Hello World: 6]
16: [test_executable_1] I heard: [Hello World: 7]
16: [test_executable_0] Publishing: 'Hello World: 8'
16: [test_executable_0] Publishing: 'Hello World: 9'
16: [test_executable_0] Publishing: 'Hello World: 10'
16: [test_executable_0] Publishing: 'Hello World: 11'
16: [test_executable_0] Publishing: 'Hello World: 12'
16: [test_executable_0] Publishing: 'Hello World: 13'
16: [test_executable_0] Publishing: 'Hello World: 14'
16: [test_executable_0] Publishing: 'Hello World: 15'
16: [test_executable_0] Publishing: 'Hello World: 16'
16: [test_executable_0] Publishing: 'Hello World: 17'
16: [test_executable_0] Publishing: 'Hello World: 18'
16: [test_executable_0] Publishing: 'Hello World: 19'
16: [test_executable_0] Publishing: 'Hello World: 20'
16: [test_executable_0] Publishing: 'Hello World: 21'
16: [test_executable_0] Publishing: 'Hello World: 22'
16: [test_executable_0] Publishing: 'Hello World: 23'
16: [test_executable_0] Publishing: 'Hello World: 24'
16: [test_executable_0] Publishing: 'Hello World: 25'
16: [test_executable_0] Publishing: 'Hello World: 26'
16: [test_executable_0] Publishing: 'Hello World: 27'
16: [test_executable_0] Publishing: 'Hello World: 28'
16: [test_executable_0] Publishing: 'Hello World: 29'
16: [test_executable_0] Publishing: 'Hello World: 30'
16: [test_executable_0] Publishing: 'Hello World: 31'
16: [test_executable_0] Publishing: 'Hello World: 32'
16: [test_executable_0] Publishing: 'Hello World: 33'
16: [test_executable_0] Publishing: 'Hello World: 34'
16: [test_executable_0] Publishing: 'Hello World: 35'
16: [test_executable_0] Publishing: 'Hello World: 36'
16: [test_executable_0] Publishing: 'Hello World: 37'
16: [test_executable_0] Publishing: 'Hello World: 38'
16: [test_executable_0] Publishing: 'Hello World: 39'
16: [test_executable_0] Publishing: 'Hello World: 40'
16: [test_executable_0] Publishing: 'Hello World: 41'
16: [test_executable_0] Publishing: 'Hello World: 42'
16: [test_executable_0] Publishing: 'Hello World: 43'
16: [test_executable_0] Publishing: 'Hello World: 44'
16: [test_executable_0] Publishing: 'Hello World: 45'
16: [test_executable_0] Publishing: 'Hello World: 46'
16: [test_executable_0] Publishing: 'Hello World: 47'
16: [test_executable_0] Publishing: 'Hello World: 48'
16: [test_executable_0] Publishing: 'Hello World: 49'
16: [test_executable_0] Publishing: 'Hello World: 50'
16: [test_executable_0] Publishing: 'Hello World: 51'
16: [test_executable_0] Publishing: 'Hello World: 52'
16: [test_executable_0] Publishing: 'Hello World: 53'
16: [test_executable_0] Publishing: 'Hello World: 54'
16: [test_executable_0] Publishing: 'Hello World: 55'
16: [test_executable_0] Publishing: 'Hello World: 56'
16: [test_executable_0] Publishing: 'Hello World: 57'
16: [test_executable_0] Publishing: 'Hello World: 58'
16: [test_executable_0] Publishing: 'Hello World: 59'
16/18 Test #16: test_tutorial_talker_listener__rmw_fastrtps_cpp .................................***Timeout  30.01 sec

The test output shows the subscriber only receives messages for a short period of time, however the test does have the required output in the console. I would expect the test to have been marked as passed.

new test failures for osx release nightlies and windows release nightlies

The tests that are failing are all test communication tests between rclpy and rclcpp, some facts about them:

affects only macOS and Windows release and repeated nightlies
- Windows Debug is still failing to buld
- macOS debug is green
affects both fastrtps and connext
started June 9th, June 8th is clean
- search for merged prs: https://github.com/search?p=1&q=user%3Aament+user%3Aros2+merged%3A%222017-06-08T22%3A00%3A00-08%3A00+..+2017-06-12T23%3A30%3A00-08%3A00%22&type=Issues&utf8=✓
fails due to timeout (15 sec for most), so probably hanging

Things I have tried:

macOS CI passes
running the ci scripts on my macbook with same settings as macOS release nightly passes too

Right now I'm trying to get a new run of the job to fail, then I'll take that machine off line while the workspace is in place and rerun on that machine. Other ideas are welcome.

rtiddsgen sometimes crashes with IO Error when building in parallel on linux

ERROR com.rti.ndds.nddsgen.Main I/O error generating code
java.net.BindException: Address already in use
	at java.net.PlainSocketImpl.socketBind(Native Method)
	at java.net.AbstractPlainSocketImpl.bind(Unknown Source)
	at java.net.ServerSocket.bind(Unknown Source)
	at java.net.ServerSocket.<init>(Unknown Source)
	at java.net.ServerSocket.<init>(Unknown Source)
	at com.rti.ndds.nddsgen.CodeGeneratorServer.<init>(Unknown Source)
	at com.rti.ndds.nddsgen.Main.main(Unknown Source)

linux_slave_on_master will not run jobs

I am now getting:

containerd: container did not start before the specified timeout

-- http://ci.ros2.org/job/ci_linux/2722/console

It has happened twice in a row, so I'm guessing it borked until we figure out why. I'm just disabling it for now. I'll look into it as soon as I can.

Empty xunit output for test that timed out

A test didn't generate its xunit file correctly on timeout. It looks like it generated an empty file.

http://ci.ros2.org/view/nightly/job/nightly_linux_repeated/628/

# BEGIN SUBSECTION: ament test_results
11:25:28 (RTI)
11:25:28 (venv)
11:25:28 ==> . "/home/rosbuild/rti_connext_dds-5.2.3/resource/scripts/rtisetenv_x64Linux3gcc4.8.2.bash" && "/home/rosbuild/ci_scripts/venv/bin/python" -u "src/ament/ament_tools/scripts/ament.py" test_results "build"
11:25:28 ###############################################################################
11:25:28 
11:25:28          (c) Copyright, Real-Time Innovations, All rights reserved.            
11:25:28                                                                                
11:25:28                            RTI Connext DDS 5.2.3                       
11:25:28 
11:25:28 ###############################################################################
11:25:29 Skipping 'test_communication/test_results/test_communication/test_publisher_subscriber__StaticArrayNested__rclcpp__rmw_fastrtps_cpp__rmw_connext_cpp.xunit.xml': File "<string>", line None, xml.etree.ElementTree.ParseError: no element found: line 1, column 0
11:25:29 composition/test_results/composition/test_composition__rmw_fastrtps_cpp.xunit.xml: 1 tests, 0 skipped, 0 errors, 1 failures
11:25:29 demo_nodes_cpp/test_results/demo_nodes_cpp/test_tutorial_add_two_ints_server_add_two_ints_client__rmw_fastrtps_cpp.xunit.xml: 1 tests, 0 skipped, 0 errors, 1 failures
11:25:29 intra_process_demo/test_results/intra_process_demo/test_demo_cyclic_pipeline__rmw_fastrtps_cpp.xunit.xml: 1 tests, 0 skipped, 0 errors, 1 failures
11:25:29 pendulum_control/test_results/pendulum_control/test_pendulum__rmw_connext_cpp.xunit.xml: 1 tests, 0 skipped, 1 errors, 0 failures
11:25:29 rcl/test_results/rcl/test_services__rmw_fastrtps_cpp.xunit.xml: 1 tests, 0 skipped, 0 errors, 1 failures
11:25:29 rclcpp_lifecycle/test_results/rclcpp_lifecycle/test_state_machine_info.gtest.xml: 2 tests, 0 skipped, 0 errors, 1 failures
11:25:29 test_rclcpp/test_results/test_rclcpp/test_client_scope_consistency_cpp__rmw_fastrtps_cpp.xunit.xml: 1 tests, 0 skipped, 0 errors, 1 failures
11:25:29 test_rclcpp/test_results/test_rclcpp/test_services_cpp__rmw_fastrtps_cpp.xunit.xml: 1 tests, 0 skipped, 0 errors, 1 failures
11:25:29 Summary: 754 tests, 1 errors, 7 failures, 26 skipped
11:25:29 
11:25:29 II> ament.py test_results returned: '1'
11:25:29 # END SUBSECTION
11:25:29 + echo # END SECTION
11:25:29 # END SECTION
11:25:29 $ ssh-agent -k
11:25:29 unset SSH_AUTH_SOCK;
11:25:29 unset SSH_AGENT_PID;
11:25:29 echo Agent pid 9361 killed;
11:25:29 [ssh-agent] Stopped.
11:25:29 [WARNINGS] Parsing warnings in console log with parser CMake
11:25:30 [WARNINGS] Found 0 warnings after exclusion.
11:25:30 [WARNINGS] Parsing warnings in console log with parser GNU C Compiler 4 (gcc)
11:25:31 [WARNINGS] Found 0 warnings after exclusion.
11:25:31 [WARNINGS] Computing warning deltas based on reference build #627
11:25:31 [WARNINGS] Ignore new warnings since this is the first valid build
11:25:31 [WARNINGS] Plug-in Result: Success - no threshold has been exceeded
11:25:31 [Cobertura] Publishing Cobertura coverage report...
11:25:32 Publishing Cobertura coverage results...
11:25:32 Cobertura coverage report found.
11:25:32 [xUnit] [INFO] - Starting to record.
11:25:32 [xUnit] [INFO] - Processing CTest-Version N/A (default)
11:25:33 [xUnit] [INFO] - [CTest-Version N/A (default)] - 86 test report file(s) were found with the pattern 'ws/build/*/Testing/*/Test.xml' relative to '/home/jenkins/workspace/nightly_linux_repeated' for the testing framework 'CTest-Version N/A (default)'.
11:25:33 [xUnit] [INFO] - Processing GoogleTest-1.6
11:25:33 [xUnit] [INFO] - [GoogleTest-1.6] - 88 test report file(s) were found with the pattern 'ws/build/*/test_results/**/*.gtest.xml' relative to '/home/jenkins/workspace/nightly_linux_repeated' for the testing framework 'GoogleTest-1.6'.
11:25:34 [xUnit] [INFO] - Processing JUnit
11:25:34 [xUnit] [INFO] - [JUnit] - 258 test report file(s) were found with the pattern 'ws/build/*/test_results/**/*.xunit.xml' relative to '/home/jenkins/workspace/nightly_linux_repeated' for the testing framework 'JUnit'.
11:25:34 [xUnit] [ERROR] - The result file '/home/jenkins/workspace/nightly_linux_repeated/ws/build/test_communication/test_results/test_communication/test_publisher_subscriber__StaticArrayNested__rclcpp__rmw_fastrtps_cpp__rmw_connext_cpp.xunit.xml' for the metric 'JUnit' is empty. The result file has been skipped.
11:25:34 [xUnit] [INFO] - Failing BUILD because 'set build failed if errors' option is activated.
11:25:34 [xUnit] [INFO] - There are errors when processing test results.
11:25:34 [xUnit] [INFO] - Skipping tests recording.
11:25:34 [xUnit] [INFO] - Stop build.
11:25:34 Build step 'Publish xUnit test result report' changed build result to FAILURE
11:25:34 Sending e-mails to: [email protected]
11:25:35 Finished: FAILURE

The test was supposed to timeout after 15s but it took closer to 30 total (might have still been killed after 15s but took a while to die?)

11:08:05 165: Test command: /home/rosbuild/ci_scripts/venv/bin/python3 "-u" "/home/rosbuild/ci_scripts/ws/install/ament_cmake_test/share/ament_cmake_test/cmake/run_test.py" "/home/rosbuild/ci_scripts/ws/build/test_communication/test_results/test_communication/test_publisher_subscriber__StaticArrayNested__rclcpp__rmw_fastrtps_cpp__rmw_connext_cpp.xunit.xml" "--output-file" "/home/rosbuild/ci_scripts/ws/build/test_communication/ament_cmake_nose/test_publisher_subscriber__StaticArrayNested__rclcpp__rmw_fastrtps_cpp__rmw_connext_cpp.txt" "--append-env" "PYTHONPATH="/home/rosbuild/ci_scripts/ws/build/test_communication/rosidl_generator_py:/home/rosbuild/ci_scripts/ws/src/ros2/system_tests/test_communication:/home/rosbuild/ci_scripts/ws/build/test_communication/../../rclpy"" "LD_LIBRARY_PATH=/home/rosbuild/ci_scripts/ws/build/test_communication" "--command" "/home/rosbuild/ci_scripts/venv/bin/python3" "/usr/bin/nosetests3" "/home/rosbuild/ci_scripts/ws/build/test_communication/test_publisher_subscriber__StaticArrayNested__rclcpp__rmw_fastrtps_cpp__rmw_connext_cpp_.py" "--with-xunit" "--xunit-file=/home/rosbuild/ci_scripts/ws/build/test_communication/test_results/test_communication/test_publisher_subscriber__StaticArrayNested__rclcpp__rmw_fastrtps_cpp__rmw_connext_cpp.xunit.xml" "--xunit-testsuite-name=test_communication.nosetests"
11:08:05 165: Test timeout computed to be: 15
11:08:05 165: -- run_test.py: extra environment variables to append:
11:08:05 165:  - PYTHONPATH="/home/rosbuild/ci_scripts/ws/build/test_communication/rosidl_generator_py:/home/rosbuild/ci_scripts/ws/src/ros2/system_tests/test_communication:/home/rosbuild/ci_scripts/ws/build/test_communication/../../rclpy"
11:08:05 165:  - LD_LIBRARY_PATH=/home/rosbuild/ci_scripts/ws/build/test_communication
11:08:05 165: -- run_test.py: invoking following command in '/home/rosbuild/ci_scripts/ws/build/test_communication/rosidl_generator_py':
11:08:05 165:  - /home/rosbuild/ci_scripts/venv/bin/python3 /usr/bin/nosetests3 /home/rosbuild/ci_scripts/ws/build/test_communication/test_publisher_subscriber__StaticArrayNested__rclcpp__rmw_fastrtps_cpp__rmw_connext_cpp_.py --with-xunit --xunit-file=/home/rosbuild/ci_scripts/ws/build/test_communication/test_results/test_communication/test_publisher_subscriber__StaticArrayNested__rclcpp__rmw_fastrtps_cpp__rmw_connext_cpp.xunit.xml --xunit-testsuite-name=test_communication.nosetests
11:08:34 165/216 Test #165: test_publisher_subscriber__StaticArrayNested__rclcpp__rmw_fastrtps_cpp__rmw_connext_cpp ...............***Timeout  28.75 sec
11:08:34 test 166
11:08:34         Start 166: test_publisher_subscriber__StaticArrayPrimitives__rclcpp__rmw_fastrtps_cpp__rmw_connext_cpp

It's not usual for this type of test to timeout. If anything the tests from that package fail when they take too long e.g. http://ci.ros2.org/view/nightly/job/nightly_linux_repeated/587/testReport/junit/(root)/test_publisher_subscriber__Nested__rclcpp__rmw_connext_cpp__rmw_fastrtps_cpp_/test_publisher_subscriber/

It's especially unusual for an empty xunit file to be generated. This is the first time I've seen a build fail because of an empty file.

image demo not working on eatable

Since eatable has been turned back on it is failing the image demo tests.
@clalancette to give context about what has been done on the node and its reactivation

CI failing tests:
http://ci.ros2.org/job/ci_windows/2534/
http://ci.ros2.org/job/ci_windows/2555/
list of builds here: http://ci.ros2.org/computer/windows_slave_eatable/builds

Need to install libpcl development libraries on OSX from homebrew

ros2/pcl_conversions#1 needs the libpcl development libraries installed to work, so we need to do this on the OSX build farm machines.

Build farmer/cop handoff 2017-08-15

Farmer:

Pending tasks:
- update java on OSX and Windows nodes (#43)
- update packages on master and linux2:
  - may fix #30 now that docker 17.05 available on trusty, I updated linux3 and havent seen the docker failure since
- investigate packet.net node failure (#42)
Nice to have: Try VS2017 and assess if we can migrate Windows nodes to it
Recurrent tasks:
- make system updates on MacOS and Windows machines
- upgrade eatable to windows 10
- keep chocolatey packages up to date on Windows nodes

Cop:

Outstanding issues:
- New failing test: time operators (tracked here)
- rcl_get_node_names became flakier than before on all platforms (example here), mainly tracked here. Still confused of why it started failing much more often recently
- rcl_test_wait / test_timeout became flakier on Windows (has always been flaky but now fail on most CI not only repeated nightlies)
Job status:
- Packaging jobs: green
- Nightlies: green except time operators.
- Repeated: all yellow, <15 failing tests per night per platform 🎉. Still 58 different tests failed in the last 5 days (script here)

Error generating repository hashes causing most jobs to fail

Not looking into this just yet but I definitely got the emails and am going to when I have the time.

OSX nightly debug failed cloning from GitHub on mini1 and mini2

This job: http://ci.ros2.org/view/nightly/job/nightly_osx_debug/435/

failed to clone from github with an error:

07:39:33 error: RPC failed; curl 56 SSLRead() return error -9806

We've seen this from time-to-time, and it usually goes away. Still, we've seen it enough that I think we should probably look into it. I think @dhood had done some research before, but I thought I would re-report it here.