See http://build.ros2.org/view/Fci/job/Fci__nightly-performance_ubuntu_focal_amd64/66/gcc/

Currently the plots can be quite confusing to read as the colors are often close enough to not be able to distinguish at a quick glance. But even harder is that between two plots in the same result the color selection for the different implementations are inconsistent between side by side plots.

For example at: http://build.ros2.org/view/Eci/job/Eci__nightly-performance_ubuntu_bionic_amd64/plot/

The rmw_connext is yellow for the first 2 and last 2 graphs, but for the middle 2 FastRTPS is yellow.

Implementing a consistent coloring would be really helpful for quick reading of the graphs. They all have the same 7 items in the legend.

• Static load - create one process and spin for 1 minute
  • x86_64/EC2

• Simple pub/sub - create two processes, the publisher publishes a 1kArray every 200ms, for 30 seconds (150 messages in total)
  • x86_64/EC2

Metrics

• CPU usage per-process collected every 1 second
• Virtual Memory usage per-process collected every 1 second
• Physical Memory usage per-process collected every 1 second

Per-Test Data

• [Graph] metrics variation over time for 1 test run.
  • X = time, Y = metric value
• [Text] Test summary with per-process aggregated values: mean/median/P95
  • Text report / log output
  • *Metrics should not be aggregated between processes.

Over-Time Data

• [Graph] Variation of the metrics commit-after-commit
• X = commit ID, Y = aggregated metric value (mean/median/P95)
• One graph per process.
• Each value on X axis means a “run” of the job on a particular commit. Not all commits need to be
  evaluated.

E.g. sync for Array4M in http://build.ros2.org/view/Rci/job/Rci__nightly-performance_ubuntu_focal_amd64/96/

FastRTPS:

• Latency ~14ms:
  

• Throughput ~875Mbits/s:
  

rmw_fastrtps_cpp:

• Latency ~3.5ms:
  

• Throughput ~3650Mbits/s:
  

As commented in this issue #21 (comment) there are some test failing

Dashing

It is not working yet.

• Waiting for this PR ros2/launch#393

Eloquent

• This build has been failing the last 4 times: http://build.ros2.org/view/Eci/job/Eci__nightly-performance_ubuntu_bionic_amd64/177/testReport/
  • there is a known issue with sync mode and big messages. This PR skips some tests #32

Foxy

The latest Foxy build shows numerous test failures: http://build.ros2.org/view/Fci/job/Fci__nightly-performance_ubuntu_focal_amd64/10/testReport/

refactor the calls to get_filename_component, configure_file, add_launch_test, etc... into a new function.

Should allow data aggregation and presentation by other processes.

Inspired by #26 (comment)

The performance testing jobs that were running on build.ros2.org have temporarily been made private while the test suite is improved to provide more informative results and avoid potential confusion or premature conclusions from the preliminary results. The test suite remains available for anyone to use.

The jobs will be made public again after the following issues have been addressed:

• add additional performance tests covering more usage scenarios

  • for example: varying the message sizes, publish frequency, and number of entities involved

• improve the visibility of context surrounding the test results

  • make sure this information is prominently displayed next to graphs
  • include more details about each scenario, e.g., how data were gathered, what QoS settings were used, other important factors in the test

• resolve known discrepancies between rmw implementations

  • async vs sync publishing
    • see: ros2/rmw_cyclonedds#89
    • see: ros2/rmw_fastrtps#343

The plots shown by the image gallery plugin on a single build (e.g. http://build.ros2.org/view/Rci/job/Rci__nightly-performance_ubuntu_focal_amd64/96/) are hard to navigation since there are 888 images without an index or anything else. All text is embedded in the images and therefore can't be used to search / used to find specific images.

Please add CycloneDDS to the following tests:

• Performance Two Processes Test Results (Array1k)
• Performance Two Processes Test Results (multisize messages)

CycloneDDS is already included in the following tests:

• Performance One Process Test Results (Array1k)
• Performance One Process Test Results (multisize messages)

rmw_cyclonedds is already in all tests.
@cottsay @dirk-thomas @eboasson @k0ekk0ek
http://build.ros2.org/job/Fci__nightly-performance_ubuntu_focal_amd64/plot/

The following message is shown in CI result.
https://build.ros2.org/view/Rci/job/Rci__nightly-performance_ubuntu_focal_amd64/lastBuild/console

07:00:37 288: Warning: Passing ready_fn as an argument to generate_test_description will be removed in a future release.  Include a launch_testing.actions.ReadyToTest action in the LaunchDescription instead.

This PR looks related.
ros2/launch#346

As discussed here #14 we should refactor the calls to get_filename_component, configure_file, add_launch_test, etc... into a new function.

Hi @ahcorde @cottsay,

While comparing the ROS 2 performance test tools here there were some questions about different approaches used by each tool to collect the system resources.

It seems that buildfarm_perf_tests use system_metric_collector to collect the system data (#12), while performance_test uses getrusage (see analysis_result.cpp). I think it makes sense to re-use an existing tool such as system_metric_collector and join efforts to add new features there. First, I would like understand what are the pros and cons of each approach. Hopefully, this could be a step forward to reduce divergences between all the exiting tools.

cc @fadi-labib @SniderThanYou

I compared several benchmark programs before (ros-realtime/community#6) to discuss real-time tests. Additionally, I hope we can use this repository to compare the performances between several machines (for example, to answer what machine we should use).

Now, I am trying to build&run buildfarm_perf_test on my Raspberry Pi4 and found some lacks in README.md.
If my memory is correct, they happened when I tried to build&run before.

OS: Ubuntu 20.04-raspi
ROS2: build, master
buildfarm_perf_test: build, master(bc4e002)

1. build failure
   1. boost not found
   2. -D option error: PERF_TEST_MAX_RUNTIME or PERF_TEST_TOPIC. Their name has been chaned at 5d88c11, for example.
2. test failue
   1. pandas and matplotlib not found althogh I set up my RPi4 with https://index.ros.org/doc/ros2/Installation/Rolling/Linux-Development-Setup/ or README.md. (launch_test.py: error: No module named 'matplotlib' for example)
   2. colcon test -D option at README.md (colcon test: error: argument --packages-select: unrecognized argument: -DPERF_TEST_MAX_RUNTIME=30 for example)

I'm going to send PR after this.

Thank you.

The functions _raw_to_csv and _raw_to_png are called directly from the test which prevent the user to e.g. run the test manually (a publisher and subscriber on two machines) and then call a script to generate the graphs.