self.atf.set_user_result(AtfResult())

make recording fail with exception/timeout message when timeout is exceeded...

needed for:

• wait_for_action
• wait_for_topic
• wait_for_service
• wait_for_diagnostics

because the cmake macro tries to run generate_tests.py from @(CMAKE_SOURCE_DIR)/scripts/generate_tests.py

  Errors     << mojin_atf_kevin:cmake /root/target_ws/logs/mojin_atf_kevin/build.cmake.000.log
  ATF: executing test generation macro with 'atf/test_generation_config.yaml' and EXECUTE_TESTS=True
  /usr/bin/python2: can't open file '/root/git/app_ws/src/atf/atf_core/scripts/generate_tests.py': [Errno 2] No such file or directory
  CMake Error at /root/git/app_ws/install/share/atf_core/cmake/atf_core-extras.cmake:39 (message):
    -- ATF: generating test files failed: exit_code='2'
  Call Stack (most recent call first):
    CMakeLists.txt:27 (atf_test)

I got this error:
/home/lvb/git/care-o-bot/src/atf/atf_core/src/atf_core/configuration_parser.py:181: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. doc = yaml.load(stream)

which points to https://msg.pyyaml.org/load

in atf_test some jerk values deviate from the others. Problems seems more relevant if static transforms are involved.

e.g. for

• path position/velocity/acceleration/jerks profiles
• joint position/velocity/acceleration/jerk profiles
• battery profile
• ...

currently the result data is limited to a single float (https://github.com/floweisshardt/atf/blob/master/atf_msgs/msg/MetricResult.msg#L4).
This could be used for plotting a time series this would need to be

recording and analyzing

• (prefered) add an optional time series field to all span metrics (e.g. tf_length_translation) additionally to the single float. data type needs to be a list of stamped floats
• frequency needs to be configurable in testblockset.yaml
  • none
  • either on each update step
  • fixed rate (trigger mechanism needed! interpolation needed?)
• can be expanded with fields for
  • min/max
  • avg/std_dev

visualization

• plot time series

when running multiple testsuite in the same package (e.g. with multiple CMake macros set to TRUE during catkin run_tests) there are name collission for

• recorded bag files ts*
• test results txt files ts*
• test results bag files ts*
• overall test results atf_result.txt, atf_result.bag

Solution:

• apply date/time suffix
• symlink latest recordings and test results to latest (for easy plotter terminal commands)

discovered by @HannesBachter

this can be reproduzed by calling self.atf.stop(TESTBLOCK) without calling self.atf.start(TESTBLOCK) before.

rosrun atf_core analyser.py mojin_atf_mbl

analysing for package mojin_atf_mbl
[ROSUNIT] Outputting test results to /home/hyb/.ros/test_results/atf_core/rosunit-analysing.xml
0 errors detected during test processing
Processing tests took 8.9079 sec
ATF analyser: done!
An error occured during analysis of testblock 'testblock_shelf', no useful results available.
[Testcase: test_analysing] ... ERROR!
ERROR: global name 'ATFAnalyserError' is not defined
   File "/usr/lib/python2.7/unittest/case.py", line 329, in run
     testMethod()
   File "/home/hyb/git/care-o-bot/src/atf/atf_core/scripts/analyser.py", line 190, in test_analysing
     atf_result = analyser.get_result()
   File "/home/hyb/git/care-o-bot/src/atf/atf_core/scripts/analyser.py", line 122, in get_result
     test_result = test.get_result()
   File "/home/hyb/git/care-o-bot/src/atf/atf_core/src/atf_core/test.py", line 34, in get_result
     testblock_result = testblock.get_result()
   File "/home/hyb/git/care-o-bot/src/atf/atf_core/src/atf_core/testblock.py", line 45, in get_result
     raise ATFAnalyserError("Analysing failed, no testblock result available for testblock '%s'."%testblock_result.name)
--------------------------------------------------------------------------------
-------------------------------------------------------------
SUMMARY:
  * RESULT: FAIL
  * TESTS: 1
  * ERRORS: 1 [test_analysing]
  * FAILURES: 0 []

currently disabled for atf_test. see #153 (comment).

Fails tf and publish_rate metrics if system load is to high (?!)

minimum is to have

• 1x testblocksets

optional

• envs
• robots
• tests

this is a bug :)

atf sm blockes forever if one testblock is not stopped when calling atf.shutdown()

also, I need to call atf.stop(testblock) for testblocks that have not even started (no atf.start(testblock) called before)

The installation instructions under the step labeled Get ATF sources from github instruct to wstool merge https://raw.githubusercontent.com/ipa-fmw/atf/master/.travis.rosinstall into a catkin workspace to get the sources.

The referenced file only contains a stanza for ros_comm, which seems to not have any relation to this project:

- git:
    uri: 'https://github.com/ros/ros_comm.git'
    local-name: ros_comm
    version: indigo-devel

I just git cloned the repository to my workspace, which seems to work as well.

The metrics related to TF (distance, jerk, velocity, length) have a lot of code duplication between them.

In eg. CalculateTfAccelerationTranslation, vs CalculateTfDistanceTranslation only a few members are added and a few lines in get_data are different but mostly the same. Some basic inheritance can reduce a lot of duplication and reduce maintenance.

Might be related to #58 as well.

The parameter robot_bringup_launch to define your specific robot-dependent launchfile is not actually used by atf

for a large number of similar testblocks (>10-100) it's not very comfortable to configure every testblock individually in the testblockset.yaml files. Instead it would be nice to configure metrics for a master testblock and then use the same configuration in all other testblocks.

Questions:

• how to handle different groundtruth settings?

in our scenarios, we want the metrics to check the tf distances at a certain time, e.g. when putting a box to the internal storage
analyzing this distance as the minimum over the whole recording period is unreliable as we might move closer to that goal pose during some other phases of the scenario

the minimum approach only works for groundtruth.data: 0.0 - but when there is no tf frame at the goal pose this is not the case
#179 might also be a solution for the problem

During tests in https://github.com/mojin-robotics/mojin_atf/pull/3 sometimes the distance was small (as expected), but sometimes larger than the epsilon. Investigating the bag-file, I realized that the atf/status topic was published shortly after the last movement - so maybe there is an issue with getting the latest tf transform

The current TF-based metrics use the distance between origins of TF frames. But to check whether a robot eg. passed through some waypoint etc that doesn't have a TF frame, calculating the distance between two poses would be very handy.

like client.wait_for_server()

obviously there is a (small) difference in just listening to goal/status topic and waiting for client.wait_for_server() discovered by @HannesBachter

analog to #117

covering dummies for

• start
• stop
• pause
• update
• get result
  ...

if the recording gets aborted (equvivalent to CTRL-C)

max time [360.0s] allotted for test [atf_recording_ts0_c0_r0_e0_s0_0] of type [atf_core/sm_test.py]

we get somthing like

 - test 'ts0_c0_r0_e0_s0_0' (cob4-2, ipa-apartment, test_default, testblockset_ipa-apartment): 
   - testblock 'testblock_forward': 
     - metric 'interface': no result
     - metric 'tf_length_translation': no result
     - metric 'tf_length_translation': no result
     - metric 'user_result': no result
     - metric 'time': no result
   - testblock 'testblock_all': 
     - metric 'interface': no result
     - metric 'tf_length_translation': no result
     - metric 'tf_length_translation': no result
     - metric 'user_result': no result
     - metric 'time': no result
   - testblock 'testblock_backward': 
     - metric 'interface': no result
     - metric 'tf_length_translation': no result
     - metric 'tf_length_translation': no result
     - metric 'user_result': no result
     - metric 'time': no result

Nevertheless there should be an (unindexed) bag file which can be used to evaluate metrics like path length, time etc. test result however should be False and error message should be test did not finish corretly

example is metric interface which is currently spread to atf_recorder_plugins for the recording phase and atf_metrics for the analysing phase

Hi guys,

If I define several testblocksets and only start one in the application script, this test fails even though it succeeds on all metrics. If I comment out all other testblocksets and run the analyzer again, it succeeds.

For example:
In testblocksets in testblock_1.yaml I define:

• testblock_1: ...
• testblock_2: ...

and start it in application.py:

• self.atf.start(testblock_1)

The test will fail.

But if I define in testblocksets in testblock_1.yaml:

• testblock_1: ...

and start it in application.py:

• self.atf.start(testblock_1)

The test will succeed.

should be able to

• visualize tests results given an atf_result.txt file (CI)

  • visualize data and groundtruth for a given metric in a given testblock for a given test, e.g. time in testblock_small from atf_test/ts0_c0_r0_e0_s0_0
  • visualize data for a all metrics in a given testblock for a given test
  • visualize data for all metrics in all testblocks in all tests
  • visualize aggregated data for all test repetitions for a given test, e.g. atf_test/ts0_c0_r0_e0_s0_0..10

• visualize comparisons given a single atf_result.txt file (Benchmarking)

  • visualize comparison for a given metric in various testblocks of a given test, e.g. path_length in testblock testblock_small and testblock testblock_large from atf_test/ts0_c0_r0_e0_s0_0
  • visualize comparision for all repetitions for a given test, e.g. atf_test/ts0_c0_r0_e0_s0_0..10

• visualize comparisons given multiple atf_result.txt files of the same test package from different test runs(CI-Series)

  • visualize time series data for a given metric in a given testblock for a given test, e.g. time in testblock_small from atf_test/ts0_c0_r0_e0_s0_0 from all atf_result.txt files.

traceback (most recent call last):
  File "/opt/ros/indigo/share/rostest/cmake/../../../bin/rostest", line 36, in <module>
    rostestmain()
  File "/opt/ros/indigo/lib/python2.7/dist-packages/rostest/__init__.py", line 273, in rostestmain
    _main()
  File "/opt/ros/indigo/lib/python2.7/dist-packages/rostest/rostest_main.py", line 167, in rostestmain
    results_file = xmlResultsFile(pkg, outname, is_rostest, env=env)
  File "/opt/ros/indigo/lib/python2.7/dist-packages/rosunit/core.py", line 102, in xml_results_file
    raise IOError("cannot create test results directory [%s]. Please check permissions."%(test_dir))
IOError: cannot create test results directory [/home/travis/catkin_ws/build/test_results/atf_test]. Please check permissions.

see https://travis-ci.org/ipa-fmw/atf/builds/132988829

probably two tests are trying to create directory at the same time. --> add check if directory exists. if yes: do noting

The ATF test case fails due to the delay in termination of the running simulation through ATF. I chased this into gazebo and noticed that there is a setUserQuit method that gets called on any signal and interrupts the server only if it is headless. Otherwise the setUserQuit method does nothing to the WX app loop. The delay escalation to sig-term after sending sig-int to roslaunch is not configurable, so changing this behaviour means editing :
/opt/ros/kinetic/lib/python2.7/dist-packages/roslaunch/nodeprocess.py
_TIMEOUT_SIGINT = 15.0 #seconds
_TIMEOUT_SIGTERM = 2.0 #seconds
Therefore, we need to have better escalation to sig-term after sending sig-int in ATF that will speed-up the Gazebo shutdown. For now no other test case fails except the one for which testcase run-time exceeds allocation.

running

rosrun atf_core record_tests.py atf_test

prints an ERROR which in fact is a INFO if the package atf_test is not found in the root workspace of a chained catkin workspace.

recording all in package 'atf_test'
ERROR: Requested path '/home/.../git/my_workspace/build/atf_test' does not exist.
found package 'atf_test' in '/home/.../git/my_chained_workspace/build/atf_test'

It is unclear what metrics take what parameters and what they do exactly.

The path_length metric is also replaced by tf_length_translation and tf_length_rotation but this is not yet documented

should preempt all testblocks and set their status to error if not already in the succeeded state

see #140

• explain usage
• explain arguments
  • style
  • sharey
  • filters for tests, testblocks and metrics
  • hide-groundtruth
  • hide-min-max
  • output pdf and png
• example commands and plots
• result is saved as /tmp/test.png

sometimes tests are aborted (e.g. due to time limit), this results in unindexed bag files altought there is (partially) usefull data recorded. The analyzer skips this bag files and adds no result to all the testblocks, athough some testblocks finished successfully.

if reindexing the bag file and the running the analyzer will result in successfull testblocks and partially executed testblocks, see

    name: "ts0_c1_r0_e0_s0_0"
    robot: "diff"
    env: "office"
    test_config: "dwa_local_planner"
    testblockset: "testblockset1"
    groundtruth_result: False
    groundtruth_error_message: "\n   - testblock 'testblock_rooms': \n     - metric 'tf_length_translation': no result\n\
  \     - metric 'tf_length_rotation': no result\n     - metric 'time': no result\n\
  \   - testblock 'testblock_all': \n     - metric 'tf_length_translation': no result\n\
  \     - metric 'interface': no result\n     - metric 'tf_length_rotation': no result\n\
  \     - metric 'time': no result"
    results: 
      - 
        name: "testblock_rooms"
        groundtruth_result: False
        groundtruth_error_message: "\n     - metric 'tf_length_translation': no result\n     - metric 'tf_length_rotation':\
  \ no result\n     - metric 'time': no result"
        results: 
          - 
            name: "tf_length_translation"
            started: True
            finished: False
            data: None
            groundtruth_result: False
            groundtruth_error_message: "no result"
            groundtruth: 50
            groundtruth_epsilon: 49
            details: []
          - 
            name: "tf_length_rotation"
            started: True
            finished: False
            data: None
            groundtruth_result: False
            groundtruth_error_message: "no result"
            groundtruth: 50
            groundtruth_epsilon: 49
            details: []

For the partially executed testblocks it's ok to have data: None but the error message should be no result (did not finish testblock) to indicate.

if running large testsuites (e.g. atf_test_app_navigation with 360 tests) the configuration_parser takes ~1min to complete. And the configuration_parser is used in application.py, in sm_test.py, analyser.py and plot.py. So we spend several times 1min each.

The slowdown is probably due to the many for loops. Maybe there is some room for optimization.

• catkin run_tests with FALSE, TRUE in CMakeLists.txt
• manual testing with non-default test_generation.yaml

#127 sometimes breaks Melodic builds: Sometimes there is an error on Travis Melodic builds, but Kinetic is always fine:

Traceback (most recent call last):
  File "/opt/ros/melodic/share/catkin/cmake/test/run_tests.py", line 41, in <module>
    sys.exit(main())
  File "/opt/ros/melodic/share/catkin/cmake/test/run_tests.py", line 18, in main
    remove_junit_result(args.results)
  File "/opt/ros/melodic/lib/python2.7/dist-packages/catkin/test_results.py", line 46, in remove_junit_result
    os.remove(filename)
OSError: [Errno 2] No such file or directory: '/root/target_ws/build/atf_test/test_results/atf_test/rostest-test_generated_atf_test_generation_config.xml'
make[3]: *** [CMakeFiles/_run_tests_atf_test_rostest_test_generated_atf_test_generation_config.yaml_recording_ts0_c0_r1_e0_s0_0.test] Error 1
make[2]: *** [CMakeFiles/_run_tests_atf_test_rostest_test_generated_atf_test_generation_config.yaml_recording_ts0_c0_r1_e0_s0_0.test.dir/all] Error 2
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/run_tests.dir/rule] Error 2
make: *** [run_tests] Error 2

full log: https://gist.github.com/floweisshardt/5a1f03a8517ca8e9c9b4ddaae4653af4

e.g.

• robot
• robot_env
• test_config
• tested package name

fixes #119

see e.g. #140

• add description

• list default values

• list example configs

• groundtruth and groundtruth_epsilon

• mode SNAP/SPAN

• series_mode

• min_window_size

the image below could e.g. be be grouped via

• r1 and r2
• c0, c1 and c2
  and split into subplots (new columns)

grouping should be an (optinal) parameter for plot.py

rosrun atf_plotter plot.py plot-benchmark -g r1,r2 /tmp/atf_test_app*/results_txt/atf_results.bag
rosrun atf_plotter plot.py plot-benchmark -g c0,c1,c2 /tmp/atf_test_app*/results_txt/atf_results.bag

is currently limited in travis.yml to ROS_PARALLEL_TEST_JOBS="-j1", see https://github.com/floweisshardt/atf/blob/master/.travis.yml#L14

it should be possible to configure the tf_length_X-metrics to only take translation/rotation around x axis into account.

Possible configuration examples are:

tf_length_translation/tf_distance_translation:

• default: calculate for all axes (x, y, z in root_frame)
• xyz: same as default
• x: only calculate for x
• y: only calculate for y
• z: only calculate for z
• xz: only calculate for x and z

tf_length_rotation/tf_distance_rotation:

• default: calculate for all axes (roll, pitch, yaw in root_frame)
• xyz: same as default
• x: only calculate for roll
• y: only calculate for pitch
• z: only calculate for yaw
• xz: only calculate for roll and yaw

currently needed after ATF constructor for all python subscribers and publisher to initialize correctly.
if needed at all, sleep should go into atf_core.ATF instead of user code

class Application:
    def __init__(self):
        self.atf = atf_core.ATF()
        #TODO WAIT FOR ATF to be initialized inside ATFController 
        rospy.sleep(3)

currently the travis job is spammed by a lot of terminal output from ATF

• rosparameters printed by roslaunch/rostest
• test result in case of failure in atf_results.txt

inspired by https://github.com/mojin-robotics/mojin_atf/pull/2/files#diff-834a870f3e7c3ec988cc7f8c5a5a893eR24

new optinal parameter: wait_for_tf which waits for a transformation between two frames to become available.

implement similar to the wait_for_topics and wait_for_service and wait_for_action #120