Git Product home page Git Product logo

atf's Introduction

The Automated Test Framework (ATF)

CI-Status master: Build Status

CI-Status atf_test_apps/master: Build Status

The ATF is a testing framework written for ROS which supports executing integration and system tests, running benchmarks and monitor the code behaviour over time. The ATF provides basic building blocks for easy integration of the tests into your application. Furthermore the ATF provides everything to automate the execution and analysis of tests as well as a graphical web-based frontend to visualize the results.

Auto-generated TOC with https://imthenachoman.github.io/nGitHubTOC/.

Architecture

Overview

There is a presentation for a brief ATF introduction.

Recording data

Analysing metrics

Visualising results

Implemented metrics

The following metrics have been implemented so far:

Metric Description Unit Mode (span, snap)
interface The interface metric checks if an interfaces (nodes, publishers, subscribers, service servers, action servers) matches its specification. [bool] snap
publish_rate The publish_rate metric measures the publising rate of a topic [1/sec] span
tf_acceleration_translation The tf_acceleration_translation metric measures the cartesian acceleration of a TF frame with respect to another frame. [m/s^2] span
tf_distance_rotation The tf_distance_rotation metric measures the cartesian angular distance of a TF frame with respect to another frame at the end of a testblock. [rad] snap
tf_distance_translation The tf_distance_translation metric measures the cartesian distance of a TF frame with respect to another frame at the end of a testblock. [m] snap
tf_jerk_translation The tf_jerk_translation metric measures the cartesian jerk of a TF frame with respect to another frame. [m/s^3] span
tf_length_rotation The tf_length_rotation metric measures the cartesian angular path (angular distance integrated over time) of a TF frame with respect to another frame. [rad] span
tf_length_translation The tf_length_translation metric measures the cartesian path (distance integrated over time) of a TF frame with respect to another frame. [m] span
tf_velocity_translation The tf_velocity_translation metric measures the cartesian velocity of a TF frame with respect to another frame. [m/s] span
time The time metric measures the elapsed time. [sec] span
topic_data The topic_data metric fetches a numerical data filed from a message [any] span, snap
user_result The result for the user_result metric can be set from the user within the application.py. [any] span, snap
See full list of implemented metrics at https://github.com/floweisshardt/atf/tree/master/atf_metrics.

Further metrics (in development):

Metric Description Unit Mode (span, snap)
resources The resources metric measures the resource consumption of a node on the operating system level (CPU, RAM, IO). [%], [MB], [MB/sec] snap
obstacle_distance The obstacle_distance metric measures the distance between two meshes [m] snap
message_match The message_match metric checks if a message content matches its desired content. [bool] snap

Installation

For installation instruction see ATF Installation.

Using the ATF (by examples)

For examples how to use the ATF see ATF Examples

Contributing to the ATF

For examples how to extend the ATF with your own contribution see ATF Contribution

Acknowledgements

The work leading to these results has received funding from the European Community's Seventh Framework Program (FP7/2007-2013) under grant agreement no 609206 Factory-in-a-Day and the German Federal Ministry for Economic Affairs and Energy under grant agreement no 01MA13001A ReApp.

atf's People

Contributors

acuda avatar awesomebytes avatar chamango90 avatar floweisshardt avatar fmessmer avatar hannesbachter avatar koellsch avatar loyvanbeek avatar maralbus avatar souravran avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

atf's Issues

redord_tests: do not print ERROR while searching package in $CMAKE_PREFIX_PATH

running

rosrun atf_core record_tests.py atf_test

prints an ERROR which in fact is a INFO if the package atf_test is not found in the root workspace of a chained catkin workspace.

recording all in package 'atf_test'
ERROR: Requested path '/home/.../git/my_workspace/build/atf_test' does not exist.
found package 'atf_test' in '/home/.../git/my_chained_workspace/build/atf_test'

extend docu

  • catkin run_tests with FALSE, TRUE in CMakeLists.txt
  • manual testing with non-default test_generation.yaml

metrics: make tf metrics configurable for x,y,z,r,p,y

it should be possible to configure the tf_length_X-metrics to only take translation/rotation around x axis into account.

Possible configuration examples are:

tf_length_translation/tf_distance_translation:

  • default: calculate for all axes (x, y, z in root_frame)
  • xyz: same as default
  • x: only calculate for x
  • y: only calculate for y
  • z: only calculate for z
  • xz: only calculate for x and z

tf_length_rotation/tf_distance_rotation:

  • default: calculate for all axes (roll, pitch, yaw in root_frame)
  • xyz: same as default
  • x: only calculate for roll
  • y: only calculate for pitch
  • z: only calculate for yaw
  • xz: only calculate for roll and yaw

travis error for concurrent tests

traceback (most recent call last):
  File "/opt/ros/indigo/share/rostest/cmake/../../../bin/rostest", line 36, in <module>
    rostestmain()
  File "/opt/ros/indigo/lib/python2.7/dist-packages/rostest/__init__.py", line 273, in rostestmain
    _main()
  File "/opt/ros/indigo/lib/python2.7/dist-packages/rostest/rostest_main.py", line 167, in rostestmain
    results_file = xmlResultsFile(pkg, outname, is_rostest, env=env)
  File "/opt/ros/indigo/lib/python2.7/dist-packages/rosunit/core.py", line 102, in xml_results_file
    raise IOError("cannot create test results directory [%s]. Please check permissions."%(test_dir))
IOError: cannot create test results directory [/home/travis/catkin_ws/build/test_results/atf_test]. Please check permissions.

see https://travis-ci.org/ipa-fmw/atf/builds/132988829

probably two tests are trying to create directory at the same time. --> add check if directory exists. if yes: do noting

atf.shutdown() does not shutdown all testblocks

this is a bug :)

atf sm blockes forever if one testblock is not stopped when calling atf.shutdown()

also, I need to call atf.stop(testblock) for testblocks that have not even started (no atf.start(testblock) called before)

Concept for Snapshot Tf Metrics

in our scenarios, we want the metrics to check the tf distances at a certain time, e.g. when putting a box to the internal storage
analyzing this distance as the minimum over the whole recording period is unreliable as we might move closer to that goal pose during some other phases of the scenario

the minimum approach only works for groundtruth.data: 0.0 - but when there is no tf frame at the goal pose this is not the case
#179 might also be a solution for the problem

Metric for distance to PoseStamped

The current TF-based metrics use the distance between origins of TF frames. But to check whether a robot eg. passed through some waypoint etc that doesn't have a TF frame, calculating the distance between two poses would be very handy.

Docs for metrics missing

It is unclear what metrics take what parameters and what they do exactly.

The path_length metric is also replaced by tf_length_translation and tf_length_rotation but this is not yet documented

[recording] solve name collissions

when running multiple testsuite in the same package (e.g. with multiple CMake macros set to TRUE during catkin run_tests) there are name collission for

  • recorded bag files ts*
  • test results txt files ts*
  • test results bag files ts*
  • overall test results atf_result.txt, atf_result.bag

Solution:

  • apply date/time suffix
  • symlink latest recordings and test results to latest (for easy plotter terminal commands)

[plotter] group subplots via test names

the image below could e.g. be be grouped via

  • r1 and r2
  • c0, c1 and c2
    and split into subplots (new columns)

image

grouping should be an (optinal) parameter for plot.py

rosrun atf_plotter plot.py plot-benchmark -g r1,r2 /tmp/atf_test_app*/results_txt/atf_results.bag
rosrun atf_plotter plot.py plot-benchmark -g c0,c1,c2 /tmp/atf_test_app*/results_txt/atf_results.bag

atf presenter with pyplot

should be able to

  • visualize tests results given an atf_result.txt file (CI)

    • visualize data and groundtruth for a given metric in a given testblock for a given test, e.g. time in testblock_small from atf_test/ts0_c0_r0_e0_s0_0
    • visualize data for a all metrics in a given testblock for a given test
    • visualize data for all metrics in all testblocks in all tests
    • visualize aggregated data for all test repetitions for a given test, e.g. atf_test/ts0_c0_r0_e0_s0_0..10
  • visualize comparisons given a single atf_result.txt file (Benchmarking)

    • visualize comparison for a given metric in various testblocks of a given test, e.g. path_length in testblock testblock_small and testblock testblock_large from atf_test/ts0_c0_r0_e0_s0_0
    • visualize comparision for all repetitions for a given test, e.g. atf_test/ts0_c0_r0_e0_s0_0..10
  • visualize comparisons given multiple atf_result.txt files of the same test package from different test runs(CI-Series)

    • visualize time series data for a given metric in a given testblock for a given test, e.g. time in testblock_small from atf_test/ts0_c0_r0_e0_s0_0 from all atf_result.txt files.

Delay in Gazebo shutdown if started through ATF.

The ATF test case fails due to the delay in termination of the running simulation through ATF. I chased this into gazebo and noticed that there is a setUserQuit method that gets called on any signal and interrupts the server only if it is headless. Otherwise the setUserQuit method does nothing to the WX app loop. The delay escalation to sig-term after sending sig-int to roslaunch is not configurable, so changing this behaviour means editing :
/opt/ros/kinetic/lib/python2.7/dist-packages/roslaunch/nodeprocess.py
_TIMEOUT_SIGINT = 15.0 #seconds
_TIMEOUT_SIGTERM = 2.0 #seconds
Therefore, we need to have better escalation to sig-term after sending sig-int in ATF that will speed-up the Gazebo shutdown. For now no other test case fails except the one for which testcase run-time exceeds allocation.

inherit testblock configuration

for a large number of similar testblocks (>10-100) it's not very comfortable to configure every testblock individually in the testblockset.yaml files. Instead it would be nice to configure metrics for a master testblock and then use the same configuration in all other testblocks.

Questions:

  • how to handle different groundtruth settings?

get rid of rospy.sleep() in application code

currently needed after ATF constructor for all python subscribers and publisher to initialize correctly.
if needed at all, sleep should go into atf_core.ATF instead of user code

class Application:
    def __init__(self):
        self.atf = atf_core.ATF()
        #TODO WAIT FOR ATF to be initialized inside ATFController 
        rospy.sleep(3)

install: merging travis.rosinstall does not clone atf sources

The installation instructions under the step labeled Get ATF sources from github instruct to wstool merge https://raw.githubusercontent.com/ipa-fmw/atf/master/.travis.rosinstall into a catkin workspace to get the sources.

The referenced file only contains a stanza for ros_comm, which seems to not have any relation to this project:

- git:
    uri: 'https://github.com/ros/ros_comm.git'
    local-name: ros_comm
    version: indigo-devel

I just git cloned the repository to my workspace, which seems to work as well.

implement atf.shutdown()

should preempt all testblocks and set their status to error if not already in the succeeded state

[analyser] analyse metrics in case of timeout during recording phase (broken bag file)

if the recording gets aborted (equvivalent to CTRL-C)

max time [360.0s] allotted for test [atf_recording_ts0_c0_r0_e0_s0_0] of type [atf_core/sm_test.py]

we get somthing like

 - test 'ts0_c0_r0_e0_s0_0' (cob4-2, ipa-apartment, test_default, testblockset_ipa-apartment): 
   - testblock 'testblock_forward': 
     - metric 'interface': no result
     - metric 'tf_length_translation': no result
     - metric 'tf_length_translation': no result
     - metric 'user_result': no result
     - metric 'time': no result
   - testblock 'testblock_all': 
     - metric 'interface': no result
     - metric 'tf_length_translation': no result
     - metric 'tf_length_translation': no result
     - metric 'user_result': no result
     - metric 'time': no result
   - testblock 'testblock_backward': 
     - metric 'interface': no result
     - metric 'tf_length_translation': no result
     - metric 'tf_length_translation': no result
     - metric 'user_result': no result
     - metric 'time': no result

Nevertheless there should be an (unindexed) bag file which can be used to evaluate metrics like path length, time etc. test result however should be False and error message should be test did not finish corretly

timeout concept for wait_for features

make recording fail with exception/timeout message when timeout is exceeded...

needed for:

  • wait_for_action
  • wait_for_topic
  • wait_for_service
  • wait_for_diagnostics

atf cannot be used from install space

because the cmake macro tries to run generate_tests.py from @(CMAKE_SOURCE_DIR)/scripts/generate_tests.py

  Errors     << mojin_atf_kevin:cmake /root/target_ws/logs/mojin_atf_kevin/build.cmake.000.log
  ATF: executing test generation macro with 'atf/test_generation_config.yaml' and EXECUTE_TESTS=True
  /usr/bin/python2: can't open file '/root/git/app_ws/src/atf/atf_core/scripts/generate_tests.py': [Errno 2] No such file or directory
  CMake Error at /root/git/app_ws/install/share/atf_core/cmake/atf_core-extras.cmake:39 (message):
    -- ATF: generating test files failed: exit_code='2'
  Call Stack (most recent call first):
    CMakeLists.txt:27 (atf_test)

concept for analyzing and visualizing time series

e.g. for

  • path position/velocity/acceleration/jerks profiles
  • joint position/velocity/acceleration/jerk profiles
  • battery profile
  • ...

currently the result data is limited to a single float (https://github.com/floweisshardt/atf/blob/master/atf_msgs/msg/MetricResult.msg#L4).
This could be used for plotting a time series this would need to be

recording and analyzing

  • (prefered) add an optional time series field to all span metrics (e.g. tf_length_translation) additionally to the single float. data type needs to be a list of stamped floats
  • frequency needs to be configurable in testblockset.yaml
    • none
    • either on each update step
    • fixed rate (trigger mechanism needed! interpolation needed?)
  • can be expanded with fields for
    • min/max
    • avg/std_dev

visualization

  • plot time series

test fails for several testblocksets that are not used

Hi guys,

If I define several testblocksets and only start one in the application script, this test fails even though it succeeds on all metrics. If I comment out all other testblocksets and run the analyzer again, it succeeds.

For example:
In testblocksets in testblock_1.yaml I define:

  • testblock_1: ...
  • testblock_2: ...

and start it in application.py:

  • self.atf.start(testblock_1)

The test will fail.

But if I define in testblocksets in testblock_1.yaml:

  • testblock_1: ...

and start it in application.py:

  • self.atf.start(testblock_1)

The test will succeed.

[analyser] Exception for incomplete testblock data

discovered by @HannesBachter

this can be reproduzed by calling self.atf.stop(TESTBLOCK) without calling self.atf.start(TESTBLOCK) before.

rosrun atf_core analyser.py mojin_atf_mbl
analysing for package mojin_atf_mbl
[ROSUNIT] Outputting test results to /home/hyb/.ros/test_results/atf_core/rosunit-analysing.xml
0 errors detected during test processing
Processing tests took 8.9079 sec
ATF analyser: done!
An error occured during analysis of testblock 'testblock_shelf', no useful results available.
[Testcase: test_analysing] ... ERROR!
ERROR: global name 'ATFAnalyserError' is not defined
   File "/usr/lib/python2.7/unittest/case.py", line 329, in run
     testMethod()
   File "/home/hyb/git/care-o-bot/src/atf/atf_core/scripts/analyser.py", line 190, in test_analysing
     atf_result = analyser.get_result()
   File "/home/hyb/git/care-o-bot/src/atf/atf_core/scripts/analyser.py", line 122, in get_result
     test_result = test.get_result()
   File "/home/hyb/git/care-o-bot/src/atf/atf_core/src/atf_core/test.py", line 34, in get_result
     testblock_result = testblock.get_result()
   File "/home/hyb/git/care-o-bot/src/atf/atf_core/src/atf_core/testblock.py", line 45, in get_result
     raise ATFAnalyserError("Analysing failed, no testblock result available for testblock '%s'."%testblock_result.name)
--------------------------------------------------------------------------------
-------------------------------------------------------------
SUMMARY:
  * RESULT: FAIL
  * TESTS: 1
  * ERRORS: 1 [test_analysing]
  * FAILURES: 0 []

[configuration_parser] is slow for large teststuites

if running large testsuites (e.g. atf_test_app_navigation with 360 tests) the configuration_parser takes ~1min to complete. And the configuration_parser is used in application.py, in sm_test.py, analyser.py and plot.py. So we spend several times 1min each.

The slowdown is probably due to the many for loops. Maybe there is some room for optimization.

verify jerk calculation

in atf_test some jerk values deviate from the others. Problems seems more relevant if static transforms are involved.
image

Unstable Melodic builds with multiple cmake macros

#127 sometimes breaks Melodic builds: Sometimes there is an error on Travis Melodic builds, but Kinetic is always fine:

Traceback (most recent call last):
  File "/opt/ros/melodic/share/catkin/cmake/test/run_tests.py", line 41, in <module>
    sys.exit(main())
  File "/opt/ros/melodic/share/catkin/cmake/test/run_tests.py", line 18, in main
    remove_junit_result(args.results)
  File "/opt/ros/melodic/lib/python2.7/dist-packages/catkin/test_results.py", line 46, in remove_junit_result
    os.remove(filename)
OSError: [Errno 2] No such file or directory: '/root/target_ws/build/atf_test/test_results/atf_test/rostest-test_generated_atf_test_generation_config.xml'
make[3]: *** [CMakeFiles/_run_tests_atf_test_rostest_test_generated_atf_test_generation_config.yaml_recording_ts0_c0_r1_e0_s0_0.test] Error 1
make[2]: *** [CMakeFiles/_run_tests_atf_test_rostest_test_generated_atf_test_generation_config.yaml_recording_ts0_c0_r1_e0_s0_0.test.dir/all] Error 2
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/run_tests.dir/rule] Error 2
make: *** [run_tests] Error 2

full log: https://gist.github.com/floweisshardt/5a1f03a8517ca8e9c9b4ddaae4653af4

Add documentation for plotter

see #140

  • explain usage
  • explain arguments
    • style
    • sharey
    • filters for tests, testblocks and metrics
    • hide-groundtruth
    • hide-min-max
    • output pdf and png
  • example commands and plots
  • result is saved as /tmp/test.png

analyzer: reindex bag files if needed

sometimes tests are aborted (e.g. due to time limit), this results in unindexed bag files altought there is (partially) usefull data recorded. The analyzer skips this bag files and adds no result to all the testblocks, athough some testblocks finished successfully.

if reindexing the bag file and the running the analyzer will result in successfull testblocks and partially executed testblocks, see

    name: "ts0_c1_r0_e0_s0_0"
    robot: "diff"
    env: "office"
    test_config: "dwa_local_planner"
    testblockset: "testblockset1"
    groundtruth_result: False
    groundtruth_error_message: "\n   - testblock 'testblock_rooms': \n     - metric 'tf_length_translation': no result\n\
  \     - metric 'tf_length_rotation': no result\n     - metric 'time': no result\n\
  \   - testblock 'testblock_all': \n     - metric 'tf_length_translation': no result\n\
  \     - metric 'interface': no result\n     - metric 'tf_length_rotation': no result\n\
  \     - metric 'time': no result"
    results: 
      - 
        name: "testblock_rooms"
        groundtruth_result: False
        groundtruth_error_message: "\n     - metric 'tf_length_translation': no result\n     - metric 'tf_length_rotation':\
  \ no result\n     - metric 'time': no result"
        results: 
          - 
            name: "tf_length_translation"
            started: True
            finished: False
            data: None
            groundtruth_result: False
            groundtruth_error_message: "no result"
            groundtruth: 50
            groundtruth_epsilon: 49
            details: []
          - 
            name: "tf_length_rotation"
            started: True
            finished: False
            data: None
            groundtruth_result: False
            groundtruth_error_message: "no result"
            groundtruth: 50
            groundtruth_epsilon: 49
            details: []

For the partially executed testblocks it's ok to have data: None but the error message should be no result (did not finish testblock) to indicate.

Code duplication in TF metrics

The metrics related to TF (distance, jerk, velocity, length) have a lot of code duplication between them.

In eg. CalculateTfAccelerationTranslation, vs CalculateTfDistanceTranslation only a few members are added and a few lines in get_data are different but mostly the same. Some basic inheritance can reduce a lot of duplication and reduce maintenance.

Might be related to #58 as well.

Yaml loading deprecation

I got this error:
/home/lvb/git/care-o-bot/src/atf/atf_core/src/atf_core/configuration_parser.py:181: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. doc = yaml.load(stream)

which points to https://msg.pyyaml.org/load

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.