robotology / robot-testing-framework Goto Github PK
View Code? Open in Web Editor NEWRobot Testing Framework (RTF)
Home Page: http://robotology.github.io/robot-testing-framework/index.html
License: GNU Lesser General Public License v2.1
Robot Testing Framework (RTF)
Home Page: http://robotology.github.io/robot-testing-framework/index.html
License: GNU Lesser General Public License v2.1
Hi @apaikan , thanks for providing this tool.
I'm just curious why in some cases the expression "suit" is used (here), whereas other times it's "suite" (here).
If it was just a typo, perhaps it's worth changing it to be consistent everywhere?
Feel free to close if you consider this a minor/irrelevant issue. :)
I had a fatal error of this type:
pegua@pareto:~/src/icub-tests/suits$ testrunner --suit basics-icubSim.xml --verbose
Loading basics-icubSim.xml
Staring test runner.
Test suit Basic Tests Suite started...
[INFO] (Basic Tests Suite) reports: yarpmanager is setuping the fixture...
Returning true from yarpManager setup*** stack smashing detected ***: testrunner terminated
Aborted (core dumped)
This is probably caused by some wrong parameters passed to the yarpmanager fixture plugin. I have disabled the GCC stack protection compiling RTF
with the -fno-stack-protector
, and I still get some error because it is not able to find the configuration files.
the RTF_TEST_FAIL_IF macro fail if the condition is false. example:
int a = 5;
int b = 2;
RTF_TEST_FAIL_IF(a > b) //this call does not fail as it should..
On Windows the test FixturePluginLoader
get stuck if RTF
is compiled in Debug.
We should also enable tests on appveyor and the compilation in debug.
https://travis-ci.org/robotology/robot-testing/jobs/96977269
Actual error:
[ 60%] Building CXX object src/plugins/python/CMakeFiles/RTF_python.dir/src/PythonPluginLoader.cpp.o
/home/travis/build/robotology/robot-testing/src/plugins/python/src/PythonPluginLoader.cpp: In member function ‘RTF::TestCase* RTF::plugin::PythonPluginLoaderImpl::open(std::string)’:
/home/travis/build/robotology/robot-testing/src/plugins/python/src/PythonPluginLoader.cpp:112:49: error: ‘PyString_FromString’ was not declared in this scope
/home/travis/build/robotology/robot-testing/src/plugins/python/src/PythonPluginLoader.cpp:135:52: error: ‘Py_InitModule4’ was not declared in this scope
/home/travis/build/robotology/robot-testing/src/plugins/python/src/PythonPluginLoader.cpp: In member function ‘std::string RTF::plugin::PythonPluginLoaderImpl::getPythonErrorString()’:
/home/travis/build/robotology/robot-testing/src/plugins/python/src/PythonPluginLoader.cpp:183:37: error: ‘PyString_Check’ was not declared in this scope
/home/travis/build/robotology/robot-testing/src/plugins/python/src/PythonPluginLoader.cpp:184:44: error: ‘PyString_AsString’ was not declared in this scope
/home/travis/build/robotology/robot-testing/src/plugins/python/src/PythonPluginLoader.cpp:191:37: error: ‘PyString_Check’ was not declared in this scope
/home/travis/build/robotology/robot-testing/src/plugins/python/src/PythonPluginLoader.cpp:192:60: error: ‘PyString_AsString’ was not declared in this scope
/home/travis/build/robotology/robot-testing/src/plugins/python/src/PythonPluginLoader.cpp: In member function ‘virtual bool RTF::plugin::PythonPluginLoaderImpl::setup(int, char**)’:
/home/travis/build/robotology/robot-testing/src/plugins/python/src/PythonPluginLoader.cpp:231:52: error: ‘PyString_FromString’ was not declared in this scope
make[2]: *** [src/plugins/python/CMakeFiles/RTF_python.dir/src/PythonPluginLoader.cpp.o] Error 1
make[1]: *** [src/plugins/python/CMakeFiles/RTF_python.dir/all] Error 2
make: *** [all] Error 2
Googling for similar errors it appears that the bug is caused by a mismatch between the Python versions:
https://bugs.launchpad.net/opencog/+bug/1063039
http://sourceforge.net/p/robotcub/mailman/robotcub-hackers/thread/[email protected]/
Probably now the Travis images have been update to have multiple versions of Python, and there is a bug in the CMake code handling Python versions. I will look into it.
According to @traversaro it would be important to control the order of execution of both tests and fixtures.
While usual unit testing has to be order independent, in a robot testing scenario it would be useful to test small components at first (like encoder response, plugin response etc) and then larger behaviors like for instance navigation and grasping.
For fixtures, it is necessary where some fixtures (speaking of the yarp framework a yarp device, yarp application or the yarpmanager itself for example) need other utilities (like yarp server) in order to start correctly.
As far as I know, this functionality is not available in RTF. The way to implement it is the following:
In my opinion, the simplest and most intuitive solution from both the user and the development side is to maintain the order of the xml tags (top-bottom). This can be done by simply change the tests and fixtures data storage for from std::set to std vector. While the loss in terms of performance is negligible in my opinion an inconsistency with the xml parsing policy of yarp (where only the depth order of the tags is relevant). However being RTF and YARP two completely separate frameworks this shouldn't be a problem.
There is no way to control the fixture or test execution order in RTF while in some cases this is a requirement. Let's discuss on how to implement it.
RTF.testCheck
in the binding language needs to be modified to come up with the recent change in RTF_TEST_CHECK
macro from C++.
To integrate nicely with the traditional CMake/CTest add_test
command, we need that the testrunner
returns a number different from 0
on failure.
For this are necessary a bunch of modifications:
return 0
in https://github.com/robotology/robot-testing/blob/master/src/testrunner/src/main.cpp#L232 should be substituted with return collector.failedCount();
load*
method) I guess it make sense to simply return 1.All the RTF headers are installed in the include
directory, so they are used as:
#include <TestCase.h>
#include <TestAssert.h>
#include <Arguments.h>
Could it make sense to move them in include/rtf
(or include/RTF
), so we will have:
#include <rtf/TestCase.h>
#include <rtf/TestAssert.h>
#include <rtf/Arguments.h>
cc @apaikan
This code
if(true)
RTF_TEST_REPORT(Asserter::format("Starting with SystemClock"));
else
RTF_TEST_REPORT(Asserter::format("Starting with Network Clock from port %s", clockPort_EnvName.c_str()));
Give this compilation error:
In member function ‘virtual void ClockTest::run()’:
/usr/local/src/robot/icub-src/icub-tests/src/clock/ClockTest.cpp:84:5: error: ‘else’ without a previous ‘if’
else
^
While using brackets it works fine
if(true)
{ RTF_TEST_REPORT(Asserter::format("Starting with SystemClock")); }
else
{ RTF_TEST_REPORT(Asserter::format("Starting with Network Clock from port %s", clockPort_EnvName.c_str())); }
Need to discuss how to place this c++11 tag.
The description of this issue is still be completed.
Four tests:
Robots verification table:
http://wiki.icub.org/images/c/c5/Signs.xls
Matlab code:
available in icub-main\app\robots\experimentalSetups\test_utils\matlab\verify_encoders.m
Similar to how it is done for YARP : https://github.com/robotology/yarp/tree/master/doc/release .
To simplify adding tests using testrunner
to CTest using the add_test
cmake command, the CMake infrastructure of RTF should provide a variable containing the absolute path to the testrunner binary.
Helper CMake functions to automatically add a CTest test from a suite would also be convenient.
cc @S-Dafarra
It seems that tests may fail because when they start, fixture is not ready yet to execute the test.
Example: I start a motor test on the robot simulator. Even if the simulator is immediately launched (and control ports are responsive), it take approx 10 seconds to perform an initial "calibration". PositionMove() commands are not correctly executed until the parts reach their final home position.
Of course this issue could be addressed by improving the state-machine of the simulator, however I'm wondering if it worth implementing a parametric delay after the fixture startup for applications (e.g. third party modules) which do not provide a better handshaking mechanism...
If I have a Syntax Error in my test, the test fails with the following message:
1: Test case geometry.py started...
1: [INFO] (geometry.py) reports: Running test testInnerProductInvariance
1: [INFO] (geometry.py) reports: Running test testTransformInverse
1: [ERROR] (geometry.py) asserts error with exception: Cannot call the run() method because Syntax Error!
1: Test case geometry.py failed!
I still have to understand why it does not print a more helpful message.
E.g.:
RTF_TEST_CHECK(value, "checking value")
should be logged as:
checking value...OK
or
checking value... NOT OK
At the moment the macro prints the string only if the tests fails.
cxxabi.h, included by robot-testing\src\testrunner\include\cmdline.h is not standard (gcc only?). testrunner does not compile on windows VS2010.
For now robot-testing supports just test that are hard coded (and directly linked) into the main application robot-testing
.
This is undesirable for the following reasons:
robot-testing
.yarp has a nice feature of allowing dynamically loaded device drivers. I guess we can borrow most of this yarp device drivers infrastructure/logic, in a way that robot-testing
will play a role similar to what yarpdev
is doing for devices.
In a nutshell, the ideal scenario is the one where the CI jobs will simply call robot-testing testListSimulation.ini
.
Similarly a human user that want to test a real robot will call robot-testing testListRobot.ini
.
For some application, it is convenient to set a specific environmental variable for just a test, and do not have this environmental variable set to the rest of the fixture in the suite or for other tests.
The doxygen generated documentation available at http://robotology.github.io/robot-testing/index.html is currently outdated (it contains the documentation of the 1.0 version and it was generate the last time the 5th of January).
While compiling under macOS the following warning are raised:
src/rtf/src/Arguments.cpp:17:17: warning: using directive refers to implicitly-defined namespace
'std'
using namespace std;
^
src/testrunner/src/SuitRunner.cpp:170:44: warning: comparison of unsigned expression >= 0 is
always true [-Wtautological-compare]
if (endptr == 0 && rep >= 0) {
~~~ ^ ~
We need some helpers for testing approximate equality of floating point values.
An important feature to have is to print the values when the comparison fails, for easy debugging of failing tests (as in gtest EXPECT_NEAR
, ASSERT_NEAR
in https://code.google.com/p/googletest/wiki/AdvancedGuide ),
Just cloned the clean repo and I get 2 compilation errors:
src/UnitTest.c -> fabs not defined
TestMotors.cpp:365:52: error: ‘isTrue’ was not declared in this scope
doneAll=isTrue(done_vector, m_NumJoints);
Am I the only one? :$
Hi,
The links under Tutorials and Examples are broken.
Cheers,
Michele
...for logging purposes please use yInfo(), yWarning(), yError(), including < yarp/os/LogStream.h > and < yarp/os/Log.h > as appropriate.
If I tried to load with the yarpmanager fixture plugin an application with a name (contained in the xml <name>
tag) longer or equal to 16 characters, I got a failure message:
pegua@pareto:~/src/icub-tests/build$ testrunner --suit ../suits/basics-icubGazeboSim.xml --verbose
Loading ../suits/basics-icubGazeboSim.xml
Staring test runner.
Test suit Basic Tests Suite started...
[INFO] (Basic Tests Suite) reports: yarpmanager is setuping the fixture...
[ERROR] (Basic Tests Suite) asserts error on (ret) with exception: yarpmanager cannot setup the fixture because Application is not loaded.
[INFO] (Basic Tests Suite) reports: yarpmanager is tearing down the fixture...
Test suit Basic Tests Suite failed!
Ending test runner.
If one of the modules of a fixture is not launched because its <dependencies>
are not satisfied, yarpmanager consider the fixture to be successful anyway, launching the tests.
In case you are debugging a failure of setting up a fixture using the yarpmanager, it is quite difficult to understand what is going on. This because the yarpmanager fixture manager does not print any information on which application it is launching, on which dependency is waiting, etc etc. It does not print anything even if the testrunner was launched with the verbose
option.
This does make sense if using the yarpmanager application (because the user can check on the gui what is going on) but is a bit more complicated for use in rtf.
cc @apaikan What is a good strategy to add some (optional) verbose print to yarpmanager ? I saw that libYARP_manager
is relativly yarp-indipendent, so I don't know it it make sense to use the yarp output macros in it.
While configuring CMake, I get the following warning:
CMake Warning (dev) at src/rtf/CMakeLists.txt:8 (project):
Policy CMP0048 is not set: project() command manages VERSION variables.
Run "cmake --help-policy CMP0048" for policy details. Use the cmake_policy
command to set the policy and suppress this warning.
The following variable(s) would be set to empty:
RTF_VERSION
RTF_VERSION_MAJOR
RTF_VERSION_MINOR
RTF_VERSION_PATCH
RTF_VERSION_TWEAK
This warning is for project developers. Use -Wno-dev to suppress it.
During generation I get:
Policy CMP0042 is not set: MACOSX_RPATH is enabled by default. Run "cmake
--help-policy CMP0042" for policy details. Use the cmake_policy command to
set the policy and suppress this warning.
MACOSX_RPATH is not specified for the following targets:
RTF
RTF_dll
This warning is for project developers. Use -Wno-dev to suppress it.
System: macOS 10.11.6 (El Capitan), CMake 3.7.0.
I am wondering that no one has wondered where are RTF self tests? 😜
A unit testing framework without unit tests .... 😶
TODO:
make tests
can run tests using testrunner
testrunner
without testrunner
!Currently in TestFTSensors we are reading just the sensor port, we should also add the test of use the AnalogSensorClient to read its value, check its state and the timestamps. Reading the raw port should however remain because it is how it is used in a lot of code.
cc @barbalberto any documentation on how to properly use AnalogSensorClient ?
In a nutshell, I propose we move the test name from the main configuration file, as in:
[REFERENCES]
user "alessandro scalzo"
comment "this is a wild bunch of tests"
outfile iCubTestReport.xml
[TESTS]
TestCamera right_camera.ini
TestCamera left_camera.ini
TestMotors test_left_arm.ini
TestMotors test_right_arm.ini
TestFTSensors test_ft_left_arm.ini
TestFTSensors test_ft_right_arm.ini
TestFTSensors test_ft_left_leg.ini
TestFTSensors test_ft_right_leg.ini
TestFTSensors test_ft_left_foot.ini
TestFTSensors test_ft_right_foot.ini
to the test configuration file, so we will have:
[REFERENCES]
user "alessandro scalzo"
comment "this is a wild bunch of tests"
outfile iCubTestReport.xml
[TESTS]
right_camera.ini
left_camera.ini
test_left_arm.ini
test_right_arm.ini
test_ft_left_arm.ini
test_ft_right_arm.ini
test_ft_left_leg.ini
test_ft_right_leg.ini
test_ft_left_foot.ini
test_ft_right_foot.ini
This will permit to reuse more easily the same .ini file in multiple main configuration files, and is more consistent with the device use (test and device share a lot of commonality in my opinion).
Example: if I write a fixture like this one:
<application>
<name>iCub Simulator</name>
<description>A fixture to prepare iCub Simulator for the test cases</description>
<version>1.0</version>
<authors>
<author email="[email protected]">Ali Paikan</author>
</authors>
<module>
<name>iCub_SIM</name>
<parameters></parameters>
<node>localhost</node>
<ensure>
<wait>1</wait>
</ensure>
</module>
<module>
<name>yarpview</name>
<parameters></parameters>
<node>localhost</node>
<ensure>
<wait>100</wait>
</ensure>
<dependencies>
<port timeout="5.0">/dsaadsdada</port>
</dependencies>
</module>
</application>
Nor the ensure tag neither the dependency tag are considered for yarpview
, and yarpview
is simply launched immediately and the fixture is considered loaded.
It will be very useful to have timeout
for running the tests using testrunner
.
As it is for now, if one of the test cases gets blocked, the testrunner
is blocked too and no more tests are running until the CTRL+C
is pressed from the testrunner
terminal.
--timeout <seconds>
param to the testrunner
<?xml version="1.0" encoding="UTF-8"?>
<suit name="my suit">
<description> a simple example </description>
<!-- default timeout for all tests -->
<timeout> 5.0 </timeout>
<test type="dll"> mytest2 </test>
...
<!-- timeout for a specific test -->
<test type="dll" timeout="10.0"> mytest1 </test>
...
</suit>
It will be useful (and in some cases necessary) to have multiple fixture manager for each suites. Example:
<suit name="Basic Tests Suite">
<description>Testing basic features</description>
<fixture param="--param myparam"> fixturemanager1 </fixture>
<fixture param="--param myparam"> fixturemanager2 </fixture>
...
<!-- tests -->
...
</suit>
There is still a compilation problem on windows due to lack of proper configuration (dllexport) of the shared libraries. It can be fixed as described here http://www.cmake.org/Wiki/BuildingWinDLL.
Different robots either real or simulated (ODE or gazebo) have different functionality. Even in case in which functionalities are the same, tests may need different parameters (thresholds, speeds etc).
#5 addresses how to propagate variables from the main configuration file to individual test files. This is for sure useful to propagate the name of the robot as a prefix for port names.
At the moment using the environment functionality (#5) we can use different configuration files (test-ode.ini/test-gazebo.ini)
I wonder if in the end we will need to build different directories/contexts, i.e.:
app/iCubSim --> test ode simulator
app/icub --> test real robot
app/icubGazeboSim --> gazebo simulator
If the testrunner
fails to find the plugin for a given tests, it prints the following erro message:
2: [testrunner] cannot load plugin myTest.so; error (load) : myTest.so: cannot open shared object file: No such file or directory
But the final output is:
2: ---------- results -----------
2: Total number of test suites : 1
2: Number of passed test suites : 1
2: Number of failed test suites : 0
2: Total number of test cases : 0
2: Number of passed test cases : 0
2: Number of failed test cases : 0
A test that was impossible to load should count as a failed test.
RTF_ADD_PLUGIN should force the exporting of the factory symbols
Ref: robotology/icub-tests#8
Example in Travis: https://travis-ci.org/robotology/gazebo-yarp-plugins/jobs/62661440
Coverall has been already setup for robot-testting (see .coveralls.yml
). However the Travis file and the related compile flags should be configured to run the code coverage!
Working on the tests, I realized that in some error situations, I would not only stop the current test, but avoid that the framework continues to launch tests of the same suit.
Therefore I propose to add a new error type in the framework for satisfy this need.
we could call it RTF_ESSERTER_SUIT_ERROR or RTF_ASSERT_CRITICALERROR.
Anyone has any suggestions?
By writing a class similar to TextOutputter
but that dumps the test results in a format compatible with Jenkins.
For now, yarp middleware plugin in RTF depends on yarp. This means that that it will be very complicated to use RTF to develop units tests within yarp simply because of mutual dependency!
1) The solution suggested by @drdanz is to move the RTF yarp middleware plugin to the yarp repository. In this way one can:
make tests
)2) The second option is to have the yarp unit tests in a separate repository. This needs a bit of more work to compile and run the tests especially if we want to run the new tests (i.e. written using RTF) along with the available tests in YARP.
In this Travis Job https://travis-ci.org/robotology/gazebo-yarp-plugins/jobs/62666628 :
testrunner --verbose --suit ../../icub-tests/suits/basics-icubGazeboSim.xml
Loading ../../icub-tests/suits/basics-icubGazeboSim.xml
[testrunner] cannot load plugin FtSensorTest.so; error (load) : FtSensorTest.so: cannot open shared object file: No such file or directory
[testrunner] cannot load plugin FtSensorTest.so; error (load) : FtSensorTest.so: cannot open shared object file: No such file or directory
[testrunner] cannot load plugin FtSensorTest.so; error (load) : FtSensorTest.so: cannot open shared object file: No such file or directory
Staring test runner.
Test suit Basic Tests Suite started...
[INFO] (Basic Tests Suite) reports: yarpmanager is setuping the fixture...
||| did not find fixtures/icubgazebosim-fixture.xml
[ERROR] (Basic Tests Suite) asserts error on (appfile.size()) with exception: yarpmanager cannot find apllication file icubgazebosim-fixture.xml. Is it in the 'fixtures' folder?
[INFO] (Basic Tests Suite) reports: yarpmanager is tearing down the fixture...
Test suit Basic Tests Suite failed!
Ending test runner.
-------- results ---------
Total number of tests : 0
Number of passed tests: 0
Number of failed tests: 0
The command "testrunner --verbose --suit ../../icub-tests/suits/basics-icubGazeboSim.xml" exited with 0.
The testrunner is returning 0 because the number of failed tests is zero, but we should make him aware of the ERROR in [ERROR] (Basic Tests Suite) asserts error on (appfile.size()) with exception: yarpmanager cannot find apllication file icubgazebosim-fixture.xml. Is it in the 'fixtures' folder?
and return a value different from 0.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.