coriander.cs.wisc.edu has the 32-bit libc libraries installed, so it's the only x86 platform that is running those tests. There are many crashes and failures.

bgq, bgp, irix, aix

If the environment variable DYNINST_REWRITER_NO_UNLINK is set and the entire testsuite is run several tests fail, due to the instrumented libc.so being reused.

Anyone seen this or similar on Ubuntu?

[ 27%] Built target test_snip_remove.dyn_gcc_64_none_none
[ 27%] Built target test3_7.dyn_gcc_64_none_none
[ 27%] Built target dyninst_group_test.dyn_gcc_64_none_none
[ 27%] Built target test1_19.dyn_gcc_64_none_none
[ 27%] Linking C executable test_mem_1.dyn_gcc_64_none_none
/usr/bin/ld: CMakeFiles/test_mem_1.dyn_gcc_64_none_none.dir/__/__/__/__/__/__/__/src/dyninst/test6LS-x86_64.s.o: relocation R_X86_64_32 against `.text' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: Nonrepresentable section on output
collect2: error: ld returned 1 exit status
x86_64-unknown-linux2.4/dyninst/gcc/64/dyn/none/none/CMakeFiles/test_mem_1.dyn_gcc_64_none_none.dir/build.make:190: recipe for target 'x86_64-unknown-linux2.4/dyninst/gcc/64/dyn/none/none/test_mem_1.dyn_gcc_64_none_none' failed
make[2]: *** [x86_64-unknown-linux2.4/dyninst/gcc/64/dyn/none/none/test_mem_1.dyn_gcc_64_none_none] Error 1
CMakeFiles/Makefile2:8928: recipe for target 'x86_64-unknown-linux2.4/dyninst/gcc/64/dyn/none/none/CMakeFiles/test_mem_1.dyn_gcc_64_none_none.dir/all' failed
make[1]: *** [x86_64-unknown-linux2.4/dyninst/gcc/64/dyn/none/none/CMakeFiles/test_mem_1.dyn_gcc_64_none_none.dir/all] Error 2
Makefile:116: recipe for target 'all' failed
make: *** [all] Error 2

Sometimes the binaries installed into a separate installation directory do not match the ones in the build directory. Bolo found that

As far as installed binaries not matching built binaries, that is
all due to cmake_install.cmake hacking up binaries to change
rpath and other attributes ... and then not changing the modification
time on the, now different, installed binary.

This behavior occurs if anyone uses an "installed"
testsuite.   If you just run the testsuite in the
build dir, it is not installed, so the rewrite
doesn't occur.

We need to force CMake to always copy the files from the build directory into the install directory

In rewriter mode test, the mutator executes the rewritten mutatee and uses waitpid to get the return status of the mutatee, checking whether the mutatee crashed or finished normally.

However, the rewritten mutatee can crash after exit(). For example, global destructor code is executed after exit(). In this case, waitpid seems to return a normal exit code, causing the mutator believes the test finished normally.

I observed this behavior when no tests are reported as crashed, but there are core files generated. The bug in Dyninst that caused the crash is fixed, but the test suite needs to be able to detect such failure mode.

https://bottle.cs.wisc.edu/results/CRASHED/351

We only started running on this platform in Oct 2019, so these crashes haven't been addressed before.

http://jenkins.dyninst.org/jenkins/job/testsuite-build/lastFailedBuild/console

When we're adding dependencies to the tests like this, they should be wrapped in a cap_XXX condition, so that only code that's merged in the requisite Dyninst features will try to use them in the test.

Each of dyninst, testsuite, and Tests should have a $x.PASSED and a $x.FAILED file. Clean out log and temp files from the tests directory before restarting test run. Clean out build directory before restarting a build.

https://bottle.cs.wisc.edu/results/CRASHED/350

It was killed off by 49830f6 in 2011.

Action
Configure testsuite and run make.

Expected Result
Build completes

Actual Result
Build fails with

fatal error: tbb/concurrent_vector.h: No such file or directory
#include <tbb/concurrent_vector.h>

Dyninst builds fine, so we likely just need to point to the ${DYNINST_ROOT}/cmake/FindTBB.cmake

The current version of testsuite no longer compiles on PPC64LE. I am seeing the following error:

$ make
[  0%] Built target testB_static
[  0%] Built target testB
[  0%] Built target testA_static
[  0%] Building CXX object CMakeFiles/testSuite.dir/src/JUnitOutputDriver.cpp.o
/g/g0/martymcf/devel/src/testsuite_9.2/src/JUnitOutputDriver.cpp: In constructor 'JUnitOutputDriver::JUnitOutputDriver(void*)':
/g/g0/martymcf/devel/src/testsuite_9.2/src/JUnitOutputDriver.cpp:13:50: error: 'getpid' was not declared in this scope
     results_log_name << "test_results" << getpid() << ".xml";
                                                  ^
make[2]: *** [CMakeFiles/testSuite.dir/src/JUnitOutputDriver.cpp.o] Error 1
make[1]: *** [CMakeFiles/testSuite.dir/all] Error 2
make: *** [all] Error 2

On arm64, verifying type my_union fails leaving messages below:

test_type_info.C[496]:  bad sizes for expected fields for type my_union
test_type_info.C[497]:  got 0, expected 2
test_type_info.C[590]:  verify union my_union failing
test_type_info.C[757]:  could not verify union
test_type_info.C[1025]:  specific type test failed... 

On arm64, is this expected still to be checked?

The test suite fails to build is gcc 11 is used due to a new requirement of C++17 that comparison function object's operator() be a const method.

hi,

I wonder if I can exec this testsuite without connctiong to github, without doing some PR jobs? I just want to check whether the dyninst is ok after building. Is this testcase suitable for me for that purpose?

sorry, English is my non mother language so I maybe misunderstood the README.
can somebody tell me ? thanks!

Just that! It's been deprecated.

test_instruction_read_writ,"",none,64,disk,NA,dynamic,nonPIC,FAILED

@wrwilliams @ssunny7
I'm on the master of testsuite and here's the CMake output

-- Checking for C++11 flags for GNU
-- Disabling Boost's own CMake--known buggy in many cases
-- Boost version: 1.60.0
-- Dyninst includes: /usr/local/lib/cmake/Dyninst/../../../include
-- Project source dir: /home/itaru.kitayama/projects/testsuite
-- Mutatee gcc: /usr/bin/gcc
-- Mutatee g++: /usr/bin/g++
-- Mutatee icc: M_intel_cc-NOTFOUND
-- Mutatee icpc: M_intel_CC-NOTFOUND
-- Mutatee pgcc: M_pg_cc-NOTFOUND
-- Mutatee pgCC: M_pg_cxx-NOTFOUND
-- Mutatee mpicc: /usr/lib64/openmpi/bin/mpicc
-- Mutatee mpicxx: /usr/lib64/openmpi/bin/mpicxx
-- Mutatee mpixlc: M_mpi_xlc-NOTFOUND
-- Mutatee mpixlcxx: M_mpi_xlcxx-NOTFOUND
-- Configuring done
-- Generating done
-- Build files have been written to: /home/itaru.kitayama/projects/testsuite

dyninst/dyninst#610 introduces better checking and enforcement of the C and C++ standards being used. In the Testsuite, #71, this causes build failures because we are using non-standard compiler extensions for reader-writer locks (first error was in src/mutatee_driver.c:64). This means that the Testsuite cannot be compiled without the gnu extensions enabled. Eventually, this will be a problem as we test Dyninst under more diverse platforms under CI.

In addition to cloning the testsuite as part of the dyninst travis build and running it against the just-built dyninst, would it make sense to have a travis build of the testsuite repo that clones dyninst from head and/or a known-good tag and runs the testsuite against it to catch errors introduced by changes to the testsuite?

The testsuite currently disables pie executables (even if the default executable type by the compiler). This was done to prevent Dyninst failures due to its incorrect handling of PIE. The testsuite should be enhanced to test non-PIC, PIC but non-PIE, and PIE executable types and in the future static PIE executables.

Tests run on ARM:

but not x86

None

but not Power

test_instruction_profile
test_instruction_read_write

Tests run on x86:

but not ARM

Amd64_7_arg_call
fucompp
init_fini_callback
mov_size_details
pc_hw_breakpoint
power_cft
power_decode
test_exception
test_instruction_bind_eval
test_instruction_farcall
test_write_param

but not Power

amd64_7_arg_call
fucompp
mov_size_details
pc_hw_breakpoint
test_exception
test_instruction_bind_eval
test_instruction_farcall
test_instruction_profile
test_instruction_read_writ
test_write_param

Tests run on Power:

but not ARM

init_fini_callback
power_cft
power_decode

but not x86

None

X86-only

Amd64_7_arg_call
fucompp         
mov_size_details
pc_hw_breakpoint
test_exception  
test_instruction
test_instruction
test_write_param

ARM-only

None

Power-only

None

I am getting the following build issue when I build dyninst (att_syntax_formerge branch) and testsuite (master branch):

[ 16%] Building CXX object CMakeFiles/aarch64_decode_ldst.dir/src/instruction/aarch64_decode_ldst.C.o
/home/d/detter/Workspace/dyninst/testsuite/src/instruction/aarch64_decode_ldst.C: In member function ‘virtual test_results_t aarch64_decode_ldst_Mutator::executeTest()’:
/home/d/detter/Workspace/dyninst/testsuite/src/instruction/aarch64_decode_ldst.C:529:40: error: ‘zr’ is not a member of ‘Dyninst::aarch64’

When rebuilding the testsuite after the 9.2.X update, I'm getting:

 CMake Error at /usr/lib/cmake/Dyninst/dynC_APITargets.cmake:61 (message):
   The imported target "dynC_API" references the file

      "/usr/lib/libdynC_API.so.9.1.0"

   but this file does not exist.  Possible reasons include:

   * The file was deleted, renamed, or moved to another location.

   * An install or uninstall procedure did not complete successfully.

   * The installation package was faulty and contained

      "/usr/lib/cmake/Dyninst/dynC_APITargets.cmake"

   but not all the files it references.

Does the testsuite need to be updated or is this a local issue?

Serialization was removed by dyninst/dyninst#805, so remove the corresponding (tests test_ser_anno, test_symtab_ser_funcs, test_symtab_ser_funcs_mutatee, and test_ser_anno_mutatee).

closes #22

Action
Provide no value for INSTALL_DIR.

Expected Result
LIBRARY DESTINATION provided to install is CMAKE_INSTALL_PREFIX.

Actual Result

CMake errors out with

CMake Error at CMakeLists.txt:190 (install):
install TARGETS given no LIBRARY DESTINATION for shared library target "testdyninst".

plus more for each use of 'LIBRARY DESTINATION'.

1. Parameters without names are ignored (bug!)
2. Reference parameters have an invalid name string

Must be done after dyninst/dyninst#563.

Currently, we are selecting the system Boost even when Dyninst built Boost from source.

@wrwilliams I'd like to regenerate or redefine an instruction test group on arm64. Executing this:

make -f Makefile.genspec aarch64-unknown-linux

seems to give me a proper set of directories and files, though is this all you do whenever
a new test has been added and editing files under the src/specification?

A regression was found in dyninst/dyninst#146 that attaching to a process without specifying the executable was broken. The documentation for Process::attachProcess does state that the executable is optional. AFAICS this is never tested anywhere.

It's not clear to me the best way to add this. Currently, all tests can run in either create or attach mode. It seems undesirable to add an entire test matrix of attach-no-exe. Is there a good way to write a targeted attach test for this?

It should run on ARM because we have cross-platform instrumentation (e.g., it is run on Power). Also, the InstructionDecoder is using 32-bit x86, but should be using (at least) 64-bit.

test_instruction_read_writ should also be run on ARM.

Building commit e5ec233. The main error is

relocation R_X86_64_32 against `.text' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: final link failed: Nonrepresentable section on output

See gist for output details, ld, and gcc versions. This is the first time I've had the build completely fail. This my first time building on 'cayenne.cs.wisc.edu', though.

Check that all tests are executing in at least 64-bit mode.

https://bottle.cs.wisc.edu/regressions?id=25

All but one is a static rewriter issue.

power_decode none 64 disk NA dynamic nonPIC FAILED (Running Test)
power_decode none 32 disk NA dynamic nonPIC FAILED (Running Test)

I ran the tests using "./runTests -all -log all.log".

@sashanicolas could you take a look?

test3_6 doesn't clean up after itself, and leaves a bunch of suspended processes lying around. Some have to be killed with root (on my local machine, anyway).

Currently, the Testsuite relies on LD_LIBRARY_PATH to find the correct third-party libraries (if built from source). This can be fixed with rpaths. However, using global rpaths introduces regressions, so a target-specific approach is needed. This is not a critical change as LD_LIBRARY_PATH works, but is inconvenient.

When trying to build master I get the following build error:

[  4%] Building CXX object CMakeFiles/testproccontrol.dir/src/proccontrol/proccontrol_comp.C.o
/home/detter/prefix/src/testsuite/src/proccontrol/proccontrol_comp.C: In member function ‘uint64_t ProcControlComponent::adjustFunctionEntryAddress(Dyninst::ProcControlAPI::Process::const_ptr, uint64_t)’:
/home/detter/prefix/src/testsuite/src/proccontrol/proccontrol_comp.C:343:16: error: ‘class Dyninst::SymReader’ has no member named ‘getABIVersion’
       if (rdr->getABIVersion(major, minor)) {
                ^
CMakeFiles/testproccontrol.dir/build.make:62: recipe for target 'CMakeFiles/testproccontrol.dir/src/proccontrol/proccontrol_comp.C.o' failed
make[2]: *** [CMakeFiles/testproccontrol.dir/src/proccontrol/proccontrol_comp.C.o] Error 1
CMakeFiles/Makefile2:6211: recipe for target 'CMakeFiles/testproccontrol.dir/all' failed
make[1]: *** [CMakeFiles/testproccontrol.dir/all] Error 2
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2

I have dyninst installed at /usr/local and the cmake file is configured with the dyninst cmake information directory.

@wrwilliams Is this a testsuite issue or an issue with my dyninst install?

gethostbyname* is considered obsolete. From man gethostbyname2
DESCRIPTION
The gethostbyname*(), gethostbyaddr*(), herror(), and hstrerror() functions are obsolete. Applications should use getaddrinfo(3), getnameinfo(3), and gai_strerror(3) instead.

(The linux build process flags this)

This is a not well tested patch that converts from gethostbyname2 to getaddrinfo.

testsuite-ghbn.txt

Add a test to test this:
dyninst/dyninst#668

Tested with Fedora 25's DynInst packages under Docker on x86.

https://gist.github.com/ikitayama/b265f7aa02d6107447e634261985b371

@cuviper it is great that Fedora maintains and updates the set of packages, as
it'd be always a reference for other architectures. Should I file this bug at
the Bugzilla as well?

We are compiling on 1.68 fine, but need dyninst/dyninst#541 to compile with 1.69

When ThreadDB is not compiled into Dyninst, we need to check if the thread tests were skipped to prevent waiting on non-existent threads and causing a hang. See the discussion at dyninst/dyninst#658 (comment) for details.

When a test in a group fails, the entire group fails. Dyninst does not yet implement static linking
on arm64 / aarch64... but was running those tests. Due to this combination, we could see many
test failures, depending on when the static linking tests were executed.

runTest always hangs on ARM. Running the tests individually using the --single-stepping flag in the build script works fine.

Observed on arm64, this prevents from executing all the tests under the src/symtab directory
using the -symtab option of the test_driver script.

$ ./test_driver -test test_symtab_ser_funcs
test_driver: /home/itaru.kitayama/projects/testsuite/src/test_lib.C:502: bool shouldRunTest(RunGroup*, TestInfo*): Assertion `test->results[i] == UNKNOWN || test->results[i] == PASSED' failed.
Aborted (core dumped)