Base tests fail under windows with -O3 about sgpp HOT 12 CLOSED

In GitLab by @valentjn on Mar 23, 2017, 01:20

@fabianfranzelin told me the default changed from OPT=0 (master) to OPT=1 (codingdays). And Jenkins only tests the default value. That's why we didn't see this bug before.

I'm not able to reproduce the bug when I extract the TestBsplineClenshawCurtisBasis function to a separate standalone source file and compile without Boost. Argh 😒

from sgpp.

In GitLab by @valentjn on Mar 23, 2017, 01:36

I'm getting weird backtraces with gdb. E.g., it now claims that p = 1875934688 in TestBsplineClenshawCurtisBasis::test_method and boundTest (lowerBound=2.9643938750474793e-323, upperBound=9.2688104077161441e-315, i=512, l=<optimized out>, basis=...)--these p, lowerBound, and upperBound values are garbage. Somewhere, we write into memory where we should not write to...

from sgpp.

In GitLab by @valentjn on Mar 23, 2017, 04:49

I'm genuinely lost on this one. I installed the trial version of a 3rd party memory checker (https://www.softwareverify.com/cpp-memory.php), and even managed to get it to run on the crashing Boost test executable, to no avail--it displays lots of memory leaks, but no errors. It may well be that it searches only specific memory areas and not the ones where the stuff happens.

I'm not going to lie: Part of the problem is that we formally "support" a platform for which, at least currently, we simply lack the means, that is tools* and people who know the platform, to support it properly. It's very difficult to support an OS that nobody of the developers uses. This leads to cases like this in which none of the other developers (besides Fabian and David) feels like helping to investigate, because they are not affected (who cares about Windows, right?)...

(* in this case: valgrind or similar)

from sgpp.

In GitLab by @valentjn on Mar 23, 2017, 04:53

Tried another memory checker (http://www.drmemory.org/), doesn't find anything either. The first message is about reading from unaccessible memory in BsplineClenshawCurtisBasis::nonUniformBSpline... But if it's a memory error, the memory must've been corrupted before.

C:\Users\valentjn\Desktop\sgpp\base\tests>"C:\Program Files (x86)\Dr. Memory\bin\drmemory.exe" -- test_base_boost.exe --log_level=test_suite --run_test=TestBsplineClenshawCurtisBasis
~~Dr.M~~ Dr. Memory version 1.11.0
~~Dr.M~~ (Uninitialized read checking is not yet supported for 64-bit)
~~Dr.M~~ Running "test_base_boost.exe --log_level=test_suite --run_test=TestBsplineClenshawCurtisBasis"
Running 1 test case...
Entering test module "SGppBaseModule"
base\tests\test_algorithms.cpp(327): Entering test case "TestBsplineClenshawCurtisBasis"
BOOM2: p = 2
BOOM2: p = 3
~~Dr.M~~
~~Dr.M~~ Error #1: UNADDRESSABLE ACCESS: reading 0x0000000000000008-0x0000000000000010 8 byte(s)
~~Dr.M~~ # 0 TestBsplineClenshawCurtisBasis::test_method
        [base/src/sgpp/base/operation/hash/common/basis/BsplineClenshawCurtisBasis.hpp:72]
~~Dr.M~~ # 1 TestBsplineClenshawCurtisBasis_invoker
        [base/tests/test_algorithms.cpp:327]
~~Dr.M~~ # 2 libboost_unit_test_framework-mgw!boost::detail::function::function_obj_invoker0<>::invoke [../../../../../src/gcc-5.2.0/libgcc/config/i386/cygwin.S:158]
~~Dr.M~~ # 3 libboost_unit_test_framework-mgw!boost::debug::under_debugger
~~Dr.M~~ # 4 libboost_unit_test_framework-mgw!boost::execution_monitor::execute
~~Dr.M~~ # 5 libstdc++-6.dll!?
       +0x0      (0x000000006fcadcba <libstdc++-6.dll+0x6dcba>)
~~Dr.M~~ # 6 libstdc++-6.dll!?
       +0x0      (0x000000006fcade22 <libstdc++-6.dll+0x6de22>)
~~Dr.M~~ # 7 libboost_unit_test_framework-mgw!std::_Rb_tree<>::_M_insert_unique_<>
~~Dr.M~~ # 8 libboost_unit_test_framework-mgw!boost::unit_test::unit_test_monitor_t::execute_and_translate
~~Dr.M~~ # 9 libboost_unit_test_framework-mgw!boost::unit_test::results_collector_t::test_unit_start
~~Dr.M~~ #10 libboost_unit_test_framework-mgw!boost::unit_test::unit_test_log_t::test_unit_start
~~Dr.M~~ #11 libboost_unit_test_framework-mgw!boost::unit_test::framework::state::execute_test_tree [../../../../../src/gcc-5.2.0/libgcc/config/i386/cygwin.S:158]
~~Dr.M~~ #12 libboost_unit_test_framework-mgw!boost::runtime::arguments_store::get<>    [../../../../../src/gcc-5.2.0/libgcc/config/i386/cygwin.S:158]
~~Dr.M~~ #13 libstdc++-6.dll!?
       +0x0      (0x000000006fcade22 <libstdc++-6.dll+0x6de22>)
~~Dr.M~~ #14 libboost_unit_test_framework-mgw!boost::unit_test::output::compiler_log_formatter::test_unit_start
~~Dr.M~~ #15 libboost_unit_test_framework-mgw!std::_Rb_tree<>::_M_insert_unique_<>
~~Dr.M~~ #16 libboost_unit_test_framework-mgw!boost::unit_test::framework::run
~~Dr.M~~ #17 libboost_unit_test_framework-mgw!boost::debug::under_debugger
~~Dr.M~~ #18 libboost_unit_test_framework-mgw!std::_Rb_tree<>::_M_erase
        [../../../../../src/gcc-5.2.0/libgcc/config/i386/cygwin.S:158]
~~Dr.M~~ #19 libboost_unit_test_framework-mgw!boost::unit_test::framework::finalize_setup_phase
~~Dr.M~~ Note: @0:00:00.904 in thread 315212
~~Dr.M~~ Note: instruction: movsd  0x08(%r11) -> %xmm13

from sgpp.

In GitLab by @valentjn on Mar 23, 2017, 05:26

I replaced, just for fun,

      return (x - xi[k]) / (xi[k + p] - xi[k])
             * nonUniformBSpline(x, p - 1, k)
             + (1.0 - (x - xi[k + 1]) / (xi[k + p + 1] - xi[k + 1]))
             * nonUniformBSpline(x, p - 1, k + 1);

in BsplineClenshawCurtisBasis::nonUniformBSpline with

      const double a = xi[k];
      const double b = xi[k + p];
      const double c = xi[k + 1];
      const double d = xi[k + p + 1];
      const double e = nonUniformBSpline(x, p - 1, k);
      const double f = nonUniformBSpline(x, p - 1, k + 1);
      return (x - a) / (b - a)
             * e
             + (1.0 - (x - c) / (d - c))
             * f;

At first, it worked, but only for my debug version, where there are lots of changes (reduced loops etc.). For the original codingdays branch, this doesn't seem to work.

I've attached the assembler generated for that function for the top (asm1.txt) and the bottom (asm2.txt) version. Maybe someone with basic assembler knowledge look if there's a compiler error in there? (That's how desperate I am. I got no better idea at the moment.)

• asm1.txt
• asm2.txt

from sgpp.

In GitLab by @valentjn on Mar 23, 2017, 05:33

Oh, and I checked all the xi assignments in the constructKnots in the same class with assert statements, whether they lie between 0 and xi.size() - 1. Needless to say, no assertion errors, and it didn't even crash. This bug is highly non-reproducible.

from sgpp.

In GitLab by @fabianfranzelin on Mar 23, 2017, 11:20

As an experienced Windows user ;) I do what Windows users usually to: Install the newest versions of boost (1.63) and mingw (6.3.0, build 0). After a first try I got no error. Everything alright? No, of course not, its Windows. Now the python interface does compile anymore! It crashes with

g++ -o .sconf_temp\conftest_3.o -c -O3 -g -Wall -pedantic -Wextra -Wcast-qual -Wconversion -Wformat=2 -Wformat-nonliteral -Wformat-security -Winit-self -Wmissing-format-attribute -Wmissing-include-dirs -Wpacked -Wredundant-decls -Wswitch-default -Wswitch-enum -Wunreachable-code -Wunused -Wno-unused-parameter -fno-strict-aliasing -funroll-loops -mfpmath=sse -fopenmp -fno-omit-frame-pointer -mavx -Wno-switch-enum -Wno-deprecated-declarations -D__NO_INLINE__ -std=c++11 -Ibase\src -IC:\local\boost_1_63_0 -IC:\local\Python27_amd64\include .sconf_temp\conftest_3.cpp
In file included from C:/local/mingw-w64/x86_64-6.3.0-posix-seh-rt_v5-rev0/mingw64/lib/gcc/x86_64-w64-mingw32/6.3.0/include/c++/math.h:36:0,
from C:\local\Python27_amd64\include/pyport.h:325,
from C:\local\Python27_amd64\include/Python.h:58,
from .sconf_temp\conftest_3.cpp:2:
C:/local/mingw-w64/x86_64-6.3.0-posix-seh-rt_v5-rev0/mingw64/lib/gcc/x86_64-w64-mingw32/6.3.0/include/c++/cmath:1157:11: error: '::hypot' has not been declared
using ::hypot;
^~~~~
This seems to be a known issue see

http://stackoverflow.com/questions/10660524/error-building-boost-1-49-0-with-gcc-4-7-0

As suggested in this thread I added "-D_hypot=hypot" to the compiler flags, and the interface is building again.

I updated the software on winsgpp and pushed the changes to the build scripts. The builds are running right now...

from sgpp.

In GitLab by @valentjn on Mar 23, 2017, 11:45

Actually a good idea. Great news! :man_dancing_tone1:

Let's just hope it really was a compiler bug that was fixed with the MinGW upgrade. The other option is that the bug is still lurking around, not triggered at the moment (non-reproducible), just waiting around the corner when everything seems fine...

from sgpp.

In GitLab by @fabianfranzelin on Mar 23, 2017, 13:15

I would say, let's hope the best and close this issue 😄

from sgpp.

In GitLab by @fabianfranzelin on Mar 23, 2017, 13:15

closed

from sgpp.

In GitLab by @valentjn on May 1, 2017, 09:27

Note for the future: Don't forget to change the configuration of the Jenkins jobs to use the different paths for the new MinGW and Boost versions.

from sgpp.

In GitLab by @valentjn on May 1, 2017, 09:54

mentioned in merge request !16

from sgpp.