Dataracebench test cases:
DRB005-indirectaccess1-orig-yes.exe, DRB006-indirectaccess2-orig-yes.exe DRB007-indirectaccess3-orig-yes.exe, DRB008-indirectaccess4-orig-yes.exe

int main (int argc, char* argv[])
{
  // max index value is 2013. +12 to obtain a valid xa2[idx] after xa1+12.
  // +1 to ensure a reference like base[2015] is within the bound.
  double * base = (double*) malloc(sizeof(double)* (2013+12+1));
  if (base == 0)
  {
    printf ("Error in malloc(). Aborting ...\n");
    return 1;  
  }

  double * xa1 = base;
  double * xa2 = xa1 + 12;
  int i;

  // initialize segments touched by indexSet
  for (i =521; i<= 2025; ++i)
  {
    base[i]=0.5*i;
  }
// default static even scheduling may not trigger data race, using static,1 instead.
#pragma omp parallel for schedule(static,1)
  for (i =0; i< N; ++i) 
  {
    int idx = indexSet[i];
    xa1[idx]+= 1.0 + i;
    xa2[idx]+= 3.0 + i;
  }

  printf("x1[999]=%f xa2[1285]=%f\n", xa1[999], xa2[1285]);
  free (base);
  return  0;
}

This crash only occurs on a server with a skylake processor:
OS: Windows 10 Enterprise, 10.0.17134
CPU: Intel Xeon Silver 4116
RAM: 80 GB

on a similar system with Kaby Lake Refresh arch, the bug does not happen.
OS: Windows 10 Enterprise, 10.0.17134
CPU: Intel Core i5-8350U
RAM: 16 GB

# RetAddr           : Args to Child                                                           : Call Site
00 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x0

The crash is reproducible using this application and happens here:
https://github.com/siemens/drace/blob/master/test/mini-apps/cs-sync/main.cs#L115

The crash is not directly related to DRace, but might be an DR issue. Running the wrap sample client also crashes:

C:\opt\DynamoRIO-Windows-7.91.18219-0\bin64\drrun.exe -c C:\opt\DynamoRIO-Windows-7.91.18219-0\samples\bin64\wrap.dll -- gp-cs-sync-clr.exe monitor

Workaround: comment out the line containing HeapAlloc in drace.ini

Probably related issues:

• DynamoRIO/dynamorio#2666 (mentioned assertion fires)

Many OpenMP functions internally map to the windows sync api, which is fine. However, some are implemented in VCOMPxxx.dll directly. These have to be intercepted and mapped to happens-before. This especially applies for the barriers at the end of each parallel region.

To debug a detector using drace-test.exe or drrun.exe with the drace-client.dll the sources of the detector must be mentioned in the according cmake files.

Is it okay to just mention them statically in the cmake file (like shown in the pictures)

(CMakeLists of drace-test)

(CMake of drace-client)

Add Jobs to the CI-Pipeline which build and execute the unit tests of the cross platform tools also under linux.

We observe sporadic crashes in large managed applications. When running in debug mode, the crashes happen more often and mostly after loading System.Linq.Expressions.dll.

<Application C:\Program Files\dotnet\dotnet.exe (30488).  Internal Error: DynamoRIO debug check failure: ..\..\core\translate.c:948 false
(Error occurred @245994 frags)
version 7.91.18137, custom build
-no_dynamic_options -client_lib 'C:\Users\felix\CMakeBuilds\a40b8e65-abeb-fe34-9f21-0ec53bca8898\build\x64-Debug-TSAN\drace-client\drace-client.dll;0;"-c" "C:\Users\felix\source\repos\drace\drace.ini" "-d" "dummy" "-i" "0"' -code_api -probe_api -stack_size 56K -max_elide_jmp 0 -max_elide_call 0 -no_inline_ignored_s
0x00000001e597ae20 0x000001f42a426b60

On Linux, module names look like libc.so.2.6 or libfoo.1.0.0.so. To reliably exclude these modules using the config file (or to add custom logic like function interception), we need wildcard matching support.

For example, libc* could be use to match libc.so.x.

We have to port the function interception and wrapping logic to Linux. This includes the following steps:

• add sync function names to the corresponding section in drace.ini (or drace_linux.ini)
• implement symbol lookup for non-exported symbols on linux

The thread ids reported by tsan on a data race should match the thread ids of DynamoRIO.

C:\Users\Z0040SWB\source\repos\drace\test\src\DetectorTest.cpp(190): error: Expected equality of these values:
a1.thread_id + a2.thread_id
Which is: 2657508032
90 + 91
Which is: 181

Unit test added in 3b68707

Chrome warns the user that the drace binary release is a "unsafe" download. We should find a way to avoid this message if possible. Otherwise we should document it in the readme.

We further should make our build process fully transparent and repeatable which implies to generate a manifest with used VS version + compiler + OS version and checksums.

DynamoRIO 7.91.271 (with client) crashes in non-debug mode during startup on linux i386. This applies to all dynamorio clients, hence it must be a DR bug. In debug mode, DRace runs successfully.

DynamoRIO internal crash at PC 0x00000035.  Please report this at http://dynamorio.org/issues/.  Program aborted.
Received SIGSEGV at unknown pc 0x00000035 in thread 78868
Base: 0xf7d5f000
Registers:eax=0x00000000 ebx=0xfff18e14 ecx=0x00000035 edx=0xf78a5a40
        esi=0xf7f5c09c edi=0xfff18e08 esp=0xfff18dec ebp=0xfff18e0e
        eflags=0x00010286
version 7.91.18271, custom build

When running on managed (CLR) applications, DRace crashes in the thread_exit event. This is possibly related to printing the per-thread statistics.

Some sections of the documentation (Readme, Contributing, ...) are a bit outdated.

• Readme: how to run tests
• Standalone: Supported environments

... list will be continued

some names slightly changed:

• copy draceGUI.py to ./bin
• copy resources to ./bin/..

The inline instrumentation has a serious performance impact on managed applications.
A possible mitigation is to move more code to the code cache.

• investigate which parts can be moved to cache
• find benchmark application
• implement proof of concept

Currently, we use std::system to trigger an execution of DR + DRace + MSR in the integration tests. This works as long as the test does not crash. Otherwise, the MSR does not terminate.

With std::system we cannot send a signal to a process to force termination, as we do not get the PID. With boost::process this is possible in a cross-platform way.

As stated in #11, using TSAN imposes multiple limitations and technical issues on the detection backend. This is mainly due to the direct-address mapping strategy, implemented in TSAN.

In the thesis, I proposed to also evaluate other race-detection backends like Fasttrack2. This was not possible due to the limited time, as well as due to the focus on the instrumentation part.
However, DRace is already prepared for this scenario as the detector is connected using a generic interface.

For Fasttrack2, there currently exists only an OSS implementation for Java, which would have to be ported to C++.
Fortunately this code is well documented and the logic behind is described in this paper, so a port should not be too time consuming.

When debugging the drace-test.exe with fasttrack or tsan, gp-concurrent-inc throws an exception in exe_common.inl just after drintegration.h issued the drrun command.

To get the exception. Run a with TSAN or FASTTRACK compiled drace-test.exe with the vs-debugger and wait until the first integration test has started

Create a new binary release, including the fasttrack component

Tests on large managed applications indicate that the shadow stack becomes a performance bottleneck.

Tasks:

• profile DRace with Windows Performance Toolkit
• check if fast-path can be implemented in shadow stack
• use faster hash-maps
  • early tests show that this is not trivial due to DR limitations
  • give DR tables a try

A high-performant tree-like data structure is needed for the tracking of the stack traces of the threads in fasttrack.

Debugging has shown that using std::stringstream leads to sporadic crashes of DRace. We observed these crashes mainly on managed applications.

We use the stringstream as a convenient way to format things, but this should be easily replacable by C++ strings or better C strings.

Debug

The crash happens at the following address:

drace_client!__acrt_release_locale_ref+0x11 [minkernel\crts\ucrt\src\appcrt\locale\locale_refcounting.cpp @ 76]:
00007ff7`8c0103a1 f044014910      lock add dword ptr [rcx+10h],r9d
00007ff7`8c0103a6 488b81e0000000  mov     rax,qword ptr [rcx+0E0h]
00007ff7`8c0103ad 4885c0          test    rax,rax
00007ff7`8c0103b0 7404            je      drace_client!__acrt_release_locale_ref+0x26 (00007ff7`8c0103b6)
00007ff7`8c0103b2 f0440108        lock add dword ptr [rax],r9d
00007ff7`8c0103b6 488b81f0000000  mov     rax,qword ptr [rcx+0F0h]
00007ff7`8c0103bd 4885c0          test    rax,rax
00007ff7`8c0103c0 7404            je      drace_client!__acrt_release_locale_ref+0x36 (00007ff7`8c0103c6)

Application paths which contain spaces are currently not correctly processed.

Currently, memory that is manipulated using these functions is either out-of-scope, or leads to many false-positives. To counter this, it's better to not instrument the implementation of these functions, but to intercept them and feed the processed data directly into the detector.

Be aware, just wrapping the exported symbols in ucrtbase, kernel, etc. is not sufficient, as some parts are directly inlined into the application, or aliased, or dispatched to specific implementations.

See also the function interception in DrMemory: https://github.com/DynamoRIO/drmemory/blob/d261a4dc254016355f64ebf5eff9187dccb34eb2/drmemory/replace.c#L955

• extctrl flag is missing
• msr must be started in an own powershell instance

Early tests on large C# (CLR) projects show a crash of the MSR after the first or a few data-races.

Callstack entries like ::``scalar deleting destructor' break the code preview as this leads to a javascript crash.

Possible solution: escape all callstack entries, file and directory names.

An option (e.g. checkbox, or button) should be added to the GUI to run a powershell with the MSR.
Here, we should consider, that the MSR is long-running, i.e. it is not terminated after each run. If implemented using a checkbox, we could set the --once flag to terminate after drace terminates.

Write a end-to-end tutorial for new DRace users. This should cover the following topics:

• Get DRace + DynamoRIO
• Non-trivial sample application with a data-race
• How to use the GUI (with screenshots)
• How to interpret the generated report
• How to fix the data-race

The tutorial can be part of either the documentation, or the github repository.

Error observed on a machine with 80GB MEM

C:\opt\DynamoRIO-Windows-7.91.18137-0\bin64\drrun.exe -debug -- C:\Temp\x64-Release-TSAN\bin\test\mini-apps\concurrent-inc\gp-concurrent-inc.exe
<Starting application C:\Temp\x64-Release-TSAN\bin\test\mini-apps\concurrent-inc\gp-concurrent-inc.exe (3536)>
<Early threads found>
<Initial options = -no_dynamic_options -code_api -probe_api -stack_size 56K -max_elide_jmp 0 -max_elide_call 0 -no_inline_ignored_syscalls -native_exec_default_list '' -no_native_exec_managed_code -no_indcall2direct -pad_jmps_mark_no_trace >
<Application C:\Temp\x64-Release-TSAN\bin\test\mini-apps\concurrent-inc\gp-concurrent-inc.exe (3536) DynamoRIO usage error : encode error: rip-relative reference out of 32-bit reach>
<Usage error: encode error: rip-relative reference out of 32-bit reach (..\..\core\arch\x86\encode.c, line 3112)
version 7.91.18137, custom build
-no_dynamic_options -code_api -probe_api -stack_size 56K -max_elide_jmp 0 -max_elide_call 0 -no_inline_ignored_syscalls -native_exec_default_list '' -no_native_exec_managed_code -no_indcall2direct -pad_jmps_mark_no_trace >

To be able to load the fasttrack-dll into the drace-tests.exe, some dynamrio dlls must be in the folder of drace-tests.exe. Otherwise all the tests are failing.

These dlls (in .../DynamoRIO/lib64):

To properly wrap .Net synchronization functions, debug information beyond exports is required. The debugging information is queried using the MSR which internally uses dbghelp.dll.

If this dll is not copied to the msr folder, only local and cached symbols are available as for requests to a MS symbol server symsrv.dll is required to reside in the same directory. With the default version shipped with windows, this is not the case.

Due to unclear redistribution policies we cannot just copy these files and ship it with DRace.

Possible solution:

• document external dependency to Debugging Tools For Windows
• add install directory to dll search path
• issue a warning if target application is managed but sync functions cannot be found

Due to the aligned-stack (which inherits from aligned buffer), the per_thread_t is not standard layout anymore. Hence, it might be unsafe to use offsetof() on it's members. For most compilers, this should be not an issue, but we should fix that anyways.

A possible solution would be to carve out the shadow-stack and put it into it's own TLS slot.

Diagnostics (both clang and gcc-8)

drace/drace-client/src/instr/instr-mem-full.cpp:169:44: warning: offsetof within non-standard-layout type ‘drace::per_thread_t’ is conditionally-supported [-Winvalid-offsetof]
  opnd2 = OPND_CREATE_MEMPTR(reg3, offsetof(per_thread_t, buf_end));

The testing code (unit + integration) should be parameterized to test all implemented detector backends.

Note: For the unit-tests, this is not trivial, as the current TSAN detector must only be initialized and finalized once per process. It is also not possible to perform a clear unload of the library.

Xref #15

Currently, only the lower 32 bit of each memory address are considered for race-detection. For the PoC this was sufficient, but for real applications the full addresses have to be analyzed.

The reason behind this decision was to avoid changes to the memory mapping table of TSAN. To properly handle all accesses, we have to make sure that all memory we want to track is inside either the EXECUTABLE region or inside a heap.

Thereto, we have to shift all heaps inside a shadowable region or alternatively change the regions in TSAN.

Currently the xml report is missing line information. Add it with a new tag <offset>.

Premises for a detector back-end (what one can assume, to be true; for adding in the documentation):

• the first appearance of a thread in the detector will be a fork
• a read or write will never contain a tid, which was not forked before
• reads can happen before writes
• ?a first lock acquire will always happen before the first release?
• happens_after may arrive before a corresponding, happens before arrives -> but no backward sync need
  (means no backward synchronisation of a happens_after needed??)
• no double forks of same thread as child
• ...

Reviews and extension to be made

On dotnet applications we observed the following instruction combination to implement a call:

push addr; ret

See also: https://groups.google.com/d/topic/dynamorio-users/8cPY5B7FZ74/discussion

Which data structures to use for caching the threads, variables, stack traces and so on ...

To port the MSR to Linux, the shared-memory system has to be ported as well. After that, remove the #ifdef WINDOWS around the calls to the MSR.

• Load/Save of a configuration
• open html report after creation
• possibility to build gui without boost (Note: boost is now mandatory to be able to build the gui app)

change string return values in const char *

see here:
https://stackoverflow.com/questions/4446620/returning-strings-from-dll-functions

• Implement a custom pool allocator for global allocations in fasttrack.
• Implement a thread local allocator for thread local allocations (thread data, shadow stacks)

https://code.siemens.com/multicore/drace/blob/master/drace-client/src/drace-client.cpp#L217

When the detector-dll is unloaded no destructor of the class is called, but just the finalize function. Therefore all allocated members must be destructed manually in the finalize function.

During workshops, I discovered that it is quite hard for first-time users of DRace to get the tool up and running. This includes the following problem sites:

• Paths: DynamoRIO, DRace, config file, application
• Parameters: difference between parameters for DynamoRIO, DRace and the application
• Workflow: To get an HTML report, the report generator python script has to be executed manually

Further pitfalls:

• Naming: DRaceGUI.exe is not a GUI itself, but converts the xml report to a HTML version
• Dependencies: Our prebuild version should not depend on python. Some early tests with PyInstaller look promising. We should auto-build this in CI as well and include it in the bundle.

Possible solution

• Config Path: Most people do not need to change the config. Hence, load it from the drace-client binary dir if it's not found in current relative path.
• Naming: Just rename the report generator to e.g. ReportConverter, also provide hints on the CLI if the tool is not correctly used (e.g. no or wrong parameters, or '-h')
• Dependencies: Build exe in CI and include in bundle (either along with python script, or just the exe).
• Workflow & Paths: Implement Qt5 based GUI where all paths and options can be specified. Then, the full command can be displayed and we could also directly start the application from the GUI.

Please show a warning-message if the reportConverter.exe is not available (e.g. in prebuild packages) and python is also not found in the path.