Git Product home page Git Product logo

junction's Introduction

Junction is a library of concurrent data structures in C++. It contains several hash map implementations:

junction::ConcurrentMap_Crude
junction::ConcurrentMap_Linear
junction::ConcurrentMap_Leapfrog
junction::ConcurrentMap_Grampa

CMake and Turf are required. See the blog post New Concurrent Hash Maps for C++ for more information.

License

Junction uses the Simplified BSD License. You can use the source code freely in any project, including commercial applications, as long as you give credit by publishing the contents of the LICENSE file in your documentation somewhere.

Getting Started

If you just want to get the code and look around, start by cloning Junction and Turf into adjacent folders, then run CMake on Junction's CMakeLists.txt. You'll want to pass different arguments to cmake depending on your platform and IDE.

$ git clone https://github.com/preshing/junction.git
$ git clone https://github.com/preshing/turf.git
$ cd junction
$ mkdir build
$ cd build
$ cmake <additional options> ..

On Unix-like environments, cmake will generate a Makefile by default. On Windows, it will create a Visual Studio solution. To use a specific version of Visual Studio:

$ cmake -G "Visual Studio 14 2015" ..

To generate an Xcode project on OS X:

$ cmake -G "Xcode" ..

To generate an Xcode project for iOS:

$ cmake -G "Xcode" -DCMAKE_TOOLCHAIN_FILE=../../turf/cmake/toolchains/iOS.cmake ..

The generated build system will contain separate targets for Junction, Turf, and some sample applications.

Solution Explorer

Alternatively, you can run CMake on a specific sample only:

$ cd junction/samples/MapCorrectnessTests
$ mkdir build
$ cd build
$ cmake <additional options> ..

Adding Junction to Your Project

There are several ways to add Junction to your own C++ project.

  1. Add Junction as a build target in an existing CMake-based project.
  2. Use CMake to build Junction and Turf, then link the static libraries into your own project.
  3. Grab only the source files you need from Junction, copy them to your project and hack them until they build correctly.

Some developers will prefer approach #3, but I encourage you to try approach #1 or #2 instead. It will be easier to grab future updates that way. There are plenty of files in Junction (and Turf) that you don't really need, but it won't hurt to keep them on your hard drive either. And if you link Junction statically, the linker will exclude the parts that aren't used.

Adding to an Existing CMake Project

If your project is already based on CMake, clone the Junction and Turf source trees somewhere, then call add_subdirectory on Junction's root folder from your own CMake script. This will add both Junction and Turf targets to your build system.

For a simple example, see the junction-sample repository.

Building the Libraries Separately

Generate Junction's build system using the steps described in the Getting Started section, then use it to build the libraries you need. Add these to your own build system. Make sure to generate static libraries to avoid linking parts of the library that aren't needed.

If you build the install target provided by Junction's CMake script, the build system will output a clean folder containing only the headers and libs that you need. You can add this to your own project using a single include path. Choose the output directory by specifying the CMAKE_INSTALL_PREFIX variable to CMake. Additionally, you can specify JUNCTION_WITH_SAMPLES=OFF to avoid building the samples. For example:

$ cmake -DCMAKE_INSTALL_PREFIX=~/junction-install -DJUNCTION_WITH_SAMPLES=OFF ..
$ cmake --build . --target install --config RelWithDebInfo

Notes:

  • Instead of running the second cmake command, which runs the build system, you could run your build system directly. For example, make install on Unix, or build the INSTALL project in Visual Studio.
  • If using makefiles, you'll probably want to pass the additional option -DCMAKE_BUILD_TYPE=RelWithDebInfo to the first cmake command.

This will create the following file structure:

Install folder

Configuration

When you first run CMake on Junction, Turf will detect the capabilities of your compiler and write the results to a file in the build tree named turf/include/turf_config.h. Similarly, Junction will write include/junction_config.h to the build tree. You can modify the contents of those files by setting variables when CMake runs. This can be done by passing additional options to cmake, or by using an interactive GUI such as cmake-gui or ccmake.

For example, to configure Turf to use the C++11 standard library, you can set the TURF_PREFER_CPP11 variable on the command line:

$ cmake -DTURF_PREFER_CPP11=1 ..

Or, using the CMake GUI:

CMake GUI

Many header files in Turf, and some in Junction, are configurable using preprocessor definitions. For example, turf/Thread.h will switch between turf::Thread implementations depending on the values of TURF_IMPL_THREAD_PATH and TURF_IMPL_THREAD_TYPE. If those macros are not defined, they will be set to default values based on information from the environment. You can set them directly by providing your own header file and passing it in the TURF_USERCONFIG variable when CMake runs. You can place this file anywhere; CMake will copy it to Turf's build tree right next to include/turf_config.h.

$ cmake -DTURF_USERCONFIG=path/to/custom/turf_userconfig.h.in ..

The JUNCTION_USERCONFIG variable works in a similar way. As an example, take a look at the Python script junction/samples/MapScalabilityTests/TestAllMaps.py. This script invokes cmake several times, passing a different junction_userconfig.h.in file each time. That's how it builds the same test application using different map implementations.

Rules and Behavior

Currently, Junction maps only work with keys and values that are pointers or pointer-sized integers. The hash function must be invertible, so that every key has a unique hash. Out of all possible keys, a null key must be reserved, and out of all possible values, null and redirect values must be reserved. The defaults are 0 and 1. You can override those defaults by passing custom KeyTraits and ValueTraits parameters to the template.

Every thread that manipulates a Junction map must periodically call junction::DefaultQSBR.update, as mentioned in the blog post. If not, the application will leak memory.

Otherwise, a Junction map is a lot like a big array of std::atomic<> variables, where the key is an index into the array. More precisely:

  • All of a Junction map's member functions, together with its Mutator member functions, are atomic with respect to each other, so you can safely call them from any thread without mutual exclusion.
  • If an assign happens before a get with the same key, the get will return the value it inserted, except if another operation changes the value in between. Any synchronizing operation will establish this relationship.
  • For Linear, Leapfrog and Grampa maps, assign is a release operation and get is a consume operation, so you can safely pass non-atomic information between threads using a pointer. For Crude maps, all operations are relaxed.
  • In the current version, you must not assign while concurrently using an Iterator.

Feedback

If you have any feedback on improving these steps, feel free to open an issue on GitHub, or send a direct message using the contact form on my blog.

junction's People

Contributors

mdw55189 avatar preshing avatar zuyu avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

junction's Issues

Crash in first table migration (QSBR.enqueue)

gcc 4.8.2 c++11 (g++ (GCC) 4.8.2 20140120 (Red Hat 4.8.2-15))

I declare a Leapfrog map and begin inserting into the map. After a few successful inserts, a migration is attempted and the program is faulting.

The call stack is below. If I break prior to the crash, say in the call to enqueue, the m_deferredActions._M_impl seems to have been corrupted, with _M_start = 0x1,_M_finish = 0x0, _M_end_of_storage = 0x0. My gut is telling me that the swap calls in QSBR.update(), but i haven't caught it with a watch point. Watch indicates the _M_start member alternating between 0x1 and 0x0.

It is possible that I have missed some step in getting the QSBR set up? I created a single context and call update() from the thread that is adding to the table when it is quiet. Currently I haven't enabled consumers of the data.

Callstack below:


[Switching to Thread 0x7ffff63d3700 (LWP 13122)]

0x0000000000716fef in __gnu_cxx::new_allocator<junction::QSBR::Action>::construct<junction::QSBR::Action<junction::QSBR::Action> > (

    this=0xb95578 <junction::DefaultQSBR+88>, __p=0xfffffffffffffff8)

    at /opt/rh/devtoolset-2/root/usr/include/c++/4.8.2/ext/new_allocator.h:120

120 in /opt/rh/devtoolset-2/root/usr/include/c++/4.8.2/ext/new_allocator.h

(gdb) bt

#0  0x0000000000716fef in __gnu_cxx::new_allocator<junction::QSBR::Action>::construct<junction::QSBR::Action<junction::QSBR::Action> > (

    this=0xb95578 <junction::DefaultQSBR+88>, __p=0xfffffffffffffff8)

    at /opt/rh/devtoolset-2/root/usr/include/c++/4.8.2/ext/new_allocator.h:120

#1  0x0000000000716e03 in std::allocator_traits<std::allocator<junction::QSBR::Action> >::_S_construct<junction::QSBR::Action<junction::QSBR::Action> >(std::allocator<junction::QSBR::Action>&, std::allocator_traits<std::allocator<junction::QSBR::Action> >::__construct_helper*, (junction::QSBR::Action<junction::QSBR::Action>&&)...) (__a=..., __p=0xfffffffffffffff8)

    at /opt/rh/devtoolset-2/root/usr/include/c++/4.8.2/bits/alloc_traits.h:254

#2  0x0000000000716b77 in std::allocator_traits<std::allocator<junction::QSBR::Action> >::construct<junction::QSBR::Action<junction::QSBR::Action> >(std::allocator<junction::QSBR::Action>&, junction::QSBR::Action<junction::QSBR::Action>*, (junction::QSBR::Action<junction::QSBR::Action>&&)...) (__a=..., 

    __p=0xfffffffffffffff8)

    at /opt/rh/devtoolset-2/root/usr/include/c++/4.8.2/bits/alloc_traits.h:393

#3  0x0000000000716c0f in std::vector<junction::QSBR::Action, std::allocator<junction::QSBR::Action> >::_M_emplace_back_aux<junction::QSBR::Action>(junction::QSBR::Action&&) (this=0xb95578 <junction::DefaultQSBR+88>)

    at /opt/rh/devtoolset-2/root/usr/include/c++/4.8.2/bits/vector.tcc:408

#4  0x0000000000716add in std::vector<junction::QSBR::Action, std::allocator<junction::QSBR::Action> >::emplace_back<junction::QSBR::Action>(junction::QSBR::Action&&) (this=0xb95578 <junction::DefaultQSBR+88>)

    at /opt/rh/devtoolset-2/root/usr/include/c++/4.8.2/bits/vector.tcc:101

#5  0x0000000000716996 in std::vector<junction::QSBR::Action, std::allocator<junction::QSBR::Action> >::push_back(junction::QSBR::Action&&) (

    this=0xb95578 <junction::DefaultQSBR+88>, 

    __x=<unknown type in /home/matt.weiss/poc/engine_debug, CU 0x3a0c7d, DIE 0x3f3621>)

    at /opt/rh/devtoolset-2/root/usr/include/c++/4.8.2/bits/stl_vector.h:920

#6  0x0000000000716799 in junction::QSBR::enqueue<junction::details::Leapfrog<junction::ConcurrentMap_Leapfrog<unsigned long, engine::Vehicle*, junction::DefaultKeyTraits<unsigned long>, junction::DefaultValueTraits<engine::Vehicle*> > >::TableMigration> (this=0xb95520 <junction::DefaultQSBR>, 

    pmf=(void (junction::details::Leapfrog<junction::ConcurrentMap_Leapfrog<unsigned long, engine::Vehicle*, junction::DefaultKeyTraits<unsigned long>, junction::DefaultValueTraits<engine::Vehicle*> > >::TableMigration::*)(junction::details::Leapfrog<junction::ConcurrentMap_Leapfrog<unsigned long, engine::Vehicle*, junction::DefaultKeyTraits<unsigned long>, junction::DefaultValueTraits<engine::Vehicle*> > >::TableMigration * const)) 0x7165f6 <junction::details::Leapfrog<junction::ConcurrentMap_Leapfrog<unsigned long, engine::Vehicle*, junction::DefaultKeyTraits<unsigned long>, junction::DefaultValueTraits<engine::Vehicle*> > >::TableMigration::destroy()>, target=0x7fffe0003160)

    at /home/mweiss/src/simplified_simplification/build/engine/Debug/../../Debug/include/junction/QSBR.h:80

#7  0x0000000000715e16 in junction::details::Leapfrog<junction::ConcurrentMap_Leapfrog<unsigned long, engine::Vehicle*, junction::DefaultKeyTraits<unsigned long>, junction::DefaultValueTraits<engine::Vehicle*> > >::TableMigration::run (

    this=0x7fffe0003160)

    at /home/mweiss/src/simplified_simplification/build/engine/Debug/../../Debug/include/junction/details/Leapfrog.h:568

#8  0x000000000070c705 in junction::SimpleJobCoordinator::participate (

    this=0xb99410)

    at /home/mweiss/src/simplified_simplification/build/engine/Debug/../../Debug/include/junction/SimpleJobCoordinator.h:69

#9  0x00000000007108d7 in junction::ConcurrentMap_Leapfrog<unsigned long, engine::Vehicle*, junction::DefaultKeyTraits<unsigned long>, junction::DefaultValueTraits<engine::Vehicle*> >::Mutator::Mutator (this=0x7ffff63d2b20, map=..., 

    key=3314346713558739690)

    at /home/mweiss/src/simplified_simplification/build/engine/Debug/../../Debug/include/junction/ConcurrentMap_Leapfrog.h:128

#10 0x000000000070e8c1 in junction::ConcurrentMap_Leapfrog<unsigned long, engine::Vehicle*, junction::DefaultKeyTraits<unsigned long>, junction::DefaultValueTra---Type <return> to continue, or q <return> to quit---

its<engine::Vehicle*> >::insertOrFind (this=0xb993b8, key=3314346713558739690)

    at /home/mweiss/src/simplified_simplification/build/engine/Debug/../../Debug/include/junction/ConcurrentMap_Leapfrog.h:250

#11 0x000000000070d557 in engine::Cache::impl::addToLookups (this=0xb99380, 

    veh=0x7fffe00039e0)

    at /home/mweiss/src/simplified_simplification/engine/src/Cache.cpp:174```

can't compile with g++ 4.4.7

in turf/cmake/Macros.cmake, you set(CMAKE_CXX_FLAGS "-g -std=c++11...)
this doesn't make sense. because this lib suppose to work without c++11

Iterator over entries

How do you iterate over entries. As a basic example, how would you iterator over SingleMap_Linear's entries?

compile error with gcc-5.3

error like this

[ 88%] Linking CXX executable AtomicTests
CMakeFiles/AtomicTests.dir/generated/TestExchange8.cpp.o: In function `std::thread::thread<void* (*&)(void*), void*&>(void* (*&)(void*), void*&)':
/usr/local/include/c++/5.3.0/thread:137: undefined reference to `std::thread::_M_start_thread(std::shared_ptr<std::thread::_Impl_base>, void (*)())'
../../libturf.a(Affinity_Linux.cpp.o): In function `turf::Affinity_Linux::Affinity_Linux()':
/home/chenyun/workspace/turf/turf/impl/Affinity_Linux.cpp:29: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string()'
/home/chenyun/workspace/turf/turf/impl/Affinity_Linux.cpp:30: undefined reference to `std::basic_istream<char, std::char_traits<char> >& std::getline<char, std::char_traits<char>, std::allocator<char> >(std::basic_istream<char, std::char_traits<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)'
/home/chenyun/workspace/turf/turf/impl/Affinity_Linux.cpp:31: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::find_first_of(char, unsigned long) const'
/home/chenyun/workspace/turf/turf/impl/Affinity_Linux.cpp:33: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::find_last_not_of(char const*, unsigned long) const'
/home/chenyun/workspace/turf/turf/impl/Affinity_Linux.cpp:34: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::substr(unsigned long, unsigned long) const'
/home/chenyun/workspace/turf/turf/impl/Affinity_Linux.cpp:36: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::c_str() const'
/home/chenyun/workspace/turf/turf/impl/Affinity_Linux.cpp:34: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string()'
/home/chenyun/workspace/turf/turf/impl/Affinity_Linux.cpp:29: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string()'
../../libturf.a(Affinity_Linux.cpp.o): In function `bool std::operator==<char, std::char_traits<char>, std::allocator<char> >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*)':
/usr/local/include/c++/5.3.0/bits/basic_string.h:4941: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::compare(char const*) const'
collect2: error: ld returned 1 exit status
make[2]: *** [samples/AtomicTests/AtomicTests] Error 1
make[1]: *** [samples/AtomicTests/CMakeFiles/AtomicTests.dir/all] Error 2
make: *** [all] Error 2

Coredump in turf heap free when TURF_REPLACE_OPERATOR_NEW=1

@preshing I use junction and turf static lib, TURF_REPLACE_OPERATOR_NEW is open by default, but when integrate into application, the application coredump at operator delete. Does it's need recompile all the dependent libraries for this program?

Program terminated with signal 6, Aborted.
#0  0x00007f4c6e5971f7 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.17-196.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-8.el7.x86_64 libcom_err-1.42.9-10.el7.x86_64 libgcc-4.8.5-16.el7.x86_64 libselinux-2.5-11.el7.x86_64 libstdc++-4.8.5-16.el7.x86_64 pcre-8.32-17.el7.x86_64
(gdb) bt
#0  0x00007f4c6e5971f7 in raise () from /lib64/libc.so.6
#1  0x00007f4c6e5988e8 in abort () from /lib64/libc.so.6
#2  0x00007f4c6e5d6f47 in __libc_message () from /lib64/libc.so.6
#3  0x00007f4c6e5dcb54 in malloc_printerr () from /lib64/libc.so.6
#4  0x00007f4c6e5de7aa in _int_free () from /lib64/libc.so.6
#5  0x0000000000b0d138 in turf::Heap_CRT::Operator::free (this=0x7f4c4f7fcbff, ptr=0x7f4c1000e670)
    at /home/jlh/oin/src/apex_adapter_3rd/../../include/turf/impl/Heap_CRT.h:34
#6  0x000000000115bc79 in operator delete (ptr=0x7f4c1000e670) at /home/jlh/opensource/turf/turf/Heap.cpp:33
#7  0x0000000000d08014 in biz_common::FixParam::~FixParam (this=0x7f4c1000e670, __in_chrg=<optimized out>)
    at /home/jlh/oin/proto/biz_common.pb.cc:6361
#8  0x0000000000e7c652 in biz_adapter::QueryFundMaxWithdrawRequest::SharedDtor (this=0x7f4c1000ff30)
    at /home/jlh/oin/proto/biz_adapter.pb.cc:202661
#9  0x0000000000e7c581 in biz_adapter::QueryFundMaxWithdrawRequest::~QueryFundMaxWithdrawRequest (this=0x7f4c1000ff30,
    __in_chrg=<optimized out>) at /home/jlh/oin/proto/biz_adapter.pb.cc:202654
#10 0x0000000000e7c5cc in biz_adapter::QueryFundMaxWithdrawRequest::~QueryFundMaxWithdrawRequest (this=0x7f4c1000ff30,
    __in_chrg=<optimized out>) at /home/jlh/oin/proto/biz_adapter.pb.cc:202655
#11 0x0000000000aec4ea in std::_Sp_counted_ptr<google::protobuf::Message*, (__gnu_cxx::_Lock_policy)2>::_M_dispose (this=0x7f4c1000f170)
    at /usr/include/c++/4.8.2/bits/shared_ptr_base.h:290
#12 0x0000000000aa3532 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x7f4c1000f170)
    at /usr/include/c++/4.8.2/bits/shared_ptr_base.h:144
#13 0x0000000000a86f93 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x7f4c4f7fcd28,
    __in_chrg=<optimized out>) at /usr/include/c++/4.8.2/bits/shared_ptr_base.h:546
#14 0x0000000000a804fc in std::__shared_ptr<google::protobuf::Message, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=0x7f4c4f7fcd20,
    __in_chrg=<optimized out>) at /usr/include/c++/4.8.2/bits/shared_ptr_base.h:781
#15 0x0000000000a899b3 in std::__shared_ptr<google::protobuf::Message, (__gnu_cxx::_Lock_policy)2>::reset (this=0x7f4c1000ffb0)
    at /usr/include/c++/4.8.2/bits/shared_ptr_base.h:882
#16 0x0000000000a80955 in MsgCommBase::DeInit (this=0x7f4c1000ff70)
    at /home/jlh/oin/src/apex_adapter_3rd/../../include/adapter_common/adapter_bizcomm_msg.h:75
#17 0x0000000000a81eb6 in Msg::DeInit (this=0x7f4c1000ff70) at /home/jlh/oin/src/apex_adapter_3rd/src/apex_adapter_msg.h:59
#18 0x0000000000a82042 in PoolableMsgFactory::passivate_object (this=0x2ddf5b8, msg=0x7f4c1000ff70)
    at /home/jlh/oin/src/apex_adapter_3rd/src/apex_adapter_msg.h:188
#19 0x0000000000adedca in GenericObjectPool<Msg>::return_object (this=0x2dd7f48, obj=0x7f4c1000ff70)
    at /home/jlh/oin/src/apex_adapter_3rd/../../include/adapter_common/object_pool.h:133
#20 0x0000000000adb8ed in GenericObjectPool<Msg>::Deleter::operator() (this=0x7f4c10011718, obj=0x7f4c1000ff70)
    at /home/jlh/oin/src/apex_adapter_3rd/../../include/adapter_common/object_pool.h:66
#21 0x0000000000aec015 in std::_Sp_counted_deleter<Msg*, GenericObjectPool<Msg>::Deleter, std::allocator<int>, (__gnu_cxx::_Lock_policy)2>::_M_dispose (this=0x7f4c10011700) at /usr/include/c++/4.8.2/bits/shared_ptr_base.h:347
#22 0x0000000000aa3532 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x7f4c10011700)
    at /usr/include/c++/4.8.2/bits/shared_ptr_base.h:144
#23 0x0000000000a86f93 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x46aca50, __in_chrg=<optimized out>)
    at /usr/include/c++/4.8.2/bits/shared_ptr_base.h:546
---Type <return> to continue, or q <return> to quit---
#24 0x0000000000a83c92 in std::__shared_ptr<Msg, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=0x46aca48, __in_chrg=<optimized out>)
    at /usr/include/c++/4.8.2/bits/shared_ptr_base.h:781

#25 0x0000000000a83cac in std::shared_ptr<Msg>::~shared_ptr (this=0x46aca48, __in_chrg=<optimized out>)
    at /usr/include/c++/4.8.2/bits/shared_ptr.h:93
#26 0x0000000000cd5b30 in std::pair<long, std::shared_ptr<Msg> >::~pair (this=0x46aca40, __in_chrg=<optimized out>)
    at /usr/include/c++/4.8.2/bits/stl_pair.h:96
#27 0x0000000000cd5b4e in __gnu_cxx::new_allocator<std::pair<long const, std::shared_ptr<Msg> > >::destroy<std::pair<long, std::shared_ptr<Msg> > > (this=0x4606470, __p=0x46aca40) at /usr/include/c++/4.8.2/ext/new_allocator.h:124
#28 0x0000000000cd061d in std::allocator_traits<std::allocator<std::pair<long const, std::shared_ptr<Msg> > > >::_S_destroy<std::pair<long, std::shared_ptr<Msg> > > (__a=..., __p=0x46aca40) at /usr/include/c++/4.8.2/bits/alloc_traits.h:281
#29 0x0000000000ccb4fa in std::allocator_traits<std::allocator<std::pair<long const, std::shared_ptr<Msg> > > >::destroy<std::pair<long, std::shared_ptr<Msg> > > (__a=..., __p=0x46aca40) at /usr/include/c++/4.8.2/bits/alloc_traits.h:405
#30 0x0000000000cc4b8f in libcuckoo::bucket_container<long, std::shared_ptr<Msg>, std::allocator<std::pair<long const, std::shared_ptr<Msg> > >, unsigned char, 4ul>::eraseKV (this=0x4606470, ind=192, slot=2)
    at /home/jlh/oin/src/apex_adapter_3rd/../../include/libcuckoo/bucket_container.hh:219
#31 0x0000000000cb7b36 in libcuckoo::cuckoohash_map<long, std::shared_ptr<Msg>, std::hash<long>, std::equal_to<long>, std::allocator<std::pair<long const, std::shared_ptr<Msg> > >, 4ul>::del_from_bucket (this=0x4606468, bucket_ind=192, slot=2)
    at /home/jlh/oin/src/apex_adapter_3rd/../../include/libcuckoo/cuckoohash_map.hh:1963
#32 0x0000000000c9f496 in libcuckoo::cuckoohash_map<long, std::shared_ptr<Msg>, std::hash<long>, std::equal_to<long>, std::allocator<std::pair<long const, std::shared_ptr<Msg> > >, 4ul>::erase_fn<long, bool libcuckoo::cuckoohash_map<long, std::shared_ptr<Msg>, std::hash<long>, std::equal_to<long>, std::allocator<std::pair<long const, std::shared_ptr<Msg> > >, 4ul>::erase<long>(long const&)::{lambda(std::shared_ptr<Msg>&)#1}>(long const&, bool libcuckoo::cuckoohash_map<long, std::shared_ptr<Msg>, std::hash<long>, std::equal_to<long>, std::allocator<std::pair<long const, std::shared_ptr<Msg> > >, 4ul>::erase<long>(long const&)::{lambda(std::shared_ptr<Msg>&)#1}) (
    this=0x4606468, key=@0x7f4c4f7fd090: 139964327177408, fn=...)
    at /home/jlh/oin/src/apex_adapter_3rd/../../include/libcuckoo/cuckoohash_map.hh:518
#33 0x0000000000c553ee in libcuckoo::cuckoohash_map<long, std::shared_ptr<Msg>, std::hash<long>, std::equal_to<long>, std::allocator<std::pair<long const, std::shared_ptr<Msg> > >, 4ul>::erase<long> (this=0x4606468, key=@0x7f4c4f7fd090: 139964327177408)
    at /home/jlh/oin/src/apex_adapter_3rd/../../include/libcuckoo/cuckoohash_map.hh:650
#34 0x0000000000c1a637 in CuckooMap<long, std::shared_ptr<Msg> >::Delete (this=0x4606468, key=@0x7f4c4f7fd090: 139964327177408)
    at /home/jlh/oin/src/apex_adapter_3rd/../../include/libcuckoo/cuckoo_map.h:30
#35 0x0000000000c16aeb in ApexAdapterWorker::ReleaseMsg (this=0x4606418, session=139964327177408)
    at /home/jlh/oin/src/apex_adapter_3rd/src/apex_adapter_worker.h:92
#36 0x0000000000c01798 in RspMsgWorker::dojob (this=0x4642938) at /home/jlh/oin/src/apex_adapter_3rd/src/apex_adapter_worker.cpp:6012
#37 0x0000000000b0bab9 in ec::cThread::mainloop (this=0x4642938) at /home/jlh/oin/src/apex_adapter_3rd/../../include/ec/c11_thread.h:93
#38 0x0000000000b0b9f6 in ec::cThread::ThreadProcess (pargs=0x4642938)
    at /home/jlh/oin/src/apex_adapter_3rd/../../include/ec/c11_thread.h:81
#39 0x0000000000b48202 in std::_Bind_simple<void (*(ec::cThread*))(void*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x4642a50)
    at /usr/include/c++/4.8.2/functional:1732
#40 0x0000000000b47e0b in std::_Bind_simple<void (*(ec::cThread*))(void*)>::operator()() (this=0x4642a50)
    at /usr/include/c++/4.8.2/functional:1720
#41 0x0000000000b4799a in std::thread::_Impl<std::_Bind_simple<void (*(ec::cThread*))(void*)> >::_M_run() (this=0x4642a38)
---Type <return> to continue, or q <return> to quit---
    at /usr/include/c++/4.8.2/thread:115
#42 0x00007f4c6eef22b0 in ?? () from /lib64/libstdc++.so.6
#43 0x00007f4c72ce9e25 in start_thread () from /lib64/libpthread.so.0
#44 0x00007f4c6e65a34d in clone () from /lib64/libc.so.6

Junction as headers only

Hi
Just seen this comment on
https://preshing.com/20160201/new-concurrent-hash-maps-for-cpp
:
"People would obviously prefer a header-only, C++11 style implementation of Junction that they can just drop into any project. The thing is, the code is pretty experimental, so I wouldn't feel comfortable if a lot of people did that anyway."

Could indeed be Junction converted to some headers only lib ?

Congrats for the nice lib anyway,
Kind

Limitations

  1. "Hash function must be invertible, so that every key has a unique hash". This seems like a severe limitation. std::map deals with collisions without issues. Why not use a thread-safe queue to remove the limitation ?

  2. How about TTL, LRU and LFU ?

Building with MinGW (MinGW Builds gcc 5.3.0)

Interesting proposal. 👍 I would wish to try it in a "real" project
Any possibility of official support for MinGW? Thanks! :-)

> cmake -G "MinGW Makefiles" ..
-- The C compiler identification is GNU 5.3.0
-- The CXX compiler identification is GNU 5.3.0
-- Check for working C compiler: V:/MinGW-Builds/mingw64/bin/gcc.exe
-- Check for working C compiler: V:/MinGW-Builds/mingw64/bin/gcc.exe -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: V:/MinGW-Builds/mingw64/bin/g++.exe
-- Check for working CXX compiler: V:/MinGW-Builds/mingw64/bin/g++.exe -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMAKE_CXX_FLAGS = -g -std=c++11 -fno-stack-protector
Checking for <stdint.h> -- yes
Checking for noexcept keyword -- yes
Checking for constexpr keyword -- yes
Checking for override keyword -- yes
Checking for long long -- yes
-- Configuring done
-- Generating done
-- Build files have been written to: P:/Plataformas/x32-x64/junction/junction/buildunix

> mingw32-make
Scanning dependencies of target junction
[  2%] Building CXX object CMakeFiles/junction.dir/junction/ConcurrentMap_Grampa.cpp.obj
In file included from P:/Plataformas/x32-x64/junction/turf/turf/c/core.h:17:0,
                 from P:/Plataformas/x32-x64/junction/turf/turf/Core.h:18,
                 from P:/Plataformas/x32-x64/junction/junction/junction/Core.h:25,
                 from P:/Plataformas/x32-x64/junction/junction/junction/ConcurrentMap_Grampa.h:16,
                 from P:\Plataformas\x32-x64\junction\junction\junction\ConcurrentMap_Grampa.cpp:13:
P:/Plataformas/x32-x64/junction/turf/turf/c/platform_detect.h:37:10: error: #error "Unrecognized platform!"
         #error "Unrecognized platform!"
          ^
In file included from P:/Plataformas/x32-x64/junction/turf/turf/impl/Atomic_Native.h:17:0,
                 from P:/Plataformas/x32-x64/junction/turf/turf/Atomic.h:47,
                 from P:/Plataformas/x32-x64/junction/junction/junction/Core.h:27,
                 from P:/Plataformas/x32-x64/junction/junction/junction/ConcurrentMap_Grampa.h:16,
                 from P:\Plataformas\x32-x64\junction\junction\junction\ConcurrentMap_Grampa.cpp:13:
P:/Plataformas/x32-x64/junction/turf/turf/c/atomic.h:424:6: error: #error "TURF_PTR_SIZE not set!"
     #error "TURF_PTR_SIZE not set!"
      ^
In file included from P:/Plataformas/x32-x64/junction/turf/turf/c/atomic.h:33:0,
                 from P:/Plataformas/x32-x64/junction/turf/turf/impl/Atomic_Native.h:17,
                 from P:/Plataformas/x32-x64/junction/turf/turf/Atomic.h:47,
                 from P:/Plataformas/x32-x64/junction/junction/junction/Core.h:27,
                 from P:/Plataformas/x32-x64/junction/junction/junction/ConcurrentMap_Grampa.h:16,
                 from P:\Plataformas\x32-x64\junction\junction\junction\ConcurrentMap_Grampa.cpp:13:
P:/Plataformas/x32-x64/junction/turf/turf/c/impl/atomic_msvc.h: In function 'uint8_t turf_compareExchange8Relaxed(turf_atomic8_t*, uint8_t, uint8_t)':
P:/Plataformas/x32-x64/junction/turf/turf/c/impl/atomic_msvc.h:77:74: error: '_InterlockedCompareExchange8' was not declared in this scope
     return _InterlockedCompareExchange8((char*) object, desired, expected);
                                                                          ^
P:/Plataformas/x32-x64/junction/turf/turf/c/impl/atomic_msvc.h: In function 'intreg_t turf_compareExchangeWeak8Relaxed(turf_atomic8_t*, uint8_t*, uint8_t)':
P:/Plataformas/x32-x64/junction/turf/turf/c/impl/atomic_msvc.h:82:79: error: '_InterlockedCompareExchange8' was not declared in this scope
     uint8_t previous = _InterlockedCompareExchange8((char*) object, desired, e);
                                                                               ^
P:/Plataformas/x32-x64/junction/turf/turf/c/impl/atomic_msvc.h: In function 'uint8_t turf_exchange8Relaxed(turf_atomic8_t*, uint8_t)':
P:/Plataformas/x32-x64/junction/turf/turf/c/impl/atomic_msvc.h:90:57: error: '_InterlockedExchange8' was not declared in this scope
     return _InterlockedExchange8((char*) object, desired);
                                                         ^
P:/Plataformas/x32-x64/junction/turf/turf/c/impl/atomic_msvc.h: In function 'uint8_t turf_fetchAdd8Relaxed(turf_atomic8_t*, int8_t)':
P:/Plataformas/x32-x64/junction/turf/turf/c/impl/atomic_msvc.h:94:60: error: '_InterlockedExchangeAdd8' was not declared in this scope
     return _InterlockedExchangeAdd8((char*) object, operand);
                                                            ^
P:/Plataformas/x32-x64/junction/turf/turf/c/impl/atomic_msvc.h: In function 'uint16_t turf_exchange16Relaxed(turf_atomic16_t*, uint16_t)':
P:/Plataformas/x32-x64/junction/turf/turf/c/impl/atomic_msvc.h:132:59: error: '_InterlockedExchange16' was not declared in this scope
     return _InterlockedExchange16((short*) object, desired);
                                                           ^
P:/Plataformas/x32-x64/junction/turf/turf/c/impl/atomic_msvc.h: In function 'uint16_t turf_fetchAdd16Relaxed(turf_atomic16_t*, int16_t)':
P:/Plataformas/x32-x64/junction/turf/turf/c/impl/atomic_msvc.h:136:62: error: '_InterlockedExchangeAdd16' was not declared in this scope
     return _InterlockedExchangeAdd16((short*) object, operand);
                                                              ^
P:/Plataformas/x32-x64/junction/turf/turf/c/impl/atomic_msvc.h: In function 'uint64_t turf_load64Relaxed(const turf_atomic64_t*)':
P:/Plataformas/x32-x64/junction/turf/turf/c/impl/atomic_msvc.h:203:11: error: expected '(' before '{' token
     __asm {
           ^
P:/Plataformas/x32-x64/junction/turf/turf/c/impl/atomic_msvc.h:204:9: error: 'mov' was not declared in this scope
         mov esi, object;
         ^
P:/Plataformas/x32-x64/junction/turf/turf/c/impl/atomic_msvc.h:205:13: error: expected ';' before 'ebx'
         mov ebx, eax;
             ^
P:/Plataformas/x32-x64/junction/turf/turf/c/impl/atomic_msvc.h:206:13: error: expected ';' before 'ecx'
         mov ecx, edx;
             ^
P:/Plataformas/x32-x64/junction/turf/turf/c/impl/atomic_msvc.h:207:9: error: 'lock' was not declared in this scope
         lock cmpxchg8b [esi];
         ^
P:/Plataformas/x32-x64/junction/turf/turf/c/impl/atomic_msvc.h:208:13: error: expected ';' before 'dword'
         mov dword ptr result, eax;
             ^
P:/Plataformas/x32-x64/junction/turf/turf/c/impl/atomic_msvc.h:209:13: error: expected ';' before 'dword'
         mov dword ptr result[4], edx;
             ^
P:/Plataformas/x32-x64/junction/turf/turf/c/impl/atomic_msvc.h: In function 'void turf_store64Relaxed(turf_atomic64_t*, uint64_t)':
P:/Plataformas/x32-x64/junction/turf/turf/c/impl/atomic_msvc.h:228:11: error: expected '(' before '{' token
     __asm {
           ^
P:/Plataformas/x32-x64/junction/turf/turf/c/impl/atomic_msvc.h:229:9: error: 'mov' was not declared in this scope
         mov esi, object;
         ^
P:/Plataformas/x32-x64/junction/turf/turf/c/impl/atomic_msvc.h:230:13: error: expected ';' before 'ebx'
         mov ebx, dword ptr value;
             ^
P:/Plataformas/x32-x64/junction/turf/turf/c/impl/atomic_msvc.h:231:13: error: expected ';' before 'ecx'
         mov ecx, dword ptr value[4];
             ^
P:/Plataformas/x32-x64/junction/turf/turf/c/impl/atomic_msvc.h:233:9: error: 'cmpxchg8b' was not declared in this scope
         cmpxchg8b [esi];
         ^
P:/Plataformas/x32-x64/junction/turf/turf/c/impl/atomic_msvc.h:233:20: error: 'esi' was not declared in this scope
         cmpxchg8b [esi];
                    ^
P:/Plataformas/x32-x64/junction/turf/turf/c/impl/atomic_msvc.h:234:9: error: 'jne' was not declared in this scope
         jne retry;
         ^
In file included from P:/Plataformas/x32-x64/junction/turf/turf/Atomic.h:47:0,
                 from P:/Plataformas/x32-x64/junction/junction/junction/Core.h:27,
                 from P:/Plataformas/x32-x64/junction/junction/junction/ConcurrentMap_Grampa.h:16,
                 from P:\Plataformas\x32-x64\junction\junction\junction\ConcurrentMap_Grampa.cpp:13:
P:/Plataformas/x32-x64/junction/turf/turf/impl/Atomic_Native.h: In member function 'T* turf::Atomic_Native<T*>::load(turf::MemoryOrder) const':
P:/Plataformas/x32-x64/junction/turf/turf/impl/Atomic_Native.h:174:76: error: there are no arguments to 'turf_loadPtr' that depend on a template parameter, so a declaration of 'turf_loadPtr' must be available [-fpermissive]
         return (T*) turf_loadPtr(&m_value, (turf_memoryOrder_t) memoryOrder);
                                                                            ^
P:/Plataformas/x32-x64/junction/turf/turf/impl/Atomic_Native.h:174:76: note: (if you use '-fpermissive', G++ will accept your code, but allowing the use of an undeclared name is deprecated)
In file included from P:/Plataformas/x32-x64/junction/turf/turf/ConditionVariable.h:39:0,
                 from P:/Plataformas/x32-x64/junction/junction/junction/striped/ConditionPair.h:18,
                 from P:/Plataformas/x32-x64/junction/junction/junction/striped/ConditionBank.h:17,
                 from P:/Plataformas/x32-x64/junction/junction/junction/striped/AutoResetEvent.h:17,
                 from P:/Plataformas/x32-x64/junction/junction/junction/striped/Mutex.h:23,
                 from P:/Plataformas/x32-x64/junction/junction/junction/details/Grampa.h:18,
                 from P:/Plataformas/x32-x64/junction/junction/junction/ConcurrentMap_Grampa.h:17,
                 from P:\Plataformas\x32-x64\junction\junction\junction\ConcurrentMap_Grampa.cpp:13:
P:/Plataformas/x32-x64/junction/turf/turf/impl/ConditionVariable_Win32.h: In constructor 'turf::ConditionVariable_Win32::ConditionVariable_Win32()':
P:/Plataformas/x32-x64/junction/turf/turf/impl/ConditionVariable_Win32.h:27:47: error: 'InitializeConditionVariable' was not declared in this scope
         InitializeConditionVariable(&m_condVar);
                                               ^
P:/Plataformas/x32-x64/junction/turf/turf/impl/ConditionVariable_Win32.h: In member function 'void turf::ConditionVariable_Win32::wait(turf::LockGuard<turf::Mutex_Win32>&)':
P:/Plataformas/x32-x64/junction/turf/turf/impl/ConditionVariable_Win32.h:31:81: error: 'SleepConditionVariableCS' was not declared in this scope
         SleepConditionVariableCS(&m_condVar, &guard.getMutex().m_mutex, INFINITE);
                                                                                 ^
P:/Plataformas/x32-x64/junction/turf/turf/impl/ConditionVariable_Win32.h: In member function 'void turf::ConditionVariable_Win32::timedWait(turf::LockGuard<turf::Mutex_Win32>&, turf::intTypes::ureg)':
P:/Plataformas/x32-x64/junction/turf/turf/impl/ConditionVariable_Win32.h:36:87: error: 'SleepConditionVariableCS' was not declared in this scope
             SleepConditionVariableCS(&m_condVar, &guard.getMutex().m_mutex, waitMillis);
                                                                                       ^
P:/Plataformas/x32-x64/junction/turf/turf/impl/ConditionVariable_Win32.h: In member function 'void turf::ConditionVariable_Win32::wakeOne()':
P:/Plataformas/x32-x64/junction/turf/turf/impl/ConditionVariable_Win32.h:40:41: error: 'WakeConditionVariable' was not declared in this scope
         WakeConditionVariable(&m_condVar);
                                         ^
P:/Plataformas/x32-x64/junction/turf/turf/impl/ConditionVariable_Win32.h: In member function 'void turf::ConditionVariable_Win32::wakeAll()':
P:/Plataformas/x32-x64/junction/turf/turf/impl/ConditionVariable_Win32.h:44:44: error: 'WakeAllConditionVariable' was not declared in this scope
         WakeAllConditionVariable(&m_condVar);
                                            ^
In file included from P:/Plataformas/x32-x64/junction/turf/turf/Atomic.h:47:0,
                 from P:/Plataformas/x32-x64/junction/junction/junction/Core.h:27,
                 from P:/Plataformas/x32-x64/junction/junction/junction/ConcurrentMap_Grampa.h:16,
                 from P:\Plataformas\x32-x64\junction\junction\junction\ConcurrentMap_Grampa.cpp:13:
P:/Plataformas/x32-x64/junction/turf/turf/impl/Atomic_Native.h: In instantiation of 'T* turf::Atomic_Native<T*>::load(turf::MemoryOrder) const [with T = junction::striped::ConditionPair]':
P:/Plataformas/x32-x64/junction/junction/junction/striped/ConditionBank.h:40:58:   required from here
P:/Plataformas/x32-x64/junction/turf/turf/impl/Atomic_Native.h:174:33: error: 'turf_loadPtr' was not declared in this scope
         return (T*) turf_loadPtr(&m_value, (turf_memoryOrder_t) memoryOrder);
                                 ^
CMakeFiles\junction.dir\build.make:62: recipe for target 'CMakeFiles/junction.dir/junction/ConcurrentMap_Grampa.cpp.obj' failed
mingw32-make[2]: *** [CMakeFiles/junction.dir/junction/ConcurrentMap_Grampa.cpp.obj] Error 1
CMakeFiles\Makefile2:66: recipe for target 'CMakeFiles/junction.dir/all' failed
mingw32-make[1]: *** [CMakeFiles/junction.dir/all] Error 2
Makefile:126: recipe for target 'all' failed
mingw32-make: *** [all] Error 2

Multithreaded Migration and the implicit lock ?

I am asking this based on the article http://preshing.com/20160222/a-resizable-concurrent-map/ not based on the code. If you coordinate multithreaded migration with a lock, the whole thing can't be fully concurrent anymore (by definition ?).

Also you write: "In the current version of Linear, even get calls don’t read from the new table until the migration is complete." That sounds like a (b)lock to me, if your get API does not want to return the redirect marker.

My current belief is, that you have to do things optimistically otherwise, you lose being concurrent.

Is there exist a fixed-size hash table's scenes ?

I have already seen the source code, When erase it do not set the key to null and I guess it's in order to keep the chain. This cell will not be occupied by others, resulting in a larger hash table until it is rewritten.
In our project, we assign a fixed-size hash table at initialization time and when running, it can not be changed.
Is there a fixed-size hash table implement
Or How to handle the deletion ?

Correct use of Garbage collection

In the blog/docs it says

To make QSBR work, each thread must periodically call junction::DefaultQSBR.update at a moment when that thread is quiescent – that is, not in the middle of an operation that uses the map. In a game engine, you could call it between iterations of the main loop.

I am experimenting with the leapfrog map, and inserted a call to junction::DefaultQSBR.flush() (since there is no update()). However, when I add the flush call, my code throws exceptions due to empty states in some objects retrieved from the map. There is no single place in the code where I can guarantee that the map is not accessed since everything is completely asynchronous. When one thread is calling flush() another thread might decide to insert or remove something from the map.

Is there a recommended way of dealing with this? (Also please clarify "each thread must periodically call" - do you mean that all threads must call it individually, or can any one single thread call it from time to time).

Junction on ARM ?

Great work. Does Junction work out of the box with ARM32/ARM64 CPUs ?

bad_alloc in QSBR.h with Visual Studio

Hi, I am facing a problem when using Leapfrog concurrent map.
I have created Visual Studio project with cmake, linked libs to my project and added include folders.
After that I did a small setup to test if everything works and it stared failing on bad_alloc, even when I used different versions of Visual Studio or switched to 32/64bit.
With initial size of the map equal to 8, after adding 10th element to map it fails on bad_alloc at QSBR.h:80 (notice the m_deferredActions capacity at first screenshot) when push_back is executed.
I have also attached whole project (2.5MB) with all the libraries prepared.
Please help me in resolving this issue.
Thanks!

  1. screenshot (QSBR.h)
    image

  2. screenshot (bad_alloc)
    image

  3. screenshot (main.cpp)
    image

DummyProject.zip

Make turf a git submodule

Why not make turf a git submodule? Seems like it would be best to pin a specific version of turf to a specific version of junction to avoid breaking the compilation and tracking the API interface between projects with version control.

issue when inserting into concurrent map

Hi,

I have a seemingly simple test program which hangs on my machine under certain circumstances:

#include <junction/ConcurrentMap_Leapfrog.h>
#include <iostream>

struct Value
{
    int v;
};

using ConcurrentMap = junction::ConcurrentMap_Leapfrog<int,Value*>;

int main()
{
    ConcurrentMap cmap(10000);
    Value value{ 0 };

    //for (int i = 0; i < 1000000; ++i) // locks in 158th assign
    for (int i = 999999; i >= 0; --i) // works as expected
    {
        cmap.assign(i, &value);
        std::cout << "inserted " << i << "\n";
    }

    std::cout << "done\n";
}

As you can see from the comment around the for loops, one case works as expected (completes without error) while the other one hangs. I am at a loss to explain why. Is this a bug, or a known issue? Or am I doing something stupid?

Thanks for your time.

supporting C++11 rules issue

hi , when i try to run the example in junction ,
IDE throw error which in the junction libs
q -20160229152448
q -20160229152414

system: win7 64bit
IDE: VS 2013

i did some digs , noted that the reason cause error is the IDE is not completely compatibility with C++ 11.
VS2013 IDE doesn't support the following usage

class UnitTest    // in VS2013
{
    const static int a = 10;    // VALID 
    const static string c = "good moringint";  // invalid 
    const static float d = 1.254;  // invalid 
    const static char e = '4';  // invalid 
};

Upgrading IDE VS2013 to VS2015 is not my option which is not free.
Modifying source code to fix the issue in VS2013 is my only way to apply junction .

How to use a non-number type as a key for maps (for example std::bitset)

I'm certain that this is a feature unsupported by junction, but just in case, I'll provide some system details.

OS: Ubuntu 20.04.2 LTS
Build system: CMake (used through CLion IDE). Not sure which CMake version it uses under the hood though.

I'm trying to create something like a transposition table for a chess engine. So the following code compiles:

#include <junction/ConcurrentMap_Leapfrog.h>

struct Foo {};

int main() {
    typedef junction::ConcurrentMap_Leapfrog<unsigned long long, Foo*> ConcurrentMap;
    ConcurrentMap map;
    return 0;
}

However, in my case, I'm unable to use unsigned long long as the position hash, because it can't hold enough values. So I use std::bitset. But there is a problem, the following won't compile:

#include <bitset>
#include <junction/ConcurrentMap_Leapfrog.h>

struct Foo {};

int main() {
    typedef junction::ConcurrentMap_Leapfrog<std::bitset<100>, Foo*> ConcurrentMap;
    ConcurrentMap map;
    return 0;
}

The error is the following:

In file included from /path/to/project/main.cpp:2:
/path/to/project/junction/junction/ConcurrentMap_Leapfrog.h: In instantiation of ‘class junction::ConcurrentMap_Leapfrog<std::bitset<100>, Foo*>’:
/path/to/project/main.cpp:8:19:   required from here
/path/to/project/junction/junction/ConcurrentMap_Leapfrog.h:33:57: error: invalid use of incomplete type ‘struct turf::util::BestFit<std::bitset<100> >’
   33 |     typedef typename turf::util::BestFit<Key>::Unsigned Hash;
      |                                                         ^~~~
In file included from /path/to/project/junction/junction/details/Leapfrog.h:20,
                 from /path/to/project/junction/junction/ConcurrentMap_Leapfrog.h:17,
                 from /path/to/project/main.cpp:2:
/path/to/project/turf/turf/Util.h:22:8: note: declaration of ‘struct turf::util::BestFit<std::bitset<100> >’
   22 | struct BestFit;
      |        ^~~~~~~
In file included from /path/to/project/junction/junction/ConcurrentMap_Leapfrog.h:17,
                 from /path/to/project/main.cpp:2:
/path/to/project/junction/junction/details/Leapfrog.h: In instantiation of ‘static junction::details::Leapfrog<Map>::Table* junction::details::Leapfrog<Map>::Table::create(turf::intTypes::ureg) [with Map = junction::ConcurrentMap_Leapfrog<std::bitset<100>, Foo*>; turf::intTypes::ureg = long long unsigned int]’:
/path/to/project/junction/junction/ConcurrentMap_Leapfrog.h:40:97:   required from ‘junction::ConcurrentMap_Leapfrog<K, V, KT, VT>::ConcurrentMap_Leapfrog(turf::intTypes::ureg) [with K = std::bitset<100>; V = Foo*; KT = junction::DefaultKeyTraits<std::bitset<100> >; VT = junction::DefaultValueTraits<Foo*>; turf::intTypes::ureg = long long unsigned int]’
/path/to/project/main.cpp:8:19:   required from here
/path/to/project/junction/junction/details/Leapfrog.h:81:37: error: ‘struct junction::details::Leapfrog<junction::ConcurrentMap_Leapfrog<std::bitset<100>, Foo*> >::Cell’ has no member named ‘hash’
   81 |                     group->cells[j].hash.storeNonatomic(KeyTraits::NullHash);
      |                     ~~~~~~~~~~~~~~~~^~~~
In file included from /path/to/project/junction/junction/details/Leapfrog.h:21,
                 from /path/to/project/junction/junction/ConcurrentMap_Leapfrog.h:17,
                 from /path/to/project/main.cpp:2:
/path/to/project/junction/junction/MapTraits.h: In instantiation of ‘struct junction::DefaultKeyTraits<std::bitset<100> >’:
/path/to/project/junction/junction/details/Leapfrog.h:81:21:   required from ‘static junction::details::Leapfrog<Map>::Table* junction::details::Leapfrog<Map>::Table::create(turf::intTypes::ureg) [with Map = junction::ConcurrentMap_Leapfrog<std::bitset<100>, Foo*>; turf::intTypes::ureg = long long unsigned int]’
/path/to/project/junction/junction/ConcurrentMap_Leapfrog.h:40:97:   required from ‘junction::ConcurrentMap_Leapfrog<K, V, KT, VT>::ConcurrentMap_Leapfrog(turf::intTypes::ureg) [with K = std::bitset<100>; V = Foo*; KT = junction::DefaultKeyTraits<std::bitset<100> >; VT = junction::DefaultValueTraits<Foo*>; turf::intTypes::ureg = long long unsigned int]’
/path/to/project/main.cpp:8:19:   required from here
/path/to/project/junction/junction/MapTraits.h:24:55: error: invalid use of incomplete type ‘struct turf::util::BestFit<std::bitset<100> >’
   24 |     typedef typename turf::util::BestFit<T>::Unsigned Hash;
      |                                                       ^~~~
In file included from /path/to/project/junction/junction/details/Leapfrog.h:20,
                 from /path/to/project/junction/junction/ConcurrentMap_Leapfrog.h:17,
                 from /path/to/project/main.cpp:2:
/path/to/project/turf/turf/Util.h:22:8: note: declaration of ‘struct turf::util::BestFit<std::bitset<100> >’
   22 | struct BestFit;
      |        ^~~~~~~
In file included from /path/to/project/junction/junction/details/Leapfrog.h:21,
                 from /path/to/project/junction/junction/ConcurrentMap_Leapfrog.h:17,
                 from /path/to/project/main.cpp:2:
/path/to/project/junction/junction/MapTraits.h:25:22: error: ‘constexpr’ needed for in-class initialization of static data member ‘const Key junction::DefaultKeyTraits<std::bitset<100> >::NullKey’ of non-integral type [-fpermissive]
   25 |     static const Key NullKey = Key(0);
      |                      ^~~~~~~
In file included from /path/to/project/junction/junction/ConcurrentMap_Leapfrog.h:17,
                 from /path/to/project/main.cpp:2:
/path/to/project/junction/junction/details/Leapfrog.h: In instantiation of ‘static junction::details::Leapfrog<Map>::Table* junction::details::Leapfrog<Map>::Table::create(turf::intTypes::ureg) [with Map = junction::ConcurrentMap_Leapfrog<std::bitset<100>, Foo*>; turf::intTypes::ureg = long long unsigned int]’:
/path/to/project/junction/junction/ConcurrentMap_Leapfrog.h:40:97:   required from ‘junction::ConcurrentMap_Leapfrog<K, V, KT, VT>::ConcurrentMap_Leapfrog(turf::intTypes::ureg) [with K = std::bitset<100>; V = Foo*; KT = junction::DefaultKeyTraits<std::bitset<100> >; VT = junction::DefaultValueTraits<Foo*>; turf::intTypes::ureg = long long unsigned int]’
/path/to/project/main.cpp:8:19:   required from here
/path/to/project/junction/junction/details/Leapfrog.h:81:21: error: ‘NullHash’ is not a member of ‘junction::details::Leapfrog<junction::ConcurrentMap_Leapfrog<std::bitset<100>, Foo*> >::KeyTraits’ {aka ‘junction::DefaultKeyTraits<std::bitset<100> >’}
   81 |                     group->cells[j].hash.storeNonatomic(KeyTraits::NullHash);
      |                     ^~~~~
make[3]: *** [CMakeFiles/test.dir/build.make:82: CMakeFiles/test.dir/main.cpp.o] Error 1
make[2]: *** [CMakeFiles/Makefile2:136: CMakeFiles/test.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:143: CMakeFiles/test.dir/rule] Error 2
make: *** [Makefile:137: test] Error 2

To my understanding, the error is caused by turf being unable to hash std::bitset. Also, C++11 supports hashing std::bitset and a few other types by default so maybe using std::hash in junction implementation could help. I could submit a PR, if you gave me a bit of guidance.

Unexpected behavior on unit test with assign/erase/assign

Apologies if this turns out to be due to my own mishandling of the library.

I've written a test in some of my own code to see what container I should use for a concurrent hash map. The test is as follow (adapted to junction):

template <typename Pool>
void bash_junction_map(Pool& pool, const char* name)
{
	using namespace ::testing;
	using namespace stk;

	junction::ConcurrentMap_Leapfrog<int, int> m;

	auto nItems = 10000;
	for (auto i = 2; i < nItems + 2; ++i)
	{
		m.assign(i, i * 10);
	}

	using future_t = typename Pool::template future<void>;
	std::vector<future_t> fs;
	fs.reserve(100000);
	{
		GEOMETRIX_MEASURE_SCOPE_TIME(name);
		for (unsigned i = 2; i < 100000 + 2; ++i) {
			fs.emplace_back(pool.send([&m, i]() -> void
			{
				for (int q = 0; q < nsubwork; ++q)
				{
					m.assign(i, i * 20);
					m.erase(i);
					m.assign(i, i * 20);
				}
			}));
		}
		boost::for_each(fs, [](const future_t& f) { f.wait(); });
	}

	for (auto i = 2; i < 100000 + 2; ++i)
	{
		auto r = m.find(i);
		EXPECT_EQ(i * 20, r.getValue());
	}
}

My understanding is that an assign operation followed by an erase and then another assign ought to essentially cancel each other out. However, I'm getting spurious failures with junction:

Note: Google Test filter = timing.work_stealing_thread_pool_moodycamel_concurrentQ_bash_junction
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from timing
[ RUN      ] timing.work_stealing_thread_pool_moodycamel_concurrentQ_bash_junction
G:\Projects\simulation_suite\test\fiber_timing_tests.cpp(150): error: Expected equality of these values:
  i * 20
    Which is: 1032520
  r.getValue()
    Which is: 0
G:\Projects\simulation_suite\test\fiber_timing_tests.cpp(150): error: Expected equality of these values:
  i * 20
    Which is: 540200
  r.getValue()
    Which is: 0
G:\Projects\simulation_suite\test\fiber_timing_tests.cpp(150): error: Expected equality of these values:
  i * 20
    Which is: 1032520
  r.getValue()
    Which is: 0
[  FAILED  ] timing.work_stealing_thread_pool_moodycamel_concurrentQ_bash_junction (35310 ms)
[----------] 1 test from timing (35311 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (35313 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] timing.work_stealing_thread_pool_moodycamel_concurrentQ_bash_junction

 1 FAILED TEST
Press any key to continue . . .

Are my expectations incorrect?

SingleMap_Linear: bug when migrating?

In SingleMap_Linear.h:123: you break out of the loop without resetting m_cell. After the migration in line 122 the old table is destroyed and m_cell is dangling, isn't it? isValid() will report true and you can set values on the dangling pointer.

Error in samples/MapMemoryBench/MapMemoryBench.cpp

Line 47 of samples/MapMemoryBench/MapMemoryBench.cpp reads like follows
map->insert(key, (void*) uptr(key));

but, I guess, it should be:
map->assign(key, (void*) uptr(key));

The error when executing python python TestAllMaps.py . says:

MapMemoryBench.cpp:47:22:
error: junction::extra::MapAdapter::Map’ has no member named ‘insert’
                 map->insert(key, (void*) uptr(key));

Linker Errors: undefined reference to std::__cxx11::basic_string<>...

I statically-linked Junction and Turf into a shared library and am getting the following linker errors when linking my shared library into another executable. Generally these errors are due to compiling the library with an older GLIBCXX version than the final app, but I'm building everything with g++ 7.3.0

../lib/libreference_data_driver.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::c_str() const'
../lib/libreference_data_driver.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::find_first_of(char, unsigned long) const'
../lib/libreference_data_driver.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::substr(unsigned long, unsigned long) const'
../lib/libreference_data_driver.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string()'
../lib/libreference_data_driver.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&)'
../lib/libreference_data_driver.so: undefined reference to `std::basic_istream<char, std::char_traits<char> >& std::getline<char, std::char_traits<char>, std::allocator<char> >(std::basic_istream<char, std::char_traits<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)'
../lib/libreference_data_driver.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string()'
../lib/libreference_data_driver.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::find_last_not_of(char const*, unsigned long) const'
../lib/libreference_data_driver.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::compare(char const*) const'
../lib/libreference_data_driver.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::compare(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const'
../lib/libreference_data_driver.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&)'

My Junction/Turf config:

#define JUNCTION_WITH_FOLLY 0
#define JUNCTION_WITH_CDS 0
#define JUNCTION_WITH_NBDS 0
#define JUNCTION_WITH_TBB 0
#define JUNCTION_WITH_TERVEL 0
#define JUNCTION_WITH_LIBCUCKOO 0
#define NBDS_USE_TURF_HEAP 0
#define TBB_USE_TURF_HEAP 0
#define JUNCTION_TRACK_GRAMPA_STATS 0
#define JUNCTION_USE_STRIPING 1
---
#define TURF_HAS_STDINT 1
#define TURF_PREFER_CPP11 0
#define TURF_WITH_BOOST 0
#define TURF_PREFER_BOOST 0
#define TURF_WITH_EXCEPTIONS 0
#define TURF_HAS_NOEXCEPT 1
#define TURF_HAS_CONSTEXPR 1
#define TURF_HAS_OVERRIDE 1
#define TURF_HAS_LONG_LONG 1
#define TURF_HAS_STATIC_ASSERT 1
#define TURF_HAS_MOVE 1
#define TURF_REPLACE_OPERATOR_NEW 0
#define TURF_USE_DLMALLOC 0
#define TURF_DLMALLOC_DEBUG_CHECKS 0
#define TURF_DLMALLOC_FAST_STATS 0

Has anyone else run into this?

Freeing dynamically allocated values

How do you free dynamically allocated values when, for example, SingleMap_Linear is destroyed. I see that the cell's destructors are run, but that doesn't free the Value if the Value is a pointer. And without a way to iterate over entries, it seems there would just me a memory leak.

Cmake error when doing map performance test

I had the issue that the Cmake reported error "cannot find CDS"
In the directory: /Github/junction/samples/MapPerformanceTests/build-michael
CMake Error at ../../cmake/modules/FindCDS.cmake:23 (message)
The operating system I am using is macOS 10.12.
How should I solve it?

Type limitation on values arbitrary?

I'm trying to use the leapfrog hashmap in the context of packet processing and flow tracking.
Ideally the flow data would be stored directly in the hash table to mitigate the whole issue of memory allocations. But according to your blog posts and the readme, value types are limited to integers and pointers.
I understand the restrictions on key types because of the hash reversibility requirement, but do not see why value types are limited as well. As long as I provide a ValueTraits::NullValue and ValueTraits::Redirect any value could work, right?

For reference, a value would essentially be a POD struct:

struct flow_data {
    uint64_t timestamp;
    bool someflag;
    uint16_t another_value;
};

Please document what source files are needed for those who want to do that way

You acknowledge in the docs that some devs prefer just including the required headers + cpp files in their project. However it would be extremely useful if you, for those devs, could clearly explain precisely what headers + cpp files are required in order to do this (plus any compile flags or anything additional to be wary of).

Note - there are many many reasons / use-cases why (manually adding the required files to whatever ones build setup is) can be strongly preferred and often even as in my case really the only viable option. It's worth being aware that, whatever your opinions on the topic, many devs like me will simply write-off as 'not worth taking the time to work-out how to integrate it in order to try it' any library that doesn't make such a mode of inclusion clear+simple (thus especially favouring [single] header-only libraries. though that's likely not a sensible option here) ... and thus adding such an explanation would surely broaden the amount that your by all appearances rather useful work will be used.

How to process memory leak?

Hi @preshing

This is a great lib!

My situation is here:

  1. There is a global hashmap for managing sessions;
  2. There are several thread to create session and insert the session to the hashmap from different threads;
  3. In each thread, when its work finish, the session is erased from the global hashmap;

Can you tell me:

  1. when to call junction::QSBR::Context context = junction::DefaultQSBR.createContext();;
  2. when to call junction::DefaultQSBR.update(context);;
  3. when to call junction::DefaultQSBR.destroyContext(context);;

Thank you!

Conan package

Hello,
Do you know about Conan?
Conan is modern dependency manager for C++. And will be great if your library will be available via package manager for other developers.

Here you can find example, how you can create package for the library.

If you have any questions, just ask :-)

Deadlock when using Iterator

I found out that this example deadlocks in insert():

uint32_t random(uint32_t min, uint32_t max) {
    return min + (((double) rand() / (double(RAND_MAX) + 1)) * (max - min));
}
int main() {
    struct Data {
        Data(uint64_t id) : Id(id) { }
        uint64_t Id;
    };

    junction::ConcurrentMap_Leapfrog<uint64_t, Data *> map;
    std::mutex mut;

    std::vector<std::thread> threads;
    for (int i = 0; i < 16; ++i) {
        threads.push_back(std::thread([&] {
            auto qsbrCtx = junction::DefaultQSBR.createContext();
            while (true) {
                auto type = random(0, 5);
                switch (type) {
                    case 0: {
                        auto item = new Data(random(0, 1000));
                        std::lock_guard<std::mutex> lock(mut);
                        map.assign(item->Id, item);
                    } break;
                    case 1: {
                        if (auto *item = map.get(random(0, 1000))) {
                            // std::lock_guard<std::mutex> lock(mut);
                            map.erase(item->Id);
                        }
                    } break;
                    case 2: {
                        std::lock_guard<std::mutex> lock(mut);
                        junction::ConcurrentMap_Leapfrog<uint64_t, Data *>::Iterator it(map);
                        while (it.isValid()) {
                            if (it.getValue()->Id < 2) putchar('a');
                            it.next();
                        }
                    } break;
                    default: {
                        if (random(0, 10000) < 2)
                            printf("Thread %lx is running\n", std::hash<std::thread::id>()(std::this_thread::get_id()));  
                    } break;
                }
                junction::DefaultQSBR.update(qsbrCtx);
            }
        }));
    }
    threads.push_back(std::thread([&] {
        auto qsbrCtx = junction::DefaultQSBR.createContext();
        auto j = 0u;
        while (j++ < 10000000u) {
            if (auto *item = map.get(random(0, 1000))) {
                if (item->Id > 998) {
                    putchar('b');
                }
            }
            junction::DefaultQSBR.update(qsbrCtx);
        }
    }));
    for (auto &t : threads) {
        t.join();
    }

    return 0;
}

After running this deadlocks in junction/details/Leapfrog.h:235:

do {
    probeHash = cell->hash.load(turf::Acquire);
} while (probeHash == KeyTraits::NullHash);

If I remove case 2: from the example, there is no deadlock, so it appears that the iterator is causing some race condition. I used mutex on insert and iterator since comment for Iterator says

// The easiest way to implement an Iterator is to prevent all Redirects.
// The currrent Iterator does that by forbidding concurrent inserts.

In my case I insert and erase values very frequently from different threads, but occasionally I need to be able to iterate through all items (and inserts and erases may happen and that time too, hence the mutex).

Could you investigate and see if this is a bug in Junction or my mistake?

I'm attaching my complete example with makefile: junction-deadlock.zip

Any reason to call a member function of the object in QSBR?

TURF_CALL_MEMBER (*self->target, self->pmf)();

Since QSBR accepts a member function of the object here it's not clear how we can use this for object destruction. We cannot pass it the destructor of the object since that is forbidden in C++. It would be nice if we could pass arbitrary function here which takes the pointer (i.e. map element's value) as a parameter like this:

void enqueue(void (T::*func)(T*), T* target) {
       struct Closure {
            void (T::*func)(T*);
            T* target;
            static void thunk(void* param) {
                Closure* self = (Closure*) param;
                self->(*func)(target);
            }
        };
        Closure closure = {func, target};
        turf::LockGuard<turf::Mutex> guard(m_mutex);
        TURF_RACE_DETECT_GUARD(m_flushRaceDetector);
        m_deferredActions.push_back(Action(Closure::thunk, &closure, sizeof(closure)));
}

Then we could just call delete value in the callback function to free the object. For example:

struct Foo {
  static void destroy(Foo *ptr) {
    delete ptr;
  }
}

auto value= map.erase(key);
junction::DefaultQSBR.enqueue(&Foo::destroy, value);

I was wondering if there is any reason to enforce a member function in enqueue().

A possible deadlock

Hello,

The following code will be stuck at the second get() operation.
Is it a bug or I misuse Junction's API?

#include "junction/ConcurrentMap_Grampa.h"
int main() {
  junction::ConcurrentMap_Grampa<uint64_t, uint64_t> m;
  m.set(1, 0);
  m.get(1);
  m.set(1, 1);
  m.get(1);
}

According to my investigation, it is stuck at

table->jobCoordinator.participate() @ ConcurrentMap_Grampa::get()
    pair.condVar.wait(guard) @ SimpleJobCoordinator::participate()

Thx!

BUG : Stucked while use mutator.getValue, but no error or exit.

when i use these codes , it stucked at the ninth data, mutator.getValue() returns nothing and no error message.

            junction::ConcurrentMap_Leapfrog<turf::u32 ,turf::u32> HashTable;
            for(int k=0; k<9; ++k)
            {
                turf::u32 model_index = newI[k];
                turf::u32 scene_index = newJ[k];
                junction::ConcurrentMap_Leapfrog<turf::u32, unsigned int>::Mutator mutator = HashTable.insertOrFind(scene_index);
                if (mutator.getValue())
                {
                  //  turf::u32 thash = mutator.getValue();

                  //  if(distances[thash] > distances[model_index])
                    {
                        HashTable.exchange(scene_index,model_index);
                    }
                }
                else
                {
                    HashTable.exchange(scene_index,model_index);
                }
            }

where int newI[9] = {0,1,2,3,4,5,7,8,9};
int newJ[9] = {24814,75961,46974,77578,26819,70099,70420,31706,69132};

Currently using of iterator and assign.

While I am applying get() and assign() operations to LinearMap concurrently, deadlock always occurs with 4 thread. Then I checked readme, the Rules and Behavior said that

  • All of a Junction map's member functions, together with its Mutator member functions, are atomic with respect to each other, so you can safely call them from any thread without mutual exclusion.
  • In the current version, you must not assign while concurrently using an Iterator.

Above two entries seem contradict each other.

ConcurrentMap_Leapfrog::Hash decl

I think perhaps

using Hash = typename KT::Hash;

is more suitable that current

typedef typename turf::util::BestFit<Key>::Unsigned Hash;

insertOrFind unexpected behavior

Hi,

First of all, thank you for the great library and blog, they are great material !
I'm currently using ConcurrentMap_Leapfrog and got excellent performances and results.
But, when I integrated with one of my libraries, my tests started to fail spuriously.
I started looking at the issue, and after a look of debugging, I was surprised by the behavior of insertOrFind .
I expect insertOrFind to work as follow:

mut = insertOrFind(k)
assert(mut.getValue() == ValueTraits<>::NullValue)
mut = insertOrFind(k)
assert(mut.getValue() == ValueTraits<>::Redirect)

But, I encounter sometimes, that on the second getValue I get a NullValue instead of Redirect.

I made the following concrete :

#include <mutex>
#include <thread>
#include <vector>
#include <atomic>
#include <cassert>
#include <junction/ConcurrentMap_Leapfrog.h>

using Key = size_t;
using Value = int64_t;

template <typename T>
struct ValueTraits {
    using IntType = T;
    using Value = T;

    static constexpr IntType NullValue = 0;
    static constexpr IntType Redirect = static_cast<T>(-1); // I want -1, not 1 by default for redirect value
};

using hash_map = junction::ConcurrentMap_Leapfrog<Key, Value, junction::DefaultKeyTraits<Key>,
      ValueTraits<Value>>;
using vec_t = std::vector<std::atomic<size_t>>;

void insert(hash_map& m, vec_t& vec, size_t len) {
    for (size_t i = 0; i < len ; ++i) {
        auto mutator = m.insertOrFind(i + 1);
        auto to_insert = mutator.getValue() == ValueTraits<Value>::NullValue;

        if (to_insert)
        {
            mutator.exchangeValue(i + 1);
            ++vec[i];
        }
        assert(vec[i]!=2); // can assert here
    }
}

int main()
{
    constexpr size_t map_size = 2048;
    hash_map m(map_size<<1);
    vec_t vec(map_size);

    for (auto& v : vec)
        v = 0;

    auto t1 = std::thread(insert, std::ref(m), std::ref(vec), map_size);
    auto t2 = std::thread(insert, std::ref(m), std::ref(vec), map_size);

    t1.join();
    t2.join();

    return 0;
}

The example does not use directly the value of Redirect but I should normally only insert one time in the map (It reproduce the behavior of my use case).
When you run this example, sometimes in does thrown an assert, sometimes it fails in the assert.

Do I make a wrong assumption somewhere or did I implement something wrong?

Thank you for any help you can provide me with.
I tried with Grampa too, same behavior.

Best,
Julián

Help reproducing benchmark values

I came across this post and I'm trying to run the benchmarks on my machine (2x16-core xeon e5 2620v4 @2.1Ghz). I expect to get worse performance at lower core-counts due to the lower clock speed, but I currently get substantially (>2x) worse performance than the post running MapScalabilityTest (where I believe time is reported in seconds):

{
'mapType': 'Junction Grampa map',
'population': 32000,
'readsPerWrite': 4,
'itersPerChunk': 10000,
'chunks': 200,
'keepChunkFraction': 1.000000,
'labels': ('numThreads', 'mapOpsDone', 'totalTime'),
'points': [
    (1, 20000000, 3.130913),
    (2, 39271012, 6.792609),
    (3, 60127378, 10.934400),
    (4, 80936978, 15.173613),
    (5, 102136528, 20.520561),
    (6, 122053852, 25.645662),
    (7, 142280994, 30.308151),
    (8, 162408667, 35.968076),
    (9, 183720392, 44.163389),
    (10, 203532431, 52.606002),
    (11, 223287034, 60.265506),
    (12, 241929131, 68.742479),
    (13, 260322337, 75.837455),
    (14, 278162335, 82.871782),
    (15, 297701998, 88.851133),
    (16, 316862143, 96.579740),
],
}

Are there any tests I can perform to ensure that I'm building this correctly (e.g. that optimizations are turned on)?

Crash in Debug lib.

Hi,
I have a crash when I linked the lib which build following.

$ git clone https://github.com/preshing/junction.git
$ git clone https://github.com/preshing/turf.git
$ cd junction
$ cmake -DCMAKE_INSTALL_PREFIX=~/junction-install -DJUNCTION_WITH_SAMPLES=OFF ..
$ cmake --build . --target install --config Debug

And the test file is 'MapScalabilityTests'.
Here is the callstack.
image

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.