(gdb) bt
#0 0x00007ffff51430ed in cxa_throw () from /lib64/libstdc++.so.6
#1 0x00007ffff2125889 in impl::table_base<persist_fixed_string, persist_fixed_string, (anonymous namespace)::pstr_hash, std::equal_to<persist_fixed_string >, pobj_cache_aligned_allocator<std::pair<persist_fixed_string const, persist_fixed_string > >, dummy::shared_mutex>::make_space_for_insert (this=this@entry=0x7fffee17f8b0,
bi=3321, b_dst_lock=...) at /home/danielwaddington/comanche/src/components/store/hstore/src/table.tcc:347
#2 0x00007ffff21285fd in impl::table_base<persist_fixed_string, persist_fixed_string, (anonymous namespace)::pstr_hash, std::equal_to<persist_fixed_string >, pobj_cache_aligned_allocator<std::pair<persist_fixed_string const, persist_fixed_string > >, dummy::shared_mutex>::emplace<std::piecewise_construct_t const&, std::tuple<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > >&&, __gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > >&&, pmemobjpool* const&, unsigned long const&, char const (&) [4]>, std::tuple<char const*&, char const*&&, pmemobjpool* const&, unsigned long const&, char const (&) [6]> > (this=0x7fffee17f8b0)
at /home/danielwaddington/comanche/src/components/store/hstore/src/table.tcc:470
#3 table<persist_fixed_string, persist_fixed_string, (anonymous namespace)::pstr_hash, std::equal_to<persist_fixed_string >, pobj_cache_aligned_allocator<std::pair<persist_fixed_string const, persist_fixed_string > >, dummy::shared_mutex>::emplace<std::piecewise_construct_t const&, std::tuple<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > >&&, __gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > >&&, pmemobjpool* const&, unsigned long const&, char const (&) [4]>, std::tuple<char const*&, char const*&&, pmemobjpool* const&, unsigned long const&, char const (&) [6]> > (this=) at /home/danielwaddington/comanche/src/components/store/hstore/src/hop_hash.h:565
#4 hstore::put (this=, pool=, key=..., value=0x7fffe1800039, value_len=64)
at /home/danielwaddington/comanche/src/components/store/hstore/src/hstore.cpp:562
#5 0x00000000004162bf in Dawn::Shard::process_message_IO_request (this=this@entry=0x7fffffffe1f0,
handler=handler@entry=0x7fffec0093f0, msg=0x7fffe1800000)
at /home/danielwaddington/comanche/src/servers/dawn/src/shard.cpp:345
#6 0x000000000041665b in Dawn::Shard::main_loop (this=this@entry=0x7fffffffe1f0)
at /home/danielwaddington/comanche/src/servers/dawn/src/shard.cpp:133
#7 0x000000000040b0ad in Dawn::Shard::thread_entry (this=0x7fffffffe1f0, po=...)
at /home/danielwaddington/comanche/src/servers/dawn/src/shard.h:80
#8 0x00007ffff516f53f in execute_native_thread_routine () from /lib64/libstdc++.so.6
#9 0x00007ffff6b8050b in start_thread () from /lib64/libpthread.so.0
#10 0x00007ffff489038f in clone () from /lib64/libc.so.6

I have successfully built comanche, but I don't know how to run fio with it.

In SPDK, I used this command to run fio.
LD_PRELOAD=<path/to/spdk>/examples/nvme/fio_plugin/fio_plugin <path/to/fio>/fio <path/to/spdk>/examples/nvme/fio_plugin/example_config.fio

With SPDK, I specified spdk as the ioengine for my fio script. However, I can't figure out what I should specify as the ioengine for comanche. Is there a specific ioengine for comanche to use with fio?

If I use the same command, will there be any difference between using spdk and comanche?

README.md says that Ubuntu 16.04 is the (only) tested environment, yet I can no longer get cmake to work there (in unstable branch):

CMake Error at CMakeLists.txt:1 (cmake_minimum_required):
CMake 3.6 or higher is required. You are running version 3.5.1

Currently for each worker thread, the tool shares a connection (i.e. instance of the storage component created through the factory). This limits scaling.

What we need is for each worker thread, a new instance of the component is created for that thread.

As a temporary measure, pools for Dawn are called "/dev/dax0.0", "/dev/dax0.1", etc.

I'd like to tweak kvstore-perf so that I can provide a prefix (with the --path option?) e.g., "/dev/dax0." and it will add a value counting from 0 for each worker thread.

Thus for option --path "/dev/dax0."
worker 0 will use path /dev/dax0.0
worker 1 will use path /dev/dax0.1 etc.

This is a temporary measure until we have a dax allocator.

Do we need special configuration for fabric component in dawn server? (drivers, linux kernel, etc?)
I am having this error in ribbit 4 (full log at dawn_fabric_err.log)

:

(base) fengggli@ribbit5(:):~/WorkSpace/comanche/build$./src/servers/dawn/dawn --config ./nvmestore-dawn.conf
[LOG]:config_file: (./nvmestore-dawn.conf)
[LOG]:shards type:Array
[LOG]:shard: core(0) port(11912) net(mlx5_0)
[+] launching shard: core(0) port(11912) net(mlx5_0)
[LOG]:Loaded component: libcomanche-fabric.so
terminate called after throwing an instance of 'fabric_runtime_error'
  what():  fabric_runtime_error (61) "No data available" at /home/fengggli/WorkSpace/comanche/src/components/net/fabric/src/fabric.cpp:111 fi_info:

My dawn configuration file

{
    "shards" :
    [
        {
            "core": 0,
            "port": 11912,
            "net" : "mlx5_0",
            "default_backend" : "mapstore"
        }
    ],
    "net_providers" : "verbs"
}

kernel configuration

(base) fengggli@ribbit5(:):~/WorkSpace/comanche/build$uname -a
Linux ribbit5 4.15.0-50-generic #54-Ubuntu SMP Mon May 6 18:46:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
(base) fengggli@ribbit5(:):~/WorkSpace/comanche/build$cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.15.0-50-generic root=UUID=0fed1c41-5b15-11e9-a62a-40f2e92c3c5a ro intel_iommu=on hugepagesz=2M hugepages=4096 text memmap=2G!4G

I also noticed the --enable-mlx=no in libfabric CMakelists.txt, is this intended?

I'd like an option on kvstore-perf (e.g., --multi-endpoint) which will open up a different connection for each worker thread where each connection is on a different port that is incremented from the base port.

I need this so we can test dawn's server side scale out which means starting multiple shards (one thread per shard) on a different network port.

Travis are now recommending removing the sudo tag.

"If you currently specify sudo: false in your .travis.yml, we recommend removing that configuration"

I added the following test function in the "src/components/block/posix/unit_test/test1.cpp"(before the "BasicAsync" test).
It uses one 4k page(mem) as a buffer and first writes 0xcc into each byte of the first block,and 0xff into each byte of the second block.
It's more like a stress test where the values are frequently read in and written back between one memory page and two logical blocks. (it also represents one extreme case in Pager_Simple: there is only once physical page and two blocks for swapping)
But the test cannot pass in my qemu vm.
Could you help to confirm it in your side?

Thanks

TEST_F(Block_posix_test, AsyncSwap)                                                                                                                                                                                                           
{   
    uint64_t nr_iterations = 10000;
    io_buffer_t mem = _block->allocate_io_buffer(4096,4096,Component::NUMA_NODE_ANY);
    void * ptr = _block->virt_addr(mem);
    char *p = (char*) ptr;
    
    uint64_t tag;
    
    memset(ptr, 0x0, 4096);
    
    /* two blocks, 1 filled with 0xcc and the other with 0xff*/
    for(unsigned i=0;i<4096;i++) p[i] = 0xcc;
    
    tag = _block->async_write(mem, 0, 1, 1);
    
    for(unsigned i=0;i<4096;i++) p[i] = 0xff;
    
    for(uint64_t i = 0; i < nr_iterations; i++){
        /*
         * write second blk and read first blk
         */
        while(!_block->check_completion(tag)) {
            cpu_relax();
        }
        tag = _block->async_write(mem, 0, 2, 1);
        
        _block->read(mem, 0, 1, 1);
        
        if(p[0] !=0xcc){
            PWRN("eeek on blk 1!! p[0]=0x%x%x",0xf&(p[0]>>4), 0xf &p[0]);
            ASSERT_TRUE(0);
        }
        
        /*
         * write first blk and read second blk
         */
        while(!_block->check_completion(tag)) {
            cpu_relax();
        }
    
        tag = _block->async_write(mem, 0, 1, 1);
        _block->read(mem, 0, 2, 1);
    
        if(p[0] !=0xff){
            PWRN("eeek on blk 2!! p[0]=0x%x%x",0xf&(p[0]>>4), 0xf &p[0]);
            ASSERT_TRUE(0);
        }
    
    }
    _block->free_io_buffer(mem);
    PMAJOR("> async swap test OK");

}

The first byte of block 1 should be read as 0xcc, but 0xff is read instead:

root@node0:/home/lifeng/comanche/src/components/block/posix/unit_test# ./blockposix-test1 /home/lifeng/comanche/blockfile.dat 
[==========] Running 9 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 9 tests from Block_posix_test
[ RUN      ] Block_posix_test.InstantiateBlockDevice
[LOG]:Load path: libcomanche-blkposix.so
[LOG]:Creating 'Block_posix' factory.
[LOG]:Block_device factory loaded OK.
[LOG]:config: {"path":"/home/lifeng/comanche/blockfile.dat","size_in_blocks" : 100 }
[LOG]:Block_posix: path=(/home/lifeng/comanche/blockfile.dat) size=(100)
[LOG]:Block_posix: XMS available!
[LOG]:Using API defined memory limit 256 MB
[LOG]:CPU count: 4
EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles !
# DPDK EAL set maps & initialized OK.
# DPDK EAL initialized ok (--proc-type=primary).
[LOG]:unloading component (0x1f26f50)
POSIX-based block-layer component loaded OK.
[       OK ] Block_posix_test.InstantiateBlockDevice (825 ms)
[ RUN      ] Block_posix_test.AsyncSwap
[WRN]: eeek on blk 1!! p[0]=0xff
/home/lifeng/comanche/src/components/block/posix/unit_test/test1.cpp:121: Failure
Value of: 0
  Actual: false
Expected: true

Currently the tool only supports N number of cores, counting from core 0. We need to be able to support a range of specific cores. They should be of the format..

1,2,3,6-10,12-40

Hey there! I'm a TSC member of JSON Schema focused on building better and faster JSON Schema utilities for C++.

I saw this repo makes use of RapidJSON and its (limited) JSON Schema implementation. How is your use of RapidJSON going? Anything you like or dislike? Did you find a need for supporting newer dialects of JSON Schema than Draft 4?

Any honest feedback would be super appreciated to make JSON Schema and my C++ efforts in there more valuable! 🙏🏻

We want all apps/components use the tbb installed to $INSTALL. But the deps/apt-get.sh trys to install libtbb to system directory. I have removed it in the latest commit in unstable 31dd3eb .

It will work out of box for new clone. But if you pull this commit from a existing build, you might need to delete all the contents in the build directory and rerun cmake+make bootstrap+make +make install. (cmake might have cached the tbb install path as /usr/lib/libtbb.so, which you don't want)

TBB should be once off build and install.

The Google "benchmark" submodule test cxx03_test.cc fails to compile when it detects that the compiler is C++11 or newer. Although Google benchmark specified -std=c++03 for that test, someone overrides that with -std=gnu++11.

Workaround:
cmake -DBENCHMARK_ENABLE_TESTING=off ..

this(1d88cb5) failed because no gperftools installed:

I see two version of gperftools in comanche tree:

1. comanche/deps/fetch-deps.sh has two optional packages (google-perftools libgoogle-perftools-dev), but neither of them are enabled.
2. gperftools submodule at comanche/src/lib/gperftools, is this a must to use the most updated version(compared with the shipped version with ubuntu)? Also there is no build script for this in Comanche CMake for gperftool.

CUDA should not be compulsory. If its not found, then the CUDA components should not be built.

CUDA_TOOLKIT_ROOT_DIR not found or specified
-- Could NOT find CUDA (missing: CUDA_TOOLKIT_ROOT_DIR CUDA_NVCC_EXECUTABLE CUDA_INCLUDE_DIRS CUDA_CUDART_LIBRARY)
-- Configuring incomplete, errors occurred!

src/components/examples/sample and src/components/examples/python-sample specify specify -fdump-class-hierarchy, which is refused by clang

Recent build passed the CI but if you look at the build log(https://travis-ci.com/IBM/comanche/jobs/159882003#L8497-L8542), only "bootstrap" targets are built (and actually failed), component related targets are not..
This might also be the reason successful build in comanche but failing building in local machines..

• It looks like libmemkind failed to build because of "virtual memory exhausted: Cannot allocate memory".
  https://github.com/dwaddington/memkind/blob/629f1da866c6a7ade35e55bf825e827b66b90a96/build.sh#L47
  libmemkind/build.sh lauches as many tasks as possible by default.
  We should disable it by passing "MAKEOPTS=#tasks" in the comanche CMakelists.txt

As well as IOPS the tool should report MiB/s throughput (2 decimal points).

a put should replace any existing k,v pairs that match. If the new put is a different size, then the object should be reallocated. If the new put is the same size, then it should be updated in place.

The function rte_malloc_virt2phy() is renamed to rte_malloc_virt2iova().

for small put on DAWN+hstore, my unit test reports 134K ish where as your kvstore-perf tool reports less, ~90K IOPs. Somebody is measuring something incorrectly.

A fresh build today gave me this...; its not the first time I have hit this problem. Do we need a branch/tag on the flatbuffer version?

[ 93%] Building CXX object src/fuse/ustack/CMakeFiles/iotest.dir/src/iotest.cpp.o
In file included from /home/danielwaddington/comanche/src/fuse/ustack/src/ustack_client.h:14:0,
from /home/danielwaddington/comanche/src/fuse/ustack/src/iotest.cpp:9:
/home/danielwaddington/comanche/build/src/fuse/ustack/src/protocol_generated.h: In member function ‘bool Protocol::ElementMemoryReply::Verify(flatbuffers::Verifier&) const’:
/home/danielwaddington/comanche/build/src/fuse/ustack/src/protocol_generated.h:300:21: error: ‘class flatbuffers::Verifier’ has no member named ‘VerifyString’
verifier.VerifyString(shmem_id()) &&
^
/home/danielwaddington/comanche/build/src/fuse/ustack/src/protocol_generated.h: In member function ‘bool Protocol::ElementChannelReply::Verify(flatbuffers::Verifier&) const’:
/home/danielwaddington/comanche/build/src/fuse/ustack/src/protocol_generated.h:361:21: error: ‘class flatbuffers::Verifier’ has no member named ‘VerifyString’
verifier.VerifyString(uipc_id()) &&
^
src/fuse/ustack/CMakeFiles/iotest.dir/build.make:62: recipe for target 'src/fuse/ustack/CMakeFiles/iotest.dir/src/iotest.cpp.o' failed
make[2]: *** [src/fuse/ustack/CMakeFiles/iotest.dir/src/iotest.cpp.o] Error 1
CMakeFiles/Makefile2:3595: recipe for target 'src/fuse/ustack/CMakeFiles/iotest.dir/all' failed
make[1]: *** [src/fuse/ustack/CMakeFiles/iotest.dir/all] Error 2
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2

PMDK depends on a sys present libfabric e.g., /usr/local/lib

It should use the libfabric in the install prefix /lib