fengggli / comanche Goto Github PK

This project forked from ibm/comanche

comanche

License: Apache License 2.0

CMake 0.79% C++ 59.71% Cuda 0.10% Python 0.04% Shell 2.80% Makefile 0.43% C 32.50% M4 0.02% Batchfile 0.01% HTML 3.16% Perl 0.05% Roff 0.24% Java 0.01% Emacs Lisp 0.11% Dockerfile 0.02% CSS 0.01%

comanche's People

Contributors

Watchers

Forkers

dwaddington

comanche's Issues

fs-integration

TODO

File operations:
*remove file
*append to file
Capture system call
- open/close
- read/write
systemutils
fio test
- simple implementation
- dual-channel version.
Intercept
- read/write can be interepted.
- experiment 1 show overhead of fuse by tracing. (why set max_write as 128KB? is the default lower than this? see this issue 161 in libfuse )
- experiment 2 can be comparing performance of dual path or all direct.
- Explain differences of this with direct-fuse and fusionfs. (zero-copy in userspace?)
- dd/ perf

mcas memory register in nvmestore

The second mmap has MAP_PRIVATE, so dont memset. otherwise another private copy(some un-aligned physical address will be used)
I verified both virtual addr and phys_addr are 2M aligned, but still failed in regestration.
The previous cuda-dma seems to work, what happened to this hugepage memory?

Previously I had tried to use newer dpdk(fengggli/dpdk#1), but failed

Couldn't get fd no hugepage file

This happens sometime if I didn' exit dpdk application peacefully.

(py36) lifen@sievert(:):~/Workspace/comanche/build$./src/components/block/nvme/unit_test/blocknvme-test1 20:00.0       │[  +0.000002] mcas: exposure added (pid=7788, token=feeb, paddr=79f400000, size=4194304)
[==========] Running 3 tests from 1 test case.                                                                         │[  +0.000000] walk result: 79f400000
[----------] Global test environment set-up.                                                                           │[  +0.000008] mcas: mmap called --------------------
[----------] 3 tests from Block_nvme_test                                                                              │[  +0.000001] mcas: mmap flags=fb token=feeb
[ RUN      ] Block_nvme_test.InstantiateBlockDevice                                                                    │[  +0.000001] mcas: mmap remapping (79f400000,4194304)
rte_virtio_pmd_init(): IOPL call failed - cannot use virtio PMD                                                        │[  +0.000001] mcas: vm_start=700000000 vm_end=700400000 delta=4194304
[LOG]:Using API defined memory limit 48 MB                                                                             │[  +0.000019] mcas: mmap OK
[LOG]:CPU count: 24                                                                                                    │[  +2.088501] mcas: vm_close
EAL: No free hugepages reported in hugepages-1048576kB                                                                 │[  +0.000010] mcas: ioctl (9) (arg=7fffa0bbeef0)
EAL: Couldn't get fd on hugepage file                                                                                  │[  +0.000001] mcas: calling remove_exposure
EAL: FATAL: Cannot init memory                                                                                         │[  +0.000000] mcas: remove_exposure (token=feeb)
                                                                                                                       │[  +0.000001] mcas: removed exposure feeb
EAL: Cannot init memory                                                                                                │(py36) lifen@sievert(:):~/Workspace/comanche$ls /dev/
                                                                                                                       │Display all 223 possibilities? (y or n)
Exception:API error                                                                                                    │(py36) lifen@sievert(:):~/Workspace/comanche$ls /dev/ -alt|grep huge
Trace/breakpoint trap (core dumped)

virtual machine using vagrant

spdk provides a script to run nvme-enabled virtual machine in vagrant. See the instructions

Steps.

install vbox and vbox extension instruction
install vagrant instruction
Follow the instruction [here] (https://spdk.io/doc/vagrant.html)

kernel

needs to install kernel which has linux-headers provided by ubuntu. to compile kernel module

sudo apt-get install linux-generic

to allocate hugepage, use spdk/scripts/setup.sh

ndctl

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/storage_administration_guide/ch-persistent-memory-nvdimms

nvmestore performance(also with DAWN)

nvme component

put/get/get_direct of 4k, 64k, 1M, 16M, 256M
latency bins.
latency across time

dawn+ hstore

put/get/get_direct 4k, 64k (1M&16M 256M needs special attention because of the two-stage)
lantency bins.
latency across time

vfio related

kernel doc (https://github.com/torvalds/linux/blob/master/Documentation/vfio.txt)
a video(https://www.youtube.com/watch?v=WFkdTFTOTpA) and discussion(https://www.redhat.com/archives/vfio-users/2018-February/msg00010.html)
VFIO_IOMMU_MAP_DMA succeeds only on second try

[illegal free after upgrading to new dpdk/spdk]

After upgrading dpdk/spdk, the src/lib/core unit test failed.

Debug output

(base) fengggli@ribbit5(:):~/WorkSpace/comanche/build$gdb ./src/lib/core/unit_test/core-test1
GNU gdb (Ubuntu 8.1-0ubuntu3) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./src/lib/core/unit_test/core-test1...done.
(gdb) r
Starting program: /home/fengggli/WorkSpace/comanche/build/src/lib/core/unit_test/core-test1
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[==========] Running 3 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 3 tests from Core_test
[ RUN      ] Core_test.DPDKInit
[LOG]:Using API defined memory limit 32 MB
[LOG]:CPU count: 48
[New Thread 0x7ffff53f1700 (LWP 47948)]
[New Thread 0x7ffff4bf0700 (LWP 47949)]
EAL: No available hugepages reported in hugepages-1048576kB
...
...
[LOG]:##- freeing DPDK memory: 0x1011e5000

Thread 1 "core-test1" received signal SIGSEGV, Segmentation fault.
0x00007ffff6f06898 in malloc_heap_free (elem=<optimized out>)
    at /home/fengggli/WorkSpace/comanche/src/lib/dpdk/dpdk-19.05/lib/librte_eal/common/malloc_heap.c:888
888                     if (tmp->flags & RTE_MEMSEG_FLAG_DO_NOT_FREE) {
(gdb) bt
#0  0x00007ffff6f06898 in malloc_heap_free (elem=<optimized out>)
    at /home/fengggli/WorkSpace/comanche/src/lib/dpdk/dpdk-19.05/lib/librte_eal/common/malloc_heap.c:888
#1  0x00007ffff6f0079c in rte_free (addr=addr@entry=0x1011e5000)
    at /home/fengggli/WorkSpace/comanche/src/lib/dpdk/dpdk-19.05/lib/librte_eal/common/rte_malloc.c:34
#2  0x00007ffff74843df in Core::Physical_memory::free_io_buffer (this=<optimized out>, io_mem=4313731072)
    at /home/fengggli/WorkSpace/comanche/src/lib/core/src/physical_memory.cpp:118
#3  0x000055555555ea37 in (anonymous namespace)::Core_test_MemoryAllocation_Test::TestBody (this=<optimized out>)
    at /home/fengggli/WorkSpace/comanche/src/lib/core/unit_test/test1.cpp:64
#4  0x00005555555890f1 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) ()
#5  0x0000555555583097 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) ()
#6  0x00005555555668d6 in testing::Test::Run() ()
#7  0x000055555556720c in testing::TestInfo::Run() ()
#8  0x0000555555567896 in testing::TestCase::Run() ()
#9  0x000055555556e762 in testing::internal::UnitTestImpl::RunAllTests() ()
#10 0x000055555558a2b7 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) ()
#11 0x0000555555583ed3 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) ()
#12 0x000055555556d326 in testing::UnitTest::Run() ()
#13 0x000055555555c2ef in RUN_ALL_TESTS () at /usr/local/include/gtest/gtest.h:2233
#14 main (argc=<optimized out>, argv=<optimized out>)
    at /home/fengggli/WorkSpace/comanche/src/lib/core/unit_test/test1.cpp:104

[hstore instructions]

This tracks how I could use hstore

minimal memory mapping test

The code is in https://github.com/fengggli/comanche/tree/mapping-for-clem
There is a readme file(how to prepare, build and test)at src/components/store/nvmestore/testing/mcas-nvmestore/README.md

BlockAllocator Selector for nvmestore

Let nvmestore use different blockallocator during factorization.
if blkalloc_ikv is used, the same _meta_store is used to store block allocation information

fengggli / comanche Goto Github PK

comanche's People

Contributors

Watchers

Forkers

comanche's Issues

TODO

Steps.

kernel

nvme component

dawn+ hstore

Notes for dawn.

Shards

Recommend Projects

Recommend Topics

Recommend Org