couchbaselabs / cbforest Goto Github PK
View Code? Open in Web Editor NEWC++ wrapper library around ForestDB, for use in Couchbase Lite.
C++ wrapper library around ForestDB, for use in Couchbase Lite.
This is used in a number of places by both the ForestDB storage engine itself and query logic
04-07 17:06:13.559 1663-1672/com.couchbase.cbforest.test A/libc: Fatal signal 11 (SIGSEGV) at 0x000000d0 (code=1), thread 1672 (FinalizerDaemon)
04-07 17:06:13.643 1663-1676/com.couchbase.cbforest.test I/TestRunner: finished: testIndexVersion(com.couchbase.cbforest.C4ViewTest)
04-07 17:06:13.643 1663-1676/com.couchbase.cbforest.test I/TestRunner: passed: testIndexVersion(com.couchbase.cbforest.C4ViewTest)
04-07 17:06:13.643 1663-1676/com.couchbase.cbforest.test I/TestRunner: started: testQueryIndex(com.couchbase.cbforest.C4ViewTest)
04-07 17:06:13.659 146-146/? I/DEBUG: *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
04-07 17:06:13.659 146-146/? I/DEBUG: Build fingerprint: 'generic/vbox86p/vbox86p:4.4.4/KTU84P/eng.buildbot.20151118.000452:userdebug/test-keys'
04-07 17:06:13.659 146-146/? I/DEBUG: Revision: '0'
04-07 17:06:13.659 146-146/? I/DEBUG: pid: 1663, tid: 1672, name: FinalizerDaemon >>> com.couchbase.cbforest.test <<<
04-07 17:06:13.659 146-146/? I/DEBUG: signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 000000d0
04-07 17:06:13.663 534-609/system_process D/dalvikvm: GC_FOR_ALLOC freed 709K, 18% free 5713K/6908K, paused 6ms, total 6ms
04-07 17:06:13.811 146-146/? I/DEBUG: eax 000000c8 ebx 951e55bc ecx 000000c8 edx 00000000
04-07 17:06:13.811 146-146/? I/DEBUG: esi 95b48930 edi 95b489dc
04-07 17:06:13.811 146-146/? I/DEBUG: xcs 00000073 xds 0000007b xes 0000007b xfs 00000000 xss 0000007b
04-07 17:06:13.811 146-146/? I/DEBUG: eip 94ff515b ebp 95b489f8 esp 95b488f8 flags 00210202
04-07 17:06:13.811 146-146/? E/Corkscrew: unrecognized dwarf lower part encoding: 0x37
04-07 17:06:13.811 146-146/? I/DEBUG: #00 pc 0016515b /data/app-lib/com.couchbase.cbforest.test-1/libCouchbaseLiteJavaForestDB.so (cbforest::Index::isBusy() const+11)
04-07 17:06:13.811 146-146/? I/DEBUG: #01 pc 0019e9c7 /data/app-lib/com.couchbase.cbforest.test-1/libCouchbaseLiteJavaForestDB.so (c4View::checkNotBusy(C4Error*)+55)
04-07 17:06:13.811 146-146/? I/DEBUG: #02 pc 0019e7c6 /data/app-lib/com.couchbase.cbforest.test-1/libCouchbaseLiteJavaForestDB.so (c4view_close+54)
04-07 17:06:13.811 146-146/? I/DEBUG: 95b488b8 00000000
04-07 17:06:13.811 146-146/? I/DEBUG: 95b488bc b4da7729 /system/lib/libdvm.so (dvmDecodeIndirectRef(Thread*, _jobject*)+233)
04-07 17:06:13.811 146-146/? I/DEBUG: 95b488c0 b937bd00 [heap]
04-07 17:06:13.811 146-146/? I/DEBUG: 95b488c4 1d800001
04-07 17:06:13.811 146-146/? I/DEBUG: 95b488c8 b939e238 [heap]
04-07 17:06:13.811 146-146/? I/DEBUG: 95b488cc b4ef2cb4 /system/lib/libdvm.so
04-07 17:06:13.811 146-146/? I/DEBUG: 95b488d0 00000000
04-07 17:06:13.811 146-146/? I/DEBUG: 95b488d4 00000000
04-07 17:06:13.811 146-146/? I/DEBUG: 95b488d8 b937bc58 [heap]
04-07 17:06:13.811 146-146/? I/DEBUG: 95b488dc b4ef2cb4 /system/lib/libdvm.so
04-07 17:06:13.811 146-146/? I/DEBUG: 95b488e0 00000000
04-07 17:06:13.811 146-146/? I/DEBUG: 95b488e4 00000000
04-07 17:06:13.811 146-146/? I/DEBUG: 95b488e8 b937bc58 [heap]
04-07 17:06:13.811 146-146/? I/DEBUG: 95b488ec b4dab542 /system/lib/libdvm.so (SetLongField(_JNIEnv*, _jobject*, _jfieldID*, long long)+130)
04-07 17:06:13.811 146-146/? I/DEBUG: 95b488f0 b937bc58 [heap]
04-07 17:06:13.811 146-146/? I/DEBUG: 95b488f4 00000007
04-07 17:06:13.811 146-146/? I/DEBUG: #00 95b488f8 000000c8
04-07 17:06:13.811 146-146/? I/DEBUG: 95b488fc 9502e9c8 /data/app-lib/com.couchbase.cbforest.test-1/libCouchbaseLiteJavaForestDB.so (c4View::checkNotBusy(C4Error*)+56)
04-07 17:06:13.811 146-146/? I/DEBUG: #01 95b48900 000000c8
04-07 17:06:13.811 146-146/? I/DEBUG: 95b48904 b939f398 [heap]
04-07 17:06:13.811 146-146/? I/DEBUG: 95b48908 ffff0208
04-07 17:06:13.811 146-146/? I/DEBUG: 95b4890c 951e55bc /data/app-lib/com.couchbase.cbforest.test-1/libCouchbaseLiteJavaForestDB.so
04-07 17:06:13.811 146-146/? I/DEBUG: 95b48910 b4dabd00 /system/lib/libdvm.so (GetLongField(_JNIEnv*, _jobject*, _jfieldID*))
04-07 17:06:13.811 146-146/? I/DEBUG: 95b48914 1d800001
04-07 17:06:13.811 146-146/? I/DEBUG: 95b48918 951e55bc /data/app-lib/com.couchbase.cbforest.test-1/libCouchbaseLiteJavaForestDB.so
04-07 17:06:13.811 146-146/? I/DEBUG: 95b4891c 95b489b8 [stack:1672]
04-07 17:06:13.811 146-146/? I/DEBUG: 95b48920 00000000
04-07 17:06:13.811 146-146/? I/DEBUG: 95b48924 1d800001
04-07 17:06:13.811 146-146/? I/DEBUG: 95b48928 951e55bc /data/app-lib/com.couchbase.cbforest.test-1/libCouchbaseLiteJavaForestDB.so
04-07 17:06:13.811 146-146/? I/DEBUG: 95b4892c 9502e7c7 /data/app-lib/com.couchbase.cbforest.test-1/libCouchbaseLiteJavaForestDB.so (c4view_close+55)
04-07 17:06:13.811 146-146/? I/DEBUG: 95b48930 00000000
04-07 17:06:13.811 146-146/? I/DEBUG: 95b48934 95b489b8 [stack:1672]
04-07 17:06:13.811 146-146/? I/DEBUG: 95b48938 9e1245a4 /dev/ashmem/dalvik-LinearAlloc (deleted)
04-07 17:06:13.811 146-146/? I/DEBUG: 95b4893c 95035759 /data/app-lib/com.couchbase.cbforest.test-1/libCouchbaseLiteJavaForestDB.so (_JNIEnv::SetLongField(_jobject*, _jfieldID*, long long)+105)
04-07 17:06:13.867 534-556/system_process I/BootReceiver: Copying /data/tombstones/tombstone_03 to DropBox (SYSTEM_TOMBSTONE)
04-07 17:06:13.871 534-1680/system_process W/ActivityManager: Error in app com.couchbase.cbforest.test running instrumentation ComponentInfo{com.couchbase.cbforest.test/android.test.InstrumentationTestRunner}:
04-07 17:06:13.871 534-1680/system_process W/ActivityManager: Native crash
04-07 17:06:13.871 534-1680/system_process W/ActivityManager: Native crash: Segmentation fault
@borrrden ran into a crash where a C4QueryEnumerator is used (or freed) after its C4View is closed. Closing the C4View calls fdb_close
so the database handle is invalidated, including any fdb_iterators.
It would be a good idea to add some internal refcounting on C4View and C4Database so an enumerator keeps the reference alive.
This is needed to load the appropriate document body when a placeholder ("*") is found.
Please see attached compilation error message
/Users/hideki/github/couchbase-lite-android/libraries/couchbase-lite-java-forestdb/jni/../vendor/cbforest/C/c4Document.cc:481:5: error: use of undeclared identifier 'SecureRandomize'
SecureRandomize({r, sizeof(r)});
^
/Users/hideki/github/couchbase-lite-android/libraries/couchbase-lite-java-forestdb/jni/../vendor/cbforest/C/c4Document.cc:500:9: error: unknown type name 'md5Context'
md5Context ctx;
^
/Users/hideki/github/couchbase-lite-android/libraries/couchbase-lite-java-forestdb/jni/../vendor/cbforest/C/c4Document.cc:502:35: error: no matching function for call to 'min'
uint8_t revLen = (uint8_t)std::min(parentRevID.size, 255ul);
^~~~~~~~
/Users/hideki/Library/Android/sdk/ndk-bundle/sources/cxx-stl/gnu-libstdc++/4.8/include/bits/algorithmfwd.h:367:5: note: candidate template ignored: deduced conflicting types for parameter '_Tp' ('unsigned int' vs. 'unsigned long')
min(const _Tp&, const _Tp&);
^
/Users/hideki/Library/Android/sdk/ndk-bundle/sources/cxx-stl/gnu-libstdc++/4.8/include/bits/stl_algo.h:4226:5: note: candidate template ignored: could not match 'initializer_list<type-parameter-0-0>' against 'unsigned int'
min(initializer_list<_Tp> __l, _Compare __comp)
^
/Users/hideki/Library/Android/sdk/ndk-bundle/sources/cxx-stl/gnu-libstdc++/4.8/include/bits/algorithmfwd.h:371:5: note: candidate function template not viable: requires 3 arguments, but 2 were provided
min(const _Tp&, const _Tp&, _Compare);
^
/Users/hideki/Library/Android/sdk/ndk-bundle/sources/cxx-stl/gnu-libstdc++/4.8/include/bits/stl_algo.h:4221:5: note: candidate function template not viable: requires single argument '__l', but 2 arguments were provided
min(initializer_list<_Tp> __l)
^
/Users/hideki/github/couchbase-lite-android/libraries/couchbase-lite-java-forestdb/jni/../vendor/cbforest/C/c4Document.cc:513:9: error: unknown type name 'sha1Context'
sha1Context ctx;
^
/Users/hideki/github/couchbase-lite-android/libraries/couchbase-lite-java-forestdb/jni/../vendor/cbforest/C/c4Document.cc:515:35: error: no matching function for call to 'min'
uint8_t revLen = (uint8_t)std::min(parentRevID.size, 255ul);
^~~~~~~~
/Users/hideki/Library/Android/sdk/ndk-bundle/sources/cxx-stl/gnu-libstdc++/4.8/include/bits/algorithmfwd.h:367:5: note: candidate template ignored: deduced conflicting types for parameter '_Tp' ('unsigned int' vs. 'unsigned long')
min(const _Tp&, const _Tp&);
^
/Users/hideki/Library/Android/sdk/ndk-bundle/sources/cxx-stl/gnu-libstdc++/4.8/include/bits/stl_algo.h:4226:5: note: candidate template ignored: could not match 'initializer_list<type-parameter-0-0>' against 'unsigned int'
min(initializer_list<_Tp> __l, _Compare __comp)
^
/Users/hideki/Library/Android/sdk/ndk-bundle/sources/cxx-stl/gnu-libstdc++/4.8/include/bits/algorithmfwd.h:371:5: note: candidate function template not viable: requires 3 arguments, but 2 were provided
min(const _Tp&, const _Tp&, _Compare);
^
/Users/hideki/Library/Android/sdk/ndk-bundle/sources/cxx-stl/gnu-libstdc++/4.8/include/bits/stl_algo.h:4221:5: note: candidate function template not viable: requires single argument '__l', but 2 arguments were provided
min(initializer_list<_Tp> __l)
^
5 errors generated.
make: *** [/Users/hideki/github/couchbase-lite-android/libraries/couchbase-lite-java-forestdb/obj/local/armeabi-v7a/objs/CouchbaseLiteJavaForestDB//Users/hideki/github/couchbase-lite-android/libraries/couchbase-lite-java-forestdb/jni/__/vendor/cbforest/C/c4Document.o] Error 1
ForestDB rekeys a database by compacting it, however it disallows the compaction of a database file opened with FDB_SEQTREE_NOT_USE
so the rekey operation cannot succeed.
(For reference, the location of this check is here)
As the title, I'd be really interested in a portable C++ implementation of Couchbase Lite, wonder if this is pointing in that direction?
In CBSE-3058 Sundar wrote:
ForestDB circular block reuse … was designed to solve exactly this type of small-batch-write-space-explosion problem ... The feature was done around December 2015 and was first shipped in 4.5 Couchbase Server watson release. It has been tested extensively by secondary index use cases for both 4.5.1 as well as 4.6 releases. Among other things I believe that it should significantly benefit many mobile device use cases simply because it cuts down the app's write amplification by half.
All we need to do is pull the ForestDB stable branch to the latest. And test...
I just checked out the ForestDB branch of Couchbase-Lite, but I'm having trouble building the project. When selecting CBForest static and attempting to build, or when building my OS X framework with CBForest static as a dependency, I get the following error:
/Users/nathan/Projects/DEVONthink-To-Go/DEVONthink To Go 2/Submodules/Sync/Sync/External/TouchDB/vendor/CBForest/CBForest/RevTree.cc
/Users/nathan/Projects/DEVONthink-To-Go/DEVONthink To Go 2/Submodules/Sync/Sync/External/TouchDB/vendor/CBForest/CBForest/RevTree.cc:18:10: 'forestdb.h' file not found with <angled> include; use "quotes" instead
Normally, I'd assume this is because of some minor incompatibility in the project file -- you're sending headers to some place, I'm expecting to find them in another. But opening up the project, I found this:
Highlighting a randomly-selected file, I see:
Note that submodules appear to be checked out correctly:
Leilah:CBForest nathan$ pwd
/Users/nathan/Projects/Sync/Sync/External/TouchDB/vendor/CBForest
Leilah:CBForest nathan$ ls -1 vendor/forestdb/src/
api_wrapper.cc
arch.h
avltree.cc
avltree.h
blockcache.cc
blockcache.h
I'm of course not ruling out the possibility that I'm irredeemably stupid, but I thought this might be an issue with the project rather than me :-)
With new information regarding platform support
As reported in couchbase/couchbase-lite-ios#1379 , after a CBForest view is indexed, the lastSequenceIndexed stored in the view may still be less than the database's lastSequence. This marks the view as not being up to date, so the next time it's queried it will get indexed again (with the same results, if the db hasn't changed.) This is bad for performance.
Original Ticket:
https://github.com/couchbase/couchbase-lite-java-core/issues/1177
From following crash log, KeyStore::lastSequence()
could throw the exception. It should be caught.
03-31 17:15:31.664 F/DEBUG ( 3010): Build fingerprint: 'samsung/zeroflteuc/zeroflteatt:6.0/MRA58K/G920AUCU3COJA:user/release-keys'
03-31 17:15:31.664 F/DEBUG ( 3010): Revision: '11'
03-31 17:15:31.664 F/DEBUG ( 3010): ABI: 'arm'
03-31 17:15:31.664 F/DEBUG ( 3010): pid: 6827, tid: 7181, name: Acme.Utils.Thre >>> com.xyz.company <<<
03-31 17:15:31.664 F/DEBUG ( 3010): signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
03-31 17:15:31.684 F/DEBUG ( 3010): r0 00000000 r1 00001c0d r2 00000006 r3 c58d2978
03-31 17:15:31.684 F/DEBUG ( 3010): r4 c58d2980 r5 c58d2930 r6 00000002 r7 0000010c
03-31 17:15:31.684 F/DEBUG ( 3010): r8 00000000 r9 00000000 sl 00000001 fp 00000000
03-31 17:15:31.684 F/DEBUG ( 3010): ip 00000006 sp c58d1e88 lr f71af285 pc f71b09cc cpsr 400f0010
03-31 17:15:31.694 F/DEBUG ( 3010):
03-31 17:15:31.694 F/DEBUG ( 3010): backtrace:
03-31 17:15:31.694 F/DEBUG ( 3010): #00 pc 000419cc /system/lib/libc.so (tgkill+12)
03-31 17:15:31.694 F/DEBUG ( 3010): #01 pc 00040281 /system/lib/libc.so (pthread_kill+32)
03-31 17:15:31.694 F/DEBUG ( 3010): #02 pc 0001ca3b /system/lib/libc.so (raise+10)
03-31 17:15:31.694 F/DEBUG ( 3010): #03 pc 00019cb9 /system/lib/libc.so (__libc_android_abort+34)
03-31 17:15:31.694 F/DEBUG ( 3010): #04 pc 000174fc /system/lib/libc.so (abort+4)
03-31 17:15:31.694 F/DEBUG ( 3010): #05 pc 001a64d9 /data/app/com.xyz.company-1/lib/arm/libCouchbaseLiteJavaForestDB.so (__gnu_cxx::__verbose_terminate_handler()+220)
03-31 17:15:31.694 F/DEBUG ( 3010): #06 pc 00176247 /data/app/com.xyz.company-1/lib/arm/libCouchbaseLiteJavaForestDB.so (__cxxabiv1::__terminate(void (*)())+2)
03-31 17:15:31.694 F/DEBUG ( 3010): #07 pc 00176277 /data/app/com.xyz.company-1/lib/arm/libCouchbaseLiteJavaForestDB.so (std::terminate()+10)
03-31 17:15:31.694 F/DEBUG ( 3010): #08 pc 00175f41 /data/app/com.xyz.company-1/lib/arm/libCouchbaseLiteJavaForestDB.so (__cxa_throw+132)
03-31 17:15:31.694 F/DEBUG ( 3010): #09 pc 000e3a41 /data/app/com.xyz.company-1/lib/arm/libCouchbaseLiteJavaForestDB.so (cbforest::error::_throw(fdb_status)+92)
03-31 17:15:31.694 F/DEBUG ( 3010): #10 pc 000f8183 /data/app/com.xyz.company-1/lib/arm/libCouchbaseLiteJavaForestDB.so
03-31 17:15:31.694 F/DEBUG ( 3010): #11 pc 000f8241 /data/app/com.xyz.company-1/lib/arm/libCouchbaseLiteJavaForestDB.so (cbforest::KeyStore::lastSequence() const+36)
03-31 17:15:31.694 F/DEBUG ( 3010): #12 pc 0010ae2d /data/app/com.xyz.company-1/lib/arm/libCouchbaseLiteJavaForestDB.so (c4db_getLastSequence+44)
03-31 17:15:31.694 F/DEBUG ( 3010): #13 pc 00112629 /data/app/com.xyz.company-1/lib/arm/libCouchbaseLiteJavaForestDB.so (Java_com_couchbase_cbforest_Database_getLastSequence+24)
03-31 17:15:31.694 F/DEBUG ( 3010): #14 pc 00e39a41 /data/app/com.xyz.company-1/oat/arm/base.odex (offset 0xd81000)
All outputs from WarnError()
, Warn()
and Log()
are logged, no matter what logLevel is specified.
I love what I see here and was planning on trying this on an app that is currently using LevelDB. Is this lib ready for production apps?
Thanks for the great work!
John
If two threads try to create a transaction on the same Database object, the second one will hit this assertion failure. The mutex works for two threads on different Database objects (on the same db file), but not when they're the same object.
(Note that this is only a valid operation when C4DB_THREADSAFE
is defined at build time, since otherwise threads can't access the same objects at all.)
Something I haven't fixed in my latest pull request #6 is the use of strlcpy(), which doesn't exist on Linux/glibc and presumably other platforms. Maybe there's a different way to implement the same thing with a more portable function?
This either needs to be fixed on the native side, or an API needs to be introduced to rekey the views.
See https://github.com/couchbase/couchbase-lite-android/issues/909#issuecomment-229565872 . @hideki wrote:
06-29 22:21:03.483 1804-1822/? E/CBFOREST: Forestdb Handle is being used by another thread (-39)
06-29 22:21:03.483 1804-1822/? W/CBFOREST: android stack dump
06-29 22:21:03.497 1804-1822/? W/CBFOREST: 000: 0x0xa28bc3ba dump_stack()
06-29 22:21:03.498 1804-1822/? W/CBFOREST: 001: 0x0xa28bc534 cbforest::error::_throw(fdb_status)
06-29 22:21:03.498 1804-1822/? W/CBFOREST: 002: 0x0xa28cf8e0
06-29 22:21:03.499 1804-1822/? W/CBFOREST: 003: 0x0xa28cf992 cbforest::KeyStore::lastSequence() const
06-29 22:21:03.499 1804-1822/? W/CBFOREST: 004: 0x0xa28d9408 cbforest::MapReduceIndexer::startingSequence()
06-29 22:21:03.499 1804-1822/? W/CBFOREST: 005: 0x0xa28ed190 c4indexer_enumerateDocuments
06-29 22:21:03.499 1804-1822/? W/CBFOREST: 006: 0x0xa28f5874 Java_com_couchbase_cbforest_Indexer_iterateDocuments
06-29 22:21:03.499 1804-1822/? W/CBFOREST: android stack dump done
cbforest/CBForest/MapReduceIndex.cc
Line 390 in d6d38d9
sequence MapReduceIndexer::startingSequence() {
_latestDbSequence = _writers[0]->index->sourceStore().lastSequence();
...
...
}
add(...)
method calls beginTransaction()
and endTransaction()
. In cbforest level, beginTransaction()
and endTransaction()
, forestdb is protected by _transactionMutex
.C4Indexer
parameter of c4indexer_enumerateDocuments(C4Indexer *indexer, C4Error *outError)
is protected by view->_mutex.lock();
. But it does not protect forestdb instance.Something in f033035 broke the database unit tests. It looks like the sequence on selected revision is not getting set properly anymore (perhaps other things as well).
CBForest is currently using a ForestDB commit from April 23. We should update, since a number of bugs have been fixed. I tried updating to the latest stable
branch commit (28feef96), but found two regressions, one critical:
I'll try again once these are fixed.
See couchbase/couchbase-lite-ios#1164. This problem doesn't actually occur in SQLite based databases, but it does in ForestDB-based ones, and can be fixed at the CBForest level.
I found some more functions that can create an unlimited number of JNI local references, by converting an array of Java objects into a native array:
These are subject to the same crash as in couchbase-lite-android#782.
They can be fixed the same way as in #64.
@borrrden has been working on getting CBForest to build in MSVC on Windows. Via email:
Alright, it turns out the errors were not as bad as I thought, just a lot of leaning on GNU extensions. I was able to get cbforest to “build” on Windows. The reason I put it in quotes is because I only progressed to the point where all the compile errors were gone. I didn’t set it up to link with forestdb yet, so I still have linker errors and such to deal with but the core of forestdb compiles now (not sure about the tokenizer library). I’ve successfully compiled forestDB on Windows before without much of a problem so I don’t imagine it will be much more to do.
I summarized all the issues I found and how I solved them in this document: https://docs.google.com/spreadsheets/d/1up0TGzCwKXm7AfYozUgRpQ0zeE707NSxI-xU1dLoaPg/edit?usp=sharing
I also committed the fixes (minus one from the fts3_unicodesn repo that I don’t have access to) into the feature/c_api _windows branch.
It doesn't appear that the _version variable in C4View does anything, so is the caller responsible for tracking it and erasing the index on change. If so, what is the point of the index parameter in the c4view_open call?
The full-text search implementation in CBForest is pretty basic. It doesn't have any fancy query features like boolean operators or even word-prefix matching, and it's not very fast. The best way to improve it is probably to replace it with an existing already-proven implementation.
The best choice, ironically, is SQLite. Its FTS4 is pretty good, and is what Couchbase Lite already uses with the default SQLite storage engine. It basically comes for free on iOS/Mac/Android.
Another alternative is CLucene, a C++ port of Lucene. It's rather a lot of code though (about 80,000 lines of C++.)
The result would be that the FTS storage for a view wouldn't be stored in ForestDB but in the external database used by the FTS library. The CBForest API would need to make this transparent so clients can use the same API for queries.
The RefCounted helper class (c4Impl.hh) has a destructor with an assertion in it. An assertion failure throws an exception, but throwing from a destructor is illegal and triggers immediate termination.
This seems to be the cause of couchbase/couchbase-lite-ios#1422; the assertion failure happens because the RefCounted instance hasn't been constructed yet.
To reproduce this I added a throw
call at the start of MapReduceIndex::readState()
to simulate a DB read error. This produces the following crash during the CBL ViewInternal test suite:
13:55:51.779| WARNING: ForestDB error: unknown error (-9999)
{at FDBLogCallback:68}
13:55:51.780| WARNING: ForestDB error: Assertion failed: _refCount == 0 (/Couchbase/CouchbaseLite/vendor/CBForest/C/c4Impl.hh:115, in ~RefCounted) {at FDBLogCallback:68}
libc++abi.dylib: terminating with uncaught exception of type cbforest::error: unknown error
The unit tests are written in Objective-C++ and use Xcode's test framework, so they can't be used on non-Apple platforms.
The other remaining issue with my Linux build (at least as far as I made it in the build) is this piece of code (from KeyStore.cc) and a similar one two functions down:
fdb_doc doc = {
.key = (void*)key.buf,
.keylen = key.size,
.meta = (void*)meta.buf,
.metalen = meta.size,
.body = (void*)body.buf,
.bodylen = body.size,
};
Which g++ (GCC 4.9.2 20150304 (prerelease)) doesn't quite like too much:
FAILED: /usr/bin/c++ -std=c++11 -fomit-frame-pointer -O2 -g -DNDEBUG -I../vendor/forestdb/include -I.. -MMD -MT CMakeFiles/cbforest.dir/CBForest/KeyStore.cc.o -MF CMakeFiles/cbforest.dir/CBForest/KeyStore.cc.o.d -o CMakeFiles/cbforest.dir/CBForest/KeyStore.cc.o -c ../CBForest/KeyStore.cc
../CBForest/KeyStore.cc: In member function ‘forestdb::sequence forestdb::KeyStoreWriter::set(forestdb::slice, forestdb::slice, forestdb::slice)’:
../CBForest/KeyStore.cc:116:9: sorry, unimplemented: non-trivial designated initializers not supported
};
^
../CBForest/KeyStore.cc: In member function ‘bool forestdb::KeyStoreWriter::del(forestdb::slice)’:
../CBForest/KeyStore.cc:149:9: sorry, unimplemented: non-trivial designated initializers not supported
};
^
The reason is that this is not valid C++, neither C++11 nor C++14, although it is valid C99. (C++ initializer lists got adopted, but designated initializers didn't.) In the interest of portability, maybe it can be swapped out with something more standard.
By running View unit test, the test crashes with same place frequently enough. See following crash log.
04-12 20:52:41.658 19453-19466/? A/libc: Fatal signal 11 (SIGSEGV) at 0x00000000 (code=128), thread 19466 (ationTestRunner)
04-12 20:52:41.710 147-147/? I/DEBUG: *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
04-12 20:52:41.710 147-147/? I/DEBUG: Build fingerprint: 'generic/vbox86p/vbox86p:4.4.4/KTU84P/eng.buildbot.20151118.000452:userdebug/test-keys'
04-12 20:52:41.710 147-147/? I/DEBUG: Revision: '0'
04-12 20:52:41.710 147-147/? I/DEBUG: pid: 19453, tid: 19466, name: ationTestRunner >>> com.couchbase.lite.test <<<
04-12 20:52:41.710 147-147/? I/DEBUG: signal 11 (SIGSEGV), code 128 (SI_KERNEL), fault addr 00000000
04-12 20:52:41.866 147-147/? I/DEBUG: eax 949a953c ebx 949a953c ecx b976e11c edx 94ab661c
04-12 20:52:41.866 147-147/? I/DEBUG: esi 94753100 edi 947522a0
04-12 20:52:41.866 147-147/? I/DEBUG: xcs 00000073 xds 0000007b xes 0000007b xfs 00000000 xss 0000007b
04-12 20:52:41.866 147-147/? I/DEBUG: eip 9473cc8d ebp b976e108 esp 94ab661c flags 00010282
04-12 20:52:41.866 147-147/? E/Corkscrew: unrecognized dwarf lower part encoding: 0x2f
04-12 20:52:41.866 147-147/? I/DEBUG: backtrace:
04-12 20:52:41.866 147-147/? I/DEBUG: #00 pc 000eac8d /data/app-lib/com.couchbase.lite.test-1/libCouchbaseLiteJavaForestDB.so (filemgr_is_fully_resident+141)
04-12 20:52:41.866 147-147/? I/DEBUG: #01 pc 0012e4f2 /data/app-lib/com.couchbase.lite.test-1/libCouchbaseLiteJavaForestDB.so
04-12 20:52:41.866 147-147/? I/DEBUG: #02 pc 0012e426 /data/app-lib/com.couchbase.lite.test-1/libCouchbaseLiteJavaForestDB.so (wal_flush+118)
04-12 20:52:41.866 147-147/? I/DEBUG: #03 pc 000f9213 /data/app-lib/com.couchbase.lite.test-1/libCouchbaseLiteJavaForestDB.so (_fdb_commit+1571)
04-12 20:52:41.866 147-147/? I/DEBUG: #04 pc 00101c3b /data/app-lib/com.couchbase.lite.test-1/libCouchbaseLiteJavaForestDB.so (fdb_commit+91)
04-12 20:52:41.866 147-147/? I/DEBUG: #05 pc 0013c77c /data/app-lib/com.couchbase.lite.test-1/libCouchbaseLiteJavaForestDB.so (cbforest::Database::close()+92)
04-12 20:52:41.866 147-147/? I/DEBUG: #06 pc 0013c6e4 /data/app-lib/com.couchbase.lite.test-1/libCouchbaseLiteJavaForestDB.so (cbforest::Database::~Database()+196)
04-12 20:52:41.866 147-147/? I/DEBUG: stack:
ForestDB disallows skipping keys if the keys are empty, which means that when CBForest uses an empty Collatable then any query with InclusiveEnd=false (or descending and InclusiveStart=false) will fail because the enumerator will be NULL
due to an error unless the end (or start in the case of descending) key is specified.
WARNING: ForestDB error: invalid arguments (-1)
Values don't need to be sorted, which what Collatable is for. And especially in Java it is more efficient to work with JSON than to have to go through JNI calls to C4KeyReader to parse the binary data.
Passing class pointer to native could improve performance.
Uses CommonCrypto. For now it will be good enough to have a compiler directive to either compile or not compile it.
There is currently no way to delete documents in a non-tombstone fashion.
Can we make the interface below support multiple view indexing?
cbforest/Java/jni/native_view.cc
Lines 124 to 134 in d0afb82
Log("DB %p: added %s --> %s (meta %s) (seq %llu)\n",
_handle, key.hexCString(), body.hexCString(), meta.hexCString(), doc.seqnum);
[arm64-v8a] Compile++ : CouchbaseLiteJavaForestDB <= KeyStore.cc
/Users/hideki/github/couchbase-lite-android/libraries/couchbase-lite-java-forestdb/jni/../vendor/cbforest/CBForest/KeyStore.cc:148:78: warning: format specifies type 'unsigned long long' but the argument has type 'fdb_seqnum_t' (aka 'unsigned long') [-Wformat]
_handle, key.hexCString(), body.hexCString(), meta.hexCString(), doc.seqnum);
^~~~~~~~~~
../vendor/cbforest/CBForest/LogInternal.hh:40:77: note: expanded from macro 'Log'
#define Log(...) __android_log_print(ANDROID_LOG_INFO, LOG_TAG, __VA_ARGS__)
^~~~~~~~~~~
1 warning generated.
Trying to load a body when it doesn't exist results in an error condition. It would be less cumbersome to expose this method to the C API for places like getPossibleAncestors and other places that iterate through c4doc_selectNextRevision.
Unit test here
The all docs enumerator is including deleted documents no matter what.
Need a C4DocEnumerator that works by sequence for the UpdateIndexes logic.
The iOS implementation uses some ForestDB copying functions that are not exposed to the C API. Either the copy mechanism or a simple "rekey" API will suffice.
C4Error error;
error is not initialized with 0. So C4error.domain is set random number.
In case of end of leaf, selectNextLeaf()
returns false
with C4Error.error = 0 without updating domain variable.
04-07 19:18:40.723 3181-3194/? E/Database: Error in changesSince()
ForestException{domain=-1642724708, code=0}
at com.couchbase.cbforest.Document.selectNextLeaf(Native Method)
at com.couchbase.cbforest.Document.selectNextLeaf(Document.java:47)
at com.couchbase.lite.store.ForestBridge.getCurrentRevisionIDs(ForestBridge.java:86)
at com.couchbase.lite.store.ForestDBStore.changesSince(ForestDBStore.java:731)
at com.couchbase.lite.Database.getAllDocs(Database.java:1428)
at com.couchbase.lite.Database.queryViewNamed(Database.java:2079)
at com.couchbase.lite.Query.run(Query.java:433)
at com.couchbase.lite.ApiTest.test12_AllDocumentsBySequence(ApiTest.java:399)
at java.lang.reflect.Method.invokeNative(Native Method)
at java.lang.reflect.Method.invoke(Method.java:515)
at junit.framework.TestCase.runTest(TestCase.java:168)
at junit.framework.TestCase.runBare(TestCase.java:134)
at com.couchbase.lite.LiteTestCase.runBare(LiteTestCase.java:20)
at com.couchbase.lite.LiteTestCaseWithDB.runBare(LiteTestCaseWithDB.java:95)
at junit.framework.TestResult$1.protect(TestResult.java:115)
at junit.framework.TestResult.runProtected(TestResult.java:133)
at junit.framework.TestResult.run(TestResult.java:118)
at junit.framework.TestCase.run(TestCase.java:124)
at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:191)
at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:176)
at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:554)
at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1701)
I found this while investigating the 1.2.0-forestdb Android LiteServ forestdb issues.
Running the cbl-webhook_events_not_posted.js node.js (https://github.com/couchbaselabs/sync-gateway-tests/blob/master/tests/cbl-webhook_events_not_posted.js#L120-L123) test, I see the below exception right before the test fails with 'unable to get doc rev'
https://gist.github.com/sethrosetter/7c87809454471b90bd8a5ecb05e29a36#file-gistfile1-txt-L90-L107
Collatable.cc
C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\xlocale(337) : warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify /EHsc
C:\couchbase-lite-java\libraries\couchbase-lite-java-forestdb\vendor\cbforest\CBForest\Collatable.cc(77) : error C2039: 'max' : is not a member of 'std'
C:\couchbase-lite-java\libraries\couchbase-lite-java-forestdb\vendor\cbforest\CBForest\Collatable.cc(77) : error C3861: 'max': identifier not found
C:\couchbase-lite-java\libraries\couchbase-lite-java-forestdb\vendor\cbforest\CBForest\Collatable.cc(141) : warning C4068: unknown pragma
C:\couchbase-lite-java\libraries\couchbase-lite-java-forestdb\vendor\cbforest\CBForest\Collatable.cc(375) : warning C4068: unknown pragma
slice.cc
C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\xlocale(337) : warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify /EHsc
C:\couchbase-lite-java\libraries\couchbase-lite-java-forestdb\vendor\cbforest\CBForest\slice.cc(25) : error C2059: syntax error : ':'
C:\couchbase-lite-java\libraries\couchbase-lite-java-forestdb\vendor\cbforest\CBForest\slice.cc(27) : error C2059: syntax error : ':'
Needed by the "Compact" method of the storage engine
There is always an issue that I've noticed when assigning bools to 1 bit sections of a bit field. The bit field representation the compiler chooses is undefined and in the case of Windows it will truncate down to the lowest bit. The bool value however is 8 bits, which means that there is a 50% chance that the bool will turn into an incorrect value. For instance in DocEnumerator.cc
there is this struct:
struct Options {
unsigned skip;
unsigned limit;
bool descending :1;
bool inclusiveStart :1;
bool inclusiveEnd :1;
bool includeDeleted :1;
KeyStore::contentOptions contentOptions :4;
static const Options kDefault;
};
And in C4DocEnumerator* c4db_enumerateAllDocs
in c4Database.cc
there is this logic:
options.descending = c4options->descending;
options.inclusiveStart = c4options->inclusiveStart;
options.inclusiveEnd = c4options->inclusiveEnd;
options.includeDeleted = c4options->includeDeleted;
In the unit test run, all four values in c4options are set to true
and they are all bonafide bool
types. However this is less important than what is actually stored inside them, which in this case is (in order) 233, 238, 112, and 5. To a bool, any value other than ZERO is true. So why is this a problem? Well in the bitfield the values get stored as follows: 233 & 1 (1), 238 & 1 (0), 112 & 1 (0), 5 & 1 (1) which turns them into true, false, false, true instead of all true. Since the comparison is effectively X & 1, this means that the miss rate is about 50%.
There are two ways to deal with this:
options.descending = c4options->descending != false
Looks like it may be due to compaction.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.