couchbaselabs / cbforest Goto Github PK

C++ wrapper library around ForestDB, for use in Couchbase Lite.

Objective-C++ 5.63% C++ 33.01% C 6.81% C# 20.23% Java 3.52% Makefile 1.04% Shell 0.09% Objective-C 29.67%

cbforest's Introduction

CBForest is a higher-level C and C++ wrapper around ForestDB, a new key-value storage engine based on a hierarchical B+-tree trie data structure. ForestDB is similar in functionality to the CouchStore engine that currently underlies Couchbase Server, but it should be faster and more space-efficient, especially on solid-state disks (SSDs).

CBForest adds an idiomatic object-oriented C++ API for ForestDB, and also some new functionality (see below.) Recently a C API with the same functionality has been added, and Java and C# bindings to that C API.

The immediate purpose of CBForest is to serve as the storage engine of the next generation of Couchbase Lite (on all platforms), replacing SQLite. But it may find other uses too, perhaps for applications that want a fast minimalist data store with map/reduce indexing, but don't need any of the fancy features of Couchbase Lite like replication.

Features

ForestDB features, available via idiomatic C++, Java and C# APIs:
- Fast key-value storage, where keys and values are both opaque blobs.
- Extremely robust append-only file format with write-ahead log and automatic compaction.
- Reads are never blocked, even while writes or transactions are in progress.
- Iteration by key order.
- Iteration by sequence, reflecting the order in which changes were made to the database. (This is useful for tasks like updating indexes and replication.)
- Database encryption using AES-256.
New features implemented by CBForest:
- Optional multi-version document format that keeps a revision tree of the history of each document (as in Couchbase Lite or CouchDB.)
- Index API that uses a database as an index of an external data set.
- Map-reduce indexes that update incrementally as documents are changed in the source DB (as in Couchbase Lite or CouchDB.)
- Limited full-text indexing.
- Limited geo-indexing.
- Support for JSON-compatible structured keys in indexes, sorted according to CouchDB's JSON collation spec.

Platform Support

CBForest runs on Mac OS, iOS, tvOS, Android, various other flavors of Unix, and Windows.

CBForest has been in use since mid-2015 in the iOS/Mac version of Couchbase Lite 1.1, and since early 2016 in the 1.2 release on all the above platforms.

License

Like all Couchbase source code, this is released under the Apache 2 license.

cbforest's People

Contributors

Stargazers

Watchers

Forkers

mindis jpetso lordkev buzzfeed shaobofeng bkway chrisdurtschi parastoo-62 zyamusic 5b5 mugabe tophatch

cbforest's Issues

Reimplement FTS using an external library (SQLite? CLucene?)

The full-text search implementation in CBForest is pretty basic. It doesn't have any fancy query features like boolean operators or even word-prefix matching, and it's not very fast. The best way to improve it is probably to replace it with an existing already-proven implementation.

The best choice, ironically, is SQLite. Its FTS4 is pretty good, and is what Couchbase Lite already uses with the default SQLite storage engine. It basically comes for free on iOS/Mac/Android.

Another alternative is CLucene, a C++ port of Lucene. It's rather a lot of code though (about 80,000 lines of C++.)

The result would be that the FTS storage for a view wouldn't be stored in ForestDB but in the external database used by the FTS library. The CBForest API would need to make this transparent so clients can use the same API for queries.

RefCounted destructor can throw, triggering abort

The RefCounted helper class (c4Impl.hh) has a destructor with an assertion in it. An assertion failure throws an exception, but throwing from a destructor is illegal and triggers immediate termination.

This seems to be the cause of couchbase/couchbase-lite-ios#1422; the assertion failure happens because the RefCounted instance hasn't been constructed yet.

To reproduce this I added a throw call at the start of MapReduceIndex::readState() to simulate a DB read error. This produces the following crash during the CBL ViewInternal test suite:

13:55:51.779| WARNING: ForestDB error: unknown error (-9999)
 {at FDBLogCallback:68}
13:55:51.780| WARNING: ForestDB error: Assertion failed: _refCount == 0 (/Couchbase/CouchbaseLite/vendor/CBForest/C/c4Impl.hh:115, in ~RefCounted) {at FDBLogCallback:68}
libc++abi.dylib: terminating with uncaught exception of type cbforest::error: unknown error

c4db_rekey leaves view db files encrypted with old key

This either needs to be fixed on the native side, or an API needs to be introduced to rekey the views.

Update ForestDB to get improved compaction / space reuse

In CBSE-3058 Sundar wrote:

ForestDB circular block reuse … was designed to solve exactly this type of small-batch-write-space-explosion problem ... The feature was done around December 2015 and was first shipped in 4.5 Couchbase Server watson release. It has been tested extensively by secondary index use cases for both 4.5.1 as well as 4.6 releases. Among other things I believe that it should significantly benefit many mobile device use cases simply because it cuts down the app's write amplification by half.

All we need to do is pull the ForestDB stable branch to the latest. And test...

KeyStore::lastSequence() could throw the exception. It should be caught.

Original Ticket:
https://github.com/couchbase/couchbase-lite-java-core/issues/1177

From following crash log, KeyStore::lastSequence() could throw the exception. It should be caught.

03-31 17:15:31.664 F/DEBUG   ( 3010): Build fingerprint: 'samsung/zeroflteuc/zeroflteatt:6.0/MRA58K/G920AUCU3COJA:user/release-keys'
03-31 17:15:31.664 F/DEBUG   ( 3010): Revision: '11'
03-31 17:15:31.664 F/DEBUG   ( 3010): ABI: 'arm'
03-31 17:15:31.664 F/DEBUG   ( 3010): pid: 6827, tid: 7181, name: Acme.Utils.Thre  >>> com.xyz.company                                                                           <<<
03-31 17:15:31.664 F/DEBUG   ( 3010): signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
03-31 17:15:31.684 F/DEBUG   ( 3010):     r0 00000000  r1 00001c0d  r2 00000006  r3 c58d2978
03-31 17:15:31.684 F/DEBUG   ( 3010):     r4 c58d2980  r5 c58d2930  r6 00000002  r7 0000010c
03-31 17:15:31.684 F/DEBUG   ( 3010):     r8 00000000  r9 00000000  sl 00000001  fp 00000000
03-31 17:15:31.684 F/DEBUG   ( 3010):     ip 00000006  sp c58d1e88  lr f71af285  pc f71b09cc  cpsr 400f0010
03-31 17:15:31.694 F/DEBUG   ( 3010): 
03-31 17:15:31.694 F/DEBUG   ( 3010): backtrace:
03-31 17:15:31.694 F/DEBUG   ( 3010):     #00 pc 000419cc  /system/lib/libc.so (tgkill+12)
03-31 17:15:31.694 F/DEBUG   ( 3010):     #01 pc 00040281  /system/lib/libc.so (pthread_kill+32)
03-31 17:15:31.694 F/DEBUG   ( 3010):     #02 pc 0001ca3b  /system/lib/libc.so (raise+10)
03-31 17:15:31.694 F/DEBUG   ( 3010):     #03 pc 00019cb9  /system/lib/libc.so (__libc_android_abort+34)
03-31 17:15:31.694 F/DEBUG   ( 3010):     #04 pc 000174fc  /system/lib/libc.so (abort+4)
03-31 17:15:31.694 F/DEBUG   ( 3010):     #05 pc 001a64d9  /data/app/com.xyz.company-1/lib/arm/libCouchbaseLiteJavaForestDB.so (__gnu_cxx::__verbose_terminate_handler()+220)
03-31 17:15:31.694 F/DEBUG   ( 3010):     #06 pc 00176247  /data/app/com.xyz.company-1/lib/arm/libCouchbaseLiteJavaForestDB.so (__cxxabiv1::__terminate(void (*)())+2)
03-31 17:15:31.694 F/DEBUG   ( 3010):     #07 pc 00176277  /data/app/com.xyz.company-1/lib/arm/libCouchbaseLiteJavaForestDB.so (std::terminate()+10)
03-31 17:15:31.694 F/DEBUG   ( 3010):     #08 pc 00175f41  /data/app/com.xyz.company-1/lib/arm/libCouchbaseLiteJavaForestDB.so (__cxa_throw+132)
03-31 17:15:31.694 F/DEBUG   ( 3010):     #09 pc 000e3a41  /data/app/com.xyz.company-1/lib/arm/libCouchbaseLiteJavaForestDB.so (cbforest::error::_throw(fdb_status)+92)
03-31 17:15:31.694 F/DEBUG   ( 3010):     #10 pc 000f8183  /data/app/com.xyz.company-1/lib/arm/libCouchbaseLiteJavaForestDB.so
03-31 17:15:31.694 F/DEBUG   ( 3010):     #11 pc 000f8241  /data/app/com.xyz.company-1/lib/arm/libCouchbaseLiteJavaForestDB.so (cbforest::KeyStore::lastSequence() const+36)
03-31 17:15:31.694 F/DEBUG   ( 3010):     #12 pc 0010ae2d  /data/app/com.xyz.company-1/lib/arm/libCouchbaseLiteJavaForestDB.so (c4db_getLastSequence+44)
03-31 17:15:31.694 F/DEBUG   ( 3010):     #13 pc 00112629  /data/app/com.xyz.company-1/lib/arm/libCouchbaseLiteJavaForestDB.so (Java_com_couchbase_cbforest_Database_getLastSequence+24)
03-31 17:15:31.694 F/DEBUG   ( 3010):     #14 pc 00e39a41  /data/app/com.xyz.company-1/oat/arm/base.odex (offset 0xd81000)

Android: LogLevel does not effect for Android.

All outputs from WarnError(), Warn() and Log() are logged, no matter what logLevel is specified.

[C API] Expose doc sequence via C4QueryEnumerator

This is needed to load the appropriate document body when a placeholder ("*") is found.

Could this mean a C++ port of Couchbase Lite?

As the title, I'd be really interested in a portable C++ implementation of Couchbase Lite, wonder if this is pointing in that direction?

C4Error: variables of struct is not initialized with 0 on Android platform

C4Error error; error is not initialized with 0. So C4error.domain is set random number.
In case of end of leaf, selectNextLeaf() returns false with C4Error.error = 0 without updating domain variable.

04-07 19:18:40.723 3181-3194/? E/Database: Error in changesSince()
 ForestException{domain=-1642724708, code=0}
     at com.couchbase.cbforest.Document.selectNextLeaf(Native Method)
     at com.couchbase.cbforest.Document.selectNextLeaf(Document.java:47)
     at com.couchbase.lite.store.ForestBridge.getCurrentRevisionIDs(ForestBridge.java:86)
     at com.couchbase.lite.store.ForestDBStore.changesSince(ForestDBStore.java:731)
     at com.couchbase.lite.Database.getAllDocs(Database.java:1428)
     at com.couchbase.lite.Database.queryViewNamed(Database.java:2079)
     at com.couchbase.lite.Query.run(Query.java:433)
     at com.couchbase.lite.ApiTest.test12_AllDocumentsBySequence(ApiTest.java:399)
     at java.lang.reflect.Method.invokeNative(Native Method)
     at java.lang.reflect.Method.invoke(Method.java:515)
     at junit.framework.TestCase.runTest(TestCase.java:168)
     at junit.framework.TestCase.runBare(TestCase.java:134)
     at com.couchbase.lite.LiteTestCase.runBare(LiteTestCase.java:20)
     at com.couchbase.lite.LiteTestCaseWithDB.runBare(LiteTestCaseWithDB.java:95)
     at junit.framework.TestResult$1.protect(TestResult.java:115)
     at junit.framework.TestResult.runProtected(TestResult.java:133)
     at junit.framework.TestResult.run(TestResult.java:118)
     at junit.framework.TestCase.run(TestCase.java:124)
     at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:191)
     at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:176)
     at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:554)
     at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1701)

[C API] enumerate by sequence

Need a C4DocEnumerator that works by sequence for the UpdateIndexes logic.

Multiple Transactions on same Database* cause assertion failure in thread-safe mode

If two threads try to create a transaction on the same Database object, the second one will hit this assertion failure. The mutex works for two threads on different Database objects (on the same db file), but not when they're the same object.

(Note that this is only a valid operation when C4DB_THREADSAFE is defined at build time, since otherwise threads can't access the same objects at all.)

Ready for production?

I love what I see here and was planning on trying this on an app that is currently using LevelDB. Is this lib ready for production apps?

Thanks for the great work!

John

Compilation Error on Windows with VC++

Collatable.cc
C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\xlocale(337) : warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify /EHsc
C:\couchbase-lite-java\libraries\couchbase-lite-java-forestdb\vendor\cbforest\CBForest\Collatable.cc(77) : error C2039: 'max' : is not a member of 'std'
C:\couchbase-lite-java\libraries\couchbase-lite-java-forestdb\vendor\cbforest\CBForest\Collatable.cc(77) : error C3861: 'max': identifier not found
C:\couchbase-lite-java\libraries\couchbase-lite-java-forestdb\vendor\cbforest\CBForest\Collatable.cc(141) : warning C4068: unknown pragma
C:\couchbase-lite-java\libraries\couchbase-lite-java-forestdb\vendor\cbforest\CBForest\Collatable.cc(375) : warning C4068: unknown pragma

slice.cc
C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\xlocale(337) : warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify /EHsc
C:\couchbase-lite-java\libraries\couchbase-lite-java-forestdb\vendor\cbforest\CBForest\slice.cc(25) : error C2059: syntax error : ':'
C:\couchbase-lite-java\libraries\couchbase-lite-java-forestdb\vendor\cbforest\CBForest\slice.cc(27) : error C2059: syntax error : ':'

View index should store values as JSON not Collatable

Values don't need to be sorted, which what Collatable is for. And especially in Java it is more efficient to work with JSON than to have to go through JNI calls to C4KeyReader to parse the binary data.

[C API] Need a purge mechanism

There is currently no way to delete documents in a non-tombstone fashion.

Crash in view

04-07 17:06:13.559 1663-1672/com.couchbase.cbforest.test A/libc: Fatal signal 11 (SIGSEGV) at 0x000000d0 (code=1), thread 1672 (FinalizerDaemon)
04-07 17:06:13.643 1663-1676/com.couchbase.cbforest.test I/TestRunner: finished: testIndexVersion(com.couchbase.cbforest.C4ViewTest)
04-07 17:06:13.643 1663-1676/com.couchbase.cbforest.test I/TestRunner: passed: testIndexVersion(com.couchbase.cbforest.C4ViewTest)
04-07 17:06:13.643 1663-1676/com.couchbase.cbforest.test I/TestRunner: started: testQueryIndex(com.couchbase.cbforest.C4ViewTest)
04-07 17:06:13.659 146-146/? I/DEBUG: *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
04-07 17:06:13.659 146-146/? I/DEBUG: Build fingerprint: 'generic/vbox86p/vbox86p:4.4.4/KTU84P/eng.buildbot.20151118.000452:userdebug/test-keys'
04-07 17:06:13.659 146-146/? I/DEBUG: Revision: '0'
04-07 17:06:13.659 146-146/? I/DEBUG: pid: 1663, tid: 1672, name: FinalizerDaemon  >>> com.couchbase.cbforest.test <<<
04-07 17:06:13.659 146-146/? I/DEBUG: signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 000000d0
04-07 17:06:13.663 534-609/system_process D/dalvikvm: GC_FOR_ALLOC freed 709K, 18% free 5713K/6908K, paused 6ms, total 6ms
04-07 17:06:13.811 146-146/? I/DEBUG:     eax 000000c8  ebx 951e55bc  ecx 000000c8  edx 00000000
04-07 17:06:13.811 146-146/? I/DEBUG:     esi 95b48930  edi 95b489dc
04-07 17:06:13.811 146-146/? I/DEBUG:     xcs 00000073  xds 0000007b  xes 0000007b  xfs 00000000  xss 0000007b
04-07 17:06:13.811 146-146/? I/DEBUG:     eip 94ff515b  ebp 95b489f8  esp 95b488f8  flags 00210202
04-07 17:06:13.811 146-146/? E/Corkscrew: unrecognized dwarf lower part encoding: 0x37
04-07 17:06:13.811 146-146/? I/DEBUG:     #00  pc 0016515b  /data/app-lib/com.couchbase.cbforest.test-1/libCouchbaseLiteJavaForestDB.so (cbforest::Index::isBusy() const+11)
04-07 17:06:13.811 146-146/? I/DEBUG:     #01  pc 0019e9c7  /data/app-lib/com.couchbase.cbforest.test-1/libCouchbaseLiteJavaForestDB.so (c4View::checkNotBusy(C4Error*)+55)
04-07 17:06:13.811 146-146/? I/DEBUG:     #02  pc 0019e7c6  /data/app-lib/com.couchbase.cbforest.test-1/libCouchbaseLiteJavaForestDB.so (c4view_close+54)
04-07 17:06:13.811 146-146/? I/DEBUG:          95b488b8  00000000  
04-07 17:06:13.811 146-146/? I/DEBUG:          95b488bc  b4da7729  /system/lib/libdvm.so (dvmDecodeIndirectRef(Thread*, _jobject*)+233)
04-07 17:06:13.811 146-146/? I/DEBUG:          95b488c0  b937bd00  [heap]
04-07 17:06:13.811 146-146/? I/DEBUG:          95b488c4  1d800001  
04-07 17:06:13.811 146-146/? I/DEBUG:          95b488c8  b939e238  [heap]
04-07 17:06:13.811 146-146/? I/DEBUG:          95b488cc  b4ef2cb4  /system/lib/libdvm.so
04-07 17:06:13.811 146-146/? I/DEBUG:          95b488d0  00000000  
04-07 17:06:13.811 146-146/? I/DEBUG:          95b488d4  00000000  
04-07 17:06:13.811 146-146/? I/DEBUG:          95b488d8  b937bc58  [heap]
04-07 17:06:13.811 146-146/? I/DEBUG:          95b488dc  b4ef2cb4  /system/lib/libdvm.so
04-07 17:06:13.811 146-146/? I/DEBUG:          95b488e0  00000000  
04-07 17:06:13.811 146-146/? I/DEBUG:          95b488e4  00000000  
04-07 17:06:13.811 146-146/? I/DEBUG:          95b488e8  b937bc58  [heap]
04-07 17:06:13.811 146-146/? I/DEBUG:          95b488ec  b4dab542  /system/lib/libdvm.so (SetLongField(_JNIEnv*, _jobject*, _jfieldID*, long long)+130)
04-07 17:06:13.811 146-146/? I/DEBUG:          95b488f0  b937bc58  [heap]
04-07 17:06:13.811 146-146/? I/DEBUG:          95b488f4  00000007  
04-07 17:06:13.811 146-146/? I/DEBUG:     #00  95b488f8  000000c8  
04-07 17:06:13.811 146-146/? I/DEBUG:          95b488fc  9502e9c8  /data/app-lib/com.couchbase.cbforest.test-1/libCouchbaseLiteJavaForestDB.so (c4View::checkNotBusy(C4Error*)+56)
04-07 17:06:13.811 146-146/? I/DEBUG:     #01  95b48900  000000c8  
04-07 17:06:13.811 146-146/? I/DEBUG:          95b48904  b939f398  [heap]
04-07 17:06:13.811 146-146/? I/DEBUG:          95b48908  ffff0208  
04-07 17:06:13.811 146-146/? I/DEBUG:          95b4890c  951e55bc  /data/app-lib/com.couchbase.cbforest.test-1/libCouchbaseLiteJavaForestDB.so
04-07 17:06:13.811 146-146/? I/DEBUG:          95b48910  b4dabd00  /system/lib/libdvm.so (GetLongField(_JNIEnv*, _jobject*, _jfieldID*))
04-07 17:06:13.811 146-146/? I/DEBUG:          95b48914  1d800001  
04-07 17:06:13.811 146-146/? I/DEBUG:          95b48918  951e55bc  /data/app-lib/com.couchbase.cbforest.test-1/libCouchbaseLiteJavaForestDB.so
04-07 17:06:13.811 146-146/? I/DEBUG:          95b4891c  95b489b8  [stack:1672]
04-07 17:06:13.811 146-146/? I/DEBUG:          95b48920  00000000  
04-07 17:06:13.811 146-146/? I/DEBUG:          95b48924  1d800001  
04-07 17:06:13.811 146-146/? I/DEBUG:          95b48928  951e55bc  /data/app-lib/com.couchbase.cbforest.test-1/libCouchbaseLiteJavaForestDB.so
04-07 17:06:13.811 146-146/? I/DEBUG:          95b4892c  9502e7c7  /data/app-lib/com.couchbase.cbforest.test-1/libCouchbaseLiteJavaForestDB.so (c4view_close+55)
04-07 17:06:13.811 146-146/? I/DEBUG:          95b48930  00000000  
04-07 17:06:13.811 146-146/? I/DEBUG:          95b48934  95b489b8  [stack:1672]
04-07 17:06:13.811 146-146/? I/DEBUG:          95b48938  9e1245a4  /dev/ashmem/dalvik-LinearAlloc (deleted)
04-07 17:06:13.811 146-146/? I/DEBUG:          95b4893c  95035759  /data/app-lib/com.couchbase.cbforest.test-1/libCouchbaseLiteJavaForestDB.so (_JNIEnv::SetLongField(_jobject*, _jfieldID*, long long)+105)
04-07 17:06:13.867 534-556/system_process I/BootReceiver: Copying /data/tombstones/tombstone_03 to DropBox (SYSTEM_TOMBSTONE)
04-07 17:06:13.871 534-1680/system_process W/ActivityManager: Error in app com.couchbase.cbforest.test running instrumentation ComponentInfo{com.couchbase.cbforest.test/android.test.InstrumentationTestRunner}:
04-07 17:06:13.871 534-1680/system_process W/ActivityManager:   Native crash
04-07 17:06:13.871 534-1680/system_process W/ActivityManager:   Native crash: Segmentation fault

Java View.beginIndex doesn't support multiple view indexing

Can we make the interface below support multiple view indexing?

cbforest/Java/jni/native_view.cc

Lines 124 to 134 in d0afb82

 JNIEXPORT jlong JNICALL Java_com_couchbase_cbforest_View_beginIndex 

 (JNIEnv *env, jobject self, jlong dbHandle, jlong viewHandle) 

 { 

 C4View* view = (C4View*)viewHandle; 

 C4Error error; 

 C4Indexer* indexer = c4indexer_begin((C4Database*)dbHandle, &view, 1, &error); 

 if (!indexer) 

 throwError(env, error); 

 return (jlong)indexer; 

 }

Task: Update ForestDB to latest stable-branch commit

CBForest is currently using a ForestDB commit from April 23. We should update, since a number of bugs have been fixed. I tried updating to the latest stable branch commit (28feef96), but found two regressions, one critical:

MB-19970 Opening database with wrong password succeeds
MB-19969 Log messages going to stderr, bypassing log callback

I'll try again once these are fixed.

InclusiveEnd=false query does not work unless end key is specified

ForestDB disallows skipping keys if the keys are empty, which means that when CBForest uses an empty Collatable then any query with InclusiveEnd=false (or descending and InclusiveStart=false) will fail because the enumerator will be NULL due to an error unless the end (or start in the case of descending) key is specified.

WARNING: ForestDB error: invalid arguments (-1)

ForestDB storage cause native crush during push replication performance test with 200 docs

See https://github.com/couchbase/couchbase-lite-java-core/issues/731

JNI local-refs overflow

I found some more functions that can create an unlimited number of JNI local references, by converting an array of Java objects into a native array:

Java_com_couchbase_cbforest_DocumentIterator_initEnumerateSomeDocs
Java_com_couchbase_cbforest_Indexer_emit

These are subject to the same crash as in couchbase-lite-android#782.
They can be fixed the same way as in #64.

[C API] Missing method / logic to rekey a database

The iOS implementation uses some ForestDB copying functions that are not exposed to the C API. Either the copy mechanism or a simple "rekey" API will suffice.

C API for full-text search

Update README.md

With new information regarding platform support

Need cross-platform unit tests

The unit tests are written in Objective-C++ and use Xcode's test framework, so they can't be used on non-Apple platforms.

Passing class pointer to native could improve performance.

Accessing from native to java code is more expensive.
To obtain class pointer from native requires one additional JNI call.
http://www.ibm.com/developerworks/java/library/j-jni/#reaching

Passing class pointer to native could improve performance.

[C API] How to handle changes in a C4View version

It doesn't appear that the _version variable in C4View does anything, so is the caller responsible for tracking it and erasing the index on change. If so, what is the point of the index parameter in the c4view_open call?

Bitfields cause major issues on Windows

There is always an issue that I've noticed when assigning bools to 1 bit sections of a bit field. The bit field representation the compiler chooses is undefined and in the case of Windows it will truncate down to the lowest bit. The bool value however is 8 bits, which means that there is a 50% chance that the bool will turn into an incorrect value. For instance in DocEnumerator.cc there is this struct:

struct Options {
    unsigned                 skip;
    unsigned                 limit;
    bool                     descending     :1;
    bool                     inclusiveStart :1;
    bool                     inclusiveEnd   :1;
    bool                     includeDeleted :1;
    KeyStore::contentOptions contentOptions :4;

    static const Options kDefault;
};

And in C4DocEnumerator* c4db_enumerateAllDocs in c4Database.cc there is this logic:

options.descending = c4options->descending;
options.inclusiveStart = c4options->inclusiveStart;
options.inclusiveEnd = c4options->inclusiveEnd;
options.includeDeleted = c4options->includeDeleted;

In the unit test run, all four values in c4options are set to true and they are all bonafide bool types. However this is less important than what is actually stored inside them, which in this case is (in order) 233, 238, 112, and 5. To a bool, any value other than ZERO is true. So why is this a problem? Well in the bitfield the values get stored as follows: 233 & 1 (1), 238 & 1 (0), 112 & 1 (0), 5 & 1 (1) which turns them into true, false, false, true instead of all true. Since the comparison is effectively X & 1, this means that the miss rate is about 50%.

There are two ways to deal with this:

Don't use bit fields
Do the awkward dance of options.descending = c4options->descending != false

Crash in ForestDB (filemgr_is_fully_resident)

By running View unit test, the test crashes with same place frequently enough. See following crash log.

04-12 20:52:41.658 19453-19466/? A/libc: Fatal signal 11 (SIGSEGV) at 0x00000000 (code=128), thread 19466 (ationTestRunner)
04-12 20:52:41.710 147-147/? I/DEBUG: *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
04-12 20:52:41.710 147-147/? I/DEBUG: Build fingerprint: 'generic/vbox86p/vbox86p:4.4.4/KTU84P/eng.buildbot.20151118.000452:userdebug/test-keys'
04-12 20:52:41.710 147-147/? I/DEBUG: Revision: '0'
04-12 20:52:41.710 147-147/? I/DEBUG: pid: 19453, tid: 19466, name: ationTestRunner  >>> com.couchbase.lite.test <<<
04-12 20:52:41.710 147-147/? I/DEBUG: signal 11 (SIGSEGV), code 128 (SI_KERNEL), fault addr 00000000
04-12 20:52:41.866 147-147/? I/DEBUG:     eax 949a953c  ebx 949a953c  ecx b976e11c  edx 94ab661c
04-12 20:52:41.866 147-147/? I/DEBUG:     esi 94753100  edi 947522a0
04-12 20:52:41.866 147-147/? I/DEBUG:     xcs 00000073  xds 0000007b  xes 0000007b  xfs 00000000  xss 0000007b
04-12 20:52:41.866 147-147/? I/DEBUG:     eip 9473cc8d  ebp b976e108  esp 94ab661c  flags 00010282
04-12 20:52:41.866 147-147/? E/Corkscrew: unrecognized dwarf lower part encoding: 0x2f
04-12 20:52:41.866 147-147/? I/DEBUG: backtrace:
04-12 20:52:41.866 147-147/? I/DEBUG:     #00  pc 000eac8d  /data/app-lib/com.couchbase.lite.test-1/libCouchbaseLiteJavaForestDB.so (filemgr_is_fully_resident+141)
04-12 20:52:41.866 147-147/? I/DEBUG:     #01  pc 0012e4f2  /data/app-lib/com.couchbase.lite.test-1/libCouchbaseLiteJavaForestDB.so
04-12 20:52:41.866 147-147/? I/DEBUG:     #02  pc 0012e426  /data/app-lib/com.couchbase.lite.test-1/libCouchbaseLiteJavaForestDB.so (wal_flush+118)
04-12 20:52:41.866 147-147/? I/DEBUG:     #03  pc 000f9213  /data/app-lib/com.couchbase.lite.test-1/libCouchbaseLiteJavaForestDB.so (_fdb_commit+1571)
04-12 20:52:41.866 147-147/? I/DEBUG:     #04  pc 00101c3b  /data/app-lib/com.couchbase.lite.test-1/libCouchbaseLiteJavaForestDB.so (fdb_commit+91)
04-12 20:52:41.866 147-147/? I/DEBUG:     #05  pc 0013c77c  /data/app-lib/com.couchbase.lite.test-1/libCouchbaseLiteJavaForestDB.so (cbforest::Database::close()+92)
04-12 20:52:41.866 147-147/? I/DEBUG:     #06  pc 0013c6e4  /data/app-lib/com.couchbase.lite.test-1/libCouchbaseLiteJavaForestDB.so (cbforest::Database::~Database()+196)
04-12 20:52:41.866 147-147/? I/DEBUG: stack:

c4view_rekey implementation returns error -30

ForestDB rekeys a database by compacting it, however it disallows the compaction of a database file opened with FDB_SEQTREE_NOT_USE so the rekey operation cannot succeed.

(For reference, the location of this check is here)

C API for geo-query

Multithreaded exception when indexing views

See https://github.com/couchbase/couchbase-lite-android/issues/909#issuecomment-229565872 . @hideki wrote:

Condition that causes ForestDB error -39

Thread-1: keep adding document into forestdb.
Thread-2: keep run query

Where error -39 is thrown from

06-29 22:21:03.483 1804-1822/? E/CBFOREST: Forestdb Handle is being used by another thread (-39)
06-29 22:21:03.483 1804-1822/? W/CBFOREST: android stack dump
06-29 22:21:03.497 1804-1822/? W/CBFOREST: 000: 0x0xa28bc3ba dump_stack()
06-29 22:21:03.498 1804-1822/? W/CBFOREST: 001: 0x0xa28bc534 cbforest::error::_throw(fdb_status)
06-29 22:21:03.498 1804-1822/? W/CBFOREST: 002: 0x0xa28cf8e0 
06-29 22:21:03.499 1804-1822/? W/CBFOREST: 003: 0x0xa28cf992 cbforest::KeyStore::lastSequence() const
06-29 22:21:03.499 1804-1822/? W/CBFOREST: 004: 0x0xa28d9408 cbforest::MapReduceIndexer::startingSequence()
06-29 22:21:03.499 1804-1822/? W/CBFOREST: 005: 0x0xa28ed190 c4indexer_enumerateDocuments
06-29 22:21:03.499 1804-1822/? W/CBFOREST: 006: 0x0xa28f5874 Java_com_couchbase_cbforest_Indexer_iterateDocuments
06-29 22:21:03.499 1804-1822/? W/CBFOREST: android stack dump done

cbforest/CBForest/MapReduceIndex.cc

Line 390 in d6d38d9

_latestDbSequence = _writers[0]->index->sourceStore().lastSequence();

    sequence MapReduceIndexer::startingSequence() {
        _latestDbSequence = _writers[0]->index->sourceStore().lastSequence();
        ...
        ...
    }

Analysis

To insert document, add(...) method calls beginTransaction() and endTransaction(). In cbforest level, beginTransaction() and endTransaction(), forestdb is protected by _transactionMutex.
C4Indexer parameter of c4indexer_enumerateDocuments(C4Indexer *indexer, C4Error *outError) is protected by view->_mutex.lock();. But it does not protect forestdb instance.

master branch has compilation error for non-Apple platforms

Please see attached compilation error message

/Users/hideki/github/couchbase-lite-android/libraries/couchbase-lite-java-forestdb/jni/../vendor/cbforest/C/c4Document.cc:481:5: error: use of undeclared identifier 'SecureRandomize'
    SecureRandomize({r, sizeof(r)});
    ^
/Users/hideki/github/couchbase-lite-android/libraries/couchbase-lite-java-forestdb/jni/../vendor/cbforest/C/c4Document.cc:500:9: error: unknown type name 'md5Context'
        md5Context ctx;
        ^
/Users/hideki/github/couchbase-lite-android/libraries/couchbase-lite-java-forestdb/jni/../vendor/cbforest/C/c4Document.cc:502:35: error: no matching function for call to 'min'
        uint8_t revLen = (uint8_t)std::min(parentRevID.size, 255ul);
                                  ^~~~~~~~
/Users/hideki/Library/Android/sdk/ndk-bundle/sources/cxx-stl/gnu-libstdc++/4.8/include/bits/algorithmfwd.h:367:5: note: candidate template ignored: deduced conflicting types for parameter '_Tp' ('unsigned int' vs. 'unsigned long')
    min(const _Tp&, const _Tp&);
    ^
/Users/hideki/Library/Android/sdk/ndk-bundle/sources/cxx-stl/gnu-libstdc++/4.8/include/bits/stl_algo.h:4226:5: note: candidate template ignored: could not match 'initializer_list<type-parameter-0-0>' against 'unsigned int'
    min(initializer_list<_Tp> __l, _Compare __comp)
    ^
/Users/hideki/Library/Android/sdk/ndk-bundle/sources/cxx-stl/gnu-libstdc++/4.8/include/bits/algorithmfwd.h:371:5: note: candidate function template not viable: requires 3 arguments, but 2 were provided
    min(const _Tp&, const _Tp&, _Compare);
    ^
/Users/hideki/Library/Android/sdk/ndk-bundle/sources/cxx-stl/gnu-libstdc++/4.8/include/bits/stl_algo.h:4221:5: note: candidate function template not viable: requires single argument '__l', but 2 arguments were provided
    min(initializer_list<_Tp> __l)
    ^
/Users/hideki/github/couchbase-lite-android/libraries/couchbase-lite-java-forestdb/jni/../vendor/cbforest/C/c4Document.cc:513:9: error: unknown type name 'sha1Context'
        sha1Context ctx;
        ^
/Users/hideki/github/couchbase-lite-android/libraries/couchbase-lite-java-forestdb/jni/../vendor/cbforest/C/c4Document.cc:515:35: error: no matching function for call to 'min'
        uint8_t revLen = (uint8_t)std::min(parentRevID.size, 255ul);
                                  ^~~~~~~~
/Users/hideki/Library/Android/sdk/ndk-bundle/sources/cxx-stl/gnu-libstdc++/4.8/include/bits/algorithmfwd.h:367:5: note: candidate template ignored: deduced conflicting types for parameter '_Tp' ('unsigned int' vs. 'unsigned long')
    min(const _Tp&, const _Tp&);
    ^
/Users/hideki/Library/Android/sdk/ndk-bundle/sources/cxx-stl/gnu-libstdc++/4.8/include/bits/stl_algo.h:4226:5: note: candidate template ignored: could not match 'initializer_list<type-parameter-0-0>' against 'unsigned int'
    min(initializer_list<_Tp> __l, _Compare __comp)
    ^
/Users/hideki/Library/Android/sdk/ndk-bundle/sources/cxx-stl/gnu-libstdc++/4.8/include/bits/algorithmfwd.h:371:5: note: candidate function template not viable: requires 3 arguments, but 2 were provided
    min(const _Tp&, const _Tp&, _Compare);
    ^
/Users/hideki/Library/Android/sdk/ndk-bundle/sources/cxx-stl/gnu-libstdc++/4.8/include/bits/stl_algo.h:4221:5: note: candidate function template not viable: requires single argument '__l', but 2 arguments were provided
    min(initializer_list<_Tp> __l)
    ^
5 errors generated.
make: *** [/Users/hideki/github/couchbase-lite-android/libraries/couchbase-lite-java-forestdb/obj/local/armeabi-v7a/objs/CouchbaseLiteJavaForestDB//Users/hideki/github/couchbase-lite-android/libraries/couchbase-lite-java-forestdb/jni/__/vendor/cbforest/C/c4Document.o] Error 1

forestdb source files referenced incorrectly by xcode?

I just checked out the ForestDB branch of Couchbase-Lite, but I'm having trouble building the project. When selecting CBForest static and attempting to build, or when building my OS X framework with CBForest static as a dependency, I get the following error:

/Users/nathan/Projects/DEVONthink-To-Go/DEVONthink To Go 2/Submodules/Sync/Sync/External/TouchDB/vendor/CBForest/CBForest/RevTree.cc
/Users/nathan/Projects/DEVONthink-To-Go/DEVONthink To Go 2/Submodules/Sync/Sync/External/TouchDB/vendor/CBForest/CBForest/RevTree.cc:18:10: 'forestdb.h' file not found with <angled> include; use "quotes" instead

Normally, I'd assume this is because of some minor incompatibility in the project file -- you're sending headers to some place, I'm expecting to find them in another. But opening up the project, I found this:

Highlighting a randomly-selected file, I see:

Note that submodules appear to be checked out correctly:

Leilah:CBForest nathan$ pwd
/Users/nathan/Projects/Sync/Sync/External/TouchDB/vendor/CBForest
Leilah:CBForest nathan$ ls -1 vendor/forestdb/src/
api_wrapper.cc
arch.h
avltree.cc
avltree.h
blockcache.cc
blockcache.h

I'm of course not ruling out the possibility that I'm irredeemably stupid, but I thought this might be an issue with the project rather than me :-)

Build errors on Windows

@borrrden has been working on getting CBForest to build in MSVC on Windows. Via email:

Alright, it turns out the errors were not as bad as I thought, just a lot of leaning on GNU extensions. I was able to get cbforest to “build” on Windows. The reason I put it in quotes is because I only progressed to the point where all the compile errors were gone. I didn’t set it up to link with forestdb yet, so I still have linker errors and such to deal with but the core of forestdb compiles now (not sure about the tokenizer library). I’ve successfully compiled forestDB on Windows before without much of a problem so I don’t imagine it will be much more to do.

I summarized all the issues I found and how I solved them in this document: https://docs.google.com/spreadsheets/d/1up0TGzCwKXm7AfYozUgRpQ0zeE707NSxI-xU1dLoaPg/edit?usp=sharing

I also committed the fixes (minus one from the fts3_unicodesn repo that I don’t have access to) into the feature/c_api _windows branch.

[C API] Missing method for manual compaction

Needed by the "Compact" method of the storage engine

ForestDB Error: null

I found this while investigating the 1.2.0-forestdb Android LiteServ forestdb issues.

Running the cbl-webhook_events_not_posted.js node.js (https://github.com/couchbaselabs/sync-gateway-tests/blob/master/tests/cbl-webhook_events_not_posted.js#L120-L123) test, I see the below exception right before the test fails with 'unable to get doc rev'

https://gist.github.com/sethrosetter/7c87809454471b90bd8a5ecb05e29a36#file-gistfile1-txt-L90-L107

Allow enumerators to keep working after db or view is closed

@borrrden ran into a crash where a C4QueryEnumerator is used (or freed) after its C4View is closed. Closing the C4View calls fdb_close so the database handle is invalidated, including any fdb_iterators.

It would be a good idea to add some internal refcounting on C4View and C4Database so an enumerator keeps the reference alive.

ForestDB performance tanking on non SSD Window 10 machine

Looks like it may be due to compaction.

strlcpy() is BSD-only

Something I haven't fixed in my latest pull request #6 is the use of strlcpy(), which doesn't exist on Linux/glibc and presumably other platforms. Maybe there's a different way to implement the same thing with a more portable function?

Commit f033035 broke the database unit tests

Something in f033035 broke the database unit tests. It looks like the sequence on selected revision is not getting set properly anymore (perhaps other things as well).

[C API] isBodyAvailable()

Trying to load a body when it doesn't exist results in an error condition. It would be less cumbersome to expose this method to the C API for places like getPossibleAncestors and other places that iterate through c4doc_selectNextRevision.

After indexing a view, its lastSequenceIndexed may be less than db's lastSequence

As reported in couchbase/couchbase-lite-ios#1379 , after a CBForest view is indexed, the lastSequenceIndexed stored in the view may still be less than the database's lastSequence. This marks the view as not being up to date, so the next time it's queried it will get indexed again (with the same results, if the db hasn't changed.) This is bad for performance.

Failed to begin transaction - com.couchbase.cbforest.Database.beginTransaction

Found the following issue running (https://github.com/couchbaselabs/sync-gateway-tests/blob/master/tests/cbl-revisions.js) -https://gist.github.com/sethrosetter/b00106e612f6069d33cc07e335720059#file-gistfile1-txt-L66-L118

c4doc_enumerateAllDocs not honoring includeDeleted

Unit test here

The all docs enumerator is including deleted documents no matter what.

[C API] Missing method to get document by sequence

This is used in a number of places by both the ForestDB storage engine itself and query logic

After purging a doc, its rows are left behind in view indexes

See couchbase/couchbase-lite-ios#1164. This problem doesn't actually occur in SQLite based databases, but it does in ForestDB-based ones, and can be fixed at the CBForest level.

Non-trivial designated initializers not supported

The other remaining issue with my Linux build (at least as far as I made it in the build) is this piece of code (from KeyStore.cc) and a similar one two functions down:

        fdb_doc doc = {
            .key = (void*)key.buf,
            .keylen = key.size,
            .meta = (void*)meta.buf,
            .metalen = meta.size,
            .body  = (void*)body.buf,
            .bodylen = body.size,
        };

Which g++ (GCC 4.9.2 20150304 (prerelease)) doesn't quite like too much:

FAILED: /usr/bin/c++    -std=c++11 -fomit-frame-pointer -O2 -g -DNDEBUG -I../vendor/forestdb/include -I.. -MMD -MT CMakeFiles/cbforest.dir/CBForest/KeyStore.cc.o -MF CMakeFiles/cbforest.dir/CBForest/KeyStore.cc.o.d -o CMakeFiles/cbforest.dir/CBForest/KeyStore.cc.o -c ../CBForest/KeyStore.cc
../CBForest/KeyStore.cc: In member function ‘forestdb::sequence forestdb::KeyStoreWriter::set(forestdb::slice, forestdb::slice, forestdb::slice)’:
../CBForest/KeyStore.cc:116:9: sorry, unimplemented: non-trivial designated initializers not supported
         };
         ^
../CBForest/KeyStore.cc: In member function ‘bool forestdb::KeyStoreWriter::del(forestdb::slice)’:
../CBForest/KeyStore.cc:149:9: sorry, unimplemented: non-trivial designated initializers not supported
         };
         ^

The reason is that this is not valid C++, neither C++11 nor C++14, although it is valid C99. (C++ initializer lists got adopted, but designated initializers didn't.) In the interest of portability, maybe it can be swapped out with something more standard.

Encryption is platform-dependent

Uses CommonCrypto. For now it will be good enough to have a compiler directive to either compile or not compile it.

Compilation warning with 64bit Android - type mismatch between unsigned long and unsigned long long

        Log("DB %p: added %s --> %s (meta %s) (seq %llu)\n",
            _handle, key.hexCString(), body.hexCString(), meta.hexCString(), doc.seqnum);

[arm64-v8a] Compile++      : CouchbaseLiteJavaForestDB <= KeyStore.cc

/Users/hideki/github/couchbase-lite-android/libraries/couchbase-lite-java-forestdb/jni/../vendor/cbforest/CBForest/KeyStore.cc:148:78: warning: format specifies type 'unsigned long long' but the argument has type 'fdb_seqnum_t' (aka 'unsigned long') [-Wformat]
            _handle, key.hexCString(), body.hexCString(), meta.hexCString(), doc.seqnum);
                                                                             ^~~~~~~~~~
../vendor/cbforest/CBForest/LogInternal.hh:40:77: note: expanded from macro 'Log'
    #define Log(...)        __android_log_print(ANDROID_LOG_INFO,  LOG_TAG, __VA_ARGS__)
                                                                            ^~~~~~~~~~~
1 warning generated.

	JNIEXPORT jlong JNICALL Java_com_couchbase_cbforest_View_beginIndex
	(JNIEnv *env, jobject self, jlong dbHandle, jlong viewHandle)
	{
	C4View* view = (C4View*)viewHandle;
	C4Error error;
	C4Indexer* indexer = c4indexer_begin((C4Database*)dbHandle, &view, 1, &error);
	if (!indexer)
	throwError(env, error);
	return (jlong)indexer;
	}