warrenfalk / rocksdb-sharp Goto Github PK

View Code? Open in Web Editor NEW

206.0 14.0 65.0 464 KB

.net bindings for the rocksdb by facebook

License: Other

C# 99.80% Batchfile 0.20%

rocksdb wrapper mono database linux dll sln dotnet dotnet-core dotnetcore

rocksdb-sharp's Introduction

rocksdb-sharp

RocksDb for C#

RocksDB is a key-value database with a log-structured-merge design, optimized for flash and RAM storage, which can be tuned to balance write-, read-, and space-amplification factors.

RocksDB is developed by Facebook and is based on LevelDB. For more information about RocksDB, visit RocksDB and on GitHub

This library provides C# bindings for rocksdb, implemented as a wrapper for the native rocksdb DLL (unmanaged C++) via the rocksdb C API.

This is a multi-level binding, providing direct access to the C API functions (low level) plus some helper wrappers on those to aid in marshaling and exception handling (mid level) plus an idiomatic C# class hierarchy for ease of use (high level).

Example (High Level)

var options = new DbOptions()
    .SetCreateIfMissing(true);
using (var db = RocksDb.Open(options, path))
{
    // Using strings below, but can also use byte arrays for both keys and values
	// much care has been taken to minimize buffer copying
    db.Put("key", "value");
    string value = db.Get("key");
    db.Remove("key");
}

Usage

Using NuGet:

install-package RocksDbSharp

This will install the managed library which will use the unmanaged library installed on the machine at runtime. If you do not want to install the managed library, you can include it by additionally installing the "RocksDbNative" package.

install-package RocksDbNative

Requirements

On Linux and Mac, the snappy library (libsnappy) must be installed.

Caveats and Warnings:

64-bit only (Especially on Windows)

RocksDb is supported only in 64-bit mode. Although I contributed a fix that allows it to compile in 32-bit mode, this is untested and unsupported, may not work at all, and almost certainly will have at least some major issues and should not be attempted in production.

Extras

Windows Build Script: Building rocksdb for Windows is hard; this project contains a build script to make it easy.

Building Native Library

Pre-built native binaries can be downloaded from the releases page. You may also build them yourself.

(This is only a high level (and low level) wrapper on the unmanaged library and is not a managed C# port of the rocksdb database. The difficulty of a potential managed port library almost certainly far exceeds any usefulness of such a thing and so is not planned and will probably never exist).

This is now buildable on Windows thanks to the Bing team at Microsoft who are actively using rocksdb. Rocksdb-sharp should work on any platform provided the native unmanaged library is available.

Windows Native Build Instructions

Prerequisities:

Git for Windows (specifically, the git bash environment)
CMake
Visual Studio 2017 (older versions may work but are not tested)

Build Instructions:

Open "Developer Command Prompt for VS2017"
Run git's bash.exe
cd to the native-build folder within the repository
execute ./build-rocksdb.sh

This will create a rocksdb.dll and copy it to the where the .sln file is expecting it to be. (If you only need to run this in Windows, you can remove the references to the other two platform binaries from the .sln)

Linux Native Build Instructions

cd native-build
./build-rocksdb.sh

Mac Native Build Instructions

cd native-build
./build-rocksdb.sh

Note: On a Mac, although a change I contributed now allows RocksDb to compile in 32 bit, it is not supported and may not work. You should definitely only run 64-bit Mono to use RocksDb with Mono on Mac.

TODO

Many of the less-commonly-used C-API functions imports are yet to be included.

rocksdb-sharp's People

Contributors

Stargazers

Watchers

rocksdb-sharp's Issues

Create function for multiGet

Thus far this library has been working great, but I'm curious if there are any plans on implementing a function do handle the multiGet functionality of RocksDB, as they have in the Java or Python versions.

TTL support

Is there a way to set TTL? I found TTL option for WAL, but i think it's not the same as this:
https://github.com/facebook/rocksdb/wiki/Time-to-Live

Do you have plans to add it?

Native build with Zlib or LZ4 linked

Issue:
The current native build doesn't have Zlib or LZ4 linked in. There doesn't appear to be a way to plug-in these compression methods without directly linking into the native build. Zlib and LZ4 have perf characteristics that are better than Snappy in many workloads.

Request:
Please link in additional compression methods with the native build, preferably Zlib and/or LZ4. Thanks!

Question: how do I prevent LOG.old.XXXX to grow out of control

I use RocksDb.OpenReadOnly(options, config.GetString("akka.rocksdb.path"), false); Every 10 seconds or so. This way I keep a few recent readers present at all time. Problem is I end up with quite a lot of log.OLD files.

How do I keep my directory somewhat clean?

Update to newer upstream

We at https://github.com/Azure/iotedge use RocksDBSharp, and are interested in getting this library working against the latest upstream RocksDB version (v5.17.2 as of now). We're worried that we're missing out on security fixes made since the version that this library uses (v5.4), and other folks at Microsoft had difficulty porting this old version to support ARM.

I took a stab at it locally. Following the comment in Native.Raw.cs, I diffed upstream's c.h between the two versions. It looks like new API has been added but the existing API has mostly not changed. Only one rocksdb_options_set_use_direct_writes function was renamed, but this library did not wrap it in the first place. (I've pasted the full diff at the end.)

Based on that, I updated build-rocksdb.sh with

-ROCKSDBVERSION=6e05979
+ROCKSDBVERSION=v5.17.2
 ROCKSDBVNUM=5.4.6
 ROCKSDBSHARPVNUM=5.4.6.11
 SNAPPYVERSION=37aafc9e
 
-ROCKSDBREMOTE=https://github.com/warrenfalk/rocksdb
-SNAPPYREMOTE=https://github.com/warrenfalk/snappy
+ROCKSDBREMOTE=https://github.com/facebook/rocksdb
+SNAPPYREMOTE=https://github.com/google/snappy

and was able to build and have a successful run from the RocksDbSharpTest tests. I did not test on Windows.

I'm not familiar with RocksDB development, so I have some questions:

Do you foresee any problems with updating to v5.17 ? It seemed to me that the tests in FunctionalTests.cs are quite comprehensive.
If we submit a PR to update the build scripts to use v5.17, would you require that PR to also include wrappers for the new API added between v5.4 and v5.17 ? Or would it be sufficient to keep the exising wrappers as-is? (I want to scope what we need to do to get the update upstreamed into this repo.)

Diff of upstream C API

$ git diff 6e05979...v5.17.2 ./include/rocksdb/c.h

diff --git a/include/rocksdb/c.h b/include/rocksdb/c.h
index 3b087f305..0899ed625 100644
--- a/include/rocksdb/c.h
+++ b/include/rocksdb/c.h
@@ -1,8 +1,9 @@
-/*  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
-  This source code is licensed under the BSD-style license found in the
-  LICENSE file in the root directory of this source tree. An additional grant
-  of patent rights can be found in the PATENTS file in the same directory.
- Copyright (c) 2011 The LevelDB Authors. All rights reserved.
+//  Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under both the GPLv2 (found in the
+//  COPYING file in the root directory) and Apache 2.0 License
+//  (found in the LICENSE.Apache file in the root directory).
+
+/* Copyright (c) 2011 The LevelDB Authors. All rights reserved.
   Use of this source code is governed by a BSD-style license that can be
   found in the LICENSE file. See the AUTHORS file for names of contributors.
 
@@ -41,9 +42,6 @@
   (5) All of the pointer arguments must be non-NULL.
 */
 
-#ifndef STORAGE_ROCKSDB_INCLUDE_C_H_
-#define STORAGE_ROCKSDB_INCLUDE_C_H_
-
 #pragma once
 
 #ifdef _WIN32
@@ -81,6 +79,7 @@ typedef struct rocksdb_compactionfiltercontext_t
 typedef struct rocksdb_compactionfilterfactory_t
     rocksdb_compactionfilterfactory_t;
 typedef struct rocksdb_comparator_t      rocksdb_comparator_t;
+typedef struct rocksdb_dbpath_t          rocksdb_dbpath_t;
 typedef struct rocksdb_env_t             rocksdb_env_t;
 typedef struct rocksdb_fifo_compaction_options_t rocksdb_fifo_compaction_options_t;
 typedef struct rocksdb_filelock_t        rocksdb_filelock_t;
@@ -111,12 +110,30 @@ typedef struct rocksdb_envoptions_t      rocksdb_envoptions_t;
 typedef struct rocksdb_ingestexternalfileoptions_t rocksdb_ingestexternalfileoptions_t;
 typedef struct rocksdb_sstfilewriter_t   rocksdb_sstfilewriter_t;
 typedef struct rocksdb_ratelimiter_t     rocksdb_ratelimiter_t;
+typedef struct rocksdb_perfcontext_t     rocksdb_perfcontext_t;
+typedef struct rocksdb_pinnableslice_t rocksdb_pinnableslice_t;
+typedef struct rocksdb_transactiondb_options_t rocksdb_transactiondb_options_t;
+typedef struct rocksdb_transactiondb_t rocksdb_transactiondb_t;
+typedef struct rocksdb_transaction_options_t rocksdb_transaction_options_t;
+typedef struct rocksdb_optimistictransactiondb_t
+    rocksdb_optimistictransactiondb_t;
+typedef struct rocksdb_optimistictransaction_options_t
+    rocksdb_optimistictransaction_options_t;
+typedef struct rocksdb_transaction_t rocksdb_transaction_t;
+typedef struct rocksdb_checkpoint_t rocksdb_checkpoint_t;
+typedef struct rocksdb_wal_iterator_t rocksdb_wal_iterator_t;
+typedef struct rocksdb_wal_readoptions_t rocksdb_wal_readoptions_t;
+typedef struct rocksdb_memory_consumers_t rocksdb_memory_consumers_t;
+typedef struct rocksdb_memory_usage_t rocksdb_memory_usage_t;
 
 /* DB operations */
 
 extern ROCKSDB_LIBRARY_API rocksdb_t* rocksdb_open(
     const rocksdb_options_t* options, const char* name, char** errptr);
 
+extern ROCKSDB_LIBRARY_API rocksdb_t* rocksdb_open_with_ttl(
+    const rocksdb_options_t* options, const char* name, int ttl, char** errptr);
+
 extern ROCKSDB_LIBRARY_API rocksdb_t* rocksdb_open_for_read_only(
     const rocksdb_options_t* options, const char* name,
     unsigned char error_if_log_file_exist, char** errptr);
@@ -127,6 +144,10 @@ extern ROCKSDB_LIBRARY_API rocksdb_backup_engine_t* rocksdb_backup_engine_open(
 extern ROCKSDB_LIBRARY_API void rocksdb_backup_engine_create_new_backup(
     rocksdb_backup_engine_t* be, rocksdb_t* db, char** errptr);
 
+extern ROCKSDB_LIBRARY_API void rocksdb_backup_engine_create_new_backup_flush(
+    rocksdb_backup_engine_t* be, rocksdb_t* db, unsigned char flush_before_backup,
+    char** errptr);
+
 extern ROCKSDB_LIBRARY_API void rocksdb_backup_engine_purge_old_backups(
     rocksdb_backup_engine_t* be, uint32_t num_backups_to_keep, char** errptr);
 
@@ -137,6 +158,10 @@ extern ROCKSDB_LIBRARY_API void rocksdb_restore_options_destroy(
 extern ROCKSDB_LIBRARY_API void rocksdb_restore_options_set_keep_log_files(
     rocksdb_restore_options_t* opt, int v);
 
+extern ROCKSDB_LIBRARY_API void
+rocksdb_backup_engine_verify_backup(rocksdb_backup_engine_t* be,
+    uint32_t backup_id, char** errptr);
+
 extern ROCKSDB_LIBRARY_API void
 rocksdb_backup_engine_restore_db_from_latest_backup(
     rocksdb_backup_engine_t* be, const char* db_dir, const char* wal_dir,
@@ -169,6 +194,16 @@ extern ROCKSDB_LIBRARY_API void rocksdb_backup_engine_info_destroy(
 extern ROCKSDB_LIBRARY_API void rocksdb_backup_engine_close(
     rocksdb_backup_engine_t* be);
 
+extern ROCKSDB_LIBRARY_API rocksdb_checkpoint_t*
+rocksdb_checkpoint_object_create(rocksdb_t* db, char** errptr);
+
+extern ROCKSDB_LIBRARY_API void rocksdb_checkpoint_create(
+    rocksdb_checkpoint_t* checkpoint, const char* checkpoint_dir,
+    uint64_t log_size_for_flush, char** errptr);
+
+extern ROCKSDB_LIBRARY_API void rocksdb_checkpoint_object_destroy(
+    rocksdb_checkpoint_t* checkpoint);
+
 extern ROCKSDB_LIBRARY_API rocksdb_t* rocksdb_open_column_families(
     const rocksdb_options_t* options, const char* name, int num_column_families,
     const char** column_family_names,
@@ -271,6 +306,12 @@ extern ROCKSDB_LIBRARY_API void rocksdb_multi_get_cf(
 extern ROCKSDB_LIBRARY_API rocksdb_iterator_t* rocksdb_create_iterator(
     rocksdb_t* db, const rocksdb_readoptions_t* options);
 
+extern ROCKSDB_LIBRARY_API rocksdb_wal_iterator_t* rocksdb_get_updates_since(
+        rocksdb_t* db, uint64_t seq_number,
+        const rocksdb_wal_readoptions_t* options,
+        char** errptr
+);
+
 extern ROCKSDB_LIBRARY_API rocksdb_iterator_t* rocksdb_create_iterator_cf(
     rocksdb_t* db, const rocksdb_readoptions_t* options,
     rocksdb_column_family_handle_t* column_family);
@@ -374,6 +415,14 @@ extern ROCKSDB_LIBRARY_API const char* rocksdb_iter_value(
 extern ROCKSDB_LIBRARY_API void rocksdb_iter_get_error(
     const rocksdb_iterator_t*, char** errptr);
 
+extern ROCKSDB_LIBRARY_API void rocksdb_wal_iter_next(rocksdb_wal_iterator_t* iter);
+extern ROCKSDB_LIBRARY_API unsigned char rocksdb_wal_iter_valid(
+        const rocksdb_wal_iterator_t*);
+extern ROCKSDB_LIBRARY_API void rocksdb_wal_iter_status (const rocksdb_wal_iterator_t* iter, char** errptr) ;
+extern ROCKSDB_LIBRARY_API rocksdb_writebatch_t* rocksdb_wal_iter_get_batch (const rocksdb_wal_iterator_t* iter, uint64_t* seq) ;
+extern ROCKSDB_LIBRARY_API uint64_t rocksdb_get_latest_sequence_number (rocksdb_t *db);
+extern ROCKSDB_LIBRARY_API void rocksdb_wal_iter_destroy (const rocksdb_wal_iterator_t* iter) ;
+
 /* Write batch */
 
 extern ROCKSDB_LIBRARY_API rocksdb_writebatch_t* rocksdb_writebatch_create();
@@ -457,6 +506,8 @@ extern ROCKSDB_LIBRARY_API void rocksdb_writebatch_set_save_point(
     rocksdb_writebatch_t*);
 extern ROCKSDB_LIBRARY_API void rocksdb_writebatch_rollback_to_save_point(
     rocksdb_writebatch_t*, char** errptr);
+extern ROCKSDB_LIBRARY_API void rocksdb_writebatch_pop_save_point(
+    rocksdb_writebatch_t*, char** errptr);
 
 /* Write batch with index */
 
@@ -601,6 +652,18 @@ rocksdb_block_based_options_set_block_size_deviation(
 extern ROCKSDB_LIBRARY_API void
 rocksdb_block_based_options_set_block_restart_interval(
     rocksdb_block_based_table_options_t* options, int block_restart_interval);
+extern ROCKSDB_LIBRARY_API void
+rocksdb_block_based_options_set_index_block_restart_interval(
+    rocksdb_block_based_table_options_t* options, int index_block_restart_interval);
+extern ROCKSDB_LIBRARY_API void
+rocksdb_block_based_options_set_metadata_block_size(
+    rocksdb_block_based_table_options_t* options, uint64_t metadata_block_size);
+extern ROCKSDB_LIBRARY_API void
+rocksdb_block_based_options_set_partition_filters(
+    rocksdb_block_based_table_options_t* options, unsigned char partition_filters);
+extern ROCKSDB_LIBRARY_API void
+rocksdb_block_based_options_set_use_delta_encoding(
+    rocksdb_block_based_table_options_t* options, unsigned char use_delta_encoding);
 extern ROCKSDB_LIBRARY_API void rocksdb_block_based_options_set_filter_policy(
     rocksdb_block_based_table_options_t* options,
     rocksdb_filterpolicy_t* filter_policy);
@@ -620,6 +683,7 @@ extern ROCKSDB_LIBRARY_API void rocksdb_block_based_options_set_format_version(
 enum {
   rocksdb_block_based_table_index_type_binary_search = 0,
   rocksdb_block_based_table_index_type_hash_search = 1,
+  rocksdb_block_based_table_index_type_two_level_index_search = 2,
 };
 extern ROCKSDB_LIBRARY_API void rocksdb_block_based_options_set_index_type(
     rocksdb_block_based_table_options_t*, int);  // uses one of the above enums
@@ -630,8 +694,14 @@ extern ROCKSDB_LIBRARY_API void
 rocksdb_block_based_options_set_cache_index_and_filter_blocks(
     rocksdb_block_based_table_options_t*, unsigned char);
 extern ROCKSDB_LIBRARY_API void
+rocksdb_block_based_options_set_cache_index_and_filter_blocks_with_high_priority(
+    rocksdb_block_based_table_options_t*, unsigned char);
+extern ROCKSDB_LIBRARY_API void
 rocksdb_block_based_options_set_pin_l0_filter_and_index_blocks_in_cache(
     rocksdb_block_based_table_options_t*, unsigned char);
+extern ROCKSDB_LIBRARY_API void
+rocksdb_block_based_options_set_pin_top_level_index_and_filter(
+    rocksdb_block_based_table_options_t*, unsigned char);
 extern ROCKSDB_LIBRARY_API void rocksdb_options_set_block_based_table_factory(
     rocksdb_options_t* opt, rocksdb_block_based_table_options_t* table_options);
 
@@ -670,6 +740,9 @@ extern ROCKSDB_LIBRARY_API void rocksdb_options_optimize_level_style_compaction(
 extern ROCKSDB_LIBRARY_API void
 rocksdb_options_optimize_universal_style_compaction(
     rocksdb_options_t* opt, uint64_t memtable_memory_budget);
+extern ROCKSDB_LIBRARY_API void
+rocksdb_options_set_allow_ingest_behind(rocksdb_options_t*,
+                                                   unsigned char);
 extern ROCKSDB_LIBRARY_API void rocksdb_options_set_compaction_filter(
     rocksdb_options_t*, rocksdb_compactionfilter_t*);
 extern ROCKSDB_LIBRARY_API void rocksdb_options_set_compaction_filter_factory(
@@ -693,6 +766,9 @@ extern ROCKSDB_LIBRARY_API void rocksdb_options_set_error_if_exists(
     rocksdb_options_t*, unsigned char);
 extern ROCKSDB_LIBRARY_API void rocksdb_options_set_paranoid_checks(
     rocksdb_options_t*, unsigned char);
+extern ROCKSDB_LIBRARY_API void rocksdb_options_set_db_paths(rocksdb_options_t*,
+                                                             const rocksdb_dbpath_t** path_values,
+                                                             size_t num_paths);
 extern ROCKSDB_LIBRARY_API void rocksdb_options_set_env(rocksdb_options_t*,
                                                         rocksdb_env_t*);
 extern ROCKSDB_LIBRARY_API void rocksdb_options_set_info_log(rocksdb_options_t*,
@@ -705,6 +781,8 @@ extern ROCKSDB_LIBRARY_API void rocksdb_options_set_db_write_buffer_size(
     rocksdb_options_t*, size_t);
 extern ROCKSDB_LIBRARY_API void rocksdb_options_set_max_open_files(
     rocksdb_options_t*, int);
+extern ROCKSDB_LIBRARY_API void rocksdb_options_set_max_file_opening_threads(
+    rocksdb_options_t*, int);
 extern ROCKSDB_LIBRARY_API void rocksdb_options_set_max_total_wal_size(
     rocksdb_options_t* opt, uint64_t n);
 extern ROCKSDB_LIBRARY_API void rocksdb_options_set_compression_options(
@@ -737,6 +815,9 @@ rocksdb_options_set_max_bytes_for_level_multiplier_additional(
     rocksdb_options_t*, int* level_values, size_t num_levels);
 extern ROCKSDB_LIBRARY_API void rocksdb_options_enable_statistics(
     rocksdb_options_t*);
+extern ROCKSDB_LIBRARY_API void
+rocksdb_options_set_skip_stats_update_on_db_open(rocksdb_options_t* opt,
+                                                 unsigned char val);
 
 /* returns a pointer to a malloc()-ed, null terminated string */
 extern ROCKSDB_LIBRARY_API char* rocksdb_options_statistics_get_string(
@@ -749,6 +830,12 @@ rocksdb_options_set_min_write_buffer_number_to_merge(rocksdb_options_t*, int);
 extern ROCKSDB_LIBRARY_API void
 rocksdb_options_set_max_write_buffer_number_to_maintain(rocksdb_options_t*,
                                                         int);
+extern ROCKSDB_LIBRARY_API void rocksdb_options_set_enable_pipelined_write(
+    rocksdb_options_t*, unsigned char);
+extern ROCKSDB_LIBRARY_API void rocksdb_options_set_max_subcompactions(
+    rocksdb_options_t*, uint32_t);
+extern ROCKSDB_LIBRARY_API void rocksdb_options_set_max_background_jobs(
+    rocksdb_options_t*, int);
 extern ROCKSDB_LIBRARY_API void rocksdb_options_set_max_background_compactions(
     rocksdb_options_t*, int);
 extern ROCKSDB_LIBRARY_API void rocksdb_options_set_base_background_compactions(
@@ -804,8 +891,9 @@ extern ROCKSDB_LIBRARY_API void rocksdb_options_set_allow_mmap_writes(
     rocksdb_options_t*, unsigned char);
 extern ROCKSDB_LIBRARY_API void rocksdb_options_set_use_direct_reads(
     rocksdb_options_t*, unsigned char);
-extern ROCKSDB_LIBRARY_API void rocksdb_options_set_use_direct_writes(
-    rocksdb_options_t*, unsigned char);
+extern ROCKSDB_LIBRARY_API void
+rocksdb_options_set_use_direct_io_for_flush_and_compaction(rocksdb_options_t*,
+                                                           unsigned char);
 extern ROCKSDB_LIBRARY_API void rocksdb_options_set_is_fd_close_on_exec(
     rocksdb_options_t*, unsigned char);
 extern ROCKSDB_LIBRARY_API void rocksdb_options_set_skip_log_error_on_recovery(
@@ -820,6 +908,10 @@ extern ROCKSDB_LIBRARY_API void rocksdb_options_set_use_adaptive_mutex(
     rocksdb_options_t*, unsigned char);
 extern ROCKSDB_LIBRARY_API void rocksdb_options_set_bytes_per_sync(
     rocksdb_options_t*, uint64_t);
+extern ROCKSDB_LIBRARY_API void rocksdb_options_set_wal_bytes_per_sync(
+        rocksdb_options_t*, uint64_t);
+extern ROCKSDB_LIBRARY_API void
+rocksdb_options_set_writable_file_max_buffer_size(rocksdb_options_t*, uint64_t);
 extern ROCKSDB_LIBRARY_API void
 rocksdb_options_set_allow_concurrent_memtable_write(rocksdb_options_t*,
                                                     unsigned char);
@@ -910,6 +1002,99 @@ extern ROCKSDB_LIBRARY_API rocksdb_ratelimiter_t* rocksdb_ratelimiter_create(
     int64_t rate_bytes_per_sec, int64_t refill_period_us, int32_t fairness);
 extern ROCKSDB_LIBRARY_API void rocksdb_ratelimiter_destroy(rocksdb_ratelimiter_t*);
 
+/* PerfContext */
+enum {
+  rocksdb_uninitialized = 0,
+  rocksdb_disable = 1,
+  rocksdb_enable_count = 2,
+  rocksdb_enable_time_except_for_mutex = 3,
+  rocksdb_enable_time = 4,
+  rocksdb_out_of_bounds = 5
+};
+
+enum {
+  rocksdb_user_key_comparison_count = 0,
+  rocksdb_block_cache_hit_count,
+  rocksdb_block_read_count,
+  rocksdb_block_read_byte,
+  rocksdb_block_read_time,
+  rocksdb_block_checksum_time,
+  rocksdb_block_decompress_time,
+  rocksdb_get_read_bytes,
+  rocksdb_multiget_read_bytes,
+  rocksdb_iter_read_bytes,
+  rocksdb_internal_key_skipped_count,
+  rocksdb_internal_delete_skipped_count,
+  rocksdb_internal_recent_skipped_count,
+  rocksdb_internal_merge_count,
+  rocksdb_get_snapshot_time,
+  rocksdb_get_from_memtable_time,
+  rocksdb_get_from_memtable_count,
+  rocksdb_get_post_process_time,
+  rocksdb_get_from_output_files_time,
+  rocksdb_seek_on_memtable_time,
+  rocksdb_seek_on_memtable_count,
+  rocksdb_next_on_memtable_count,
+  rocksdb_prev_on_memtable_count,
+  rocksdb_seek_child_seek_time,
+  rocksdb_seek_child_seek_count,
+  rocksdb_seek_min_heap_time,
+  rocksdb_seek_max_heap_time,
+  rocksdb_seek_internal_seek_time,
+  rocksdb_find_next_user_entry_time,
+  rocksdb_write_wal_time,
+  rocksdb_write_memtable_time,
+  rocksdb_write_delay_time,
+  rocksdb_write_pre_and_post_process_time,
+  rocksdb_db_mutex_lock_nanos,
+  rocksdb_db_condition_wait_nanos,
+  rocksdb_merge_operator_time_nanos,
+  rocksdb_read_index_block_nanos,
+  rocksdb_read_filter_block_nanos,
+  rocksdb_new_table_block_iter_nanos,
+  rocksdb_new_table_iterator_nanos,
+  rocksdb_block_seek_nanos,
+  rocksdb_find_table_nanos,
+  rocksdb_bloom_memtable_hit_count,
+  rocksdb_bloom_memtable_miss_count,
+  rocksdb_bloom_sst_hit_count,
+  rocksdb_bloom_sst_miss_count,
+  rocksdb_key_lock_wait_time,
+  rocksdb_key_lock_wait_count,
+  rocksdb_env_new_sequential_file_nanos,
+  rocksdb_env_new_random_access_file_nanos,
+  rocksdb_env_new_writable_file_nanos,
+  rocksdb_env_reuse_writable_file_nanos,
+  rocksdb_env_new_random_rw_file_nanos,
+  rocksdb_env_new_directory_nanos,
+  rocksdb_env_file_exists_nanos,
+  rocksdb_env_get_children_nanos,
+  rocksdb_env_get_children_file_attributes_nanos,
+  rocksdb_env_delete_file_nanos,
+  rocksdb_env_create_dir_nanos,
+  rocksdb_env_create_dir_if_missing_nanos,
+  rocksdb_env_delete_dir_nanos,
+  rocksdb_env_get_file_size_nanos,
+  rocksdb_env_get_file_modification_time_nanos,
+  rocksdb_env_rename_file_nanos,
+  rocksdb_env_link_file_nanos,
+  rocksdb_env_lock_file_nanos,
+  rocksdb_env_unlock_file_nanos,
+  rocksdb_env_new_logger_nanos,
+  rocksdb_total_metric_count = 68
+};
+
+extern ROCKSDB_LIBRARY_API void rocksdb_set_perf_level(int);
+extern ROCKSDB_LIBRARY_API rocksdb_perfcontext_t* rocksdb_perfcontext_create();
+extern ROCKSDB_LIBRARY_API void rocksdb_perfcontext_reset(
+    rocksdb_perfcontext_t* context);
+extern ROCKSDB_LIBRARY_API char* rocksdb_perfcontext_report(
+    rocksdb_perfcontext_t* context, unsigned char exclude_zero_counters);
+extern ROCKSDB_LIBRARY_API uint64_t rocksdb_perfcontext_metric(
+    rocksdb_perfcontext_t* context, int metric);
+extern ROCKSDB_LIBRARY_API void rocksdb_perfcontext_destroy(
+    rocksdb_perfcontext_t* context);
+
 /* Compaction Filter */
 
 extern ROCKSDB_LIBRARY_API rocksdb_compactionfilter_t*
@@ -1009,16 +1194,29 @@ extern ROCKSDB_LIBRARY_API void rocksdb_readoptions_set_snapshot(
     rocksdb_readoptions_t*, const rocksdb_snapshot_t*);
 extern ROCKSDB_LIBRARY_API void rocksdb_readoptions_set_iterate_upper_bound(
     rocksdb_readoptions_t*, const char* key, size_t keylen);
+extern ROCKSDB_LIBRARY_API void rocksdb_readoptions_set_iterate_lower_bound(
+    rocksdb_readoptions_t*, const char* key, size_t keylen);
 extern ROCKSDB_LIBRARY_API void rocksdb_readoptions_set_read_tier(
     rocksdb_readoptions_t*, int);
 extern ROCKSDB_LIBRARY_API void rocksdb_readoptions_set_tailing(
     rocksdb_readoptions_t*, unsigned char);
+// The functionality that this option controlled has been removed.
+extern ROCKSDB_LIBRARY_API void rocksdb_readoptions_set_managed(
+    rocksdb_readoptions_t*, unsigned char);
 extern ROCKSDB_LIBRARY_API void rocksdb_readoptions_set_readahead_size(
     rocksdb_readoptions_t*, size_t);
+extern ROCKSDB_LIBRARY_API void rocksdb_readoptions_set_prefix_same_as_start(
+    rocksdb_readoptions_t*, unsigned char);
 extern ROCKSDB_LIBRARY_API void rocksdb_readoptions_set_pin_data(
     rocksdb_readoptions_t*, unsigned char);
 extern ROCKSDB_LIBRARY_API void rocksdb_readoptions_set_total_order_seek(
     rocksdb_readoptions_t*, unsigned char);
+extern ROCKSDB_LIBRARY_API void rocksdb_readoptions_set_max_skippable_internal_keys(
+    rocksdb_readoptions_t*, uint64_t);
+extern ROCKSDB_LIBRARY_API void rocksdb_readoptions_set_background_purge_on_iterator_cleanup(
+    rocksdb_readoptions_t*, unsigned char);
+extern ROCKSDB_LIBRARY_API void rocksdb_readoptions_set_ignore_range_deletions(
+    rocksdb_readoptions_t*, unsigned char);
 
 /* Write options */
 
@@ -1030,6 +1228,12 @@ extern ROCKSDB_LIBRARY_API void rocksdb_writeoptions_set_sync(
     rocksdb_writeoptions_t*, unsigned char);
 extern ROCKSDB_LIBRARY_API void rocksdb_writeoptions_disable_WAL(
     rocksdb_writeoptions_t* opt, int disable);
+extern ROCKSDB_LIBRARY_API void rocksdb_writeoptions_set_ignore_missing_column_families(
+    rocksdb_writeoptions_t*, unsigned char);
+extern ROCKSDB_LIBRARY_API void rocksdb_writeoptions_set_no_slowdown(
+    rocksdb_writeoptions_t*, unsigned char);
+extern ROCKSDB_LIBRARY_API void rocksdb_writeoptions_set_low_pri(
+    rocksdb_writeoptions_t*, unsigned char);
 
 /* Compact range options */
 
@@ -1040,6 +1244,9 @@ extern ROCKSDB_LIBRARY_API void rocksdb_compactoptions_destroy(
 extern ROCKSDB_LIBRARY_API void
 rocksdb_compactoptions_set_exclusive_manual_compaction(
     rocksdb_compactoptions_t*, unsigned char);
+extern ROCKSDB_LIBRARY_API void
+rocksdb_compactoptions_set_bottommost_level_compaction(
+    rocksdb_compactoptions_t*, unsigned char);
 extern ROCKSDB_LIBRARY_API void rocksdb_compactoptions_set_change_level(
     rocksdb_compactoptions_t*, unsigned char);
 extern ROCKSDB_LIBRARY_API void rocksdb_compactoptions_set_target_level(
@@ -1066,6 +1273,11 @@ rocksdb_cache_get_usage(rocksdb_cache_t* cache);
 extern ROCKSDB_LIBRARY_API size_t
 rocksdb_cache_get_pinned_usage(rocksdb_cache_t* cache);
 
+/* DBPath */
+
+extern ROCKSDB_LIBRARY_API rocksdb_dbpath_t* rocksdb_dbpath_create(const char* path, uint64_t target_size);
+extern ROCKSDB_LIBRARY_API void rocksdb_dbpath_destroy(rocksdb_dbpath_t*);
+
 /* Env */
 
 extern ROCKSDB_LIBRARY_API rocksdb_env_t* rocksdb_create_default_env();
@@ -1096,8 +1308,19 @@ extern ROCKSDB_LIBRARY_API void rocksdb_sstfilewriter_open(
 extern ROCKSDB_LIBRARY_API void rocksdb_sstfilewriter_add(
     rocksdb_sstfilewriter_t* writer, const char* key, size_t keylen,
     const char* val, size_t vallen, char** errptr);
+extern ROCKSDB_LIBRARY_API void rocksdb_sstfilewriter_put(
+    rocksdb_sstfilewriter_t* writer, const char* key, size_t keylen,
+    const char* val, size_t vallen, char** errptr);
+extern ROCKSDB_LIBRARY_API void rocksdb_sstfilewriter_merge(
+    rocksdb_sstfilewriter_t* writer, const char* key, size_t keylen,
+    const char* val, size_t vallen, char** errptr);
+extern ROCKSDB_LIBRARY_API void rocksdb_sstfilewriter_delete(
+    rocksdb_sstfilewriter_t* writer, const char* key, size_t keylen,
+    char** errptr);
 extern ROCKSDB_LIBRARY_API void rocksdb_sstfilewriter_finish(
     rocksdb_sstfilewriter_t* writer, char** errptr);
+extern ROCKSDB_LIBRARY_API void rocksdb_sstfilewriter_file_size(
+    rocksdb_sstfilewriter_t* writer, uint64_t* file_size);
 extern ROCKSDB_LIBRARY_API void rocksdb_sstfilewriter_destroy(
     rocksdb_sstfilewriter_t* writer);
 
@@ -1117,6 +1340,10 @@ extern ROCKSDB_LIBRARY_API void
 rocksdb_ingestexternalfileoptions_set_allow_blocking_flush(
     rocksdb_ingestexternalfileoptions_t* opt,
     unsigned char allow_blocking_flush);
+extern ROCKSDB_LIBRARY_API void
+rocksdb_ingestexternalfileoptions_set_ingest_behind(
+    rocksdb_ingestexternalfileoptions_t* opt,
+    unsigned char ingest_behind);
 extern ROCKSDB_LIBRARY_API void rocksdb_ingestexternalfileoptions_destroy(
     rocksdb_ingestexternalfileoptions_t* opt);
 
@@ -1213,12 +1440,267 @@ extern ROCKSDB_LIBRARY_API void rocksdb_delete_file_in_range_cf(
     const char* start_key, size_t start_key_len, const char* limit_key,
     size_t limit_key_len, char** errptr);
 
+/* Transactions */
+
+extern ROCKSDB_LIBRARY_API rocksdb_column_family_handle_t*
+rocksdb_transactiondb_create_column_family(
+    rocksdb_transactiondb_t* txn_db,
+    const rocksdb_options_t* column_family_options,
+    const char* column_family_name, char** errptr);
+
+extern ROCKSDB_LIBRARY_API rocksdb_transactiondb_t* rocksdb_transactiondb_open(
+    const rocksdb_options_t* options,
+    const rocksdb_transactiondb_options_t* txn_db_options, const char* name,
+    char** errptr);
+
+extern ROCKSDB_LIBRARY_API const rocksdb_snapshot_t*
+rocksdb_transactiondb_create_snapshot(rocksdb_transactiondb_t* txn_db);
+
+extern ROCKSDB_LIBRARY_API void rocksdb_transactiondb_release_snapshot(
+    rocksdb_transactiondb_t* txn_db, const rocksdb_snapshot_t* snapshot);
+
+extern ROCKSDB_LIBRARY_API rocksdb_transaction_t* rocksdb_transaction_begin(
+    rocksdb_transactiondb_t* txn_db,
+    const rocksdb_writeoptions_t* write_options,
+    const rocksdb_transaction_options_t* txn_options,
+    rocksdb_transaction_t* old_txn);
+
+extern ROCKSDB_LIBRARY_API void rocksdb_transaction_commit(
+    rocksdb_transaction_t* txn, char** errptr);
+
+extern ROCKSDB_LIBRARY_API void rocksdb_transaction_rollback(
+    rocksdb_transaction_t* txn, char** errptr);
+
+extern ROCKSDB_LIBRARY_API void rocksdb_transaction_set_savepoint(
+    rocksdb_transaction_t* txn);
+
+extern ROCKSDB_LIBRARY_API void rocksdb_transaction_rollback_to_savepoint(
+    rocksdb_transaction_t* txn, char** errptr);
+
+extern ROCKSDB_LIBRARY_API void rocksdb_transaction_destroy(
+    rocksdb_transaction_t* txn);
+
+// This snapshot should be freed using rocksdb_free
+extern ROCKSDB_LIBRARY_API const rocksdb_snapshot_t*
+rocksdb_transaction_get_snapshot(rocksdb_transaction_t* txn);
+
+extern ROCKSDB_LIBRARY_API char* rocksdb_transaction_get(
+    rocksdb_transaction_t* txn, const rocksdb_readoptions_t* options,
+    const char* key, size_t klen, size_t* vlen, char** errptr);
+
+extern ROCKSDB_LIBRARY_API char* rocksdb_transaction_get_cf(
+    rocksdb_transaction_t* txn, const rocksdb_readoptions_t* options,
+    rocksdb_column_family_handle_t* column_family, const char* key, size_t klen,
+    size_t* vlen, char** errptr);
+
+extern ROCKSDB_LIBRARY_API char* rocksdb_transaction_get_for_update(
+    rocksdb_transaction_t* txn, const rocksdb_readoptions_t* options,
+    const char* key, size_t klen, size_t* vlen, unsigned char exclusive,
+    char** errptr);
+
+extern ROCKSDB_LIBRARY_API char* rocksdb_transactiondb_get(
+    rocksdb_transactiondb_t* txn_db, const rocksdb_readoptions_t* options,
+    const char* key, size_t klen, size_t* vlen, char** errptr);
+
+extern ROCKSDB_LIBRARY_API char* rocksdb_transactiondb_get_cf(
+    rocksdb_transactiondb_t* txn_db, const rocksdb_readoptions_t* options,
+    rocksdb_column_family_handle_t* column_family, const char* key,
+    size_t keylen, size_t* vallen, char** errptr);
+
+extern ROCKSDB_LIBRARY_API void rocksdb_transaction_put(
+    rocksdb_transaction_t* txn, const char* key, size_t klen, const char* val,
+    size_t vlen, char** errptr);
+
+extern ROCKSDB_LIBRARY_API void rocksdb_transaction_put_cf(
+    rocksdb_transaction_t* txn, rocksdb_column_family_handle_t* column_family,
+    const char* key, size_t klen, const char* val, size_t vlen, char** errptr);
+
+extern ROCKSDB_LIBRARY_API void rocksdb_transactiondb_put(
+    rocksdb_transactiondb_t* txn_db, const rocksdb_writeoptions_t* options,
+    const char* key, size_t klen, const char* val, size_t vlen, char** errptr);
+
+extern ROCKSDB_LIBRARY_API void rocksdb_transactiondb_put_cf(
+    rocksdb_transactiondb_t* txn_db, const rocksdb_writeoptions_t* options,
+    rocksdb_column_family_handle_t* column_family, const char* key,
+    size_t keylen, const char* val, size_t vallen, char** errptr);
+
+extern ROCKSDB_LIBRARY_API void rocksdb_transactiondb_write(
+    rocksdb_transactiondb_t* txn_db, const rocksdb_writeoptions_t* options,
+    rocksdb_writebatch_t *batch, char** errptr);
+
+extern ROCKSDB_LIBRARY_API void rocksdb_transaction_merge(
+    rocksdb_transaction_t* txn, const char* key, size_t klen, const char* val,
+    size_t vlen, char** errptr);
+
+extern ROCKSDB_LIBRARY_API void rocksdb_transactiondb_merge(
+    rocksdb_transactiondb_t* txn_db, const rocksdb_writeoptions_t* options,
+    const char* key, size_t klen, const char* val, size_t vlen, char** errptr);
+
+extern ROCKSDB_LIBRARY_API void rocksdb_transaction_delete(
+    rocksdb_transaction_t* txn, const char* key, size_t klen, char** errptr);
+
+extern ROCKSDB_LIBRARY_API void rocksdb_transaction_delete_cf(
+    rocksdb_transaction_t* txn, rocksdb_column_family_handle_t* column_family,
+    const char* key, size_t klen, char** errptr);
+
+extern ROCKSDB_LIBRARY_API void rocksdb_transactiondb_delete(
+    rocksdb_transactiondb_t* txn_db, const rocksdb_writeoptions_t* options,
+    const char* key, size_t klen, char** errptr);
+
+extern ROCKSDB_LIBRARY_API void rocksdb_transactiondb_delete_cf(
+    rocksdb_transactiondb_t* txn_db, const rocksdb_writeoptions_t* options,
+    rocksdb_column_family_handle_t* column_family, const char* key,
+    size_t keylen, char** errptr);
+
+extern ROCKSDB_LIBRARY_API rocksdb_iterator_t*
+rocksdb_transaction_create_iterator(rocksdb_transaction_t* txn,
+                                    const rocksdb_readoptions_t* options);
+
+extern ROCKSDB_LIBRARY_API rocksdb_iterator_t*
+rocksdb_transaction_create_iterator_cf(
+    rocksdb_transaction_t* txn, const rocksdb_readoptions_t* options,
+    rocksdb_column_family_handle_t* column_family);
+
+extern ROCKSDB_LIBRARY_API rocksdb_iterator_t*
+rocksdb_transactiondb_create_iterator(rocksdb_transactiondb_t* txn_db,
+                                      const rocksdb_readoptions_t* options);
+
+extern ROCKSDB_LIBRARY_API void rocksdb_transactiondb_close(
+    rocksdb_transactiondb_t* txn_db);
+
+extern ROCKSDB_LIBRARY_API rocksdb_checkpoint_t*
+rocksdb_transactiondb_checkpoint_object_create(rocksdb_transactiondb_t* txn_db,
+                                               char** errptr);
+
+extern ROCKSDB_LIBRARY_API rocksdb_optimistictransactiondb_t*
+rocksdb_optimistictransactiondb_open(const rocksdb_options_t* options,
+                                     const char* name, char** errptr);
+
+extern ROCKSDB_LIBRARY_API rocksdb_optimistictransactiondb_t*
+rocksdb_optimistictransactiondb_open_column_families(
+    const rocksdb_options_t* options, const char* name, int num_column_families,
+    const char** column_family_names,
+    const rocksdb_options_t** column_family_options,
+    rocksdb_column_family_handle_t** column_family_handles, char** errptr);
+
+extern ROCKSDB_LIBRARY_API rocksdb_t*
+rocksdb_optimistictransactiondb_get_base_db(
+    rocksdb_optimistictransactiondb_t* otxn_db);
+
+extern ROCKSDB_LIBRARY_API void rocksdb_optimistictransactiondb_close_base_db(
+    rocksdb_t* base_db);
+
+extern ROCKSDB_LIBRARY_API rocksdb_transaction_t*
+rocksdb_optimistictransaction_begin(
+    rocksdb_optimistictransactiondb_t* otxn_db,
+    const rocksdb_writeoptions_t* write_options,
+    const rocksdb_optimistictransaction_options_t* otxn_options,
+    rocksdb_transaction_t* old_txn);
+
+extern ROCKSDB_LIBRARY_API void rocksdb_optimistictransactiondb_close(
+    rocksdb_optimistictransactiondb_t* otxn_db);
+
+/* Transaction Options */
+
+extern ROCKSDB_LIBRARY_API rocksdb_transactiondb_options_t*
+rocksdb_transactiondb_options_create();
+
+extern ROCKSDB_LIBRARY_API void rocksdb_transactiondb_options_destroy(
+    rocksdb_transactiondb_options_t* opt);
+
+extern ROCKSDB_LIBRARY_API void rocksdb_transactiondb_options_set_max_num_locks(
+    rocksdb_transactiondb_options_t* opt, int64_t max_num_locks);
+
+extern ROCKSDB_LIBRARY_API void rocksdb_transactiondb_options_set_num_stripes(
+    rocksdb_transactiondb_options_t* opt, size_t num_stripes);
+
+extern ROCKSDB_LIBRARY_API void
+rocksdb_transactiondb_options_set_transaction_lock_timeout(
+    rocksdb_transactiondb_options_t* opt, int64_t txn_lock_timeout);
+
+extern ROCKSDB_LIBRARY_API void
+rocksdb_transactiondb_options_set_default_lock_timeout(
+    rocksdb_transactiondb_options_t* opt, int64_t default_lock_timeout);
+
+extern ROCKSDB_LIBRARY_API rocksdb_transaction_options_t*
+rocksdb_transaction_options_create();
+
+extern ROCKSDB_LIBRARY_API void rocksdb_transaction_options_destroy(
+    rocksdb_transaction_options_t* opt);
+
+extern ROCKSDB_LIBRARY_API void rocksdb_transaction_options_set_set_snapshot(
+    rocksdb_transaction_options_t* opt, unsigned char v);
+
+extern ROCKSDB_LIBRARY_API void rocksdb_transaction_options_set_deadlock_detect(
+    rocksdb_transaction_options_t* opt, unsigned char v);
+
+extern ROCKSDB_LIBRARY_API void rocksdb_transaction_options_set_lock_timeout(
+    rocksdb_transaction_options_t* opt, int64_t lock_timeout);
+
+extern ROCKSDB_LIBRARY_API void rocksdb_transaction_options_set_expiration(
+    rocksdb_transaction_options_t* opt, int64_t expiration);
+
+extern ROCKSDB_LIBRARY_API void
+rocksdb_transaction_options_set_deadlock_detect_depth(
+    rocksdb_transaction_options_t* opt, int64_t depth);
+
+extern ROCKSDB_LIBRARY_API void
+rocksdb_transaction_options_set_max_write_batch_size(
+    rocksdb_transaction_options_t* opt, size_t size);
+
+extern ROCKSDB_LIBRARY_API rocksdb_optimistictransaction_options_t*
+rocksdb_optimistictransaction_options_create();
+
+extern ROCKSDB_LIBRARY_API void rocksdb_optimistictransaction_options_destroy(
+    rocksdb_optimistictransaction_options_t* opt);
+
+extern ROCKSDB_LIBRARY_API void
+rocksdb_optimistictransaction_options_set_set_snapshot(
+    rocksdb_optimistictransaction_options_t* opt, unsigned char v);
+
 // referring to convention (3), this should be used by client
 // to free memory that was malloc()ed
 extern ROCKSDB_LIBRARY_API void rocksdb_free(void* ptr);
 
+extern ROCKSDB_LIBRARY_API rocksdb_pinnableslice_t* rocksdb_get_pinned(
+    rocksdb_t* db, const rocksdb_readoptions_t* options, const char* key,
+    size_t keylen, char** errptr);
+extern ROCKSDB_LIBRARY_API rocksdb_pinnableslice_t* rocksdb_get_pinned_cf(
+    rocksdb_t* db, const rocksdb_readoptions_t* options,
+    rocksdb_column_family_handle_t* column_family, const char* key,
+    size_t keylen, char** errptr);
+extern ROCKSDB_LIBRARY_API void rocksdb_pinnableslice_destroy(
+    rocksdb_pinnableslice_t* v);
+extern ROCKSDB_LIBRARY_API const char* rocksdb_pinnableslice_value(
+    const rocksdb_pinnableslice_t* t, size_t* vlen);
+
+extern ROCKSDB_LIBRARY_API rocksdb_memory_consumers_t*
+    rocksdb_memory_consumers_create();
+extern ROCKSDB_LIBRARY_API void rocksdb_memory_consumers_add_db(
+    rocksdb_memory_consumers_t* consumers, rocksdb_t* db);
+extern ROCKSDB_LIBRARY_API void rocksdb_memory_consumers_add_cache(
+    rocksdb_memory_consumers_t* consumers, rocksdb_cache_t* cache);
+extern ROCKSDB_LIBRARY_API void rocksdb_memory_consumers_destroy(
+    rocksdb_memory_consumers_t* consumers);
+extern ROCKSDB_LIBRARY_API rocksdb_memory_usage_t*
+rocksdb_approximate_memory_usage_create(rocksdb_memory_consumers_t* consumers,
+                                       char** errptr);
+extern ROCKSDB_LIBRARY_API void rocksdb_approximate_memory_usage_destroy(
+    rocksdb_memory_usage_t* usage);
+
+extern ROCKSDB_LIBRARY_API uint64_t
+rocksdb_approximate_memory_usage_get_mem_table_total(
+    rocksdb_memory_usage_t* memory_usage);
+extern ROCKSDB_LIBRARY_API uint64_t
+rocksdb_approximate_memory_usage_get_mem_table_unflushed(
+    rocksdb_memory_usage_t* memory_usage);
+extern ROCKSDB_LIBRARY_API uint64_t
+rocksdb_approximate_memory_usage_get_mem_table_readers_total(
+    rocksdb_memory_usage_t* memory_usage);
+extern ROCKSDB_LIBRARY_API uint64_t
+rocksdb_approximate_memory_usage_get_cache_total(
+    rocksdb_memory_usage_t* memory_usage);
+
 #ifdef __cplusplus
 }  /* end extern "C" */
 #endif
-
-#endif  /* STORAGE_ROCKSDB_INCLUDE_C_H_ */

Opening RocksDB in read only mode

one process open rocksdb with openReadOnly, anther process open the same db with nomal modle。if the writing process update data of db，process of openReadOnly donot read the updated data？

Remove does not work when using ColumnFamilies

The following API does not work -
public void Remove(byte[] key, ColumnFamilyHandle cf = null, WriteOptions writeOptions = null)

It seems like it takes in the ColumnFamilyHandle but does not use it in the remove, causing the remove to fail.

ColumnFamilyHandle design is too confusing and requires special knowledge to avoid AccessViolations

I'm using iterator to create enumerator like that:

internal class MetadataEnumerator : IEnumerator<Tuple<long, long>>
            {
                private Iterator _iterator;
                private ColumnFamilyHandle _metadataColumn;

                public MetadataEnumerator(RocksDb dbContext)
                {
                    _metadataColumn = dbContext.GetColumnFamily("metadata");
                    _iterator = dbContext.NewIterator(_metadataColumn);
                    _iterator.SeekToFirst();
                }

                public void Dispose()
                {
                    _metadataColumn.Dispose();
                    _iterator.Dispose();
                }

                public bool MoveNext()
                {
                    _iterator.Next();
                    return _iterator.Valid();
                }

                public void Reset()
                {
                    _iterator.SeekToFirst();
                }

                public Tuple<long, long> Current
                {
                    get
                    {
                        Byte[] key = _iterator.Key();
                        Byte[] value = _iterator.Value();
                        return new Tuple<long, long>(BitConverter.ToInt64(key, 0), BitConverter.ToInt64(value, 0));
                    }
                }

                object IEnumerator.Current
                {
                    get { return Current; }
                }
            }

But i'm getting AccessViolation line _iterator = dbContext.NewIterator(_metadataColumn); is hit for the second time. You might have any idea how can i bypass this?

snapshots

It seems there is no support for snapshots in the current release. Is this due to some problems or simply just not done yet?

Contact request

Hi Warren,

Please forgive me this way of reaching out to you, but I could not find any contact details online.
We would greatly benefit from hiring you to do a few days of consulting to help with the way we use RocksDB / rocksdb sharp in our project (https://github.com/NethermindEth/nethermind).

Please let me know if that is something that you could potentially be interested in.
[email protected]

Best Regards,
Tomasz

API for listing all column families through C#

First of all, thank you for this wonderful binding! I am beginning to play around with it and so far it has been a breeze.

As I see it currently, there is a ListColumnFamilies call missing in the C# implementation compared to the native implementation.

DB::ListColumnFamilies(const DBOptions& db_options, const std::string& name, std::vector<std::string>* column_families)

I see that you already added this call into the native wrapper here, but did not expose it in the RocksDb C# class.

Could you kindly update this when you get a chance? I am using v5.1.4 published on Nuget. Thank you!

Can't delete checkpoint directory (Windows)

Can't delete checkpoint directory from C# (Net Core 2.1/2.2) code.
I can move/set attributes but can't delete or rewrite .sst files.
P.S.
Process have all permission. I can easily delete it from explorer.

Open database as read-only

Great project! I'm so glad I found it.

Any plans to add the ability to open a database as read-only?

A dynamic link library (DLL) initialization routine failed

When I install the 5.4.6 NuGet packages for RocksDbSharp and RocksDbNative and create a small sample application, I get a TypeInitializationException that seems to be caused by a Win32Exception, specifically "A dynamic link library (DLL) initialization routine failed".

I've tried replacing the native libraries obtained via NuGet with those in native-6e05979.zip from the v5.4.6 release, same result.

I've tried cloning the entire repository and running SimpleExampleLowLevel, same result.

I also note that there's a bug in download-native.cmd and download-native.sh. They should both point at https://github.com/warrenfalk/rocksdb-sharp/releases/download/v5.4.6/native-6e05979.zip but both point at different incorrect urls that result in 404s.

Nuget package does not install the rocksdb.dll (dotnet core)

C:\code\RulesLmdb\rocksdb-sharp\examples\PrefixExample>dotnet run

Unhandled Exception: System.TypeInitializationException: The type initializer for 'RocksDbSharp.Native' threw an exception. ---> NativeImport.NativeLoadException: Unable to locate rocksdb native library, either install it, or use RocksDbNative nuget package
Searched:C:\code\RulesLmdb\rocksdb-sharp\examples\PrefixExample\bin\Debug\netcoreapp2.0\native\amd64\rocksdb-5.4.6.dll: (Win32Exception) The specified module could not be found
C:\code\RulesLmdb\rocksdb-sharp\examples\PrefixExample\bin\Debug\netcoreapp2.0\native\amd64\rocksdb-5.4.dll: (Win32Exception) The specified module could not be found
C:\code\RulesLmdb\rocksdb-sharp\examples\PrefixExample\bin\Debug\netcoreapp2.0\native\amd64\rocksdb-5.dll: (Win32Exception) The specified module could not be found
C:\code\RulesLmdb\rocksdb-sharp\examples\PrefixExample\bin\Debug\netcoreapp2.0\native\amd64\rocksdb.dll: (Win32Exception) The specified module could not be found
C:\code\RulesLmdb\rocksdb-sharp\examples\PrefixExample\bin\Debug\netcoreapp2.0\native\rocksdb-5.4.6.dll: (Win32Exception) The specified module could not be found
C:\code\RulesLmdb\rocksdb-sharp\examples\PrefixExample\bin\Debug\netcoreapp2.0\native\rocksdb-5.4.dll: (Win32Exception) The specified module could not be found
C:\code\RulesLmdb\rocksdb-sharp\examples\PrefixExample\bin\Debug\netcoreapp2.0\native\rocksdb-5.dll: (Win32Exception) The specified module could not be found
C:\code\RulesLmdb\rocksdb-sharp\examples\PrefixExample\bin\Debug\netcoreapp2.0\native\rocksdb.dll: (Win32Exception) The specified module could not be found
C:\code\RulesLmdb\rocksdb-sharp\examples\PrefixExample\bin\Debug\netcoreapp2.0\amd64\rocksdb-5.4.6.dll: (Win32Exception) The specified module could not be found
C:\code\RulesLmdb\rocksdb-sharp\examples\PrefixExample\bin\Debug\netcoreapp2.0\amd64\rocksdb-5.4.dll: (Win32Exception) The specified module could not be found
C:\code\RulesLmdb\rocksdb-sharp\examples\PrefixExample\bin\Debug\netcoreapp2.0\amd64\rocksdb-5.dll: (Win32Exception) The specified module could not be found
C:\code\RulesLmdb\rocksdb-sharp\examples\PrefixExample\bin\Debug\netcoreapp2.0\amd64\rocksdb.dll: (Win32Exception) The specified module could not be found
C:\code\RulesLmdb\rocksdb-sharp\examples\PrefixExample\bin\Debug\netcoreapp2.0\rocksdb-5.4.6.dll: (Win32Exception) The specified module could not be found
C:\code\RulesLmdb\rocksdb-sharp\examples\PrefixExample\bin\Debug\netcoreapp2.0\rocksdb-5.4.dll: (Win32Exception) The specified module could not be found
C:\code\RulesLmdb\rocksdb-sharp\examples\PrefixExample\bin\Debug\netcoreapp2.0\rocksdb-5.dll: (Win32Exception) The specified module could not be found
C:\code\RulesLmdb\rocksdb-sharp\examples\PrefixExample\bin\Debug\netcoreapp2.0\rocksdb.dll: (Win32Exception) The specified module could not be found
rocksdb-5.4.6.dll: (Win32Exception) The specified module could not be found
rocksdb-5.4.dll: (Win32Exception) The specified module could not be found
rocksdb-5.dll: (Win32Exception) The specified module could not be found
rocksdb.dll: (Win32Exception) The specified module could not be found
C:\Users\asarkar.nuget\packages\rocksdbsharp\5.4.6\lib\netstandard1.6\native\amd64\rocksdb-5.4.6.dll: (Win32Exception) The specified module could not be found
C:\Users\asarkar.nuget\packages\rocksdbsharp\5.4.6\lib\netstandard1.6\native\amd64\rocksdb-5.4.dll: (Win32Exception) The specified module could not be found
C:\Users\asarkar.nuget\packages\rocksdbsharp\5.4.6\lib\netstandard1.6\native\amd64\rocksdb-5.dll: (Win32Exception) The specified module could not be found
C:\Users\asarkar.nuget\packages\rocksdbsharp\5.4.6\lib\netstandard1.6\native\amd64\rocksdb.dll: (Win32Exception) The specified module could not be found
C:\Users\asarkar.nuget\packages\rocksdbsharp\5.4.6\lib\netstandard1.6\native\rocksdb-5.4.6.dll: (Win32Exception) The specified module could not be found
C:\Users\asarkar.nuget\packages\rocksdbsharp\5.4.6\lib\netstandard1.6\native\rocksdb-5.4.dll: (Win32Exception) The specified module could not be found
C:\Users\asarkar.nuget\packages\rocksdbsharp\5.4.6\lib\netstandard1.6\native\rocksdb-5.dll: (Win32Exception) The specified module could not be found
C:\Users\asarkar.nuget\packages\rocksdbsharp\5.4.6\lib\netstandard1.6\native\rocksdb.dll: (Win32Exception) The specified module could not be found
C:\Users\asarkar.nuget\packages\rocksdbsharp\5.4.6\lib\netstandard1.6\amd64\rocksdb-5.4.6.dll: (Win32Exception) The specified module could not be found
C:\Users\asarkar.nuget\packages\rocksdbsharp\5.4.6\lib\netstandard1.6\amd64\rocksdb-5.4.dll: (Win32Exception) The specified module could not be found
C:\Users\asarkar.nuget\packages\rocksdbsharp\5.4.6\lib\netstandard1.6\amd64\rocksdb-5.dll: (Win32Exception) The specified module could not be found
C:\Users\asarkar.nuget\packages\rocksdbsharp\5.4.6\lib\netstandard1.6\amd64\rocksdb.dll: (Win32Exception) The specified module could not be found
C:\Users\asarkar.nuget\packages\rocksdbsharp\5.4.6\lib\netstandard1.6\rocksdb-5.4.6.dll: (Win32Exception) The specified module could not be found
C:\Users\asarkar.nuget\packages\rocksdbsharp\5.4.6\lib\netstandard1.6\rocksdb-5.4.dll: (Win32Exception) The specified module could not be found
C:\Users\asarkar.nuget\packages\rocksdbsharp\5.4.6\lib\netstandard1.6\rocksdb-5.dll: (Win32Exception) The specified module could not be found
C:\Users\asarkar.nuget\packages\rocksdbsharp\5.4.6\lib\netstandard1.6\rocksdb.dll: (Win32Exception) The specified module could not be found
rocksdb-5.4.6.dll: (Win32Exception) The specified module could not be found
rocksdb-5.4.dll: (Win32Exception) The specified module could not be found
rocksdb-5.dll: (Win32Exception) The specified module could not be found
rocksdb.dll: (Win32Exception) The specified module could not be found
at NativeImport.Importers.Import[T](INativeLibImporter importer, String libName, String version, Boolean suppressUnload)
at NativeImport.Auto.Import[T](String name, String version, Boolean suppressUnload) in C:\Users\wfalk\source\rocksdb-sharp\RocksDbSharp\AutoNativeImport.cs:line 39
at RocksDbSharp.Native..cctor() in C:\Users\wfalk\source\rocksdb-sharp\RocksDbSharp\Native.Load.cs:line 18
--- End of inner exception stack trace ---
at RocksDbSharp.BlockBasedTableOptions..ctor() in C:\Users\wfalk\source\rocksdb-sharp\RocksDbSharp\BlockBasedTableOptions.cs:line 18
at PrefixExample.Program.Main(String[] args) in C:\code\RulesLmdb\rocksdb-sharp\examples\PrefixExample\Program.cs:line 15

Live Unit Testing support

I'm using MS Tests with Visual Studio 2019 16.0.0 and 5.17.2 version of this library.

Running test in "standard" mode - right click -> Run Selected Tests is working as expected. However, when I enable Live Unit Testing, Visual Studio is unable to locate RocksDbSharp.Native library:

> System.TypeInitializationException: The type initializer for 'RocksDbSharp.Native' threw an exception. ---> NativeImport.NativeLoadException: Unable to locate rocksdb native library, either install it, or use RocksDbNative nuget package
> Searched:
> C:\PathToProject\.vs\ProjectName\lut\0\t\ProjectName.Tests\bin\Debug\native\amd64\rocksdb-5.17.2.dll: (Win32Exception) The specified module could not be found
> C:\PathToProject\.vs\ProjectName\lut\0\t\ProjectName.Tests\bin\Debug\native\amd64\rocksdb-5.17.dll: (Win32Exception) The specified module could not be found
> C:\PathToProject\.vs\ProjectName\lut\0\t\ProjectName.Tests\bin\Debug\native\amd64\rocksdb-5.dll: (Win32Exception) The specified module could not be found
> C:\PathToProject\.vs\ProjectName\lut\0\t\ProjectName.Tests\bin\Debug\native\amd64\rocksdb.dll: (Win32Exception) The specified module could not be found
> C:\PathToProject\.vs\ProjectName\lut\0\t\ProjectName.Tests\bin\Debug\native\rocksdb-5.17.2.dll: (Win32Exception) The specified module could not be found
> C:\PathToProject\.vs\ProjectName\lut\0\t\ProjectName.Tests\bin\Debug\native\rocksdb-5.17.dll: (Win32Exception) The specified module could not be found
> C:\PathToProject\.vs\ProjectName\lut\0\t\ProjectName.Tests\bin\Debug\native\rocksdb-5.dll: (Win32Exception) The specified module could not be found
> C:\PathToProject\.vs\ProjectName\lut\0\t\ProjectName.Tests\bin\Debug\native\rocksdb.dll: (Win32Exception) The specified module could not be found
> C:\PathToProject\.vs\ProjectName\lut\0\t\ProjectName.Tests\bin\Debug\amd64\rocksdb-5.17.2.dll: (Win32Exception) The specified module could not be found
> C:\PathToProject\.vs\ProjectName\lut\0\t\ProjectName.Tests\bin\Debug\amd64\rocksdb-5.17.dll: (Win32Exception) The specified module could not be found
> C:\PathToProject\.vs\ProjectName\lut\0\t\ProjectName.Tests\bin\Debug\amd64\rocksdb-5.dll: (Win32Exception) The specified module could not be found
> C:\PathToProject\.vs\ProjectName\lut\0\t\ProjectName.Tests\bin\Debug\amd64\rocksdb.dll: (Win32Exception) The specified module could not be found
> C:\PathToProject\.vs\ProjectName\lut\0\t\ProjectName.Tests\bin\Debug\rocksdb-5.17.2.dll: (Win32Exception) The specified module could not be found
> C:\PathToProject\.vs\ProjectName\lut\0\t\ProjectName.Tests\bin\Debug\rocksdb-5.17.dll: (Win32Exception) The specified module could not be found
> C:\PathToProject\.vs\ProjectName\lut\0\t\ProjectName.Tests\bin\Debug\rocksdb-5.dll: (Win32Exception) The specified module could not be found
> C:\PathToProject\.vs\ProjectName\lut\0\t\ProjectName.Tests\bin\Debug\rocksdb.dll: (Win32Exception) The specified module could not be found
> rocksdb-5.17.2.dll: (Win32Exception) The specified module could not be found
> rocksdb-5.17.dll: (Win32Exception) The specified module could not be found
> rocksdb-5.dll: (Win32Exception) The specified module could not be found
> rocksdb.dll: (Win32Exception) The specified module could not be found

When I examine this folder, I couldn't find 'native" directory as well.

RocksDb.ListColumnFamilies returns garbage characters on Linux

ListColumnFamilies seems to return incorrect strings on Linux. For example, I updated the ColumnFamilyExample and updated it to ListColumnFamilies and printed them, and here is what I get - "PJ ,L".
So I am not able to use the list of returned Column families and do a GetColumnFamily on them.
This seems to work fine on Windows.

Options files support

RocksDB 4.3 added options that would be very useful for robustness of rocksdb-sharp based applications which may have changes to their RocksDB options over time. See here.

Stop static linking against VC Redist

Static linking against VC allows for a zero-config deployment, but has drawbacks.

Copying this from note by @mkosieradzki on issue #24.

What are the reasons not to link statically against vcredist:

Often RocksDB isn't the only one native library inside entire application. If all libraries were linked statically it would basically mean a lot duplication and memory waste inside a single application.

VCredists are serviceable - it means they are subject to security updates.

It decreases the ability to reuse memory image of vcredists between different applications as well.

IMO it would be a good idea to stop static linking. If one wants zero-config deployment they can add nuget package with vcredist DLLs (again if we have switched to P/Invoke it will work out-of-the-box).

I know it's controversial and I've seen you have put some effort into static linking however I think it's worth the consideration. Even project that huge like dotnet Core requires VCRedist. (Maybe it's a good idea to link against the same VCRedist version required by the .NET Core already? :)

Static linking was added by @wangjia184, so I'd like to make sure he's able to weigh in.

I'm most concerned about the second drawback given above. Increasing the surface for attack vectors is not something I had previously considered.

Is block cache supported?

Is setting block cache supported?

https://github.com/facebook/rocksdb/wiki/Block-Cache

I can't find anywhere to make a NewLRUCache call.

AccessViolationException then using StringComparator with ColumnFamily

This code example throw exception with RocksDbSharp.StringComparator but without it works correctly.

            var options = new RocksDbSharp.DbOptions()
              .SetCreateIfMissing(true)
              .SetCreateMissingColumnFamilies(true);

            var bbto = new RocksDbSharp.BlockBasedTableOptions()
               .SetFilterPolicy(RocksDbSharp.BloomFilterPolicy.Create(20, false))
               .SetWholeKeyFiltering(false)
               .SetCacheIndexAndFilterBlocks(true);

            var columnFamilies = new RocksDbSharp.ColumnFamilies
            {
                 { "cf1", new RocksDbSharp.ColumnFamilyOptions()
                    .SetCompression(RocksDbSharp.CompressionTypeEnum.rocksdb_no_compression)
                    .SetComparator( new RocksDbSharp.StringComparator(StringComparer.InvariantCultureIgnoreCase)) //if commented - works fine
                    .SetBlockBasedTableFactory(bbto)
                },
            };
            using (var db = RocksDbSharp.RocksDb.Open(options, dbPath, columnFamilies))
            {
                Random rand = new Random(42);

                var cf1 = db.GetColumnFamily("cf1");

                for (int i = 0; i < 100000; ++i)
                {
                    string k = rand.Next(100000000).ToString();
                    byte[] key = Encoding.UTF8.GetBytes(k);
                    byte[] value = db.Get(key, cf1);
                    if (value == null)
                    {
                        db.Put(key, BitConverter.GetBytes(i), cf1);
                    }
                    else
                        Console.WriteLine(BitConverter.ToInt32(value, 0));
                }
            }

This error is not always reproduced!!!
PS Windows x64

Why patching thirdparty.inc and CMakeLists.txt instead of environment variables?

I am using your scripts to build newer version of rocksdb and I have noticed some strange things inside. For example you are patching thirdparty.inc and CMakeLists.txt.

The thirdparty.inc file has been very nicely designed so you can either hardcode paths or provide them using environment variables... My suggestion would be to use env variables and provide full paths - then you will have no issues building tools and you will not need to maintain patchfiles...

For example in my powershell script (I have ported bash to powershell, if you also prefer powershell I can share my script) I use following lines:

$env:SNAPPY_INCLUDE = Resolve-Path ../snappy/build
$env:SNAPPY_LIB_DEBUG = Join-Path $env:SNAPPY_INCLUDE Debug/libsnappy
$env:SNAPPY_LIB_RELEASE = Join-Path $env:SNAPPY_INCLUDE Release/libsnappy

You can provide paths to GFLAGS the same way.

any chance of having 32-bit version for Windows?

It's fantastic that you've made .NET binding to RocksDB. However, is it possible to make a 32-bit lib and binding to it to run on Windows? We are looking for a 32-bit version for using RocksDB as embedded db in our WPF client app.

Seems like .sh script makes it possible to build 32-bit experimental version only when running under linux

Add support for Merge

@warrenfalk
I have created a prototype for merge operation handling (basically just expanded your:
rocksdb_mergeoperator_create and rocksdb_mergeoperator_destroy
and added proper declarations for delegates)

I've got couple of questions before submiting a PR:

Should I remove ROCKSDB_MERGE_OPERATOR constant if I create a full low-level implementation
Should I start with low-level + tests or you expect a full high-level implementation?

I also understand you expect at least a functional smoke test for this feature.

Alpine support

Please, add Alpine (3.7, 3.8, edge) support.

In current version 5.17.2 it does not work. Depends of libbz2 and finally I've got ".../bin/Release/netcoreapp2.2/native/amd64/librocksdb.so: (NativeLoadException) dlopen: Error relocating .../bin/Release/netcoreapp2.2/native/amd64/librocksdb.so: backtrace: symbol not found"

If I tried Alpine package rocksdb 5.15.10-r4, I've got "librocksdb.so: (NativeLoadException) dlopen: Error relocating /usr/lib/librocksdb.so: _ZdlPvSt11align_val_t: symbol not found"

Undetectable truncation in Get(byte[], byte[]) when key does not exist

rocksdb-sharp/RocksDbSharp/RocksDb.cs

Lines 111 to 121 in 8818ab4

 public long Get(byte[] key, long keyLength, byte[] buffer, long offset, long length, ColumnFamilyHandle cf = null, ReadOptions readOptions = null) 

 { 

 unsafe 

 { 

 var ptr = Native.Instance.rocksdb_get(Handle, (readOptions ?? DefaultReadOptions).Handle, key, keyLength, out long valLength, cf); 

 valLength = Math.Min(length, valLength); 

 Marshal.Copy(ptr, buffer, (int)offset, (int)valLength); 

 Native.Instance.rocksdb_free(ptr); 

 return valLength; 

 } 

 }

The code will truncate the value if the passed in buffer is not big enough. Furthermore, since it returns valLength which is reassigned to Min(length, valLength), this truncation cannot be detected.

In my opinion, the best way to address this would be to return the actual db length. This allows users to get a truncated result while they can easily check if a truncation occurred by checking if the method result (valLength) is bigger than the length they passed in.

Can't run tests in netcoreapp1.1 project

I've created a test project and have added RocksDbNative package there. But I can't run tests. I've tried many different options

Resharper Test Runner
VS 2017 test runner
dotnet xunit
dotnet test

Exception was thrown

    TestExample.TestRocks.Test [FAIL]
      System.TypeInitializationException : The type initializer for 'RocksDbSharp.Native' threw an exception.
      ---- NativeImport.NativeLoadException : Unable to locate rocksdb native library, either install it, or use RocksDbNative nuget package
Searched:C:\Users\ravenger\.nuget\packages\dotnet-xunit\2.3.0-beta2-build3683\tools\netcoreapp1.0\native\amd64\rocksdb-5.4.6.dll: (Win32Exception) The specified module could not be found
C:\Users\ravenger\.nuget\packages\dotnet-xunit\2.3.0-beta2-build3683\tools\netcoreapp1.0\native\amd64\rocksdb-5.4.dll: (Win32Exception) The specified module could not be found
C:\Users\ravenger\.nuget\packages\dotnet-xunit\2.3.0-beta2-build3683\tools\netcoreapp1.0\native\amd64\rocksdb-5.dll: (Win32Exception) The specified module could not be found
C:\Users\ravenger\.nuget\packages\dotnet-xunit\2.3.0-beta2-build3683\tools\netcoreapp1.0\native\amd64\rocksdb.dll: (Win32Exception) The specified module could not be found
C:\Users\ravenger\.nuget\packages\dotnet-xunit\2.3.0-beta2-build3683\tools\netcoreapp1.0\native\rocksdb-5.4.6.dll: (Win32Exception) The specified module could not be found
C:\Users\ravenger\.nuget\packages\dotnet-xunit\2.3.0-beta2-build3683\tools\netcoreapp1.0\native\rocksdb-5.4.dll: (Win32Exception) The specified module could not be found
C:\Users\ravenger\.nuget\packages\dotnet-xunit\2.3.0-beta2-build3683\tools\netcoreapp1.0\native\rocksdb-5.dll: (Win32Exception) The specified module could not be found
C:\Users\ravenger\.nuget\packages\dotnet-xunit\2.3.0-beta2-build3683\tools\netcoreapp1.0\native\rocksdb.dll: (Win32Exception) The specified module could not be found
C:\Users\ravenger\.nuget\packages\dotnet-xunit\2.3.0-beta2-build3683\tools\netcoreapp1.0\amd64\rocksdb-5.4.6.dll: (Win32Exception) The specified module could not be found
C:\Users\ravenger\.nuget\packages\dotnet-xunit\2.3.0-beta2-build3683\tools\netcoreapp1.0\amd64\rocksdb-5.4.dll: (Win32Exception) The specified module could not be found
C:\Users\ravenger\.nuget\packages\dotnet-xunit\2.3.0-beta2-build3683\tools\netcoreapp1.0\amd64\rocksdb-5.dll: (Win32Exception) The specified module could not be found
C:\Users\ravenger\.nuget\packages\dotnet-xunit\2.3.0-beta2-build3683\tools\netcoreapp1.0\amd64\rocksdb.dll: (Win32Exception) The specified module could not be found
C:\Users\ravenger\.nuget\packages\dotnet-xunit\2.3.0-beta2-build3683\tools\netcoreapp1.0\rocksdb-5.4.6.dll: (Win32Exception) The specified module could not be found
C:\Users\ravenger\.nuget\packages\dotnet-xunit\2.3.0-beta2-build3683\tools\netcoreapp1.0\rocksdb-5.4.dll: (Win32Exception) The specified module could not be found
C:\Users\ravenger\.nuget\packages\dotnet-xunit\2.3.0-beta2-build3683\tools\netcoreapp1.0\rocksdb-5.dll: (Win32Exception) The specified module could not be found
C:\Users\ravenger\.nuget\packages\dotnet-xunit\2.3.0-beta2-build3683\tools\netcoreapp1.0\rocksdb.dll: (Win32Exception) The specified module could not be found
rocksdb-5.4.6.dll: (Win32Exception) The specified module could not be found
rocksdb-5.4.dll: (Win32Exception) The specified module could not be found
rocksdb-5.dll: (Win32Exception) The specified module could not be found
rocksdb.dll: (Win32Exception) The specified module could not be found
C:\Users\ravenger\.nuget\packages\rocksdbsharp\5.4.6\lib\netstandard1.6\native\amd64\rocksdb-5.4.6.dll: (Win32Exception) The specified module could not be found
C:\Users\ravenger\.nuget\packages\rocksdbsharp\5.4.6\lib\netstandard1.6\native\amd64\rocksdb-5.4.dll: (Win32Exception) The specified module could not be found
C:\Users\ravenger\.nuget\packages\rocksdbsharp\5.4.6\lib\netstandard1.6\native\amd64\rocksdb-5.dll: (Win32Exception) The specified module could not be found
C:\Users\ravenger\.nuget\packages\rocksdbsharp\5.4.6\lib\netstandard1.6\native\amd64\rocksdb.dll: (Win32Exception) The specified module could not be found
C:\Users\ravenger\.nuget\packages\rocksdbsharp\5.4.6\lib\netstandard1.6\native\rocksdb-5.4.6.dll: (Win32Exception) The specified module could not be found
C:\Users\ravenger\.nuget\packages\rocksdbsharp\5.4.6\lib\netstandard1.6\native\rocksdb-5.4.dll: (Win32Exception) The specified module could not be found
C:\Users\ravenger\.nuget\packages\rocksdbsharp\5.4.6\lib\netstandard1.6\native\rocksdb-5.dll: (Win32Exception) The specified module could not be found
C:\Users\ravenger\.nuget\packages\rocksdbsharp\5.4.6\lib\netstandard1.6\native\rocksdb.dll: (Win32Exception) The specified module could not be found
C:\Users\ravenger\.nuget\packages\rocksdbsharp\5.4.6\lib\netstandard1.6\amd64\rocksdb-5.4.6.dll: (Win32Exception) The specified module could not be found
C:\Users\ravenger\.nuget\packages\rocksdbsharp\5.4.6\lib\netstandard1.6\amd64\rocksdb-5.4.dll: (Win32Exception) The specified module could not be found
C:\Users\ravenger\.nuget\packages\rocksdbsharp\5.4.6\lib\netstandard1.6\amd64\rocksdb-5.dll: (Win32Exception) The specified module could not be found
C:\Users\ravenger\.nuget\packages\rocksdbsharp\5.4.6\lib\netstandard1.6\amd64\rocksdb.dll: (Win32Exception) The specified module could not be found
C:\Users\ravenger\.nuget\packages\rocksdbsharp\5.4.6\lib\netstandard1.6\rocksdb-5.4.6.dll: (Win32Exception) The specified module could not be found
C:\Users\ravenger\.nuget\packages\rocksdbsharp\5.4.6\lib\netstandard1.6\rocksdb-5.4.dll: (Win32Exception) The specified module could not be found
C:\Users\ravenger\.nuget\packages\rocksdbsharp\5.4.6\lib\netstandard1.6\rocksdb-5.dll: (Win32Exception) The specified module could not be found
C:\Users\ravenger\.nuget\packages\rocksdbsharp\5.4.6\lib\netstandard1.6\rocksdb.dll: (Win32Exception) The specified module could not be found
rocksdb-5.4.6.dll: (Win32Exception) The specified module could not be found
rocksdb-5.4.dll: (Win32Exception) The specified module could not be found
rocksdb-5.dll: (Win32Exception) The specified module could not be found
rocksdb.dll: (Win32Exception) The specified module could not be found
      Stack Trace:
        C:\Users\wfalk\source\rocksdb-sharp\RocksDbSharp\Options.cs(26,0): at RocksDbSharp.OptionsHandle..ctor()
           at RocksDbSharp.DbOptions..ctor()
        C:\Projects\test\RocksDbTestSample\TestRocks.cs(11,0): at TestExample.TestRocks.Test()
        ----- Inner Stack Trace -----
           at NativeImport.Importers.Import[T](INativeLibImporter importer, String libName, String version, Boolean suppressUnload)
        C:\Users\wfalk\source\rocksdb-sharp\RocksDbSharp\AutoNativeImport.cs(39,0): at NativeImport.Auto.Import[T](String name, String version, Boolean suppressUnload)
        C:\Users\wfalk\source\rocksdb-sharp\RocksDbSharp\Native.Load.cs(18,0): at RocksDbSharp.Native..cctor()
  Finished:    RocksDbTestSample

I've tried 5.2.1 and 5.4.6 versions
There is my project: https://github.com/alexvaluyskiy/RocksDbTestSample

Support for Docker containers

Have you considered supporting docker containers officially? I came across a few issues while running rocksdb-sharp in a container.

Libdl not found. This is not included by default in the microsoft/dotnet base container. So I had to use the following commands to install

RUN apt-get update && apt-get install -y libcap2-bin libsnappy1v5 &&
ln -s /lib/x86_64-linux-gnu/libdl.so.2 /usr/lib/x86_64-linux-gnu/libdl.so &&
ln -s /lib/x86_64-linux-gnu/libc.so.6 /usr/lib/x86_64-linux-gnu/libc.so &&
rm -rf /var/lib/apt/lists/*

Even after this I get the following exception when calling RocksDb.Open

RocksDbSharp.RocksDbException
HResult=0x80131500
Message=IO error: While fsync: a directory: Invalid argument
Source=RocksDbSharp
StackTrace:
at RocksDbSharp.Native.rocksdb_open(IntPtr options, String name)
at RocksDbSharp.RocksDb.Open(OptionsHandle options, String path)
at Hexastore.Rocks.RocksTest..ctor()

SST file Writer Class

I want a distributed computing and make one Generalized dictionary key value store so every body can access that db for that i can use the Concept of SST File Writer Class
Every PC in distributed make own SST files and then Further all SST files are ingest into One DB
Here is the links that i can accomplish my task
https://github.com/facebook/rocksdb/wiki/Creating-and-Ingesting-SST-files

is that class or API available in Rocksdb-Sharp
or how can i accomplish this concept using Rocksdb C#

Unable to load DLL 'libc'

Hello
I currently have the following error message:
Unable to load DLL 'libc': The specified module could not be found. (Exception from HRESULT: 0x8007007E)
System.DllNotFoundException: at NativeImport.Importers+PosixImporter.uname (RocksDbSharp, Version=5.4.6.10, Culture=neutral, PublicKeyToken=null) at NativeImport.Importers+PosixImporter.GetPlatform (RocksDbSharp, Version=5.4.6.10, Culture=neutral, PublicKeyToken=null) at NativeImport.Importers+PosixImporter..ctor (RocksDbSharp, Version=5.4.6.10, Culture=neutral, PublicKeyToken=null) at NativeImport.Importers..cctor (RocksDbSharp, Version=5.4.6.10, Culture=neutral, PublicKeyToken=null) at NativeImport.Auto.Import (RocksDbSharp, Version=5.4.6.10, Culture=neutral, PublicKeyToken=null) at RocksDbSharp.Native..cctor (RocksDbSharp, Version=5.4.6.10, Culture=neutral, PublicKeyToken=null) at RocksDbSharp.OptionsHandle..ctor (RocksDbSharp, Version=5.4.6.10, Culture=neutral, PublicKeyToken=null)
I run it on a windows machine.
At first I thought that it is a problem with the OS. I logged the values and the result was:
Environment.OSVersion.Platform: Win32NT. So it should take the Importers.Windows. I think the problem is somewere in the method public static T Import Method. Does anyone have an idea what the problem could be?
Thanks a lot!

GetColumnFamily() call in C#

I have a couple of somewhat trivial questions so please excuse my ignorance. I see a call in C# GetColumnFamily() that I cannot see in the CPP implementation.

I have a couple of clarification questions:

What is the expected behavior if the column family cannot be found? Instead of throwing an exception (since you do return columnFamilies[name];), does it make sense to return null or do you expect the application to handle the exception?
Since this call is not in the CPP implementation, what is your general take on API parity between CPP and C#?

Thank you!

Add prefix_same_as_start to ReadOptions

I wonder if you could add prefix_same_as_start:

https://github.com/facebook/rocksdb/blob/64324e329eb0a9b4e77241a425a1615ff524c7f1/include/rocksdb/options.h#L1086

to ReadOptions in this library. It was added to RocksDb api after your last update.

Instantiating SstFileWriter causes a memory allocation exception

SstFileWriter sfw = new SstFileWriter();

crashes the system, regardless of whether it's made before a call to DB.Open or after.

Included low level rocksdb_mergeoperator_create

First, thanks for the great project!

I've been using your NuGet packages on Linux, and they have been working wonderfully.

I've been using WriteBatch to do Read/Modify/Write and wanted to switch to using WriteBatch.Merge but from the documentation, it seems I need to be able to call rocksdb_mergeoperator_create, which does not appear to be included in the NuGet package.

Thoughts?

Garbage collection and Comparator callbacks

Hi,

I am experiencing some issues with the use of custom key comparators. An exception is thrown by rocksdb_write:

Managed Debugging Assistant 'CallbackOnCollectedDelegate' : 'A callback was made on a garbage collected delegate of type 'RocksDbSharp!RocksDbSharp.CompareFunc::Invoke'. This may cause application crashes, corruption and data loss. When passing delegates to unmanaged code, they must be kept alive by the managed application until it is guaranteed that they will never be called.'

To rule out any issue related to my implementation of Comparator, I have used the included StringComparator and the exception still remains. Is this to be expected?

Thanks to everyone involved with the project!

size_t is marshalled as ulong not UIntPtr

Please correct me if I am wrong, but AFAIK size_t should be marshalled as UIntPtr not ulong.

How to buildin rocksdb.dll in windows

@warrenfalk
You can give more infomation or steps how to build in windows. Something like snappy
1, Open "Developer command prompt for vs2015"
2, Run git's bash.exe
3, cd to the native-build folder within the repository
4, execute ./build-rocksdb.sh

Any chance for NuGet in near future?

Like in title, is there any chance you'll release this as a nuget package soon?

PS. Library is great, keep up the good work ;)

Huge leaks

I'm not sure if it's C# wrappers fault or RocksDB issue itself.

I started a huge bulk seed for our production database. After running for about 5 hours RocksDB was taking 7,5 GB of RAM. ANTS 8 shows that this memory is unmanaged memory taken by librocksdb.dll. Same thing when i leave application working in backgroud. We are doing a lot of random access on database with about 550M keys. Application slowly but progressively is taking all ram on computer. I tried releasing database - helps little bit, but still memory gets "eaten".

Seems like sth isn't closing up, i can't really find out what ;/

My (simplified) code of the bulk seed is here:

            ColumnFamilies _columnFamilies;
            DbOptions _dbOptions;

            _dbOptions = new DbOptions()
                .SetCreateIfMissing(true)
                .SetCreateMissingColumnFamilies(true)
                .PrepareForBulkLoad()
                .IncreaseParallelism(4);
            _columnFamilies = new ColumnFamilies
            {
                {"index", new ColumnFamilyOptions().OptimizeForPointLookup(512)},
                {"store", new ColumnFamilyOptions().OptimizeForPointLookup(512)},
                {"metadata", new ColumnFamilyOptions()}
            };

            RocksDb _dbContext = null;
            RocksDBSerializer _rocksDBSerializer = null;
            ColumnFamilyHandle _indexColumn = null;
            ColumnFamilyHandle _storeColumn = null;
            ColumnFamilyHandle _metadataColumn = null;
            int counter = 0;
            Int64 trackId = -1;

            foreach (var path in files)
            {
                // every 100 passes release database handle
                // not much result unfortunetely, memory is leaking somewhere else 
                if (counter == 99) counter = 0;
                if (counter == 0) {
                    if (_dbContext != null)
                    {
                        _rocksDBSerializer = null;
                        _indexColumn.Dispose();
                        _storeColumn.Dispose();
                        _metadataColumn.Dispose();
                        _dbContext.Dispose();
                    }
                    _dbContext = RocksDb.Open(_dbOptions, "storeDatabase", _columnFamilies);
                    _indexColumn = _dbContext.GetColumnFamily("index");
                    _storeColumn = _dbContext.GetColumnFamily("store");
                    _metadataColumn = _dbContext.GetColumnFamily("metadata");

                    _rocksDBSerializer = new RocksDBSerializer(_dbContext, _indexColumn, _storeColumn, _metadataColumn);

                    trackId = _rocksDBSerializer.GetLastTrackId();
                }
                counter++;

                var dirs = path.Split('\\');
                var fileName = dirs[dirs.Length - 1];
                Int64 mediaObjectId;
                if(!Int64.TryParse(fileName , out mediaObjectId))
                {
                    Console.WriteLine("Error bad media object id");
                    Console.WriteLine("File name: {0} id: {1}", fileName, idString);
                    return;
                }

                // shortCodes is a array of short ints
                // we have 70k files, each one generates about 8000 shortCodes

                foreach (var shortCode in shortCodes)
                {
                    trackId++;
                    Byte[] key = BitConverter.GetBytes(trackId);

                    _rocksDBSerializer.AddToStore(key, shortCode);

                    _rocksDBSerializer.AddToMetadata(mediaObjectId, trackId, key);

                    _rocksDBSerializer.AddToIndex(trackId, shortCode);
                }
            }
            _dbContext.Dispose();

RocksDBSerializer is here:

        public void AddToIndex(long trackId, List<CodeSegment> shortCode)
        {
            foreach (var codeSegment in shortCode)
            {
                Byte[] indexKey = BitConverter.GetBytes(codeSegment.Code);
                Byte[] indexValue = _dbContext.Get(indexKey, _indexColumn);
                if (indexValue != null)
                {
                    List<Int64> tracks = Enumerable.Range(0, indexValue.Length / 8)
                        .Select(i => BitConverter.ToInt64(indexValue, i * 8))
                        .ToList();
                    if (!tracks.Contains(trackId))
                    {
                        tracks.Add(trackId);
                        indexValue = tracks.SelectMany(BitConverter.GetBytes).ToArray();
                        _dbContext.Put(indexKey, indexValue, _indexColumn);
                    }
                }
                else
                {
                    List<Int64> tracks = new List<Int64> { trackId };
                    indexValue = tracks.SelectMany(BitConverter.GetBytes).ToArray();
                    _dbContext.Put(indexKey, indexValue, _indexColumn);
                }
            }
        }

        public void AddToMetadata(long mediaObjectId, long trackId, byte[] key)
        {
            Byte[] mediaObjectIdAsByteArray = BitConverter.GetBytes(mediaObjectId);
            Byte[] metadataValue = _dbContext.Get(key, _metadataColumn);
            if (metadataValue != null && metadataValue.Equals(mediaObjectIdAsByteArray))
            {
                throw new Exception(
                    String.Format("This mapping already exists and has diferent value. TrackId: {0}; MediaObjectId: {1}", trackId, mediaObjectId)
                    );
            }

            _dbContext.Put(key, mediaObjectIdAsByteArray, _metadataColumn);
        }

        public void AddToStore(Byte[] key, List<CodeSegment> codeSegments)
        {
            Byte[] value;

            using (var ms = new MemoryStream())
            {
                Serializer.Serialize(ms, codeSegments);
                value = ms.ToArray();
            }

            _dbContext.Put(key, value, _storeColumn);
        }

        public Int64 GetLastTrackId()
        {
            Iterator iter = _dbContext.NewIterator(_storeColumn);
            iter.SeekToLast();
            Int64 trackId = -1;
            if (iter.Valid())
            {
                Byte[] firstKey = iter.Key();
                trackId = BitConverter.ToInt64(firstKey, 0);
            }
            iter.Dispose();
            return trackId;
        }

        #endregion

I tried this code with WriteBatch, performance is better but leak is still there :(

Sign nuget package

It could be possible to sign the nuget package?

[Suggest] Set default encoding

I notice that RocksDB-Sharp uses UTF-8 as default encoding. This is better than ASCII as default. But I sill wonder whether there is a way to set the default encoding besides UTF-8?

I also suggest the Encoding should applied to Key & Value separately, both for the default encoding & per command encoding. It would be better to be able to assign an Encoding other to the default encoding for Key & Value separately.

Multiple Threads break SetIterateUpperBound

SetIterateUpperBound does not seem to be enforced correctly when there are multiple iterators being used concurrently on the same database.

To test, I created a small program that seems to demonstrate the issue. This test creates a database containing essentially 5 values: 5, 6, 7, 8, 9. It then runs many tasks, which each separately iterate between 6 and 8, which should mean iterating over the values 6 and 7 count many times.

I have made the values multiple bytes, as the bug didn't seem to surface using single byte keys. I haven't been able to observe the bug with just a single task / thread either.

If the test operates correctly, you should see Running followed by Done. If it prints any numbers out, it has failed.

You may need to run the test numerous times to see a failure. You can also try changing the number of tasks and number of iterations and try changing between debug and release builds. Merely increasing the number of tasks / iterations significantly does not help, as the errors tend to occur near the start of execution.

I've put the test into a loop so you can run it multiple times without restarting it. Just press ENTER to start a test round. Once it is complete, you can press ENTER to start the next round of testing.

Don't run this code if you have important files in the directory: C:/TempRock/. It will be deleted.

using System;
using System.IO;
using System.Linq;
using System.Threading.Tasks;
using RocksDbSharp;

namespace Experiments
{
    public class Program
    {
        static void Main(string[] args)
        {
            while(true)
            {
                Directory.Delete("C:/TempRock/", true);

                Console.Out.WriteLine("Ready (press ENTER to start test)");

                Console.In.ReadLine();

                var options = new DbOptions()
                    .SetCreateIfMissing(true);

                using(var db = RocksDb.Open(options, "C:/TempRock/"))
                {
                    var v5 = value(5);
                    var v6 = value(6);
                    var v7 = value(7);
                    var v8 = value(8);
                    var v9 = value(9);

                    db.Put(v5, v5);
                    db.Put(v6, v6);
                    db.Put(v7, v7);
                    db.Put(v8, v8);
                    db.Put(v9, v9);

                    var start = v6;
                    var end = v8;

                    var tasks = Enumerable
                        .Repeat(1, 19)
                        .Select(i => Task.Run(() => iterate(db, start, end, 110_000)))
                        .ToArray();

                    Console.Out.WriteLine("Running");

                    Task.WaitAll(tasks);

                    Console.Out.WriteLine("Done");
                }
            }
        }

        static byte[] value(byte number)
        {
            return new byte[] { 1, 2, number, 4 };
        }

        static void iterate(RocksDb db, byte[] start, byte[] end, int count)
        {
            for(int i = 0; i < count; i++)
            {
                using(var iterator = db.NewIterator(
                    readOptions: new ReadOptions().SetIterateUpperBound(end)))
                {
                    iterator.Seek(start);

                    while(iterator.Valid())
                    {
                        var v = (int)iterator.Key()[2];
                        
                        if(v != 6 && v != 7)
                        {
                            Console.Out.WriteLine(v);
                        }

                        iterator.Next();
                    }
                }
            }
        }
    }
}

ArgumentNullException in Get(byte[], byte[]) when key does not exist

When calling public long Get(byte[] key, long keyLength, byte[] buffer, long offset, long length, ColumnFamilyHandle cf = null, ReadOptions readOptions = null) or the other Get()s which redirect there, the code in

rocksdb-sharp/RocksDbSharp/RocksDb.cs

Lines 111 to 121 in 8818ab4

 public long Get(byte[] key, long keyLength, byte[] buffer, long offset, long length, ColumnFamilyHandle cf = null, ReadOptions readOptions = null) 

 { 

 unsafe 

 { 

 var ptr = Native.Instance.rocksdb_get(Handle, (readOptions ?? DefaultReadOptions).Handle, key, keyLength, out long valLength, cf); 

 valLength = Math.Min(length, valLength); 

 Marshal.Copy(ptr, buffer, (int)offset, (int)valLength); 

 Native.Instance.rocksdb_free(ptr); 

 return valLength; 

 } 

 }

tries to unconditionally copy the from the result ptr. That pointer might be a nullptr, however, if the key is not found, in which case the Marshall.Copy throws an ArgumentNullException.

Instead, the method should either throw a more appropriate exception, or return a proper result code.
Just returning 0 would be wrong, as you cannot distinguish that from a an empty/null db value, so returning something less than 0 might work better.

Assembly is not signed

Adding strong name to the assembly makes it much easier to consume.

[Question] Why "native" not "runtimes"?

Related to #23

AFAIK nuget toolchain introduces some mechanisms to deal with cross platform native libraries. I think LibUV used by Kestrel can be a decent example.

@warrenfalk
If I understand correctly you are the author of https://github.com/warrenfalk/auto-native-import. In documentation I can read that intention of your library is to add automatic x86/x64 support at the runtime.

I believe this is no longer a problem for example (again...) LibUV provides six different runtimes debian-x64, osx, rhel-x64, win7-arm, win7-x64, win7-x86 and is able to load proper platform library.

DotNet 4.0 support

Can you make support 4.0 framework? Yes, thats old. But some old projects cannot/want to switch to 4.5, but need a good key-value storage.

[Question] Why not P/Invoke?

Current implementation of native interop uses dynamically generated IL. While this approach should be quite efficient, there are some implications of dynamic IL making library impossible to use in some scenarios.

For example it cannot be used in some environments with prohibited dynamic code generation or AOT scenarios.

Also IL generation adds some bloat to the .NET Core applications.

Are there still any issues with DllImport performance that justify dynamically generated IL instead of old good P/Invoke?

Support .net core with nuget

Tried to use RocksDbSharp with .net core. Does not work.

error: Package RocksDbSharp 4.13.1 is not compatible with netcoreapp1.0 (.NETCoreApp,Version=v1.0). Package RocksDbSharp 4.13.1 supports: net40 (.NETFramework,Version=v4.0)
error: One or more packages are incompatible with .NETCoreApp,Version=v1.0.
log : Lock file has not changed. Skipping lock file write. Path: /home/param/tensorX/project.lock.json
log : project.json
log : Restore failed in 3009ms.

Errors in project.json
Package RocksDbSharp 4.13.1 is not compatible with netcoreapp1.0 (.NETCoreApp,Version=v1.0). Package RocksDbSharp 4.13.1 supports: net40 (.NETFramework,Version=v4.0)
One or more packages are incompatible with .NETCoreApp,Version=v1.0.

Target NetStandard1.x

The library targets netcoreapp1.0, but it should target one of netstandard1.x profiles.
netcoreapp - for executables
`netstandard1 - for libraries

Adding native ARM RocksDB binaries to nuget?

Would it be possible to get arm binaries put into the nuget package for this project? I'm specifically looking for ARM32 binaries ("arm" architecture).

	public long Get(byte[] key, long keyLength, byte[] buffer, long offset, long length, ColumnFamilyHandle cf = null, ReadOptions readOptions = null)
	{
	unsafe
	{
	var ptr = Native.Instance.rocksdb_get(Handle, (readOptions ?? DefaultReadOptions).Handle, key, keyLength, out long valLength, cf);
	valLength = Math.Min(length, valLength);
	Marshal.Copy(ptr, buffer, (int)offset, (int)valLength);
	Native.Instance.rocksdb_free(ptr);
	return valLength;
	}
	}