iondbproject / iondb Goto Github PK
View Code? Open in Web Editor NEWIonDB, a key-value datastore for resource constrained systems.
License: BSD 3-Clause "New" or "Revised" License
IonDB, a key-value datastore for resource constrained systems.
License: BSD 3-Clause "New" or "Revised" License
Currently, working with IonDB "feels like" a C library. Write a C++ wrapper that simplifies the usage of IonDB to bring it more in line with how a typical Arduino library is used. The goal is to remove burden from the user and free them from tasks such as having to manually malloc
memory for keys and values.
Currently, we use a locally defined macro called ION_ARDUINO to do conditional compilation on device. However, there are also references in our code to an ARDUINO macro, which comes from the headers included when compiling on device.
We need to resolve this conflict and standardize on one consistent usage throughout the codebase.
The benchmarking suite, ionbench
, currently assumes the implementation under test is fully functional. Some checks should be added to assert the return code from the methods tested (such as dictionary_create
, dictionary_insert
, etc) are returning err_ok
, and not something bad like err_out_of_memory
.
Possible talking points:
Currently, the above three implementations leak memory, as their cursors are created but never destroyed. The reason why they're never destroyed is due to a conflict in how they're written: In certain cases, stack memory is bound to the cursor, which cursor->destroy
then attempts to free
. In most cases this results in a segfault. As a temporary measure, most cursor->destroy
calls have been commented out.
The relevant tests should be refactored to avoid the situation of free
-ing stack memory.
Reference: test_oadictionaryhandler.c:353
and 380.
Currently, cursor->next
expects a record
struct to be passed in, which it populates with a record retrieved from the dictionary. This is cumbersome to the user, as they are expected to manually malloc
the members of the struct. This also requires knowledge of how to retrieve the key and value size from within the dictionary.
Example:
ion_record_t record;
record.key = (ion_key_t) malloc(dict.instance->record.key_size);
record.value = (ion_value_t) malloc(dict.instance->record.value_size);
while(cursor->next(cursor, &record) != cs_end_of_results);
Proposal: Possible remove the need for the record struct, or develop a helper method to build the record for the user, or any other idea that might be a better option.
Currently, we have a set of tests called generic_dictionary_test
. These set of tests operate on black box style dictionaries, and only assert at the dictionary level the correctness of the dictionary. However, these tests only define the individual tests themselves, but no suite of tests has been made yet.
A suite of tests should be written that utilize the generic_dictionary_test
functionality, covering all use cases and edge cases. The goal is to be able to pass in the handler for different implementations, and it run the tests. This is useful in answering the question, "Does this implementation support all the functionality we need it to?"
Refer to B+ tree tests, as it does something very similar already. We just need to pull it out into a more generalized test loader.
Do I understand it right that size of keys and values is fixed per DB ("dictionary")? But that doesn't scale at all!
I really appreciate this code, nice work.
Is your support for duplicate keys fully supported?
Can you describe the method you used to support this feature?
Thanks
The following things have been added (or changed) and should be reflected in our public README:
The B+ tree currently uses the ID directly as its filename for the two files it creates. This needs to be changed to use the file encoder.
Currently, the B+ Tree only tests for functional correctness at the dictionary level (via the generic_dictionary_test
utility). Proper unit tests should be written for the implementation level that asserts the internal correctness of the tree.
Currently, any time we do conditional compilation, it is done using the #ifdef directive. To allow for compound conditionals such as A && !B, switch all usage of #ifdef to be #if defined() instead.
https://github.com/iondbproject/iondb/blob/development/src/dictionary/dictionary.c#L520-L523
https://github.com/iondbproject/iondb/blob/development/src/dictionary/dictionary.h#L269
https://github.com/iondbproject/iondb/blob/development/src/dictionary/flat_file/flat_file_dictionary_handler.c#L78-L81
https://github.com/iondbproject/iondb/blob/development/src/dictionary/flat_file/flat_file_dictionary_handler.c#L340-L343
https://github.com/iondbproject/iondb/blob/development/src/dictionary/skip_list/skip_list_handler.c#L281-L284
In the BPP_tree.c bUpdateKey function, why bother checking for space in the root, and child nodes?
Can the checks and subsequent gather and scatter calls be removed from this function?
iondb is awesome library for embedded devices, but I meet a problem, can someone help me? Thanks in advance:)
delete/get failed after delete first 3 records when key_type is key_type_null_terminated_string, here is the test code:
#include <stdio.h>
#include <assert.h>
#include "../src/key_value/kv_system.h"
#include "../src/dictionary/dictionary.h"
#include "../src/dictionary/bpp_tree/bpp_tree_handler.h"
#include "../src/dictionary/ion_master_table.h"
int
main(
void
) {
int key_size, value_size;
ion_key_type_t key_type;
ion_dictionary_handler_t bpp_tree_handler;
ion_dictionary_t dictionary;
ion_status_t status;
key_type = key_type_null_terminated_string;
key_size = 12;
value_size = 20;
ion_byte_t key[key_size];
ion_byte_t value[value_size];
ion_byte_t new_value[value_size];
if (ion_init_master_table() != err_ok) {
printf("Failed to create the master table\n");
return 1;
}
bpptree_init(&bpp_tree_handler);
if (ion_master_table_create_dictionary(&bpp_tree_handler, &dictionary, key_type, key_size, value_size, -1) != err_ok) {
printf("Failed to create the dictionary\n");
return 1;
}
int i;
int n = 100;
printf("stress test begin\n");
for (i = 0; i < n; i++) {
sprintf(key, "%d", i);
sprintf((char *) value, "Hello World %d", i);
printf("%d insert %s %s\n", i, key, value);
status = dictionary_insert(&dictionary, key, value);
assert(status.error == err_ok);
status = dictionary_get(&dictionary, key, new_value);
assert(status.error == err_ok && strcmp(value, new_value) == 0);
}
for (i = 0; i < n; i++) {
sprintf(key, "%d", i);
status = dictionary_delete(&dictionary, key);
printf("%d delete %s return %d\n", i, key, status.error);
assert(status.error == err_ok);
}
printf("stress test end\n");
if ((ion_close_master_table() != err_ok) || (ion_delete_master_table() != err_ok)) {
printf("Failed to close and delete the master table\n");
return 1;
}
printf("Done\n");
return 0;
}
if n < 36, it works well. Did I miss something?
The assertion of 0 != feof(file_ptr)
is failing. This causes the flatfile to not be closed, and thus file.bin
is not deleted, which causes a segfault in later tests (as the dictionary cannot be successfully created if file.bin
exists).
To fix: Investigate whether or not the assertion is semantically valid, and adjust code to suit.
Right now it appears that IonDB assumes that when opening a file it is at the root of the filesystem. This may not always be the case. There are any number of reasons why one might like a database placed in a specific folder.
For example, in my case I am using IonDB on a ARM microcontroller using the mBed RTOS. mBed uses mountable filesystems. In my case I have a SPI Flash based LittleFileSystem mounted at /ifs
(and the SD card, if there is one, at /sd
), so I need to make sure that all databases are created at /ifs
if not lower in the directory structure.
I am happy to tackle this, but would appreciate some guidance as to the "IonDB-way" approach
First we check if there's already a duplicate node. If there is, we're going to do a modified insert instead. TODO write unit cpp_wrapper to check this
Currently, there exists a re-direct for translating C-style file calls (such as fwrite
, fread
, ...) into Arduino SD file calls (like SD.begin()
, SD.open()
, ...). This should be refactored to be more platform independent, and to open up flexibility to support other file systems, such as TEFS, FATFS, and others.
Currently, our comment documentation is not fully compliant with the doxygen syntax. A refactoring pass is needed to address this.
The generalized file interface, ion_file
, is currently only used by the B+ tree. With the introduction of the SD File IO Intercept, ion_file
is redundant and no longer needed.
ion_file
should be removed, and any previous references (most likely only in the B+ tree) to ion_file
must be resolved to use standard C file IO.
Also refer to issue "#ifdef ION_ARDUINO in B+ tree", as removing ion_file
will resolve that issue as well. #3
The ion_master_table
implementation currently allows lookup by providing the configuration of the dictionary to retrieve it from storage. By team consensus, it has been decided that this is not a good way to use the master table and should be removed. The only way to retrieve a dictionary should be by providing the ID.
In the meeting of July 28, 2015, the naming of dictionary_close
was discussed. The agreement was that close
is an inaccurate name for what it does, and might better be called flush
. The specifics of this are still up for discussion.
ion_status_t
is currently partially implemented. To fully implement ion_status_t
, the following must be done:
err
will now read the err
member of the ion_status_t
struct. These unit tests should also assert the rows affected is reported correctly.dictonary_destroy_predicate_range
and similar have a typo in the word "dictonary".
CuTest is obsolete, as we have fully ported to PlanckUnit. Remove the CuTest files and any references leftover in code and comments.
There are unit test(s) in the LinearHash that currently fail. This must be addressed before the LinearHash can be accepted into stable release.
To be consistent, filenames should be lowercased and snake_cased.
Hello,
i copy IonDB to my esp32 arduino directory "components\arduino\libraries\IonDB",
add include line #include "IonDB.h" or #include <IonDB.h> before #include "Arduino.h" or after,
but get error "main/example.h:12:19: fatal error: IonDB.h: No such file or directory",
if write full path, then not found other IonDB source files,
where i can get help for build with ESP32 FreeRTOS SDK?
Right now iond isn't using a prefix for most functions, this leaving them in the "global" namespace โ at least by C convention.
As such all functions and datatypes should be refactored to have the iond_ prefix to avoid this
There are lots of TODOs and FIXMEs scattered in our code base. Lots of them haven't been resolved because they require discussion. Some of them are old and have already been resolved.
In general, if you encounter a TODO that you can fix reasonably quickly, do so.
If it is not in an area you're familiar with, post an issue with the TODO, reference the line(s) of code, and remove the TODO from within the code. Do the same for TODOs that require further discussion from the group.
If you encounter a TODO that has already been finished, remove it from the code.
The Arduino math library does not contain a definition for the log2 math function. This needs to be addressed for the LinearHash to compile on device.
It would be great if you add a example in plain C.
Cheers Mathias
Currently, the B+ Tree implementation we have was not written by our developers, but found as an open-source implementation that was adapted to our needs. However, the code style and standard followed in the implementation are not in line with IonDB standards, and certain design decisions made render it unusable on the smaller Arduino devices, such as the Uno.
If the interest is present, a new B+ Tree implementation should be written that addresses these issues. Specifically, the implementation should dump as much as it can onto disk and minimize the memory footprint.
The serial_c_iface
printf
intercept must be included after the standard library definitions, otherwise it is overwritten. Currently, there are parts of our library that are effectively "dark" to the intercept, and any prints originating from such locations are not shown correctly.
To fix this issue, stdlib.h
and stdio.h
includes should be centralized to one location (probably kv_system
) so we are aware of when it is being implicitly included. Then, serial_c_iface.h
should be explicitly included in any file that utilizes printf
.
Hi,
I tired you library, but I am getting fatal error:
Currently, all IonDB unit tests use the CuTest library. We now have an in-house unit testing library named PlanckUnit, which can be found here.
The goal is to migrate all of our unit tests to use PlanckUnit. Realistically, this should just amount to changing method calls from the CuTest name to the Planck name.
Steps:
Is iinq working? Can I actually use it, if so there should be documentation. If not, then the current state should be documented
Currently, our #includes are sometimes redundant.
Goal: Streamline #includes.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.