Hello,
i copy IonDB to my esp32 arduino directory "components\arduino\libraries\IonDB",
add include line #include "IonDB.h" or #include <IonDB.h> before #include "Arduino.h" or after,
but get error "main/example.h:12:19: fatal error: IonDB.h: No such file or directory",
if write full path, then not found other IonDB source files,
where i can get help for build with ESP32 FreeRTOS SDK?

To be consistent, filenames should be lowercased and snake_cased.

Currently, we use a locally defined macro called ION_ARDUINO to do conditional compilation on device. However, there are also references in our code to an ARDUINO macro, which comes from the headers included when compiling on device.

We need to resolve this conflict and standardize on one consistent usage throughout the codebase.

The B+ tree currently uses the ID directly as its filename for the two files it creates. This needs to be changed to use the file encoder.

Currently, cursor->next expects a record struct to be passed in, which it populates with a record retrieved from the dictionary. This is cumbersome to the user, as they are expected to manually malloc the members of the struct. This also requires knowledge of how to retrieve the key and value size from within the dictionary.

Example:

        ion_record_t record;
        record.key      = (ion_key_t) malloc(dict.instance->record.key_size);
        record.value    = (ion_value_t) malloc(dict.instance->record.value_size);

        while(cursor->next(cursor, &record) != cs_end_of_results);

Proposal: Possible remove the need for the record struct, or develop a helper method to build the record for the user, or any other idea that might be a better option.

https://github.com/iondbproject/iondb/blob/development/src/file/ion_file.h
https://github.com/iondbproject/iondb/blob/development/src/file/ion_file.c

https://github.com/iondbproject/iondb/blob/development/cmake/Platform/Arduino.cmake#L957-L959

https://github.com/iondbproject/iondb/blob/development/src/dictionary/dictionary.c#L198-L207
https://github.com/iondbproject/iondb/blob/development/src/dictionary/dictionary.c#L248-L256

iondb is awesome library for embedded devices, but I meet a problem, can someone help me? Thanks in advance:)

delete/get failed after delete first 3 records when key_type is key_type_null_terminated_string, here is the test code:

#include <stdio.h>
#include <assert.h>
#include "../src/key_value/kv_system.h"
#include "../src/dictionary/dictionary.h"
#include "../src/dictionary/bpp_tree/bpp_tree_handler.h"
#include "../src/dictionary/ion_master_table.h"

int
main(
    void
) {
    int                         key_size, value_size;
    ion_key_type_t              key_type;
    ion_dictionary_handler_t    bpp_tree_handler;
    ion_dictionary_t            dictionary;
    ion_status_t                status;

    key_type    = key_type_null_terminated_string;
    key_size    = 12;
    value_size  = 20;

    ion_byte_t key[key_size];
    ion_byte_t  value[value_size];
    ion_byte_t  new_value[value_size];

    if (ion_init_master_table() != err_ok) {
        printf("Failed to create the master table\n");
        return 1;
    }

    bpptree_init(&bpp_tree_handler);
    if (ion_master_table_create_dictionary(&bpp_tree_handler, &dictionary, key_type, key_size, value_size, -1) != err_ok) {
        printf("Failed to create the dictionary\n");
        return 1;
    }

    int i;
    int n = 100;
    printf("stress test begin\n");
    for (i = 0; i < n; i++) {
        sprintf(key, "%d", i);
        sprintf((char *) value, "Hello World %d", i);
        printf("%d insert %s %s\n", i, key, value);
        status = dictionary_insert(&dictionary, key, value);
        assert(status.error == err_ok);
        status = dictionary_get(&dictionary, key, new_value);
        assert(status.error == err_ok && strcmp(value, new_value) == 0);
    }

    for (i = 0; i < n; i++) {
        sprintf(key, "%d", i);
        status = dictionary_delete(&dictionary, key);
        printf("%d delete %s return %d\n", i, key, status.error);
        assert(status.error == err_ok);
    }
    printf("stress test end\n");

    if ((ion_close_master_table() != err_ok) || (ion_delete_master_table() != err_ok)) {
        printf("Failed to close and delete the master table\n");
        return 1;
    }

    printf("Done\n");
    return 0;
}

if n < 36, it works well. Did I miss something?

Currently, our #includes are sometimes redundant.

Goal: Streamline #includes.

Right now iond isn't using a prefix for most functions, this leaving them in the "global" namespace — at least by C convention.

As such all functions and datatypes should be refactored to have the iond_ prefix to avoid this

Currently, all IonDB unit tests use the CuTest library. We now have an in-house unit testing library named PlanckUnit, which can be found here.

The goal is to migrate all of our unit tests to use PlanckUnit. Realistically, this should just amount to changing method calls from the CuTest name to the Planck name.

Steps:

1. Pull the master IonDB branch and make sure you can build and run the existing unit tests and have them all pass.
2. Pull Planck Unit and make sure you can build and run the example for Planck Unit.
3. branch off IonDB/master to a planck branch. Integrate Planck Unit.
4. Once all tests pass once again, merge the planck branch back into master.

Currently, any time we do conditional compilation, it is done using the #ifdef directive. To allow for compound conditionals such as A && !B, switch all usage of #ifdef to be #if defined() instead.

The serial_c_iface printf intercept must be included after the standard library definitions, otherwise it is overwritten. Currently, there are parts of our library that are effectively "dark" to the intercept, and any prints originating from such locations are not shown correctly.

To fix this issue, stdlib.h and stdio.h includes should be centralized to one location (probably kv_system) so we are aware of when it is being implicitly included. Then, serial_c_iface.h should be explicitly included in any file that utilizes printf.

Hi,

I tired you library, but I am getting fatal error:

ion_status_t is currently partially implemented. To fully implement ion_status_t, the following must be done:

• The return types of all affected methods need to be adjusted across the dictionary, handler, and implementation levels.
• Any unit test affected will also need to be changed to accommodate. Reading an err will now read the err member of the ion_status_t struct. These unit tests should also assert the rows affected is reported correctly.
• All implementations must properly support and return the struct, with rows affected.

Currently, working with IonDB "feels like" a C library. Write a C++ wrapper that simplifies the usage of IonDB to bring it more in line with how a typical Arduino library is used. The goal is to remove burden from the user and free them from tasks such as having to manually malloc memory for keys and values.

Right now it appears that IonDB assumes that when opening a file it is at the root of the filesystem. This may not always be the case. There are any number of reasons why one might like a database placed in a specific folder.

For example, in my case I am using IonDB on a ARM microcontroller using the mBed RTOS. mBed uses mountable filesystems. In my case I have a SPI Flash based LittleFileSystem mounted at /ifs (and the SD card, if there is one, at /sd), so I need to make sure that all databases are created at /ifs if not lower in the directory structure.

I am happy to tackle this, but would appreciate some guidance as to the "IonDB-way" approach

The Arduino math library does not contain a definition for the log2 math function. This needs to be addressed for the LinearHash to compile on device.

In the meeting of July 28, 2015, the naming of dictionary_close was discussed. The agreement was that close is an inaccurate name for what it does, and might better be called flush. The specifics of this are still up for discussion.

Currently, we have a set of tests called generic_dictionary_test. These set of tests operate on black box style dictionaries, and only assert at the dictionary level the correctness of the dictionary. However, these tests only define the individual tests themselves, but no suite of tests has been made yet.

A suite of tests should be written that utilize the generic_dictionary_test functionality, covering all use cases and edge cases. The goal is to be able to pass in the handler for different implementations, and it run the tests. This is useful in answering the question, "Does this implementation support all the functionality we need it to?"

Refer to B+ tree tests, as it does something very similar already. We just need to pull it out into a more generalized test loader.

Currently, there exists a re-direct for translating C-style file calls (such as fwrite, fread, ...) into Arduino SD file calls (like SD.begin(), SD.open(), ...). This should be refactored to be more platform independent, and to open up flexibility to support other file systems, such as TEFS, FATFS, and others.

The generalized file interface, ion_file, is currently only used by the B+ tree. With the introduction of the SD File IO Intercept, ion_file is redundant and no longer needed.

ion_file should be removed, and any previous references (most likely only in the B+ tree) to ion_file must be resolved to use standard C file IO.

Also refer to issue "#ifdef ION_ARDUINO in B+ tree", as removing ion_file will resolve that issue as well. #3

https://github.com/iondbproject/iondb/blob/development/cmake/Platform/Arduino.cmake#L407

Currently, the B+ Tree only tests for functional correctness at the dictionary level (via the generic_dictionary_test utility). Proper unit tests should be written for the implementation level that asserts the internal correctness of the tree.

In the BPP_tree.c bUpdateKey function, why bother checking for space in the root, and child nodes?
Can the checks and subsequent gather and scatter calls be removed from this function?

I really appreciate this code, nice work.

Is your support for duplicate keys fully supported?
Can you describe the method you used to support this feature?

Thanks

The following things have been added (or changed) and should be reflected in our public README:

• IONIZE_VAL macro added
• Cursor and Predicate API to be described
• write_concern added (But perhaps this is best kept hidden)
• Dictionary open/close functionality
• Master table functionality

There are unit test(s) in the LinearHash that currently fail. This must be addressed before the LinearHash can be accepted into stable release.

https://github.com/arduino/Arduino/wiki/Library-Manager-FAQ

https://github.com/iondbproject/iondb/blob/development/src/dictionary/flat_file/flat_file_dictionary_handler.c#L59-L82

https://github.com/iondbproject/iondb/blob/development/src/dictionary/flat_file/flat_file_dictionary_handler.c#L230-L352

First we check if there's already a duplicate node. If there is, we're going to do a modified insert instead. TODO write unit cpp_wrapper to check this

https://github.com/iondbproject/iondb/blob/development/src/dictionary/skip_list/skip_list.c#L122-L125

https://github.com/iondbproject/iondb/blob/development/src/dictionary/dictionary.c#L520-L523
https://github.com/iondbproject/iondb/blob/development/src/dictionary/dictionary.h#L269
https://github.com/iondbproject/iondb/blob/development/src/dictionary/flat_file/flat_file_dictionary_handler.c#L78-L81
https://github.com/iondbproject/iondb/blob/development/src/dictionary/flat_file/flat_file_dictionary_handler.c#L340-L343
https://github.com/iondbproject/iondb/blob/development/src/dictionary/skip_list/skip_list_handler.c#L281-L284

Is iinq working? Can I actually use it, if so there should be documentation. If not, then the current state should be documented

It would be great if you add a example in plain C.

Cheers Mathias

Do I understand it right that size of keys and values is fixed per DB ("dictionary")? But that doesn't scale at all!

Currently, the B+ Tree implementation we have was not written by our developers, but found as an open-source implementation that was adapted to our needs. However, the code style and standard followed in the implementation are not in line with IonDB standards, and certain design decisions made render it unusable on the smaller Arduino devices, such as the Uno.

If the interest is present, a new B+ Tree implementation should be written that addresses these issues. Specifically, the implementation should dump as much as it can onto disk and minimize the memory footprint.

The assertion of 0 != feof(file_ptr) is failing. This causes the flatfile to not be closed, and thus file.bin is not deleted, which causes a segfault in later tests (as the dictionary cannot be successfully created if file.bin exists).

To fix: Investigate whether or not the assertion is semantically valid, and adjust code to suit.

There are lots of TODOs and FIXMEs scattered in our code base. Lots of them haven't been resolved because they require discussion. Some of them are old and have already been resolved.

In general, if you encounter a TODO that you can fix reasonably quickly, do so.

If it is not in an area you're familiar with, post an issue with the TODO, reference the line(s) of code, and remove the TODO from within the code. Do the same for TODOs that require further discussion from the group.

If you encounter a TODO that has already been finished, remove it from the code.

The benchmarking suite, ionbench, currently assumes the implementation under test is fully functional. Some checks should be added to assert the return code from the methods tested (such as dictionary_create, dictionary_insert, etc) are returning err_ok, and not something bad like err_out_of_memory.

Possible talking points:

• These checks will cause a small amount of delay, resulting in slightly higher timing measurements. Is this acceptable? Do we adjust for this in any way?
• These checks could be made redundant if we were to deploy solid unit tests on device. Then the behavior of the implementation can be verified before we benchmark.

@todo Try changing the predicate to be an enum-and-switch to eliminate the function
call. Benchmark the performance gain and decide which strategy to use.
@todo Consider changing to @p SEEK_CUR whenever possible. Benchmark this and see
if the performance gain (if any) is worth it.

https://github.com/iondbproject/iondb/blob/development/src/dictionary/flat_file/flat_file.h#L195-L204

this and other places have hacky clauses put in to handle compilation on device. Code should be refactored in such a way that the clauses are no longer needed.

This ties into the other issue "Delete ion_file". When ion_file is removed, this issue will be resolved as well. #14

Currently, the above three implementations leak memory, as their cursors are created but never destroyed. The reason why they're never destroyed is due to a conflict in how they're written: In certain cases, stack memory is bound to the cursor, which cursor->destroy then attempts to free. In most cases this results in a segfault. As a temporary measure, most cursor->destroy calls have been commented out.

The relevant tests should be refactored to avoid the situation of free-ing stack memory.

Reference: test_oadictionaryhandler.c:353 and 380.

CuTest is obsolete, as we have fully ported to PlanckUnit. Remove the CuTest files and any references leftover in code and comments.

dictonary_destroy_predicate_range and similar have a typo in the word "dictonary".

https://github.com/iondbproject/iondb/blob/development/src/file/SD_stdio_c_iface.cpp#L95-L163

Currently, our comment documentation is not fully compliant with the doxygen syntax. A refactoring pass is needed to address this.

https://github.com/iondbproject/iondb/blob/development/src/dictionary/bpp_tree/bpp_tree_handler.c#L154-L157

The ion_master_table implementation currently allows lookup by providing the configuration of the dictionary to retrieve it from storage. By team consensus, it has been decided that this is not a good way to use the master table and should be removed. The only way to retrieve a dictionary should be by providing the ID.