siridb / siridb-server Goto Github PK

When using a huge amount of recursion in a query this can cause a stack overflow.

We can add a counter on the fifo buffer.

One should be able to use an IPv6 address to specify a server_name in the configuration file.
The IPv6 address should then be wrapped in square brackets.

For example:

server_name = [::1]:9010

When using multiple regular expressions in one query, a memory leak can occur.

Reported by @Koos85 .

When dropping a group which does not exist its possible the SiriDB crashes.

List, count and select statements should return an error message when a series is queried by name and the series does not exist. This works correctly when the receiving server should have the series but fails if the series should exist in another pool.

When using a select statement with merge and a string filter, this generates an error message because string filters are not allowed on number series. The message is correctly returned most of the time, but sometimes can cause a crash of SiriDB.

When a start time greater than the end time is parsed in a query, SiriDB cannot respond to this request and seems to hang.

When a shard has bytes left at the end and the number of bytes are less than one header size, we do not mark the shard as corrupt. The end result is that we start writing to the shard and only if the shard is optimized before actually turning SiriDB off, the new points are lost since we cannot read the shard after the invalid bytes. (optimizing solves this because only valid data is written to the optimized shard).

All queries are check for the correct permissions but inserts can be done even with a user account without insert permissions.

An int64_t on a 64bit system is a long int, but a long long on a 32bit system.

Formatting an int64_t with "%ld" will raise a warning on 32bit.
Formatting an int64_t with "%lld" will raise a warning on 64bit.

see cesbit/libcleri#1

On the google cloud platform the case showed up where a request send to another SiriDB server was received by its own process. We now accept this request because the UUID is valid. We should however reject this request.

SiriDB will try to load databases from all directories inside the configured database path.
The first thing SiriDB does is create a lock file, and then finds its not a valid DB path.

Queries with a huge size > 10MB can cause a stack overflow.

Since inserts are done asynchronous, it can happen that data points are missing when creating a new replica. We decide to send series to the replica and then start the asynchronous task.

One possible solution is to check if inserts are busy and only send the next series when no insert tasks are running. Another option is to decide to write data for the replica on each insert iteration.

Creating an imap slist is not thread safe and should have an appropriate lock when the list can be accessed by multiple treads. The main thread missed a lock which could cause the database to crash.

When connecting to another server we do not correctly resolve DNS and for some reason localhost is used as fallback.

The correct behavior should be to first test for an IPv4 address, next an IPv6 address, then try DNS and if all fail we should simple write a log and free the socket. (on the next heart-beat, SiriDB will then try to connect again)

Drop series works fine, except when using only a 'where' statement. If used in combination with a regular expression of group everything works fine.

We only check DNS for an ipv4 address but we should include support for ipv6.

(change AF_INET to AF_UNSPEC and make the callback function ipv6 compatible)

A series object reference counter is saved in a uint16_t, which is enough for a series object. Shards on the other hand can have more references and are stored in an uint32_t. We need to map a general function to increment and decrement the references on these objects. On little endian systems this works bun on big endian a problem might occur since bytes are stored in the reverse order.

When using median, a segmentation fault can occur when using a large points set.

Thanks @Koos85 for reporting this issue!

We should replace the default value 'localhost' in the configuration file with a variable, for example %HOSTNAME which will be replaced with the systems host name. Localhost is never a good choice since this is the address we send to 'other' SiriDB servers which will then try to connect to this address.

When send a CPROTO_REQ_FILE_GROUPS, the logging report and UNKNOWN type.

Memory leak in difference function. (imap object)

Possible memory leak found in alter group set expression. The reference counter on series in the group are not decremented correctly.

With issue #14 we have set dns request to be compatible with bot IPv4 and IPv6. Since we now have a property ip_support which can be set to IPV4ONLY, IPV6ONLY or ALL, we should honor this setting in DNS requests.

You should be able to enable the IPv6 stack only, and the same is true for IPv4 only.

We should add the option ip_support with the options ALL, IPV4ONLY and IPV6ONLY with the default setting to ALL

We should also update the grammar the view the ip_support setting on each server.

An assertion is hit when listing or counting server when querying with properties REINDEX_PROGRESS and SYNC_PROGRESS.

When the buffer path exceeds PATH_MAX (4096 characters) we should still get a null terminated string. Otherwise a segmentation fault can occur.

When multiple aggregation functions are used together with a merge statement SiriDB crashes.

When multiple packages are send in a stream, SiriDB crashes with a memory fault.

Using valgrind some errors showed up while enabling backup mode.

uninitialized data is written by server send package function.

An unused include existed in auth.h and clserver.c for msgpack. MsgPack was used in early development but is now replaced with QPack.

When creating a group fails, for example by using an incorrect name, the temporary group is not cleared correctly.

When ending a set expression with a space character after the expression, the groups are not saved correctly so fail to read at next startup.

A double free on a socket can occur when receiving data on a socket which is closed already be the on_data function. We should prevent the on_data to close a socket twice.

When using a select query, some work must be done by uv_queue_work and therefore needs space in the thread pool. The default UV_THREADPOOL_SIZE is set to 4 but each database requires at least one thread.

When a task to remove shard is initiated, and next the optimize task will start, we can optionally drop the 'wrong', say 'old' shard.

SiriDB writes a log entry when an invalid package (or too large package) is received. This line does currently not include the source IP which makes it hard to debug such packages.

Include the source IP so we can see who had send the illegal package.

Now we can re-use dropped buffer space its also possible to use the same technique for pre-allocating space in the buffer for new series.

SiriDB can set another buffer_path using the configuration file but the buffer is not created inside the given path.

When having multiple fifo files, emptying can fail when opening the next file.

We can list, query or count shards and its size property. SiriDB currently uses a function to read the current shard size. This is rather slow since we read the file size from the operation system, or in case the file is open, we return the file size using an fseeko function call. This last function call is not thread safe and can conflict with the optimize thread.

We better keep a size property which is lightweight so we can return the property a lot faster and thread safe.

An optimize interval or zero should be possible to disable the optimize task.

Some objects like series, shards, promises (maybe servers and groups?) are incremented and decrements a lot and can be written as macros. We still need a normal function call available since we sometimes need to parse the decref function as callback.

When we drop a series, we can reclaim the allocated buffer space. This is faster compared to extending the buffer and saves space on a running SiriDB instance.

When asking for a median, median_low or median_high on large values (close to 2**63) wrong values are returned.

Instead of solving this bug we better change the configuration option from
listen_client to listen_client_port and just accept a port number.

What we additionally can do is change listen_server to server_name and make
clear in the description that we listen on any address (0.0.0.0). We still need the
address because this is the address and port which other servers are using to connect to.

When using median, median_high or median_low on increasing data set, lets say for example having the values 1,2,3,4,5 etc, the current algorithm hit its worst case scenario.

Since these series are quite common (for example counter series) we should improve the algorithm for these types of series.