Structured value about immudb HOT 13 CLOSED

Please note that GetSV and SetSV methods are not implemented in Server, so to implement SafeGet/SafeSet methods in any client (gw, drivers - our tasks, CLI - your), the client should append timestamp to payload bytestream (see /pkg/client/convert.go)

from immudb.

The inner type of any is []Byte, so you shouldn't have any problems because the old value is of the same type.

@mmeloni , I think you're not correct: we have to implement separate marshalling of incoming data (for Set and SafeSet) in unstructured format {bytes[] key, bytes[] value}. If we'll use just autogenerated marshaller for new StructuredValue {bytes[] key, protobuf.Any value} then immugw will be able to receive only protobuf-prepared json in format

{
    "Key": "KEY",
    "Value": {
         "TypeUrl": "googleApisAny", 
          "Value": "VALUE"
      }
}

So it is not just small refactoring.

from immudb.

@nredko You are right. I was talking from an immudb point of view. Inside immugw methods's have to wrap complexity and let keep required only the strict necessary data.

from immudb.

We got very significant issue with this approach. protobuf.any is not suitable as replacement for payload byte array, since protobuf doesn't guarantee deterministic serialization: bytes array presenting any structure enveloping StructuredValue can be serialized to different bytes arrays on different platforms, in different languages, different builds and even different runs of the same code. See https://github.com/protocolbuffers/protobuf/releases/tag/v3.0.0 - deterministic serialization requires special methods and is available only for C++.

We found this since we've got different results of hash function for any structures with same values. This makes such approach unusable.

So current proposal is to just append additional field timestamp to protocol, and implement our own serialization for compatibility with badger (i.e. first 8 bytes of badger's value []bytes will be deserialized as timestamp).

from immudb.

Probably more proper approach is to replace 'payload' type with structure {timestamp, bytes}

from immudb.

@nredko it's a pity that any doesn't work. However if we replace payload type and we will still be able to manage value without refactorize immudb core should not be a problem.
I mean that I think we will able to manage value without unmarshal on immudb side, except for the specifics cases that we need to open the payload. Whats your opinion on that?

from immudb.

@mmeloni If we change only KeyValue (transport structure) but leave Item (db structure) as is (bytestream as value) - we can use our own (very simple) serialization between transport and db, so it will not affect any budger data - so we will not get any decrease in performance.

message KeyValue {
	bytes key = 1;
	Content value = 2;
}

message Content {
	uint64 timestamp = 1;
	bytes payload = 2;
}

message Index {
	uint64 index = 1;
}

message Item {
	bytes key = 1;
	bytes value = 2;
	uint64 index = 3;
}

from immudb.

Ok,
only to be sure:
methods like:

func (s *ImmuServer) Set(ctx context.Context, kv *schema.KeyValue) (*schema.Index, error) {

Will not change.
Only if we want to explore timestamp, for example, we will need to unmarshal content.

from immudb.

In my understanding, there should be difference of models for server and for client. For now they all are messed and are used only according the set of required fields, not according layer of execution.

Server methods should use Item (probably with another structure without index like NewItem) and only client methods should use KeyValue.

message Index {
	uint64 index = 1;
}

message Item {
	bytes key = 1;
	bytes value = 2;
	uint64 index = 3;
}

message NewItem {
	bytes key = 1;
	bytes value = 2;
}

So that method should look like

func (s *ImmuServer) Set(ctx context.Context, ni *schema.NewItem) (*schema.Index, error) {

Otherwise we'll need to propagate serialization/deserialization down to server/database layer.

from immudb.

Well, I’m a bit overworked :)
Just replace ‘NewItem’ with ‘KeyValue’ and call my modified ‘KeyValue’ ‘StructuredValue’.
In this case signature of server functions will not change in a char.

message KeyValue {
	bytes key = 1;
	bytes value = 2;
}
message StructuredKeyValue {
	bytes key = 1;
	Content value = 2;
}
message Content {
	uint64 timestamp = 1;
	bytes payload = 2;
}

message Index {
	uint64 index = 1;
}

message Item {
	bytes key = 1;
	bytes value = 2;
	uint64 index = 3;
}

from immudb.

Ok no problem Nik, thank you very much. When you are ready could you please share a prototype with a working Set so we can make a tour?

from immudb.

@mmeloni please look at prototype at https://github.com/nredko/immudb

from immudb.

@nredko Thank you very much.
@padurean please take a look and give us a feedback in case you see something wrong in order to merge your work on #40 when ready

from immudb.