Comments (13)
Please note that GetSV and SetSV methods are not implemented in Server, so to implement SafeGet/SafeSet methods in any client (gw, drivers - our tasks, CLI - your), the client should append timestamp to payload bytestream (see /pkg/client/convert.go)
from immudb.
The inner type of any is []Byte, so you shouldn't have any problems because the old value is of the same type.
@mmeloni , I think you're not correct: we have to implement separate marshalling of incoming data (for Set and SafeSet) in unstructured format {bytes[] key, bytes[] value}
. If we'll use just autogenerated marshaller for new StructuredValue {bytes[] key, protobuf.Any value}
then immugw will be able to receive only protobuf-prepared json in format
{
"Key": "KEY",
"Value": {
"TypeUrl": "googleApisAny",
"Value": "VALUE"
}
}
So it is not just small refactoring.
from immudb.
@nredko You are right. I was talking from an immudb point of view. Inside immugw methods's have to wrap complexity and let keep required only the strict necessary data.
from immudb.
We got very significant issue with this approach. protobuf.any
is not suitable as replacement for payload
byte array, since protobuf doesn't guarantee deterministic serialization: bytes array presenting any
structure enveloping StructuredValue
can be serialized to different bytes arrays on different platforms, in different languages, different builds and even different runs of the same code. See https://github.com/protocolbuffers/protobuf/releases/tag/v3.0.0 - deterministic serialization requires special methods and is available only for C++.
We found this since we've got different results of hash function for any
structures with same values. This makes such approach unusable.
So current proposal is to just append additional field timestamp
to protocol, and implement our own serialization for compatibility with badger (i.e. first 8 bytes of badger's value []bytes
will be deserialized as timestamp).
from immudb.
Probably more proper approach is to replace 'payload' type with structure {timestamp, bytes}
from immudb.
@nredko it's a pity that any doesn't work. However if we replace payload type and we will still be able to manage value without refactorize immudb core should not be a problem.
I mean that I think we will able to manage value without unmarshal on immudb side, except for the specifics cases that we need to open the payload. Whats your opinion on that?
from immudb.
@mmeloni If we change only KeyValue
(transport structure) but leave Item
(db structure) as is (bytestream as value) - we can use our own (very simple) serialization between transport and db, so it will not affect any budger
data - so we will not get any decrease in performance.
message KeyValue {
bytes key = 1;
Content value = 2;
}
message Content {
uint64 timestamp = 1;
bytes payload = 2;
}
message Index {
uint64 index = 1;
}
message Item {
bytes key = 1;
bytes value = 2;
uint64 index = 3;
}
from immudb.
Ok,
only to be sure:
methods like:
func (s *ImmuServer) Set(ctx context.Context, kv *schema.KeyValue) (*schema.Index, error) {
Will not change.
Only if we want to explore timestamp, for example, we will need to unmarshal content.
from immudb.
In my understanding, there should be difference of models for server and for client. For now they all are messed and are used only according the set of required fields, not according layer of execution.
Server methods should use Item
(probably with another structure without index like NewItem
) and only client methods should use KeyValue
.
message Index {
uint64 index = 1;
}
message Item {
bytes key = 1;
bytes value = 2;
uint64 index = 3;
}
message NewItem {
bytes key = 1;
bytes value = 2;
}
So that method should look like
func (s *ImmuServer) Set(ctx context.Context, ni *schema.NewItem) (*schema.Index, error) {
Otherwise we'll need to propagate serialization/deserialization down to server/database layer.
from immudb.
Well, I’m a bit overworked :)
Just replace ‘NewItem’ with ‘KeyValue’ and call my modified ‘KeyValue’ ‘StructuredValue’.
In this case signature of server functions will not change in a char.
message KeyValue {
bytes key = 1;
bytes value = 2;
}
message StructuredKeyValue {
bytes key = 1;
Content value = 2;
}
message Content {
uint64 timestamp = 1;
bytes payload = 2;
}
message Index {
uint64 index = 1;
}
message Item {
bytes key = 1;
bytes value = 2;
uint64 index = 3;
}
from immudb.
Ok no problem Nik, thank you very much. When you are ready could you please share a prototype with a working Set so we can make a tour?
from immudb.
@mmeloni please look at prototype at https://github.com/nredko/immudb
from immudb.
@nredko Thank you very much.
@padurean please take a look and give us a feedback in case you see something wrong in order to merge your work on #40 when ready
from immudb.
Related Issues (20)
- Set admin password on AWS EC2 image using user data
- New immudb instance with S3 options turned on cause a startup error HOT 1
- The immudb logs are not availabe HOT 6
- immudb replica uses too much RAM
- Deleted data is still accessible using history
- Key commited in a transaction is not visible in the next transaction HOT 3
- tx read conflict error on a concurrent read HOT 1
- Error deploying immudb on EKS cluster with EBS added HOT 1
- Getting Error: indexed key of unsupported type or exceeded length: can not create index HOT 3
- etcd API
- Select statement gets stuck due to indexing issue
- Add audit logs
- immuadmin login doesn't work when running immuadmin in docker image
- Get Retention-Period of an already created database
- Document deletion from a collection
- SQL (HISTORY OF TABLE) should have "updated_at ", "db_username" for auditing. HOT 3
- clients report: Already Closed HOT 9
- Load testing immudb hits `max active snapshots` error
- This download does NOT match the one reported by the checksum server HOT 2
- CRASH: Immudb crashes when modifying the same table in the same transaction HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from immudb.