pascaldekloe / colfer Goto Github PK
View Code? Open in Web Editor NEWbinary serialization format
License: Creative Commons Zero v1.0 Universal
binary serialization format
License: Creative Commons Zero v1.0 Universal
Poor performance as seen at #37 seems fixable with WASM.
Hi, is there any chance for Python support?
I am interested in seeing a simple Java code example that uses Colfer for serialization into a ByteStream. I am currently conducting a study on the energy efficiency of serialization/deserialization on Android devices, specifically with regards to large numerical data types such as vectors, and will like to include Colfer in my study. Short of asking @eishay, and because it will provide a quick head-start, a link to a very basic use case would be appreciated.
This is the equals method of my class (compiled with colfer):
public final boolean equals(Metadata o) {
return o != null
&& o.getClass() == Metadata.class
&& this.creationTime == null ? o.creationTime == null : this.creationTime.equals(o.creationTime)
&& this.statistic == null ? o.statistic == null : this.statistic.equals(o.statistic)
&& this.source == null ? o.source == null : this.source.equals(o.source)
&& this.agent == null ? o.agent == null : this.agent.equals(o.agent)
&& this.accessControl == null ? o.accessControl == null : this.accessControl.equals(o.accessControl);
}
If o == null
i get a NPE at the "creationTime" line. From my understanding the ?
condition are evaluated before testing the &&
condition. So in this method this.creationTime.equals
triggers a NPE.
I tried adding brackets to see if i could have the VM to evaluate the null check before all the others and it does work:
public final boolean equals(Metadata o) {
return o != null
&& o.getClass() == Metadata.class
&& (this.creationTime == null ? o.creationTime == null : this.creationTime.equals(o.creationTime))
&& (this.statistic == null ? o.statistic == null : this.statistic.equals(o.statistic))
&& (this.source == null ? o.source == null : this.source.equals(o.source))
&& (this.agent == null ? o.agent == null : this.agent.equals(o.agent))
&& (this.accessControl == null ? o.accessControl == null : this.accessControl.equals(o.accessControl));
}
I am using the latest (1.11.2) colfer maven plugin to generate Java objects for serialization and I noticed that when String fields are explicitly set to null, a NullPointerException is thrown. It looks like the generated code is generating the following code for String name title:
if (! this.title.isEmpty()) {
Can this be changed to check for null first before calling isEmpty()
? Something like
if (this.title!=null && ! this.title.isEmpty()) {
Hi,
I hope all are great.
Scenario:
I have the following mapping in my chappee.colf file.
package chappee
type mappee {
maptype uint32
msgtype uint32
callmsg text
debugleve text
minseed uint64
maxseed uint64
}
During packing, if I set the last two values, the packing is successful.
Now, if during packing I pack first 4 (or any of them) and don't pack the last 2, I get a segmentation fault.
I think it would be great that Colfer itself should take care of the values which are not set.
It can be taken as a bug, or it can be taken as a feature request.
Cheers,
infoginx.com
Please help me i look a stats of library for read / write versus other, You have a good performance because i don't undersand how use this. Thank for read.
Unmarshalling can skip unknown fields / headers when the binary's size is known.
-- @awalterschulze
We have a problem: when maven generate javascript api our parameters with int64 type have equals and incorrect index. Please fix it, as soon as possible please, it is very important for us.
In example serverEventDatetime parameter is int64.
this.Journal.prototype.marshal = function () {
var segs = [];
if (this.serverEventDatetime) {
var seg = [4]; // PROBLEM IN THIS LINE, MUST BE: var seg = [0];
if (this.serverEventDatetime < 0) {
seg[0] |= 128;
if (this.serverEventDatetime < Number.MIN_SAFE_INTEGER)
fail('colfer: api/Journal field serverEventDatetime exceeds Number.MIN_SAFE_INTEGER');
encodeVarint(seg, -this.serverEventDatetime);
} else {
if (this.serverEventDatetime > Number.MAX_SAFE_INTEGER)
fail('colfer: api/Journal field serverEventDatetime exceeds Number.MAX_SAFE_INTEGER');
encodeVarint(seg, this.serverEventDatetime);
}
segs.push(seg);
}
if (this.subsystemCode) {
var utf = encodeUTF8(this.subsystemCode);
var seg = [1];
encodeVarint(seg, utf.length);
segs.push(seg);
segs.push(utf)
}
if (this.code) {
var utf = encodeUTF8(this.code);
var seg = [2];
encodeVarint(seg, utf.length);
segs.push(seg);
segs.push(utf)
}
// ................................................................................
this.Journal.prototype.unmarshal = function (data) {
// ................................................................................
//IN UNMARSHAL HEADER CORRECT
if (header == 0) {
var x = readVarint();
if (x < 0) fail('colfer: api/IJournal field serverEventDatetime exceeds Number.MAX_SAFE_INTEGER');
this.serverEventDatetime = x;
readHeader();
} else if (header == (0 | 128)) {
var x = readVarint();
if (x < 0) fail('colfer: api/Journal field serverEventDatetime exceeds Number.MAX_SAFE_INTEGER');
this.serverEventDatetime = -1 * x;
readHeader();
}
if (header == 1) {
var size = readVarint();
if (size < 0)
fail('colfer: api.Journal.subsystemCode size exceeds Number.MAX_SAFE_INTEGER');
else if (size > colferSizeMax)
fail('colfer: api.Journal.subsystemCode size ' + size + ' exceeds ' + colferSizeMax + ' UTF-8 bytes');
var start = i;
i += size;
if (i > data.length) fail(EOF);
this.subsystemCode = decodeUTF8(data.subarray(start, i));
readHeader();
}
if (header == 2) {
var size = readVarint();
if (size < 0)
fail('colfer: api.Journal.code size exceeds Number.MAX_SAFE_INTEGER');
else if (size > colferSizeMax)
fail('colfer: api.Journal.code size ' + size + ' exceeds ' + colferSizeMax + ' UTF-8 bytes');
var start = i;
i += size;
if (i > data.length) fail(EOF);
this.code = decodeUTF8(data.subarray(start, i));
readHeader();
}
// ................................................................................
For data structures without text, binaries and lists the serial size limit is constant.
Hi,
I used 'go get' to get the Colfer tool yesterday and tried today to generate a simple schema. Unfortunately the generated code does not compile under x86 32-bit due to uint overflow errors. Specifically the timestamp causes the issue:
if v := o.From; !v.IsZero() { if s := uint(v.Unix()); s < 1<<32 { l += 9 } else { l += 13 } }
Colfer.go:160: constant 4294967296 overflows uint
I'm running go1.7.3 linux/386 if that helps. I've attached the .colf file and I'll gladly help test the fix!
Br.
Andreas
Can you support the Object-C language?
The generated code may include code for parsing the version 1 format. To distinguish, version 2 could start with the reserved header byte value 255.
Hello,
A question. I have my serialized colfer byte array inside a ByteBuffer
. And i would like to access just some specific fields. Is this possible without loading everything or i should write a customised version of generated unmarshal method?
Regards
Tamer
The order for struct equality could be improved. Also for C and Go the ordering of the fields is relevant (memory alignment).
Seems to be more convenient for the Java user.
Hey again,
While porting the code to Rust, I've come accross a very weird logic difference in two (functionally) identical sanity checks in generated Go code.
Here and here it checks if l
is not overflowing ColferMaxSize
, which seems logical and reasonable.
But a bit before (here) it performs the same check against x
which is only the length of the list of nested objects.
Shouldn't it check against l
instead like the others?
Maybe the generated test is not up to date and the bug has already been fixed, I have no idea, but in any case this might need a change.
Hi. I have a problem with generated JS code. I use Golang as a server and JS as client. Golang version makes valid marshal and unmarshal. JS version returns wrong packet after unmarshalling of packet from Golang server.
It's my code:
type update struct {
id uint16
x float32
y float32
rotation float32
DT float32
HP uint8
commandId uint32
maxBulletId uint16
}
type update_arr struct {
packetType uint8
players []update
}
Exactly update_arr packet works wrong in JS version.
On Golang server I have this data:
{
"id": 0,
"x": 100,
"y": 100,
"rotation": 0,
"DT": 0,
"HP": 100,
"commandId": 0,
"maxBulletId": 0
}
But on JS client:
{
"id": 0,
"x": 2.4262940954800533e35,
"y": -131072.03125,
"rotation": 0,
"DT": 0,
"HP": 100,
"commandId": 0,
"maxBulletId": 0
}
The schema could specify other defaults than the zero value. @magiconair
Hi, I work for a large mobile app company that is interested in replacing JSON with something faster. The one disappointing thing is that colfer is written in Go and JS it looks like, which we can't use for iOS or Android (without a fair amount of finagling at least), so are there any plans for it to get ported to C/C++? Also, are there any companies that use it? It would be great to see a list of some companies that use or even their experiences, as that would really sell the format. How much security auditing has occurred?
The numbers on https://github.com/eishay/jvm-serializers/wiki are tantalizing, so I'm curious to hear more :)
The Features section on the project page mentions that this serialization format is "Robust against malicious input".
Could it be a little more detailed than that? To what degree and how this robustness is achieved?
How can we integrate with Spring Rest Controllers
currently all "setValue(Value x)" methods return void , if they would return the object then code-chaining would become possible.
example current:
SomeClass sc = new SomeClass();
sc.setValue(42);
sc.setStringValue("string");
example builder pattern:
SomeClass sc = new SomeClass().setValue(42).setStringValue("string");
I am building a big/important project and I need a stable option for serialization. I am considering Msgpack and Protobuf as alternatives but Colfer has a nice schema and seems to be more performant. How stable/maintained is this project, would you say its production ready?
Thanks!
Hi,
colfer place the name of schema file to the comment it generates via Maven Plugin, I am using a Windows machine and colfer does not escape the "" in file path and java compiler then complains about 'illegal escape character', is there a way to disable this comment or let it the colfer escape it before putting in the comment?
Thx
Why is uint16's compressed and uncompressed flipping the behavior of (index | 0x80)? This flag sort of is the anti-pattern of what uint32 and uint64 do. It's a bit confusing.
This is Java-specific, I'm not sure how other generations work. When unmarshalling arrays (any list or byte array) a new array is always constructed. This needlessly creates garbage when the old array has the exact same size as the new array and can be directly re-used. This can be a common use-case for some applications (fixed size images or buffers, for example).
It'd be great to be able to install a colfer JS lib with npm
.
would you please refer golang/go#13560
The current spec is ambiguous. e.g.
"Data structures consist of zero or more field value definitions followed by a termination byte 0x7f."
Could you please write an EBNF spec? EBNF is unambiguous and uses less characters than descriptions. A good example is The Go Programming Language Specification.
Protobuf allows creation of Messages which can hold one element of a certain set.
message Message {
oneof value { // <-- creates field and enum with {A, B}
MsgA A = 1;
MsgB B = 2;
}
}
Message MsgA {
....
}
Message MsgB {
....
}
which allows a switch over the generated enum-field:
Message m = Message.parse(inputstream);
switch(m.getValueCase())
{
case A: m.getA(); ...
case B: m.getB(); ...
}
which is good, because java checkstyles can issue warnings when not all enum fields are checked in a switch-statement
from my understanding colfer supports this by doing something like:
type Message struct {
head text
body binary
}
type A struct {
...
}
and by constructing java objects like this:
A a = new A(); a.set(....)....;
byte[] b = new byte[1024];
a.marshal(b,0);
Message m = new Message(); m.setHead(A.class.getSimpleName()); m.setBody(b)
which in return allows switching over the class-string
switch(t.getHead()) {
case "A": new A().unmarschal(t.getBody())
}
but i find the creation of these java object a little cumbersome, just so i can easily switch over them
@Colfer
could be used to identify pojo classes. Then the annotation processor would scan the class and generate the code for serialization / deserialization.
This is the most popular java way to do it. Annotation processors are a standard part of the java compiler.
Colf has a great potential on Android. And annotation processors are very common in Android builds.
Hello,
I wanted to ask about rust support.
Considering how such code is usually generated with macros, should we just create a crate that just implements the protocol/ser/deser logic?
Also, conforming with Serde to get automatic ser/deser of rust struct
would be a very nice addition as well.
Cheers,
Mathieu
Would love to see how the built-in JSON and Protobuf fare against colfer. Its kinda unfair to see C, Go, and Java make it to the benchmark games and JS stays at home. Could it be that the JS version couldn't live up to the Colfer performance promise?
I believe uint32/uint64 list is quite common and useful, will colfer support it in the future?
Hi, I'm Yevgeny Pats Founder of Fuzzit - Continuous fuzzing as a service platform.
I saw that you implemented Fuzz targets but they are currently not running as part of the CI.
We have a free plan for OSS and I would be happy to contribute a PR if that's interesting.
The PR will include the following
You can see our basic example here and you can see an example of "in the wild" integration here.
Let me know if this is something worth working on.
Cheers,
Yevgeny
The IDL is so similar to Go. Why not just use a subset of Go directly? e.g.
I wish it had been documented that a colf type cannot have more than 126 fields, ever, and this includes "deleted" fields.
(Why 126? "Each definition starts with an 8-bit header. The 7 least significant bits identify the field by its (0-based position) index in the schema." and 127 is reserved for the termination byte.)
Related, I wish https://github.com/pascaldekloe/colfer/#compatibility had explicitly said "Fields can never be removed."
During the marshalling this part throws the NPE
if (!this.property.isEmpty()) {
It would be great if this could be supported. Right now I have to serialise extra fields to rematerialize it correctly.
I think, unless it is very hard, that I can give it a stab if you point me in the right direction.
For most languages this prevents memory allocation as the content can be embedded into the struct. There are many good uses such as IPv6 addresses, UUIDs and binary signatures.
It would be nice to serialize maps, maybe as a simple 2D array ?
I tried different things. How can I set the package name to e.g. "com.cedarsoft.test.colfer"?
Maybe I missed something, but it looks like a bug:
In Java, in generated class there is writeObject(ObjectOutputStream out)
method which uses buf
array of size 1024 by default. In case of java.nio.BufferUnderflowException
size of it is enlarged 4 times.
But in case when buf
array is too small to handle marshalled object, java.nio.BufferOverflowException
is thrown by marshal(byte[] buf, int offset)
method, not java.nio.BufferUnderflowException
, so buf
is not enlarged and writeObject
ends with exception.
I tried it with the downloaded compiler and this colf file:
package asdf
type PrimArraysColfer struct {
id []uint64
time []uint64
valueInteger []int32
valueInteger64 []int64
valueFloat []float32
valueDouble []float64
}
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.