Git Product home page Git Product logo

google / flatbuffers Goto Github PK

View Code? Open in Web Editor NEW
21.9K 639.0 3.1K 17.63 MB

FlatBuffers: Memory Efficient Serialization Library

Home Page: https://flatbuffers.dev/

License: Apache License 2.0

CMake 0.65% Shell 0.28% Makefile 0.01% C++ 39.60% Go 1.49% Java 6.27% C# 5.99% Python 6.99% Batchfile 0.02% JavaScript 5.67% PHP 1.90% TypeScript 3.66% C 0.01% Dart 4.16% Lua 0.98% Rust 10.70% Kotlin 4.53% Roff 0.01% Swift 6.61% Starlark 0.48%
flatbuffers serialization serialization-library json-parser marshalling rpc zero-copy mmap cross-platform c-plus-plus

flatbuffers's Introduction

logo FlatBuffers

Build status BuildKite status Fuzzing Status OpenSSF Scorecard Join the chat at https://gitter.im/google/flatbuffers Discord Chat Twitter Follow Twitter Follow

FlatBuffers is a cross platform serialization library architected for maximum memory efficiency. It allows you to directly access serialized data without parsing/unpacking it first, while still having great forwards/backwards compatibility.

Quick Start

  1. Build the compiler for flatbuffers (flatc)

    Use cmake to create the build files for your platform and then perform the compliation (Linux example).

    cmake -G "Unix Makefiles"
    make -j
    
  2. Define your flatbuffer schema (.fbs)

    Write the schema to define the data you want to serialize. See monster.fbs for an example.

  3. Generate code for your language(s)

    Use the flatc compiler to take your schema and generate language-specific code:

    ./flatc --cpp --rust monster.fbs
    

    Which generates monster_generated.h and monster_generated.rs files.

  4. Serialize data

    Use the generated code, as well as the FlatBufferBuilder to construct your serialized buffer. (C++ example)

  5. Transmit/store/save Buffer

    Use your serialized buffer however you want. Send it to someone, save it for later, etc...

  6. Read the data

    Use the generated accessors to read the data from the serialized buffer.

    It doesn't need to be the same language/schema version, FlatBuffers ensures the data is readable across languages and schema versions. See the Rust example reading the data written by C++.

Documentation

Go to our landing page to browse our documentation.

Supported operating systems

  • Windows
  • macOS
  • Linux
  • Android
  • And any others with a recent C++ compiler (C++ 11 and newer)

Supported programming languages

Code generation and runtime libraries for many popular languages.

  1. C
  2. C++ - snapcraft.io
  3. C# - nuget.org
  4. Dart - pub.dev
  5. Go - go.dev
  6. Java - Maven
  7. JavaScript - NPM
  8. Kotlin
  9. Lobster
  10. Lua
  11. PHP
  12. Python - PyPI
  13. Rust - crates.io
  14. Swift - swiftpackageindex
  15. TypeScript - NPM
  16. Nim

Versioning

FlatBuffers does not follow traditional SemVer versioning (see rationale) but rather uses a format of the date of the release.

Contribution

To contribute to this project, see CONTRIBUTING.

Community

Security

Please see our Security Policy for reporting vulnerabilities.

Licensing

Flatbuffers is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.


flatbuffers's People

Contributors

aardappel avatar aeneid avatar austinschuh avatar bjornharrtell avatar caspern avatar dbaileychess avatar dependabot[bot] avatar enum-class avatar fbenkstein avatar iceboy233 avatar jkuszmaul avatar jonsimantov avatar jsanmiya avatar krojew avatar le-michael avatar maxburke avatar mustiikhalil avatar parnic avatar paulovap avatar pjulien avatar rgilles avatar rw avatar schoetbi avatar sssooonnnggg avatar stefan301 avatar stewartmiles avatar tgishib avatar tira-misu avatar vaind avatar vglavnyy avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

flatbuffers's Issues

Publish Benchmarks for Java and Android VMs

Currently, there are only impressive numbers available for the C++ implementation. The Java API is different in a couple of ways, so it would be very interesting how it performs on popular VMs, e.g. Java 7 and Android 4.4.

C++ GenerateText incurs a lot of overhead for outputs larger as 1024 bytes

std::string is used as a accumulator for the output string in GenerateText(...) in idl_gen_text.cpp. Each time one byte is added, the entire string is copied in a new container. At least internally a better way of handling this is probably to use std::ostream &_text as argument for GenStruct (in idl_gen_text.cpp), and wrap in GenerateText the call to GenStruct with

std::stringstream stringaccumulator;
GenStruct(..., stringaccumulator);
_text = stringaccumulator.str();

Although text.reserve(1024); is a nice trick for micro benchmarks, this output method will incur unexpected large execution times for larger outputs.

Another benefit of using std::ostream as function signature is should be rather easy to change the output routine so accept calls like GenerateText(..., std::cout) (avoiding composing the entire buffer in memory).

Table should extend Struct and not Constants in Java

I don't think there is a relation between the Table and the Constants classes, and it should not extend it, If it is just to use the constants then use:
import static flatbuffers.Constants.*;
However there is strong relation between Table and Struct. Table should extend Struct.
Struct has bb, bb_pos and Table has them also with additional function, and in fact a Table is a variable length Struct, so there is a relation between them.
If Table extends Struct then we will be able to add Struct in union also.

Support nested tables and structs in Java as follows

Lets say I have two Java peers that exchange message like these :
table Challenge{x:int;y:int;token:int;}
struct Point{x:int;y:int;}
If a message arrives I have to know its type first, so I want to be able to warp them in:
table Message{type:int;content:nested;}
(the compiler should generate fieldNameType and fieldNameOffset for nested fields and TYPE constant for tables and structs)

and use the following code to write them:

        FlatBufferBuilder fbb = new FlatBufferBuilder(100);
        Challenge.startChallenge(fbb);
        Challenge.addX(fbb, x);
        Challenge.addY(fbb, y);
        Challenge.addToken(fbb, t);
        int c = Challenge.endChallenge(fbb);
        Message.startMessage(fbb);
        Message.addContentType(fbb, Challeng.type); 
        Message.addContentOffset(fbb, c);
        int m = Message.endMessage(fbb);
        fbb.finish(im);

and I want to be able to parse it follows:

        Message msg=Message.getRootAsMessage(bb, 0);
        int c=msg.contentOffset();
        int t=msg.contentType();
        switch(t){
           case Point.TYPE:
                Point p=Point.getRootAsPoint(bb, c);
                processPoint(p);
                break;
           case Challenge.TYPE:
                Challenge ch = Challenge.getRootAsChallenge(bb, c)
                processChallenge(ch);
                break;
        }

I hope that my idea is clear and useful.

Don't just blindly double the buffer size

In the Java growByteBuffer, you blindly double the array size. This is a bug waiting to happen for some poor soul who overflows int.
One option is to investigate the internals of java.util.HashMap to see a nice way to grow arrays without overflowing. If nothing else, at least try to fail gracefully in this case.

Java code generation for monster.fbs causes segmentation fault.

how to reproduction:

Currently, flatc does not seem to understand relative paths. So, I've copied monster.fbs from sample to source root. Then, I executed flatc as follows:

hyunsik@Hyunsiks-MacBook-Pro:flatbuffers$ ./flatc -j monster.fbs 
Segmentation fault: 11

LLDB debugging result:

(lldb) run -j monster.fbs
Process 85655 launched: './flatc' (x86_64)
Process 85655 stopped
* thread #1: tid = 0x101ba30, 0x00000001000730f2 flatc`flatbuffers::java::GenStructArgs(struct_def=0x0000000100204380, code_ptr=0x00007fff5fbfe5b8, nameprefix=0x00007fff5f4006f1) + 530 at idl_gen_java.cpp:127, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x7fff5f3fffa8)
    frame #0: 0x00000001000730f2 flatc`flatbuffers::java::GenStructArgs(struct_def=0x0000000100204380, code_ptr=0x00007fff5fbfe5b8, nameprefix=0x00007fff5f4006f1) + 530 at idl_gen_java.cpp:127
   124         it != struct_def.fields.vec.end();
   125         ++it) {
   126      auto &field = **it;
-> 127      if (IsStruct(field.value.type)) {
   128        // Generate arguments for a struct inside a struct. To ensure names
   129        // don't clash, and to make it obvious these arguments are constructing
   130        // a nested struct, prefix the name with the struct name.
(lldb) p field.value.type
(flatbuffers::Type) $0 = {
  base_type = BASE_TYPE_FLOAT
  element = BASE_TYPE_NONE
  struct_def = 0x0000000000000000
  enum_def = 0x0000000000000000
}

go language support

anyone thinking about go language support or know where one is baking ?

this seems like an excellent way to pass data on a grid of servers.

Allow parsing json with quote

Currently, GeneratorOptions.strict_json allows serializing fields with quotes.
Parser.Parse fails when the json source contains fields with quotes.

file based database

this is a bit of a crazy idea perhaps :)

thinking about how to use flat buffers as a simple database.
As a database, you can ONLY search against pre indexed fields, so you woudl be reading the actual indexes, and then scooping up an object from the FB database which is basically just a file system with locks.
On saving a index would need to be updated. that is orthagonal to the main fB code.
On searching a map reduce style hadoop or the apache storm, and any other map reduce type system.
locking can easily be handled in various ways.

I am wondering though is the properties of flat buffers pay off for read and write to file systems ?

Multiple build failures with recent commits

I am using a vanilla version of gcc 4.8.3 and recent commits to flatbuffers have caused builds to fail:

First off, commit ebac1e1 caused this build failure:

[  7%] Building CXX object CMakeFiles/flatc.dir/src/idl_parser.cpp.o
In file included from /home/amehta/workspace/external/flatbuffers/src/idl_parser.cpp:21:0:
/home/amehta/workspace/external/flatbuffers/include/flatbuffers/util.h: In function ‘int flatbuffers::ToUTF8(uint32_t, std::string*)’:
/home/amehta/workspace/external/flatbuffers/include/flatbuffers/util.h:157:29: error: comparison between signed and unsigned integer expressions [-Werror=sign-compare]
     if (ucc < (1 << max_bits)) {  // does it fit?
                             ^
cc1plus: all warnings being treated as errors

This is pretty easy to fix if we cast the comparison explicitly, ie.

  if (ucc < static_cast<uint32_t>(1 << max_bits)) {  // does it fit?

However, it is not clear to me why the original code failed nor why this cast would fix it. That is why I didn't create a pull request with the patch.

Secondly, commit 11b7436 caused this failure:

[amehta@amehta-ld1 build]$ make
Scanning dependencies of target flatc
[  7%] Building CXX object CMakeFiles/flatc.dir/src/idl_parser.cpp.o
[ 15%] Building CXX object CMakeFiles/flatc.dir/src/idl_gen_cpp.cpp.o
[ 23%] Building CXX object CMakeFiles/flatc.dir/src/idl_gen_java.cpp.o
[ 30%] Building CXX object CMakeFiles/flatc.dir/src/idl_gen_go.cpp.o
[ 38%] Building CXX object CMakeFiles/flatc.dir/src/idl_gen_text.cpp.o
[ 46%] Building CXX object CMakeFiles/flatc.dir/src/flatc.cpp.o
Linking CXX executable flatc
[ 46%] Built target flatc
Scanning dependencies of target flatsamplebinary
[ 53%] Building CXX object CMakeFiles/flatsamplebinary.dir/samples/sample_binary.cpp.o
In file included from /home/amehta/workspace/external/flatbuffers/samples/sample_binary.cpp:19:0:
/home/amehta/workspace/external/flatbuffers/samples/monster_generated.h: In member function ‘bool MyGame::Sample::Monster::Verify(const flatbuffers::Verifier&) const’:
/home/amehta/workspace/external/flatbuffers/samples/monster_generated.h:65:32: error: ‘VerifyTable’ was not declared in this scope
     return VerifyTable(verifier) &&
                                ^
/home/amehta/workspace/external/flatbuffers/samples/monster_generated.h: In function ‘bool MyGame::Sample::VerifyAny(const flatbuffers::Verifier&, const void*, uint8_t)’:
/home/amehta/workspace/external/flatbuffers/samples/monster_generated.h:111:95: error: no matching function for call to ‘flatbuffers::Verifier::VerifyTable(const MyGame::Sample::Monster*) const’
     case Any_Monster: return verifier.VerifyTable(reinterpret_cast<const Monster *>(union_obj));
                                                                                               ^
/home/amehta/workspace/external/flatbuffers/samples/monster_generated.h:111:95: note: candidate is:
In file included from /home/amehta/workspace/external/flatbuffers/samples/sample_binary.cpp:17:0:
/home/amehta/workspace/external/flatbuffers/include/flatbuffers/flatbuffers.h:693:29: note: bool flatbuffers::Verifier::VerifyTable(const T*) [with T = MyGame::Sample::Monster] <near match>
   template<typename T> bool VerifyTable(const T *table) {
                             ^
/home/amehta/workspace/external/flatbuffers/include/flatbuffers/flatbuffers.h:693:29: note:   no known conversion for implicit ‘this’ parameter from ‘const flatbuffers::Verifier*’ to ‘flatbuffers::Verifier*’
In file included from /home/amehta/workspace/external/flatbuffers/samples/sample_binary.cpp:19:0:
/home/amehta/workspace/external/flatbuffers/samples/monster_generated.h: In function ‘bool MyGame::Sample::VerifyMonsterBuffer(const flatbuffers::Verifier&)’:
/home/amehta/workspace/external/flatbuffers/samples/monster_generated.h:118:112: error: no matching function for call to ‘flatbuffers::Verifier::VerifyBuffer() const’
 inline bool VerifyMonsterBuffer(const flatbuffers::Verifier &verifier) { return verifier.VerifyBuffer<Monster>(); }
                                                                                                                ^
/home/amehta/workspace/external/flatbuffers/samples/monster_generated.h:118:112: note: candidate is:
In file included from /home/amehta/workspace/external/flatbuffers/samples/sample_binary.cpp:17:0:
/home/amehta/workspace/external/flatbuffers/include/flatbuffers/flatbuffers.h:748:29: note: bool flatbuffers::Verifier::VerifyBuffer() [with T = MyGame::Sample::Monster] <near match>
   template<typename T> bool VerifyBuffer() {
                             ^
/home/amehta/workspace/external/flatbuffers/include/flatbuffers/flatbuffers.h:748:29: note:   no known conversion for implicit ‘this’ parameter from ‘const flatbuffers::Verifier*’ to ‘flatbuffers::Verifier*’
make[2]: *** [CMakeFiles/flatsamplebinary.dir/samples/sample_binary.cpp.o] Error 1
make[1]: *** [CMakeFiles/flatsamplebinary.dir/all] Error 2
make: *** [all] Error 2

I have not been able to successfully debug this. Any help would be appreciated!

Build error in Visual Stuio

I opend build/VS2010/FlatBuffers.sln in Visual Studio.
A release build fails because warnings are treated as error.
Visual Studio reports 2 warnings in samples/sample_binary.cpp because "inv" and "pos" are initialized but not referenced.

Version is 1.0.1

Why the need for a Root

What is the reason of using a root type. If i have Request and Response tables to be sent between two parties, so why we could not create them separately. A buffer could contain either a Request or a Response. In fact any message(Table) should be able to a root in a buffer.

Indices for non-linear look-ups

OK, this one is really for Version 2 or something in the distant future, but anyway...

The idea came while writing about FlatBuffers, so I just copy my post here:
My favorite feature of FlatBuffers is it's random access to parts of the data. If you have a big data file, it should be relatively efficient to access just some small parts of it. Of course, you need to have the complete byte buffer in memory, but besides that, there's only overhead for the data you actually look at. This makes it an interesting database format for read-only data. Hmm, if it also had indices for non-linear look-ups...

A Parser::Parse for json with bounds checking

Apart from:

Parser::Parse(const char *source, const char *filepath)

Consider:

Parser::ParseJson(const char *source, const char *end) // or size_t len

The former code waits for kTokenEof.
The latter would allow callers to provide json inside a buffer.
If the buffer is the same exact size as the json input, there would be no room to insert the null char.
Would be inefficient to allocate a new buffer + 1 just for the null delim.
This is in the context of taking json input from the network/web.

Json unicode escape support

Got "unknown escape code in string constant" on Parser.Parse

Under node repl, it works as expected:

JSON.parse('{"a":"\u0061"}')
{ a: 'a' }

Schemes in directories generate wrong include guard

Include guards do not strip directory seperators. Running flatc test/foo.fbs will generate the include guard FLATBUFFERS_GENERATED_TEST/FOO which doesn't compile.

I could just run the command in the directory test but that would make my build script a lot more complicated.

Java: convenient API for reading/writing primitive vectors

It'd be nice if accessing and creating byte vectors was as easy as it is for Strings. AFAICT, if you want to create a vector of bytes you have to iterate and add one at a time.

If the API below seems like a reasonable approach I will try to create a PR this weekend.

Creation:

Under the covers this would just delegate to a new method on FlatBufferBuilder that would use System.arraycopy to add the bytes.

byte[] data = getData();
Event.createDataVector(fbb, data);

Reading

Event e = Event.getRootAsEvent(bb);
byte[] data = e.dataVector();

IndexOutOfBoundsException on endObject in Java

I guess that putShort should be replaced by addShort?

java.lang.IndexOutOfBoundsException: null
    at java.nio.Buffer.checkIndex(Buffer.java:546) ~[na:1.8.0_05]
    at java.nio.HeapByteBuffer.putShort(HeapByteBuffer.java:330) ~[na:1.8.0_05]
    at flatbuffers.FlatBufferBuilder.putShort(FlatBufferBuilder.java:102) ~[classes/:na]
    at flatbuffers.FlatBufferBuilder.endObject(FlatBufferBuilder.java:195) ~[classes/:na]

support fixed length arrays within structs?

Feature request: fixed length arrays, especially allowed within structs.

If I want to store a 256-bit hash in a struct, I'd either need to convert to a table in order to use [ubyte], or else store 4 ulongs (which is ugly). Would be convenient if there was a way to store a fixed-length array in a struct.

CreateVectorOfStructures calls wrong function.

When calling the CreateVectorOfStructures function it then passes your std vector through the internal CreateVector function rather than the internal CreateVectorOfStructures one.

Original Code:

template Offset<Vector<const T *>> CreateVectorOfStructs(
const std::vector &v) {
return CreateVector(&v[0], v.size());
}

Changed to:

template Offset<Vector<const T *>> CreateVectorOfStructs(
const std::vector &v) {
return CreateVectorOfStructs(&v[0], v.size());
}

Add support for unknow legnth of vector

Sometimes we don't know the length of a vector before processing it, so it will be nice to move the num_elems from the startVector to the endVector function, like that we add the length after finishing the vector because at that moment the length should be known.
This is useful for example if we are reading vector data from a stream, or copying only positive elements of an array etc.

Creating an empty vector causes a segmentation fault

When you create an empty vector, the eventual call of CreateVector(const T *v, size_t len) leads to a segmentation fault. The error is caused by a do while loop that decrements i = len = 0 and tries to access the element at that position, although its condition is already false.

The do while loop should be changed to a while loop, or a comment should indicate a non-empty vector as a precondition for the CreateVector function family.

flattests.exe fails under Win8.1 with mingw

I have built the master I downloaded today and everything built fine with mingw-gcc 4.9-x64 under Windows 8.1-64. But when I run the flattests.exe it fails with a runtime crash and the messages below. Also I do not know how to run and use the other Exes: "flatsamplebinary.exe" and "flatsampletext.exe".

Fail Message:
TEST FAILED: C:\Users\xxx\Downloads\Serializer-Libs\FlatBuffers\flatbuffers-master\tests\test.cpp:155, flatbuffers::LoadFile( "tests/monster_test.fbs", false, &schemafile) (0) != 1
Assertion failed!

Program: C:\Users\xxx\Downloads\Serializer-Libs\FlatBuffers\flatbuffers-master\tests\flattests.exe
File: C:\Users\xxx\Downloads\Serializer-Libs\FlatBuffers\flatbuffers-master\tests\test.cpp, Line 45

Expression: 0

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

namespaces handled incorrectly when parsing multiple schema files

If I use flatc -c /.fbs to generate code for multiple schemas, each of which begins with "namespace foo.bar;", then each generated files gains another level of namespace nesting depending on where it fell in the command line.

The first file contains definitions in namespace foo::bar, the second in foo::bar::foo::bar, and so on.

createString must be placed in front of startMonster?

/*
 * Copyright 2014 Google Inc. All rights reserved.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

import Sample.Monster;
import Sample.Vec3;
import flatbuffers.FlatBufferBuilder;

import java.nio.ByteBuffer;

class JavaTest {
    public static void main(String[] args) {
        FlatBufferBuilder fbb = new FlatBufferBuilder(1024);


        //int str = fbb.createString("MyMonster");
        Monster.startMonster(fbb);
        Monster.addPos(fbb, Vec3.createVec3(fbb, 1.0f, 1.0f, 1.0f));
        Monster.addHp(fbb, (short) 80);

        Monster.addName(fbb, fbb.createString("MyMonster"));
        Monster.addInventory(fbb, 1);

        Monster.addColor(fbb, (byte) 255);
        int mon = Monster.endMonster(fbb);
        fbb.finish(mon);

        ByteBuffer data = fbb.dataBuffer();
        Monster monster = Monster.getRootAsMonster(data, fbb.dataStart());

        System.out.println(monster.color());
        System.out.println(monster.hp());
        System.out.println(monster.name());
        System.out.println(monster.pos());
    }
}
Exception in thread "main" java.lang.AssertionError: FlatBuffers: object serialization must not be nested.
    at flatbuffers.FlatBufferBuilder.notNested(FlatBufferBuilder.java:148)
    at flatbuffers.FlatBufferBuilder.startVector(FlatBufferBuilder.java:126)
    at flatbuffers.FlatBufferBuilder.createString(FlatBufferBuilder.java:139)
    at JavaTest.main(JavaTest.java:33)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)

bug: force_align for struct

if there a struct Vec3 in flatbuffer idl file monstor:
struct Vec3 (force_align : 8 ) {
x : float;
y : float;
z : float;
}
after run the command "flatc -c monster.fbs", i got the following codes:
MANUALLY_ALIGNED_STRUCT(8) Vec3 {
private:
float x_;
float y_;
float z_;

public:
Vec3(float x, float y, float z)
: x_(flatbuffers::EndianScalar(x)), y_(flatbuffers::EndianScalar(y)), z_(flatbuffers::EndianScalar(z)) {}

float x() const { return flatbuffers::EndianScalar(x_); }
float y() const { return flatbuffers::EndianScalar(y_); }
float z() const { return flatbuffers::EndianScalar(z_); }
};
STRUCT_END(Vec3, 12);

apparently, the size "12" is not right which should be "16". if i define a object "Vec3 v3", the gcc gives the following prompting “error: static assertion failed: compiler breaks packing rules”

waiting for the author checks this bug.

Errors trying to compile flatc

I'm trying to compile flatc.

From the root folder 'flatbuffers-master' I use the command "cmake ." which works.

Then I use "make".

Here I get an error message about multiple definitions of flatbuffer_version_string, being first defined in idl_gen_cpp.cpp

I'm new to this so any help would be appreciated. Is there not a flatc.exe I can just download instead of having to compile?

Regards.

Compile warnings with VS2013 (-W4)

flatbuffers.h yields several compile warnings under VS2013 Update 2 with warning level 4, both when used to compile flatc, as well as use a generated header file. Ignoring them when building flatc is less of an issue, but it'd be nice if including a generated header (which then includes flatbuffers.h) didn't force us to back off or ignore warnings.

For example, when just including a generated header (haven't actually used code from it yet):
...flatbuffers.h(410) : error C2220: warning treated as error - no 'object' file generated
...flatbuffers.h(410) : warning C4244: 'argument' : conversion from 'flatbuffers::uoffset_t' to 'flatbuffers::voffset_t', possible loss of data
...flatbuffers.h(419) : warning C4244: 'argument' : conversion from 'flatbuffers::uoffset_t' to 'flatbuffers::voffset_t', possible loss of data

And when compiling flatc (also some come from CPP files):
...flatbuffers\include\flatbuffers/flatbuffers.h(410) : error C2220: warning treated as error - no 'object' file generated (...flatbuffers\src\idl_gen_text.cpp)
...flatbuffers\include\flatbuffers/flatbuffers.h(410) : warning C4244: 'argument' : conversion from 'flatbuffers::uoffset_t' to 'flatbuffers::voffset_t', possible loss of data (...flatbuffers\src\idl_gen_text.cpp)
...flatbuffers\include\flatbuffers/flatbuffers.h(410) : error C2220: warning treated as error - no 'object' file generated (...flatbuffers\src\flatc.cpp)
...flatbuffers\include\flatbuffers/flatbuffers.h(410) : warning C4244: 'argument' : conversion from 'flatbuffers::uoffset_t' to 'flatbuffers::voffset_t', possible loss of data (...flatbuffers\src\flatc.cpp)
...flatbuffers\include\flatbuffers/flatbuffers.h(419) : warning C4244: 'argument' : conversion from 'flatbuffers::uoffset_t' to 'flatbuffers::voffset_t', possible loss of data (...flatbuffers\src\idl_gen_text.cpp)
...flatbuffers\include\flatbuffers/flatbuffers.h(419) : warning C4244: 'argument' : conversion from 'flatbuffers::uoffset_t' to 'flatbuffers::voffset_t', possible loss of data (...flatbuffers\src\flatc.cpp)
...flatbuffers\include\flatbuffers/flatbuffers.h(410) : error C2220: warning treated as error - no 'object' file generated (...flatbuffers\src\idl_gen_cpp.cpp)
...flatbuffers\include\flatbuffers/flatbuffers.h(410) : warning C4244: 'argument' : conversion from 'flatbuffers::uoffset_t' to 'flatbuffers::voffset_t', possible loss of data (...flatbuffers\src\idl_gen_cpp.cpp)
...flatbuffers\include\flatbuffers/flatbuffers.h(640) : warning C4100: 'other' : unreferenced formal parameter (...flatbuffers\src\idl_gen_text.cpp)
...flatbuffers\include\flatbuffers/flatbuffers.h(419) : warning C4244: 'argument' : conversion from 'flatbuffers::uoffset_t' to 'flatbuffers::voffset_t', possible loss of data (...flatbuffers\src\idl_gen_cpp.cpp)
...flatbuffers\include\flatbuffers/flatbuffers.h(640) : warning C4100: 'other' : unreferenced formal parameter (...flatbuffers\src\flatc.cpp)
...flatbuffers\include\flatbuffers/flatbuffers.h(652) : warning C4245: 'return' : conversion from 'int' to 'size_t', signed/unsigned mismatch (...flatbuffers\src\idl_gen_text.cpp)
...flatbuffers\include\flatbuffers/flatbuffers.h(652) : warning C4245: 'return' : conversion from 'int' to 'size_t', signed/unsigned mismatch (...flatbuffers\src\flatc.cpp)
...flatbuffers\include\flatbuffers/flatbuffers.h(640) : warning C4100: 'other' : unreferenced formal parameter (...flatbuffers\src\idl_gen_cpp.cpp)
...flatbuffers\include\flatbuffers/flatbuffers.h(652) : warning C4245: 'return' : conversion from 'int' to 'size_t', signed/unsigned mismatch (...flatbuffers\src\idl_gen_cpp.cpp)
...flatbuffers\include\flatbuffers/flatbuffers.h(410) : error C2220: warning treated as error - no 'object' file generated (...flatbuffers\src\idl_gen_java.cpp)
...flatbuffers\include\flatbuffers/flatbuffers.h(410) : warning C4244: 'argument' : conversion from 'flatbuffers::uoffset_t' to 'flatbuffers::voffset_t', possible loss of data (...flatbuffers\src\idl_gen_java.cpp)
...flatbuffers\include\flatbuffers/flatbuffers.h(419) : warning C4244: 'argument' : conversion from 'flatbuffers::uoffset_t' to 'flatbuffers::voffset_t', possible loss of data (...flatbuffers\src\idl_gen_java.cpp)
...flatbuffers\include\flatbuffers/flatbuffers.h(640) : warning C4100: 'other' : unreferenced formal parameter (...flatbuffers\src\idl_gen_java.cpp)
...flatbuffers\include\flatbuffers/flatbuffers.h(410) : error C2220: warning treated as error - no 'object' file generated (...flatbuffers\src\idl_parser.cpp)
...flatbuffers\include\flatbuffers/flatbuffers.h(652) : warning C4245: 'return' : conversion from 'int' to 'size_t', signed/unsigned mismatch (...flatbuffers\src\idl_gen_java.cpp)
...flatbuffers\include\flatbuffers/flatbuffers.h(410) : warning C4244: 'argument' : conversion from 'flatbuffers::uoffset_t' to 'flatbuffers::voffset_t', possible loss of data (...flatbuffers\src\idl_parser.cpp)
...flatbuffers\include\flatbuffers/flatbuffers.h(419) : warning C4244: 'argument' : conversion from 'flatbuffers::uoffset_t' to 'flatbuffers::voffset_t', possible loss of data (...flatbuffers\src\idl_parser.cpp)
...flatbuffers\include\flatbuffers/flatbuffers.h(640) : warning C4100: 'other' : unreferenced formal parameter (...flatbuffers\src\idl_parser.cpp)
...flatbuffers\include\flatbuffers/flatbuffers.h(652) : warning C4245: 'return' : conversion from 'int' to 'size_t', signed/unsigned mismatch (...flatbuffers\src\idl_parser.cpp)
...flatbuffers\src\idl_gen_text.cpp(157) : warning C4244: 'argument' : conversion from 'const int' to 'flatbuffers::voffset_t', possible loss of data
...flatbuffers\src\idl_gen_text.cpp(158) : warning C4244: 'argument' : conversion from 'const int' to 'flatbuffers::voffset_t', possible loss of data
...flatbuffers\src\idl_gen_text.cpp(176) : warning C4244: 'argument' : conversion from 'int' to 'flatbuffers::voffset_t', possible loss of data
...flatbuffers\src\idl_gen_text.cpp(204) : warning C4244: 'argument' : conversion from 'int' to 'flatbuffers::voffset_t', possible loss of data
...flatbuffers\src\idl_parser.cpp(105) : warning C4244: 'argument' : conversion from 'int' to 'char', possible loss of data
...flatbuffers\src\idl_gen_java.cpp(81) : warning C4244: 'argument' : conversion from 'int' to 'char', possible loss of data
...flatbuffers\src\idl_gen_java.cpp(82) : warning C4244: 'argument' : conversion from 'int' to 'char', possible loss of data
...flatbuffers\src\idl_parser.cpp(371) : warning C4244: 'argument' : conversion from 'const int' to 'flatbuffers::voffset_t', possible loss of data
...flatbuffers\src\idl_parser.cpp(417) : warning C4244: 'argument' : conversion from 'int' to 'flatbuffers::voffset_t', possible loss of data
...flatbuffers\src\idl_parser.cpp(429) : warning C4244: 'argument' : conversion from 'int' to 'flatbuffers::voffset_t', possible loss of data
...flatbuffers\src\idl_gen_java.cpp(376) : warning C4100: 'file_name' : unreferenced formal parameter
...flatbuffers\src\idl_gen_text.cpp(141) : warning C4244: 'argument' : conversion from 'const int' to 'flatbuffers::voffset_t', possible loss of data
...flatbuffers\src\idl_gen_text.cpp(191) : see reference to function template instantiation 'void flatbuffers::GenField<uint8_t>(const flatbuffers::FieldDef &,const flatbuffers::Table *,bool,int,int,std::string *)' being compiled
...flatbuffers\include\flatbuffers/util.h(35) : warning C4127: conditional expression is constant (...flatbuffers\src\idl_gen_cpp.cpp)
...flatbuffers\src\idl_gen_cpp.cpp(96) : see reference to function template instantiation 'std::string flatbuffers::NumToString(T)' being compiled
with
[
T=int
]
...flatbuffers\include\flatbuffers/util.h(35) : warning C4127: conditional expression is constant (...flatbuffers\src\idl_gen_java.cpp)
...flatbuffers\src\idl_gen_java.cpp(106) : see reference to function template instantiation 'std::string flatbuffers::NumToString(T)' being compiled
with
[
T=int
]
...flatbuffers\src\idl_gen_text.cpp(37) : warning C4100: 'indent_step' : unreferenced formal parameter
...flatbuffers\src\idl_gen_text.cpp(60) : see reference to function template instantiation 'void flatbuffers::Print(T,flatbuffers::Type,int,int,flatbuffers::StructDef *,std::string *)' being compiled
with
[
T=unsigned char
]
...flatbuffers\src\idl_gen_text.cpp(126) : see reference to function template instantiation 'void flatbuffers::PrintVector<uint8_t>(const flatbuffers::Vector<uint8_t> &,flatbuffers::Type,int,int,std::string *)' being compiled
...flatbuffers\src\idl_gen_text.cpp(37) : warning C4100: 'indent' : unreferenced formal parameter
...flatbuffers\src\idl_gen_text.cpp(37) : warning C4100: 'type' : unreferenced formal parameter
...flatbuffers\include\flatbuffers/util.h(35) : warning C4127: conditional expression is constant (...flatbuffers\src\idl_parser.cpp)
...flatbuffers\src\idl_parser.cpp(48) : see reference to function template instantiation 'std::string flatbuffers::NumToString<size_t>(T)' being compiled
with
[
T=size_t
]
...flatbuffers\include\flatbuffers/util.h(35) : warning C4127: conditional expression is constant (...flatbuffers\src\idl_gen_text.cpp)
...flatbuffers\src\idl_gen_text.cpp(40) : see reference to function template instantiation 'std::string flatbuffers::NumToString(T)' being compiled
with
[
T=unsigned char
]
...flatbuffers\src\idl_gen_text.cpp(60) : see reference to function template instantiation 'void flatbuffers::Print(T,flatbuffers::Type,int,int,flatbuffers::StructDef *,std::string *)' being compiled
with
[
T=unsigned char
]

Explain why C++ and Java serializations differ (in test suite)

The C++-generated and Java-generated serializations in the tests/ directory differ. Is this explained anywhere? If not, I don't see how to tell whether it's a bug or expected behavior.

More details:

$ diff tests/monsterdata_test_wire.bin tests/monsterdata_java_wire.bin
Binary files tests/monsterdata_test_wire.bin and tests/monsterdata_java_wire.bin differ
$ xxd tests/monsterdata_test_wire.bin
0000000: 2800 0000 0000 0000 0000 0000 1c00 3c00  (.............<.
0000010: 0800 0000 0600 2c00 0000 3000 0000 0500  ......,...0.....
0000020: 3400 3800 0000 0000 1c00 0000 0001 5000  4.8...........P.
0000030: 0000 803f 0000 0040 0000 4040 0000 0000  ...?...@..@@....
0000040: 0000 0000 0000 0840 0400 0500 0600 0000  .......@........
0000050: 0000 0000 4c00 0000 3c00 0000 3000 0000  ....L...<...0...
0000060: 0400 0000 0200 0000 0a00 1400 1e00 2800  ..............(.
0000070: 1c00 0800 0000 0000 0600 0000 0000 0000  ................
0000080: 0000 0000 0000 0000 0000 0000 1c00 0000  ................
0000090: 0000 1400 0500 0000 0001 0203 0400 0000  ................
00000a0: 0900 0000 4d79 4d6f 6e73 7465 7200 0000  ....MyMonster...
$ xxd tests/monsterdata_java_wire.bin
0000000: 2400 0000 0000 0000 1c00 4400 1c00 0000  $.........D.....
0000010: 1a00 1400 0000 1000 0000 0f00 0800 0400  ................
0000020: 0000 0000 1c00 0000 4000 0000 6400 0000  [email protected]...
0000030: 0000 0001 6300 0000 6800 0000 0000 5000  ....c...h.....P.
0000040: 0000 803f 0000 0040 0000 4040 0000 0000  ...?...@..@@....
0000050: 0000 0000 0000 0840 0400 0500 0600 0000  .......@........
0000060: 0000 0000 0000 0000 0200 0000 1e00 2800  ..............(.
0000070: 0a00 1400 1c00 0700 0000 0000 0400 0000  ................
0000080: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000090: 1c00 0000 1400 0005 0000 0000 0102 0304  ................
00000a0: 0900 0000 4d79 4d6f 6e73 7465 7200 0000  ....MyMonster...

flat buffers and Javascript

i am writing lots of webgl code

i need to transport lots of data to visualise point clouds and objects.

with web gl. Collada and gltf are used to represent the data.
Collada is xml and so highly verbose
GLTF is much more compact, and the current king of 3d file format compression, without loosing ability to manipulation the data on the client.

Obviously i would need to write a data mapper from collada to FB.

Would it make sense to have flat buffers work i javascript ?

Constants cause integer overflow (Test suite)

In lines 227 and 229 of tests/test.cpp, constants are declared that overflow their types. The int32 code is:

const int32_t  int_val    = 0x83333333;

and the int64 code is:

const int64_t  long_val   = 0x8444444444444444;

The two values overflow. For reference, see these calculations:

int32_t: (2**31 - 1) - 0x83333333 = -53687092
int64_t: (2**63 - 1) - 0x8444444444444444 = -307445734561825861

Source:
https://github.com/google/flatbuffers/blob/master/tests/test.cpp#L227-L229

Lua support in roadmap?

Beacuse our game develop with lua,But I don't how to binding it with lua.Is there any paln for lua?

Why flatbuffers instead of capnp?

You mention memory efficiency as the main selling point of flatbuffers. However, capnp already exists, with zero copy "serialization," and a range of compelling features (RPC, etc). Still you saw the need to build flatbuffers, which means this library either (a) does something capnp doesn't, or (b) beats capnp in some respect. Could you please describe somewhere what this rationale is? Why should someone pick flatbuffers instead of capnp?

Why c++ generated header in sample skip field id 12?

New to this amazing lib, I'm very curious about the generated header of monster.fbs
add_name calls AddOffset(10, name);
add_inventory calls AddOffset(14, inventory);

I manually changed 14 to 12, and next color field from 16 to 12, everything works jsut fine!
and from the source code I read, AddOffset just PushElement of length sizeof(voffset_t), so there shouldn't be any problem with that.
And I print all the voffset_t in vtable offsets, field 12 is 0.
Forgive me if I missing something really obviously...

Name conflicts for "vector length" and "union type" generated functions

Given a schema like this:

namespace Foo;

table A {
  BLength:int;
  B:[int];
} 

root_type A;

Generated function names in Java include:

public int BLength() { int o = __offset(4); return o != 0 ? bb.getInt(o + bb_pos) : 0; }
public int BLength() { int o = __offset(6); return o != 0 ? __vector_len(o) : 0; }

The first function is for fetching the value of the key "BLength", and the second is for fetching the length of the vector "B". These names conflict and thus are invalid Java code.

Given a schema like this:

namespace Foo;

table T {
}

union U { T }

table A {
  BType:int;
  B:U;
} 

root_type A;

Generated function names in Java include:

public int BType() { int o = __offset(4); return o != 0 ? bb.getInt(o + bb_pos) : 0; }
public byte BType() { int o = __offset(6); return o != 0 ? bb.get(o + bb_pos) : 0; }

The first function is for fetching the value of the key "BType", and the second is for fetching the type of the union "B". These names conflict and thus are invalid Java code.

The problem is also present in the WIP Go port discussed in issue #36.

C++ Vector should have size

To geht the length of for example std::vector one has to call the size() function.
The template class Vector should have a function with the same name, calling Length() looks strange.

add text output from binary input

The help message from flatc may lead one to expect that the "-t" option will parse a binary flatbuffer file and produce text (the inverse of -b). This is not the case - it seems to parse text and produce text as output. Please provide an option to parse binary and product text, so that it is easy to decode binary blobs produced in flatbuffer format.

Build error on windows (MSVC Compiler)

CMakeLists.txt file is trying to set -Werror and -Wextra compiler flags. These flags do not exist in MSVC. These flags should be set in case of MSVC. The workaround I did was to replace the following line:

set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -pedantic -Werror -Wextra")

with:

if(CMAKE_COMPILER_IS_GNUCXX)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -pedantic -Werror -Wextra")
endif(CMAKE_COMPILER_IS_GNUCXX)

It started compiling after that on windows, I have not tried compiling it on Linux though with these changes.

Thanks,
Zarian.

Fail early when json fields appear more than once

Currently, when a json field appears more than once, we get an assertion failure in FlatBufferBuilder.EndTable (line 501).

Here's my workaround:
dyu@01a0bc5

Not sending it as a PR as it is not a complete solution (limits the table to 64 fields for it to work).

Add support for a vector to be returned as a ByteBuffer

A vector (of byte,int,short,char) in a Table should be returned as a (Byte,Int,Short,Char)Buffer in addition of returning it element by element. The new ByteBuffer will share the same backing array with the original one but with different start and limit positions, like that we can process it separately or send it to the networking without additional bytes. The code could be something like:

public ByteBuffer inventoryAsBB(){int o = __offset(14); bb.position(__vector(o));return o != 0 ? (ByteBuffer)bb.slice().limit(__vector_len(o)) : null;}

This method might be also useful to use in getRootAs if the vectore data is a nested table

ByteBuffer bb=rootMonster.enemyAsBB();
Monter enemy=Monster.getRootAsMonster(bb,0);

Higher level API for creating flat buffers

With non-trivial schemas, it becomes cumbersome to create a flat buffer. The main reasons are:

  • Objects may not be nested
  • Offsets need to be stored for later reference

For deeper data structures this process is not straight forward, because you cannot just traverse your object tree just once. You have to do it multiple times and add data objects to store the offset information.

It would be cool to have this (boiler plate) code generated. Managing offsets will become an internal implementation detail, the developer does not need to worry about. By having some intermediate data structures it should be possible to allow nested creation of object hierarchies. The actual creation of the low level flat buffer might be deferred until all data is complete.

This higher level API could built up on and coexist with the existing API.

ByteBuffer pooling and reuse

I want to use flatbuffers with a ByteBuffer pooler or reuse an existing ByteBuffer, so it will be nice if add a constructor to the FlatBufferBuilder that accepts a ByteBuffer as parameter.

scientiffic notation in json

Scientiffic notation is not handled in json.
Here is an example:

flatc.exe -b monster.fbs monsterdata.json

monsterdata.json:

{
  pos: {
    x: 0.001,
    y: 2,
    z: 3
  },
  hp: 80,
  name: "MyMonster"
}

The abowe works, but this doesn't:

flatc.exe -b monster.fbs monsterdata.json
flatc.exe: line 3: expecting: , instead got: ♦

monsterdata.json:

{
  pos: {
    x: 1e-3,
    y: 2,
    z: 3
  },
  hp: 80,
  name: "MyMonster"
}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.