wc-duck / datalibrary Goto Github PK
View Code? Open in Web Editor NEWOpen Source Data Library for data serialization.
License: Other
Open Source Data Library for data serialization.
License: Other
If I try to feed dl with an instance generated from an old header-file (mismatch sizes/members etc), then DL does not generate an error. Instead it freezes. Some sort of version-tagging would be nice.
"members" : [
{ "name" : "mystuff", "type" : "previously_declared_struct_type*[10]" },
]
generates the following code:
const struct previously_declared_struct_type* mystuff;
Is this really intended? I would assume that we would get this:
struct previously_declared_struct_type* mystuff[10];
Otherwise it cleans / overwrites the output file even if the input file is corrupt / not possible to parse.
I have the following type library and data file
https://gist.github.com/joeldevahl/91a6c4a3568531cc3053
and doing a dl_pack on cube.mesh with the compiled type library gives me a crash in dl_internal_store_string in the strlen.
datalibrary is at commit a59f48e
Currently all instances in the text-format is specified as structs. This is the best solution in most cases but for example vec3:s etc is really cumbersome.
Today that would be:
{ "x" : 1, "y" : 2, "z" : 3 }
Try that with a mat4x4 and you will just give up ;)
By adding support for parsing structs from arrays one can get a lot nicer.
[ 1, 2, 3 ] or mat4x4
[ 1, 1, 1, 1,
1, 1, 1, 1,
1, 1, 1, 1,
1, 1, 1, 1 ]
This should be possible by just adding the rules
if an array is encountered where a struct is expected the array will be parsed as if the members where in order in the array and all members need to be set.
The following json:
{
"module" : "test",
"types" : {
"test_type" : {
"members" : []
},
"test_type2" : {
"members" : [
{ "name" : "test_instance", "type" : "test_type[]" }
]
}
}
}
Seems to generate the following struct declarations:
struct test_type
{
#if defined( __cplusplus )
static const uint32_t TYPE_ID = 0xE077D27A;
#endif // defined( __cplusplus )
};
struct test_type2
{
#if defined( __cplusplus )
static const uint32_t TYPE_ID = 0xFBE19D9B;
#endif // defined( __cplusplus )
struct
{
test_type* data;
uint32_t count;
#if defined( __cplusplus )
test_type& operator[] ( size_t index ) { return data[index]; }
const test_type& operator[] ( size_t index ) const { return data[index]; }
#endif // defined( __cplusplus )
} test_instance;
};
Where the type-pointer in the sub-struct for the array-data says "test_type* data" which should probably be "struct test_type* data;". This does not compile in c. It does compile in c++
I have the following types:
"dl_material_mapping" :
{
"members" :
[
{ "name" : "material", "type" : "string" },
{ "name" : "submesh", "type": "int8" }
]
},
"dl_model" :
{
"members" :
[
{ "name" : "mesh", "type": "string" },
{ "name" : "materials", "type": "dl_material_mapping[]"}
]
}
Now when I pack the following data file dl_pack.exe crashes.
{
"type": "dl_model",
"data": {
"mesh": "1234",
"materials": [
{
"submesh": 2,
"material": "apan.ymesjh"
}
]
}
}
If I change the line
"mesh": "meshname",
to
"mesh": "meshna",
it doesn't crash.
break the file into smaller parts.
The current reflection interface is far from good and should be redone. Today the interface forces the user to manage memory by them self when a simpler interface would work just as well.
Example:
dl_reflect_type( dl_ctx_t ctx, int type_index, dl_type_info* info );
Add support for multi-dimensional arrays such as "uint32[][]", "uint32[5][]" and uint32[][6]".
Prerequisite is to rewrite json-parse since that would simplify this feature ALOT!
Extra , or missing , for example.
{
"module" : "example",
"types" : {
"vec3" : {
"members" : [
{ "name" : "x", "type" : "fp32" },
]
}
}
}
Yields
Traceback (most recent call last):
File "src/external/dl/tool/dl_tlc/dl_tlc.py", line 91, in <module>
tl = dl.typelibrary.TypeLibrary( options.input )
File "/home/kma/projects/demo13/src/external/dl/tool/dl_tlc/dl/typelibrary.py", line 313, in __init__
self.read( open(lib, 'r').read() )
File "/home/kma/projects/demo13/src/external/dl/tool/dl_tlc/dl/typelibrary.py", line 324, in read
self.from_text( lib )
File "/home/kma/projects/demo13/src/external/dl/tool/dl_tlc/dl/typelibrary.py", line 398, in from_text
data = json.loads( re.sub(pattern, replacer, lib ) )
File "/usr/lib/python2.7/json/__init__.py", line 326, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
It would be nice if DL would support data deduplication when serializing as a user flag when the user does not care about data being unique.
References to other values in enums, such as:
enum A {
B
};
enum C {
D = B,
};
Same goes for dl_util_store_to_file etc.
Investigate if it is a good idea to, when compiling as c++, add, optional to use, accessors for union types to get and set members.
something like:
enum my_union_type
{
my_union_type_item1 = ...;
my_union_type_item2 = ...;
};
struct my_union
{
union
{
int32_t item1;
struct apa item2;
} value;
my_union_type_item1 type;
// things above is already generated.
int32_t& item1()
{
ASSERT( type == my_union_type_item1 );
return value.item1;
}
int32_t& as_item1()
{
type = my_union_type_item1;
return value.item1;
}
struct apa& item2()
{
ASSERT( type == my_union_type_item2 );
return value.item2;
}
int32_t& as_item2()
{
type = my_union_type_item2;
return value.item2;
}
};
Biggest questions:
such as this:
enum MyTestEnum : uint8_t {
MY_TEST_ENUM_VAL = 1,
};
{
"module" : "test",
"types" : {
"mytype_1" : {
"members" : [
{ "name" : "submember1", "type" : "string[]" },
{ "name" : "submember2", "type" : "uint32" }
]
},
"mytype_2" : {
"members" : [
{ "name" : "member1", "type" : "mytype_1", "default" : { "submember1" : [ "apa" ], "submember2":1337 }}
]
}
}
}
Should work but the string-array for submember1 seems to be set incorrectly.
{
"module" : "example",
"types" : {
"my_type" : {
"members" : [
{ "name" : "x", "type" : "float" }
]
}
}
}
Yields
Traceback (most recent call last):
File "src/external/dl/tool/dl_tlc/dl_tlc.py", line 91, in <module>
tl = dl.typelibrary.TypeLibrary( options.input )
File "/home/kma/projects/demo13/src/external/dl/tool/dl_tlc/dl/typelibrary.py", line 313, in __init__
self.read( open(lib, 'r').read() )
File "/home/kma/projects/demo13/src/external/dl/tool/dl_tlc/dl/typelibrary.py", line 324, in read
self.from_text( lib )
File "/home/kma/projects/demo13/src/external/dl/tool/dl_tlc/dl/typelibrary.py", line 412, in from_text
self.__calc_type_order()
File "/home/kma/projects/demo13/src/external/dl/tool/dl_tlc/dl/typelibrary.py", line 379, in __calc_type_order
if ready_to_remove( temp, type ):
File "/home/kma/projects/demo13/src/external/dl/tool/dl_tlc/dl/typelibrary.py", line 362, in ready_to_remove
base_type_name = member.type.base_type().name
AttributeError: 'NoneType' object has no attribute 'base_type'
Support for union types has been requested from both users and myself, so that should be added. Here is my suggestion for interface.
typelib:
{
// add new section "unions" to typelib.
"unions" : {
"my_union_type" : {
// this would be the actual name of the union, see below. ( not the best name for the key =/ )
"name" : "v",
// defined as dict, it could also be defined as a [] to match "types"?
"members" : {
"member_1" : "other_defined_type1",
"member_2" : "other_defined_type2"
}
}
}
}
this would generate this type-header
struct my_union_type
{
dl_typeid_t type; // type currently present in union
union
{
other_defined_type1 member_1;
other_defined_type2 member_2;
} v;
};
However I have no really good idea for how to specify this in the text format yet =/ Lets say that we have a type with a member "union_member" of the above defined type, like this:
struct the_type
{
union_type union_member;
};
how would that be specified in text-data ( how would the text-data specify what member to actually initialize? Maybe like this?
{
"the_type" : {
"union_member" : {
"member_1" : {
// ... other_defined_type1 data here ...
},
// ... setting more than one member would be an error ...
}
}
}
Getting this compile error with VS 2015 on the following generated code:
struct resource_array
{
#if defined( __cplusplus )
static const uint32_t TYPE_ID = 0xE9279B86;
#endif // defined( __cplusplus )
struct
{
const char** data;
uint32_t count;
#if defined( __cplusplus )
const char*& operator[] (size_t index) { return data[index]; }
const char*& operator[] (size_t index) const { return data[index]; }
#endif // defined( __cplusplus )
} file;
struct
{
DL_ALIGN(8) uint64_t* data;
uint32_t count;
#if defined( __cplusplus )
DL_ALIGN(8) uint64_t& operator[] (size_t index) { return data[index]; }
const DL_ALIGN(8) uint64_t& operator[] ( size_t index ) const { return data[index]; }
#endif // defined( __cplusplus )
} id;
int32_t max;
};
Specifically lines like this:
DL_ALIGN(8) uint64_t& operator[](size_t index) { return data[index]; }
Creating type a with a member as a string array and then creating an array of these breaks the strings.
example:
"type_a" : {
"members" : [
{ "name" : "strings", "type" : "array", "subtype" : "string" }
]
}
"type_b" : {
"members" : [
{ "name" : "structs", "type" : "array", "subtype" : "type_a" }
]
}
As said above enums are not yet supported in python-bindings.
And integrate with coveralls?
There has been some requests of "loosening" the json input to DL and I can absolutely understand why.
The only thing that I do not like about it is that by doing that DL will not be readable by "any conforming json parser" such as pythons json module etc. However that might not be a problem and it is worth more to have a simple format for humans.
The format would still be writable by any tool so that should be fine.
If there is anyone reading, feel free to comment :)
Add support for "apa_[]" and "apa_[6]"
Prerequisite is to rewrite json-parse since that would simplify this feature ALOT!
The following tld file generates a c-header file where the structs are place in an incorrect order which leads to compile errors.
{
"module_name":"test_model",
"module_types":
{
"vertex_element_desc" :
{
"members" :
[
{ "name" : "usage", "type" : "string"},
{ "name" : "format" , "type" : "uint32"},
{ "name" : "offset" , "type" : "uint8"},
{ "name" : "size" , "type" : "uint8"}
]
},
"vertex_buffer" :
{
"members" :
[
{ "name" : "vertex_count", "type" : "uint32" },
{ "name" : "stride", "type" : "uint32" },
{ "name" : "data", "type" : "array","subtype":"uint8" },
{ "name" : "elements", "type" : "array", "subtype": "vertex_element_desc" }
]
},
"index_buffer":
{
"members" :
[
{ "name" : "index_count", "type" : "uint32" },
{ "name" : "data", "type" : "array","subtype":"uint16" }
]
},
"sub_mesh_desc":
{
"members" :
[
{ "name" : "index_buffer_offset", "type" : "uint32" },
{ "name" : "primitive_count", "type" : "uint32" },
{ "name" : "vertex_buffer_offset", "type" : "int32" },
{ "name" : "verex_count", "type" : "uint32" }
]
}
}
}
So if you write an atom inline array of structs with members that have pointers (such as strings), the storage fails to redirect the pointers to offsets.
Enums and structs are not on-par with the C-standard for how to define / use types. Example:
struct myStruct {};
struct myStruct2 {
myStruct m_a;
}
Is incorrect. The two solutions are: The struct "myStruct" must be typedef'ed to myStruct, or the m_a variable must be typed as struct myStruct. The same goes for enums.
dl_tlc.exe -c test.h -o test.bin test.json
With the test json from the documentation fails to read test.h
dl_tlc.exe -c test.h test.json
Spews the generated header to stdout.
Expected behavior for case 1 would be that it creates the file if it doesn't exist, and also outputs the generated header there. Creating the file test.h yields only a test.bin generated.
Expected behavior for case 2 would be that it creates the file if it doesn't exists and overwrites any data inside it with the generated header.
Could be something wrong with my setup, but this is what I get from a clean directory:
workdir$ git clone https://github.com/wc-duck/datalibrary
...
workdir$ git clone https://github.com/matricks/bam.git
...
workdir$ cd bam/
workdir/bam$ ./make_unix.sh
compiling using gcc...
/tmp/ccHhCuEn.o: In function `os_tmpname':
loslib.c:(.text+0x29d): warning: the use of `tmpnam' is dangerous, better use `mkstemp'
workdir/bam$ cd ../datalibrary/
workdir/datalibrary$ gcc --version
gcc (Ubuntu 6.2.0-5ubuntu12) 6.2.0 20161005
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
workdir/datalibrary$ ../bam/bam compiler=gcc platform=linux_x86_64 config=debug -r sc
compiler used "gcc"
[ 1/65] [1] c tool/dl_pack/getopt/getopt.c
[ 2/65] [2] c++ src/dl_alloc.cpp
[ 3/65] [3] c++ src/dl_reflect.cpp
[ 4/65] [4] c++ src/dl_typelib_write_c_header.cpp
[ 5/65] [5] c++ src/dl_typelib_read_bin.cpp
[ 6/65] [6] c++ src/dl_txt_pack.cpp
[ 7/65] [7] c++ src/dl_convert.cpp
[ 8/65] [8] c++ src/dl.cpp
[ 9/65] [2] c++ src/dl_txt_unpack.cpp
[10/65] [1] c++ src/dl_util.cpp
[11/65] [3] c++ src/dl_typelib_write_txt.cpp
[12/65] [4] c++ src/dl_patch_ptr.cpp
[13/65] [1] c++ src/dl_typelib_write_bin.cpp
[14/65] [8] c++ src/dl_typelib_read_txt.cpp
[15/65] [5] c++ tool/dl_tlc/dltlc.cpp
[16/65] [3] c++ external/gtest/src/gtest-all.cc
[17/65] [1] c++ src/dl_alloc.cpp
[18/65] [2] c++ src/dl_reflect.cpp
[19/65] [4] c++ src/dl_typelib_write_c_header.cpp
[20/65] [6] c++ src/dl_typelib_read_bin.cpp
[21/65] [1] c++ src/dl_txt_pack.cpp
[22/65] [2] c++ src/dl_convert.cpp
[23/65] [6] c++ src/dl.cpp
[24/65] [4] c++ src/dl_txt_unpack.cpp
[25/65] [8] c++ src/dl_util.cpp
[26/65] [5] c++ src/dl_typelib_write_txt.cpp
[27/65] [7] lib local/linux_x86_64/gcc/debug/libdl.a
ar: `u' modifier ignored since `D' is the default (see `U')
[28/65] [7] link local/linux_x86_64/gcc/debug/dltlc
[29/65] [8] c++ src/dl_patch_ptr.cpp
[30/65] [4] c++ src/dl_typelib_write_bin.cpp
[31/65] [1] c++ src/dl_typelib_read_txt.cpp
[32/65] [6] c++ tool/dl_pack/dl_pack.cpp
[33/65] [7] tlc local/generated/unittest2.h
[34/65] [7] tlc local/generated/unittest.h
[35/65] [7] tlc local/generated/dlbench.bin
[36/65] [7] tlc local/generated/small.bin
[37/65] [7] tlc local/generated/unittest2.bin
[38/65] [7] tlc local/generated/unittest2.bin.h
[39/65] [7] tlc local/generated/unittest.bin
[40/65] [4] tlc local/generated/dlbench.bin.h
[41/65] [8] tlc local/generated/dlbench.h
[42/65] [7] tlc local/generated/unittest.bin.h
[43/65] [5] tlc local/generated/unittest.bin.h
[44/65] [4] tlc local/generated/small.bin.h
[45/65] [4] c tests/dl_test_valid_c.c
[46/65] [6] link local/linux_x86_64/gcc/debug/dl_pack
[47/65] [7] c++ tests/dl_tests_string.cpp
[48/65] [5] c++ tests/dl_tests_typelib.cpp
In file included from tests/dl_test_valid_c.c:1:0:
local/generated/unittest2.h:66:26: error: ‘vec3_test’ undeclared here (not in a function)
DL_STATIC_ASSERT( sizeof(vec3_test) == 12, "size of external type vec3_test do not match what was specified in tld." );
^
local/generated/unittest.h:41:63: note: in definition of macro ‘DL_STATIC_ASSERT’
# define DL_STATIC_ASSERT( cond, msg ) _Static_assert( cond, msg )
^~~~
local/generated/unittest2.h:66:19: error: expression in static assertion is not an integer
DL_STATIC_ASSERT( sizeof(vec3_test) == 12, "size of external type vec3_test do not match what was specified in tld." );
^
local/generated/unittest.h:41:63: note: in definition of macro ‘DL_STATIC_ASSERT’
# define DL_STATIC_ASSERT( cond, msg ) _Static_assert( cond, msg )
^~~~
local/generated/unittest2.h:67:30: error: expected specifier-qualifier-list before ‘vec3_test’
DL_STATIC_ASSERT( DL_ALIGNOF(vec3_test) == 4, "alignment of external type vec3_test do not match what was specified in tld." );
^
local/generated/unittest.h:41:63: note: in definition of macro ‘DL_STATIC_ASSERT’
# define DL_STATIC_ASSERT( cond, msg ) _Static_assert( cond, msg )
^~~~
local/generated/unittest2.h:67:19: note: in expansion of macro ‘DL_ALIGNOF’
DL_STATIC_ASSERT( DL_ALIGNOF(vec3_test) == 4, "alignment of external type vec3_test do not match what was specified in tld." );
^~~~~~~~~~
local/generated/unittest.h:21:51: error: ‘struct <anonymous>’ has no member named ‘x’
# define DL_ALIGNOF(type) ((sizeof(type) > 1)? offsetof(struct { char c; type x; }, x) : 1)
^
local/generated/unittest.h:41:63: note: in definition of macro ‘DL_STATIC_ASSERT’
# define DL_STATIC_ASSERT( cond, msg ) _Static_assert( cond, msg )
^~~~
local/generated/unittest2.h:67:19: note: in expansion of macro ‘DL_ALIGNOF’
DL_STATIC_ASSERT( DL_ALIGNOF(vec3_test) == 4, "alignment of external type vec3_test do not match what was specified in tld." );
^~~~~~~~~~
local/generated/unittest.h:21:30: error: expression in static assertion is not an integer
# define DL_ALIGNOF(type) ((sizeof(type) > 1)? offsetof(struct { char c; type x; }, x) : 1)
^
local/generated/unittest.h:41:63: note: in definition of macro ‘DL_STATIC_ASSERT’
# define DL_STATIC_ASSERT( cond, msg ) _Static_assert( cond, msg )
^~~~
local/generated/unittest2.h:67:19: note: in expansion of macro ‘DL_ALIGNOF’
DL_STATIC_ASSERT( DL_ALIGNOF(vec3_test) == 4, "alignment of external type vec3_test do not match what was specified in tld." );
^~~~~~~~~~
[49/65] [8] c++ benchmark/dlbench.cpp
local/generated/unittest2.h:68:38: error: expected expression before ‘)’ token
DL_STATIC_ASSERT( sizeof(((vec3_test*) 0)->x) == 4, "sizeof of member vec3_test::x in external type do not match what was specified in tld." );
^
local/generated/unittest.h:41:63: note: in definition of macro ‘DL_STATIC_ASSERT’
# define DL_STATIC_ASSERT( cond, msg ) _Static_assert( cond, msg )
^~~~
local/generated/unittest2.h:68:19: error: expression in static assertion is not an integer
DL_STATIC_ASSERT( sizeof(((vec3_test*) 0)->x) == 4, "sizeof of member vec3_test::x in external type do not match what was specified in tld." );
^
local/generated/unittest.h:41:63: note: in definition of macro ‘DL_STATIC_ASSERT’
# define DL_STATIC_ASSERT( cond, msg ) _Static_assert( cond, msg )
^~~~
local/generated/unittest2.h:69:28: error: expected specifier-qualifier-list before ‘vec3_test’
DL_STATIC_ASSERT( offsetof(vec3_test, x) == 0, "offset of member vec3_test::x in external type do not match what was specified in tld." );
^
local/generated/unittest.h:41:63: note: in definition of macro ‘DL_STATIC_ASSERT’
# define DL_STATIC_ASSERT( cond, msg ) _Static_assert( cond, msg )
^~~~
local/generated/unittest2.h:69:19: error: expression in static assertion is not an integer
DL_STATIC_ASSERT( offsetof(vec3_test, x) == 0, "offset of member vec3_test::x in external type do not match what was specified in tld." );
^
local/generated/unittest.h:41:63: note: in definition of macro ‘DL_STATIC_ASSERT’
# define DL_STATIC_ASSERT( cond, msg ) _Static_assert( cond, msg )
^~~~
local/generated/unittest2.h:70:38: error: expected expression before ‘)’ token
DL_STATIC_ASSERT( sizeof(((vec3_test*) 0)->y) == 4, "sizeof of member vec3_test::y in external type do not match what was specified in tld." );
^
local/generated/unittest.h:41:63: note: in definition of macro ‘DL_STATIC_ASSERT’
# define DL_STATIC_ASSERT( cond, msg ) _Static_assert( cond, msg )
^~~~
local/generated/unittest2.h:70:19: error: expression in static assertion is not an integer
DL_STATIC_ASSERT( sizeof(((vec3_test*) 0)->y) == 4, "sizeof of member vec3_test::y in external type do not match what was specified in tld." );
^
local/generated/unittest.h:41:63: note: in definition of macro ‘DL_STATIC_ASSERT’
# define DL_STATIC_ASSERT( cond, msg ) _Static_assert( cond, msg )
^~~~
local/generated/unittest2.h:71:28: error: expected specifier-qualifier-list before ‘vec3_test’
DL_STATIC_ASSERT( offsetof(vec3_test, y) == 4, "offset of member vec3_test::y in external type do not match what was specified in tld." );
^
local/generated/unittest.h:41:63: note: in definition of macro ‘DL_STATIC_ASSERT’
# define DL_STATIC_ASSERT( cond, msg ) _Static_assert( cond, msg )
^~~~
local/generated/unittest2.h:71:19: error: expression in static assertion is not an integer
DL_STATIC_ASSERT( offsetof(vec3_test, y) == 4, "offset of member vec3_test::y in external type do not match what was specified in tld." );
^
local/generated/unittest.h:41:63: note: in definition of macro ‘DL_STATIC_ASSERT’
# define DL_STATIC_ASSERT( cond, msg ) _Static_assert( cond, msg )
^~~~
local/generated/unittest2.h:72:38: error: expected expression before ‘)’ token
DL_STATIC_ASSERT( sizeof(((vec3_test*) 0)->z) == 4, "sizeof of member vec3_test::z in external type do not match what was specified in tld." );
^
local/generated/unittest.h:41:63: note: in definition of macro ‘DL_STATIC_ASSERT’
# define DL_STATIC_ASSERT( cond, msg ) _Static_assert( cond, msg )
^~~~
local/generated/unittest2.h:72:19: error: expression in static assertion is not an integer
DL_STATIC_ASSERT( sizeof(((vec3_test*) 0)->z) == 4, "sizeof of member vec3_test::z in external type do not match what was specified in tld." );
^
local/generated/unittest.h:41:63: note: in definition of macro ‘DL_STATIC_ASSERT’
# define DL_STATIC_ASSERT( cond, msg ) _Static_assert( cond, msg )
^~~~
local/generated/unittest2.h:73:28: error: expected specifier-qualifier-list before ‘vec3_test’
DL_STATIC_ASSERT( offsetof(vec3_test, z) == 8, "offset of member vec3_test::z in external type do not match what was specified in tld." );
^
local/generated/unittest.h:41:63: note: in definition of macro ‘DL_STATIC_ASSERT’
# define DL_STATIC_ASSERT( cond, msg ) _Static_assert( cond, msg )
^~~~
local/generated/unittest2.h:73:19: error: expression in static assertion is not an integer
DL_STATIC_ASSERT( offsetof(vec3_test, z) == 8, "offset of member vec3_test::z in external type do not match what was specified in tld." );
^
local/generated/unittest.h:41:63: note: in definition of macro ‘DL_STATIC_ASSERT’
# define DL_STATIC_ASSERT( cond, msg ) _Static_assert( cond, msg )
^~~~
bam: 'c tests/dl_test_valid_c.c' error 256
[50/65] [4] c++ tests/dl_tests_union.cpp
[51/65] [2] c++ tests/dl_tests_struct.cpp
[52/65] [1] c++ tests/dl_tests_dl.cpp
[53/65] [6] c++ tests/dl_tests_txt.cpp
[54/65] [8] c++ tests/dl_tests_error.cpp
[55/65] [5] c++ tests/dl_tests_ptr.cpp
[56/65] [8] c++ tests/dl_tests_reflect.cpp
[57/65] [6] c++ tests/dl_tests_bitfield.cpp
[58/65] [7] c++ tests/dl_tests_inline_array.cpp
[59/65] [8] c++ tests/dl_tests_base.cpp
[60/65] [3] lib local/linux_x86_64/gcc/debug/libgtest.a
ar: `u' modifier ignored since `D' is the default (see `U')
[61/65] [3] c++ tests/dl_tests_util.cpp
[62/65] [2] c++ tests/dl_tests_array.cpp
[63/65] [4] link local/linux_x86_64/gcc/debug/dlbench
[64/65] [1] dll local/linux_x86_64/gcc/debug/dlsh.so
[65/65] [2] link local/linux_x86_64/gcc/debug/dl_tests
bam: error: a build step failed
Currently pointers in text data is an integer:
0 - root-item
1 > items in the sub_data-section where each key is the matching number as a string. Such as
"subdata" : {
"1" : { ... },
"2" : { ... },
}
The bad thing here is the discrepancy between string and int depending on position in text-data and its hard to read.
My idea is to replace pointer-data as strings all over the board, so when setting a pointer-member it becomes:
{
"my_ptr_member1" : "my_first_sub_ptr",
"my_ptr_member2" : "my_second_sub_ptr",
"my_ptr_member2" : "root"
}
and in subdata
"subdata" : {
"my_first_sub_ptr" : { ... },
"my_second_sub_ptr" : { ... }
}
One potential problem with this, but some thing I accept, is that the pointer-names will not be kept when unpacking, so when doing a dl_txt_unpack() the pointers will be named "ptr1", "ptr2" etc.
Since we parse our header-files to extract certain information and generate things, it would be very beneficial if we could add comments to types and members. Such as:
// my_comment_1
enum A {
...
};
// my_other_comment
struct struten {
float mymember; // my_member_comment
};
A struct with the format:
struct Parent
{
struct Child
{
some_type* ptr;
}
}
ptr will not be patched since both parent-instance and child-instance has the same offset in the stored data.
I think it would be really nice if you could do this in your type declaration:
{ "name" : "blah", "type" : "struct mystruct*[5]" }
This means that the struct can hold a list of pointers so that you don't have to do void*'s and can keep type safety somewhat reasonable.
It's a part of the standard library and simply defining it for all msc-versions won't work on msvc2015 (at least).
Currently pointers are stored in packed data as the offset from instance. This could be reworked to make patching a lot faster and simpler.
Currently patching is done by traversing the struct with stored type-info.
If the storeage instead was stored as x bits offset and y bits offset-to-next-pointer a simple loop as this could be used to patch pointers:
uint64 iter = instance->first_ptr;
while( iter & NEXT_MASK )
{
patch_pointer( iter, iter & OFFSET_MASK );
iter = iter + ( iter & NEXT_MASK );
}
this will however force instances to be less than 4GB on 64bit and less than 65kb on 32bit.
To begin with I'll save the old parse-code, set a flag in instance how the pointers is patched, and dispatch depending on that flag.
Enum feature: shifted values (1ULL << 34ULL) etc.
Why?
One dependency less ( the last one! )
Bit most of all I think that it would open up for lots of optimizations a general json parser can't do that dl could since we have the type info.
However first some benchmarks need to be written, code can never be called optimized without numbers :)
There should be an error that DL doesn't find the pointer pointed to (since 0 is never used)
If you have your typelibrary definition that ends with a line that has the ending scope in it (a simple line with the last } on it) the typelibrary compiler fails with:
50> expected '{' but got '@' at 114
50> at line 0, col 0:
50> @�n���
50> ^
50> failed to load ty
This should either be checked and given a proper error rather than printing a bad buffer, OR it should be accepted.
Example:
{
"module" : "test",
"types" : {
"mytype_1" : {
"members" : [
{ "name" : "submember1", "type" : "string[]", "default" : [ "apa" ] },
{ "name" : "submember2", "type" : "uint32" },
]
},
"mytype_2" : {
"members" : [
{ "name" : "member1", "type" : "mytype_1", "default" : { "submember2":1337 }}
]
}
}
}
Fails on two parts:
1 The default for member1 in mytype_2 tries to apply a default-value that is the size of the entire member.
2 Defaults for the type mytype_1 is never applied to member1 in mytype_2
It should be possible to add "tags" to each of the above mentioned parts of a typelibrary for the user to use as they see fit. ( mostly for editor/tools use ).
A tag should be any string and dl will put no meaning to these. On each of the aboved items a list of these tags should be valid.
Tags use-case examples:
"file" to indicate that a string should be a file
"range(0,100)" to say the range of an int.
"slider" should be edited by a slider in some editor.
Note: This will NOT be implemented by dl, only the tag-support.
Using a union in a type is valid in DL but generates a header where the union type is declared after the struct that uses it.
I can work around this by using a pointer to the union in the type but preferably I would like it built in. One solution would be to sort the types and unions on dependencies (types and union only depending on PODs first etc.)
This test case crashes the dl_tlc when running it as:
dl_tlc.exe -c -o output.h input.json
{
"module" : "component_resources",
"enums" : {
"damage_types" : {
"DAMAGE_SLASHING" : 0,
"DAMAGE_CRUSHING" : 1,
"DAMAGE_PIERCING" : 2,
"DAMAGE_HEAT" : 3,
"DAMAGE_COLD" : 4,
"DAMAGE_ELECTRIC" : 5,
"DAMAGE_CORROSIVE" : 6,
"DAMAGE_MENTAL" : 7,
"DAMAGE_POISON" : 8,
"DAMAGE_RADIATION" : 9,
"DAMAGE_NUM_TYPES" : 10
}
},
"types" : {
"damage_component_resource" : {
"members" : [
]
}
}
}
Warning is:
50> expected '{' but got 'P' at 114
50> at line 0, col 0:
50> P�����
50> ^
Should either not allow definition and print a more appropriate error or produce an empty typelibrary.
This would be very nice
{
"module" : "my_module",
"enums" : {
"some_enum" : {
"SOME_THING",
"SOME_THING2",
"NUM_THINGS"
}
},
"types" : {
"my_type" : {
"members" : [
{ "name" : "some_things", "type" : "some_enum[NUM_THINGS]" }
]
}
}
}
So that if you go in and modify the enum, the result will still be good and not missing to increase the value on your static array.
{
"module" : "stuff",
"enums" : {
"my_enum" : {
"THIS_IS_ENUM" : 0
}
},
"types" : {
"my_thing" : {
"members" : [
{ "name" : "this_is_it", "type" : "my_enum" }
]
}
}
}
generates a struct that looks like this:
struct my_thing
{
my_enum this_is_it;
};
Which unfortunately does not compile in c. I assume that the same thing happens when you do the same thing with structs. I'd suggest either adding the "enum" / "struct" keyword to the types (preferably) or do the c-style of declaration with typedef enum my_enum {} my_enum;
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.