V2 is soon about aqlaserializer HOT 81 CLOSED

That is really great news!

Especially number 9 is very interesting for me as although the current version works with extended stack size it is quite slow.

When you think the first usable version will be ready?

The reason I'm asking this is that we will release the product where I'm currently using AqlaSerializer very extensively in about 2-3 weeks:
After this release I basically cannot change the serialization anymore as long as this is not backwards compatible for reading.

Or is there a chance that we will have V2 backwards compatibility - just for reading the V1 format?

from aqlaserializer.

@ab-tools, unfortunately V2 won't be wire compatible with V1. In V2 the serialization flow is significantly changed.

Actually all the promised changes are already implemented but without compilation support yet. You may check the preview version:
https://github.com/AqlaSolutions/AqlaSerializer/releases/tag/v.dev-preview-v2

The emit (compilation) part is very large... I set the milestone to February 25.

from aqlaserializer.

First thanks for your quick reply as always! :-)

unfortunately V2 won't be wire compatible with V1. In V2 the serialization flow is significantly changed.

Yes, I guessed that already, but thought maybe you could implement a fallback like if it detects V1 format, it processes it like in V1 version (just read-only). But if V2 comes so quick, it's not needed for me, then I will use V2 directly from my first product release.

Actually all the promised changes are already implemented but without compilation support yet.

I don't need any compilation support, just a "stable" version that I can use:
The most important part for me to know is the point when there will be no more format changes. I mean that every V2 file I created can be read with upcoming V2 versions.

Would you consider the current pre-release version already "format fixed" or do you expect some format incompatibility changes till the release on February 25?

from aqlaserializer.

@ab-tools

1. There will be a change in collection subtypes serialization soon. I'm not planning to change the format afterwise.
2. I still can't guarantee that there are no hidden bugs which may make me to change the format. But the tests are mostly green (except those relying on emit).
3. Attributes and default settings changes are planned and they may affect which settings are actually applied (e.g. null and reference support on members).
4. After the final release you may want to change some setttings (like AdvancedVersioning mode) for better performance/size which will may the format incompatible.

I would wait as long as possible before making the final decision about migrating to V2 format.

from aqlaserializer.

OK, good, I'll wait as long as possible, but also need to get the product release finally done (is already a bit late as always ;-) ).

It would be great if you could do a quick comment here when point 1 is done as it seems I at least need to wait for that.

Thanks again a lot
Andreas

P. S.:
If everything goes well with V2 and I really can used that for my product which would be according to your description above for me really a BIG step forward, I definitely need your PayPal e-mail address for a donation - I'm very thankful for all your efforts you put in this and want at least to give a little bit back!

from aqlaserializer.

@ab-tools

Especially number 9 is very interesting for me as although the current version works with extended stack size it is quite slow.

I should warn you that the number 9 goal is not to speed things up but to avoid the necessity to create another thread with big stack size. It may be even slightly slower than the recursive version because it does more things like boxing, maintaining reference list, etc.. Although thread creation itself is a slow procedure and big stack size doesn't fit good in CPU cache so number 9 may still turn up faster for you.

Good news!

I've finished those changes to lists and now I'm not going to introduce any format changes for V2 release. Note that the API is planned to be changed soon. You may try to build the last version from the V2 branch. This https://github.com/AqlaSolutions/runsharp is included to the solution as a project reference so you'll have to clone it too.

Now its defaults are suitable only for testing so you need to disable some things for your real use model like this:

var comp = ProtoCompatibilitySettings.None;
comp.AllowExtensionDefinitions &= ~NetObjectExtensionTypes.AdvancedVersioning;
comp.AllowExtensionDefinitions &= ~NetObjectExtensionTypes.LateReference;
var model = TypeModel.Create(false, comp);

"LateReference" is number 9 but for now it's model-wide setting and can't be toggled per type. You'd want it only for some specific properties so better disable for now. The same is true for "AdvancedVersioning".

Null support is also enabled (where applicable) model-wide in this version. Attribute option "SupportsNull" doesn't change anything yet.

If you setup this way you should be able to read the data in V2. But if you change later those settings per type in V2 (e.g. enable LateReference for some members) you won't be able to read the old data so you'll need to setup two models - 1st for reading the old data and 2nd for using the new features. This gives me an idea of a feature where each attribute may have a "set-id" option allowing to auto-add the same field in different way to multiple models.

Also note that AqlaSerializer now supports plain Google Protocol Buffers format the same way like protobuf-net does (without referencing, null handling, etc) and it will be always possible to read it in any upcoming V2 versions. (protobuf-net specific features are not compatible, only plain Google Protocol Buffers)

from aqlaserializer.

@ab-tools I made another format change today. I hope it's not too late. Please notify me when you release your app and it's really impossible to change the format anymore.

from aqlaserializer.

First I'm very sorry to not get back to you earlier - I really appreciate your great work and also keeping me up-to-date on the status.

In fact we're currently very busy to get everything ready for release, but we did not release yet. So the last format change was not a problem.

I will do some tests (especially regarding read performance which is important for us) today evening and will let you know the results.

Thanks again
Andreas

from aqlaserializer.

Just wanted to compile the latest V2 version, but I only have Visual Studio 2013 and it seems you are using a lot VS 2015 features now.

Could you therefore please provide the latest V2 version any binary?
Then I can do a test with them later today.

Thanks!
Andreas

from aqlaserializer.

@ab-tools https://github.com/AqlaSolutions/AqlaSerializer/releases/tag/v.dev-preview-2-2-v2

from aqlaserializer.

Saw that build already, but it seems to be already 20 days old.

Doesn't it make more sense to test with your latest version?

from aqlaserializer.

@ab-tools it's new, it's just somehow attached to the master branch dates but I'm working in V2 now.

from aqlaserializer.

@ab-tools oops, now I know why ;)
I've just reuploaded it correctly.

from aqlaserializer.

Now it looks much better, thanks! :-)

from aqlaserializer.

Just wanted to do a first test with V2, but I cannot compile the project anymore after referencing the new DLL due to "missing System.Runtime.CompilerServices.ExtensionAttribute..ctor" compilation error.

Sorry, but looks like I need a .NET 4 version of the DLL to reference it in my project. I compile against .NET 4.0 x86.

from aqlaserializer.

@ab-tools Oh, I see. Let me check it.

from aqlaserializer.

@ab-tools I created a new .NET 4.0 project, referenced the dll, used it in Program.cs and I see no compiler errors there. I'm not sure what may cause your issue.

from aqlaserializer.

@ab-tools please download the updated binaries from the same link and try to compile using them. I have a guess what may cause the error. But I still can't reproduce it.

from aqlaserializer.

I really just removed the old AqlaSerializer reference and added the new one. While trying to compile again afterwards, I got this compilation error.

I will try it with your binary again directly tomorrow morning, thanks!

from aqlaserializer.

@ab-tools I uploaded the full build, you can use net35/net40 version.

from aqlaserializer.

Just wanted to let you know first that I still get the same compilation error also when I use the .NET 4 version. Maybe not both of the dependencies are .NET 4 as well?

Anyway, it works find with the work-around you mentioned on the download page.
So I can do some first tests now and let you know the results then.

from aqlaserializer.

@ab-tools I'm absolutely sure (and I've just checked via reflector) that net40 dlls version doesn't include its own declaration of ExtensionAttribute (which was the cause). Either you are not referencing the correct versions of mscorlib and System.Core inside your project or you are still referencing that net30 version.

from aqlaserializer.

OK, I will double check it once more, maybe I just selected the wrong file after the first test (which is currently running) has finished.

from aqlaserializer.

I'm sorry, this time it was really my fault - just used the DLL from the wrong folder.
With the .NET 4.0 version everything compiles just fine now!

I also finished first test with the parameters

var comp = ProtoCompatibilitySettings.None;
comp.AllowExtensionDefinitions &= ~NetObjectExtensionTypes.AdvancedVersioning;
comp.AllowExtensionDefinitions &= ~NetObjectExtensionTypes.LateReference;
var model = TypeModel.Create(false, comp);

as you suggested above and here are my results:

• Saving and loading my 2 database files in the project works fine without any code change except using the options above.
• It seems to be a bit slower, but that's probably because "compilation support" is not enabled yet, right?
• Then I changed my code to no longer create a new thread with a bigger thread size, but load the database on the main thread: Result was that is throws again a StackOverflowException at me. I thought this problem would be gone now, isn't it?

Thanks again
Andreas

from aqlaserializer.

@ab-tools

1. I'm glad to hear it :)
2. yes, with the compiler support it should be much faster than without.
3. if you remove this line

comp.AllowExtensionDefinitions &= ~NetObjectExtensionTypes.LateReference;

then it should work without new thread (can you check pls?) but it will write all the references using LateReference technique which is not optimal. I suggest you to keep it disabled until I add a possibility to specify LateReference mode per member. Also LateRefernce switch is not compatible between versions,

if you change later those settings per type in V2 (e.g. enable LateReference for some members) you won't be able to read the old data so you'll need to setup two models - 1st for reading the old data and 2nd for using the new features. This gives me an idea of a feature where each attribute may have a "set-id" option allowing to auto-add the same field in different way to multiple models.

from aqlaserializer.

First thanks for your very quick reply as always. :-)

1. (empty - stupid Markdown syntax)
2. Very good. Although it's usable at the moment, program start takes some seconds longer which is not so good as it's quite slow already anyway (due to the amount of data that needs to be loaded), but if that gets better with the "compilation support" again that's OK.
3. Just for my understanding "LateReference" is something different than the "ByReference" stuff which I can enable/disable type based by "NotAsReferenceDefault", right?

I did a test with "LateReference" enabled now and yes, now it works without a new thread!
But it seems to be again considerable slower then unfortunately and the resulting file size is about 16 % bigger (doesn't matter for me, just wanted to mention it - for me only speed is relevant, not size). By the way in the first test with "LateReference" disabled the file size was even a bit smaller than with V1, but only 2 % or so.

Also LateRefernce switch is not compatible between versions

As our product is a bit delayed now anyway and we still need to fix some remaining issues and get everything ready (manual and stuff like that) we postponed the official release now not till end of this month.

Although that's annoying for me it should fit very well to be after your official release of V2 and everything should be ready then including the "LateReference" switch. :-)

from aqlaserializer.

@ab-tools 2 - there is NotAsReference on members which is also affected by NotAsReferenceDefault on a member type. LateReference may be used only for members which have reference tracking enabled. It changes reference tracking behavior to not use deep traversal ("to not go inside member recursively") but it has to write additional data and therefore it's slower. I recommend to use it only on concrete members which should not be traversed recursively.

It's called "LateReference" because it's processed later, outside of its container object.

from aqlaserializer.

That makes totally sense, I need this "LateReference" only on a single member of a single type and this member, of course, will also use storing by reference.

I will test that as soon as I can set "LateReference" for members individually.

Just let me know as soon as it make sense to test a new version.

from aqlaserializer.

@ab-tools

As our product is a bit delayed now anyway and we still need to fix some remaining issues and get everything ready (manual and stuff like that) we postponed the official release now not till end of this month.

So no problems then in case I need to change something in the format before that time, right?

I have an idea on how to add versioning support for LateReference switch.

from aqlaserializer.

Yes, the current time plan is to have a final version ready by 22th of February - ideally also with V2 of AqlaSerializer. This would be then one week before the official release, so our beta testers have one week to find hopefully all remaining bugs. ;-)

So at least till 22th of February a format change is no problem, thanks for asking!

from aqlaserializer.

@ab-tools I've just finished with emit version and run roundtrip performance tests based on your database that you've emailed me before.

V1 runtime: 16753 ms, compiled: 9709 ms (compilation time included).
V2 runtime: 8000 ms, compiled: 3000 ms (+compilation time: 600 ms).

These results were made with automatic late-reference enabled.

I'll upload the beta release later after adding LateReference to attributes.

from aqlaserializer.

Thanks a lot - that sounds very promising!

from aqlaserializer.

Hello!
I'm really amazed with this fork. Great job! I've just pulled v2 branch, built it and replaced protobuf-net.dll with aqlaserializer.dll, applied a few straightforward changes to my code (thanks for good migration guide!) and everything just works as expected now. In just a few minutes!

I've spent so much time with hacking protobuf-net just to make it works in a proper way (you know - null/empty collections handling with surrogates, references, etc), I stuck and decided I need to start my own fork, but when I got deeper into the code mess of original repository (Marc doesn't use ReSharper?) I was depressed. But thanks Github I've found this fork!

I will keep in touch about our future experience with AqlaSerializer.

Regards! Спасибо! :-)

from aqlaserializer.

@aienabled Thanks for your feedback! Удачи)

@ab-tools
I decided to include attribute changes into the full V2 release so I postponed the milestone. But other things are ready now. I added the "LateReference" switch that you need in the recent version:

https://github.com/AqlaSolutions/AqlaSerializer/releases/tag/v.2-alpha-35-2

With this version please use the default overload of TypeModel.Create() without arguments.

The newest benchmark results for your db (hot start):

Time: 7012 ms (no compile)
Time: 2539 ms (compiled in place)
Time: 2255 ms (fully compiled)
Size: 60192782

I made only 2 changes inside the Airway class:

    //[ProtoMember(4)]
    [SerializableMember(4, EnhancedMode.LateReference)]
    public Waypoint PreviousWaypoint { get; set; }

    //[ProtoMember(5)]
    [SerializableMember(5, EnhancedMode.LateReference)]
    public Waypoint NextWaypoint { get; set; }

from aqlaserializer.

With v.2-alpha-35-2 (Full\net45) I've got this exception during deserialization (using the same type model as used for serialization):

Wrong format version, required 3 but actual 0
   at proto_124(Object , ProtoReader )
   at AqlaSerializer.Serializers.CompiledSerializer.AqlaSerializer.Serializers.IProtoSerializer.Read(Object value, ProtoReader source)
   at AqlaSerializer.Meta.RuntimeTypeModel.Deserialize(Int32 key, Object value, ProtoReader source, Boolean isRoot)
   at AqlaSerializer.Meta.TypeModel.DeserializeWithLengthPrefix(Stream source, Object value, Type type, PrefixStyle style, Int32 expectedField, TypeResolver resolver, Int32& bytesRead, Boolean& haveObject, SerializationContext context)

However, it works fine with source version from branch v2 (I've built it and take aqlaserializer.dll from AqlaSerializer\protobuf-net\bin\Release).

from aqlaserializer.

@aienabled can you provide any sample code? I don't see the issue.

from aqlaserializer.

@AqlaSolutions, sure, I will try to make a minimal sample to demonstrate the issue.

from aqlaserializer.

@AqlaSolutions: Great, so the current alpha version is format stable. I'll do some test with it today or tomorrow and if everything works fine which from V1 to V2, thanks!

Just out of interest what are these "attribute changes" about you want to include?
But these do not change the format anymore then - so everything stays compatible, right?

And sorry to ask, but I think I did not totally understand this "no compile", "compiled in place" and "fully compiled" stuff yet:
If I "just" use the AqlaSerializer DLL as it is, I guess it's "compiled in place" then, right?

Thanks a lot
Andreas

from aqlaserializer.

@ab-tools

Just out of interest what are these "attribute changes" about you want to include?

This

Attributes to specify behavior on collection and collection elements (and nested) separately.

And

• More Default-X member options for [SerializableType].
• Strict verification for specified options.
• Big refactoring for the internal mapping system.
• Changes to ProtoCompatibilitySettings class to reflect attribute changes.

AutoCompile is automatic CompileInPlace. Full compilation creates a separate serialization dll which can be referenced and used directly instead of RuntimeTypeModel on platforms where CompileInPlace can't be used or startup time is critical.

But these do not change the format anymore then - so everything stays compatible, right?

Right. And if you still keep using attributes from protobuf-net.dll then you won't notice Aqla attribute changes.

from aqlaserializer.

@AqlaSolutions, it seems the issue was on my side. I cannot reproduce the issue now after all my fixes. Sorry for false report!

from aqlaserializer.

Update 2.0.0.37 for [SerializableType] attribute.
https://github.com/AqlaSolutions/AqlaSerializer/releases

from aqlaserializer.

@AqlaSolutions: Just got me a Visual Studio 2015 version here and tried to compile v.2-alpha-37 myself now, but got a lot of error messages.

I also got the very latest source code of your runsharp fork and made sure the "aqlaserializer" project finds the source. Then I tried to compile the "aqlaserializer" project (not the whole solution, just this single project). It first compiled the referenced "RunSharp" project which were successful, but the build of the main "aqlaserializer" project failed throwing an enormous amount of errors at me like "TriAxis" "ICodeGenContext" in the file "CompilerContext.cs" not found although "RunSharp" is referenced and compiled successfully.

Do you have any idea what I'm doing wrong?
I had no problems compiling the V1 version.

Thanks
Andreas

from aqlaserializer.

@AqlaSolutions: Got it, sorry!

I just needed to remove all the other projects (like the phone projects) from the solution and afterwards it compiled just fine.

Don't really understand why that's necessary as I just compiled the single project and not the whole solution. Anyway, it works now! :-)

I will do some performance test with my latest project version today and let you know then.

Thanks!
Andreas

from aqlaserializer.

@AqlaSolutions: Sorry to bother again, but got another question.

WIth V1 I used for most of my types (not all of course, like the two mentioned that even need late binding) the attribute "NotAsReferenceDefault = true", because they do not need to be saved by reference as I know they will only exist once. So I would consider it more efficient in terms of performance and needed space to not save them as a reference.

But in V2 it seems that this "NotAsReferenceDefault" does not exist anymore or did I miss something?

from aqlaserializer.

@ab-tools I think your compile errors were caused by those Phone8/etc projects using different RunSharp per-platform projects as dependencies - you compiled only the one main RunSharp, right?

Instead of AsReferenceDefault there are new properties on [SerializableType]. Set DefaultMemberFormat = MemberFormat.Compact to disable both reference tracking and null support or set EnhancedWriteAs = EnhancedMode.Minimal to have only null support without reference tracking.

The same properties can be set per-member right in their attribute constructor overloads.

I have to warn you that EnhancedMode.LateReference now has an effect only on member attributes so don't try it on type (this part is not done yet).

Also I'm going to rename EnhancedWriteAs to DefaultEnhancedWriteAs on type attributes.

from aqlaserializer.

@AqlaSolutions:

I think your compile errors were caused by those Phone8/etc projects using different RunSharp per-platform projects as dependencies - you compiled only the one main RunSharp, right?

Yes, I only built the main project.

But anyway, after removing all the other (for me not needed) project from the solution everything compiles fine now. Thanks!

Instead of AsReferenceDefault there are new properties on [SerializableType] . Set DefaultMemberFormat = MemberFormat.Compact to disable both reference tracking and null support or set EnhancedWriteAs = EnhancedMode.Minimal to have only null support without reference tracking.

Not sure if I got that correctly yet what I need to change:
In V1 everything was "by reference" as default and I used the NotAsReferenceDefault = true option on the types that should not be stored as reference. I did not need to change anything on the per-member area as it was enough to "say" this type (whereever it is used) should not be stored as reference.

If I understand you correctly I just need to replace the old NotAsReferenceDefault = true with EnhancedWriteAs = EnhancedMode.Minimal on the type SerializableType attribute to get the same behavior.

Is that correct?

from aqlaserializer.

@ab-tools yes, correct.

In V1 everything was "by reference" as default

It's default if you are using Aqla native attributes. If you are using ProtoContact/ProtoMember they will stay not as reference default for compatibility.

from aqlaserializer.

@AqlaSolutions: I'm using native attributes - got rid of the old protobuf stuff long time ago already. ;-)

So in this case what I wrote is correct?

If I understand you correctly I just need to replace the old NotAsReferenceDefault = true with EnhancedWriteAs = EnhancedMode.Minimal on the type SerializableType attribute to get the same behavior.

from aqlaserializer.

@ab-tools yes

from aqlaserializer.

@AqlaSolutions: Perfect, thanks!

from aqlaserializer.

@AqlaSolutions: Just finished with first performance tests and I'm really impressed by the speed boost of V2 compared to V1 - both saving and reading my database is really A LOT faster now. Great work!

Just gave the EnhancedWriteAs = EnhancedMode.Minimal setting a try and it really only had a very minimal effect on file size: 236 bytes less of 70,226,253 bytes in total. Although file size is not that important for me I just want to make sure I configured everything correctly, therefore I ask.

I use EnhancedWriteAs = EnhancedMode.Minimal really on 90 % of my types and therefore would expect that it has a much bigger effect on file size if no reference is needed as the NotAsReferenceDefault = true did for V1. So is that working as expected?

Again, I don't care much about file size, just want to make sure everything is working correctly and as expected.

from aqlaserializer.

@ab-tools there could be multiple roots of this (e.g. if your type is inside collection it has the same AsReference mode as the collection member itself) - this part is still "under construction".
There are some workarounds but (considering that you don't care much about the file size) to be sure that everything is going to be compatible with later setups I'd suggest you to just remove those EnhancedWriteAs = EnhancedMode.Minimal for now. Reference type members will be always written in enhanced mode so you will have the possibility to switch between Reference and Minimal modes as they are compatible between themselves.

from aqlaserializer.

@AqlaSolutions: You're right. Basically I also only used this in V1 to optimize everything possible regarding saving/loading speed, but as V2 is so much faster now anyway, that's no problem anymore. So I will just remove those completely, thanks!

Another minor thing:
I noticed that V2 is now also able to serialize something like "short[][]" directly without the need of a surrogate.

I gave that a try, but the resulting file size is almost 4 times as big than my surrogate produces which just writes the short values one after the other in the file.

It's not so important for me as the surrogate works fine, just wanted to know if I just need to use a specific option to improve that or what's the reason for this extremely bigger file size than needed to store the values plus the size of the array?

from aqlaserializer.

@ab-tools the reason is that each short[] array inside short[][] is handled like a normal object (with all its overhead like field number, null/reference data, array length). Later there will be an option to write it in more compact mode but still it won't be as effective as directly writing short[,] until I add a support for jagged arrays

from aqlaserializer.

@AqlaSolutions: OK, no problem, then I just keep my surrogate for now. Thanks!

from aqlaserializer.

@AqlaSolutions: After all my tests were successful I just switched to V2 for the latest beta version of my product. :-)

One thing I just wanted to mention/ask:
Do you think it would be possible to use "normal" LINQ when compiling against .NET 4.0 or newer instead of this "LinqBridge.cs"?

I just did this change as I compiled it myself anymore and this saved about 50 KB of DLL size. Although size is not so important these days it seems just redundant to include an "own" LINQ implementation instead of using the .NET one.

Maybe you could switch to "normal" LINQ for .NET 4.0 and newer via compiler directives?

from aqlaserializer.

@ab-tools I decided to use the same LINQ implementation in all targets because different behaviors may cause unexpected and not reproducable bugs - even when it looks like simple thing in reality they behave differently in corner cases. It's a kind of overcautiousness but the size difference is not that big...

When you want to have minimal binaries size you usually run your product in limited .NET environment like mobile platforms where you would need to use CoreOnly version with precompiled model dll and without RuntimeTypeModel.

I will check whether I can remove LinqBridge from CoreOnly version completely and if it's not possible then I will go the way you asked because minimizing CoreOnly build is more important than having the same implementation.

Anyway this is something that may be chosen later when more important changes like V2 release are done.

from aqlaserializer.

@AqlaSolutions: OK, didn't expected that they may behave differently.

But as I'm only using the .NET 4 version (don't have other projects with different targets it need to talk to), it should not cause any problems to use native LINQ hopefully.

from aqlaserializer.

@AqlaSolutions, is there any wiki page covering inheritance support?
AqlaSerializer is positioned as serializer for objects, not simple data. So I expect some improvements in how it handles hierarchies in comparison to protobuf-net.

For example, I have a class with complex hierarchy, and I need to be able to serialize it. I expect it to work out-of-box, but it (source from latest v2_attributes branch) doesn't serialize private fields of base classes. For network transmission this could be acceptable, but for database scenario it usually causes an unexpected data loss.

In case of protobuf-net, to serialize private fields from base classes, I need to add MetaType for all its base class chain and for each of them specify inheritors by calling AddSubType as discussed in this question on SO. That's quite hard even with code-only approach (without attributes).

So I would like to see a wiki page covering inheritance support in AqlaSerializer.

BTW, I found a typo: "Comparsion with protobuf net and migration" (Comparsion Comparison).

Regards!

from aqlaserializer.

@aienabled, there is an automatic subtype registration in attribute mapping when both base and derived type are marked with [SerializableType]. In such scenario they should be registered in the same order to have the same ids. Please see also https://github.com/AqlaSolutions/AqlaSerializer/wiki/Batch-types-registration

When you register type from code you still have to specify subtypes manually. But If you use attributes you can simply specify [SerializableType(ImplicitFields = ImplicitMode.AllFieldsAndProperties)] to implicitly add all current type fields.

It's important to have subtypes registered with field numbers because this way each hierarchy has independent type key numbers and you may add different hierarchies in any order. Instead of writing concrete type index of type in Model.Types[] serializer writes subtype index which is local for each base type. Moreover, when a concrete type is not registered in another version deserializer will still read the most concrete possible type from its hierarchy.

Also see https://github.com/AqlaSolutions/AqlaSerializer/wiki/Changing-auto-add-behavior about ImplicitFallbackMode of DefaultAutoAddStrategy. ImplicitFallbackMode also supports automatic subtypes registration. You can setup ImplicitFallbackMode and add types from code with auto = true.

The current code-only API provides very low level access so you don't have anything "automatic" there. Right now I'm not planning to make a more high-level mapping API but when I finish with attributes you will have more extension points inside DefaultAutoAddStrategy. (But please don't use those extension points now because I'm going to alter that part for attributes.)

from aqlaserializer.

@AqlaSolutions, thank you for detailed response! Please consider adding a special wiki page based on it - it could be very helpful for new developers.

I just checked and can confirm the attributes approach is working fine.
However, I found the same problem as with my code-oriented approach: exception The type cannot be changed once a serializer has been generated thrown when I try to serialize another type which uses the same base type (after serializing object of some type using the same base type). It happens because serializer for the base type is already generated (the test code if you're interested).

It seems the issue is by design... this is resolvable by cloning RuntimeTypeModel, but for developers using RuntimeTypeModel.Default is not a solution of the issue as Default is getter-only property. Maybe it would be better to add RuntimeTypeModel.Reset() (static method, for cloning Default instance). However I think it doesn't resolve the problem completely because it need to be called manually.

Going back to the my problem. I'm dynamically adding new types into the RuntimeTypeModel, and most of this types have exactly the same type names (yes, I need it to work this way to convert objects of "old" type to objects of "new" type - but type FullName exactly the same). I'm cloning/unfreezing the model before extending it with new types. Without inheritance it worked perfectly well (I've successfully used it for a few days), but with inheritance it cannot properly deserialize objects into the new types (Deserialize() returns value of old type). It seems to be expected behavior (protobuf scheme):

message BaseClass {
   optional uint32 id = 1 [default = 0];
   // the following represent sub-types; at most 1 should have a value
   optional DeferredClass DeferredClass = 128;
   optional DeferredClass DeferredClass = 129;
}

• so it cannot deserialize DeferredClass to the new type (fieldId=129) because it was serialized as field Id=128 and deserialized as old type (even if I provided new type to Deserizalize() method).

I'm stuck with it now and seeking for any ideas how I could avoid it this problem without creating separate RuntimeTypeModel. Please note I'm not using attributes and using code-only model configuration.

Is it possible to use something like "flatten hierarchy" when the serializer for the specific type itself contains logic for (de)serializing private fields of base class(es) - exactly the same way it (de)serialize self fields and properties? It will add some duplication to scheme, but it will completely resolve the problem I have because this way it's not required to add any extra fields for subtypes to the base type TypeModel (it even will not require adding base types into the model and cloning the model to add new types). Is that achievable with current architecture? If so, could you point me please on the extension points which I could use to add support for it?

Regards!

UPD. I just found a workaround by adding a method public ValueMember AddField(int fieldNumber, FieldInfo fieldInfo, Type itemType = null, Type defaultType = null, object defaultValue = null) which is basically uses the same code as current private ValueMember AddField(int fieldNumber, string memberName, Type itemType, Type defaultType, object defaultValue). With this method I can manually provide all the fields from base class(es) and it (de)serializes them correctly and without any issues. So I have working "flatten hierarchy" approach now.

from aqlaserializer.

@aienabled

It seems the issue is by design... this is resolvable by cloning RuntimeTypeModel

Yes, I added a special API method CloneAsUnfrozen for such case. But better just use TypeModel.Create to make your own RuntimeTypeModel instead of RuntimeTypeModel.Default and no need to depend on it or Reset.

Is it possible to use something like "flatten hierarchy"

It's not hard to add such option. But it will be impossible to determine which subtype you serialized so you'll have to tell deserializer registered (not base) type. Such feature won't be top-priority for me in near future. You can add it by yourself, check the DefaultAutoAddStrategy code. It looks like that you just need to remove BindingFlags.DeclaredOnly from member search options and disable base type registration.

Possible solutions for your issue:

1. Make a reusable code to setup new TypeModel.Create-d model with either old types or new. Use it to create two models exactly the same way - 1 for serialization, 2 for deserialization.
2. Use different base classes for old and new hierarchies. Map them the same way. Pass new base type to Deserialize method. It will work only if you use different types for the most abstract registered type of the hierarchy.

from aqlaserializer.

@AqlaSolutions

You can add it by yourself, check the DefaultAutoAddStrategy code. It looks like that you just need to remove BindingFlags.DeclaredOnly from member search options and disable base type registration.

Unfortunately, .NET Reflection will not return private fields of base class(es) if GetMember() is invoked for deferred class type (there is a question on SO). So my issue cannot be resolved this way.

1. Make a reusable code to setup new TypeModel.Create-d model with either old types or new. Use it to create two models exactly the same way - 1 for serialization, 2 for deserialization.

Yes, I'm also considered this sophisticated approach. However, it will require total refactoring of my architecture. Currently I'm using the same instance of RuntimeTypeModel for everything.

So it seems the easiest option for me is using fork of your code which allows adding FieldInfo directly (not by name) to field metadata as I described in UPD section of my previous message. It suits very well for my current needs.

Regards!

from aqlaserializer.

@aienabled ok, cool then )

from aqlaserializer.

Beta is out!

https://github.com/AqlaSolutions/AqlaSerializer/releases

I added mapping examples to the header in this issue.

@ab-tools
Please check the compatibility. You may need to add some attributes but the format is the same.

from aqlaserializer.

@AqlaSolutions: OK, I will give it a try, thanks!

But I'm not sure if I understood the new mapping attributes correctly yet:

1. Do I need to apply these to all collections (List, Dictionary, ...)?
2. What I need to apply for something like this: public List<CustomObject> CustomObjects { get; set; }?
3. What does the serializer uses by default if no such attributes are applied?

from aqlaserializer.

@ab-tools

What does the serializer uses by default if no such attributes are applied?

Default comes from type settings, otherwise Reference if appropriate, otherwise MinimalEnhancement for nullables, otherwise Compact.

For collections default CollectionMode is Enhanced.

What I need to apply for something like this: public List CustomObjects { get; set; }

You may leave it by default unless you want some specific behavior. Just [SerializableMember(n)] is enough.

Do I need to apply these to all collections (List, Dictionary, ...)?

Again, only when you need some specific behavior for collection items.

If you had specified non-default settings on [SerializableMember] you will need to specify the same for all nested levels with [SerializableMemberNested(level)] as now they are not copied from [SerializableMember] anymore.

from aqlaserializer.

@AqlaSolutions: I meant what I need to apply for public List CustomObjects { get; set; } to keep it compatible with the pre-release version - I guess it's this then, right?

[SerializableMember(5, ValueFormat.MinimalEnhancement)] // List<T>
[SerializableMemberNested(1, ValueFormat.Reference)] // CustomObjects
public List<CustomObject> CustomObjects { get; set; }

And it would be great if you could write a short explanation (as I guess the Wiki is not ready yet) what the different modes are all about, especially what the mode Enhanced does?

Thank you!

from aqlaserializer.

@ab-tools such explanation is already present in xmldocs, use your IDE.

1/ Suppose you had

[SerializableMember(5)]
public List<MyObject> CustomObjects { get; set; }

You don't need to change anything here. The defaults are the same.

2/ Suppose you had

[SerializableMember(5, EnhancedFormat.Minimal, WriteAsDynamicType = true)]
public List<MyObject> CustomObjects { get; set; }

It should become now

[SerializableMember(5, ValueFormat.MinimalEnhancement, WriteAsDynamicType = true)] // List<T>
[SerializableMemberNested(1, ValueFormat.MinimalEnhancement, WriteAsDynamicType = true)] )] // CustomObjects
public List<MyObject> CustomObjects { get; set; }

* edited from LateReference as it doesn't support non-default settings on a current and nested levels.

from aqlaserializer.

I added a wiki page which describes possible member formats.

Also collection formats.

Versioning and field numbers

@ab-tools

from aqlaserializer.

@ab-tools you'll also need to set ((AutoAddStrategy)model.AutoAddStrategy).UseLegacyTupleFields = true before adding any types.

I want to ensure that everything is ok with format compatibility so I can publish the final V2 release without worrying. Please check the last build.

from aqlaserializer.

@AqlaSolutions: I'm really sorry that it takes so long, but we've just a lot of work right now directly after the release of our product. :-)

By the way, as it's released now, I can also share a link here if you would like to know where your great library is used: It's a instructor station for Prepar3D and FSX flight simulators.

I'll try to give your latest version a try today evening and will post the results then!

from aqlaserializer.

@ab-tools Interesting, I downloaded your app, it looks like it doesn't detect trial FSX but loads database when I specify directories manually. So it's like a realtime mission/map editor, right?

Please check the format compatibility asap (if you still need it) because it might be impossible to change after the final release of V2 and I have everything ready for the release now.

If you see any problems try to prepare small examples of incompatible cases. Also you can use model.GetDebugSchema method to retrieve actual format settings.

from aqlaserializer.

@AqlaSolutions: Again sorry for the delay, I just started to test already and will send you feedback definitely today (probably in the next few hours already).

By the way, where you downloaded FSX trial from?
Normally it should detect that as well.

from aqlaserializer.

@AqlaSolutions: I just gave it a try now and only changed two occurances of EnhancedFormat.LateBinding to ValueFormat.LateBinding.

When I run that now, I get following exception:

Internal error; a missing key occurred
   at AqlaSerializer.NetObjectCache.GetKeyedObject(Int32 key) in D:\My Documents\Aqla\AqlaSerializer\src\protobuf-net\NetObjectCache.cs:line 38
   at AqlaSerializer.NetObjectHelpers.ReadNetObject_Start(Object& value, ProtoReader source, Type& type, NetObjectOptions options, Boolean& isDynamic, Boolean& isLateReference, Int32& typeKey, Int32& newObjectKey, Int32& newTypeRefKey, Boolean& shouldEnd) in D:\My Documents\Aqla\AqlaSerializer\src\protobuf-net\NetObjectHelpers.Read.cs:line 53
   at proto_23(Object , ProtoReader )
   at AqlaSerializer.Serializers.CompiledSerializer.AqlaSerializer.Serializers.IProtoSerializer.Read(Object value, ProtoReader source) in D:\My Documents\Aqla\AqlaSerializer\src\protobuf-net\Serializers\CompiledSerializer.cs:line 71
   at AqlaSerializer.Meta.RuntimeTypeModel.Deserialize(Int32 key, Object value, ProtoReader source, Boolean isRoot) in D:\My Documents\Aqla\AqlaSerializer\src\protobuf-net\Meta\RuntimeTypeModel.cs:line 1098
   at AqlaSerializer.ProtoReader.ReadTypedObject(Object value, Int32 key, ProtoReader reader, Type type) in D:\My Documents\Aqla\AqlaSerializer\src\protobuf-net\ProtoReader.cs:line 647
   at AqlaSerializer.ProtoReader.ReadObject(Object value, Int32 key, ProtoReader reader) in D:\My Documents\Aqla\AqlaSerializer\src\protobuf-net\ProtoReader.cs:line 633
   at proto_58(Object , ProtoReader )
   at AqlaSerializer.Serializers.CompiledSerializer.AqlaSerializer.Serializers.IProtoSerializer.Read(Object value, ProtoReader source) in D:\My Documents\Aqla\AqlaSerializer\src\protobuf-net\Serializers\CompiledSerializer.cs:line 71
   at AqlaSerializer.Meta.RuntimeTypeModel.Deserialize(Int32 key, Object value, ProtoReader source, Boolean isRoot) in D:\My Documents\Aqla\AqlaSerializer\src\protobuf-net\Meta\RuntimeTypeModel.cs:line 1098
   at AqlaSerializer.ProtoReader.ReadTypedObject(Object value, Int32 key, ProtoReader reader, Type type) in D:\My Documents\Aqla\AqlaSerializer\src\protobuf-net\ProtoReader.cs:line 647
   at AqlaSerializer.ProtoReader.ReadObject(Object value, Int32 key, ProtoReader reader) in D:\My Documents\Aqla\AqlaSerializer\src\protobuf-net\ProtoReader.cs:line 633
   at proto_4(Object , ProtoReader )
   at AqlaSerializer.Serializers.CompiledSerializer.AqlaSerializer.Serializers.IProtoSerializer.Read(Object value, ProtoReader source) in D:\My Documents\Aqla\AqlaSerializer\src\protobuf-net\Serializers\CompiledSerializer.cs:line 71
   at AqlaSerializer.Meta.RuntimeTypeModel.Deserialize(Int32 key, Object value, ProtoReader source, Boolean isRoot) in D:\My Documents\Aqla\AqlaSerializer\src\protobuf-net\Meta\RuntimeTypeModel.cs:line 1098
   at AqlaSerializer.ProtoReader.ReadTypedObject(Object value, Int32 key, ProtoReader reader, Type type) in D:\My Documents\Aqla\AqlaSerializer\src\protobuf-net\ProtoReader.cs:line 647
   at AqlaSerializer.ProtoReader.ReadObject(Object value, Int32 key, ProtoReader reader) in D:\My Documents\Aqla\AqlaSerializer\src\protobuf-net\ProtoReader.cs:line 633
   at proto_252(Object , ProtoReader )
   at AqlaSerializer.Serializers.CompiledSerializer.AqlaSerializer.Serializers.IProtoSerializer.Read(Object value, ProtoReader source) in D:\My Documents\Aqla\AqlaSerializer\src\protobuf-net\Serializers\CompiledSerializer.cs:line 71
   at AqlaSerializer.Meta.RuntimeTypeModel.Deserialize(Int32 key, Object value, ProtoReader source, Boolean isRoot) in D:\My Documents\Aqla\AqlaSerializer\src\protobuf-net\Meta\RuntimeTypeModel.cs:line 1098
   at AqlaSerializer.Meta.TypeModel.DeserializeCore(ProtoReader reader, Type type, Object value, Boolean noAutoCreate, Boolean isRoot) in D:\My Documents\Aqla\AqlaSerializer\src\protobuf-net\Meta\TypeModel.cs:line 793
   at AqlaSerializer.Meta.TypeModel.Deserialize(Stream source, Object value, Type type, Int32 length, SerializationContext context) in D:\My Documents\Aqla\AqlaSerializer\src\protobuf-net\Meta\TypeModel.cs:line 752
   at AqlaSerializer.Serializer.Deserialize[T](Stream source) in D:\My Documents\Aqla\AqlaSerializer\src\protobuf-net\Serializer.cs:line 111
   at FlightControl.Program.loadSettings(Boolean retry) in C:\Austausch\#FS\FS-Control\Program.cs:line 557

I did not use something like EnhancedFormat.Minimal, so I'm not sure what I need to change additionally to make it compatible again.

Do you have any idea what I should search for?

Thanks
Andreas

from aqlaserializer.

@ab-tools I used FSX from https://www.microsoft.com/Products/Games/FSInsider/downloads/Pages/FlightSimulatorXTrialVersion.aspx installed to a non-system drive.

Try to compare debug schemes with the latest AqlaSerializer and with alpha ( I ported debug schema over alpha: https://www.dropbox.com/s/hm9h7r9gxah0p3r/aqlaserializer%20alpha-r90.zip?dl=1 ).

model.GetDebugSchema(typeof(Database<IDatabase>);

Changing between Reference and MinimalEnhancement is ok.
Changing between Reference and LateReference is ok.
WithNullWireType format can read the alpha format (but not in opposite).
In collections PackedRead doesn't have an effect when combined with NewPacked.

I replaced attributes in the model you emailed me before and according to debug schemes everything should be compatible there if you have the same mapping as I am.

Feel free to send me the schemes if you need a help.

from aqlaserializer.

@AqlaSolutions: Just generated debug schemas for all objects that are used with AqlaSerializer for comparison and there are several differences.

Mainly WithNullWireType and AsReference are sometimes present, sometimes now.

Would be great if you could have a look - I sent them to you via e-mail.

Best regards and thanks a lot
Andreas

from aqlaserializer.

@ab-tools

Are you using the latest release?

Map - should be compatible.

Settings and Database:

Seems like you've forgotten to set

((AutoAddStrategy)model.AutoAddStrategy).UseLegacyTupleFields = true 

before adding any types.

Field #2 of tuple KeyValuePair[String,Dictionary[String,String]]: is written with NetObject in release. Same for KeyValuePair[String,Dictionary[String,Single]] and several other tuples.

from aqlaserializer.

@AqlaSolutions: Oh, yes, you're right - I just forgot the UseLegacyTupleFields setting.

After this was added, it reads the old data perfectly fine!

By the way, is it possible to enable this lagacy tuple field mode only for reading?
Just asking as I'm only using AqlaSerializer. This way it only needs to read the "old" format once, but starting from there it could write with the "new" format without problems.

Best regards
Andreas

from aqlaserializer.

@ab-tools you can use two models - with and without UseLegacyTupleFields. Just make sure you setup UseLegacyTupleFields before any types are added to model.

from aqlaserializer.

@AqlaSolutions: OK, I just left it as it is now with UseLegacyTupleFields as it works very well - reading and writing - in my further tests. So no problems here with the new version, thanks! :-)

By the way, we checked and this quite old demo version of FSX is indeed not fully compatible, but with the latest program version from our website you can at least build the database (=> use AqlaSerializer ;-) ) from the FSX Demo, but connecting to the flight simulator afterwards won't be possible.

from aqlaserializer.