aqlasolutions / aqlaserializer Goto Github PK

View Code? Open in Web Editor NEW

16.0 3.0 3.0 52.27 MB

Binary serializer with full .NET support!

Home Page: http://www.aqla.net

License: Other

CSS 0.01% C# 99.87% Batchfile 0.01% Smalltalk 0.10% ASP.NET 0.01%

serializer serialization protocol-buffers protobuf protobuf-net

aqlaserializer's Introduction

AqlaSerializer

It is a fast and portable binary serializer designed to be easily used on your existing code with minimal changes on a wide range of .NET platforms. With AqlaSerializer you can store objects as a small in size binary data (far smaller than xml). And it's more CPU effective than BinaryFormatter and other core .NET serializers (which could be unavailable on your target platform).

Basically this is a fork of well known data serializer protobuf-net.

AqlaSerializer is an object serializer, it's primary goal is to support important .NET features like nested collections, references, etc. See also comparison with protobuf-net page. And it still supports Google Protocol Buffers format in compatibility mode.

It is a free open source project in which you can participiate.

The implementation is compatible with most of the .NET family, including .NET 3.5/4.0/4.5, .NET Standard 2.1 (.NET Core 3/3.1, .NET 5, .NET 6), Windows Phone 8, Silverlight 5, Android, iOS, UAP. The code is heavily based on Marc Gravell's protobuf-net but there are a lot of improvements and fixes.

Nuget: aqlaserializer.

License is same as for protobuf-net.

Usage

The usage is very simple; at the most basic level, simply read from a stream or write to a stream:

[SerializableType]
class Person 
{
    public int Id { get; set; }
    public string Name { get; set: }
}


Serializer.Serialize(outputStream, person);

...

var person = Serializer.Deserialize<Person>(inputStream);

Read Getting started

We need your help

I'm trying to merge the current state of protobuf-net main branch but it's a lot of work. If you'd like to use the newest features of protobuf-net like proto3 support on AqlaSerializer please either contribute to the merging process by issuing a pull request or by a BTC donation so I can put more time myself into doing the merge. You can also help by spreading the word about this project so it can have more attention.

Currently I'm not much into .NET development and doing the merge just because I want to keep AqlaSerializer for the community even when I'm not actively using the serializer anymore. Nowadays performance is often neglected by using text-based serialization like json, xml and pure reflection. I want to improve this situation with AqlaSerializer. I don't know about a better alternative that is fast (I mean really fast), binary, supports objects graphs and works on AOT platforms (not mentioning other improvements) so I want to keep it up to date with the newest technologies like .NET 5, C#9 and proto3. But the lack of interest and support from the community takes away my motivation.

Note that the current version of AqlaSerializer is still viable to use. It doesn't contain the newest features and improvements from protobuf-net but remember that AqlaSerializer have a lot of its own stuff like well-done reference tracking which is unavailable in protobuf-net.

Merging details

aqlaserializer's People

Contributors

Stargazers

Watchers

Forkers

robin-- jdsheneman sunguangdong

aqlaserializer's Issues

How to support inheritance of a class and an interface

I have something like this:
public class ClassA : ClassB, Interface1

How to model this? Does Aqla support this case?

Possible recursion detected

I got the error message when I tried to deep clone an object twice:
Possible recursion detected (offset: 8 level(s)): []

The first deep clone was successful, I checked the object's fields and everything seems to be ok. However, if I deep clone the same object again, I got an exception with the above error message.

The recursion stack dump is like this:

Stack:
[]
[]
[]
[]
[]
[]
[]
[]
[]

What does this error mean and how can I trouble-shoot it?

Thanks

Progress Callback

As it's not totally related and more like a question, I thought I better open another issue here for that.

With the bigger database I have - even with the older, faster version - it takes about 30 sec. to serialize. That's not a problem to let the user wait for, but it would be great to be able to show some sort of progress to the user.

Is there any chance that AqlaSerializer could send a callback with some progress update like 1,000 of 10,000 objects done or similar?

Although I'm afraid that it won't be easily possible due to the stream serialization structure of protobuf, but I thought I just ask anyway - maybe you have an idea about it. :-)

Exception When Trying to Serialize a Bigger Graph Object

As written in the protobuf-net project already, I can reproduce an exception when trying to serialize a bigger (about 50 MB) graph object with AqlaSerializer.

To reproduce it I need to include several classes from my project which I don't want to publish here. Therefore I allowed myself to send you an e-mail to the address I found on your "Aqla Solutions" website.

Please just start this example project, click on "Load Data With Protobuf" and after it's done, click on "Save Data With AqlaSerializer" => Exception.

I really would like to get rid of Protobuf as this project seems to be dead.
So I'm very much looking forward to your reply!

Thanks in advance
Andreas

'Non-public member cannot be used with full dll compilation

Got another weird issue: 'Non-public member cannot be used with full dll compilation.

This is the code, the "DeepClone" fails

        [Fact]
        public void TestInstrumentGroupId()
        {
            MetaType metaType = RuntimeTypeModel.Default.Add(typeof(IInstrumentGroupId), false);
            metaType.DefaultFormat = ValueFormat.Reference;
            metaType.UseConstructor = false;
            metaType.IgnoreListHandling = true;

            MetaType metaType1 = RuntimeTypeModel.Default.Add(typeof(InstrumentGroupId), false);
            metaType1.DefaultFormat = ValueFormat.Reference;
            metaType1.UseConstructor = false;
            metaType1.IgnoreListHandling = true;

            metaType1.AddField(1, "_id");
            metaType1.AddField(2, "_portfolioType");
            metaType.AddSubType(1, typeof(InstrumentGroupId));

            RuntimeTypeModel.Default.CompileInPlace();

            var instrumentGroupId = new InstrumentGroupId(PortfolioType.MarketInstrument, 1);
            var clone = Serializer.DeepClone(instrumentGroupId);
            Assert.Equal(instrumentGroupId.Id, clone.Id);
            Assert.Equal(instrumentGroupId.PortfolioType, clone.PortfolioType);
        }

        public interface IInstrumentGroupId
        {
            PortfolioType PortfolioType { get; }
        }

        public enum PortfolioType
        {
            MinPortType = 0,
            CurrentHoldings = MinPortType,
            MarketInstrument,
            MaxPortType = MarketInstrument
        }

        public sealed class InstrumentGroupId : IInstrumentGroupId
        {
            private readonly PortfolioType _portfolioType;
            private readonly int _id;

            public InstrumentGroupId(PortfolioType portfolioType, int id)
            {
                _portfolioType = portfolioType;
                _id = id;
            }

            public PortfolioType PortfolioType
            {
                get { return _portfolioType; }
            }

            public int Id
            {
                get { return _id; }
            }
        }

Un-defined enum values

Protobuf-net allows un-defined enum values, but Aqla doesn't. The following code shows that:

        private enum Number { One, Two, Three }

        [Fact]
        public void TestNotDefinedEnum()
        {
            Number four = (Number)4;
         
            var clone = ProtoBuf.Serializer.DeepClone(four);
            Assert.Equal(clone, four);

            var clone2 = AqlaSerializer.Serializer.DeepClone(four);
            Assert.Equal(clone2, four);
        }

Is there a way to support undefined enum values with Aqla?

Virtual members of AutoAddStrategy should be called only when it makes sense

Hello!

The exception is thrown when I try to serialize 2D array (in that case it's double[,]):

Data of this type has inbuilt behaviour, and cannot be added to a model in this way: System.Double
Stacktrace:

>	aqlaserializer.dll!AqlaSerializer.Meta.RuntimeTypeModel.FindOrAddAuto(System.Type type, bool demand, bool addWithContractOnly, bool addEvenIfAutoDisabled, out AqlaSerializer.Meta.MetaType metaType) Line 429	C#
 	aqlaserializer.dll!AqlaSerializer.Meta.RuntimeTypeModel.FindOrAddAuto(System.Type type, bool demand, bool addWithContractOnly, bool addEvenIfAutoDisabled) Line 405	C#
 	aqlaserializer.dll!AqlaSerializer.AutoAddStrategy.ApplyDefaultBehaviour(AqlaSerializer.Meta.MetaType metaType) Line 294	C#
 	aqlaserializer.dll!AqlaSerializer.Meta.MetaType.ApplyDefaultBehaviour() Line 137	C#
 	aqlaserializer.dll!AqlaSerializer.Meta.RuntimeTypeModel.FindOrAddAuto(System.Type type, bool demand, bool addWithContractOnly, bool addEvenIfAutoDisabled, out AqlaSerializer.Meta.MetaType metaType) Line 498	C#
 	aqlaserializer.dll!AqlaSerializer.Meta.RuntimeTypeModel.FindOrAddAuto(System.Type type, bool demand, bool addWithContractOnly, bool addEvenIfAutoDisabled) Line 405	C#
 	aqlaserializer.dll!AqlaSerializer.Meta.RuntimeTypeModel.GetKey(System.Type type, bool demand, bool getBaseKey) Line 895	C#
 	aqlaserializer.dll!AqlaSerializer.Meta.RuntimeTypeModel.GetKeyImpl(System.Type type) Line 885	C#
 	aqlaserializer.dll!AqlaSerializer.Meta.TypeModel.GetKey(ref System.Type type) Line 1120	C#
 	aqlaserializer.dll!AqlaSerializer.Meta.TypeModel.SerializeWithLengthPrefix(System.IO.Stream dest, object value, System.Type type, AqlaSerializer.PrefixStyle style, int fieldNumber, AqlaSerializer.SerializationContext context) Line 713	C#

Any idea how it can be fixed?

Regards!

Does class fields get serialized by default?

According to the wiki, public class properties are serialized by default. How about class fields? Do we need to mark them with protobuf attribute?

Custom / manual serializing API?

Hello!

I'm currently working on a network system for delta-synchronization of objects from application server to client. It packs every property modification into a message and sends it to client which is then applied it to fully replicate the state of the object on the server side.

Background.
Every synchronizable object should implement BaseNetObject. The modifications are determined by injecting OnSyncPropertyChange(propertyName, value) call into each property marked with [SyncToClient] attribute, so when it's modified the method is called and modification data is generated. I'm using Roslyn (.NET Compiler platform) for injecting and compilation. Some synchronized properties could contain reference types (they're required to implement BaseNetObject) and they could be synchronized the same way. And there are also network collections types (List & Dictionary) inherited from BaseNetObject (so they synchronize every modification).

The question.
Usually my system need to send full object, but sometimes it need to translate only its surrogate (because the object is already known to the client and it should use already created instance of it). For example:

public class State: BaseNetObject
{
  [SyncToClient]
  public SomeDeferredNetObject Prop1 { get; set; }

  [SyncToClient]
  public SomeDeferredNetObject Prop2 { get; set; }
}

public class SomeDeferredNetObject: BaseNetObject
{
  [SyncToClient]
  public SomeDeferredNetObject SomeProperty { get; set; }
}

public void Scenario1(State state)
{
   // this works ok using surrogates
   state.Prop2.SomeProperty = state.Prop1;
}

public void Scenario2(State state)
{
   var newObj = new SomeDeferredNetObject() { SomeProperty = state.Prop1 } ;
   // here the problem goes
   state.Prop2 = newObj;
}

The problem is shown in Scenario2() method:

I cannot use surrogates approach in that case, because it will serialize modification of Prop2 property by using surrogate (so it will only send NetObject ID, which will be unknown to client).
I also cannot use full-object approach, because it will serialize Prop2.SomeProperty by not using surrogate and it will be deserialized on client-side into the new object (when it should be "surrogated" to NetObject ID and client could use already known instance).

So, I want to be able to manually (by context?) determine during serialization when to use surrogate and when to use full object serialization. How I could accomplish this with AqlaSerializer?

Regards!

Is nuget package 2.0.1 the latest version?

We get the nuget package 2.0.1, but it doesn't seem to have the latest enhancements. For example, it doesn't support IReadOnlyDictionary. Can you please put the latest version in nuget?

Enum With Duplicate Items

Does Aqla support enum with duplicate items? The following test fails on the assertion.

        enum EnumWithDuplicateItems
        {
            Min = 0,
            Current = Min,
            Market,
            Max = Market,
        }

        [Test]
        public void TestEnumWithDuplicateItems()
        {
            var clone = Serializer.DeepClone(EnumWithDuplicateItems.Market);
            Assert.Equal(EnumWithDuplicateItems.Market, clone);
        }

A deferred key does not have a value yet (NoteObject call missed?)

The following test throws "AqlaSerializer.ProtoException : A deferred key does not have a value yet (NoteObject call missed?)". Also, the method "CoreTableSurrogate.ToSurrogate" is called twice, the first call with non-null "coreTable" parameter, the second call with null "coreTable" parameter. It seems that the result of the second call is passed to "CoreTableSurrogate.FromSurrogate".

Any idea why it behaves like this? Thanks

using ProtoBuf;
using Xunit;

namespace PrototypeTests
{
    public class UnitTest
    {
        private class CoreTable
        {
            public static readonly CoreTable DefaultCoreTable = new CoreTable();
        }

        private class WithCoreTable
        {
            public CoreTable Node { get; set; }
        }

        private void SetupAqla()
        {
            var metaType = AqlaSerializer.Meta.RuntimeTypeModel.Default.Add(typeof(WithCoreTable), false);
            metaType.DefaultFormat = AqlaSerializer.ValueFormat.Reference;
            metaType.UseConstructor = false;

            var metaField = metaType.AddField(metaType.GetNextFreeFieldNumber(), "Node");
            metaField.SetSettings(x => x.V.Format = AqlaSerializer.ValueFormat.Reference);

            var metaType2 = AqlaSerializer.Meta.RuntimeTypeModel.Default[typeof(CoreTable)];
            metaType2.SetSurrogate(typeof(CoreTableSurrogate));

            var metaType3 = AqlaSerializer.Meta.RuntimeTypeModel.Default.Add(typeof(CoreTableSurrogate), false);
            metaType3.DefaultFormat = AqlaSerializer.ValueFormat.Reference;
            metaType3.UseConstructor = false;

            var metaField2 = metaType3.AddField(metaType3.GetNextFreeFieldNumber(), "IsCoreTableNull");
            metaField2.SetSettings(x => x.V.Format = AqlaSerializer.ValueFormat.Compact);

            var metaField3 = metaType3.AddField(metaType3.GetNextFreeFieldNumber(), "ObjectId");
            metaField3.SetSettings(x => x.V.Format = AqlaSerializer.ValueFormat.Compact);
        }

        [Fact]
        public void Test()
        {
            SetupAqla();

            var withCoreTable = new WithCoreTable { Node = new CoreTable() };
            var clone = AqlaSerializer.Serializer.DeepClone(withCoreTable);
            Assert.NotNull(clone.Node);
        }

        private class CoreTableSurrogate
        {
            private static int nextObjectId = 1;

            public bool IsCoreTableNull { get; set; }
            public int ObjectId { get; set; }

            [ProtoConverter]
            public static CoreTable FromSurrogate(CoreTableSurrogate surrogate)
            {
                if (surrogate == null)
                    return null;

                return surrogate.IsCoreTableNull ? null : new CoreTable();
            }

            [ProtoConverter]
            public static CoreTableSurrogate ToSurrogate(CoreTable coreTable)
            {
                var surrogate = new CoreTableSurrogate();
                surrogate.IsCoreTableNull = coreTable == null;
                surrogate.ObjectId = nextObjectId++;

                return surrogate;
            }
        }
    }
}

Can I serialize a tuple?

Hello together,
I had tried to serialize a simple tuple in a small test, but did not get the wanted result when deserializing. Is it possible to serialize a tuple?

namespace SerializerTest
{
    public class SerializerTest
    {
        [Fact]
        public void Can_serialize_object_with_tuple()
        {
            Tuple<int, string> tuple = new Tuple<int, string>(1, "abc");
            ClassWithTupleMember objectWithTuple = new ClassWithTupleMember() { Tuple = tuple };
            byte[] serializedObject = Serialize<ClassWithTupleMember>(objectWithTuple);
            var deserializedObject = Deserialize<ClassWithTupleMember>(serializedObject);

            deserializedObject.Tuple.ShouldBe(tuple);
        }

        public static byte[] Serialize<T>(T objectToSerialize) where T : class
        {
            if (objectToSerialize is null) return null;

            var memoryStream = new MemoryStream();
            global::AqlaSerializer.Serializer.Serialize<T>(memoryStream, objectToSerialize);
            return memoryStream.ToArray();
        }

        public static T Deserialize<T>(byte[] serializedMessage) where T : class
        {
            if (serializedMessage is null) return default;

            try
            {
                var memoryStream = new MemoryStream(serializedMessage) { Position = 0 };
                return global::AqlaSerializer.Serializer.Deserialize<T>(memoryStream);
            }
            catch (Exception)
            {
                return (T)Activator.CreateInstance(typeof(T));
            }
        }
    }

    [SerializableType()]
    public class ClassWithTupleMember
    {
        public Tuple<int, string> Tuple;
    }
}

How to setup Aqla for complicated object relationships

I have classes with the relationships depicted in the attached class diagram:

How to setup Aqla for those classes? It seems that the straightforward setup for each relationship doesn't work. Our code assigns an instance of class "NameAndId" to "Property._owner", which is type "INameAndId". When we try to serialize "Property" object, we get exception saying unexpected sub-type "NameAndId" for "Property._owner". "Property._owner" is defined as a dynamic type.

Any suggestions? Thanks

Cannot compile latest source from Master

Hello!

I just pulled Master HEAD and cannot build it:

https://raw.githubusercontent.com/AqlaSolutions/AqlaSerializer/master/protobuf-net/Serializers/WillSetFieldWireType/Collections/MultiDimensionalArrayDecorator.cs
Error on line 309

    g.If(valueArr.AsOperand != null && optionalOutOldFirstDimLength > 0);

Error message:

In order to be applicable as a short circuit operator a user-defined logical operator ('ContextualOperand.operator &(ContextualOperand, Operand)') must have the same return type and parameter type.

(FEAT_COMPILER conditional compilation symbol is used)

Regards!

Non-public member cannot be used with full dll compilation

I got the error in our tests:
Non-public member cannot be used with full dll compilation

I didn't call the "Compile" method, how could the "full dll compilation" being triggered? Any idea what could be wrong? The same code works fine with Protobuf-net 2.4.6.

Thanks

This is the call stack:

AqlaSerializer.Compiler.CompilerContext.CheckAccessibility(member)
AqlaSerializer.Compiler.CompilerContext.LoadValue(field)
AqlaSerializer.Serializers.FieldDecorator.EmitWrite(ctx, valueFrom)
AqlaSerializer.Serializers.ProtoDecoratorBase.AqlaSerializer.Serializers.IProtoSerializer.EmitWrite(ctx, valueFrom)
AqlaSerializer.Serializers.TypeSerializer.AqlaSerializer.Serializers.IProtoSerializer.EmitWrite(ctx, valueFrom)
AqlaSerializer.Serializers.CompiledSerializer.AqlaSerializer.Serializers.IProtoSerializer.EmitWrite(ctx, valueFrom)
AqlaSerializer.Meta.RuntimeTypeModel.WriteSerializers.AnonymousMethod__0(pair)
AqlaSerializer.Meta.RuntimeTypeModel.WriteSerializers.AnonymousMethod__1()
AqlaSerializer.Meta.RuntimeTypeModel.WriteSerializers(options, assemblyName, type, hasInheritance, methodPairs, ilVersion)
AqlaSerializer.Meta.RuntimeTypeModel.Compile(options)
AqlaSerializer.Meta.RuntimeTypeModel.Compile(name, path)
AqlaSerializer.Meta.RuntimeTypeModel.CompileForCheckAndValidate(rtm)
AqlaSerializer.Meta.RuntimeTypeModel.GetInvertedVersionForCheckCompiledVsNot(key, metaType)
AqlaSerializer.Meta.RuntimeTypeModel.Serialize(key, value, dest, isRoot)
AqlaSerializer.Meta.TypeModel.SerializeCore(writer, value, isRoot)
AqlaSerializer.Meta.TypeModel.Serialize(dest, value, context)
AqlaSerializer.Meta.TypeModel.Serialize(dest, value)

A deferred key does not have a value yet (NoteObject call missed?)

I got the following exception when trying to deserialize an object, any idea what's wrong?

AqlaSerializer.ProtoException
  HResult=0x80131500
  Message=A deferred key does not have a value yet (NoteObject call missed?)
  Source=aqlaserializer
  StackTrace:
   at AqlaSerializer.NetObjectCache.GetKeyedObject(Int32 key, Boolean allowMissing) in D:\AqlaSerializer-v.2.0.1.1031\src\protobuf-net\NetObjectCache.cs:line 49
   at AqlaSerializer.NetObjectHelpers.ReadNetObject_End(Object value, ReadReturnValue r, ProtoReader source, Object oldValue, Type type, NetObjectOptions options) in D:\AqlaSerializer-v.2.0.1.1031\src\protobuf-net\NetObjectHelpers.Read.cs:line 37
   at AqlaSerializer.Serializers.CompiledSerializer.AqlaSerializer.Serializers.IProtoSerializer.Read(Object value, ProtoReader source) in D:\AqlaSerializer-v.2.0.1.1031\src\protobuf-net\Serializers\CompiledSerializer.cs:line 74
   at AqlaSerializer.Meta.RuntimeTypeModel.Deserialize(Int32 key, Object value, ProtoReader source, Boolean isRoot) in D:\AqlaSerializer-v.2.0.1.1031\src\protobuf-net\Meta\RuntimeTypeModel.cs:line 1080
   at AqlaSerializer.ProtoReader.ReadTypedObject(Object value, Int32 key, ProtoReader reader, Type type) in D:\AqlaSerializer-v.2.0.1.1031\src\protobuf-net\ProtoReader.cs:line 787
   at AqlaSerializer.ProtoReader.ReadObject(Object value, Int32 key, ProtoReader reader) in D:\AqlaSerializer-v.2.0.1.1031\src\protobuf-net\ProtoReader.cs:line 773
   at AqlaSerializer.Serializers.CompiledSerializer.AqlaSerializer.Serializers.IProtoSerializer.Read(Object value, ProtoReader source) in D:\AqlaSerializer-v.2.0.1.1031\src\protobuf-net\Serializers\CompiledSerializer.cs:line 74
   at AqlaSerializer.Meta.RuntimeTypeModel.Deserialize(Int32 key, Object value, ProtoReader source, Boolean isRoot) in D:\AqlaSerializer-v.2.0.1.1031\src\protobuf-net\Meta\RuntimeTypeModel.cs:line 1080
   at AqlaSerializer.ProtoReader.ReadTypedObject(Object value, Int32 key, ProtoReader reader, Type type) in D:\AqlaSerializer-v.2.0.1.1031\src\protobuf-net\ProtoReader.cs:line 787
   at AqlaSerializer.ProtoReader.ReadObject(Object value, Int32 key, ProtoReader reader) in D:\AqlaSerializer-v.2.0.1.1031\src\protobuf-net\ProtoReader.cs:line 773
   at AqlaSerializer.Serializers.CompiledSerializer.AqlaSerializer.Serializers.IProtoSerializer.Read(Object value, ProtoReader source) in D:\AqlaSerializer-v.2.0.1.1031\src\protobuf-net\Serializers\CompiledSerializer.cs:line 74
   at AqlaSerializer.Meta.RuntimeTypeModel.Deserialize(Int32 key, Object value, ProtoReader source, Boolean isRoot) in D:\AqlaSerializer-v.2.0.1.1031\src\protobuf-net\Meta\RuntimeTypeModel.cs:line 1080
   at AqlaSerializer.ProtoReader.ReadTypedObject(Object value, Int32 key, ProtoReader reader, Type type) in D:\AqlaSerializer-v.2.0.1.1031\src\protobuf-net\ProtoReader.cs:line 787
   at AqlaSerializer.ProtoReader.ReadObject(Object value, Int32 key, ProtoReader reader) in D:\AqlaSerializer-v.2.0.1.1031\src\protobuf-net\ProtoReader.cs:line 773
   at AqlaSerializer.Serializers.CompiledSerializer.AqlaSerializer.Serializers.IProtoSerializer.Read(Object value, ProtoReader source) in D:\AqlaSerializer-v.2.0.1.1031\src\protobuf-net\Serializers\CompiledSerializer.cs:line 74
   at AqlaSerializer.Meta.RuntimeTypeModel.Deserialize(Int32 key, Object value, ProtoReader source, Boolean isRoot) in D:\AqlaSerializer-v.2.0.1.1031\src\protobuf-net\Meta\RuntimeTypeModel.cs:line 1080
   at AqlaSerializer.ProtoReader.ReadTypedObject(Object value, Int32 key, ProtoReader reader, Type type) in D:\AqlaSerializer-v.2.0.1.1031\src\protobuf-net\ProtoReader.cs:line 787
   at AqlaSerializer.ProtoReader.ReadObject(Object value, Int32 key, ProtoReader reader) in D:\AqlaSerializer-v.2.0.1.1031\src\protobuf-net\ProtoReader.cs:line 773
   at AqlaSerializer.Serializers.CompiledSerializer.AqlaSerializer.Serializers.IProtoSerializer.Read(Object value, ProtoReader source) in D:\AqlaSerializer-v.2.0.1.1031\src\protobuf-net\Serializers\CompiledSerializer.cs:line 74
   at AqlaSerializer.Meta.RuntimeTypeModel.Deserialize(Int32 key, Object value, ProtoReader source, Boolean isRoot) in D:\AqlaSerializer-v.2.0.1.1031\src\protobuf-net\Meta\RuntimeTypeModel.cs:line 1080
   at AqlaSerializer.Meta.TypeModel.DeserializeCore(ProtoReader reader, Type type, Object value, Boolean noAutoCreate, Boolean isRoot) in D:\AqlaSerializer-v.2.0.1.1031\src\protobuf-net\Meta\TypeModel.cs:line 819
   at AqlaSerializer.Meta.TypeModel.Deserialize(Stream source, Object value, Type type, Int64 length, SerializationContext context) in D:\AqlaSerializer-v.2.0.1.1031\src\protobuf-net\Meta\TypeModel.cs:line 779
   at AqlaSerializer.Meta.TypeModel.Deserialize(Stream source, Object value, Type type, SerializationContext context) in D:\AqlaSerializer-v.2.0.1.1031\src\protobuf-net\Meta\TypeModel.cs:line 703
   at AqlaSerializer.Meta.TypeModel.Deserialize(Stream source, Object value, Type type) in D:\AqlaSerializer-v.2.0.1.1031\src\protobuf-net\Meta\TypeModel.cs:line 689
   at AqlaSerializer.Serializer.Deserialize(Type type, Stream source) in D:\AqlaSerializer-v.2.0.1.1031\src\protobuf-net\Serializer.cs:line 123
 
  This exception was originally thrown at this call stack:
    AqlaSerializer.NetObjectCache.GetKeyedObject(int, bool) in NetObjectCache.cs
    AqlaSerializer.NetObjectHelpers.ReadNetObject_End(object, AqlaSerializer.NetObjectHelpers.ReadReturnValue, AqlaSerializer.ProtoReader, object, System.Type, AqlaSerializer.BclHelpers.NetObjectOptions) in NetObjectHelpers.Read.cs
    AqlaSerializer.Serializers.CompiledSerializer.AqlaSerializer.Serializers.IProtoSerializer.Read(object, AqlaSerializer.ProtoReader) in CompiledSerializer.cs
    AqlaSerializer.Meta.RuntimeTypeModel.Deserialize(int, object, AqlaSerializer.ProtoReader, bool) in RuntimeTypeModel.cs
    AqlaSerializer.ProtoReader.ReadTypedObject(object, int, AqlaSerializer.ProtoReader, System.Type) in ProtoReader.cs
    AqlaSerializer.ProtoReader.ReadObject(object, int, AqlaSerializer.ProtoReader) in ProtoReader.cs
    AqlaSerializer.Serializers.CompiledSerializer.AqlaSerializer.Serializers.IProtoSerializer.Read(object, AqlaSerializer.ProtoReader) in CompiledSerializer.cs
    AqlaSerializer.Meta.RuntimeTypeModel.Deserialize(int, object, AqlaSerializer.ProtoReader, bool) in RuntimeTypeModel.cs
    AqlaSerializer.ProtoReader.ReadTypedObject(object, int, AqlaSerializer.ProtoReader, System.Type) in ProtoReader.cs
    AqlaSerializer.ProtoReader.ReadObject(object, int, AqlaSerializer.ProtoReader) in ProtoReader.cs
    ...

Can you please publish the latest code?

I need the fix of issue:
"Unable to determine member" when running obfuscated code

Could you please publish the latest code which has the fix of the above issue?

Thanks

Dynamic type is not a contract-type

I got the following error during our tests:
Dynamic type is not a contract-type: System.Collections.Generic.ObjectComparer1`

"ObjectComparer" is a .NET internal type, which I don't think we need to do anything about it. Any idea what could be wrong?

Thanks

DeflateStream is not supported?

The following test fails with the error: System.NotSupportedException : This operation is not supported. This code works for Protobuf-net.

        [Fact]
        public void TestCompress()
        {
            using (DeflateStream dest = new DeflateStream(new MemoryStream(), CompressionMode.Compress))
            {
                Serializer.Serialize(dest, "Test");
            }
        }

Why is ISet not supported?

The following code throws an exception complaining no default constructor. I verified that IList, IDictionary, ICollection all work fine,

        [Fact]
        public void ISetTest()
        {
            MetaType metaType = RuntimeTypeModel.Default.Add(typeof(WithISet), true);
            metaType.AddField(1, "_set").SetSettings(x => x.V.Format = ValueFormat.LateReference);

            RuntimeTypeModel.Default.CompileInPlace();
        }

        private class WithISet
        {
            public readonly ISet<int> _set;
        }

Very Long Deserialization Time on Few Systems

Hello @AqlaSolutions again!

We are still using your great serialization library, just as a reminder, we are the ones have the bigger graph database as discussed some time ago here:
#1

Everything really works well with a database file size of about 100 MB and a deserialization time of about 3-5 seconds on most modern computers which is fine.

But since with changed our product to use 64 bit if available (compiled with "Any CPU"), we do have few customers - all running Windows 7 64 bit - that complain about extremely long database loading times:
Instead of 3-5 seconds it takes about 5-10 minutes for them!

For testing we then forced our product to start in 32 bit mode (via "CorFlags.exe") and the database loads "normally" fast again in just 3-5 seconds, but when switching back to 64 bit mode it gets immediately extremely slow again.

We cannot reproduce this problem ourselves, also not on a Windows 7 64 bit system. So we are running a bit out of ideas what could really cause this issue and how to track that down.

Do you have any idea what could cause such an extreme performance drop when deserialization in a 64 bit process instead of a 32 bit process?

Thanks in advance for any ideas on that
Andreas

RuntimeTypeModel.Default.CompileInPlace is too slow

We have more than 5800 classes which are registered with "RuntimeTypeModel.Default". However if I run CompileInPlace against the model, it takes more than 3 minutes. Is there a way to speed this up?

Bug: IgnoreListHandling is not works / serializing of custom collections never invokes callbacks

Hello!
I've a custom collection class which is implements IList and it's basically just a wrapper over a list instance (and the only serialized field is the inner list).

The problem is that a custom callback delegate (metaType.Callbacks.AfterDeserialize = ...) is never called in this case. But when I remove IList from class definition it works perfectly well.

I know there is a setting for ignoring default list handling - metaType.IgnoreListHandling = true - but it doesn't resolve the issue. It seems the type continue to be handled like a collection by AqlaSerializer.
I've tried with this setting and without it and still get the same proto-scheme:

message NetworkSyncList_String {
   repeated string list = 1;
}

Regards!

How to model "Lazy" field with Aqla?

I have a class which contains a "Lazy" field, how to set it up and make the test case work? Any help is highly appreciated.

[Fact]
public void LazyField()
{
    MetaType metaType = RuntimeTypeModel.Default.Add(typeof(WithLazyField), true);
    metaType.UseConstructor = false;
    ValueMember metaField = metaType.AddField(1, "_lazyField");
    metaField.AsReference = true;

    var obj = new WithLazyField();

    var clone = Serializer.DeepClone(obj);
    Assert.Equal(obj.GetValue(), clone.GetValue());
}

public class WithLazyField
{
    private readonly Lazy<double> _lazyField;

    public WithLazyField()
    {
        _lazyField = new Lazy<double>(() => Calculate());
    }

    public double Calculate()
    {
        return 1.0;
    }

    public double GetValue()
    {
        return _lazyField.Value;
    }
}

Wrong group was ended

I got the error "Wrong group was ended" when trying to deserialize an object. I narrowed it down to a class field but don't know why this happens and how to resolve it. Any suggestions is appreciated.

Very High Footprint on Serializing Numeric 2-Dimensional Array

I've tried to use AqlaSerializer for serializing a bigger 2-dimensional array of shorts simply like:

var heightMap = new short[7201, 21601];
// (filling array)

using (var fileStream = File.Create(@"path_to_file"))
    Serializer.Serialize(fileStream, heightMap);

With an array size of 7,201 * 21,601 = 155,548,801 elements of shorts with each element using 2 bytes I would have expected a file size of 311,097,602 bytes plus a little bit overhead for the serialization information. But I was shocked about the resulting file having 1,369,649,462 bytes which is more than 4 times as much as the stored data itself needs!

What's the reason for this extremely high AqlaSerializer footprint?
Is there a way to improve this?

Of course, I could serialize/deserialize such a structure very easily myself without AqlaSerializer, but as this array is part of a bigger structure where I do use AqlaSerializer already, it would be great if I could serialize/deserialize everything with AqlaSerializer.

Best regards and thanks in advance
Andreas

Support for System.ValueTuple`

There is no support for new ValueTuple (de)serialization. I've hacked my solution based on type.FullName check (where there are checks for System.Tuple) and commenting out this code:

if (!field.IsInitOnly) return null; // all public fields must be readonly to be counted a tuple

MetaType.ArrayLengthReadLimit

how set MetaType.ArrayLengthReadLimit?

"Unable to determine member" when running obfuscated code

We got the error "Unable to determine member" when running the obfuscated code with Aqla. After some investigation we found the cause:
Obfuscation could assign the same name for a field and method. This is legal in IL, but not allowed in C#. When we register a field with Aqla by calling "metaType.AddField(Index, fieldName)", Aqla internally calls the following code:

private ValueMember AddField(int fieldNumber, string memberName, Type itemType, Type defaultType, object defaultValue)
{
    if (Type.IsArray) throw new InvalidOperationException("Can't add fields to array type");
    MemberInfo mi = null;
    MemberInfo[] members = Type.GetMember(memberName, Helpers.IsEnum(Type) ? BindingFlags.Static | BindingFlags.Public : BindingFlags.Instance | BindingFlags.Public | BindingFlags.NonPublic);
    if (members != null && members.Length == 1) mi = members[0];
    if (mi == null) throw new ArgumentException("Unable to determine member: " + memberName, nameof(memberName));

If a method and a field have the same member name (which is in our case), "Type.GetMember" will return more than 1 results. This results in "mi == null" and thus the exception "Unable to determine member".
Can Aqla be enhanced to handle this case?

Unexpected sub-type

I got this error when running the underneath code. I've been struggling on this for quite a few days. Can you take a look please and see what is wrong?

System.InvalidOperationException : Unexpected sub-type: PrototypeTests.AqlaSerializationTests+CashflowListVector2

One strange thing I noticed is that if I remove these two lines, the test passes:

var metaType28 = AqlaSerializer.Meta.RuntimeTypeModel.Default.Add(typeof(ExtraRecords2), false);
metaType27.AddSubType(1, typeof(ExtraRecords2));

using System;
using System.IO;
using Xunit;

namespace PrototypeTests;

public class AqlaSerializationTests
{
    internal class CashFlowDetails
    {
        private readonly CashflowVector _faceChange = new CashflowVector();
        private readonly ICompactableBuffer[] _nonStandardCashFlows;

        public CashFlowDetails()
        {
            _nonStandardCashFlows = new ICompactableBuffer[] { _faceChange };
        }
    }

    public interface ICompactableBuffer
    {
        int Size { get; }
    }

    private abstract class Indexable
    {
        public abstract double this[int i] { get; set; }
    }

    private class CashflowVector : Indexable, ICompactableBuffer
    {
        public override double this[int i] { get => throw new NotImplementedException(); set => throw new NotImplementedException(); }
        public int Size => throw new NotImplementedException();
    }

    private class ExtraRecords2 : ICompactableBuffer
    {
        public int Size => throw new NotImplementedException();
    }

    [Fact]
    public void Test()
    {
        SetupModel();

        var cfdtl2 = new CashFlowDetails();
        using (var stream = new MemoryStream())
        {
            AqlaSerializer.Serializer.Serialize(stream, cfdtl2);
            stream.Seek(0, SeekOrigin.Begin);
            var clone = AqlaSerializer.Serializer.Deserialize<CashFlowDetails>(stream);

            Assert.NotNull(clone);
        }
    }

    private void SetupModel()
    {
        var metaType7 = AqlaSerializer.Meta.RuntimeTypeModel.Default.Add(typeof(CashflowVector), false);
        metaType7.DefaultFormat = AqlaSerializer.ValueFormat.Reference;
        metaType7.UseConstructor = false;

        var metaType8 = AqlaSerializer.Meta.RuntimeTypeModel.Default.Add(typeof(Indexable), false);
        metaType8.DefaultFormat = AqlaSerializer.ValueFormat.Reference;
        metaType8.UseConstructor = false;

        metaType8.AddSubType(1, typeof(CashflowVector));

        var metaType27 = AqlaSerializer.Meta.RuntimeTypeModel.Default.Add(typeof(ICompactableBuffer), false);
        metaType27.DefaultFormat = AqlaSerializer.ValueFormat.Reference;
        metaType27.UseConstructor = false;

        var metaType28 = AqlaSerializer.Meta.RuntimeTypeModel.Default.Add(typeof(ExtraRecords2), false);
        metaType27.AddSubType(1, typeof(ExtraRecords2));

        var metaType57 = AqlaSerializer.Meta.RuntimeTypeModel.Default.Add(typeof(CashFlowDetails), false);
        metaType57.DefaultFormat = AqlaSerializer.ValueFormat.Reference;
        metaType57.UseConstructor = false;

        AqlaSerializer.Meta.MetaType metaType58 = AqlaSerializer.Meta.RuntimeTypeModel.Default[typeof(CashFlowDetails)];
        var metaField35 = metaType58.AddField(1, "_nonStandardCashFlows");
        metaField35.SetSettings(x => { x.V.Format = AqlaSerializer.ValueFormat.Reference; });
        metaField35.SetSettings(x => x.V.WriteAsDynamicType = true, 0);
        AqlaSerializer.Meta.MetaType metaType59 = AqlaSerializer.Meta.RuntimeTypeModel.Default[typeof(CashFlowDetails)];
        var metaField36 = metaType59.AddField(2, "_faceChange");
        metaField36.SetSettings(x => { x.V.Format = AqlaSerializer.ValueFormat.Reference; });
    }
}

How to unregister or re-register a type?

In unit tests, each test may want to setup their own types. The default model contains configuration for all types, which doesn't fit into the case of unit tests. Is there a way to erase everything registered, or override the existing registration with a different configuration?

Problem with serializing nullable enum value

Hello!

I've noticed than when we serialize nullable enum value with null value and casted to object we have a NullReferenceException.
I mean something like:

FooEnum? nullableEnumValue = null;
var obj = (object)nullableEnumValue;
this.Model.SerializeWithLengthPrefix(
				stream,
				obj,
				typeof(FooEnum?),
				PrefixStyle.Base128,
				0,
				serializationContext);

Could you check this please?

TODO

Merging protobuf-net changes
No-versioning mode for most compact output size.
Delta synchronization and Objects tree comparsion

DeserializeWithLengthPrefix() overload with SerializationContext is absent

Hello!
I cannot find overload of DeserializeWithLengthPrefix() public method which accepts SerializationContext.
I've added it myself and it works fine: (TypeModel.cs)

public object DeserializeWithLengthPrefix(
    Stream source, object value, Type type, PrefixStyle style, int fieldNumber, SerializationContext context)
{
    int bytesRead;
    bool haveObject;
    return DeserializeWithLengthPrefix(source, value, type, style, fieldNumber, null, out bytesRead, out haveObject, context);
}

Regards!

Invalid serialization operation with wire-type None

The following test throws AqlaSerializer.ProtoException : AqlaSerializer.ProtoException : Invalid serialization operation with wire-type None, waiting for wire type of field number 2, position 5

Any idea why? Thanks

        [Fact]
        public void TestNamedTuples()
        {
            (int Number1, int? Number2) obj = (1, null);

            var clone = Serializer.DeepClone(obj);

            Assert.Equal(clone.Number1, obj.Number1);
            Assert.Equal(clone.Number2, obj.Number2);
        }

Recursion depth exceeded safe limit

The following test "TestDistribution" results in an exception.
AqlaSerializer.ProtoException : Recursion depth exceeded safe limit. See TypeModel.RecursionDepthLimit.

Any idea what is wrong? Thanks

    private void SetupModel()
    {
        var metaType2 = AqlaSerializer.Meta.RuntimeTypeModel.Default.Add(typeof(Surrogate), false);
        metaType2.DefaultFormat = AqlaSerializer.ValueFormat.Reference;
        metaType2.UseConstructor = false;
        var metaField1 = metaType2.AddField(1, "_distribution");
        metaField1.SetSettings(x => { x.V.Format = AqlaSerializer.ValueFormat.Reference; });

        var metaType3 = AqlaSerializer.Meta.RuntimeTypeModel.Default.Add(typeof(Distribution), false);
        metaType3.DefaultFormat = AqlaSerializer.ValueFormat.Reference;
        metaType3.UseConstructor = false;

        AqlaSerializer.Meta.MetaType metaType4 = AqlaSerializer.Meta.RuntimeTypeModel.Default[typeof(ILoader)];
        metaType4.AddSubType(1, typeof(Distribution));
        metaType4.SetSurrogate(typeof(Surrogate));

        var metaType6 = AqlaSerializer.Meta.RuntimeTypeModel.Default.Add(typeof(Instrument), false);
        metaType6.DefaultFormat = AqlaSerializer.ValueFormat.Reference;
        metaType6.UseConstructor = false;
        var metaField2 = metaType6.AddField(1, "_loader");
        metaField2.SetSettings(x => { x.V.Format = AqlaSerializer.ValueFormat.Reference; });
    }

    [Test]
    public void TestDistribution()
    {
        SetupModel();

        var inst = new Instrument()
        {
            Loader = new Distribution()
        };

        var clone = AqlaSerializer.Serializer.DeepClone(inst);

        Assert.NotNull(clone);
    }

    private class Instrument
    {
        private ILoader _loader;

        public ILoader Loader
        {
            get { return _loader; }
            set { _loader = value; }
        }
    }

    private class Distribution : ILoader
    {
        public IReadOnlyDictionary<int, double> GetDistribution()
        {
            throw new NotImplementedException();
        }
    }

    private class DefaultLoader : ILoader
    {
        public IReadOnlyDictionary<int, double> GetDistribution()
        {
            return new ReadOnlyDictionary<int, double>(new Dictionary<int, double>());
        }
    }

    private interface ILoader
    {
        IReadOnlyDictionary<int, double> GetDistribution();
    }

    private class Surrogate
    {
        private Distribution _distribution;

        [ProtoConverter]
        public static ILoader FromSurrogate(Surrogate surrogate)
        {
            if (surrogate == null)
                return null;

            if (surrogate._distribution != null)
                return surrogate._distribution;

            return new DefaultLoader();
        }

        [ProtoConverter]
        public static Surrogate ToSurrogate(ILoader loader)
        {
            if (loader == null)
                return null;

            var distribution = loader as Distribution;
            return new Surrogate { _distribution = distribution };
        }
    }

Merging from protobuf-net: need help

Note that the current version of AqlaSerializer is still viable to use. It doesn't contain the newest features and improvements from protobuf-net but remember that AqlaSerializer have a lot of its own stuff like well-done reference tracking which is unavailable in protobuf-net.
I've already managed to merge to master this protobuf-net/protobuf-net@e74cb24 , it kind of works (tests green in Debug) but might need some small changes for Release build. This version is not available yet on nuget.
Then started merging protobuf-net/protobuf-net@0bd2541 into a separate branch. Many small changes were merged automatically with Plastic SCM language-aware merging. In order to complete the merge we have to look this diff and apply the changes manually if they are not there. I've already done this for ProtoReader/Writer, BclHelpers and Helpers.
Instead of completing that merge I decided to skip directly to merging the current protobuf-net version into a child branch. So to complete the merge we need to look this diff except for ProtoReader/Writer, BclHelpers and Helpers it's this diff. Same as before, moved, renamed and deleted files were mostly handled automatically.
Moved, renamed and deleted files were handled automatically. Mostly we need to look only for changed and added cs files.
We don't need to merge changed tests like migration from NUnit to XUnit but the new added protobuf-net tests should be kept.
During the merge keep the PORTABLE conditional stuff, perhaps we need to update the Portable .NET profile to a newer one like .NETPortable4.5-Profile111 Windows 8.0, WindowsPhoneApp 8.1, but the Portable target is good to keep. I'd like to keep the support for Android and iOS but using Portable instead of separate build configurations.
Also keep the IKVM if possible.
The merged version should support all the current runtimes of protobuf-net (so no legacy platforms anymore) plus Portable. Except that NetStandard <2.1 is not planned to support because only .NET Framework needs the old NetStandard nowadays and there is a separate target for .NET Framework already.
Note that switching between Debug and Release configurations might require restarting Visual Studio due to different TargetFrameworks list specified.

Currently I'm not much into .NET development and doing the merge just because I want to keep AqlaSerializer for the community even when I'm not actively using the serializer anymore. Nowadays performance is often neglected by using text-based serialization like json, xml and pure reflection. I want to improve this situation with AqlaSerializer. I I don't know about a better alternative that is fast (I mean really fast), binary, supports objects graphs and works on AOT platforms (not mentioning other improvements) so I want to keep it up to date with the newest technologies like .NET 5, C#9 and proto3. But the lack of interest and support from the community takes away my motivation.

If you like my vision of .NET serialization please assist.

AqlaSerializer.ProtoException: Trap count > 1, will be mismatched with next NoteObject

I got the following exception when trying to deserialize an object. Basically I serialized an object into MemoryStream and then trying to deserialize it, but got this error. What does the error message mean? Is it something we didn't do well?

AqlaSerializer.ProtoException: Trap count > 1, will be mismatched with next NoteObject
   at AqlaSerializer.ProtoReader.TrapNextObject(Int32 newObjectKey)
   at AqlaSerializer.NetObjectHelpers.ReadNetObject_Start(Object& value, ProtoReader source, Type& type, NetObjectOptions options, Int32& typeKey, Boolean handleMissingKeys, Boolean replaceOldValue)
   at proto_478(Object , ProtoReader )
   at AqlaSerializer.Meta.RuntimeTypeModel.Deserialize(Int32 key, Object value, ProtoReader source, Boolean isRoot)
   at AqlaSerializer.ProtoReader.ReadTypedObject(Object value, Int32 key, ProtoReader reader, Type type)
   at proto_1512(Object , ProtoReader )
   at AqlaSerializer.Meta.RuntimeTypeModel.Deserialize(Int32 key, Object value, ProtoReader source, Boolean isRoot)
   at AqlaSerializer.Meta.TypeModel.DeserializeCore(ProtoReader reader, Type type, Object value, Boolean noAutoCreate, Boolean isRoot)
   at AqlaSerializer.Meta.TypeModel.Deserialize(Stream source, Object value, Type type, Int64 length, SerializationContext context)
   at AqlaSerializer.Serializer.Deserialize(Type type, Stream source)

Removing a SerializableMember Makes Deserializing Fail

Everything is working really well with AqlaSerializer, I just met following problem now.

I have a serialized object (class attributes [SerializableType(ImplicitFields = ImplicitFieldsMode.None, EnumPassthru = true)]) with a data member like this:

        [SerializableMember(5)]
        public List<CustomClass> CustomClassObjects { get; set; }

Due to some restructoring I want to completely remove this class member and I thought this is no problem as long as I do not reuse the serializable member ID 5 again.

But as soon as I remove these two lines from the code and try to deserialize the object again I get an exception telling me that

A deferred key does not have a value yet (NoteObject call missed?)

What is the reason for that?
How can I remove a serializable member without breaking the serialization?

Thanks in advance
Andreas

Wiki update

Update documentation to reflect mapping in V2
Use information from other issues
Add basic information (when applicable) from protobuf-net readme file.

Support More Useful .NET Types for Serialization

One point which I always found very bothersome with protobuf-net was their limited support for serialization of several very useful and often used .NET types in data structures.

For me especially the types from the "System.Drawing" namespace would be very interesting, in particular:

Color
Point
Rectangle
RectangleF
if possible also: Image (which should also be easy to implement by using "Image.FromStream()" and "Image.Save()" methods)

I know I can work around that with e. g.

public Color Color { get; set; }

    [SerializableMember(1)]
    private int ColorProto {
        get { return Color.ToArgb(); }
        set { Color = Color.FromArgb(value); }
    }

for Color and e. g.

    public Point MainWindowLocation { get; set; }

    [SerializableMember(101)]
    private ProtoPoint MainWindowLocationProto
    {
        get
        {
            return new ProtoPoint(MainWindowLocation.X, MainWindowLocation.Y);
        }
        set
        {
            MainWindowLocation = new Point(value.X, value.Y);
        }
    }

[SerializableType(ImplicitFields = ImplicitFieldsMode.None, NotAsReferenceDefault = true)]
internal class ProtoPoint
{
    [SerializableMember(1)]
    public int X { get; set; }

    [SerializableMember(2)]
    public int Y { get; set; }

    public ProtoPoint(int x, int y)
    {
        this.X = x;
        this.Y = y;
    }

    private ProtoPoint()
    {
    }
}

for Point, but as you can easily see that gets longish very quickly and is really bothersome to use such work-arounds here as it even make serialization/deserialization slower when such "useless" extra objects need to be created all the time.

Would you mind supporting these types directly with AqlaSerializer?

Thanks a lot in advance again for your great support
Andreas

V2 is soon

These features are already available in the V2 beta version:

Google Protocol Buffers format support is back (I've reconsidered this) - (de)serialization.
Improved reference tracking (e.g. surrogate fix and referencing arrays from inside themselves).
Improved versioning (e.g. between reference-nonreference-null-nonnull-dynamic).
Optimizations for better output size.
Nested collections.
Collection subtypes (will read subtype number to create correct concrete type).
Array types may be registered as collections and use full set of features (null support, etc) even when passed as root objects.
Primitive types are allowed to be set to fields marked as dynamic type.
Reference serialization will avoid using too much recursion #1.
Possibility to specify different attribute sets for different models (see ModelId property on attributes).
Attributes to specify behavior on collection and collection elements (and nested) separately.
Multi-dimensional arrays.

And those are planned to be added with minor 2.x releases:
12. No-versioning mode for best output size.
14. Google Protocol Buffers schema correct generation inside AqlaSerializer.
15. Protogen to generate classes from Google Protocol Buffers schema.
16. Improved performance by avoiding boxing when serializing Nullable[T] with null support or advanced versioning mode enabled.
17. CoreCLR platform support.

AppendCollection mode is not going to be supported in complex scenarios.

Migration:

Attributes mapping example:

[SerializableMember(5, ValueFormat.MinimalEnhancement)] // List<T>
[SerializableMemberNested(1, ValueFormat.Reference)] // int[]
[SerializableMemberNested(2, ValueFormat.Compact)] // int
public List<int[]> Member { get; set; }

[SerializableMemberNested] is optional and may be omitted.

Code mapping example:

MetaType mt = model[typeof(MyType)];
ValueMember valueMember = mt[5];

var s = valueMember.GetSettingsCopy(0); // List<T>
s.Format = ValueFormat.MinimalEnhancement;
valueMember.SetSettings(s, 0);

s = valueMember.GetSettingsCopy(1); // int[]
s.Format = ValueFormat.Reference;
valueMember.SetSettings(s, 1);

s = valueMember.GetSettingsCopy(2); // int
s.Format = ValueFormat.Compact;
valueMember.SetSettings(s, 2);

News: support for .NET Standard 2.1 including .NET Core 3 and .NET 5

The .NET Standard version of AqlaSerializer is released. Don't hesitate to download and try it!

Cannot serialize null string when serializing with length prefix

Hello!
When I'm using public void SerializeWithLengthPrefix(Stream dest, object value, Type type, PrefixStyle style, int fieldNumber, SerializationContext context).
for serializing null string with PrefixStyle.Base128 I've got this exception:
System.NullReferenceException: Object reference not set to an instance of an object. at AqlaSerializer.ProtoWriter.WriteObject(Object value, Int32 key, ProtoWriter writer, PrefixStyle style, Int32 fieldNumber, Boolean isRoot) in C:\Projects\Games\LibsSources\Aqla\Serializer\AqlaSerializer\protobuf-net\ProtoWriter.cs:line 160

Is that by-design or a bug?

Regards!

Type System.Double can't be handled as reference

The following code generates error message "Type System.Double can't be handled as reference", which I don't understand. I think the code should work, instead of failing with some weird messages. Can you take a look please?

    [Test]
    public void RegisterCollectionClassWithMemberFields()
    {
        var model = RuntimeTypeModel.Default;

        var metaType = model.Add(typeof(WithCollectionClassWithMemberFields), true);
        metaType.AsReferenceDefault = true;
        metaType.UseConstructor = false;

        var metaType2 = model.Add(typeof(CollectionClassWithArrayField), true);
        metaType2.AsReferenceDefault = true;
        metaType2.UseConstructor = false;

        var metaField = metaType.AddField(1, "_collectionClassWithMemberFields");
        metaField.AsReference = true;

        var metaField2 = metaType2.AddField(1, "_params");
        metaField2.AsReference = true;

        model.CompileInPlace();
    }
        
    [Serializable]
    private class WithCollectionClassWithMemberFields
    {
        private readonly CollectionClassWithArrayField _collectionClassWithArrayFields;

        public WithCollectionClassWithMemberFields(CollectionClassWithArrayField collectionClassWithMemberFields)
        {
            _collectionClassWithArrayFields = collectionClassWithMemberFields;
        }
    }

    [Serializable]
    private class CollectionClassWithArrayField : IList<double>
    {
        private double[] _params;

        public CollectionClassWithArrayField(int size)
        {
            _params = new double[size];
        }

        public double this[int index]
        {
            get
            {
                throw new NotImplementedException();
            }

            set
            {
                throw new NotImplementedException();
            }
        }

        public int Count
        {
            get
            {
                throw new NotImplementedException();
            }
        }

        public bool IsReadOnly
        {
            get
            {
                throw new NotImplementedException();
            }
        }

        public void Add(double item)
        {
            throw new NotImplementedException();
        }

        public void Clear()
        {
            throw new NotImplementedException();
        }

        public bool Contains(double item)
        {
            throw new NotImplementedException();
        }

        public void CopyTo(double[] array, int arrayIndex)
        {
            throw new NotImplementedException();
        }

        public IEnumerator<double> GetEnumerator()
        {
            throw new NotImplementedException();
        }

        public int IndexOf(double item)
        {
            throw new NotImplementedException();
        }

        public void Insert(int index, double item)
        {
            throw new NotImplementedException();
        }

        public bool Remove(double item)
        {
            throw new NotImplementedException();
        }

        public void RemoveAt(int index)
        {
            throw new NotImplementedException();
        }

        IEnumerator IEnumerable.GetEnumerator()
        {
            throw new NotImplementedException();
        }
    }

How to defined sub-type of open generic type?

I have two generic classes (A and B) defined as below. B is a subtype of A. However, my setup doesn't work. I got an error like "Unexpected sub-type B". Any suggestion on how to make it work?

public static class Test
{
    public static void Main(string[] args)
    {
        var b = new B<int>(1, 2);
        var clone = Serializer.DeepClone(b);

        if (clone.Member2 != b.Member2 || clone.Member != b.Member)
            throw new Exception("clone failed");

        Console.WriteLine("clone success");
        Console.WriteLine("Press any key to continue ...");
        Console.ReadKey();
    }
}

[SerializableType(ImplicitFields = ImplicitFieldsMode.AllFields, SkipConstructor = true)]
public class A<T>
{
    private T _member;

    public A(T member)
    {
        _member = member;
    }

    public T Member => _member;
}

[SerializableType(ImplicitFields = ImplicitFieldsMode.AllFields, SkipConstructor = true)]
[SerializeDerivedType(1, typeof(A<>))]
public class B<T> : A<T>
{
    private T member2;

    public B(T member, T member2)
        : A(member)
    {
        member2 = member2;
    }

    public T Member2 => member2;
}

How to build AqlaSerializer?

I want to use AqlaSerializer to replace the .NET binary formatter we are currently using.

It would be very helpful if I can debug into AqlaSerializer's source code. I have downloaded the source code from its github site. The download includes a very short description about the build, which I could not follow. If you can show me some instructions of how to build it, it would be great.

IList clone changes the type

The following unit test fails on the second assert:
Expected: typeof(int[])
Actual: typeof(System.Collections.Generic.List)

    [Fact]
    private void CloneListOfList()
    {
        var listOfList = new List<IList<int>> { new[] { 1, 2 } };
        var clone = Serializer.DeepClone(listOfList );
        Assert.Equal(listOfList .Count, clone.Count);
        Assert.Equal(listOfList [0].GetType(), clone[0].GetType());
    }

The problem is that the original list contains a list of int arrays, but the cloned list contains a list of "List" objects. How to solve this issue?

Thanks

MetaType.AddSubType() returns the same meta type, not the derived metatype for added sub type

Hello!

MetaType.AddSubType() returns the same meta type, not the derived metatype for added sub type.

This seems misleading. I suppose AqlaSerializer should return derived metatype with which the developer could continue working - adding fields, etc.

Regards!

aqlasolutions / aqlaserializer Goto Github PK

aqlaserializer's Introduction

AqlaSerializer

Usage

We need your help

aqlaserializer's People

Contributors

Stargazers

Watchers

Forkers

aqlaserializer's Issues

Recommend Projects

Recommend Topics

Recommend Org