pedrohm / protobuf-net Goto Github PK

First of all, thank you very much for this great piece of work :) I try to
use it as an alternative to WCF's built-in serializers, and it looks
promising in terms of speed and size.

Now I have an issue with classes relying on execution of methods marked
with [OnDeserialized]. As an example, they could be used to initialize
ReadOnlyCollections that can't be serialized by protobuf because they don't
have an empty default constructor. I can work around this by adding some
ugly extra code (cannot use automatic properties anymore; need to make sure
in getters that collections are initialized etc.), however I obviously want
to avoid this.

I guess calling OnDeserialized would be much easier to implement then
OnDeserializing, OnSerializing etc., as the StreamContext's Context
property passed to the method by the DataContractSerializer is null anyway.
So it seems to be just a bit of reflection and a method invoke.

Would be great if this could be added to the code. I looked through the
code, but I'm not really sure where would be the best place to add this call...

email: [email protected]

What steps will reproduce the problem?
1. I create 2 .proto's (see attached files) loosely taken from the Google
tutorial: http://code.google.com/apis/protocolbuffers/docs/cpptutorial.html

2. I "proto" compile using the following protogen syntax
protogen.exe -i:Person.proto -o:..\dotnet\Person.cs
protogen.exe -i:AddressBook.proto -o:..\dotnet\AddressBook.cs

3. I "compile" a dotnet class library using the 2 generated .cs files (see
attached dotnet.zip for the complete .NET project). I'm using Microsoft
Visual C# 2005 (Version 8.0.50727.762)

What is the expected output? What do you see instead?
A tutorial.dll should be created. I am either doing something incorrectly
(hopefully!) or protogen does not handle "enum" and "import" correctly. If
the former is true, I apologize - and what I am doing wrong?

The message definitions are duplicated. I get a compile error:
The type 'tutorial.Person' already contains a definition for 'PhoneType'

What version of the product are you using? On what operating system?
r217

Please provide any additional information below.
I use the same sample to compile C++ using Google's protoc.exe. I create a
static library using Microsoft C++ 2005 and everything works OK. Same with
compiling Java. Both languages create the correct import/include statements
within the auto-generated source.

This is just an enhancement request - Serializer could have non-generic
Serialize and Deserialize methods that would require to type of the proto
class to be passed as parameter. 

Please include an e-mail address if this might need a dialogue!
==============

What steps will reproduce the problem?
1. Create a TcpListener, server, and start listening for connections.
2. Create a TcpClient, clientside, and connect to the server.
3. Create another TcpClient, serverside, with server.AcceptTcpClient().
4. Get the clientside stream and serverside stream.
5. Serialize an object with the clientside stream
6. Deserialize the object on the serverside stream.

What is the expected output? What do you see instead?
I expect the object that I serialized to deserialize, instead, the
Serialize.Deserialize<T>(Stream source) method hangs.

What version of the product are you using? On what operating system?
I've reproduced this on the .Net 2.0 revision 164 download and using
revision 171.  Windows XP.

Please provide any additional information below.
I have a workaround for this.  I've attached a patch for review.

Serialization of some CLR types seems to work just fine, like DateTime and
double - it is possible to serialize the values in one culture setting and
deserialize in another.

This however does not work with DateTimeOffset. Probably internally simply
ToString and Parse are used without using invariant culture settings or
instead of using the XmlConvert class.

Here is some test code demonstrating the problem:

            CultureInfo originalCulture = Thread.CurrentThread.CurrentCulture;
            try
            {
                DateTimeOffset originalData = DateTime.Now;
                DateTimeOffset clonedData;

                #region serialization/deserialization roundtrip using
different cultures

                using (MemoryStream stream = new MemoryStream())
                {
                    // serialize with culture en-GB
                    Thread.CurrentThread.CurrentCulture = new
CultureInfo("en-GB");
                    Serializer.Serialize<DateTimeOffset>(stream, originalData);

                    // deserialize with culture de-DE (incompatible date
and number formats)
                    Thread.CurrentThread.CurrentCulture = new
CultureInfo("de-DE");
                    stream.Position = 0;
                    clonedData =
Serializer.Deserialize<DateTimeOffset>(stream); // <-- crashing here
                }

                #endregion

                // compare data
                Assert.AreEqual(originalData, clonedData);

All common scenarios, including inheritance, inferred tags, remoting, etc

had working previously with really good results; ideally all of:
* basicHttp
* MTOM basicHttp
* proto basicHttp
* proto MTOM basicHttp
* net-binding
* proto net-binding (ISerializable?)

http://stackoverflow.com/questions/583182/protocol-buffer-deserialization-
and-a-dynamically-loaded-dll

An enum and class pair, properly annotated with ProtoContractAttribute and
ProtoEnumAttribute, will correctly serialize and deserialize.  However, one
cannot generate a .proto file for the enum using the
Serializer.GetProto<T>() method because T is constrained only to classes. 
That would be okay if the Serializer.GetProto<T>() method would detect the
enum type while generating the .proto for the containing class, and dump it
out automatically; but that is not the case.

"No suitable Default CStation encoding found." after calling 
"Serializer.Serialize"

But i did define my Class CStation:

        [ProtoContract]
        public class CStation
        {
            [ProtoMember(1)]
            public int number { get; set; }

........
            [ProtoMember(8)]
            public int ticket { get; set; }

            // My public constructors there

       }

and i have another class :

[ProtoContract]
        public class CListStations
        {
            [ProtoMember(1)]
            private  List<CStation> _liststations = new List<CStation>();
            public List<CStation> liststation { get { return _liststations; 
} }

         }

And i get the exception when i try to serialize my CListStations object. 
""No suitable Default CStation encoding found.""

See NWind benchmark for length-prefixed deserialization; there is no real 
reason that this should be slower than grouped deserialization.

I have tired to use a simple array of strings but it has failed. Let me
know if you need a code sample.

See proposed usage in inheritance demo

See thread 
http://groups.google.com/group/protobuf/browse_thread/thread/50955cedea6ada
62#

Waiting on sample

http://stackoverflow.com/questions/453820

See issue 14; currently uses naive implementation - need to assess whether 
this is a problem for larger data sets (or give advice to use lists!)

Although core works for both fields and properties, only properties are 
included by the outer code. Support fields for WCF compatiblity...

What steps will reproduce the problem?
1. Create a new .NET project with SNK (strong name key) specified.
2. Reference protobuf-net.dll
3. Try building the project.

What is the expected output? What do you see instead?
Expected ouput: successful build.
Result instead: Assembly generation failed -- Referenced assembly 
'protobuf-net' does not have a strong name

What version of the product are you using? On what operating system?
Version:  1.0.0.237
OS: Windows XP SP3, .NET 3.5 SP1

I was playing about with protogen using the Person type from the Google 
website and wondered why a field declared with 

  optional int32 id = 2;

is not generated as a Nullable<int> in the corresponding c# class.

It seems to just get the default value (zero), which might not mean the 
same thing as 'missing'.

Hi! I'm currently struggling with inheritance in protobuf-net. I have the
following example:
[ProtoContract]
[ProtoInclude (2, typeof (B))]
public class A
{
    [ProtoMember (1, Name = "PropA")]
    public string PropA { get; set; }

    [ProtoMember (2, Name = "PropB")]
    public string PropB { get; set; }
}

[ProtoContract]
public class B : A
{
    [ProtoMember (1, Name = "PropAB")]
    public string PropAB { get; set; }
    [ProtoMember (2, Name = "PropBB")]
    public string PropBB { get; set; }
}

Now doing:
B b = new B {PropA = "a", PropB = "b", PropAB = "ab", PropBB = "bb"};
using (var s = new MemoryStream ())
{
   Serializer.Serialize (s, b);
   s.Position = 0;
   B bb = Serializer.Deserialize<B> (s);
   Console.WriteLine ("{0},{1},{2},{3}", bb.PropA, bb.PropB, bb.PropAB,
bb.PropBB);
}

Should produce: a,b,ab,bb
Instead I get: ,,ab,bb

So it seems like inherited members are not retrieved.

I've got a wcf duplex callback that is passing an array of objects like so:

       [OperationContract(IsOneWay = true), ProtoBehavior]
       void Foo(SomeType[] data); // is a valid [DataContract] with Order

My problem is the visualUpdates is getting set to null on the client
side.  They don't seem to be getting deserialized no matter what I do.

Minor annoyance

[deleted issue]

Using a class which has a member inheriting from IDictionary<TKey, TValue> 
causes exception to be thrown upon PropertyFactory.Create[T] call.

[csc] Compiling 79 files to 'protobuf-
net/Protobuf_Serialization_Comparison_Test/bin/Debug/ProtoBuf.dll'.
     [csc] protobuf-net/Protobuf_Serialization_Comparison_Test/protobuf-
net/protobuf-net/Property/PropertyFactory.cs(16,40): error CS0029: Cannot 
implicitly convert type `T' to `T'
     [csc] protobuf-net/Protobuf_Serialization_Comparison_Test/protobuf-
net/protobuf-net/Property/PropertyFactory.cs(16,40): The generic parameter 
`T' of `ProtoBuf.Property.PropertyFactory.GetPassThru<T>()' cannot be 
converted to the generic parameter `T' of 
`ProtoBuf.Property.PropertyFactory.<>c__CompilerGenerated9<T>' (in the 
previous error)
     [csc] protobuf-net/Protobuf_Serialization_Comparison_Test/protobuf-
net/protobuf-net/Property/PropertyFactory.cs(16,40): error CS1662: Cannot 
convert `anonymous method' to delegate type `ProtoBuf.Property.Getter<T,T>' 
because some of the return types in the block are not implicitly 
convertible to the delegate return type
     [csc] Compilation failed: 2 error(s), 0 warnings

Based on -ve int32 discussion (and how this impacts int16/byte if 
implemented), prefer to use sint32 if no preference is given.

Will need to rename SignedAttribute ;-p

Please include an e-mail address if this might need a dialogue!
==============

What steps will reproduce the problem?
1. Bulid in MonoDevelop opensuse 11.0 mono 2.01
2.
3.

What is the expected output? What do you see instead?
This is the error:
[Task:File=/home/kiminozo.eien/protobuf-net/protobuf-net/Property/
PropertyTimeSpanFixed.cs, Line=16, Column=26, Type=Error, Priority=Normal, 
Description=The type `ProtoBuf.Property.PropertyTimeSpanString<TSource>' must 
be 
convertible to 
`ProtoBuf.Property.Property<ProtoBuf.Property.PropertyTimeSpanString<TSource>>' 
in 
order to use it as parameter `T' in the generic type or method 
`ProtoBuf.Property.Property<TSource,TValue>.CreateAlternative<T>
(ProtoBuf.DataFormat)'(CS0309)]

What version of the product are you using? On what operating system?
217 
opensuse 11.0 mono 2.01

Please provide any additional information below.

I think this error is a mono's bug, I Fixed Property.cs line 132 like this:
but it is not a good way.

protected Property<TSource> CreateAlternative<T>(DataFormat format) where T : 
new
() //Property<TSource>, new()
        {
            Property<TSource> alt = new T() as Property<TSource>;
            if(alt != null){
                alt.Init(Tag, format, getValue, setValue, IsOptional, 
DefaultValue);
            }
            return alt;
        }

Fine for 32-bit, but flakey at the extreme ends of 64-bits (impacts -ve)

f you tag a collection class simply as DataContract it doesn't throw an 
exception, however, serializing an instance of the class and then 
deserializing it loses the Items.

A test code for the classes below:

ContainerClass cc = new ContainerClass()
                        {
                            ContainerID = 9,
                            Description = "foocontainer",
                            Items = new MyCollection()
                                        {
                                            {
                                                3, new MyItem()
                                                       {
                                                           ItemID = 3,
                                                           Description = 
"fooitem"
                                                       }
                                                }
                                        }
                        };


string b64 = null;
using (MemoryStream ms = new MemoryStream())
{
    ProtoBuf.Serializer.Serialize(ms, cc);
    long length = ms.Length;
    ms.Position = 0;
    b64 = Convert.ToBase64String(ms.ToArray());
}

ContainerClass cont = null;
using(MemoryStream ms = new MemoryStream())
{
    byte[] data = Convert.FromBase64String(b64);
    ms.Write(data, 0, data.Length);
    ms.Position = 0;
    cont = ProtoBuf.Serializer.Deserialize<ContainerClass>(ms);
}

Console.WriteLine("Items after deserialization: " + cont.Items.Count);

// ============================================

Also, in Serializer.IsEntityType() method you should check for 
CollectionDataContract tag if anyone else uses collection classes.

For unexpected fields; have added preliminary IExtensible interface, but 
needs implementation, examples, etc

Currently uses 5 bytes
See here:
http://groups.google.com/group/protobuf/browse_thread/thread/7f325f50d1d52c
94/24c52b9448b266ba#24c52b9448b266ba

Please include an e-mail address if this might need a dialogue!
==============
Email: [email protected]

What steps will reproduce the problem?
1. Error occurs occasionally  when deserializing message from Stream.
StackTrace I get is:

The exception is NullReferenceException

ProtoBuf.Serializer`1.Deserialize(T& instance, SerializationContext
context) in d:\\protobuf-net\\trunk\\protobuf-net\\SerializerT.cs:line
489\r\n  in ProtoBuf.Serializer`1.Deserialize(T& instance, Stream source) w
d:\\protobuf-net\\trunk\\protobuf-net\\SerializerT.cs:line 358\r\n   w
ProtoBuf.SerializerItemProxy`1.Deserialize(TItem& instance, Stream source)
in d:\\protobuf-net\\trunk\\protobuf-net\\SerializerProxy.cs:line 64\r\n  
in ProtoBuf.Serializer.Deserialize[T](Stream source) in
d:\\protobuf-net\\trunk\\protobuf-net\\Serializer.cs:line 251\r\n   w
SecPL.Return.CoreLib.NetworkServices.NetworkUtils.DeserializeMessage[T](Stream
source)"

To show core and advanced usage

Quite a big task... needs to generate data-contracts and service-contracts 
from .proto; I wonder if this can be done in Jon's ported net version via 
different templates?

Please include an e-mail address if this might need a dialogue!
[email protected] ==============

ProtoInclude causes "Attempt to read past the end of the stream" exception
when writing to IsolatedStorageFileStream in Silverlight 2.0.  This
behavior does not occur when writing to a memory stream.

Version 207 SVN

Please provide any additional information below.

View the code that produces the error at
http://memephiswebtech.com/protobuf/protobuftest.html.

I am including the test project as an attachment.

       What's the best way for me to get this to work with protobuf-net? Is 
there a way for me to inject a serializer proxy thing or something? (I was 
so sure it worked at one point, but obviously not.)

Thank again,
-Michael


   [DataContract(Namespace = FileDbSerializer.XmlDbNamespace)]
   public struct KeyPair<TKey1, TKey2>
   {
       public KeyPair(TKey1 k1, TKey2 k2)
           : this() {
           Key1 = k1;
           Key2 = k2;
       }
       // Stupid tuple class for datacontract
       [DataMember]
       public TKey1 Key1 { get;  internal set; }
       [DataMember]
       public TKey2 Key2 { get;  internal set; }

       public override string ToString() {
           return Key1.ToString() + ", " + Key2.ToString();
       }
   }

What steps will reproduce the problem?
1.
declare contract
    [ProtoContract]
    class MyClass
    {
        [ProtoMember(1), DefaultValue("Test Test TEst")]
        public string TestString;
    }
2.
get proto desription
string proto = Serializer.GetProto<MyClass>();

What is the expected output? What do you see instead?
expected:
variable proto contains
message MyClass {
   optional string TestString = 1 [default = "Test Test TEst"];
}
see instead:
message MyClass {
   optional string TestString = 1 [default = Test Test TEst];
}

Please provide any additional information below.
I correct this issue with next code:

object def = prop.DefaultValue;
if (def != null)
{
    if (prop.DefinedType == "string")
    {
        def = "\"" + def.ToString() + "\"";
    }
    string defText = Convert.ToString(def, 
CultureInfo.InvariantCulture);
    sb.Append(" [default = ").Append(defText).Append("]");
}

To avoid having to mark all the properties in existing types as 
serializable, consider a marker (perhaps at the [ProtoContract] level) to 
say "infer the tags", similar to the existing global switch for 
[DataMember].

This would be limited to public properties (no fields, no non-public), and 
would be based on the alphabetic (ordinal) sequence of the (unmarked) 
members. To allow for later extension, it must be possible to add a member 
out-of-sequence - i.e. "[ProtoMember(18)] public string Aardvark 
{get;set;}". And to hide members, there should be a [ProtoIgnore] attribute 
(that should also be respected for [XmlElement] and [DataMember] members).

Add support for 16-bit/8-bit numbers? (byte/sbyte/short/ushort)
Note that since 32-bit numbers can be processed more efficiently than 8/16,
just needs to call the 32-bit variant (signed/not) versions and cast.

Add support for fixed-length (32/64) integers? Need to check spec...

See example usage with ProtoEnum - no runtime support yet

See SimpleStreamDemo example 
It was working fine, so I probably did something stoopid when refactoring

If you import message types from other .proto file the generation with 
protogen passes without problem. When you try to build though you notice 
that the imported .proto file namespace has not been imported properly in 
the generated .cs file. As workaround you can add this manually.

The protoc.exe version shipped with r221 is 2.0.2. I am currently using 
2.0.3 to compile java/c++ protos. An error is thrown when attempting to use 
protogen with 2.0.3.

Thanks,
Dave

Similar to stream examples, but aiming at speed

[email protected]
==============

What steps will reproduce the problem?
1. My Program is running in multi-thread;
2. A haphazard occurrence. 
The value of member Serializer<T>.subclasses is null;
3.

What is the expected output? What do you see instead?
I think the "subclasses" is not threaded safe variables.
I think it must add a head "[ThreadStatic]";

What version of the product are you using? On what operating system?
1.0.0.213 win2008

Please provide any additional information below.

See http://groups.google.com/group/protobuf/browse_thread/thread/8f6e9ba3ee923b
db#

  optional double Foo = 1;

Becomes

  [ProtoBuf.ProtoMember(1, IsRequired = false, Name=@"Foo ", DataFormat = 
ProtoBuf.DataFormat.TwosComplement)]
  [System.ComponentModel.DefaultValue(default(int))]    
  public int Foo
  {
    get { return _ID0EBB; }
    set { _ID0EBB = value; }
  }

Investigate

Run the QuickStart project under CF 2.0 (tweaking the async TcpListener to 
sync); the List<T>.Sort method reports MethodAccessException when looking 
at Contact/Customer.

Suggestion.

protobuf-net DLL compiler

Compiles your proto files, and wraps them into a single .dll file, which
also includes a serialiser/deserialser.

Lightweight, and can be copied easily between your applications.