csirtgadgets / google-protocolbuffers-perl Goto Github PK

Perl 99.94% Shell 0.06%

google-protocolbuffers-perl's Introduction

Google-ProtocolBuffers

Google Protocol Buffers is a data serialization format. It is binary (and hence compact and fast for serialization) and as extendable as XML; its nearest analogues are Thrift and ASN.1. There are official mappings for C++, Java and Python languages; this library is a mapping for Perl.

INSTALLATION

To install this module type the following:

   perl Makefile.PL
   make
   make test
   make install

DEPENDENCIES

This module requires these other modules and libraries:

    Parse::RecDescent
    Math::BigInt
    Class::Accessor

Copyright (C) 2008 by Igor Gariev, <[email protected]>
Copyright (C) 2013 by the CSIRT Gadgets Foundation <csirtgadgets.org>

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself, either Perl version 5.10.0 or, at your option, any later version of Perl 5 you may have available.

google-protocolbuffers-perl's People

Contributors

Stargazers

Watchers

Forkers

herbsears-oanda benvanstaveren trinitum volant bozhom davidkaufman hessu hexfusion gurbalinder xtaran mykhailosh hugmeir

google-protocolbuffers-perl's Issues

Test failure in t/08-defaults.t

t/08-defaults.t fails on a FreeBSD 9.0 system, both with the system perl (5.12.4) and a self-compiled 5.16.2:

t/08-defaults.t .... 1/55 
       ERROR (line 9): Invalid fieldOption: Was expecting constant but found
                       ".25 ]; " instead

       ERROR (line 8): Invalid messageBody: Was expecting '}' but found
                       "required float f7 = 7 [default = .25 ]; " instead

       ERROR (line 8): Invalid message: Was expecting messageBody but found
                       "{" instead

       ERROR (line 1): Invalid proto: Was expecting message, or extend, or
                       enum, or import, or package, or option, or service, or
                       syntax, or ';' but found "message TestFP {" instead
Died at /home/e/eserte/work2/google-protocolbuffers-perl/blib/lib/Google/ProtocolBuffers/Compiler.pm line 328.
 at /home/e/eserte/work2/google-protocolbuffers-perl/blib/lib/Google/ProtocolBuffers/Compiler.pm line 328.
        Google::ProtocolBuffers::Compiler::parse('Google::ProtocolBuffers::Compiler', 'HASH(0x10830d8)', 'HASH(0xf187b8)') called at /home/e/eserte/work2/google-protocolbuffers-perl/blib/lib/Google/ProtocolBuffers.pm line 47
        Google::ProtocolBuffers::_parse('Google::ProtocolBuffers', 'HASH(0x10830d8)', 'HASH(0xf187b8)') called at /home/e/eserte/work2/google-protocolbuffers-perl/blib/lib/Google/ProtocolBuffers.pm line 27
        Google::ProtocolBuffers::parse('Google::ProtocolBuffers', '\x{a}    message TestFP {\x{a}        required float f1 = 1 [default ...') called at t/08-defaults.t line 145
# Looks like you planned 55 tests but ran 46.
# Looks like your test exited with 255 just after 46.

Regards, Slaven

Subtle corruption on float encode/decode

The final bit is being dropped in values > 2**24

0x1000000 encodes fine
0x1000001 does not (encodes as 0x1000000)

Or maybe it's the decode routine. Not sure yet.

$ perl ./test2.pl 16777216
ok

$ perl ./test2.pl 16777217
expected 16777217 got 16777216

$ perl ./test2.pl 16777218
ok

#!/usr/bin/perl -w
use strict;

my $tv = shift;
die "$0 [value]\n\ttry 16777216, 16777217, 16777218\n" if !defined($tv);

use Google::ProtocolBuffers;
Google::ProtocolBuffers->parse( "
               message Person {
                     optional float version      = 1;
               }
           ",
    { create_accessors => 1 } );

my $p = Person->encode( { version => $tv } );
my $p2 = Person->decode($p);

if ($p2->{version} != $tv) {
    print "expected $tv got " .$p2->{version} . "\n";   
    exit 0;
}

print "ok\n";

Deviation from PB spec may result in code that makes invalid assumptions

this is technically not per-spec

MessageClass->encode($hashref)

This method may be called as class or instance method. 'MessageClass' must already be created by compiler.      
Input is a hash reference. Output is a scalar (string) with serialized data. Unknown fields in hashref are ignored. 
In case of errors (e.g. required field is not set and there is no default value for the required field) an exception is  
thrown."

The issue is with "required field is not set and there is no default value for the required field". Required + default is meaningless per spec [see http://code.google.com/p/protobuf/issues/detail?id=312#c3]

'default' is only meaningful for optional fields

By emitting serialized protocol buffers and not throwing an exception for unset required fields, you are encouraging people to write code, in Perl, that is not translatable to other languages. They will assume that specifying a default for a required field in the IDL means they don't have to explicitly set the field in their code. They must.

The fix is when serializing, if a required field has not been explicitly set -> error. Default values are only used in 'getter' routines.

OpenStreet Map

Does it work with OpenStreet Map files? https://wiki.openstreetmap.org/wiki/PBF_Format

provide protoc-perl commandline utility

try to mimic what protoc does so we can ./configure check for it in other utilities.

Release 0.12 throws error for keyword `reserved` of proto2 syntax

Under the description of Protocol Buffers for syntax proto2, there is an allowed keyword reserved to mark tags no longer in use.

The keyword is used to prevent reuse of fields that may clash with previous definitions, and to also remove the chance of re-ordering fields causing the same issue.

See Google-ProtocolBuffers - Language Guide for proto2 includes the reserved word.

A test Protocol Buffer with the reserved keyword is:

283 message SQLMsg {
284
285     reserved    1, "query";
286
287 //    optional string query       = 1;    //  Query run against the data source.
291     optional string metadata    = 2;    //  Metadata resulting from the query.
292 //  What data for the table and row?
293 //  Anything else ?
294 }

But compilation throws an error - redaction included:

<prompt>:$ ./generate_pm_from_proto --proto-file <input>.proto --pm-file <output>.pm
[Tue,23-Oct-2018 12:08:24 PM ACDT] Starting Proto to PM Generator ./generate_pm_from_proto with parameters:
Proto File  = [<input>.proto]
PM File     = [<output>.pm]
Debug       = [0]
Help        = [0]

     ERROR (line 283): Invalid messageBody: Was expecting '}' but found
                       "reserved 1, "query";" instead

     ERROR (line 283): Invalid message: Was expecting messageBody but found
                       "{" instead

     ERROR (line 278): Invalid proto: Was expecting /\Z/ but found "// SQL
                       system query - generic SQL queries run against a data
                       source" instead
Died at /usr/share/perl5/Google/ProtocolBuffers/Compiler.pm line 335.
<prompt>:$

Grammar fails on [ default = float ]

I'm having an issue with default = float. I looked at $grammar and it looks ok (Compiler.pm) and I looked at the regex's for floatLit and those look Ok too. Declaring the field as a double has the same result, so it looks like the grammar is ignoring decimals and treating numbers like ints...

For example:

#!/usr/bin/perl -w
#perl ./test.pl [proto file]

use strict;
use Google::ProtocolBuffers;

Google::ProtocolBuffers->parsefile(shift,
  {
    generate_code => 'test.pm',
    create_accessors    => 1,
    follow_best_practice => 1,
  }
);

This works fine:

message MessageType {
   optional float f = 1 [default = 1];
}

but this does not:

message MessageType {
   optional float f = 1 [default = 1.23];
}

The second proto file ("1.23") throws an error (but "protoc" has no issue with the file and it conforms to the IDL):

$ perl test.pl msg.proto

   ERROR (line 3): Invalid option: Was expecting optionBody but found "al
                   float f = 1 [default = 1.23];" instead

   ERROR (line 2): Invalid messageBody: Was expecting '}' but found
                   "optional float f = 1 [default = 1.23];" instead

   ERROR (line 2): Invalid message: Was expecting messageBody but found
                   "{" instead

   ERROR (line 1): Invalid proto: Was expecting message, or extend, or
                   enum, or import, or package, or option, or service, or
                   syntax, or ';' but found "message MessageType {"
                   instead
Died at /usr/local/share/perl5/Google/ProtocolBuffers/Compiler.pm line 328.

protoc-perl: parse multiple .proto's

dies, unless an optional uint64 field is commented out of the protocol definition

A binary block that is decoded ok using the current python protocol library, causes Google::ProtocolBuffers to go off the rails, right after decoding a uint64 primitive field that is optional.
"optional uint64 FieldName = 2;"
FieldName is not shown in the decoded output of a python run, on the same binary block.

Commenting out the field from the protocol spec results in the same binary being parsed correctly.
I'm unable to work out why :(

Poor langauge support

SAXJAZMAN/protobuf/Google-ProtocolBuffers-0.12.tar.gz requires perl '5.012'; you have only 5.010001; giving up

I'm reasonably confident that there is no genuine necessity for this arbitrary restriction?

.DS_Store should not be part of the distribution

Get data in hex format, not float64

#!/usr/bin/perl -w

use Google::ProtocolBuffers;
use File::Slurp qw(read_file write_file);

Google::ProtocolBuffers->parse("
    message TestMessage {
      required double tag1 = 1;
    }
",
    {create_accessors => 1 }
);

my $enc = TestMessage->encode({
tag1 => 0, 
});

open my($fh), ">data.bin";
binmode $fh;
print $fh $enc;
close $fh;

my $dec = read_file('data.bin', { binmode => ':raw' });

my $protoc = TestMessage->decode($dec);

print $protoc->tag1;

And I get 0! Any number is rounded.
But, when I try to use the protoc command, everything is as it should.

C:\Users\user\Desktop>protoc --decode_raw < data.bin 1: 0x0000000000000000

parse error "Unknown syntax"

When I call parse() it just prints the following error and dies. Even if I specify generate_code. I'm attaching my .proto file.
MyMessage.proto.txt
How can I debug this?

Unknown syntax at (eval 76) line 13393, <> chunk 1.

MYMETA should not be included

Your distribution (SAXJAZMAN/protobuf/Google-ProtocolBuffers-0.11.tar.gz) contains files MYMETA.yml and/or MYMETA.json. These files should not be included into distribution - they are generated when installing.
See http://weblog.bulknews.net/post/44251476706/stop-shipping-mymeta-to-cpan

P.S. If this report is about old version, please delete old indexed versions mentioned here: http://cpanold.chorny.net/?a=SAXJAZMAN and contact me.

Make it possible to decode enums

At this moment, it's possible to define an enum and go from its constants to its int values. However, it would be nice if it were possible to convert a decoded int back to the string constant for printing. At the moment there is a workaround using Symbol::Tables:

my $st = Symbol::Tables->New("CODE", "EnumName");
my %type_to_str;
foreach(keys %$st) {
  $type_to_str{$st->{$_}->()} = $_;
}

Then, $type_to_str{5} = "YOUR_CONSTANT".

Strange META.yml contents

The first element of the author array looks strange, see here: https://metacpan.org/source/SAXJAZMAN/Google-ProtocolBuffers-0.11/META.yml#L4

Parsing proto files with /* .. */ style comments

ref: https://rt.cpan.org/Ticket/Display.html?id=82734

diff lib/Google/ProtocolBuffers/Compiler.pm.orig
lib/Google/ProtocolBuffers/Compiler.pm
19c19
< proto       :   <skip: qr!  (?: //.*\n | \s+ )*  !x>

---
> proto       :   <skip: qr!  (?: //.*?\n | \s+ | /\*.*?\*/\s* )*  !xs>

The added 's' in the options, makes . also match the newline symbol. I
thus added the non-greedy modifier '?' in the expression to match
//-style comments.

Cheers,
Daniel

doc missing -p flag

The 0.12 release breaks other CPAN distributions

The following CPAN distributions have failing test suites if used with Google-ProtocolBuffers-0.12:

DAMS/AnyEvent-Riak-0.93.tar.gz
RGE/Net-Riak-0.1702.tar.gz
DAMS/Riak-Client-1.96.tar.gz
EXUSSUM/Voldemort-0.11.tar.gz

The failing test log always includes something like this:

Can't use string ("1") as an ARRAY ref while "strict refs" in use at /home/cpansand/.cpan/build/2016100115/Google-ProtocolBuffers-0.12-1/blib/lib/Google/ProtocolBuffers.pm line 308.