mbarbon / google-protobuf-dynamic Goto Github PK

When attempting to install with cpanm (cpanminus), install fails with the following error:

--> Working on Alien::ProtoBuf   
Fetching http://www.cpan.org/authors/id/M/MB/MBARBON/Alien-ProtoBuf-0.09.tar.gz                                                        
-> OK                            
Unpacking Alien-ProtoBuf-0.09.tar.gz                               
Entering Alien-ProtoBuf-0.09     
Checking configure dependencies from META.json                     
Checking if you have Module::Build 0.38 ... Yes (0.4224)           
Checking if you have ExtUtils::CppGuess 0.11 ... Yes (0.20)        
Checking if you have ExtUtils::Install 1.46 ... Yes (2.04)         
Checking if you have Alien::Base 0 ... Yes (1.89)                  
Checking if you have Alien::Base::ModuleBuild 0.023 ... Yes (1.06) 
Configuring Alien-ProtoBuf-0.09  
Running Build.PL                 
+ /usr/bin/perl /usr/local/share/perl/5.26.1/PkgConfig.pm --modversion protobuf                                                        
+ /usr/bin/perl /usr/local/share/perl/5.26.1/PkgConfig.pm --modversion protobuf                                                        
Could not find an installed protobuf library at inc/AP/Build.pm line 19.                                                               
-> N/A                           
-> FAIL Configure failed for Alien-ProtoBuf-0.09. See /root/.cpanm/work/1570033972.31588/build.log for details.                        

FYI, I was able to get past the below error by adding "#undef seed" near the top of src/dynamic.h:

x86_64-linux-gnu-gcc -I/usr/lib/x86_64-linux-gnu/perl/5.22/CORE -DVERSION="0.22" -DXS_VERSION="0.22" -fPIC -D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -xc++ -I/usr/local/share/perl/5.22.1/auto/share/dist/Alien-uPB/include -DNDEBUG -pthread -I/usr/local/include -DPERL_NO_GET_CONTEXT -Isrc -Ibuildtmp -c -D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c++11 -O3 -O2 -g -o buildtmp/Dynamic.o buildtmp/Dynamic.c
In file included from /usr/include/c++/5/random:49:0,
from /usr/include/c++/5/bits/stl_algo.h:66,
from /usr/include/c++/5/algorithm:62,
from /usr/local/include/google/protobuf/stubs/common.h:38,
from /usr/local/include/google/protobuf/descriptor.h:61,
from /usr/local/include/google/protobuf/compiler/importer.h:44,
from src/dynamic.h:9,
from buildtmp/Dynamic.c:180:
/usr/include/c++/5/bits/random.h:264:17: error: macro "seed" passed 1 arguments, but takes just 0
{ seed(__s); }
^

after attempting to encode broken input (null instead of arrayref) the encoder breaks for all subsequent encode() calls
so having schema like:

message Person {
  required int32 id = 1;
  repeated SomeSubMessage foo = 6;
  message SomeSubMessage {
    repeated int32 arrayOfInts = 2;
  }
}

after attempt to encode {id => 123, foo => [{arrayOfInts => undef }]} , all subsequent encode() calls produce 0 byte outputs

encoded length: 4
Not an array reference when encoding field 'humans.Person.SomeSubMessage.arrayOfInts' at ff20.pl line 31.
encoded length: 0
encoded length: 0
encoded length: 0

script to reproduce:

use Data::Dumper;
use Google::ProtocolBuffers::Dynamic;
$dynamic = Google::ProtocolBuffers::Dynamic->new;
$dynamic->load_string("person.proto", <<'EOT');
syntax = "proto2";
 
package humans;

message Person {
  required int32 id = 1;
  repeated SomeSubMessage foo = 6;
  message SomeSubMessage {
    repeated int32 arrayOfInts = 2;
  }
}

EOT
 
$dynamic->map({ package => 'humans', prefix => 'Humans' });
 
toggle({id => 123, foo => [{}]});
toggle({id => 123, foo => [{arrayOfInts => undef }]});
toggle({id => 123, foo => [{}]});
toggle({id => 123, foo => [{}]});
toggle({id => 123, foo => [{}]});

sub toggle {
    my ($h) = @_;
    my $encoder = "Humans::Person";
    eval {
        $bytes = $encoder->encode($h);
        print "encoded length: ". length($bytes) . "\n";
        1;
    } or do {
        print $@;
    }
}

it works fine if arrayOfInts is in the top object"

message Person {
  required int32 id = 1;
  repeated int32 arrayOfInts = 2;
}

toggle({id => 123 });
toggle({id => 123, arrayOfInts => undef });
toggle({id => 123 });

output:

encoded length: 2
Not an array reference when encoding field 'humans.Person.arrayOfInts' at /home/bnikolov/ff10.pl line 28.
encoded length: 2

Just built with protobuf from github and get the following warnings:

[libprotobuf WARNING google/protobuf/compiler/parser.cc:547] No syntax specified for the proto file: scalar.proto. Please use 'syntax = "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted to proto2 syntax.)

Probably doesn't matter yet, but eventually will. :)

(Sorry for not just patching this.)

The documentation says:
$dynamic = Google::ProtocolBuffers::Dynamic->new($search_path);

Please explain what EXACTLY goes in $search_path. How do I specify more than one path? I have my project .proto file(s), and they import google .proto files such as timestamp.proto...

Logs attached.

cpan-build.txt
perl-version.txt

I believe upb supports JSON in- and output transparently while using the same protobuf schemas. I think as a not-protobuf-enabled-client compatibility layer, supporting this in GPB::Dynamic as something like encode_json/decode_json for the generated mapper classes would be awesome.

(And yeah, I have a work use case in mind for that.)

I cannot make two .proto's to work.

=== Address.proto ===
package Diadoc.Api.Proto;

message Address {
optional RussianAddress RussianAddress = 1;
optional ForeignAddress ForeignAddress = 2;
optional string AddressCode = 3;
}

message RussianAddress {
optional string ZipCode = 1;
required string Region = 2;
optional string Territory = 3;
optional string City = 4;
optional string Locality = 5;
optional string Street = 6;
optional string Building = 7;
optional string Block = 8;
optional string Apartment = 9;
}

message ForeignAddress {
required string Country = 1;
required string Address = 2;
}
=== Address.proto ===

=== Organization.proto ===
import "Address.proto";

package Diadoc.Api.Proto;

//...
message Department {
required string DepartmentId = 1;
required string ParentDepartmentId = 2;
required string Name = 3;
optional string Abbreviation = 4;
optional string Kpp = 5;
optional Address Address = 6;
optional bool IsDisabled = 7 [default = false];
}
//...
=== Organization.proto ===

Every attempt I was able to imagine yields an error

Error processing serialized protobuf descriptor: Organization.proto: Import "Address.proto" has not been loaded.
Error processing serialized protobuf descriptor: Organization.proto: ".Diadoc.Api.Proto.Address" is not defined.

Looks like either your module does not handle import properly or I'm doing some quite simple error

TBD whether they should create a fresh array every time or store it on first fetch.

Also check this does not change the behaviour of other list accessors.

For example the mysqlx protocol definition at https://github.com/mysql/mysql-server/tree/5.7/rapid/plugin/x/protocol requires separate mappings for Mysqlx, Mysqlx.Connection, ...
which is suboptimal.

This seems to be related to the new boolean values introduces in 5.36.0 and the changes in how Data::Dumper stringifies these starting with 5.37.2. See DrHyde/perl-modules-Test-Differences#21 for more context on this issue.

FYI, I was able to get past the below error by doing export CFLAGS="-std=c++11 -O3"

martin@L3-37:~/.cpanplus/5.22.1/build/liaRtZfsYD/Google-ProtocolBuffers-Dynamic-0.22$ ./Build
Building Google-ProtocolBuffers-Dynamic
Processing XS typemap files...
Generating main XS file...
x86_64-linux-gnu-gcc -I/usr/lib/x86_64-linux-gnu/perl/5.22/CORE -fPIC -D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -xc++ -I/usr/local/share/perl/5.22.1/auto/share/dist/Alien-uPB/include -DNDEBUG -pthread -I/usr/local/include -DPERL_NO_GET_CONTEXT -Isrc -Ibuildtmp -c -D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -o src/descriptorloader.o src/descriptorloader.cpp
In file included from /usr/include/c++/5/mutex:35:0,
from /usr/local/include/google/protobuf/stubs/mutex.h:33,
from /usr/local/include/google/protobuf/stubs/common.h:52,
from /usr/local/include/google/protobuf/descriptor.h:61,
from /usr/local/include/google/protobuf/compiler/importer.h:44,
from src/descriptorloader.h:9,
from src/descriptorloader.cpp:1:
/usr/include/c++/5/bits/c++0x_warning.h:32:2: error: #error This file requires compiler and library support for the ISO C++ 2011 standard. This support must be enabled with the -std=c++11 or -std=gnu++11 compiler options.

martin@L3-37:~/.cpanplus/5.22.1/build/liaRtZfsYD/Google-ProtocolBuffers-Dynamic-0.22$ ./Build test
t/001_load.t .................... 1/1
Failed test 'use Google::ProtocolBuffers::Dynamic;'
at t/001_load.t line 8.
Tried to use 'Google::ProtocolBuffers::Dynamic'.
Error: Can't load '/media/martin/91e9c4bd-9f2c-4734-a835-3af3ebd6fe41/cpanplus/5.22.1/build/liaRtZfsYD/Google-ProtocolBuffers-Dynamic-0.22/blib/arch/auto/Google/ProtocolBuffers/Dynamic/Dynamic.so' for module Google::ProtocolBuffers::Dynamic: /media/martin/91e9c4bd-9f2c-4734-a835-3af3ebd6fe41/cpanplus/5.22.1/build/liaRtZfsYD/Google-ProtocolBuffers-Dynamic-0.22/blib/arch/auto/Google/ProtocolBuffers/Dynamic/Dynamic.so: undefined symbol: _ZN6google8protobuf8internal21StringTypeHandlerBase3NewB5cxx11Ev at /usr/lib/x86_64-linux-gnu/perl/5.22/DynaLoader.pm line 187.
at t/001_load.t line 8.
Compilation failed in require at t/001_load.t line 8.
BEGIN failed--compilation aborted at t/001_load.t line 8.
Looks like you failed 1 test of 1.
t/001_load.t .................... Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/1 subtests

> protoc --pertl-gpd_out=package=test test.proto

protoc-gen-pertl-gpd: program not found or is not executable
Please specify a program using absolute path or make sure the program is available in your PATH system variable
--pertl-gpd_out: protoc-gen-pertl-gpd: Plugin failed with status code 1.


> protoc-gen-perl-gpd --help

/usr/include/c++/8/bits/stl_vector.h:950: std::vector<_Tp, _Alloc>::const_reference std::vector<_Tp, _Alloc>::operator[](std::vector<_Tp, _Alloc>::size_type) const [with _Tp = upb::Def*; _Alloc = std::allocator<upb::Def*>; std::vector<_Tp, _Alloc>::const_reference = upb::Def* const&; std::vector<_Tp, _Alloc>::size_type = long unsigned int]: Assertion '__builtin_expect(__n < this->size(), true)' failed.
Aborted

Happens with protobuf 3.11.0 and 3.19.1 - compiled from source.

Hi, I noticed that one of my scripts is leaking memory, and after investigating it seems that the G::PB::D is the reason. Consider the following script:

#!/usr/bin/env perl
use 5.014;
use strict;
use warnings;

use Google::ProtocolBuffers::Dynamic;

my $dynamic = Google::ProtocolBuffers::Dynamic->new;
$dynamic->load_string( "person.proto", <<'EOT');
syntax = "proto2";
 
package humans;
 
message Person {
  required string name = 1;
  required int32 id = 2;
  optional string email = 3;
}
EOT

$dynamic->map(
    {
        package => 'humans',
        prefix  => 'Humans'
    }
);

for ( 1 .. 100000 ) {
    my $bytes = Humans::Person->encode(
        {
            name => 'foo',
            id   => 42
        }
    );
}

when running it under valgrind I'm getting the following output (with some of it skipped):

$ valgrind --leak-check=full perl example.pl                                                                                                                                                                                                               
==9258== Memcheck, a memory error detector
==9258== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==9258== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==9258== Command: perl example.pl
==9258== 
==9258== 
==9258== HEAP SUMMARY:
==9258==     in use at exit: 10,209,010 bytes in 103,205 blocks
==9258==   total heap usage: 808,722 allocs, 705,517 frees, 237,601,122 bytes allocated
==9258== 
==9258== 8 bytes in 1 blocks are definitely lost in loss record 35 of 1,304
==9258==    at 0x4C2B0E0: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==9258==    by 0x694D5FD: XS_Google__ProtocolBuffers__Dynamic_load_string(cv*) (in /home/pshaydo/perl5/perlbrew/perls/perl-5.24.0/lib/site_perl/5.24.0/x86_64-linux/auto/Google/ProtocolBuffers/Dynamic/Dynamic.so)
==9258==    by 0x4B4CDB: Perl_pp_entersub (in /home/pshaydo/perl5/perlbrew/perls/perl-5.24.0/bin/perl)
==9258==    by 0x4ADB52: Perl_runops_standard (in /home/pshaydo/perl5/perlbrew/perls/perl-5.24.0/bin/perl)
==9258==    by 0x44166F: perl_run (in /home/pshaydo/perl5/perlbrew/perls/perl-5.24.0/bin/perl)
==9258==    by 0x41FE84: main (in /home/pshaydo/perl5/perlbrew/perls/perl-5.24.0/bin/perl)
==9258== 
... (skipped)
==9258== 9,591,072 (9,554,400 direct, 36,672 indirect) bytes in 99,525 blocks are definitely lost in loss record 1,304 of 1,304
==9258==    at 0x4C2B0E0: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==9258==    by 0x6953502: (anonymous namespace)::make_localized_environment(upb::Status*) (in /home/pshaydo/perl5/perlbrew/perls/perl-5.24.0/lib/site_perl/5.24.0/x86_64-linux/auto/Google/ProtocolBuffers/Dynamic/Dynamic.so)
==9258==    by 0x6958506: gpd::Mapper::encode(sv*) (in /home/pshaydo/perl5/perlbrew/perls/perl-5.24.0/lib/site_perl/5.24.0/x86_64-linux/auto/Google/ProtocolBuffers/Dynamic/Dynamic.so)
==9258==    by 0x6950259: XS_Google__ProtocolBuffers__Dynamic__Mapper_encode(cv*) (in /home/pshaydo/perl5/perlbrew/perls/perl-5.24.0/lib/site_perl/5.24.0/x86_64-linux/auto/Google/ProtocolBuffers/Dynamic/Dynamic.so)
==9258==    by 0x4B4CDB: Perl_pp_entersub (in /home/pshaydo/perl5/perlbrew/perls/perl-5.24.0/bin/perl)
==9258==    by 0x4ADB52: Perl_runops_standard (in /home/pshaydo/perl5/perlbrew/perls/perl-5.24.0/bin/perl)
==9258==    by 0x44166F: perl_run (in /home/pshaydo/perl5/perlbrew/perls/perl-5.24.0/bin/perl)
==9258==    by 0x41FE84: main (in /home/pshaydo/perl5/perlbrew/perls/perl-5.24.0/bin/perl)
==9258== 
==9258== LEAK SUMMARY:
==9258==    definitely lost: 9,554,424 bytes in 99,528 blocks
==9258==    indirectly lost: 36,734 bytes in 384 blocks
==9258==      possibly lost: 229,538 bytes in 338 blocks
==9258==    still reachable: 388,314 bytes in 2,955 blocks
==9258==         suppressed: 0 bytes in 0 blocks
==9258== Reachable blocks (those to which a pointer was found) are not shown.
==9258== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==9258== 
==9258== For counts of detected and suppressed errors, rerun with: -v
==9258== ERROR SUMMARY: 102 errors from 102 contexts (suppressed: 0 from 0)

The problem is present starting from module version 0.15.

The pod documentation says to call load_string($string) but I get error message showing three arguments. What are the arguments?

Usage: Google::ProtocolBuffers::Dynamic::load_string(THIS, file, sv) at bin/proto-to-module.pl line 14.

Here is my program:

use Google::ProtocolBuffers::Dynamic;
my $s = q/ message Person { required string name = 1; } /;
my $dynamic = Google::ProtocolBuffers::Dynamic->new;
$dynamic->load_string($s);

Encoding/decoding roundtrip via protobuf or JSON does produce a different result when the set value is 0. See the the following code for a failing testcase

#!/usr/bin/env perl

use strict;
use warnings;
use 5.010;

use Google::ProtocolBuffers::Dynamic;
use Test::More;

my $proto = <<PROTO;
syntax = "proto2";
package test;
message TestProto {
    optional string col1 = 1;
    optional double col2 = 2;
    optional double col3 = 3;
    optional double col4 = 4;                // Implicit zero default
    optional double col5 = 5 [default = 10]; // Explicit non zero default
    optional double col6 = 6 [default = 0];  // Explicit zero default
}
PROTO

my $dynamic = Google::ProtocolBuffers::Dynamic->new();
$dynamic->load_string('test.proto',$proto);
$dynamic->map({
    package             => 'test',
    prefix              => 'Test',
    explicit_defaults   => 0,
    #encode_defaults     => 0,
});
$dynamic->resolve_references();

my $obj = Test::TestProto->new({
    col1    => 'test',
    col2    => 1.234,
    col3    => 0,
    # col4 not set!
    # col5 not set!
    # col6 not set!
});

# Initial object
ok($obj->has_col2,'col2 is set');
ok($obj->has_col3,'col3 is set');
ok(! $obj->has_col4,'col4 is not set');
ok(! $obj->has_col5,'col5 is not set');
ok(! $obj->has_col6,'col6 is not set');
is($obj->get_col2,1.234,'col2 returns set value');
is($obj->get_col3,0,'col3 returns set value');
is($obj->get_col4,0,'col4 returns default value');
is($obj->get_col5,10,'col5 returns default value');
is($obj->get_col6,0,'col6 returns default value');

# Roundtrip via protobuf
my $roundtrip_pb = Test::TestProto->decode($obj->encode);
ok($roundtrip_pb->has_col2,'col2 is still set after protobuf roundtrip');
ok($roundtrip_pb->has_col3,'col3 is still set after protobuf roundtrip');
ok(! $roundtrip_pb->has_col4,'col4 is still not set after protobuf roundtrip');
ok(! $roundtrip_pb->has_col5,'col5 is still not set after protobuf roundtrip');
ok(! $roundtrip_pb->has_col6,'col6 is still not set after protobuf roundtrip');

is($roundtrip_pb->get_col3,0,'col3 returns set value after protobuf roundtrip');
is($roundtrip_pb->get_col4,0,'col4 returns default value after protobuf roundtrip');
is($roundtrip_pb->get_col5,10,'col5 returns default value after protobuf roundtrip');
is($roundtrip_pb->get_col6,0,'col6 returns default value after protobuf roundtrip');

# Roundtrip via JSON
my $roundtrip_json = Test::TestProto->decode_json($obj->encode_json);
ok($roundtrip_json->has_col2,'col2 is still set after JSON roundtrip');
ok($roundtrip_json->has_col3,'col3 is still set after JSON roundtrip');
ok(! $roundtrip_json->has_col4,'col4 is still not set after JSON roundtrip');
ok(! $roundtrip_json->has_col5,'col5 is still not set after JSON roundtrip');
ok(! $roundtrip_json->has_col6,'col6 is still not set after JSON roundtrip');

is($roundtrip_pb->get_col3,0,'col3 returns set value after JSON roundtrip');
is($roundtrip_pb->get_col4,0,'col4 returns default value after JSON roundtrip');
is($roundtrip_pb->get_col5,10,'col5 returns default value after JSON roundtrip');
is($roundtrip_pb->get_col6,0,'col6 returns default value after JSON roundtrip');

Tested with version 0.16 and perl 5.22.3

We have messages that contain sometimes empty string values, I would expect them to be decoded as empty strings, but we're getting undef instead. I added a test to demonstrate the problem here: e020d6b

Would be nice to have a way to resolve protobuf message names to the perl classes they are mapped to at runtime. So if I have a protobuf bytes blob and I know the protobuf message name for it, I could feed it to the appropriate perl class for decoding. For example $dynamic->resolve_message($protobuf_name) would return the perl class mapped to it.

Use case: I'm toying around implementing twirp support for perl, and basically mimicking what they do for their python support. Python protobuf stuff has a symbol db that can be used to look up things like this. For encoding there's no need for that as the given perl object can just be told to ->encode itself, but for decoding I'd need to have a class to feed the bytes to.

It's possible to map enum keys that are also keywords in the main:: package

$ cat PB_evil_enums.pm
package PB_evil_enums;

use strict;
use warnings;
use Google::ProtocolBuffers::Dynamic;

my $gpd = Google::ProtocolBuffers::Dynamic->new;
$gpd->load_string('', <<'EOD');
syntax = "proto2";
package evilenum_pb;

message console_log_line {
    // from: https://perldoc.perl.org/constant#CAVEATS
    enum perl_keywords {
        STDOUT = 1;
        STDERR = 2;
    }

    optional perl_keywords the_enum = 1;
}
EOD

$gpd->map(
   +{
      'package' => 'evilenum_pb',
      'prefix' => 'PB_evil_enums::_pb'
    },
);

print STDERR "STDERR is okay from module\n";

undef $gpd;
1;

loading this module causes misattributed errors when compiling subsequent scripts & modules:

$ perl -e 'print STDERR "hi"';
hi

$ perl -I. -MPB_evil_enums -e 'print STDERR "hi"';
STDERR is okay from module
String found where operator expected at -e line 1, near "STDERR "hi""
	(Do you need to predeclare STDERR?

this error is due to main::STDERR being converted from a fileglob to a constant sub:

$ perl -I. -MPB_evil_enums -e 'print STDERR()';
STDERR is okay from module
2

Perhaps src/dynamic.cpp: Dynamic::map_enum() should be dying on any enum field named with a known perl keyword, or prefixing/suffixing those words with some special string to avoid the collision (but perhaps cause another?).

I don't speak C++ so this looks very cryptic to me, but using g++ 4.9.2 on Debian Jessie with protobuf 3.5.1 freshly compiled and installed, the build errors out. GCC is unhappy about 4 properties being constant without there being an explicit assignment overload method:

In file included from src/dynamic.cpp:3:0:
src/servicedef.h: In instantiation of ‘void std::vector<_Tp, _Alloc>::_M_insert_aux(std::vector<_Tp, _Alloc>::iterator, const _Tp&) [with _Tp = gpd::MethodDef; _Alloc = std::allocator<gpd::MethodDef>; std::vector<_Tp, _Alloc>::iterator = __gnu_cxx::__normal_iterator<gpd::MethodDef*, std::vector<gpd::MethodDef> >; typename std::_Vector_base<_Tp, _Alloc>::pointer = gpd::MethodDef*]’:
/usr/include/c++/4.9/bits/stl_vector.h:925:28:   required from ‘void std::vector<_Tp, _Alloc>::push_back(const value_type&) [with _Tp = gpd::MethodDef; _Alloc = std::allocator<gpd::MethodDef>; std::vector<_Tp, _Alloc>::value_type = gpd::MethodDef]’
src/dynamic.cpp:212:9:   required from here
src/servicedef.h:16:12: error: non-static const member ‘const string gpd::MethodDef::name’, can’t use default assignment operator
     struct MethodDef {
            ^
src/servicedef.h:16:12: error: non-static const member ‘const string gpd::MethodDef::full_name’, can’t use default assignment operator
src/servicedef.h:16:12: error: non-static const member ‘const bool gpd::MethodDef::client_streaming’, can’t use default assignment operator
src/servicedef.h:16:12: error: non-static const member ‘const bool gpd::MethodDef::server_streaming’, can’t use default assignment operator
In file included from /usr/include/c++/4.9/vector:69:0,
                 from /usr/local/include/google/protobuf/compiler/importer.h:41,
                 from src/dynamic.h:6,
                 from src/dynamic.cpp:1:
/usr/include/c++/4.9/bits/vector.tcc:343:16: note: synthesized method ‘gpd::MethodDef& gpd::MethodDef::operator=(const gpd::MethodDef&)’ first required here 
    *__position = __x_copy;
                ^
error building src/dynamic.o from 'src/dynamic.cpp' at /usr/share/perl/5.20/ExtUtils/CBuilder/Base.pm line 175.

Do I have an out of date library somewhere or is this an actual bug?

On - CentOS Linux release 7.9.2009 (Core)

Run -

# cpanm Google::ProtocolBuffers::Dynamic

Result:

TL;DR -

g++ -I/usr/lib64/perl5/CORE -fPIC -xc++ -I/usr/local/share/perl5/auto/share/dist/Alien-uPB-Core/include -DNDEBUG -DPERL_NO_GET_CONTEXT -Isrc -Ibuildtmp -c -D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -o src/sourcetree.o src/sourcetree.cpp
src/sourcetree.cpp: In member function 'virtual std::string gpd::OverlaySourceTree::GetLastErrorMessage()':
src/sourcetree.cpp:25:22: error: 'class google::protobuf::compiler::SourceTree' has no member named 'GetLastErrorMessage'
     return fallback->GetLastErrorMessage();
                      ^
src/sourcetree.cpp:26:1: warning: control reaches end of non-void function [-Wreturn-type]
 }
 ^
error building src/sourcetree.o from 'src/sourcetree.cpp' at /usr/local/share/perl5/ExtUtils/CBuilder/Base.pm line 185.
-> FAIL Installing Google::ProtocolBuffers::Dynamic failed. See /root/.cpanm/work/1618848110.11530/build.log for details. Retry with --force to force install it.

The error is generated in this source file -

/root/.cpanm/work/1618848110.11530/Google-ProtocolBuffers-Dynamic-0.31/src/sourcetree.cpp

Full log -

cpanm (App::cpanminus) 1.6922 on perl 5.016003 built for x86_64-linux-thread-multi
Work directory is /root/.cpanm/work/1618848110.11530
You have make /usr/bin/make
You have LWP 6.05
You have /usr/bin/tar: tar (GNU tar) 1.26
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by John Gilmore and Jay Fenlason.
Searching Google::ProtocolBuffers::Dynamic on cpanmetadb ...
--> Working on Google::ProtocolBuffers::Dynamic
Fetching http://www.cpan.org/authors/id/M/MB/MBARBON/Google-ProtocolBuffers-Dynamic-0.31.tar.gz
-> OK
Unpacking Google-ProtocolBuffers-Dynamic-0.31.tar.gz
Entering Google-ProtocolBuffers-Dynamic-0.31
Checking configure dependencies from META.json
Checking if you have Alien::ProtoBuf 0.08 ... Yes (0.09)
Checking if you have Module::Build::WithXSpp 0.14 ... Yes (0.14)
Checking if you have Alien::uPB::Core 0.16 ... Yes (0.17)
Checking if you have Module::Build 0.28 ... Yes (0.4005)
Checking if you have Module::Build 0.36 ... Yes (0.4005)
Configuring Google-ProtocolBuffers-Dynamic-0.31
Running Build.PL
Created MYMETA.yml and MYMETA.json
Creating new 'Build' script for 'Google-ProtocolBuffers-Dynamic' version '0.31'
-> OK
Checking dependencies from MYMETA.json ...
Checking if you have ExtUtils::Typemaps::Default 1.05 ... Yes (1.05)
Checking if you have Test::Exception 0 ... Yes (0.43)
Checking if you have ExtUtils::ParseXS 2.30 ... Yes (3.35)
Checking if you have Test::Warn 0 ... Yes (0.36)
Checking if you have autodie 0 ... Yes (2.34)
Checking if you have ExtUtils::XSpp 0.18 ... Yes (0.18)
Checking if you have Module::Build 0.28 ... Yes (0.4005)
Checking if you have ExtUtils::Typemaps::STL::String 0 ... Yes (1.05)
Building and testing Google-ProtocolBuffers-Dynamic-0.31
Building Google-ProtocolBuffers-Dynamic
Processing XS typemap files...
Generating main XS file...
g++ -I/usr/lib64/perl5/CORE -fPIC -xc++ -I/usr/local/share/perl5/auto/share/dist/Alien-uPB-Core/include -DNDEBUG -DPERL_NO_GET_CONTEXT -Isrc -Ibuildtmp -c -D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -o src/mapper.o src/mapper.cpp
src/mapper.cpp:81:0: warning: "av_top_index" redefined [enabled by default]
     #define av_top_index(av) GPD_av_top_index(aTHX_ (av))
 ^
In file included from src/mapper.h:18:0,
                 from src/mapper.cpp:1:
src/ppport.h:4222:0: note: this is the location of the previous definition
 #  define av_top_index                   AvFILL
 ^
src/mapper.cpp: In member function 'bool gpd::Mapper::DecoderHandlers::apply_defaults_and_check()':
src/mapper.cpp:147:20: warning: enumeration value 'UPB_TYPE_MESSAGE' not handled in switch [-Wswitch]
             switch (field.field_def->type()) {
                    ^
src/mapper.cpp: In function 'bool {anonymous}::set_bigint(PerlInterpreter*, SV*, uint64_t, bool)':
src/mapper.cpp:430:13: warning: unused variable 'count' [-Wunused-variable]
         int count = call_method("new", G_SCALAR);
             ^
src/mapper.cpp: In function 'uint64_t {anonymous}::extract_bits(PerlInterpreter*, SV*, bool*)':
src/mapper.cpp:943:13: warning: unused variable 'count' [-Wunused-variable]
         int count = call_method("as_hex", G_SCALAR);
             ^
src/mapper.cpp: In member function 'bool gpd::Mapper::encode_from_perl_hash(upb::Sink*, upb::Status*, const gpd::Mapper::Field&, SV*) const':
src/mapper.cpp:1546:14: warning: unused variable 'needs_free' [-Wunused-variable]
         bool needs_free = false;
              ^
g++ -I/usr/lib64/perl5/CORE -fPIC -xc++ -I/usr/local/share/perl5/auto/share/dist/Alien-uPB-Core/include -DNDEBUG -DPERL_NO_GET_CONTEXT -Isrc -Ibuildtmp -c -D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -o src/sourcetree.o src/sourcetree.cpp
src/sourcetree.cpp: In member function 'virtual std::string gpd::OverlaySourceTree::GetLastErrorMessage()':
src/sourcetree.cpp:25:22: error: 'class google::protobuf::compiler::SourceTree' has no member named 'GetLastErrorMessage'
     return fallback->GetLastErrorMessage();
                      ^
src/sourcetree.cpp:26:1: warning: control reaches end of non-void function [-Wreturn-type]
 }
 ^
error building src/sourcetree.o from 'src/sourcetree.cpp' at /usr/local/share/perl5/ExtUtils/CBuilder/Base.pm line 185.
-> FAIL Installing Google::ProtocolBuffers::Dynamic failed. See /root/.cpanm/work/1618848110.11530/build.log for details. Retry with --force to force install it.

Hi! I'm reporting this here in case anyone else stumbles on this issue.

I notice that I have to use X.pm (generated by protoc) as early as possible in my program flow in order to avoid that warning, which I'm hoping is benign.

That said, with Apache/mod_perl I cannot get rid of that warning every time a new Apache child is forked. I tried using the module in my handler.pl, and even in the Apache configuration itself in PerlModule, PerlLoadModule, PerlRequire, PerlPostConfigRequire.

The only places I found CHECK in /usr/ were in ExtUtils/XSpp/Grammar.pm and this module's Dynamic.so. I'm not too sure how to investigate this further. There doesn't seem to be a way to look at generated code since it's generated dynamically at module load time.

Try to use protoc-gen-perl-gpd to generate module but the script never exits and prints warnings.

root@5332d16ec3b9:/client# protoc-gen-perl-gpd --perl-gpd_out=package=Avatica.Client.Protocol:lib 1.proto
\1 better written as $1 at /usr/local/lib/perl5/site_perl/5.34.0/x86_64-linux-gnu/Google/ProtocolBuffers/Dynamic/MakeModule.pm line 57.
\2 better written as $2 at /usr/local/lib/perl5/site_perl/5.34.0/x86_64-linux-gnu/Google/ProtocolBuffers/Dynamic/MakeModule.pm line 57.
^C
root@5332d16ec3b9:/client#

Version of Google::ProtocolBuffers::Dynamic is 0.32.
Version of perl is 5.34.

if I map a proto before launching-and-joining two threads (or join just one thread and then exit the main process), I get warning outputs from glibc and a coredump:

#!/usr/bin/perl
use strict;
use warnings;
use threads;
use Google::ProtocolBuffers::Dynamic;
my $dynamic = Google::ProtocolBuffers::Dynamic->new;
$dynamic->load_string("person.proto", <<'EOT');
syntax = "proto2";

package humans;

message Person {
  required string name = 1;
  required int32 id = 2;
  optional string email = 3;
}
EOT

$dynamic->map({ package => 'humans', prefix => 'Humans' });

for (1..2) {
    my $thr = threads->create({'context' => 'list', 'exit' => 'thread_only'}, sub {
        my $tid = threads->tid();
        warn "new thread: $tid";
        return $tid;
    });
}
my @results = map { $_->join(); } threads->list();
warn "joined threads: @results";

$ ./crash.pl
new thread: 1 at ./crash.pl line 24.
new thread: 2 at ./crash.pl line 24.
corrupted double-linked list
Aborted (core dumped)

cribbing from another CPAN module, I wrote this patch to fix this particular scenario:

diff --git a/src/dynamic.cpp b/src/dynamic.cpp
index 9928288..bac7ee7 100644
--- a/src/dynamic.cpp
+++ b/src/dynamic.cpp
@@ -186,6 +186,13 @@ namespace {
         return 0;
     }
 
+    int dup_refcounted(pTHX_ MAGIC *mg, CLONE_PARAMS *param) {
+        Refcounted *refcounted = (Refcounted *) mg->mg_ptr;
+        refcounted->ref();
+
+        return 0;
+    }
+
     MGVTBL manage_refcounted = {
         NULL, // get
         NULL, // set
@@ -193,7 +200,7 @@ namespace {
         NULL, // clear
         free_refcounted,
         NULL, // copy
-        NULL, // dup
+        dup_refcounted,
         NULL, // local
     };
 
@@ -209,9 +216,10 @@ namespace {
         CV *new_xs = newXS((perl_package + "::" + target).c_str(), CvXSUB(src), __FILE__);
 
         CvXSUBANY(new_xs).any_ptr = refcounted;
-        sv_magicext((SV *) new_xs, NULL,
+        MAGIC *mg = sv_magicext((SV *) new_xs, NULL,
                     PERL_MAGIC_ext, &manage_refcounted,
                     (const char *) refcounted, 0);
+        mg->mg_flags |= MGf_DUP;
         refcounted->ref();
     }

this may be too simple of a fix for users that detach threads instead of joining them. but I think that leaking is preferable to double-freeing and crashing out.

fwiw my application does not use protos in its threads but spawns them upfront for auxiliary chores. I've resorted to requiring my protos' generated modules after these threads have been spawned but this may not be doable in many applications, either because they wish to preload all code or because they spawn threads intermittently after a mapping may have been done.

The underlying upb library provides const_field_iterator which is used in mapper.cpp to iterate over FieldDefs and field information is stored in that order in a fields vector.

Later, const_oneof_iterator is used to iterate over the "oneof" fields, and field_def->index() is used to access the fields vector to delete any existing value for the "oneof".

The presumption here is that field_def->index() is usable to index the list of fields returned by const_field_iterator in the order they're returned.

This presumption turns out not to be correct. The question is, what is correct?

• Should upb's const_field_iterator return fields in field_def->index() order?

• Or is field_def->index() really unrelated to the order that const_field_iterator returns fields in, and Mapper::Mapper() should use field_def->index() to populate the vector of fields?

To recreate this issue:

tim.proto:

syntax = "proto3";

message A {

	int32 val = 1;

	oneof foo {
		B b = 2;
		C c = 3;
	}

	message B {
		int32 val = 1;
	}

	message C {
		int32 val = 1;
	}
}

tim.pl:

#!/usr/bin/perl
use strict;
use warnings;
use 5.008;
use Google::ProtocolBuffers::Dynamic;

my $dynamic = Google::ProtocolBuffers::Dynamic->new( '.' );
$dynamic->load_file( 'tim.proto' );
$dynamic->map( { message => 'A', to => 'A' } );

my $json = '{"val":5,"b":{"val":6}}';

my $data = A->decode_json( $json );

print "A.val = ".$data->get_val."\n";

output:

$ ./tim.pl
A.val = 0

epected output:

$ ./tim.pl
A.val = 5

This happens because, when decoding "b":{"val":6}, the Field definition for b has a oneof_index which causes the decoder to look for any existing value in that "oneof" group, which will need to be deleted. Unfortunately, the initial "val":5 field definition also says it's in the same "oneof" group, and so A.val is deleted.

The oneof_indexes are set up incorrectly in Mapper::Mapper(), when iterating over the oneof groups. The following code in mapper.cpp is where that happens:

        for (OneofDef::const_iterator it = oneof_def->begin(), en = oneof_def->end(); it != en; ++it) {
            const FieldDef *field_def = *it;
            Field &field = fields[field_def->index()]; // <-- CAN PICK WRONG FIELD
            field.oneof_index = oneof_index;
        }

What is the correct fix? A change in mapper.cpp? Or a change in upb?

I am trying to install Google::ProtocolBuffers::Dynamic on RHEL 8.4 using cpanm but it fails. I have protobuf-c and protobuf-devel already installed. I have pasted the protobuf lib version and build.log output below.

[root@95382e5d8906 /]# cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="8.4 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.4"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.4
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.4"

[root@95382e5d8906 /]# find / -name libprotobuf*.so
/usr/lib64/libprotobuf-c.so.1
/usr/lib64/libprotobuf.so
/usr/lib64/libprotobuf.so.15
/usr/lib64/libprotobuf.so.15.0.0
/usr/lib64/libprotobuf-c.so.1.0.0

[root@95382e5d8906 /]# cpanm Google::ProtocolBuffers::Dynamic
--> Working on Google::ProtocolBuffers::Dynamic
Fetching http://www.cpan.org/authors/id/M/MB/MBARBON/Google-ProtocolBuffers-Dynamic-0.34.tar.gz ... OK
Configuring Google-ProtocolBuffers-Dynamic-0.34 ... OK
Building and testing Google-ProtocolBuffers-Dynamic-0.34 ... FAIL
! Installing Google::ProtocolBuffers::Dynamic failed. See /root/.cpanm/work/1638441235.10898/build.log for details. Retry with --force to force install it.

[root@95382e5d8906 /]# cat /root/.cpanm/work/1638441235.10898/build.log
cpanm (App::cpanminus) 1.7044 on perl 5.026003 built for x86_64-linux-thread-multi
Work directory is /root/.cpanm/work/1638441235.10898
You have make /usr/bin/make
You have LWP 6.34
You have /usr/bin/tar: tar (GNU tar) 1.30
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later https://gnu.org/licenses/gpl.html.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by John Gilmore and Jay Fenlason.
You have /usr/bin/unzip
Searching Google::ProtocolBuffers::Dynamic () on cpanmetadb ...
--> Working on Google::ProtocolBuffers::Dynamic
Fetching http://www.cpan.org/authors/id/M/MB/MBARBON/Google-ProtocolBuffers-Dynamic-0.34.tar.gz
-> OK
Unpacking Google-ProtocolBuffers-Dynamic-0.34.tar.gz
Entering Google-ProtocolBuffers-Dynamic-0.34
Checking configure dependencies from META.json
Checking if you have Module::Build 0.38 ... Yes (0.4224)
Checking if you have Module::Build::WithXSpp 0.14 ... Yes (0.14)
Checking if you have ExtUtils::Install 1.46 ... Yes (2.14)
Checking if you have Alien::uPB::Core 0.20 ... Yes (0.20)
Checking if you have Alien::ProtoBuf 0.08 ... Yes (0.09)
Configuring Google-ProtocolBuffers-Dynamic-0.34
Running Build.PL
Created MYMETA.yml and MYMETA.json
Creating new 'Build' script for 'Google-ProtocolBuffers-Dynamic' version '0.34'
-> OK
Checking dependencies from MYMETA.json ...
Checking if you have ExtUtils::Typemaps::STL::String 0 ... Yes (1.05)
Checking if you have Module::Build 0.28 ... Yes (0.4224)
Checking if you have ExtUtils::Typemaps::Default 1.05 ... Yes (1.05)
Checking if you have autodie 0 ... Yes (2.29)
Checking if you have ExtUtils::ParseXS 2.30 ... Yes (3.35)
Checking if you have ExtUtils::XSpp 0.18 ... Yes (0.18)
Checking if you have Test::Warn 0 ... Yes (0.36)
Checking if you have Test::Exception 0 ... Yes (0.43)
Building and testing Google-ProtocolBuffers-Dynamic-0.34
Building Google-ProtocolBuffers-Dynamic
Processing XS typemap files...
Generating main XS file...
g++ -I/usr/lib64/perl5/CORE -fPIC -xc++ -I/usr/local/share/perl5/auto/share/dist/Alien-uPB-Core/include -DNDEBUG -DPERL_NO_GET_CONTEXT -Isrc -Ibuildtmp -c -D_REENTRANT -D_GNU_SOURCE -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fwrapv -fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -g -o src/descriptorloader.o src/descriptorloader.cpp
g++ -I/usr/lib64/perl5/CORE -fPIC -xc++ -I/usr/local/share/perl5/auto/share/dist/Alien-uPB-Core/include -DNDEBUG -DPERL_NO_GET_CONTEXT -Isrc -Ibuildtmp -c -D_REENTRANT -D_GNU_SOURCE -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fwrapv -fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -g -o src/upb/bridge.o src/upb/bridge.cpp
g++ -I/usr/lib64/perl5/CORE -fPIC -xc++ -I/usr/local/share/perl5/auto/share/dist/Alien-uPB-Core/include -DNDEBUG -DPERL_NO_GET_CONTEXT -Isrc -Ibuildtmp -c -D_REENTRANT -D_GNU_SOURCE -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fwrapv -fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -g -o src/sourcetree.o src/sourcetree.cpp
g++ -I/usr/lib64/perl5/CORE -fPIC -xc++ -I/usr/local/share/perl5/auto/share/dist/Alien-uPB-Core/include -DNDEBUG -DPERL_NO_GET_CONTEXT -Isrc -Ibuildtmp -c -D_REENTRANT -D_GNU_SOURCE -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fwrapv -fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -g -o src/dynamic.o src/dynamic.cpp
In file included from src/dynamic.cpp:1:
src/dynamic.h: In constructor ‘gpd::MappingOptions::MappingOptions(PerlInterpreter*, SV*)’:
src/dynamic.h:50:19: warning: ‘gpd::MappingOptions::client_services’ will be initialized after [-Wreorder]
ClientService client_services;
^~~~~~~~~~~~~~~
src/dynamic.h:48:10: warning: ‘bool gpd::MappingOptions::fail_ref_coercion’ [-Wreorder]
bool fail_ref_coercion;
^~~~~~~~~~~~~~~~~
src/dynamic.cpp:27:1: warning: when initialized here [-Wreorder]
MappingOptions::MappingOptions(pTHX_ SV options_ref) :
^~~~~~~~~~~~~~
src/dynamic.cpp: In member function ‘void gpd::Dynamic::map_message_prefix_recursive(PerlInterpreter, const google::protobuf::Descriptor*, const string&, const gpd::MappingOptions&, std::tr1::unordered_set<std::_cxx11::basic_string >&)’:
src/dynamic.cpp:376:9: warning: enumeration value ‘TYPE_DOUBLE’ not handled in switch [-Wswitch]
switch( field->type() ) {
^
src/dynamic.cpp:376:9: warning: enumeration value ‘TYPE_FLOAT’ not handled in switch [-Wswitch]
src/dynamic.cpp:376:9: warning: enumeration value ‘TYPE_INT64’ not handled in switch [-Wswitch]
src/dynamic.cpp:376:9: warning: enumeration value ‘TYPE_UINT64’ not handled in switch [-Wswitch]
src/dynamic.cpp:376:9: warning: enumeration value ‘TYPE_INT32’ not handled in switch [-Wswitch]
src/dynamic.cpp:376:9: warning: enumeration value ‘TYPE_FIXED64’ not handled in switch [-Wswitch]
src/dynamic.cpp:376:9: warning: enumeration value ‘TYPE_FIXED32’ not handled in switch [-Wswitch]
src/dynamic.cpp:376:9: warning: enumeration value ‘TYPE_BOOL’ not handled in switch [-Wswitch]
src/dynamic.cpp:376:9: warning: enumeration value ‘TYPE_STRING’ not handled in switch [-Wswitch]
src/dynamic.cpp:376:9: warning: enumeration value ‘TYPE_GROUP’ not handled in switch [-Wswitch]
src/dynamic.cpp:376:9: warning: enumeration value ‘TYPE_BYTES’ not handled in switch [-Wswitch]
src/dynamic.cpp:376:9: warning: enumeration value ‘TYPE_UINT32’ not handled in switch [-Wswitch]
src/dynamic.cpp:376:9: warning: enumeration value ‘TYPE_SFIXED32’ not handled in switch [-Wswitch]
src/dynamic.cpp:376:9: warning: enumeration value ‘TYPE_SFIXED64’ not handled in switch [-Wswitch]
src/dynamic.cpp:376:9: warning: enumeration value ‘TYPE_SINT32’ not handled in switch [-Wswitch]
src/dynamic.cpp:376:9: warning: enumeration value ‘TYPE_SINT64’ not handled in switch [-Wswitch]
src/dynamic.cpp:376:9: warning: enumeration value ‘MAX_TYPE’ not handled in switch [-Wswitch]
g++ -I/usr/lib64/perl5/CORE -fPIC -xc++ -I/usr/local/share/perl5/auto/share/dist/Alien-uPB-Core/include -DNDEBUG -DPERL_NO_GET_CONTEXT -Isrc -Ibuildtmp -c -D_REENTRANT -D_GNU_SOURCE -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fwrapv -fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -g -o src/mapper.o src/mapper.cpp
src/mapper.cpp: In member function ‘bool gpd::Mapper::DecoderHandlers::apply_defaults_and_check()’:
src/mapper.cpp:147:20: warning: enumeration value ‘UPB_TYPE_MESSAGE’ not handled in switch [-Wswitch]
switch (field.field_def->type()) {
^
src/mapper.cpp: In function ‘bool {anonymous}::set_bigint(PerlInterpreter*, SV*, uint64_t, bool)’:
src/mapper.cpp:431:13: warning: unused variable ‘count’ [-Wunused-variable]
int count = call_method("new", G_SCALAR);
^~~~~
src/mapper.cpp: In function ‘uint64_t {anonymous}::extract_bits(PerlInterpreter*, SV*, bool*)’:
src/mapper.cpp:944:13: warning: unused variable ‘count’ [-Wunused-variable]
int count = call_method("as_hex", G_SCALAR);
^~~~~
src/mapper.cpp: In member function ‘bool gpd::Mapper::encode_from_perl_hash(upb::Sink*, upb::Status*, const gpd::Mapper::Field&, SV*) const’:
src/mapper.cpp:1547:14: warning: unused variable ‘needs_free’ [-Wunused-variable]
bool needs_free = false;
^~~~~~~~~~
g++ -I/usr/lib64/perl5/CORE -DVERSION="0.34" -DXS_VERSION="0.34" -fPIC -xc++ -I/usr/local/share/perl5/auto/share/dist/Alien-uPB-Core/include -DNDEBUG -DPERL_NO_GET_CONTEXT -Isrc -Ibuildtmp -c -D_REENTRANT -D_GNU_SOURCE -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fwrapv -fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -g -o buildtmp/Dynamic.o buildtmp/Dynamic.c
buildtmp/Dynamic.c: In function ‘void XS_Google__ProtocolBuffers__Dynamic__Mapper_new(PerlInterpreter*, CV*)’:
buildtmp/Dynamic.c:2507:7: warning: unused variable ‘klass’ [-Wunused-variable]
SV * klass = ST(0)
^~~~~
buildtmp/Dynamic.c: In function ‘void XS_Google__ProtocolBuffers__Dynamic__Mapper_new_and_check(PerlInterpreter*, CV*)’:
buildtmp/Dynamic.c:2538:7: warning: unused variable ‘klass’ [-Wunused-variable]
SV * klass = ST(0)
^~~~~
buildtmp/Dynamic.c: In function ‘void XS_Google__ProtocolBuffers__Dynamic__Mapper_message_descriptor(PerlInterpreter*, CV*)’:
buildtmp/Dynamic.c:2572:7: warning: unused variable ‘klass’ [-Wunused-variable]
SV * klass = ST(0)
^~~~~
buildtmp/Dynamic.c: In function ‘void XS_Google__ProtocolBuffers__Dynamic__Mapper_enum_descriptor(PerlInterpreter*, CV*)’:
buildtmp/Dynamic.c:2595:7: warning: unused variable ‘klass’ [-Wunused-variable]
SV * klass = ST(0)
^~~~~
buildtmp/Dynamic.c: In function ‘void XS_Google__ProtocolBuffers__Dynamic__Mapper_service_descriptor(PerlInterpreter*, CV*)’:
buildtmp/Dynamic.c:2618:7: warning: unused variable ‘klass’ [-Wunused-variable]
SV * klass = ST(0)
^~~~~
buildtmp/Dynamic.c: In function ‘void XS_Google__ProtocolBuffers__Dynamic__Mapper_decode(PerlInterpreter*, CV*)’:
buildtmp/Dynamic.c:2641:7: warning: unused variable ‘klass’ [-Wunused-variable]
SV * klass = ST(0)
^~~~~
buildtmp/Dynamic.c: In function ‘void XS_Google__ProtocolBuffers__Dynamic__Mapper_decode_json(PerlInterpreter*, CV*)’:
buildtmp/Dynamic.c:2703:7: warning: unused variable ‘klass’ [-Wunused-variable]
SV * klass = ST(0)
^~~~~
In file included from buildtmp/main.xs:12:
/usr/bin/perl -MExtUtils::XSpp::Cmd -e xspp -- -t "/root/.cpanm/work/1638441235.10898/Google-ProtocolBuffers-Dynamic-0.34/xsp/main.xspt" "/root/.cpanm/work/1638441235.10898/Google-ProtocolBuffers-Dynamic-0.34/xsp/mapper.xsp": In function ‘void XS_Google__ProtocolBuffers__Dynamic__Mapper_get_list(PerlInterpreter*, CV*)’:
/usr/lib64/perl5/CORE/XSUB.h:183:28: warning: unused variable ‘targ’ [-Wunused-variable]
#define dXSTARG SV * const targ = ((PL_op->op_private & OPpENTERSUB_HASTARG)
^~~~
/usr/bin/perl -MExtUtils::XSpp::Cmd -e xspp -- -t "/root/.cpanm/work/1638441235.10898/Google-ProtocolBuffers-Dynamic-0.34/xsp/main.xspt" "/root/.cpanm/work/1638441235.10898/Google-ProtocolBuffers-Dynamic-0.34/xsp/mapper.xsp":364:5: note: in expansion of macro ‘dXSTARG’
/usr/bin/perl -MExtUtils::XSpp::Cmd -e xspp -- -t "/root/.cpanm/work/1638441235.10898/Google-ProtocolBuffers-Dynamic-0.34/xsp/main.xspt" "/root/.cpanm/work/1638441235.10898/Google-ProtocolBuffers-Dynamic-0.34/xsp/mapper.xsp": In function ‘void XS_Google__ProtocolBuffers__Dynamic__Mapper_get_extension_list(PerlInterpreter*, CV*)’:
/usr/lib64/perl5/CORE/XSUB.h:183:28: warning: unused variable ‘targ’ [-Wunused-variable]
#define dXSTARG SV * const targ = ((PL_op->op_private & OPpENTERSUB_HASTARG)
^~~~
/usr/bin/perl -MExtUtils::XSpp::Cmd -e xspp -- -t "/root/.cpanm/work/1638441235.10898/Google-ProtocolBuffers-Dynamic-0.34/xsp/main.xspt" "/root/.cpanm/work/1638441235.10898/Google-ProtocolBuffers-Dynamic-0.34/xsp/mapper.xsp":372:5: note: in expansion of macro ‘dXSTARG’
/usr/bin/perl -MExtUtils::XSpp::Cmd -e xspp -- -t "/root/.cpanm/work/1638441235.10898/Google-ProtocolBuffers-Dynamic-0.34/xsp/main.xspt" "/root/.cpanm/work/1638441235.10898/Google-ProtocolBuffers-Dynamic-0.34/xsp/mapper.xsp": In function ‘void XS_Google__ProtocolBuffers__Dynamic__Mapper_get_or_set_list(PerlInterpreter*, CV*)’:
/usr/lib64/perl5/CORE/XSUB.h:183:28: warning: unused variable ‘targ’ [-Wunused-variable]
#define dXSTARG SV * const targ = ((PL_op->op_private & OPpENTERSUB_HASTARG)
^~~~
/usr/bin/perl -MExtUtils::XSpp::Cmd -e xspp -- -t "/root/.cpanm/work/1638441235.10898/Google-ProtocolBuffers-Dynamic-0.34/xsp/main.xspt" "/root/.cpanm/work/1638441235.10898/Google-ProtocolBuffers-Dynamic-0.34/xsp/mapper.xsp":394:5: note: in expansion of macro ‘dXSTARG’
/usr/bin/perl -MExtUtils::XSpp::Cmd -e xspp -- -t "/root/.cpanm/work/1638441235.10898/Google-ProtocolBuffers-Dynamic-0.34/xsp/main.xspt" "/root/.cpanm/work/1638441235.10898/Google-ProtocolBuffers-Dynamic-0.34/xsp/mapper.xsp": In function ‘void XS_Google__ProtocolBuffers__Dynamic__Mapper_get_or_set_extension_list(PerlInterpreter*, CV*)’:
/usr/lib64/perl5/CORE/XSUB.h:183:28: warning: unused variable ‘targ’ [-Wunused-variable]
#define dXSTARG SV * const targ = ((PL_op->op_private & OPpENTERSUB_HASTARG)
^~~~
/usr/bin/perl -MExtUtils::XSpp::Cmd -e xspp -- -t "/root/.cpanm/work/1638441235.10898/Google-ProtocolBuffers-Dynamic-0.34/xsp/main.xspt" "/root/.cpanm/work/1638441235.10898/Google-ProtocolBuffers-Dynamic-0.34/xsp/mapper.xsp":405:5: note: in expansion of macro ‘dXSTARG’
/usr/bin/perl -MExtUtils::XSpp::Cmd -e xspp -- -t "/root/.cpanm/work/1638441235.10898/Google-ProtocolBuffers-Dynamic-0.34/xsp/main.xspt" "/root/.cpanm/work/1638441235.10898/Google-ProtocolBuffers-Dynamic-0.34/xsp/mapper.xsp": In function ‘void XS_Google__ProtocolBuffers__Dynamic__Mapper_get_map(PerlInterpreter*, CV*)’:
/usr/lib64/perl5/CORE/XSUB.h:183:28: warning: unused variable ‘targ’ [-Wunused-variable]
#define dXSTARG SV * const targ = ((PL_op->op_private & OPpENTERSUB_HASTARG)
^~~~
/usr/bin/perl -MExtUtils::XSpp::Cmd -e xspp -- -t "/root/.cpanm/work/1638441235.10898/Google-ProtocolBuffers-Dynamic-0.34/xsp/main.xspt" "/root/.cpanm/work/1638441235.10898/Google-ProtocolBuffers-Dynamic-0.34/xsp/mapper.xsp":442:5: note: in expansion of macro ‘dXSTARG’
/usr/bin/perl -MExtUtils::XSpp::Cmd -e xspp -- -t "/root/.cpanm/work/1638441235.10898/Google-ProtocolBuffers-Dynamic-0.34/xsp/main.xspt" "/root/.cpanm/work/1638441235.10898/Google-ProtocolBuffers-Dynamic-0.34/xsp/mapper.xsp": In function ‘void XS_Google__ProtocolBuffers__Dynamic__Mapper_get_or_set_map(PerlInterpreter*, CV*)’:
/usr/lib64/perl5/CORE/XSUB.h:183:28: warning: unused variable ‘targ’ [-Wunused-variable]
#define dXSTARG SV * const targ = ((PL_op->op_private & OPpENTERSUB_HASTARG)
^~~~
/usr/bin/perl -MExtUtils::XSpp::Cmd -e xspp -- -t "/root/.cpanm/work/1638441235.10898/Google-ProtocolBuffers-Dynamic-0.34/xsp/main.xspt" "/root/.cpanm/work/1638441235.10898/Google-ProtocolBuffers-Dynamic-0.34/xsp/mapper.xsp":457:5: note: in expansion of macro ‘dXSTARG’
In file included from /usr/lib64/perl5/CORE/perl.h:5644,
from buildtmp/main.xs:11:
/usr/lib64/perl5/CORE/embed.h: In function ‘void XS_Google__ProtocolBuffers__Dynamic__FieldDef_default_value(PerlInterpreter*, CV*)’:
/usr/lib64/perl5/CORE/embed.h:596:39: warning: ‘RETVAL’ may be used uninitialized in this function [-Wmaybe-uninitialized]
#define sv_2mortal(a) Perl_sv_2mortal(aTHX a)
^
buildtmp/Dynamic.c:1416:7: note: ‘RETVAL’ was declared here
SV * RETVAL;
^~~~~~
ExtUtils::Mkbootstrap::Mkbootstrap('blib/arch/auto/Google/ProtocolBuffers/Dynamic/Dynamic.bs')
Writing blib/arch/auto/Google/ProtocolBuffers/Dynamic/Dynamic.bs.bs
gcc -lpthread -shared -Wl,-z,relro -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -L/usr/local/lib -fstack-protector-strong -lperl -o blib/arch/auto/Google/ProtocolBuffers/Dynamic/Dynamic.so buildtmp/Dynamic.o src/descriptorloader.o src/upb/bridge.o src/sourcetree.o src/dynamic.o src/mapper.o -lstdc++ -L/usr/local/share/perl5/auto/share/dist/Alien-uPB-Core/lib -lupb_descriptor -lupb_json -lupb_pb -lupb -lprotobuf -lpthread
t/001_load.t ........................... ok
t/010_basic.t .......................... ok
t/020_primitives.t ..................... ok
t/025_bigints.t ........................ ok
t/030_repeated.t ....................... ok
t/032_map.t ............................
No subtests run
t/033_proto2_map.t .....................
No subtests run
t/035_required.t ....................... ok
t/040_messages.t ....................... ok
t/042_recursion.t ...................... ok
t/045_enum.t ........................... ok
t/050_extensions.t .....................
No subtests run
t/055_oneof.t .......................... ok
t/057_proto3_optional.t ................ skipped: Protocol Buffers v3.12 required
t/060_tie.t ............................ ok
t/065_checked_new.t .................... ok
t/067_warnings.t .......................
No subtests run
t/070_order.t .......................... ok
t/080_json.t ...........................
No subtests run
t/082_wkt.t ............................
No subtests run
t/085_wkt_json.t .......................
No subtests run
t/100_type_mapping.t ................... ok
t/102_service_mapping.t ................
No subtests run
t/105_package_mapping.t ................ ok
t/107_package_prefix_mapping.t ......... ok
t/108_message_prefix_mapping.t .........
No subtests run
t/110_mixed_mapping.t .................. ok
t/115_simplified_mapping.t .............
No subtests run
t/140_parse_string.t ................... ok
t/145_parse_binary.t ................... ok
t/200_check_required.t ................. ok
t/205_explicit_defaults.t .............. ok
t/210_encode_defaults.t ................ ok
t/215_check_enum_values.t .............. ok
t/220_accessor_style.t .................
No subtests run
t/225_decode_unblessed.t ............... ok
t/235_fail_ref_coercion.t .............. ok
t/300_scalar_accessors.t ............... ok
t/305_oneof_accessors.t ................ ok
t/310_list_accessors.t ................. ok
t/315_message_accessors.t .............. ok
t/320_map_accessors.t ..................
No subtests run
t/330_field_numbers.t .................. ok
t/335_extension_api.t ..................
No subtests run
t/350_introspection.t .................. ok
t/352_proto3_optional_introspection.t .. skipped: Protocol Buffers v3.12 required
t/355_map_introspection.t ..............
No subtests run
t/357_service_introspection.t ..........
No subtests run
t/400_issue_12.t ....................... ok
t/401_descriptor_error.t ............... ok
t/605_basic_grpc.t ..................... skipped: Grpc::XS not available
t/610_stream_client_grpc.t ............. skipped: Grpc::XS not available
t/615_stream_server_grpc.t ............. skipped: Grpc::XS not available
t/620_stream_bidi_grpc.t ............... skipped: Grpc::XS not available
t/author-pod-syntax.t .................. skipped: these tests are for testing by the author

Test Summary Report

t/032_map.t (Wstat: 6 Tests: 0 Failed: 0)
Non-zero wait status: 6
Parse errors: No plan found in TAP output
t/033_proto2_map.t (Wstat: 6 Tests: 0 Failed: 0)
Non-zero wait status: 6
Parse errors: No plan found in TAP output
t/050_extensions.t (Wstat: 6 Tests: 0 Failed: 0)
Non-zero wait status: 6
Parse errors: No plan found in TAP output
t/067_warnings.t (Wstat: 6 Tests: 0 Failed: 0)
Non-zero wait status: 6
Parse errors: No plan found in TAP output
t/080_json.t (Wstat: 6 Tests: 0 Failed: 0)
Non-zero wait status: 6
Parse errors: No plan found in TAP output
t/082_wkt.t (Wstat: 6 Tests: 0 Failed: 0)
Non-zero wait status: 6
Parse errors: No plan found in TAP output
t/085_wkt_json.t (Wstat: 6 Tests: 0 Failed: 0)
Non-zero wait status: 6
Parse errors: No plan found in TAP output
t/102_service_mapping.t (Wstat: 6 Tests: 0 Failed: 0)
Non-zero wait status: 6
Parse errors: No plan found in TAP output
t/108_message_prefix_mapping.t (Wstat: 6 Tests: 0 Failed: 0)
Non-zero wait status: 6
Parse errors: No plan found in TAP output
t/115_simplified_mapping.t (Wstat: 6 Tests: 0 Failed: 0)
Non-zero wait status: 6
Parse errors: No plan found in TAP output
t/220_accessor_style.t (Wstat: 6 Tests: 0 Failed: 0)
Non-zero wait status: 6
Parse errors: No plan found in TAP output
t/320_map_accessors.t (Wstat: 6 Tests: 0 Failed: 0)
Non-zero wait status: 6
Parse errors: No plan found in TAP output
t/335_extension_api.t (Wstat: 6 Tests: 0 Failed: 0)
Non-zero wait status: 6
Parse errors: No plan found in TAP output
t/355_map_introspection.t (Wstat: 6 Tests: 0 Failed: 0)
Non-zero wait status: 6
Parse errors: No plan found in TAP output
t/357_service_introspection.t (Wstat: 6 Tests: 0 Failed: 0)
Non-zero wait status: 6
Parse errors: No plan found in TAP output
Files=55, Tests=555, 3 wallclock secs ( 0.12 usr 0.06 sys + 2.99 cusr 0.55 csys = 3.72 CPU)
Result: FAIL
Failed 15/55 test programs. 0/555 subtests failed.
-> FAIL Installing Google::ProtocolBuffers::Dynamic failed. See /root/.cpanm/work/1638441235.10898/build.log for details. Retry with --force to force install it.
[root@95382e5d8906 /]#

**[root@95382e5d8906 /]# yum install -y protobuf-compiler \

protobuf-c \
protobuf-devel** 

Updating Subscription Management repositories.
Last metadata expiration check: 0:45:07 ago on Thu Dec 2 09:55:34 2021.
Package protobuf-compiler-3.5.0-13.el8.x86_64 is already installed.
Package protobuf-c-1.3.0-6.el8.x86_64 is already installed.
Package protobuf-devel-3.5.0-13.el8.x86_64 is already installed.
Dependencies resolved.
Nothing to do.
Complete!
[root@95382e5d8906 /]#

A nice error message would be better

Unless I'm missing something, there doesn't appear to be a very convenient method to map a message that uses message types defined in other packages.

E.g., suppose you have these two proto files:

a.proto:

syntax = "proto3";
package a;

import "b.proto";

message A {
    b.B foo = 1;
}

b.proto:

syntax = "proto3";
package b;

message B {
    string bar = 1;
}

And you wanted to map a.A, you might go as follows:

$dynamic->load_file( "a.proto" );
$dynamic->map( { message => 'a.A', to => 'PB' } );

But this raises the following error (actually from resolve_references()):

Unknown type 'b.B'

It may not always be convenient to map b.B first. For example, in the case where you are writing a generic tool, you may not know that you have to. It would be significantly more convenient to be able to load a message and map anything it needs to a perl package prefix.

The order of vector cleanup in Mapper::DecoderHandlers::on_end_sub_message() can cause the seen_oneof vector to be pop()ed too many times.

According to CPAN, version 0.23 is released. But the last tagged git release is 0.22!