oskaritimperi / nimpb Goto Github PK

View Code? Open in Web Editor NEW

37.0 37.0 6.0 335 KB

Protocol Buffers for Nim

License: MIT License

Nim 100.00%

library nim nimpb protobuf protocol-buffers serialization

nimpb's People

Contributors

Stargazers

Watchers

Forkers

timotheecour zacharycarter tmokazaki dchem drygdryg burnysc2

nimpb's Issues

text format serialization and deserialization

text format is very useful for debugging, or for cross language, human readable,, type safe, configuration files:

https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.text_format

example of text format

https://github.com/BVLC/caffe/blob/master/models/bvlc_reference_caffenet/solver.prototxt

usage

let message = newTest1()
message.a = 150

## serialize
let data = protoToText(message)
echo data # a : 150
echo message # would use `$` as alias to serializeText

## deserialize
let message2 = Test1.deserializeText(data)
assert message2 == message

## serialize with option
let data = serializeText(message, oneline = false)

WORKAROUND

same as msoucy/dproto#71 (comment) : shell out to protoc eg:

## serializeText
serialize message to /tmp/z01.pb
protoc --decode=Test1 --proto_path=tests/t29_nimpb/ tests/t29_nimpb/test1.proto < /tmp/z01.pb > /tmp/z01.txt
read /tmp/z01.txt

## deserializeText
TODO

workaround is not great, as not as efficient, and requires passing the proto file; the proper solution should involve using reflection

Compiling error with generated nim file

with the following proto file:

syntax = "proto3";

option java_package = "io.deepstream.protobuf";

enum TOPIC {
  UNKNOWN = 0;
  PARSER = 1;
  CONNECTION = 2;
  AUTH = 3;
  EVENT = 4;
  RECORD = 5;
  RPC = 6;
  PRESENCE = 7;
  MONITORING = 8;
  CLUSTER = 9;
  LOCK = 10;
  STATE_REGISTRY = 11;
  ERROR = 100;
}

message Message {
  TOPIC topic = 2;
  bytes message = 3;
}

The generated nim file has the following code:

type
    TOPIC* {.pure.} = enum
        UNKNOWN = 0
        PARSER = 1
        CONNECTION = 2
        AUTH = 3
        EVENT = 4
        RECORD = 5
        RPC = 6
        PRESENCE = 7
        MONITORING = 8
        CLUSTER = 9
        LOCK = 10
        STATE_REGISTRY = 11
        ERROR = 100
    Message* = ref MessageObj
    MessageObj* = object of Message
        topic: TOPIC
        message: seq[byte]

compile with error

Error: inheritance only works with non-final objects; to enable inheritance write 'Message of RootObj'

I guess it is conflicted with the nimpb's Message

Error: overloaded 'parseInt' leads to ambiguous calls

git clone https://github.com/oswjk/nimpb
cd nimpb
nimble install
/Users/timothee/git_clone/nim/nimpb/nimpb/json.nim(100, 6) Error: overloaded 'parseInt' leads to ambiguous calls

WORKAROUND: commented out parseInt in nimpb/nimpb/json.nim EDIT: that doesn't work with nimble run_conformance_tests

Could this be used for gRPC?

Is it currently possible, and/or what would be required, to use your Nim implementation of protobuf with gRPC?

Generate pure enums

The enums generated by the generator are not pure. This creates a problem if a value of the enum is named the same as the name of the enum, except for casing. The simplest example is

syntax = "proto3";

package example;

enum Foo {
  FOO = 0;
  NOT_FOO = 1;
}

If you run the generator on this, you get the following nim file

# Generated by protoc_gen_nim. Do not edit!

import intsets

import protobuf/gen
import protobuf/stream
import protobuf/types

const
    FooDesc = EnumDesc(
        name: "Foo",
        values: @[
            EnumValueDesc(name: "FOO", number: 0),
            EnumValueDesc(name: "NOT_FOO", number: 1),
        ]
    )
generateEnumType(FooDesc)
generateEnumProcs(FooDesc)

This file will not compile because the name of the enum (Foo) conflicts with one of the values of the enum itself (FOO). It would be nice to have the possibility to generate pure enums, so that the value FOO would need to be scoped as Foo.FOO and no conflict would arise

questions + related packages

see also: PMunch/protobuf-nim#1 in which I list other related nim packages

just wondering whether it would make sense to consolidate efforts with @PMunch or whether these have different goals/designs.

Need an option to specify if writing with length prefix into protobuf message

In the nim protobuf package there is a writeSize parameter, so we can have the message length indicator in the packet,

proc write(s: Stream; o: ds_Challenge; writeSize = false) =
  if writeSize:
    s.protoWriteInt64(o.len)
  ...

But I didn't find this parameter in the nimpb, the protobuf server I'm now communicating with need this length prefix in the packet, so it is good to have.

Compilation failed with Nim 1.4.2

Nim compiler version: 1.4.2
Build log:

$ nimble install nimpb
Downloading https://github.com/oswjk/nimpb using git
  Verifying dependencies for [email protected]
      Info: Dependency on nimpb_protoc@any version already satisfied
  Verifying dependencies for [email protected]
 Installing [email protected]
   Building nimpb/nimpb/compiler/nimpb_build using c backend
/tmp/nimble_14769/githubcom_oswjknimpb/nimpb/compiler/nimpb_build.nim(3, 8) Warning: Use the 'parseopt' module instead; parseopt2 is deprecated [Deprecated]
/tmp/nimble_14769/githubcom_oswjknimpb/nimpb/nimpb.nim(21, 19) Warning: inherit from a more precise exception type like ValueError, IOError or OSError. If these don't suit, inherit from CatchableError or Defect. [InheritFromException]
/tmp/nimble_14769/githubcom_oswjknimpb/nimpb/nimpb.nim(539, 72) Warning: `typed` will change its meaning in future versions of Nim. `void` or no return type declaration at all has the same meaning as the current meaning of `typed` as return type declaration. [Deprecated]
/tmp/nimble_14769/githubcom_oswjknimpb/nimpb/wkt/descriptor_pb.nim(3, 8) Warning: imported and not used: 'base64' [UnusedImport]
/tmp/nimble_14769/githubcom_oswjknimpb/nimpb/wkt/descriptor_pb.nim(4, 8) Warning: imported and not used: 'intsets' [UnusedImport]
/tmp/nimble_14769/githubcom_oswjknimpb/nimpb/wkt/descriptor_pb.nim(5, 8) Warning: imported and not used: 'json' [UnusedImport]
/tmp/nimble_14769/githubcom_oswjknimpb/nimpb/wkt/descriptor_pb.nim(6, 8) Warning: imported and not used: 'strutils' [UnusedImport]
/tmp/nimble_14769/githubcom_oswjknimpb/nimpb/wkt/descriptor_pb.nim(9, 19) Warning: imported and not used: 'json' [UnusedImport]
/tmp/nimble_14769/githubcom_oswjknimpb/nimpb/compiler/generator.nim(585, 32) Warning: Deprecated since v0.20, use 'toHashSet'; toSet is deprecated [Deprecated]
/tmp/nimble_14769/githubcom_oswjknimpb/nimpb/compiler/generator.nim(630, 12) Error: undeclared identifier: 'isNilOrWhitespace'
       Tip: 6 messages have been suppressed, use --verbose to show them.
     Error: Build failed for package: nimpb
        ... Execution failed with exit code 1
        ... Command: /usr/bin/nim c --colors:on --noNimblePath -d:release -d:NimblePkgVersion=0.2.0 --path:/home/victor/.nimble/pkgs/nimpb_protoc-0.1.0 --hints:off -o:/tmp/nimble_14769/githubcom_oswjknimpb/nimpb/compiler/nimpb_build /tmp/nimble_14769/githubcom_oswjknimpb/nimpb/compiler/nimpb_build.nim

It looks like isNilOrWhitespace procedure was removed in Nim version 1.4.0
https://github.com/nim-lang/Nim/blob/devel/changelogs/changelog_1_4_0.md#language-changes
Perhaps you should use isEmptyOrWhitespace (https://nim-lang.org/docs/strutils.html#isEmptyOrWhitespace%2Cstring)

allow specifying `protoc` binary instead of forcing to build one from scratch

on OSX it's just brew install protobuf ; I'd like to just be able to use that binary (eg maybe in nimcfg or initialization code or something, via protoc_binary=$homebrew_D/bin/protoc)

eg, useful in case user has some custom build of protoc, or doesn't want to recompile from scratch (time consuming: 480 seconds) etc

enum error

with the proto spec:

type
    TOPIC* {.pure.} = enum
        UNKNOWN = 0
        PARSER = 1
        CONNECTION = 2
        AUTH = 3
        EVENT = 4
        RECORD = 5
        RPC = 6
        PRESENCE = 7
        MONITORING = 8
        CLUSTER = 9
        LOCK = 10
        STATE_REGISTRY = 11
        ERROR = 100
    DPSMessage* = ref DPSMessageObj
    DPSMessageObj* = object of Message
        topic: TOPIC
        message: seq[byte]

The compiler generate the following errors:

/Users/geohuz/Coding/nimlearn/deepstream.io-protobuf/proto_files/general_pb.nim(120, 33) template/generic instantiation of `toJson` from here
/Users/geohuz/.nimble/pkgs/nimpb-0.2.0/nimpb/json.nim(42, 18) template/generic instantiation of `items` from here
/Users/geohuz/.choosenim/toolchains/nim-1.4.8/lib/system/iterators.nim(87, 19) template/generic instantiation of `..` from here
/Users/geohuz/.choosenim/toolchains/nim-1.4.8/lib/system/iterators_1.nim(85, 12) Error: type mismatch: got <TOPIC>
but expected one of: 
proc inc[T: Ordinal](x: var T; y = 1)
  first type mismatch at position: 1
  required type for x: var T: Ordinal
  but expression 'res' is of type: TOPIC

expression: inc(res)

API design discussion

on API side, I much prefer the way it's done in dproto ; that one is in D but IMO should be doable in nim too:

instead of:

let message = newTest1()
setA(message, 150)
let pbso = newProtobufStream(newFileStream(stdout))
writeTest1(pbso, message)

we could have:

let message = newTest1()
message.a = 150
assert message.a == 150
auto serializedObject = message.serialize();
auto message2 = newTest1(serializedObject);
assert message2 == message

and more generally for more complex example, see example in their readme

generated files (eg nimpb/compiler/nimpb_build) should go in a subfolder

eg:
nimpb/compiler/nimpb_build
=>
this could go under ./build/bin/nimpb_build

protobuf-all-3.5.1.zip and protobuf-3.5.1 could go under ./build/ ...

.gitignore could add ./build/ so that git status can stay clean (eg after running nimble build_protobuf git status is not clean)

nimble run_conformance_tests: CONFORMANCE SUITE FAILED: 9 unexpected failures

@oswjk did it ever work?
before #10 I ran into #4;
after #10 I get:

ERROR, test=Required.Proto3.ProtobufInput.RepeatedScalarSelectsLast.DOUBLE.JsonOutput: Output was not equivalent to reference message: modified: optional_double: 2.2250738585072014e-308 -> 2.2250738585072009e-308
. request=protobuf_payload: "a\232\231\231\231\231\231\271?a\377\377\377\377\377\377\357\177a\000\000\000\000\000\000\020\000" requested_output_format: JSON message_type: "protobuf_test_messages.proto3.TestAllTypesProto3", response=json_payload: "{\"optionalDouble\":2.225073858507201e-308}"
ERROR, test=Required.Proto3.ProtobufInput.ValidDataRepeated.DOUBLE.JsonOutput: JSON output we received from test was unparseable. request=protobuf_payload: "\321\002\232\231\231\231\231\231\271?\321\002\377\377\377\377\377\377\357\177\321\002\000\000\000\000\000\000\020\000" requested_output_format: JSON message_type: "protobuf_test_messages.proto3.TestAllTypesProto3", response=json_payload: "{\"repeatedDouble\":[0.1,1.797693134862316e+308,2.225073858507201e-308]}"
WARNING, test=Recommended.Proto3.JsonInput.TrailingCommaInAnObject: Should have failed to parse, but didn't. request=json_payload: "{\"fieldname1\":1,}" requested_output_format: JSON message_type: "protobuf_test_messages.proto3.TestAllTypesProto3", response=json_payload: "{\"fieldname1\":1}"
WARNING, test=Recommended.Proto3.JsonInput.TrailingCommaInAnObjectWithSpace: Should have failed to parse, but didn't. request=json_payload: "{\"fieldname1\":1 ,}" requested_output_format: JSON message_type: "protobuf_test_messages.proto3.TestAllTypesProto3", response=json_payload: "{\"fieldname1\":1}"
WARNING, test=Recommended.Proto3.JsonInput.TrailingCommaInAnObjectWithSpaceCommaSpace: Should have failed to parse, but didn't. request=json_payload: "{\"fieldname1\":1 , }" requested_output_format: JSON message_type: "protobuf_test_messages.proto3.TestAllTypesProto3", response=json_payload: "{\"fieldname1\":1}"
WARNING, test=Recommended.Proto3.JsonInput.TrailingCommaInAnObjectWithNewlines: Should have failed to parse, but didn't. request=json_payload: "{\n        \"fieldname1\":1,\n      }" requested_output_format: JSON message_type: "protobuf_test_messages.proto3.TestAllTypesProto3", response=json_payload: "{\"fieldname1\":1}"
WARNING, test=Recommended.Proto3.JsonInput.JsonWithComments: Should have failed to parse, but didn't. request=json_payload: "{\n        // This is a comment.\n        \"fieldname1\": 1\n      }" requested_output_format: JSON message_type: "protobuf_test_messages.proto3.TestAllTypesProto3", response=json_payload: "{\"fieldname1\":1}"

...