oskaritimperi / nimpb Goto Github PK
View Code? Open in Web Editor NEWProtocol Buffers for Nim
License: MIT License
Protocol Buffers for Nim
License: MIT License
text format is very useful for debugging, or for cross language, human readable,, type safe, configuration files:
https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.text_format
https://github.com/BVLC/caffe/blob/master/models/bvlc_reference_caffenet/solver.prototxt
let message = newTest1()
message.a = 150
## serialize
let data = protoToText(message)
echo data # a : 150
echo message # would use `$` as alias to serializeText
## deserialize
let message2 = Test1.deserializeText(data)
assert message2 == message
## serialize with option
let data = serializeText(message, oneline = false)
same as msoucy/dproto#71 (comment) : shell out to protoc eg:
## serializeText
serialize message to /tmp/z01.pb
protoc --decode=Test1 --proto_path=tests/t29_nimpb/ tests/t29_nimpb/test1.proto < /tmp/z01.pb > /tmp/z01.txt
read /tmp/z01.txt
## deserializeText
TODO
workaround is not great, as not as efficient, and requires passing the proto file; the proper solution should involve using reflection
with the following proto file:
syntax = "proto3";
option java_package = "io.deepstream.protobuf";
enum TOPIC {
UNKNOWN = 0;
PARSER = 1;
CONNECTION = 2;
AUTH = 3;
EVENT = 4;
RECORD = 5;
RPC = 6;
PRESENCE = 7;
MONITORING = 8;
CLUSTER = 9;
LOCK = 10;
STATE_REGISTRY = 11;
ERROR = 100;
}
message Message {
TOPIC topic = 2;
bytes message = 3;
}
The generated nim file has the following code:
type
TOPIC* {.pure.} = enum
UNKNOWN = 0
PARSER = 1
CONNECTION = 2
AUTH = 3
EVENT = 4
RECORD = 5
RPC = 6
PRESENCE = 7
MONITORING = 8
CLUSTER = 9
LOCK = 10
STATE_REGISTRY = 11
ERROR = 100
Message* = ref MessageObj
MessageObj* = object of Message
topic: TOPIC
message: seq[byte]
compile with error
Error: inheritance only works with non-final objects; to enable inheritance write 'Message of RootObj'
I guess it is conflicted with the nimpb's Message
git clone https://github.com/oswjk/nimpb
cd nimpb
nimble install
/Users/timothee/git_clone/nim/nimpb/nimpb/json.nim(100, 6) Error: overloaded 'parseInt' leads to ambiguous calls
WORKAROUND: commented out parseInt in nimpb/nimpb/json.nim EDIT: that doesn't work with nimble run_conformance_tests
Is it currently possible, and/or what would be required, to use your Nim implementation of protobuf with gRPC?
The enums generated by the generator are not pure. This creates a problem if a value of the enum is named the same as the name of the enum, except for casing. The simplest example is
syntax = "proto3";
package example;
enum Foo {
FOO = 0;
NOT_FOO = 1;
}
If you run the generator on this, you get the following nim file
# Generated by protoc_gen_nim. Do not edit!
import intsets
import protobuf/gen
import protobuf/stream
import protobuf/types
const
FooDesc = EnumDesc(
name: "Foo",
values: @[
EnumValueDesc(name: "FOO", number: 0),
EnumValueDesc(name: "NOT_FOO", number: 1),
]
)
generateEnumType(FooDesc)
generateEnumProcs(FooDesc)
This file will not compile because the name of the enum (Foo
) conflicts with one of the values of the enum itself (FOO
). It would be nice to have the possibility to generate pure enums, so that the value FOO
would need to be scoped as Foo.FOO
and no conflict would arise
see also: PMunch/protobuf-nim#1 in which I list other related nim packages
just wondering whether it would make sense to consolidate efforts with @PMunch or whether these have different goals/designs.
In the nim protobuf package there is a writeSize parameter, so we can have the message length indicator in the packet,
proc write(s: Stream; o: ds_Challenge; writeSize = false) =
if writeSize:
s.protoWriteInt64(o.len)
...
But I didn't find this parameter in the nimpb, the protobuf server I'm now communicating with need this length prefix in the packet, so it is good to have.
Nim compiler version: 1.4.2
Build log:
$ nimble install nimpb
Downloading https://github.com/oswjk/nimpb using git
Verifying dependencies for [email protected]
Info: Dependency on nimpb_protoc@any version already satisfied
Verifying dependencies for [email protected]
Installing [email protected]
Building nimpb/nimpb/compiler/nimpb_build using c backend
/tmp/nimble_14769/githubcom_oswjknimpb/nimpb/compiler/nimpb_build.nim(3, 8) Warning: Use the 'parseopt' module instead; parseopt2 is deprecated [Deprecated]
/tmp/nimble_14769/githubcom_oswjknimpb/nimpb/nimpb.nim(21, 19) Warning: inherit from a more precise exception type like ValueError, IOError or OSError. If these don't suit, inherit from CatchableError or Defect. [InheritFromException]
/tmp/nimble_14769/githubcom_oswjknimpb/nimpb/nimpb.nim(539, 72) Warning: `typed` will change its meaning in future versions of Nim. `void` or no return type declaration at all has the same meaning as the current meaning of `typed` as return type declaration. [Deprecated]
/tmp/nimble_14769/githubcom_oswjknimpb/nimpb/wkt/descriptor_pb.nim(3, 8) Warning: imported and not used: 'base64' [UnusedImport]
/tmp/nimble_14769/githubcom_oswjknimpb/nimpb/wkt/descriptor_pb.nim(4, 8) Warning: imported and not used: 'intsets' [UnusedImport]
/tmp/nimble_14769/githubcom_oswjknimpb/nimpb/wkt/descriptor_pb.nim(5, 8) Warning: imported and not used: 'json' [UnusedImport]
/tmp/nimble_14769/githubcom_oswjknimpb/nimpb/wkt/descriptor_pb.nim(6, 8) Warning: imported and not used: 'strutils' [UnusedImport]
/tmp/nimble_14769/githubcom_oswjknimpb/nimpb/wkt/descriptor_pb.nim(9, 19) Warning: imported and not used: 'json' [UnusedImport]
/tmp/nimble_14769/githubcom_oswjknimpb/nimpb/compiler/generator.nim(585, 32) Warning: Deprecated since v0.20, use 'toHashSet'; toSet is deprecated [Deprecated]
/tmp/nimble_14769/githubcom_oswjknimpb/nimpb/compiler/generator.nim(630, 12) Error: undeclared identifier: 'isNilOrWhitespace'
Tip: 6 messages have been suppressed, use --verbose to show them.
Error: Build failed for package: nimpb
... Execution failed with exit code 1
... Command: /usr/bin/nim c --colors:on --noNimblePath -d:release -d:NimblePkgVersion=0.2.0 --path:/home/victor/.nimble/pkgs/nimpb_protoc-0.1.0 --hints:off -o:/tmp/nimble_14769/githubcom_oswjknimpb/nimpb/compiler/nimpb_build /tmp/nimble_14769/githubcom_oswjknimpb/nimpb/compiler/nimpb_build.nim
It looks like isNilOrWhitespace
procedure was removed in Nim version 1.4.0
https://github.com/nim-lang/Nim/blob/devel/changelogs/changelog_1_4_0.md#language-changes
Perhaps you should use isEmptyOrWhitespace
(https://nim-lang.org/docs/strutils.html#isEmptyOrWhitespace%2Cstring)
on OSX it's just brew install protobuf
; I'd like to just be able to use that binary (eg maybe in nimcfg or initialization code or something, via protoc_binary=$homebrew_D/bin/protoc
)
eg, useful in case user has some custom build of protoc, or doesn't want to recompile from scratch (time consuming: 480 seconds) etc
with the proto spec:
type
TOPIC* {.pure.} = enum
UNKNOWN = 0
PARSER = 1
CONNECTION = 2
AUTH = 3
EVENT = 4
RECORD = 5
RPC = 6
PRESENCE = 7
MONITORING = 8
CLUSTER = 9
LOCK = 10
STATE_REGISTRY = 11
ERROR = 100
DPSMessage* = ref DPSMessageObj
DPSMessageObj* = object of Message
topic: TOPIC
message: seq[byte]
The compiler generate the following errors:
/Users/geohuz/Coding/nimlearn/deepstream.io-protobuf/proto_files/general_pb.nim(120, 33) template/generic instantiation of `toJson` from here
/Users/geohuz/.nimble/pkgs/nimpb-0.2.0/nimpb/json.nim(42, 18) template/generic instantiation of `items` from here
/Users/geohuz/.choosenim/toolchains/nim-1.4.8/lib/system/iterators.nim(87, 19) template/generic instantiation of `..` from here
/Users/geohuz/.choosenim/toolchains/nim-1.4.8/lib/system/iterators_1.nim(85, 12) Error: type mismatch: got <TOPIC>
but expected one of:
proc inc[T: Ordinal](x: var T; y = 1)
first type mismatch at position: 1
required type for x: var T: Ordinal
but expression 'res' is of type: TOPIC
expression: inc(res)
on API side, I much prefer the way it's done in dproto ; that one is in D but IMO should be doable in nim too:
instead of:
let message = newTest1()
setA(message, 150)
let pbso = newProtobufStream(newFileStream(stdout))
writeTest1(pbso, message)
we could have:
let message = newTest1()
message.a = 150
assert message.a == 150
auto serializedObject = message.serialize();
auto message2 = newTest1(serializedObject);
assert message2 == message
and more generally for more complex example, see example in their readme
eg:
nimpb/compiler/nimpb_build
=>
this could go under ./build/bin/nimpb_build
protobuf-all-3.5.1.zip and protobuf-3.5.1 could go under ./build/ ...
.gitignore could add ./build/
so that git status
can stay clean (eg after running nimble build_protobuf
git status is not clean)
@oswjk did it ever work?
before #10 I ran into #4;
after #10 I get:
ERROR, test=Required.Proto3.ProtobufInput.RepeatedScalarSelectsLast.DOUBLE.JsonOutput: Output was not equivalent to reference message: modified: optional_double: 2.2250738585072014e-308 -> 2.2250738585072009e-308
. request=protobuf_payload: "a\232\231\231\231\231\231\271?a\377\377\377\377\377\377\357\177a\000\000\000\000\000\000\020\000" requested_output_format: JSON message_type: "protobuf_test_messages.proto3.TestAllTypesProto3", response=json_payload: "{\"optionalDouble\":2.225073858507201e-308}"
ERROR, test=Required.Proto3.ProtobufInput.ValidDataRepeated.DOUBLE.JsonOutput: JSON output we received from test was unparseable. request=protobuf_payload: "\321\002\232\231\231\231\231\231\271?\321\002\377\377\377\377\377\377\357\177\321\002\000\000\000\000\000\000\020\000" requested_output_format: JSON message_type: "protobuf_test_messages.proto3.TestAllTypesProto3", response=json_payload: "{\"repeatedDouble\":[0.1,1.797693134862316e+308,2.225073858507201e-308]}"
WARNING, test=Recommended.Proto3.JsonInput.TrailingCommaInAnObject: Should have failed to parse, but didn't. request=json_payload: "{\"fieldname1\":1,}" requested_output_format: JSON message_type: "protobuf_test_messages.proto3.TestAllTypesProto3", response=json_payload: "{\"fieldname1\":1}"
WARNING, test=Recommended.Proto3.JsonInput.TrailingCommaInAnObjectWithSpace: Should have failed to parse, but didn't. request=json_payload: "{\"fieldname1\":1 ,}" requested_output_format: JSON message_type: "protobuf_test_messages.proto3.TestAllTypesProto3", response=json_payload: "{\"fieldname1\":1}"
WARNING, test=Recommended.Proto3.JsonInput.TrailingCommaInAnObjectWithSpaceCommaSpace: Should have failed to parse, but didn't. request=json_payload: "{\"fieldname1\":1 , }" requested_output_format: JSON message_type: "protobuf_test_messages.proto3.TestAllTypesProto3", response=json_payload: "{\"fieldname1\":1}"
WARNING, test=Recommended.Proto3.JsonInput.TrailingCommaInAnObjectWithNewlines: Should have failed to parse, but didn't. request=json_payload: "{\n \"fieldname1\":1,\n }" requested_output_format: JSON message_type: "protobuf_test_messages.proto3.TestAllTypesProto3", response=json_payload: "{\"fieldname1\":1}"
WARNING, test=Recommended.Proto3.JsonInput.JsonWithComments: Should have failed to parse, but didn't. request=json_payload: "{\n // This is a comment.\n \"fieldname1\": 1\n }" requested_output_format: JSON message_type: "protobuf_test_messages.proto3.TestAllTypesProto3", response=json_payload: "{\"fieldname1\":1}"
...
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.