There is currently no check on the validity of the values (as in, varint decoded is bigger than actually possible to fit in the struct)

Need to do this for any kind of varint.

Impact on decode_varint (if it doesnt fit, return error) and on the transformation done to fit the final structure in pb_decode

Taking this proto file as an example:

https://github.com/lightningnetwork/lnd/blob/master/lnrpc/invoicesrpc/invoices.proto#L29

The rpc services functions are not getting generated. Only the types are. Is it possible to also add this?

./build.zig:24:13: error: no member named 'path' in struct 'std.build.Pkg'
.path = std.build.FileSource{.path = "src/protobuf.zig"}

https://github.com/protocolbuffers/protobuf/blob/main/conformance/README.md

Parent field has a field_type "OneOf"

Make tagged unions, tagged union has a _union_desc_table too
Generate a function that returns the active tag's value + field descriptor

It is not running on either version 0.11.0 or the current zig master version (0.12.0-dev.2341+92211135f)

Trying to follow the instructions in the README, but I zig seems unable to find the module.

I have tried with both zig v0.11.0 and 0.12.0-dev.2587+a1b607acb

❯ zig build --help
thread 3057254 panic: unable to find module 'protobuf'
/Users/stephen/.local/bin/zig/lib/std/debug.zig:434:22: 0x10013fc5b in panicExtra__anon_18444 (build)
    std.builtin.panic(msg, trace, ret_addr);
                     ^
/Users/stephen/.local/bin/zig/lib/std/debug.zig:409:15: 0x10010cb47 in panic__anon_17783 (build)
    panicExtra(null, null, format, args);
              ^
/Users/stephen/.local/bin/zig/lib/std/Build.zig:1786:18: 0x1000dae4f in module (build)
            panic("unable to find module '{s}'", .{name});
                 ^
/Users/stephen/code/learn/zig/build.zig:54:46: 0x10009bc87 in build (build)
                .module = protobuf_dep.module("protobuf"),
                                             ^
/Users/stephen/.local/bin/zig/lib/std/Build.zig:1982:33: 0x10007ce37 in runBuild__anon_8404 (build)
        .Void => build_zig.build(b),
                                ^
/Users/stephen/.local/bin/zig/lib/build_runner.zig:310:29: 0x100078b3b in main (build)
        try builder.runBuild(root);
                            ^
/Users/stephen/.local/bin/zig/lib/std/start.zig:585:37: 0x10007e47b in main (build)
            const result = root.main() catch |err| {
                                    ^
???:?:?: 0x18c0e50df in ??? (???)
???:?:?: 0x56307fffffffffff in ??? (???)
error: the following build command crashed:
/Users/stephen/code/learn/zig/zig-cache/o/fca68244e4a64f1c935cc61c53853d6b/build /Users/stephen/.local/bin/zig/zig /Users/stephen/code/learn/zig /Users/stephen/code/learn/zig/zig-cache /Users/stephen/.cache/zig --seed 0x5b7babad -Z132fe72e70486721 --help

in append_tag, the encoded tag value can be calculated at compile-time since the wire value and tag value are both known at compile time.

Zig 0.12.0 has been released. Time to upgrade.

Already saved the output, only need to work on everything important :)

Will look into this more tomorrow, but wanted to share quickly in case it's obvious:

thread 1951688 panic: integer cast truncated bits
/home/tj/.cache/zig/p/1220d226686f6023fae80156c729382295c672d6ffaa746d2932d73f16bd9affc758/src/protobuf.zig:634:77: 0x2dc131 in decode_varint__anon_12288 (a)
        value += (@as(T, input[index] & 0x7F)) << (@as(std.math.Log2Int(T), @intCast(shift)));
                                                                            ^
/home/tj/.cache/zig/p/1220d226686f6023fae80156c729382295c672d6ffaa746d2932d73f16bd9affc758/src/protobuf.zig:729:51: 0x2ca405 in next (a)
            const tag_and_wire = try decode_varint(u32, state.input[state.current_index..]);

Debug prints:

std.math.Log2Int(T): u5 | shift: 0
std.math.Log2Int(T): u5 | shift: 7
std.math.Log2Int(T): u5 | shift: 14
std.math.Log2Int(T): u5 | shift: 21
std.math.Log2Int(T): u5 | shift: 28
std.math.Log2Int(T): u5 | shift: 35 <---- crashes here for obvious reasons

My proto is essentially this:

message m {
  optional uint32 a = 1;
  optional float b = 2;
  message C {
    optional string d = 1;
  }
  optional C c = 3;
}

And I'm sending the following over the wire:

const data = try client_server.encode(allocator);
// data: [_]u8{ 8, 240, 1, 21, 0, 112, 155, 69 }

which looks like it should decode correctly to:

Hopefully I'm not doing something silly. Thanks for the library!

Take this file as an example:

syntax = "proto3";

package test_demo;
message Foo {
  string name = 1;
  repeated string loved = 2;
  int32 birth = 3;
}

protoc --zig_out=. foo.proto

This will output

// Code generated by protoc-gen-zig
 ///! package test_demo
const std = @import("std");
const Allocator = std.mem.Allocator;
const ArrayList = std.ArrayList;

const protobuf = @import("protobuf");
const ManagedString = protobuf.ManagedString;
const fd = protobuf.fd;

pub const Foo = struct {
    name: ManagedString = .Empty,
    loved: ArrayList(ManagedString),
    birth: i32 = 0,

    pub const _desc_table = .{
        .name = fd(1, .String),
        .loved = fd(2, .{ .List = .String}),
        .birth = fd(3, .{ .Varint = .Simple }),
    };

    pub usingnamespace protobuf.MessageMixins(@This());
};

It seems every struct contains a MessageMixins to provide decode/encode methods.

Since those methods are generics over T, We can do just let users do this with protobuf.pb_encode(struct, allocator), this may be beneficial for compile time?

Happy to hear what your thoughts, thanks.

Can use a method check_validity(self: T) InvalidErrors!void { ... } to ensure the message adheres to restrictions and options before encoding and after decoding.

Can be a first step for required fields on proto2

Complex map keys/value deinit are not an easy feat to manage. If a key/value type requires a deinit call to be properly freed, it is currently not managed in any way.

The official python library for protobuf from google has a json format, and it is found in the conformance tests proto definition so this could tie with #27

Considering the zig library has a json parser, it shouldn't be too hard to do.

currently having fun with pointers, but @ bitCast could probably a better option.

Hello, sabe problem here with zig 0.12.0-dev.3161+377ecc6af. It can't find protobuf module and also complains that the project does not have a build.zig.zon file so zig fetch --save does not work.

Originally posted by @davo417 in #37 (comment)

At a low level, the protobuf encoding allows you to decode one field at a time. But usually it is more convenient to have an API where you decode one message at a time.

To do this, most protobuf libraries will require the entire encoded message to be in contiguous memory before decoding it. This is fine in most cases, but sometimes you want to be able to decode messages incrementally. This saves on copying out to intermediate buffers.

Example use case in an embedded system, or a kernel driver:
Data is received into a ringbuffer or FIFO from some hardware peripheral. This buffer could be smaller than the size of an encoded protobuf message, and depending on the head/tail position, some messages may also wrap around from the end to the start of the buffer. Normally it would need to be defragmented/copied to an intermediate buffer before decoding, which takes more memory, or you need to resort to manually parsing one field at a time. In any case some manual logic / state machine is required.

Proposal: a "throw bytes at this function" API, where the library calls you back every time it has fully decoded a message.
The decoding state machine is generated by the library.
Example usage code:

// Only need to call this once. Possibly comptime.
var decode_state = protobuf.pb_decoder_init(.{ 
    .message_type = MyMessageType,
    .allocator = allocator,
    .delimiter_parser = protobuf.VarIntDelimiterParser, // Library provided implementation, or could be a custom implementation
    .decode_callback = message_decoded_callback,
});

// Somewhere else in the codebase: we receive some bytes.
// This is a slice with any number of bytes, i.e. half a message, or three entire messages.
// message_decoded_callback is called whenever a MyMessageType is fully decoded,
// across any number of calls to pb_decode_bytes.
// This function should only use a buffer large enough to decode one field.
// It only calls the allocator when decoding dynamic data like strings or repeated fields.
try protobuf.pb_decode_bytes(&decode_state, bytes_from_somewhere());

fn message_decoded_callback(decoded: *const MyMessageType) void {
    defer decoded.deinit();
    // do stuff with decoded message 
}

I am not familiar enough with zig-protobuf internals to know how hard this to implement, but I imagine it would be a good chunk of code. From an API design perspective the hard part is going to be the delimiter parser, which would be some interface to parse the delimiters/headers. Ideally it would be flexible enough that you can implement things like checksums/CRC with it as well.

A streaming encoder would be nice to have as well, but it is not as important in my opinion.

I ended up typing out quite the story here, but it is just some draft/idea, I understand if it is outside of the scope of this project.

Take inspiration from nanopb's python generator

Aim for varint and non-plugin way first.

There is a text format too that we should be able to handle somehow.

https://protobuf.dev/reference/protobuf/textformat-spec/

As of today, the library is leaking memory for user-allocated strings.

Tests are not failing because static/const strings are not accounted as leaks.

This could be fixed by recursively deallocating strings in the .deinit funciton but two problems may arise:

1. It should be extremely clear for the user that the ownership of string memory is now passed onto the proto message. Plus, the allocator for the string MUST be exactly the same as the one used to deinitialize the message.
2. Deallocating const strings may result in runtime exceptions

Proposal:

Replace the plain []const u8 by ManagedString. Keep the ?ManagedString for optional semantics. ManagedString is an union with OwnedString and ConstString, the former is automatically deallocated, the later won't.

const ManagedStringKeys = enum {
  OwnedString,
  ConstString
};

const ManagedString = union(ManagedStringKeys) {
  // memory that needs to be allocated
  OwnedString: []const u8,
  // memory that is static/const
  ConstString: []const u8,
};

I'm currently unable to find a solution for the OneOf field. Just axe it for a 0.9.0 release and make everything work a bit.

Maybe i should just leave the idea of comptime optimisation for OneOf and have a simpler process?

Everything else works.