There seems to be big spacers around the Kotlin section for some reason. This is using Google Chrome Version 122.0.6261.137 (Official Build) (64-bit) on a Chromebook.

Docs here:

You wrote "signed int32 varint". What this means? You mean zigzag encoding?! I see two variants:

• This is zigzag encoded signed varint. But then number "7" should be encoded as 0x0e, not as 0x07. But you write 0x07
• This is normal varint without zigzag. But then why you write "signed int32 varint"? This is confusing

I. e. we see self-contradiction here.

This is second bug I found today. (First one is #61.) Why your docs have such low quality?

The web site exposes an RSS feed as link rel=alternate, at https://protobuf.dev/news/index.xml but that feed is empty instead of containing the latest news and/or content updates.

Q1: How to build the documentation?

Hello, Protobuf Team.

Recently, I tried to build the Protobuf's Documentation from this repo's source on Windows. However, after I followed the commands provided by .github/workflows/gh-pages.yml, it still failed.

First of all, I prepared Npm dependencies and Hugo executable:

C:\Users\hwhsu1231>where npm
C:\Program Files\nodejs\npm
C:\Program Files\nodejs\npm.cmd

C:\Users\hwhsu1231>npm install autoprefixer postcss postcss-cli

added 74 packages in 1m

19 packages are looking for funding
  run `npm fund` for details
npm notice
npm notice New minor version of npm available! 9.6.7 -> 9.8.1
npm notice Changelog: https://github.com/npm/cli/releases/tag/v9.8.1
npm notice Run npm install -g npm@9.8.1 to update!
npm notice

C:\Users\hwhsu1231>npm install -g npm@9.8.1

added 1 package in 10s

28 packages are looking for funding
  run `npm fund` for details

C:\Users\hwhsu1231>where hugo && hugo version
C:\Hugo\bin\hugo.exe
hugo v0.115.4-dc9524521270f81d1c038ebbb200f0cfa3427cc5+extended windows/amd64 BuildDate=2023-07-20T06:49:57Z VendorInfo=gohugoio

Then, git clone the repo and run hugo --minify:

D:\Repo\tmp>git clone https://github.com/protocolbuffers/protocolbuffers.github.io.git protobuf-docs
Cloning into 'protobuf-docs'...
remote: Enumerating objects: 9670, done.
remote: Counting objects: 100% (1797/1797), done.
remote: Compressing objects: 100% (257/257), done.
remote: Total 9670 (delta 958), reused 1744 (delta 940), pack-reused 7873
Receiving objects: 100% (9670/9670), 12.21 MiB | 3.60 MiB/s, done.
Resolving deltas: 100% (5301/5301), done.

D:\Repo\tmp>cd protobuf-docs

D:\Repo\tmp\protobuf-docs>hugo --minify
Total in 23 ms
Error: failed to load modules: failed to download modules: binary with name "go" not found

What happened? What do I miss?

Q2: How is the documentation rendered?

BTW, I also checked out to the gh-pages branch to see what the artifact looks like. However, it seems different than what https://protobuf.dev/ displays.

D:\Repo\tmp\protobuf-docs>git checkout gh-pages
Switched to a new branch 'gh-pages'
branch 'gh-pages' set up to track 'origin/gh-pages'.

If I want to see the rendered documentation locally, what should I do?

import typia, { tags } from "typia";

interface IMember {
    id: string & tags.Format<"uuid">;
    email: string & tags.Format<"email">;
    age: number 
        & tags.Type<"uint32"> 
        & tags.Minimum<20> 
        & tags.ExclusiveMaximum<100>;
    parent: IMember | null;
    children: IMember[];
}

typia.protobuf.createEncode<IMember>();

https://typia.io/playground/?script=JYWwDg9gTgLgBDAnmYBDANHA3g1BzAZzgF84AzKCEOAIiRVRoG4AoF4AOxgFMozUAxtzgBJALLcQAI17YWcBXGAATAFxwCMKJzxwAZLkIA6AGLQQqGAB4aAV1sqaAPlaK4k1MAA26zdo66BjD4BKbmljYe3s6uivjc6hy20rLybopBIUYAKsjcNg5cAMwATM5waen6hqFinKDJViUADE4VVQqZxgCiAB4CXrYEwABu3GKovQ0gVgCMza2xCmCoUNxc6uKSMlBwAD5wSV5eS3ACABbeymscmxIpUADaALqsxGz0aEZglDAQUrYyEYBGtLNxuhwBBBlPktg8nAAKACUTCAA

My library typia can encode and decode protobuf binary data by just only one line.

May I send a PR adding TypeScript tutoral chatper with my library typia?

• https://typia.io/docs/protobuf/message/
• https://typia.io/docs/protobuf/decode/
• https://typia.io/docs/protobuf/encode/

I was trying to vectorize my text from a pdf in preparation of using pinecone vector store.

!pip install --upgrade transformers
!pip install --upgrade protobuf

#for vectors
from transformers import pipeline
import pinecone

AttributeError Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/transformers/utils/import_utils.py in _get_module(self, module_name)
1085 try:
-> 1086 return importlib.import_module("." + module_name, self.name)
1087 except Exception as e:

30 frames
AttributeError: module 'google.protobuf.internal.api_implementation' has no attribute '_c_module'

The above exception was the direct cause of the following exception:

RuntimeError Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/transformers/utils/import_utils.py in _get_module(self, module_name)
1086 return importlib.import_module("." + module_name, self.name)
1087 except Exception as e:
-> 1088 raise RuntimeError(
1089 f"Failed to import {self.name}.{module_name} because of the following error (look up to see its"
1090 f" traceback):\n{e}"

RuntimeError: Failed to import transformers.pipelines because of the following error (look up to see its traceback):
module 'google.protobuf.internal.api_implementation' has no attribute '_c_module'

All links in the page were appended based on current URL, but it's wrong.

For example, in this page, all the links are https://protobuf.dev/reference/cpp/api-docs/google.protobuf.message/{here_is_the_linked_object}:

But the actual link should be https://protobuf.dev/reference/cpp/api-docs/{here_is_the_linked_object}.

I'm reading the encoding document in the section Base 128 Varints. There are several boxed examples, the last of which has five lines. The third line reads:

0000001  0010110        // Put into little-endian order.

Now, this is pseudocode, and these comments are descriptive of what is happening in the text. Endianness refers to byte order in memory, not to text on a page. But at least in left-to-right human languages, picture of memory are usually drawn increasing from left to right or top to bottom, so the lowest order byte (smaller address) is at the left or top (see Wikipedia for example).

This step is putting the most significant bits at the left. So I think the comment is wrong: this step is putting the bytes into BIG-endian order, with the more significant bits at the left.

Also, protobuf number format is very similar (although not identical) to LEB128, which is little-endian. The bytes must be reversed as shown in your example correctly position the more-significant bits.

https://protobuf.dev/overview/

Protocol buffers are not well supported in non-object-oriented languages popular in scientific computing, such as Fortran and IDL.

I disagree with the remark about Fortran being not object-oriented. Fortran (since 2003) supports derived-data-types (a.k.a data-abstraction) and type-bound-procedures (a.k.a methods that are invoked on the held data). This, of course, extends to other capabilities such as interfaces, inheritance, polymorphism, etc. Could the documentation be specific about what aspect of Fortran is incompatible with protocol-buffer, or atleast clarify the Fortran versions that are not supported, so that developers are not misinformed?

A simple example from chat-gpt:

module ShapesModule
  type :: Shape
     real :: area
     contains
        procedure :: calculateArea
  end type Shape

contains

  subroutine calculateArea(this)
    class(Shape), intent(inout) :: this
    ! This is a generic method to calculate the area for different shapes
    ! Implement specific area calculations for each shape by overriding this method
    this%area = 0.0
  end subroutine calculateArea

end module ShapesModule

program ObjectOrientedFortran
  use ShapesModule

  type :: CircleType
     type(Shape) :: baseShape
     real :: radius
  end type CircleType

  contains

  subroutine calculateArea(this)
    class(CircleType), intent(inout) :: this
    this%baseShape%area = this%radius**2 * 3.14159
  end subroutine calculateArea

  type(CircleType) :: myCircle
  myCircle%radius = 5.0

  ! Calculate area using the overridden method for circles
  call myCircle%baseShape%calculateArea()

  ! Print the calculated area
  print *, "Area of the circle: ", myCircle%baseShape%area

end program ObjectOrientedFortran

Here:

you wrote u?int32_t instead of u?int64_t.

How can I trust you format if your docs contains such typos? Did your read your document?!

Currently the Don’t Include Fields that Your Clients Can’t Possibly Have a Use for states the following:

Returning debug data in response protos used to be a common practice, but we have a better way. RPC response extensions (also called “side channels”) let you describe your client interface with one proto and your debugging surface with another.

I cannot find any information about these RPC response extensions or how to use them to achieve this goal anywhere on the side or on the internet. It'd be nice if some link to more information could be included (or the information be included there if it's not available elsewehere).

Compiler Invocation
The protocol buffer compiler produces Java output when invoked with the --java_out= command-line flag. The parameter to the --java_out= option is the directory where you want the compiler to write your Java output. For each .proto file input, the compiler creates a wrapper .java file containing a Java class which represents the .proto file itself.

This paragraph seems to be about the location of the output files, but is also introduces the wrapper class. This is very important.
Many real case scenarios are:
my_proto.proto
{
message MyProto {}
}

which will lead to my.package.MyProtoOuterClass.MyProto classes

The Java package name is chosen as described under Packages, below.

Link doesn't lead anywhere

Hello,

I'm learning about ProtoBuf and writing a POC with proto3 as protobuf syntax. While browsing the doc about Proto3 syntax and the Internet, I found some inconsistencies.
The documentation ProtobufDoc speaks of these 4 field rules : singular, optional, repeated and map, singular being the default field rule.

However, in an issue on gitHub here, it is said that optional has been removed as a field rule, and I understood that it was now the default rule (which is also claimed by both VSCode and WebStorm when I work with it).

I don't find it clear which are the proto3 field rules, and which one is the default one : is it singular ? is it optional ? And if both are still in use in proto3, what is the difference between them ?

I'm sorry if this is a duplicate, I did not found an issue about the presence of optional and singular in proto3 doc or the fact that both are claimed to be the default rule.
Thanks a lot for all the work.

Julie

On https://github.com/protocolbuffers/protocolbuffers.github.io/blob/main/content/programming-guides/encoding.md#signed-integers-signed-ints

Using some bit tricks, it's cheap to convert n into its ZigZag representation:

n + n + (n < 0)

Here, we assume that the boolean n < 0 is converted into an integer 1 if true
or an integer 0 if false.

For example -2 would encode as -2 -2 + 1 or -3, whereas the table above gives 3.

The code snippet referenced in section "The Protocol Buffer API" is no longer accurate. Section should be updated to reflect how latest python code is compiled. meta_class is no longer used.

The previous code snippet:
class Person(message.Message):
metaclass = reflection.GeneratedProtocolMessageType

class PhoneNumber(message.Message):
metaclass = reflection.GeneratedProtocolMessageType
DESCRIPTOR = _PERSON_PHONENUMBER
DESCRIPTOR = _PERSON

class AddressBook(message.Message):
metaclass = reflection.GeneratedProtocolMessageType
DESCRIPTOR = _ADDRESSBOOK

Compiled code using latest repository on 2/28/24:

"""Generated protocol buffer code."""
from google.protobuf import descriptor as _descriptor
from google.protobuf import descriptor_pool as _descriptor_pool
from google.protobuf import symbol_database as _symbol_database
from google.protobuf.internal import builder as _builder
_sym_db = _symbol_database.Default()

DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x11\x61\x64\x64ressbook.proto\x12\x08tutorial"\xd9\x02\n\x06Person\x12\x11\n\x04name\x18\x01 \x01(\tH\x00\x88\x01\x01\x12\x0f\n\x02id\x18\x02 \x01(\x05H\x01\x88\x01\x01\x12\x12\n\x05\x65mail\x18\x03 \x01(\tH\x02\x88\x01\x01\x12,\n\x06phones\x18\x04 \x03(\x0b\x32\x1c.tutorial.Person.PhoneNumber\x1a\x65\n\x0bPhoneNumber\x12\x13\n\x06number\x18\x01 \x01(\tH\x00\x88\x01\x01\x12-\n\x04type\x18\x02 \x01(\x0e\x32\x1a.tutorial.Person.PhoneTypeH\x01\x88\x01\x01\x42\t\n\x07_numberB\x07\n\x05_type"h\n\tPhoneType\x12\x1a\n\x16PHONE_TYPE_UNSPECIFIED\x10\x00\x12\x15\n\x11PHONE_TYPE_MOBILE\x10\x01\x12\x13\n\x0fPHONE_TYPE_HOME\x10\x02\x12\x13\n\x0fPHONE_TYPE_WORK\x10\x03\x42\x07\n\x05_nameB\x05\n\x03_idB\x08\n\x06_email"/\n\x0b\x41\x64\x64ressBook\x12 \n\x06people\x18\x01 \x03(\x0b\x32\x10.tutorial.Personb\x06proto3')

_globals = globals()
_builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, _globals)
_builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'addressbook_pb2', _globals)
if _descriptor._USE_C_DESCRIPTORS == False:
DESCRIPTOR._options = None
_globals['_PERSON']._serialized_start=32
_globals['_PERSON']._serialized_end=377
_globals['_PERSON_PHONENUMBER']._serialized_start=144
_globals['_PERSON_PHONENUMBER']._serialized_end=245
_globals['_PERSON_PHONETYPE']._serialized_start=247
_globals['_PERSON_PHONETYPE']._serialized_end=351
_globals['_ADDRESSBOOK']._serialized_start=379
_globals['_ADDRESSBOOK']._serialized_end=426

As an example, see this screenshot from https://protobuf.dev/reference/cpp/api-docs/google.protobuf.message/

Example - the * is interpreted as a bullet:

complement to encode negative integers. Positive integers `n` are encoded as `2
* n` (the even numbers), while negative integers `-n` are encoded as `2 * n + 1`
(the odd numbers). The encoding thus "zig-zags" between positive and negative
numbers. For example:

https://protobuf.dev/programming-guides/encoding/#signed-ints

The JSON changes (specifically in Python, but possibly others) listed from v25.3 -> v.26 contain breaking changes but are not listed as such.

This was not available in the immediately preceding version and will create runtime errors.

The listing of field presence for proto3 contains Singular Message twice with conflicting statements. The correct statement should be Yes I think.

https://protobuf.dev/programming-guides/field_presence/#presence-in-proto3-apis

Hi, I wanted to get some clarity on whether optional fields in proto are expected to generate Has* methods as described in the documentation here.

The FirstActiveYear struct field will be of type *int32, and additionally have the HasFirstActiveYear() and ClearFirstActiveYear() accessors, because it is marked optional.

When I try to generate Go code for something like:

syntax = "proto3";

package optrepro;

option go_package = "github.com/fzmoment/optrepro";

message MyMessage {
    optional int32 optint = 1;
}

by running

protoc ./opt.proto --go_out=./

the resulting output has a GetOptInt() method but not a HasOptInt(). protoc --version gives me libprotoc 3.20.3.

Looking at some of the tests seems to suggest that there are actually no Has methods, but elsewhere in the protobuf docs these Has methods are mentioned as well.

Let me know if any other information is required.
Thanks!

In this section: https://protobuf.dev/reference/cpp/cpp-generated/#oneof-numeric and this one: https://protobuf.dev/reference/cpp/cpp-generated/#oneof-enum it refers to the oneof field has_<> functions as being Proto2 only. It looks like this may be out of date. I am definitely able to use has_ functions for numeric types, enum types, and bool types in Proto3 and this stack overflow seems to indicate it was added in 3.15: https://stackoverflow.com/a/66541707

Thanks.

The section below is confusing: it says "use reserved" keyword, but the code example uses "deprecated". Could you, please, correct either the text or the example?

Removing enum values is a breaking change for persisted protos. Instead of removing a value, mark the value with the reserved keyword to prevent the enum value from being code-generated:

enum PhoneType {
  PHONE_TYPE_UNSPECIFIED = 0;
  PHONE_TYPE_MOBILE = 1;
  PHONE_TYPE_HOME = 2;
  PHONE_TYPE_WORK = 3 [deprecated=true];
}

Shouldn't SearchRequest definition say 'results_per_page' instead of 'result_per_page'?

int32 result_per_page = 3;

The "Signed Integers" section says:

Using some bit tricks, it’s cheap to convert n into its ZigZag representation:

((n + n) ^ -(n < 0)) - (n < 0)

This formula is wrong; I pasted it into some C++ code and it produced incorrect results. For example, it encodes both 0 and -1 to 0!

The correct formula is actually even simpler: ((n + n) ^ -(n < 0)).

FYI, I worked out the equivalent branchless decoding formula: (n/2) ^ -(n & 1)

Just for reference, here's the zigzag implementation I'm now using. I have a unit test that verifies it produces the same results shown in the table in the docs.

static inline uint64_t encode(int64_t n) {return (uint64_t(n) * 2) ^ -(n < 0);}
static inline int64_t decode(uint64_t n) {return int64_t((n/2) ^ -(n & 1));}

to send notification to telegram upon cross over ove sma

Search for "foo" receive some nonsense responses.

Search for "proto" receive ads :)

Page: https://protobuf.dev/getting-started/pythontutorial/

In Sections "Writing a Message" and "Reading a Message", the code samples are supposed to highlight the parts which directly call or reference code generated by the protocol compiler. It was correctly highlighted on the old site:

but not currently on the new site:

Given the following .proto definition:

syntax = "proto3";

enum Color {
  COLOR_UNSPECIFIED = 0;
  COLOR_RED = 1;
  COLOR_GREEN = 2;
  COLOR_BLUE = 3;
}

The official documentation says that:

The class will include a static const Color for each of the three values defined as well as a static const List<Color> containing all the three non-unspecified values.

But the protobuf compiler generates Dart code like this:

class Color extends $pb.ProtobufEnum {
  static const Color COLOR_UNSPECIFIED = Color._(0, _omitEnumNames ? '' : 'COLOR_UNSPECIFIED');
  static const Color COLOR_RED = Color._(1, _omitEnumNames ? '' : 'COLOR_RED');
  static const Color COLOR_GREEN = Color._(2, _omitEnumNames ? '' : 'COLOR_GREEN');
  static const Color COLOR_BLUE = Color._(3, _omitEnumNames ? '' : 'COLOR_BLUE');

  static const $core.List<Color> values = <Color> [
    COLOR_UNSPECIFIED,
    COLOR_RED,
    COLOR_GREEN,
    COLOR_BLUE,
  ];

  static final $core.Map<$core.int, Color> _byValue = $pb.ProtobufEnum.initByValue(values);
  static Color? valueOf($core.int value) => _byValue[value];

  const Color._($core.int v, $core.String n) : super(v, n);
}

which contains all four values, including COLOR_UNSPECIFIED.

Related: google/protobuf.dart#919

how to use the java script with out using internet operating system

There are two sections on oneofs:
https://protobuf.dev/reference/java/java-generated/#oneof
https://protobuf.dev/reference/java/java-generated/#oneof-fields
It's not clear whether the split is intentional -- they overlap, but the first has things the second doesn't.

It would be better to have just one section on oneof, with the differences reconciled.

According to https://cloud.google.com/endpoints/docs/grpc/transcoding

    option (google.api.http) = { get: "/v1/shelves" };

is a valid option and it is not recognized by the parser specified in language guide.

option = "option" optionName  "=" constant ";"

// Where constant is...
constant = fullIdent | ( [ "-" | "+" ] intLit ) | ( [ "-" | "+" ] floatLit ) | strLit | boolLit

Seems like we should update option to be something like this:

option = "option" optionName  "=" (constant | mapEntry) ";"

mapEntry = "{" { ident ":" constant } "}"

Things like ServiceDescriptor and RepeatedPtrField don't link to anything and searching for them doesn't seem to return any more details.

Relevant page: https://protobuf.dev/reference/cpp/cpp-generated/

The Timestamp section on "Protocol Buffers Well-Known Types" could mention the JSON representation of a timestamp. https://github.com/protocolbuffers/protobuf/blob/main/src/google/protobuf/timestamp.proto#L106-L117

There is no samples how to include properly the google well known types?

I have this:

syntax = "proto3";
import "google/protobuf/value.proto"; ->> and this line does not import at all.
package capability.query;

message Query {
enum OperatorType {
EQ = 0;
NEQ = 1;
GT = 2;
GTEQ = 3;
LT = 4;
LTEQ = 5;
NOT = 6;
}
string field = 1;
OperatorType op = 2;
Value value = 3;
repeated BooleanQuery filters = 5;
}

message BooleanQuery {
enum ConditionType {
AND = 0;
OR = 1;
}
repeated Query filters = 4;
ConditionType op = 5;
}

I get this on compile time:

• What went wrong:
  Execution failed for task ':generateProto'.

protoc: stdout: . stderr: google/protobuf/value.proto: File not found.
models/query.proto:14:1: Import "google/protobuf/value.proto" was not found or had errors.
models/query.proto:29:3: "Value" is not defined.

Some guidance in the api like a reference how to do the imports will be nice and some examples what to configure in gradle.

If I don't use the Value import my code compiles and generates the Kotlin/Java definitions.

Since we're not trying to copy a pointer for the mutex MessageState, is there any way to get the generator not generate it as a pointer OR is there something else I should be doing to handle this "problem"

sorry how do i delete?

Should there be a warning about avoiding the use of DASH characters in the Protobuf Message and/or parameter names?

I'm experience problems trying to compile my *.proto files whenever I attempt to use a parameter name like this: file-id

The make command (which uses Protoc) throws errors "cannot find field ID" for ANY lines which contain parameters named using dashes.

However, simply following to recommended naming style (i.e. file_id) seems to clear the error.

Is there a HARD reason why the dashes are messing up the compiler? Should we itherefore include a warning NOT to use them if this is the case? (I'm thinking the dash may be being confused for an argument indicator?)

Oh yes - if it matters I'm using a Ubuntu environment and trying to compile for use with Python gRPC code.

The Python Generated Code Guide page includes a section Well-Known Types that includes a hyperlink well-known types.

The hyperlink is incorrect:

https://protobuf.dev/reference/protobuf/google.protobuf.md

It should be:

https://protobuf.dev/reference/protobuf/google.protobuf/

The following two Pull Requests to the protocolbuffers/protobuf project add additional functionality for PHP Namespace when generating code with the Protocol Compiler.

• protocolbuffers/protobuf#3162
• protocolbuffers/protobuf#4609

This functionality is not documented in the PHP Generated Code Guide (https://protobuf.dev/reference/php/php-generated/)

Some time ago, a Reference Guide for JavaScript was existing.
Currently, I cannot find a link, e.g. as I do for ruby or python, and the previously working Link to JS reference guide fails with 404.

However, protoc --js_out="... is still working fine.

Can we have the auto generated docu back?
Thanks.

PS: Project was moved to its own github repo, however, documentation is missing also.

In the Encoding page, under "Signed Integers" section, it is mentioned that -500z encoded as ZigZag should be 1001 as varint.
Following the encoding formula for negative sintN (2 * n + 1) however, shouldn't it be 999 instead?

On https://protobuf.dev/programming-guides/proto3/#specifying-field-rules, the link to "Maps" in "See Maps for more on this field type" goes to https://protobuf.dev/programming-guides/encoding.md#maps which is a 404 error. (It appears that encoding.md#maps in the URL should instead be encoding#maps)

The guideline here https://protobuf.dev/programming-guides/style/#services suggests that RPC parameter names should mirrow RPC method names:
service FooService {
rpc GetSomething(GetSomethingRequest) returns (GetSomethingResponse);
rpc ListSomething(ListSomethingRequest) returns (ListSomethingResponse);
}
This makes it really inconvenient to follow when you want to re-use SomethingRequest/SomethingResponse in two different methods. Could you please change the guideline to remove Get/List prefixes, otherwise this leads to artificial limitation that each RPC function has to have a unique parameter type, which isn't mentioned anywhere in the documentation.

I am quite confused about the protoc flag option.Please make it clear and I want a dummy tutorial. I tried a lot of times and my friends told me that "protoc -I . --go_out=. helloI.proto --go-grpc_out=. helloI.proto" will work fine. I don't know how he managed to get this command. Please list all the protoc flag options. It is quite clear that the document is not perfect enough.When I follow other tutorials,everything works right except this one.

In the document https://protobuf.dev/programming-guides/enum/#python you mention about Python 4.22 and 4.21, but newest release of python is (at the moment) 3.11.4.