raystack / proton Goto Github PK

Push repo to BSR so it can be used for dependency in other Protobuf repos and for remote generation.

Also provide a way to access the generated BSR sequence ID so it can be used to generate the synthetic version used for importing remote generated packages.

We recently made a change to our managed buf.build/googleapis/googleapis repository before certifying BSR as v1. You'll need to run buf mod update to update your buf.lock.

googleapis contains over 3800 files, mostly relating to Google's core APIs. This causes numerous issues for most users, such as timeouts when installing packages with huge swaths of unused code to pull in just a few files. Not only is this a lot of code over the network and on disk, but it can cause issues in editors that try to parse all of that code!

However, there are only ~30 files used by 99.999% of developers, and these files are the most common dependency in the Protobuf ecosystem. For example:

• google.type, which defines useful messages such as DateTime and Money.-
• google.rpc, for interacting with gRPC.
• google.api, for defining APIs, such as with grpc-gateway.

We considered these commonly used packages and included a subset of them into the new, slim googleapis repository.

We hate breaking anyone, even in an alpha, and our apologies that it's causing any difficulties here. The issues with googleapis were serious enough that we felt that we had to do this in this extraordinary circumstance. We're coming out of beta extremely soon, and once we do, we take our responsibility not to break customers (free users or otherwise) very seriously, we appreciate you riding the rough road for the moment!

Is your feature request related to a problem? Please describe.
There are a lot of lint error due to the siren grpc api that were defined are very specific and might be complicated in the future. We use buf lint to do linting. Some suggestion in designing grpc api can be seen here and with buf lint we can implicitly enforcing the design.

Describe the solution you'd like
Here are some changes needed.

RPCs should not sharing request & response, each rpc should have its own request & response ref

rpc CreateProvider(CreateProviderRequest) returns (Provider) 
rpc GetProvider(GetProviderRequest) returns (Provider)

We have above rpc, since they are sharing a message response, in the near future it is harder for us to modify only for a specific RPC. We need to do this instead.

rpc CreateProvider(CreateProviderRequest) returns (CreateProviderRequest) 
rpc GetProvider(GetProviderRequest) returns (GetProviderResponse)

With this approach each message should have a data field as a placeholder of the response. In current implementation we are using domain name
Change from

message GetProviderResponse {
    Provider provider = 1;
}

to

message GetProviderResponse {
    Provider data = 1;
}

We do this to all of our API response so it is consistent and predictable. Response's body always inside the data field.

Better to have a dedicated empty request & response rather than using google.protobuf.Empty ref

rpc ListNamespaces(google.protobuf.Empty) returns (ListNamespacesResponse)

Do this instead

rpc ListNamespaces(ListNamespacesRequest) returns (ListNamespacesResponse)
message ListNamespacesRequest {}

Make this NotifyReceiver rpc more generic and handle the conversion in the service ref

Current implementation is coupling gRPC API with slack contract, we can make this more abstract and do convert the notification message to slack contract in our app.

Define more generic alert history hook API (if possible) ref

Use the buf github actions provided by bufbuild org -

https://github.com/bufbuild/buf-breaking-action
https://github.com/bufbuild/buf-push-action
https://github.com/bufbuild/buf-setup-action
https://github.com/bufbuild/buf-lint-action

Is your feature request related to a problem? Please describe.
As of now files from external sources are cloned and placed in odpf/third_party dir to be used in imports.

Describe the solution you'd like
Define the dependencies in buf.yaml and let buf handle it at time of generation.

Is your feature request related to a problem? Please describe.
We observed that sometimes the ownership information contains e-mail. We need to add a new email field in assets/facets/ownership.proto to support that.

Describe the solution you'd like
Add a new email field with string datatype in assets/facets/ownership.proto.

Is your feature request related to a problem? Please describe.
There are lint errors in odpf/assets that are being used by meteor and columbus. We need to fix the lint error by adding version v1beta1 to support release later.

Describe the solution you'd like
Fix lint and add version v1beta1.

Is your feature request related to a problem? Please describe.
In compass API, we use data field as a field that wrap results of GET API. Although the idea was to make all GET API having the same scaffolding of API response (since it will also be translated to REST response with grpc-gateway) but this might not be that flexible for future use cases. Having plural names of the type of results would make it more future-proof.

repeated Asset assets = 1;

Describe the solution you'd like
Update the response fields to use plural names instead of data

Proton is the central repository for all protobuf definitions. Which includes APIs, entities, events used across the entire ODPF ecosystem. To make the distinctions more clear. The following structure can be used.

├── odpf
│   ├── entities
│   │   ├── README.md
│   │   ├── common
│   │   │   ├── event.proto
│   │   │   ├── resource.proto
│   │   │   └── timestamp.proto
│   │   ├── facets
│   │   │   ├── columns.proto
│   │   │   ├── lineage.proto
│   │   └── resources
│   │       ├── dashboard.proto
│   │       ├── group.proto
│   ├── apis
│   │   ├── guardian
│   │   │   ├── v1
│   │   │   |    ├── guardian.proto
│   │   ├── optimus
│   │   │   ├── v1
│   │   │   |    ├── guardian.proto
  

This will also allow package naming to be more consistent.
io.odpf.apis.guardian.v1
io.odpf.enttiies.resources.topic
io.odpf.events.metadata.optimusJob

Describe the bug
buf-breaking-action in our CI is always failing (in main or in branch).

To Reproduce

• Run github action Validate / Publish

Expected behavior

• buf-breaking-action is working fine and all stages in CI passed

Screenshots

Run bufbuild/[email protected]
buf: downloading buf.build/grpc-ecosystem/grpc-gateway:de24a84aed3d4d84806[7](https://github.com/odpf/proton/runs/6948423099?check_suite_focus=true#step:5:8)40946b55d947
Failure: module had invalid dependency: empty digest.
Error: buf: downloading buf.build/grpc-ecosystem/grpc-gateway:de24a84aed3d4d84806740946b55d947
Failure: module had invalid dependency: empty digest.

Additional context
Add any other context about the problem here.

As of now none of the projects are using the java artefacts and generated FileDescriptorSet binary, hence we can remove the gradle setup from the repo.

As of now the configured category of rules is WIRE, which is the most permissive one. We should look into using one of the more stricter categories -

FILE > PACKAGE > WIRE_JSON > WIRE

One reason for us to move to WIRE_JSON would be for detecting changes that would break pipelines that use Beast.