namely / chief-of-state Goto Github PK

Is your feature request related to a problem? Please describe.
There is no clear how to get started doc at the moment

Describe the solution you'd like
Add a how to get started doc on the readme

Describe alternatives you've considered
NO

Additional context
NO

Is your feature request related to a problem? Please describe.
At the moment the readside and the writeside gRPC services are in one Service definition which compels someone to implement both

Describe the solution you'd like
Separate the two services into service gives the implementor the choice not to implement the readSide service.

Describe alternatives you've considered
No

Additional context
No

Describe the bug
At the moment unless the implementor pulls lagom-pb core proto they may not be able to unpack the meta data added the requests.

Expected behavior
Allow commands, events and readSide handler to unpack the meta data

Is your feature request related to a problem? Please describe.
when the remote gRPC fails in the command handler, we should propagate that status and description back to the original caller.

Describe the solution you'd like
catch io.grpc.StatusRuntimeException and bundle it into a proto, pack into the custom failure type as an Any, and translate that back into a gRPC call to the COS client.

should figure out how we can use the status.proto for this purpose. find out if there is a scalapb version already.

Describe alternatives you've considered
could handle each status code with the lagom-pb well-known error types, but seems brittle / too much work.

Additional context
We currently handle validation errors first class, but ideally we would just propagate all command handler errors back to the caller.

Implement local API / Handler service to demonstrate how this all works

Describe the bug
@igamid reported a bug where a gRPC error from the write handler was not propagated back to the API, and instead was just an "internal error". This was discovered when a gRPC interceptor requiring a JWT rejected the handleCommand call.

To Reproduce
TBD

Expected behavior
error should be forwarded back to original caller

Screenshots
n/a

Is your feature request related to a problem? Please describe.
In local dev, we need to set up the journal for you. as it is now, it's too annoying to copy around that .sql file for postgres startup.

Describe the solution you'd like
leverage the lagom schema setup methods to create the journal in development mode

Describe alternatives you've considered
could put the .sql file inside the COS image and share it as a docker-compose volume, but that's clunky.

Additional context
n/a

Is your feature request related to a problem? Please describe.
configuration for retry behavior on the write-side gRPC calls

Describe the solution you'd like

• a config/env var to set the number of retries with some reasonable default.
• perhaps an exponential backoff?
• perhaps it should not retry on validation errors?

Describe alternatives you've considered
n/a

Additional context
n/a

Is your feature request related to a problem? Please describe.
Need a way to put COS image in debug logging mode

Describe the solution you'd like
an env var to set this option

Describe alternatives you've considered
n/a

Additional context
n/a

Is your feature request related to a problem? Please describe.
Although it's helpful and smart to have users register the type url's for state and events, this should be optional. For example, it makes local dev quite complicated if you're designing your protobufs and playing with COS. Further, it should really be up to a user to rely on this setting or verify in separate testing.

Describe the solution you'd like

• Make the HANDLER_SERVICE_STATES_PROTO and HANDLER_SERVICE_EVENTS_PROTOS optional settings
• If not set, the validation of those URL's is not run.

Describe alternatives you've considered
We can make it mandatory in Namely's child image, but the open image should have it optional

Additional context
n/a

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
I would like to some meta data (e.g. ChangedByAccountUUID, AssumedChangedByAccountUUID, etc.) to the write side handler.

Describe the solution you'd like
Add ability to pass meta/headers to ProcessCommand() and then pass meta/headers to the HandleCommand().

Describe the bug
Application.conf fails to parse when the following deprecated env vars are set:

• READ_SIDE_HANDLER_SERVICE_HOST
• READ_SIDE_HANDLER_SERVICE_PORT

To Reproduce

1. Pull the python sample app
2. remove the deprecated env vars
3. make up
4. (once all pods up) make test

Expected behavior
Should start

Notes:

• planning when in the Lagom command lifecycle we make the HandlerService calls
• This must correctly handle gRPC errors from the HandlerService
• This is where we do basic validations like type URL’s
• probably create child issues to implement the actual command/event calls

Is your feature request related to a problem? Please describe.
It is not related with a particular problem but it may be useful to block gRPC response until new state is materialized by read side.

Describe the solution you'd like
CoS wouldn't return new state to the gRPC caller until it receives confirmation from the read side that new state is materialized.

• Add debug log message during request processing

• Add warning log message during data validation (for instance invalid state and event proto) before returning the GrpException.

• Add good error log message in case of failure.

Update the HandleCommandResponse proto (per PR below) to simplify the response type.

relates to namely/chief-of-state-protos/pull/11

Is your feature request related to a problem? Please describe.
It is difficult to debug calls going through multiple services like gRPC - CoS - write side handler

Describe the solution you'd like
Use meta/header to setup parent/child relation for span generated by different services.
As result we should be see in, for example, Jaeger that request was received by gRPC server, which called ProcessCommand(), which then called HandleCommand(), etc.

Is your feature request related to a problem? Please describe.
When the write handler returns a gRPC exception, we should properly propagate validation errors back to the caller. Right now, there does not seem to be specific handling of gRPC status codes.

Describe the solution you'd like
If a user expresses an invalid argument error (or any similar gRPC version of "invalid arguments"), COS should return that error message to the caller. Other gRPC exceptions can be treated as internal error.

I believe we already have the mechanism for this, so this work is just to audit the error types and clean it up.

Describe alternatives you've considered
We could just push all error messages to the API client, but that's probably bleeding too much implementation.

Additional context
n/a

Questions:

• What settings do we expose
• How does local dev work
• How does deploy work

Is your feature request related to a problem? Please describe.
If client wants to GET current state by ID, they should be able to avoid the handler calls. It shouldn't be necessary for a handler to implement a GET (aka, to reply with the Reply response type for command handler)

Describe the solution you'd like

• Add a GetCurrentState(<entity id>) gRPC endpoint
• implement it in COS to return state if it exists and avoid the handler calls

Describe alternatives you've considered
n/a

Additional context
n/a

• planning what type of validations we should do in the command handlers and event handlers
• As earlier discussed we should find a way to do basic validations like type URL’s

Is your feature request related to a problem? Please describe.
when lagom-pb v0.6 is released, upgrade dependency and fix the application to remove duplicate proto encryption

Describe the solution you'd like

• upgrades to lagom-pb v0.6.0
• removes duplicate proto encryption

Describe alternatives you've considered
n/a

Additional context
requires #49 #56

Is your feature request related to a problem? Please describe.
need to enable the lagom-pb logical deletion feature, which deletes events older than the oldest snapshot per entity.

Describe the solution you'd like
add an env-var like COS_JOURNAL_LOGICAL_DELETION = true/false to enable/disable this.

Describe alternatives you've considered
n/a

Additional context
super-flat/lagom-pb@11890b9

Is your feature request related to a problem? Please describe.
For instrumentation, it would be great to have the option to send out traces using either the zipkin or jaeger client

Describe the solution you'd like
COS currently ships with a jaeger client as the only tracing option, instead we would update the service to house both the jaeger and zipkin clients and use env vars to enable either of them
COS_JAEGER_ENABLED
COS_ZIPKIN_ENABLED

Is your feature request related to a problem? Please describe.
At the moment the command reply timeout is of 5 seconds.

Describe the solution you'd like
Expose an environment variable to make it configurable

Describe alternatives you've considered
No

Additional context
No

Describe the bug
first request is still sometimes slow. relates to super-flat/lagom-pb#68

most likely caused by timing of reflection

Is your feature request related to a problem? Please describe.
configuration objects are getting messy. let's move them into a folder in COS and a subpackage com.namely.chiefofstate.config.

Describe the solution you'd like
see above

Describe alternatives you've considered
not doing it?

Additional context
n/a

This ticket upgrades the lagom-pb dependency to v0.6.0.

Relates to #46

Describe the bug
Calling CreatePayGroupAssignment with x-validate-only flag set to true and getting the following error:
Rpc failed with status code 13, error message: message is nil

To Reproduce
Steps to reproduce the behavior:

1. Call CreatePayGroupAssignment with x-validate-only flag set to true
2. See error

Expected behavior
Should get back 0 "OK" with current (empty in this case) state.

Is your feature request related to a problem? Please describe.
leverage existing gRPC NOT FOUND exception when client invokes GetState and no entity found

Describe the solution you'd like
COS should abstract this from the client instead of making them handle specific error text. In future, this should be pushed upstream into lagom-pb, but near-term offer a better api by catching the error string internally in COS's gRPC implementation.

Describe alternatives you've considered

• making upstream lagom-pb changes now, but we should do a larger investigation

Additional context
https://grpc.github.io/grpc/core/md_doc_statuscodes.html

• https://grpc.io/docs/guides/error/#standard-error-model
• https://cloud.google.com/apis/design/errors#error_model

Make sure you pass through GRPC rich exceptions when raised

might require modifications to lagom common

Is your feature request related to a problem? Please describe.
support rich read side data by upgrading to lagom-pb 0.5.0

Describe the solution you'd like

• upgrade lagom-pb to 0.5.0
• expose those new attributes on the gRPC read side as headers

Describe alternatives you've considered
n/a

Additional context
coming work for journal migration would benefit from access to the shard and tag in the read side processor. lagom-pb v0.5.0 adds those features.

Is your feature request related to a problem? Please describe.
Need configuration to allow multiple grpc readsides via akka projections. today, we only support one.

Describe the solution you'd like
Some kind of configuration that allows multiple readsides, like grouped configs / env vars.

Describe alternatives you've considered
n/a

Additional context
n/a

Is your feature request related to a problem? Please describe.
Change encryption setting to be optional in preparation for lagom-pb v0.6

Describe the solution you'd like

• empty encryption setting returns a None

Describe alternatives you've considered
n/a

Additional context
This is technically a breaking change. Let's put it in v0.2.X

Is your feature request related to a problem? Please describe.
COS updated the v1 protos:

• breaking name changes to make the interface more clear (per feedback from users)
• breaking change for Metadata data type to support Any plugin state

Describe the solution you'd like
update the submodule, fix COS

Describe alternatives you've considered
n/a

Additional context

• namely/chief-of-state-protos#14
• namely/chief-of-state-protos#15

we need to implement an optional read side processor that takes each event and delegates to a gRPC service, and it should only commit the offset upon successful gRPC response

Is your feature request related to a problem? Please describe.
I would like to check that requester is changing data which belongs to the requesters company (i.e. JWT.CommandUUID = Cmd.Company_UUID).

Describe the solution you'd like
I would like to have access to the JWT in write side handler.

Is your feature request related to a problem? Please describe.
At the moment there is no unit tests to cover the various implementation

Describe the solution you'd like
Add unit test to the code base. This might involve faking a gRPC service.

Describe the bug
There are Could not create an Appender of type ..., ActionException in Action for tag, etc. errors when CoS is running in the Docker.
Note: CoS runs but log is filled with errors.

To Reproduce
Steps to reproduce the behavior:

1. Run CoS, I'm using cos-internal:0.2-beta.1 image, in the Docker
2. Look at logs of the CoS container
3. See errors:

20:29:01,318 |-ERROR in ch.qos.logback.core.joran.action.AppenderAction - Could not create an Appender of type [net.logstash.logback.appender.LoggingEventAsyncDisruptorAppender].
...
20:29:01,318 |-ERROR in ch.qos.logback.core.joran.spi.Interpreter@19:105 - ActionException in Action for tag [appender] ch.qos.logback.core.joran.spi.ActionException:
...
20:29:01,337 |-ERROR in ch.qos.logback.core.joran.conditional.IfAction - Could not find Janino library on the class path. Skipping conditional processing.
20:29:01,337 |-ERROR in ch.qos.logback.core.joran.conditional.IfAction - See also http://logback.qos.ch/codes.html#ifJanino

Expected behavior
No errors in the log unless there are actual errors :)

Is your feature request related to a problem? Please describe.
This task makes COS use the latest lagom-pb changes to encrypt the journal. This work prepares COS for proper encryption of events to comply with GDPR/CCPA.

Describe the solution you'd like

• once lagom-pb releases, update dependency
• add an env var / configuration for picking the encryptor used, but default to NoEncryption

Describe alternatives you've considered
n/a

Additional context
relates to super-flat/lagom-pb#84
relates to #30

Is your feature request related to a problem? Please describe.
propagate the "propagated headers" to handleEvent just like we do for handleCommand

Describe the solution you'd like
changes:

• use the nested "Remote Call" proto for handle event as well
• probably rename to "RemoteCall" instead of "RemoteCommand"

Describe alternatives you've considered
n/a

Additional context
relates to #62

Is your feature request related to a problem? Please describe.
We need a way to write plugins that intercept the ProcessCommand request (body and headers) like gRPC interceptors, write to an output Map (for the metadata), and a way to dynamically inject and run them in the docker container.

Describe the solution you'd like

• abstract class / trait for a CommandPlugin that can:

  • validate the request or headers and short circuit on failure
  • have access to the request (body and gRPC metadata)
  • have a unique ID for the plugin type (like auditing)
  • can output a scala Map[String, String]
  • can be found/instantiated via reflection with no arguments

• A configurable plugin pipeline that:

  • runs all provided plugins
  • reduces all plugin outputs to a single Map with keys <plugin id>|<plugin key> like auditing|company_uuid
  • exits the request early if any plugins fail

• A separate SBT project in COS that defines the base traits / classes

  • should get published to jfrog and eventually maven
  • should be a dependency of the core COS docker service

Describe alternatives you've considered

• use gRPC interceptors

Additional context
We should port the gRPC persisted headers setting to implement this

Is your feature request related to a problem? Please describe.
Remove the COS prefix on all the objects and file names, as it's redundant with the package.

Describe the solution you'd like
Remove the COS prefix on all the objects and file names, as it's redundant with the package.

Describe alternatives you've considered
n/a

Additional context
n/a

Is your feature request related to a problem? Please describe.
Need a configurable way to encrypt events in the journal and state in the snapshot store, but offer the decrypted payload to the handler service + gRPC read sides.

Describe the solution you'd like
Features:

• An ProtoEncryption trait that offers .encrypt and .decrypt methods
• a well known proto for encrypted payloads with:
  • a oneOf attribute for the successfully decrypted bytes or failure
  • a way to identify the key used for the ProtoEncryption implementation
• Specific failures that distinguish deleted keys from failures
• configurable via hocon and reflected in at boot

Behaviors:

• chief of state should return a gRPC failure when a command is sent in for which the entity cannot be decrypted
• gRPC read side processor should just skip (and ACK) a message which cannot be decrypted, as it means the key was revoked

Describe alternatives you've considered

• encrypting before sending in the proto (dangerous, harder for users)
• using the event adapter (won't handle snapshots or in-memory messages)

Additional context