adroll / exmld Goto Github PK

Add documentation and examples for:

1. The interaction with erlmld
2. The interaction with the caller code
3. Checkpointing
4. Basic setup (flow creation and important options, supervision trees, etc)

There is some great doc scattered throughout the sources. These should be exposed in the form of a comprehensive wiki or README so a dev can see upfront what this is all about.

i.e. now that Broadway exists, would you use that instead?

We currently lack unit tests, so let's start by adding them.

Ideally we wouldn't need to run erlmld and can mock/stub it (but DON'T use meck). Create some unit tests and some integration tests exercising both the code that handles what comes in/out from erlmld and what comes in/out of the user code.

It doesn't appear to be possible to set the endpoint the library points to for the aws services such as Kinesis and DynamoDb. Instead it appears that we are hard coded to point at the real aws service endpoint. It would be useful for local development to be able to set the endpoint ourselves. This way we aren't creating resources in our production aws cloud environment when running the example. We could supply the endpoint into the producer configuration map e.g:

    # a kinesis stream processor:
    kinesis_producer = %{
      kinesis_endpoint: "http://localhost:4567",
      dynamodb_endpoint: "http://localhost:8000",
      app_suffix: :k,
      kcl_appname: "erlang-processor-kinesis-test",
      stream_name: stream_name,
      stream_region: stream_region,
      stream_type: :kinesis,
    }

It would be great to have a CI automatically build the project, run unit tests, publish coverage with something like coveralls, etc.