zcox / data-replication-example Goto Github PK

#12 was implemented using the same kafka-avro serdes in RocksDB that Bottled Water used to write to the Kafka topics. It works, but is really involved due to interaction with Schema Registry and use of KafkaAvroSerializer/KafkaAvroDecoder.

It would probably be simpler to define our own Avro schemas for the keys/values in RocksDB. The Kafka consumer would use the GenericRecord from the BW Kafka topic to create the serialized key/value for RocksDB using our own schemas. Then the User lookup would serialize the userId to the key using our schema, and then deserialize the user from RocksDB lookup using our schema.

Don't wait for DB put, don't do a DB get. Just fire off a request that's done in a Future and return a 200 response.

User will provide a single query string. Search across all user and tweet fields.

Needed to support the onDeletionNotice event from Twitter Streaming API.

Support both endpoints:

• POST /api/users creates new user with generated userId
• PUT /api/users/:userId creates new user with specific userId

The POST version would normally be used by human users to create a new user account.

The PUT version will be useful when we hook into Twitter Streaming API and need to ensure there is a user created with specific userId. Note that the PUT version will update an existing user if one already exists with specific userId.

Use something (custom or Gatling) to continuously make HTTP GET requests against the API, to generate lots of metrics data.

Using a Kafka consumer.

Replace Bottled Water with JDBC Connect and see how well it works.

We can get a lot of data from the Twitter Streaming API to make things more interesting. For each tweet from Streaming API, send the user to PUT /api/users/:userId to ensure it exists, and then send the tweet to PUT /api/tweets/:tweetId.

Using a Kafka consumer.

Also run graphite & grafana Docker containers, and configure Metrics library to report to Graphite. So we can easily see metrics.