justsocialapps / holmes Goto Github PK

As an analyst of my user's behaviour and environment I want to be able to analyse which browsers and operating systems my users use to access my application. I need a breakdown of operating system, browser name, browser version and possibly other values obtained from the user agent string.

STR:

1. Start Holmes with holmes -host localhost -protocol https
2. Fetch the analytics client library: curl http://localhost:3001/analytics.js

Actual result: The base URL used in the client library is localhost:3001/track

Expected result: The base URL used in the client library is https://localhost:3001/track

Currently the tracking object written to Kafka contains the full IP address of the user. For privacy reasons the IP address should be anonymized before writing to Kafka. This behaviour shall be configurable via a command-line switch so that the old behaviour (tracking the full IP address) can still be used.

Currently holmes is logging all messages successfully sent to kafka (including the full body of the tracking event). This is spamming the logs.
Holmes should only log errors.

When using Holmes such that it can be served from multiple domains (e.g. in a development setup: 127.0.0.1, localhost, myhostname), we currently have the problem that the client library tries to issue cross-site requests (to the host provided with the -host parameter).

It should be possible to configure the client library in such a way that it is agnostic of the host/port that is used to fetch it and just make it issue tracking requests against /track instead of protocol://host:port/track.

Especially our dependency to sarama should be pinned to a specific version to guarantee compatibility with Kafka versions.

As a consumer of Holmes' Kafka messages I want to be able to determine the creation date of the message.

We want to enable this feature by adding timestamps to the Kafka messages using Kafka's new message format introduced in 0.10. The Sarama library now supports this new message format (IBM/sarama#678).

As a user, I want to know which Holmes version I currently have at hand. For this, I want to call Holmes with the command-line parameter -version. Holmes then prints the version number and exits.

The tracking object sent to Holmes via /track?t=OBJECT has a fixed set of properties that can be supplied and Holmes ignores all unknown properties. As a user of Holmes I want to be able to provide an arbitrary JSON object.

Currently, Holmes sets the string "key" as key on every message produced to Kafka. This leads to the producer sending every message to the same partition (using the hash of the key). Instead, Holmes should set no key which makes the producer randomly choose a partition.

With every new version of Holmes, binaries for various platforms are published. The authenticity and integrity of those binaries cannot be verified by users downloading them. Thus, those binaries should be signed with a GPG key with every release.

We want to ensure a high code quality with the help of gometalinter. Currently, a run of gometalinter --vendor ./... yields several warnings and errors by gocyclo, golint, vet and errcheck.

Currently, Holmes publishes a time field with every event logged to Kafka which represents a Unix timestamp (seconds since the epoch). This representation can lead to irritations as it is not explicit enough (e.g. does it represent seconds or milliseconds?). Thus, the representation of the field shall be improved so that it represents the time of the event in ISO 8601 combined date and time format understood by e.g. JavaScript's Date constructor.

https://github.com/golang/dep

This enables us to pin dependencies and have more reproducible builds.

As an administrator I want to determine which Holmes version is currently running on a system. This can already be done by invoking holmes -version but this doesn't ensure that this is the binary that is running at the moment.

Therefore, the version number should be printed out on startup.

It should be possible to add application specific fields to all tracking events

Currently the port Holmes listens on can be configured, but Holmes will always only listen on localhost. An option should be added to configure the listen host as well.

When starting, Holmes currently tries to connect to the Kafka broker 7 times within one minute. This means that when Kafka is down longer than one minute when Holmes starts, Holmes will just quit and you'd have to restart it manually.

As an admin I rather want Holmes to try to connect to Kafka indefinitely on startup so that once Kafka comes up, Holmes connects to the broker and starts its duty.

Our hoster reboot some of the servers running Holmes which led to the situation that Holmes wasn't running anymore. The cause of this situation is that Kafka needs some time to start and accept connections and Holmes is not patient enough to wait for Kafka, exiting with a failure.

When Holmes is operated behind a reverse proxy/load balancer the net.http.Request.RemoteAddr denotes the IP address of the proxy. However, we obviously want to track the IP address of the browser (or at least the user's NAT router or forwarding proxy).

Thus, we want to leverage the X-Forwarded-For HTTP header to determine the origin IP address of the user.