CarbonJ is a drop-in replacement for carbon-cache and carbon-relay. It was designed with high performance read and write throughput in mind and supports writing millions of metric data points and serve millions of metrics datapoints per minute with low query latency. CarbonJ is designed to run on medium-size instances (AWS: c5.4xlarge or smaller, 500k-1M/m metrics storage capacity per core) and standard (no provisioned iops) GP2 EBS volumes.

This project seeks to sequentialize the random write I/O typically generated by carbon-cache. Carbon-cache writes 'Whisper'-files for each metric. These files have to be updated in the frequency of arriving metrics data -- for millions of metrics data points per minute carbon-cache needs to write millions of files in random order. On aggregation these files need to be rewritten as well when higher resolution gets rolled into lower resolution, causing even more churn.

CarbonJ instead uses JVM-embedded RocksDB to store all metrics datapoints once per interval and different buckets for different resolutions, avoiding churn on aggregation.

It is currently running the Salesforce CommerceCloud Grafana/Graphite metrics stack and handles ~45M metric data points arriving every minute using 6 shards.

• Line
• Pickle

• JSON
• Pickle

• carbon-relay like metrics routing
• carbon-relay compatible blacklisting
• AWS Kinesis streams: Instead of routing metrics data to CarbonJ, Kinesis streams can be used instead. This allows for loss-free restarts of CarbonJ storage nodes.

https://github.com/salesforce/carbonj/blob/master/CarbonJ.pdf

https://engineering.salesforce.com/carbonj-a-high-performance-high-scale-drop-in-replacement-for-carbon-cache-and-carbon-relay-d9599371c51a

Run gradle build to run the test suite and create the Spring boot jar. Run ./gradle build to run the test suite and create the Spring boot jar. Run ./gradlew carbonj.service:docker -PdockerRepo="my-docker-repo.com/" to build the docker image.

Provided in this repository is a working helm chart to deploy an instance of CarbonJ. To deploy it a single partition/shard (numbered 1), you can run helm from the root of this repo with the below command.

helm install --set carbonj.shard.id=1 carbonj-p1 ./kube/helm/carbonj

• The --set carbonj.shard.id=1 option is redundant since this is the default shard id. The name is given as carbonj-p1 to match the shard id.
• If the Kubernetes Namespace or StorageClass resource already exists in the cluster, you'll see an exception during the helm install and will need to remove the resource that the exception complains about from the helm chart and attempt the helm install again.

There are several values that can be tweaked in ./kube/helm/carbonj/values.yaml. The values can be set to non-default value by updating that file or by using Helm's --set <key>=<value> from the command itself. Keys with a small description and default values are below. This current configuration will create 2 replicas of CarbonJ, one in us-east-1a and one in us-east-1b for high availability. The memory settings depend on the instance type. This configuration has proven to be run a metrics stack which a shard/partition persisting around 8 million metrics per minute.