Git Product home page Git Product logo

storm-metrics-statsd's Introduction

Storm Metrics Statsd

storm-metrics-statsd is a module for Storm that enables metrics collection and reporting to statsd.

Building/Installation

git clone https://github.com/endgameinc/storm-metrics-statsd.git
cd storm-metrics-statsd
mvn compile package install

Usage

This module can be used in two ways:

  1. Configure it for each topology by calling Conf.registerMetricsConsumer() prior to launching the topology.
  2. Deploy and configure system wide so usage of this is transparent across all topologies.

Configure each topology separately

Add this as a dependency to your pom.xml

<dependency>
  <groupId>com.endgame</groupId>
  <artifactId>storm-metrics-statsd</artifactId>
  <version>1.0.0-SNAPSHOT</version>
</dependency>

Configure the StatsdMetricConsumer when building your topology. The example below is based on the storm-starter ExclamationTopology.

import com.endgame.storm.metrics.statsd.StatsdMetricConsumer;

...

TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("word", new TestWordSpout(), 10);
builder.setBolt("exclaim1", new ExclamationBolt(), 3).shuffleGrouping("word");
builder.setBolt("exclaim2", new ExclamationBolt(), 2).shuffleGrouping("exclaim1");

#
#  Configure the StatsdMetricConsumer
#
Map statsdConfig = new HashMap();
statsdConfig.put(StatsdMetricConsumer.STATSD_HOST, "statsd.server.mydomain.com");
statsdConfig.put(StatsdMetricConsumer.STATSD_PORT, 8125);
statsdConfig.put(StatsdMetricConsumer.STATSD_PREFIX, "storm.metrics.");

Config conf = new Config();
conf.registerMetricsConsumer(StatsdMetricConsumer.class, statsdConfig, 2);
 
if (args != null && args.length > 0) {
  conf.setNumWorkers(3);
  StormSubmitter.submitTopology(args[0], conf, builder.createTopology());
}
else {
  LocalCluster cluster = new LocalCluster();
  cluster.submitTopology("test", conf, builder.createTopology());
  Utils.sleep(5*60*1000L);
  cluster.killTopology("test");
  cluster.shutdown();
}

System Wide Deployment

System wide deployment requires three steps:

1. Add this section to your $STORM_HOME/conf/storm.yaml.

topology.metrics.consumer.register:
  - class: "com.endgame.storm.metrics.statsd.StatsdMetricConsumer"
     parallelism.hint: 2
     argument:
       metrics.statsd.host: "statsd.server.mydomain.com"
       metrics.statsd.port: 8125
       metrics.statsd.prefix: "storm.metrics."

2. Install the storm-metrics-statsd and java-statsd-client JARs into $STORM_HOME/lib/ ON EACH STORM NODE.

$ mvn package
$ mvn org.apache.maven.plugins:maven-dependency-plugin:2.7:copy-dependencies -DincludeArtifactIds=java-statsd-client
$ cp target/dependency/java-statsd-client-1.0.1.jar $STORM_HOME/lib/
$ cp target/storm-metrics-statsd-*.jar $STORM_HOME/lib/

3. Restart storm and you will likely need to restart any topologies running prior to changing your $STORM_HOME/conf/storm.yaml.

Notes

You can override the topology name used when reporting to statsd by calling:

statsdConfig.put(Config.TOPOLOGY_NAME, "myTopologyName");
// OR 
statsdConfig.put("topology.name", "myTopologyName");

This will be useful if you use versioned topology names (.e.g. appending a timestamp or a version string), but only care to track them as one in statsd.

License

storm-metrics-statsd

Copyright 2014 Endgame, Inc.

Endgame, Inc.

    Licensed under the Apache License, Version 2.0 (the "License"); you may
    not use this file except in compliance with the License. You may obtain
    a copy of the License at

         http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.

Author

Jason Trost (@jason_trost)

storm-metrics-statsd's People

Contributors

crohling avatar jatrost avatar jramsdale avatar jsonghbo avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

storm-metrics-statsd's Issues

Release artifact

We can't release our artifacts with SNAPSHOT dependencies. When the code is stable, consider cutting a release and pushing it to a well-known public Maven repository.

ClassCastException

Hi,
I'm new to java and storm. But I'm getting this exception trying to use your library.

2014-09-05 17:01:33 INFO  executor:0 - Finished loading executor __metricscom.endgame.storm.metrics.statsd.StatsdMetricConsumer:[40 40]
2014-09-05 17:01:33 INFO  executor:0 - Preparing bolt __metricscom.endgame.storm.metrics.statsd.StatsdMetricConsumer:(40)
2014-09-05 17:01:33 ERROR util:0 - Async loop died!
java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.Integer
at com.endgame.storm.metrics.statsd.StatsdMetricConsumer.parseConfig(StatsdMetricConsumer.java:80)
at com.endgame.storm.metrics.statsd.StatsdMetricConsumer.prepare(StatsdMetricConsumer.java:64)
at backtype.storm.metric.MetricsConsumerBolt.prepare(MetricsConsumerBolt.java:49)
at backtype.storm.daemon.executor$fn__3352$fn__3364.invoke(executor.clj:690)
at backtype.storm.util$async_loop$fn__452.invoke(util.clj:429)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:745)

I've done everything correctly to my knowledge. So I did a little test.

    Config cong = new Config(); 
    Map statsdConfig = new HashMap();
    statsdConfig.put(StatsdMetricConsumer.STATSD_HOST, Common.STATSD_HOST);
    statsdConfig.put(StatsdMetricConsumer.STATSD_PORT, new Integer(8126));  // I've also tried raw 8126
    statsdConfig.put(StatsdMetricConsumer.STATSD_PREFIX, Common.STATSD_PREFIX);
    // This foo does NOT throw exception.  !!!!
    int foo = (Integer) statsdConfig.get(StatsdMetricConsumer.STATSD_PORT);
    conf.registerMetricsConsumer(StatsdMetricConsumer.class, statsdConfig, 2);

So, why would my int foo not throw exception but the metric-statsd one does? I found out that clojure would convert int into long http://stackoverflow.com/questions/9457537/why-does-int-10-produce-a-long-instance
Yet, this is supposed to be fixed in clojure 1.5+ the current version of storm I'm using, 0.9.2-incubating, is using clojure 1.5.1 http://repo1.maven.org/maven2/org/apache/storm/storm/0.9.2-incubating/storm-0.9.2-incubating.pom

Have you guys seen problem like this? Perhaps making it like this would help

statsdPort = ((Long) conf.get(STATSD_PORT)).intValue();

Send gauges instead of counts to Statsd?

Hi,

I see that the current implementation sends Storm's metrics to Statsd as Statsd counters, not gauges.

Counters are aggregated by Statsd in the specified time window. Gauges are not aggregated by Statsd.

This rappresents a problem for my topology. I record gauge values in my Storm metrics stream. It seems Storm's metric emit timer isn't very precise -- in my example, I set Storm's metric time window to 10 seconds, but often the Storm's metrics are emitted to the metrics consumer in 3 or even 2-second intervals, not the specified 10 second intervals, which, of course, is a problem, as Statsd will sum those two values to a single counter and emit it to the backend (Graphite), giving us the wrong value.

Since Storm's Metric emit interval is so imprecise, I think it would be better to change the implementation to emit the metrics to Statsd as gauges, not counters. Or possibly, support both types of Statsd metric types, and add a configuration option to make it user-configurable.

What do you think? Do you have similar issues with the Storm's metric emit interval varying around the specified interval? My Storm version in use is 0.9.0.1 (latest).

Thanks,
Danijel

Unable to get the storm metrics in Graphite UI

I am trying to display storm metrics into graphite UI.
We chose example Exclamation topology provided by you to test this but I am unable to see the Storm metrics in stats in Graphite UI after the program got executed. Please help me possible solutions that will me solve the situation

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.