Git Product home page Git Product logo

jmxtrans-agent's Introduction

Jmxtrans Agent

Travis GitHub issues GitHub license GitHub forks GitHub stars

What is jmxtrans-agent ?

jmxtrans-agent is a version of jmxtrans intended to be used as a java agent. JmxTrans Agent has zero dependencies to ease integration.

Java Agent Declaration

Download the latest release of jmxtrans-agent-<version>.jar

Sample setenv.sh for Apache Tomcat

export JAVA_OPTS="$JAVA_OPTS -javaagent:/path/to/jmxtrans-agent-1.2.10.jar=jmxtrans-agent.xml"
  • java agent jar path can be relative to the working dir
  • jmxtrans-agent.xml is the configuration file, can be classpath relative (classpath:…), http(s) (http(s)://...) or file system based (relative to the working dir)

Delayed startup (version >= 1.2.1)

For some application servers like JBoss, delaying premain is needed to start the agent, see WFLY-3054 This has been confirmed to be needed with JBoss 5.x, 6.x, 7.x and Wildfly 8.x. This is because a custom MBeanServer is used by programmatically setting the "javax.management.builder.initial" system property in JBoss's startup sequence. If the PlatformMBeanServer is initialized before this is set, the PlatformMBeanServer will not use the implementation JBoss expects.

For versions >=1.2.8, you can wait for the custom MBeanServer to be defined, set jmxtrans.agent.premain.waitForCustomMBeanServer=true:

# delays calling premain() in jmxtrans agent until javax.management.builder.initial is set, up to 2 minutes
java -Djmxtrans.agent.premain.waitForCustomMBeanServer=true

This usually takes less than a second. If needed, you can optionally increase the timeout to wait by setting jmxtrans.agent.premain.waitForCustomMBeanServer.timeoutInSeconds (defaults to 2 minutes):

# delays calling premain() in jmxtrans agent until javax.management.builder.initial is set, up to 5 minutes
java -Djmxtrans.agent.premain.waitForCustomMBeanServer=true -Djmxtrans.agent.premain.waitForCustomMBeanServer.timeoutInSeconds=300

For versions <1.2.8, you have to add a flat delay. To add a flat delay set jmxtrans.agent.premain.delay (value in seconds):

# delays calling premain() in jmxtrans agent for 30 seconds
java -Djmxtrans.agent.premain.delay=30

Query configuration

Selecting attributes

To select which attributes to collect, use the attribute or attributes attribute on the query element. attribute accepts a single attribute while attributes accepts a comma-separated list of attributes to collect. If you do not specify any attributes, all attributes of the MBean will be dynamically discovered and collected. Use the expression language #attribute# in the resultAlias to use the attribute name in the metric name when collecting many attributes.

Example - collect the ThreadCount attribute from the Threading MBean:

<query objectName="java.lang:type=Threading" attribute="ThreadCount"
   resultAlias="jvm.thread.count"/>

Example - collect the SystemLoadAverage gauge attribute from the OperatingSystem MBean:

<query objectName="java.lang:type=OperatingSystem" attributes="SystemLoadAverage" 
       type="gauge" resultAlias="#attribute#"/>

(i) Note that the type attribute is customizable. Output writers such as the LibratoWriter, StatsDOutputWriter and PerMinuteSummarizerOutputWriter are aware of the types counter and gauge and assume that non defined typemeans counter.

Example - collect ThreadCount and TotalStartedThreadCount from the Threading MBean:

<query objectName="java.lang:type=Threading" attributes="ThreadCount,TotalStartedThreadCount"
  resultAlias="jvm.threads.#attribute#"/>

Example - collect all attributes from the Threading MBean:

<query objectName="java.lang:type=Threading" resultAlias="jvm.threads.#attribute#"/>

Simple mono-valued attribute

Use attribute or attributes to specify the values to lookup. No additional configuration is required. See javax.management.MBeanServer.getAttribute(objectName, attribute).

<query objectName="java.lang:type=Threading" attribute="ThreadCount"
   resultAlias="jvm.thread"/>

MBean Composite Data attribute (CompositeData or Map)

Use key to specify the key of the CompositeData or Map. See javax.management.openmbean.CompositeData#get(key).

 <query objectName="java.lang:type=Memory" attribute="HeapMemoryUsage" key="used"
    resultAlias="jvm.heapMemoryUsage.used"/>
  • You can collect all the keys of the composite data or map omitting key in the <query /> declaration.
  • Use the expression language #key# (or its synonym #compositeDataKey#) in the resultAlias to use the composite data key in the metric name. Sample:
 <query objectName="java.lang:type=Memory" attribute="HeapMemoryUsage" resultAlias="jvm.heapMemoryUsage.#key#"/>

Multi-valued data (Iterable or array)

Use position to specify the value to lookup. Position is `0 based.

 <query objectName="MyApp:type=MyMBean" attribute="MyMultiValuedAttribute" position="2"
    resultAlias="myMBean.myMultiValuedAttributeValue"/>
  • position is 0 based
  • You can collect all the entries of the multi-valued data omitting position in the <query /> declaration.
  • Use the expression language #position# in the resultAlias to use the multi-valued data position in the metric name. Sample:
 <query objectName="MyApp:type=MyMBean" attribute="MyMultiValuedAttribute" resultAlias="myMBean.myMultiValuedAttributeValue.#position#"/>
  • If no resultAlias is specified, the generated metric name is suffixed by _#position#. Sample:
myMBean.myMultiValuedAttributeValue_0`

Additional Configuration

Dynamic configuration reloading

The configuration can be dynamically reloaded at runtime. To enable this feature, enable it with the reloadConfigurationCheckIntervalInSeconds element, e.g.:

<reloadConfigurationCheckIntervalInSeconds>60</reloadConfigurationCheckIntervalInSeconds>

Collection interval

The interval for collecting data using the specified queries and invocations can be specified with the element collectIntervalInSeconds, e.g.:

<collectIntervalInSeconds>20</collectIntervalInSeconds>

The collect interval can be overridden for a specific query or invocation by setting the collectIntervalInSeconds attribute, e.g.:

<query objectName="java.lang:type=Threading" attributes="ThreadCount,TotalStartedThreadCount"
   resultAlias="jvm.threads.#attribute#" collectIntervalInSeconds="5"/>

ResultNameStrategy

The ResultNameStrategy is the component in charge of building the metric name. The default implementation uses the resultAlias if provided and otherwise will build the metric name using the ObjectName.

You can use your own implementation for the ResultNameStrategy

<resultNameStrategy class="com.mycompany.jmxtrans.agent.MyResultNameStrategyImpl">
   <attrA>valA</attrA>
   <attrB>valB</attrB>
</resultNameStrategy>

You then have to make this implementation available in the classpath (adding it the the jmxtrans-agent jar, adding it to the boot classpath ...)

Sample configuration file

Sample jmxtrans-agent.xml configuration file for Tomcat:

<jmxtrans-agent>
    <queries>
        <!-- OS -->
        <query objectName="java.lang:type=OperatingSystem" attribute="SystemLoadAverage" resultAlias="os.systemLoadAverage"/>

        <!-- JVM -->
        <query objectName="java.lang:type=Memory" attribute="HeapMemoryUsage" key="used"
               resultAlias="jvm.heapMemoryUsage.used"/>
        <query objectName="java.lang:type=Memory" attribute="HeapMemoryUsage" key="committed"
               resultAlias="jvm.heapMemoryUsage.committed"/>
        <query objectName="java.lang:type=Memory" attribute="NonHeapMemoryUsage" key="used"
               resultAlias="jvm.nonHeapMemoryUsage.used"/>
        <query objectName="java.lang:type=Memory" attribute="NonHeapMemoryUsage" key="committed"
               resultAlias="jvm.nonHeapMemoryUsage.committed"/>
        <query objectName="java.lang:type=ClassLoading" attribute="LoadedClassCount" resultAlias="jvm.loadedClasses"/>

        <query objectName="java.lang:type=Threading" attribute="ThreadCount" resultAlias="jvm.thread"/>

        <!-- TOMCAT -->
        <query objectName="Catalina:type=GlobalRequestProcessor,name=*" attribute="requestCount"
               resultAlias="tomcat.requestCount"/>
        <query objectName="Catalina:type=GlobalRequestProcessor,name=*" attribute="errorCount"
               resultAlias="tomcat.errorCount"/>
        <query objectName="Catalina:type=GlobalRequestProcessor,name=*" attribute="processingTime"
               resultAlias="tomcat.processingTime"/>
        <query objectName="Catalina:type=GlobalRequestProcessor,name=*" attribute="bytesSent"
               resultAlias="tomcat.bytesSent"/>
        <query objectName="Catalina:type=GlobalRequestProcessor,name=*" attribute="bytesReceived"
               resultAlias="tomcat.bytesReceived"/>

        <!-- APPLICATION -->
        <query objectName="Catalina:type=Manager,context=/,host=localhost" attribute="activeSessions"
               resultAlias="application.activeSessions"/>
    </queries>
    <outputWriter class="org.jmxtrans.agent.GraphitePlainTextTcpOutputWriter">
        <host>localhost</host>
        <port>2003</port>
        <namePrefix>app_123456.servers.i876543.</namePrefix>
    </outputWriter>
    <outputWriter class="org.jmxtrans.agent.ConsoleOutputWriter"/>
    <collectIntervalInSeconds>20</collectIntervalInSeconds>
</jmxtrans-agent>

Note why XML and not JSON ? because XML parsing is out of the box in the JVM when JSON requires additional libraries.

OutputWriters

OutputWriters are very simple to develop, you just have to extend AbstractOutputWriter.java or to implement OutputWriter.java.

Out of the box output writers:

  • GraphitePlainTextTcpOutputWriter: output to Graphite Carbon plain text protocol on TCP. Configuration parameters:
    • enabled: to enable/disable the output writer. Optional, default value true
    • host: Graphite Carbon listener host
    • port: Graphite Carbon Plain Text TCP listener port. Optional, default value 2003
    • namePrefix; prefix of the metric name. Optional, default values servers.#hostname#. where #hostname# is the auto discovered hostname of computer with . escaped as _ (InetAddress.getLocalHost().getHostName()).
  • GraphiteUdpOutputWriter: output to Graphite Carbon plain text protocol on UDP. Supports the same configuration parameters as the GraphitePlainTextTcpOutputWriter
  • FileOverwriterOutputWriter: store the last collection of metrics in a file. Configuration parameters:
    • fileName: name of the file in which the collected metrics are stored. Optional, default value jmxtrans-agent.data (in JVM working dir, for example $TOMCAT_HOME/bin)
    • showTimeStamp: true or false value that determines if the time stamp is printed with the lines. Optional tag, default is `false.
  • SummarizingFileOverwriterOutputWriter: Similar to the FileOverwriterOutputWriter but displays "per minute" values for counters of type counter
  • ConsoleOutputWriter: output metric values to stdout
  • SummarizingConsoleOutputWriter: Similar to the ConsoleOutputWriter but displays "per minute" values for counters of type counter
  • RollingFileOutputWriter
    • fileName: Name of the file in which the collected metrics are stored. Optional, default value jmxtrans-agent.data (in JVM working dir, for example $TOMCAT_HOME/bin)
    • maxFileSize: Maximum file size in MB before file is rolled. Optional, default is 10
    • maxBackupIndex: Maximum number of backup files. Optional, default is `5
    • singleLine: true or false value that determines if all values are printed on a single line. Optional, default is false
  • StatsDOutputWriter: output to StatD using the counter metric type. Configuration parameters:
    • host: StatsD listener host
    • port: StatsD listener port
    • statsd : Optional StatsD server type, statsd, dd or sysdig
    • tags : Optional StatsD tags for dd and sysdig, i.e. serviceid:SERVICE_ID,environment:dev
    • metricName: metric name prefix. Optional, default value is machine hostname or IP (all . are scaped as _).
    • bufferSize: max buffer size. Holds data to be sent. Optional, default value is 1024.
  • InfluxDbOutputWriter: output to InfluxDb. This writer is currently experimental - behavior and options might change. See InfluxDbOutputWriter Details for more details. Configuration parameters:
    • url: url to the influxdb server, e.g. <url>http://influx.company.com:8086</url> - required
    • database: name of the database to write to - required
    • user: username for authentication - optional
    • password: password for authentication - optional
    • tags: additional tags to use for all metrics on n1=v1,n2=v2 format, e.g. <tags>#hostname#</tags> - optional
    • retentionPolicy: retention policy to use - optional
    • connectTimeoutMillis: connect timeout for the HTTP connection to influx - optional, defaults to 3000
    • readTimeoutMillis: read timeout for the HTTP connection to influx - optional, defaults to 5000

Output writers configuration support an expression language based on property placeholders with the {prop-name[:default-value]} syntax (e.g. "${graphite.port:2003}").

The default-value is optional. An exception is raised if no default value is defined and the property placeholder is not found.

Environment variables are looked-up in the following order:

  1. JVM system properties (System.getProperty("graphite.host"))
  2. JVM environment variables (System.getenv("graphite.host"))
  3. JVM environment variables after a "to-upper-case + dot-to-underscore" transformation (System.getenv("GRAPHITE_HOST"))

InfluxDbOutputWriter Details

This writer is currently in beta, it might have bugs and the behavior and options might change.

When using the InfluxDbOutputWriter, the queries' resultAlias have special semantics. The result alias is a comma-separated list where the first item is the name of the measurement and the rest of the items are tags to add to metrics collected by the query. For example, the query

<query objectName="java.lang:type=GarbageCollector,name=*"
	attributes="CollectionTime,CollectionCount"
	resultAlias="#attribute#,garbageCollector=%name%,myTag=foo" />

will result in measurements named as the attributes collected (CollectionTime and CollectionCount). The additional tag garbageCollector will be added which will correspond to the name attribute of the object name. In addition, a tag called myTag with value foo will be added.

All measurements sent to InfluxDb will have only one field called value. Multiple fields are currently not supported.

Example complete output writer configuration:

<outputWriter class="org.jmxtrans.agent.influxdb.InfluxDbOutputWriter">
	<url>http://localhost:8086</url>
	<database>mydb</database>
	<user>admin</user>
	<password>shadow</password>
	<tags>host=#hostname#</tags>
</outputWriter>

Sample of ConsoleOutputWriter

os.systemLoadAverage 1.80419921875 1366199958
jvm.heapMemoryUsage.used 20438792 1366199958
jvm.heapMemoryUsage.committed 119668736 1366199958
jvm.nonHeapMemoryUsage.used 15953560 1366199958
jvm.nonHeapMemoryUsage.committed 24313856 1366199958
jvm.loadedClasses 2162 1366199958
jvm.thread 13 1366199958
tomcat.requestCount 0 1366199958
tomcat.requestCount 0 1366199958
tomcat.errorCount 0 1366199958
tomcat.errorCount 0 1366199958
tomcat.processingTime 0 1366199958
tomcat.processingTime 0 1366199958
tomcat.bytesSent 0 1366199958
tomcat.bytesSent 0 1366199958
tomcat.bytesReceived 0 1366199958
tomcat.bytesReceived 0 1366199958
application.activeSessions 0 1366199958

Sample of FileOverwriterOutputWriter

os.systemLoadAverage 1.27734375
jvm.heapMemoryUsage.used 33436016
jvm.heapMemoryUsage.committed 133365760
jvm.nonHeapMemoryUsage.used 23623096
jvm.nonHeapMemoryUsage.committed 24707072
jvm.loadedClasses 3002
jvm.thread 21
tomcat.requestCount 27
tomcat.requestCount 0
tomcat.errorCount 0
tomcat.errorCount 0
tomcat.processingTime 881
tomcat.processingTime 0
tomcat.bytesSent 135816
tomcat.bytesSent 0
tomcat.bytesReceived 0
tomcat.bytesReceived 0
application.activeSessions 0

Release Notes

Sample ActiveMQ Configuration

  • Create directory ${ACTIVEMQ_HOME}/jmxtrans-agent/
  • Copy jmxtrans-agent-1.2.4.jar under ${ACTIVEMQ_HOME}/jmxtrans-agent/
  • Update ${ACTIVEMQ_HOME}/bin/activemq, add in invoke_start() and invoke_console():
JMXTRANS_AGENT="-javaagent:${ACTIVEMQ_HOME}/jmxtrans-agent/jmxtrans-agent-1.2.4.jar=${ACTIVEMQ_HOME}/jmxtrans-agent/jmxtrans-agent-activemq.xml"
ACTIVEMQ_OPTS="$ACTIVEMQ_OPTS $JMXTRANS_AGENT"
  • Copy to ${ACTIVEMQ_HOME}/jmxtrans-agent/ a config file similar to
<jmxtrans-agent>
    <queries>
        <!-- OS -->
        <query objectName="java.lang:type=OperatingSystem" attribute="SystemLoadAverage"
               resultAlias="os.systemLoadAverage"/>

        <!-- JVM -->
        <query objectName="java.lang:type=Memory" attribute="HeapMemoryUsage" key="used"
               resultAlias="jvm.heapMemoryUsage.used"/>
        <query objectName="java.lang:type=Memory" attribute="HeapMemoryUsage" key="committed"
               resultAlias="jvm.heapMemoryUsage.committed"/>
        <query objectName="java.lang:type=Memory" attribute="NonHeapMemoryUsage" key="used"
               resultAlias="jvm.nonHeapMemoryUsage.used"/>
        <query objectName="java.lang:type=Memory" attribute="NonHeapMemoryUsage" key="committed"
               resultAlias="jvm.nonHeapMemoryUsage.committed"/>
        <query objectName="java.lang:type=ClassLoading" attribute="LoadedClassCount" resultAlias="jvm.loadedClasses"/>

        <query objectName="java.lang:type=Threading" attribute="ThreadCount" resultAlias="jvm.thread"/>

        <!-- ACTIVE MQ -->
        <query objectName="org.apache.activemq:type=Broker,brokerName=*,destinationType=Queue,destinationName=*"
               attribute="QueueSize" resultAlias="activemq.%brokerName%.queue.%destinationName%.QueueSize"/>
        <query objectName="org.apache.activemq:type=Broker,brokerName=*,destinationType=Queue,destinationName=*"
               attribute="EnqueueCount" resultAlias="activemq.%brokerName%.queue.%destinationName%.EnqueueCount"/>
        <query objectName="org.apache.activemq:type=Broker,brokerName=*,destinationType=Queue,destinationName=*"
               attribute="ExpiredCount" resultAlias="activemq.%brokerName%.queue.%destinationName%.ExpiredCount"/>
        <query objectName="org.apache.activemq:type=Broker,brokerName=*,destinationType=Queue,destinationName=*"
               attribute="DequeueCount" resultAlias="activemq.%brokerName%.queue.%destinationName%.DequeueCount"/>

        <query objectName="org.apache.activemq:type=Broker,brokerName=*,destinationType=Topic,destinationName=*"
               attribute="EnqueueCount" resultAlias="activemq.%brokerName%.topic.%destinationName%.EnqueueCount"/>
    </queries>
    <outputWriter class="org.jmxtrans.agent.GraphitePlainTextTcpOutputWriter">
        <host>localhost</host>
        <port>2203</port>
    </outputWriter>
    <outputWriter class="org.jmxtrans.agent.ConsoleOutputWriter">
        <enabled>false</enabled>
    </outputWriter>
    <outputWriter class="org.jmxtrans.agent.RollingFileOutputWriter">
      <fileName>rollingJMXOutputFile</fileName>
      <maxFileSize>10</maxFileSize>
      <maxBackupIndex>4</maxBackupIndex>
   </outputWriter>
</jmxtrans-agent>

Release Notes

See https://github.com/jmxtrans/jmxtrans-agent/releases

jmxtrans-agent's People

Contributors

acl-oss avatar alechenninger avatar andhan avatar ayman-abdelghany avatar aymandf avatar billonahill avatar bmmchugh avatar claudiouzelac avatar cyrille-leclerc avatar dd00f avatar devmage avatar endoplasmicr avatar evmin avatar farnulfo avatar gehel avatar ggrandes avatar gliptak avatar jakobant avatar kailashhd avatar kerlandsson avatar maheshkelkar avatar nbvehrfr avatar ncolomer avatar packysauce avatar pingtimeout avatar raiderx avatar simmel avatar zbintliff avatar zhengyd2014 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

jmxtrans-agent's Issues

Append attribute and/or compositeData key name to the resultAlias

The following query exports all the attributes of the given objectName:
<query objectName="java.lang:type=Memory" resultAlias="my.localmachine"/>

This query, generates the following output:

my.localmachine false 1432740403
my.localmachine 0 1432740403
my.localmachine 257425408 1432740403
my.localmachine 268435456 1432740403
my.localmachine 3817865216 1432740403
my.localmachine 79592824 1432740403
my.localmachine 50003968 1432740403
my.localmachine 24576000 1432740403
my.localmachine 136314880 1432740403
my.localmachine 49349944 1432740403
my.localmachine java.lang:type=Memory 1432740403

It uses the resultAlias for all the exported attributes, which makes it impossible to distinguish:

Instead the jmxtrans-agent should append the attribute and compositeDataKey (if applicable) to the resultName to make it self-explanatory. E.g. :

my.localmachine.Verbose false 1432742044
my.localmachine.ObjectPendingFinalizationCount 0 1432742044
my.localmachine.HeapMemoryUsage.committed 257425408 1432742044
my.localmachine.HeapMemoryUsage.init 268435456 1432742044
my.localmachine.HeapMemoryUsage.max 3817865216 1432742044
my.localmachine.HeapMemoryUsage.used 79746712 1432742044
my.localmachine.NonHeapMemoryUsage.committed 50003968 1432742044
my.localmachine.NonHeapMemoryUsage.init 24576000 1432742044
my.localmachine.NonHeapMemoryUsage.max 136314880 1432742044
my.localmachine.NonHeapMemoryUsage.used 49385432 1432742044
my.localmachine.ObjectName java.lang:type=Memory 1432742044

Yes. The expressionLanguageEngine is available, but it does not allow us to add attribute or compositeData key to the resultName. It simple enhances the existing resultAlias.

Sponsored by: Lookout, Inc.

Provide bundled configuration files

To make it easier for users to get started with jmxtrans-agent it might be helpful to provide some bundled configuration files. That would make it possible to test jmxtrans-agent without having to go through the hassle creating a configuration file. I can also imagine that for some use cases these bundled files would be good enough.

I'm thinking a configuration that polls the typical MBeans on a vanilla HotSpot JVM like memory, GC, threads, CPU, ...

We could bundle configuration files for the most common output writers and use property expansion with reasonable defaults to allow users to configure the specifics with system properties or a properties file.

Thoughts?

Support for different collect intervals

We would like to collect some of our metrics more frequently than others. Hence we would like support for specifying different collect intervals.

If we can agree on how it should work we are prepared to implement it. The most simple solution I see is to allow for an override on query and invocation elements:

<query objectName="java.lang:type=Threading" attribute="ThreadCount" resultAlias="jvm.thread"
  collectIntervalInSeconds="10"/>

This override would be used when specified. The default would be used for queries which do not have an override.

Another option would to add support for groups of queries/invocations where some configuration parameters could be overridden on group level. Something like:

<queries>
  <group name="foo">
    <query ... />
    <invocation ... />
    <collectIntervalInSeconds>10</collectIntervalInSeconds>
  </group>
  <group name="bar">
  ...
  </group>
  <!-- defaults -->
  <collectIntervalInSeconds>20</collectIntervalInSeconds>
</queries>

While I can see the benefits of groups in overriding other parameters (e.g. distinct output writers per group), it is a larger change which adds a bit of complexity.

Any thoughts on this?

Add support for Zabbix Discovery queries

DiscoveryQuery : Used to discover a list of JMX beans matching a specific naming pattern.
Used with the Zabbix server.

For example, the following discovery rule :

<discoveryQuery 
   objectName="java.lang:type=GarbageCollector,name=*" 
   attributes="name,type" 
   resultAlias="discovery[garbageCollector]" 
   collectIntervalInSeconds="300"/>

<query objectName="java.lang:type=GarbageCollector,name=*"
   attributes="CollectionTime,CollectionCount"
   resultAlias="discovery[GarbageCollector.%name%.#attribute#]" />

May yield the following discovery output (formatted for readability) :

{"data":[
  {"{#NAME}":"PS Scavenge","{#TYPE}":"GarbageCollector"},
  {"{#NAME}":"PS MarkSweep","{#TYPE}":"GarbageCollector"}
]}

On the Zabbix side, create a "Discovery Rule" of type "Zabbix trapper"
with a "Key" that matches the result alias. You can then create "Item prototypes" that use the values.

Sample Zabbix configuration that matches the example above :

Discovery rule :
Name : Discover Garbage Collectors
Key : discovery[garbageCollector]

Item Prototype
Name : Object {#TYPE} named {#NAME}
Key : discovery[{#TYPE}.{#NAME}.CollectionTime]

Item Prototype
Name : Object {#TYPE} named {#NAME}
Key : discovery[{#TYPE}.{#NAME}.CollectionCount]

Dynamic reload of configuration

It would be awesome if the configuration could be dynamically reloaded. It would make the agent much more flexible in long-running applications.

I suggest adding a <reloadConfigurationOnChanges>true/false</reloadConfigurationOnChanges> element. Defaults to false.

Enabling this would re-initialize the JmxTransExporter when changes are detected in the configuration file.

I can see some issues where stuff like output writer connections are not cleaned up properly, but that should not be too hard to fix.

If you agree that it would be a desired feature we can look into implementing it.

JDK9 - Fails with java.sql is not being available at boot class path

JMXtrans Java Agent fails with JDK9:

Exception in thread "main" java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:500)
        at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:512)
Caused by: java.lang.NoClassDefFoundError: java/sql/Timestamp
        at org.jmxtrans.agent.util.logging.Logger.log(Logger.java:105)
        at org.jmxtrans.agent.util.logging.Logger.log(Logger.java:98)
        at org.jmxtrans.agent.util.logging.Logger.info(Logger.java:124)
        at org.jmxtrans.agent.JmxTransAgent.initializeAgent(JmxTransAgent.java:116)
        at org.jmxtrans.agent.JmxTransAgent.premain(JmxTransAgent.java:85)
        ... 6 more
Caused by: java.lang.ClassNotFoundException: java.sql.Timestamp
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:185)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:496)
        ... 11 more
FATAL ERROR in native method: processing of -javaagent failed

Add support for statsd gauge bucket in statsd output writer

Statsd allows for collection of stats in a few different data types: https://github.com/etsy/statsd/blob/master/docs/metric_types.md

I propose adding a bucketType field to the statsd output writer to allow configuring these different types of buckets - particularly the GAUGE type.

Is there any reason this hasn't been implemented? I can see in the code it is hardcoded to COUNT type. One issue I foresee is you inherently have the same bucketType for all queries, which will likely not be the case ie. you are collecting gauges and times but you've configured your output for a gauge.

I have implemented and semi-tested the GAUGE type so far.

Support for statsd metric tagging

It would be useful to be able to use statsd tags in metrics (see the Metric Tagging section here), for example:

<query objectName="..." attribute="..." resultAlias="metric-name#tag=value"/>

Currently this will be rejected by jmxtrans-agent's expression engine:

Invalid expression 'metric-name#tag=value', no ending '#' after beginning '#' at position ...

Typo in XML configuration results in lots of log noise

I added jmxtrans-agent to a Java process (a Confluence install) and I had as part of the configuration the following small typo in the configuration:

<jmxtrans-agent>
  <queries>
    <!-- 8< snip 8< -->
    <query objectname="java.nio:type=BufferPool,name=*" attribute="TotalCapacity" resultAlias="jvm.buffers.%name%.capacity"/>
  </queries>
  <!-- 8< snip 8< -->
</jmxtrans-agent>

i.e. I had objectname instead of objectName. This resulted in lots of log noise like the following:

2017-06-08 10:05:36.505 WARNING [jmxtrans-agent-1] org.jmxtrans.agent.Query - Failed to fetch attribute for 'Standalone:j2eeType=Servlet,WebModule=//localhost/,name=status-servlet,J2EEApplication=none,J2EEServer=none'#TotalCapacity, exception:  Cannot find attribute TotalCapacity for StandardEngine[Standalone].StandardHost[localhost].StandardContext[].StandardWrapper[status-servlet]
2017-06-08 10:05:36.505 WARNING [jmxtrans-agent-1] org.jmxtrans.agent.Query - Failed to fetch attribute for 'Standalone:type=MBeanFactory'#TotalCapacity, exception:  Cannot find attribute TotalCapacity for org.apache.catalina.mbeans.MBeanFactory@516ed85c
2017-06-08 10:05:36.505 WARNING [jmxtrans-agent-1] org.jmxtrans.agent.Query - Failed to fetch attribute for 'Confluence:name=MailTaskQueue'#TotalCapacity, exception: getAttribute failed: ModelMBeanAttributeInfo not found for TotalCapacity
2017-06-08 10:05:36.505 WARNING [jmxtrans-agent-1] org.jmxtrans.agent.Query - Failed to fetch attribute for 'Standalone:j2eeType=Servlet,WebModule=//localhost/,name=default,J2EEApplication=none,J2EEServer=none'#TotalCapacity, exception:  Cannot find attribute TotalCapacity for StandardEngine[Standalone].StandardHost[localhost].StandardContext[].StandardWrapper[default]

As it tries to find the TotalCapacity attribute for every bean. This gets repeated every collectIntervalInSeconds and rapidly fills the logs up.

Is there a need where the objectName attribute is ever optional? Or is it possible to have some XML validation performed on the configuration file so any unknown attributes cause an error/exception?

activemq query exampl

Hello mate,

This tool is awsome in our use case where we retrieving metric data from application jvm and feed into graphite.

Just wondering if you could provide us an example for querying ActiveMQ? such as listing every queue sizes (Enqueue count?) .

Thanks again

James

Refactoring directory structure

Currently all the files/modules are located in top package level agent directory.

@cyrille-leclerc let me know if you are ok if I restructure the modules into separate directories? It will really help with manageability.

E.g.
com.jmxtrans.agent

  • util - contains utility (as-is)
  • outputWriter - contains abstract & derived writers
  • resultNameStrategy - contains result name transform code (as-is)
  • agent, agent builder, query, invocations, etc. code resides in the top level dir

Add support for Zabbix Output Writer

I'm a big fan of jmxtrans-agent & Zabbix, so I implemented an output writer that supports the Zabbix protocol :

<outputWriter class="org.jmxtrans.agent.zabbix.ZabbixTcpOutputWriter">
    <host>zabbixserver</host>
    <port>10051</port>
    <serverName>myhostname</serverName>
</outputWriter> 

Hoping to integrate this change in the master branch.

single line per collection rolling file writer?

I'm considering writing a single line per collection rolling file write in the format:

key1=value1, key2=value2, key3=value3 . . .

This allows for improved parsing by Splunk (http://dev.splunk.com/view/logging-best-practices/SP-CAAADP6) and possibly by other (file based) log collectors.

Would you like this writer contributed? If so, do you have a preferred Java class name?

or would you possibly like to change the formatting to single line per collection for:

org\jmxtrans\agent\FileOverwriterOutputWriter.java
org\jmxtrans\agent\RollingFileOutputWriter.java

Please comment. Thanks

Expression language for result alias

Introduce expression language with:

  • Functions (e.g. #hostname#)
  • ObjectName properties (e.g. %name%)

Docs: https://github.com/jmxtrans/jmxtrans-agent/wiki/Expression-Language

Sample:

<query objectName="Catalina:type=DataSource,context=/,host=localhost,class=javax.sql.DataSource,name=*" 
   attribute="numActive" resultAlias="tomcat.datasource.%name%.numActive" />

Should match ObjectName:

Catalina:type=DataSource,context=/,host=localhost,class=javax.sql.DataSource,name="jdbc/my-datasource"

And resultAlias should be:

tomcat.datasource.jdbc_my_datasource.numActive

Add Multi Column Values to influxDB

I use JMX-Agent GraphitePlainTextTcpOutputWriter.java writing values into influxDB, but I find just insert one column, and I see the Graphite demands the Message Format is "metric_path value timestamp\n". So how could I insert multi column into influxDB.

LibratoWriter includes descriptive User-Agent

Copy of jmxtrans/embedded-jmxtrans#85

@mheffner said

I'm from the Librato team. It would help us to include a descriptive User-Agent header in the LibratoWriter that includes "jmxtrans" and the version number of jmxtrans, eg. "jmxtrans/1.2.3". This would assist in our debugging when working with users that are sending data with jmxtrans to Librato.
We have a full write-up here: http://support.metrics.librato.com/knowledgebase/articles/175825-how-to-set-a-custom-user-agent-when-integrating-wi

Support for custom properties per query (InfluxDB)

InfluxDB has a concept of multiple fields and tags per measurement for storing time series metrics in an optimized way.

The current implementation of InfluxDbOutputWriter is not taking advantage of that, it is only possible to create one field value per metric with fixed tags defined at the outputWriter declaration.

But it is not a limitation imposed by the outputWriter but at the query API.

I'd like to propose a more flexible but generic way to have more properties per query, so we could expand the implementation options in the outputWriters so not only InfluxDbOutputWriter would benefit from it. The ideia would be something like adding a propert list inside query definition as follows:

<query objectName="sample:instance=*,name=rollingCount,type=business_counters" attribute="value" resultAlias="business_counters">
 <properties>
   <property name="host" type="tag" value="${HOSTNAME}"/>
   <property name="%instance%" type="field" value="%attribute%">
 </properties>
</query>

Does it make sense to you guys? I'd like to hear your thoughts.

StatsDOutputWriter - Given data too big for the buffer size

jmxtrans-agent-1.2.4.jar StatsDOutputWriter is indefinitelly failing with "java.nio.BufferOverflowException". It seems error was triggered by random failed DNS resolution, that caused some buffer (lim=91) to fill up.

2017-03-07 03:19:29.614 SEVERE [jmxtrans-agent-1] org.jmxtrans.agent.StatsDOutputWriter - Could not send stat java.nio.HeapByteBuffer[pos=0 lim=91 cap=1024] to host telegraf:8125
java.io.IOException: Target address not resolved
        at sun.nio.ch.DatagramChannelImpl.send(DatagramChannelImpl.java:425)
        at org.jmxtrans.agent.StatsDOutputWriter.flush(StatsDOutputWriter.java:157)
        at org.jmxtrans.agent.StatsDOutputWriter.doSend(StatsDOutputWriter.java:128)
        at org.jmxtrans.agent.StatsDOutputWriter.writeQueryResult(StatsDOutputWriter.java:118)
        at org.jmxtrans.agent.OutputWriterCircuitBreakerDecorator.writeQueryResult(OutputWriterCircuitBreakerDecorator.java:90)
        at org.jmxtrans.agent.Query.processAttributeValue(Query.java:274)
        at org.jmxtrans.agent.Query.collectAndExportAttribute(Query.java:239)
        at org.jmxtrans.agent.Query.collectAndExportForObjectName(Query.java:173)
        at org.jmxtrans.agent.Query.collectAndExport(Query.java:167)
        at org.jmxtrans.agent.TimeTrackingCollector.collectIfEnoughTimeHasPassed(TimeTrackingCollector.java:54)
        at org.jmxtrans.agent.JmxTransExporter.collectAndExport(JmxTransExporter.java:209)
        at org.jmxtrans.agent.JmxTransExporter$2.run(JmxTransExporter.java:131)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)

2017-03-07 03:19:29.614 WARNING [jmxtrans-agent-1] org.jmxtrans.agent.StatsDOutputWriter - Given data too big (98bytes) for the buffer size (91bytes), skip it: jmxtransagents.xx...

2017-03-07 03:19:29.615 WARNING [jmxtrans-agent-1] org.jmxtrans.agent.Query - Exception collecting java.lang:type=OperatingSystem#MaxFileDescriptorCount
java.nio.BufferOverflowException
        at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:183)
        at java.nio.ByteBuffer.put(ByteBuffer.java:832)
        at org.jmxtrans.agent.StatsDOutputWriter.doSend(StatsDOutputWriter.java:136)
        at org.jmxtrans.agent.StatsDOutputWriter.writeQueryResult(StatsDOutputWriter.java:118)
        at org.jmxtrans.agent.OutputWriterCircuitBreakerDecorator.writeQueryResult(OutputWriterCircuitBreakerDecorator.java:90)
        at org.jmxtrans.agent.Query.processAttributeValue(Query.java:274)
        at org.jmxtrans.agent.Query.collectAndExportAttribute(Query.java:239)
        at org.jmxtrans.agent.Query.collectAndExportForObjectName(Query.java:173)
        at org.jmxtrans.agent.Query.collectAndExport(Query.java:167)
        at org.jmxtrans.agent.TimeTrackingCollector.collectIfEnoughTimeHasPassed(TimeTrackingCollector.java:54)
        at org.jmxtrans.agent.JmxTransExporter.collectAndExport(JmxTransExporter.java:209)
        at org.jmxtrans.agent.JmxTransExporter$2.run(JmxTransExporter.java:131)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)

...

2017-03-07 08:26:33.896 WARNING [jmxtrans-agent-1] org.jmxtrans.agent.StatsDOutputWriter - Given data too big (104bytes) for the buffer size (91bytes), skip it: jmxtransagents.xx...

2017-03-07 08:26:33.896 WARNING [jmxtrans-agent-1] org.jmxtrans.agent.Query - Exception collecting java.lang:type=OperatingSystem#SystemCpuLoad
java.nio.BufferOverflowException
        at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:183)
        at java.nio.ByteBuffer.put(ByteBuffer.java:832)
        at org.jmxtrans.agent.StatsDOutputWriter.doSend(StatsDOutputWriter.java:136)
        at org.jmxtrans.agent.StatsDOutputWriter.writeQueryResult(StatsDOutputWriter.java:118)
        at org.jmxtrans.agent.OutputWriterCircuitBreakerDecorator.writeQueryResult(OutputWriterCircuitBreakerDecorator.java:90)
        at org.jmxtrans.agent.Query.processAttributeValue(Query.java:274)
        at org.jmxtrans.agent.Query.collectAndExportAttribute(Query.java:239)
        at org.jmxtrans.agent.Query.collectAndExportForObjectName(Query.java:173)
        at org.jmxtrans.agent.Query.collectAndExport(Query.java:167)
        at org.jmxtrans.agent.TimeTrackingCollector.collectIfEnoughTimeHasPassed(TimeTrackingCollector.java:54)
        at org.jmxtrans.agent.JmxTransExporter.collectAndExport(JmxTransExporter.java:209)
        at org.jmxtrans.agent.JmxTransExporter$2.run(JmxTransExporter.java:131)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)

LibratoMetricsIntegrationTest fails when building current master

When building the current master branch the build fails on LibratoMetricsIntegrationTest. Both tests in that class fail in the @Before method with the following error:

java.lang.AssertionError: File src/test/resources/my-librato-config.properties is missing. See librato-config.template.properties for details
    at org.junit.Assert.fail(Assert.java:88)
    at org.junit.Assert.assertTrue(Assert.java:41)
    at org.junit.Assert.assertNotNull(Assert.java:621)
    at org.jmxtrans.agent.LibratoMetricsIntegrationTest.before(LibratoMetricsIntegrationTest.java:48)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
        ....

In librato-config.template.properties it says:

Create a file my-librato-config.properties with the following properties

Am I expected to do that myself to get the build to pass? If so, I suggest changing that behavior so that the project can be built from source without modifications.

Export everything if the query does not define attribute or key to collec

As discussed in pull request #29 by @cyrille-leclerc and @petermodzelewski, creating an issue to allow support for:

Adding more power for XML configuration

  • adding support for ALL attributes, when no attribute is specified
  • adding support for ALL keys, when no key is specified for CompositTypes

Following changes were made as a result:

  • Allow null attribute
  • Method getResultName() in ResultNameStrategy was modified to pass attributeName, because query.getAttributeName() could be null
  • Added tests for covering wildcard tests.

It will allow support for following configurations:

<queries>                     
    <query/>
</queries>
<queries>                     
    <query objectName="java.lang:type=Memory"/>
</queries>

Sponsored by: Lookout, Inc.

Problem with springboot java and jmxtrans-agent

Hello,

I'm trying to add jmxtrans-agent to internal java with springboot application.

It's started with :

/opt/jdk1.8/bin/java -javaagent:/servers/jmxtrans-agent/jmxtrans-agent.jar=/servers/jmxtrans-agent/jmxtrans-agent.xml -Dinfluxdb_instance=dots-server_01 -Dorg.jmxtrans.agent.JmxTransAgent.diagnostic=true -Dorg.jmxtrans.agent.util.logging.Logger.level=TRACE -Djavax.net.ssl.trustStore=/etc/digiplug/cacerts -Djavax.net.ssl.trustStorePassword=changeit -jar /servers/dgpapps/dots/dots-server.jar --slot.id=01

It correctly generates some point and finishes with a stack :

sept. 19 16:27:56  java[79309]: 2017-09-19 16:27:56.965 WARNING [jmxtrans-agent-1] org.jmxtrans.agent.JmxTransExporter - Ignore exception flushing metrics
sept. 19 16:27:56  java[79309]: java.net.SocketTimeoutException: Connect timed out
sept. 19 16:27:56  java[79309]: at java.net.SocksSocketImpl.readSocksReply(SocksSocketImpl.java:126)
sept. 19 16:27:56  java[79309]: at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:502)
sept. 19 16:27:56  java[79309]: at java.net.Socket.connect(Socket.java:589)
sept. 19 16:27:56  java[79309]: at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
sept. 19 16:27:56  java[79309]: at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
sept. 19 16:27:56  java[79309]: at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
sept. 19 16:27:56  java[79309]: at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
sept. 19 16:27:56  java[79309]: at sun.net.www.http.HttpClient.New(HttpClient.java:339)
sept. 19 16:27:56  java[79309]: at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1209)
sept. 19 16:27:56  java[79309]: at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1157)
sept. 19 16:27:56  java[79309]: at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032)
sept. 19 16:27:56  java[79309]: at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:966)
sept. 19 16:27:56  java[79309]: at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1316)
sept. 19 16:27:56  java[79309]: at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1291)
sept. 19 16:27:56  java[79309]: at org.jmxtrans.agent.influxdb.InfluxDbOutputWriter.writeMetrics(InfluxDbOutputWriter.java:192)
sept. 19 16:27:56  java[79309]: at org.jmxtrans.agent.influxdb.InfluxDbOutputWriter.sendMetrics(InfluxDbOutputWriter.java:160)
sept. 19 16:27:56  java[79309]: at org.jmxtrans.agent.influxdb.InfluxDbOutputWriter.sendMetrics(InfluxDbOutputWriter.java:153)
sept. 19 16:27:56  java[79309]: at org.jmxtrans.agent.influxdb.InfluxDbOutputWriter.postCollect(InfluxDbOutputWriter.java:147)
sept. 19 16:27:56  java[79309]: at org.jmxtrans.agent.OutputWriterCircuitBreakerDecorator.postCollect(OutputWriterCircuitBreakerDecorator.java:124)
sept. 19 16:27:56  java[79309]: at org.jmxtrans.agent.OutputWritersChain.postCollect(OutputWritersChain.java:70)
sept. 19 16:27:56  java[79309]: at org.jmxtrans.agent.JmxTransExporter.collectAndExport(JmxTransExporter.java:214)
sept. 19 16:27:56  java[79309]: at org.jmxtrans.agent.JmxTransExporter$2.run(JmxTransExporter.java:131)
sept. 19 16:27:56  java[79309]: at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
sept. 19 16:27:56  java[79309]: at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
sept. 19 16:27:56  java[79309]: at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
sept. 19 16:27:56  java[79309]: at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
sept. 19 16:27:56  java[79309]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
sept. 19 16:27:56  java[79309]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
sept. 19 16:27:56  java[79309]: at java.lang.Thread.run(Thread.java:748)

My xml :

<?xml version="1.0" encoding="UTF-8"?>
<jmxtrans-agent>
    <queries>
        <query objectName="java.lang:type=Threading"               attributes="DaemonThreadCount,ThreadCount,PeakThreadCount" resultAlias="#attribute#" />
        <query objectName="java.lang:type=Runtime"                 attribute="Uptime"                                         resultAlias="Uptime" />
        <query objectName="java.lang:type=OperatingSystem"         attributes="OpenFileDescriptorCount,ProcessCpuTime"        resultAlias="#attribute#" />
        <query objectName="java.lang:type=ClassLoading"            attribute="LoadedClassCount"                               resultAlias="LoadedClassCount" />
        <query objectName="java.lang:type=Compilation"             attribute="TotalCompilationTime"                           resultAlias="TotalCompilationTime" />
        <query objectName="java.lang:type=GarbageCollector,name=*" attributes="CollectionCount,CollectionTime"                resultAlias="%name%.#attribute#" />
        <query objectName="java.lang:type=Memory"                  attributes="HeapMemoryUsage,NonHeapMemoryUsage"            resultAlias="#attribute#.#key#" />
        <query objectName="java.lang:type=MemoryPool,name=*"       attributes="Usage,PeakUsage"                               resultAlias="%name%.#attribute#.#key#" />

        <query objectName="Catalina:type=GlobalRequestProcessor,name=*" attributes="bytesReceived,bytesSent,requestCount,processingTime" resultAlias="%name%.#attribute#" />
        <query objectName="Catalina:type=RequestProcessor,name=*"       attributes="bytesReceived,bytesSent,requestCount,processingTime" resultAlias="%name%.#attribute#" />
        <query objectName="Catalina:type=ThreadPool,name=*"             attribute="currentThreadCount,currentThreadsBusy"                resultAlias="%name%.#attribute#" />
        <query objectName="Catalina:type=Manager,context=*,host=*" 	attribute="activeSessions"           				 resultAlias="activeSessions" />
    </queries>

    <outputWriter class="org.jmxtrans.agent.influxdb.InfluxDbOutputWriter">
        <url>http://xx.yy.zz.aa:8086</url>
        <database>java_db</database>
        <tags>host=#hostname#,instance=${influxdb_instance}</tags>
        <reloadConfigurationCheckIntervalInSeconds>60</reloadConfigurationCheckIntervalInSeconds>
        <connectTimeoutMillis>3000</connectTimeoutMillis>
        <readTimeoutMillis>5000</readTimeoutMillis>
    </outputWriter>

    <outputWriter class="org.jmxtrans.agent.ConsoleOutputWriter">
        <enabled>false</enabled>
    </outputWriter>

    <collectIntervalInSeconds>10</collectIntervalInSeconds>

</jmxtrans-agent>

On the same server, I have a logstash that export correctly with the same xml and jar. I can graph it.

I see the "Ignore exception flushing metrics" exception at https://github.com/jmxtrans/jmxtrans-agent/blob/master/src/main/java/org/jmxtrans/agent/JmxTransExporter.java

But, I have no idea what it may mean.

I tried to debug by myself, but couldn't progress further.

Gerard.

Config reload does not report new metrics

hey there,

This is the config file we use:

<jmxtrans-agent>
    <queries>
        <query objectName="Catalina:type=GlobalRequestProcessor,name=*" attributes="bytesReceived,bytesSent,errorCount,processingTime,requestCount" resultAlias="tomcat.%name%.#attribute#"/>
        <query objectName="Catalina:type=ThreadPool,name=*" attributes="currentThreadCount,currentThreadsBusy,maxThreads" resultAlias="tomcat.%name%.#attribute#"/>
        <query objectName="java.lang:type=ClassLoading" attributes="LoadedClassCount,TotalLoadedClassCount,UnloadedClassCount" resultAlias="tomcat.ClassLoading.#attribute#"/>
        <query objectName="java.lang:type=Compilation" attribute="TotalCompilationTime" resultAlias="tomcat.Compilation.TotalCompilationTime"/>
        <query objectName="java.lang:type=GarbageCollector,name=*" attributes="CollectionCount,CollectionTime" resultAlias="tomcat.%name%.#attribute#"/>
        <query objectName="java.lang:type=Memory" attributes="HeapMemoryUsage,NonHeapMemoryUsage" resultAlias="jvm.#attribute#.#key#"/>
        <query objectName="java.lang:type=Memory" attributes="ObjectPendingFinalizationCount" resultAlias="tomcat.Memory.ObjectPendingFinalizationCount"/>
        <query objectName="java.lang:type=MemoryPool,name=*" attribute="Usage" resultAlias="tomcat.%name%.Usage.#key#"/>
        <query objectName="java.lang:type=OperatingSystem" attributes="MaxFileDescriptorCount,OpenFileDescriptorCount,ProcessCpuLoad,SystemCpuLoad,SystemLoadAverage" resultAlias="tomcat.OperatingSystem.#attribute#"/>
        <query objectName="java.lang:type=Runtime" attribute="Uptime" resultAlias="tomcat.Runtime.Uptime"/>
        <query objectName="java.lang:type=Threading" attributes="DaemonThreadCount,PeakThreadCount,ThreadCount,TotalStartedThreadCount" resultAlias="tomcat.Threading.#attribute#"/>
        <query objectName="java.nio:type=*,name=*" attributes="Count,TotalCapacity,MemoryUsed" resultAlias="tomcat.%type%.%name%.#attribute#"/>
        <query objectName="com.zaxxer.hikari:type=*" resultAlias="hikari.%type%.#attribute#"/>
    </queries>
    <outputWriter class="org.jmxtrans.agent.GraphitePlainTextTcpOutputWriter">
        <host>XXX</host>
        <port>2003</port>
        <namePrefix>jmx.undefined-env.undefined-service.#hostname#.</namePrefix>
    </outputWriter>
    <collectIntervalInSeconds>60</collectIntervalInSeconds>
    <reloadConfigurationCheckIntervalInSeconds>30</reloadConfigurationCheckIntervalInSeconds>
</jmxtrans-agent>

We use this script as a cronjob to update the config:

#!/bin/sh
cd /opt/jmxtrans-config
git reset --hard
git pull

sed -i.bak \
    -e "s#<namePrefix>.*#<namePrefix>jmx.${ECS_ENVIRONMENT}.${SERVICENAME}.${HOSTNAME}.</namePrefix>#" \
    /opt/jmxtrans-config/jmxtrans-agent.xml

touch /opt/jmxtrans-config/jmxtrans-agent.xml

Problem is that the config file does not get reloaded or is not adding new queries to be reported to our influxdb. Re-deploying the service makes reporting work. The config file definitely gets updated by the cronjob.

What could be the problem here?

Ability to exclude/filter attributes

In our applications, we use the Codahale/Dropwizard metrics library to publish our metrics to JMX. Rather than configure each metric, I'd like to grab everything under a particular domain. For example, something like this:

<query objectName="MyApplication-metrics:type=*,group=*,name=*"
       resultAlias="app.%type%.%group%.%name%.#attribute#"/>

However, because of the way the metrics are published to JMX, I end up with metrics that aren't really metrics. For example:

myapplication.app.API.ALL_statusSeries_5xx.responseTime.RateUnit events/second 1505500740
myapplication.app.API.ALL_statusSeries_5xx.responseTime.DurationUnit milliseconds 1505500740

What I'd really like to do is specify a set of attributes not to include. The configuration might look something like this:

<query objectName="MyApplication-metrics:type=*,group=*,name=*"
       excludeAttributes="RateUnit,DurationUnit"
       resultAlias="app.%type%.%group%.%name%.#attribute#"/>

I'm guessing that some might want to do something similar when processing CompositeData, though I don't personally have that need at the moment.

Thanks!

Add ElasticSearchOutputWriter

Follow-up on the email thread "Elastichsearch output writer strategies" initiated by @zepouet .

https://groups.google.com/forum/#!topic/jmxtrans/jIyYctTW2go

It would be nice to have an ElasticSearchOutputWriter.

ElasticSearch Data Structure

  • Question: One unique Index or rolling Indexes suffixing the index by the date? If we store all the metrics in the same ElasticSearch Index, it will be harder to implement a retention policy purging the old metrics.
    • @evgeniy-khist has decided to on his ElasticSearchOutputWriter have a configurable index name capable of using SimpleDateFormat to add a temporal element in the index name. Default value jmxtrans-%{yyyy.MM.dd}. See here
  • Question: What should be the data structure? A nested/document-oriented data structure or a flat structure?
    • Question: JMXTrans heavily relies on "." characters in metric names, usually to provide grouping levels. How do we do with ElasticSearch that sees "." as document nesting instructions.
    • @evgeniy-khist has decided to have very small documents (@timestamp, metricName, metricValue). See here
    • jmxtrans has decided to also have one document per collected metric but also stores the metrics meta data here

ElasticSearch Client and Protocol

  • Question: We don't want jmxtrans-agent to have dependencies that would mess with the classpath, is it reasonable to bypass standard elasticsearch java client and implement a basic HTTP POST call to send the metrics? We will probably also need to create the index mapping.

Basic function support in outputwriter configuration

I've configured my outputWriter to:

    <outputWriter class="org.jmxtrans.agent.GraphitePlainTextTcpOutputWriter">
        <host>graphite.domain.tld</host>
        <namePrefix>server.#escaped_canonical_hostname#.${jmxtrans_application_name:}</namePrefix>
    </outputWriter>

but what I receive is:

server.#escaped_canonical_hostname#.activemq.CurrentConnectionsCount 3 1457966635

Are basic functions supposed to work in the outputWriter config?

mbeans as Arrays

so I have an mBean in a proprietary app that has a value of long[60], When I poll this with jmxtrans I get the whole array, which is not ideal, but ok. Using the jmxtrans-agent the output to the graphite writer looks like:

query.test [J@6987d334 1375372603

on a side note, sending one value of the array would be super for both.

Adding filter name="http-bio-8080" instead of "*" to the query in JMXtransagent.xml to obtain metrics from Globalrequestprocessor mbean doesn't give any data

I want to get the processingTime metrics for only HTTP connector from the Globalrequestprocessor mbean. Hence I had given the query in jmxtransagent.xml like

query objectName="Catalina:type=GlobalRequestProcessor,name=http-nio-8080" attribute="processingTime" resultAlias="tomcat.processingTime"

instead of

query objectName="Catalina:type=GlobalRequestProcessor,name=*" attribute="processingTime"
resultAlias="tomcat.processingTime"

I am not getting any data related to processingTime in the outputwriter(influxdb, console or fileoutputwriter).

Can anyone let me know what wrong am I doing ?

jmxtrans-agent crashes when graphite receiver disappears

I have jmxtrans-agent sending graphite to influxdb, but if the influxdb server ever goes away or is switched out, jmxtrans-agent crashes and has to be restarted to start sending data again. It would be nice if it tried reconnecting and possibly even buffering data until it could send graphite data again.

Retry connection

We have a DNS endpoint that contains multiple IP addresses to our KairosDB servers. These servers can change at any given time, but will update DNS as new ones come online.
Taking Cassandra as an example, when cassandra starts it checks the dns entry and gets an IP address and uses that to connect jmxtrans-agent. If that IP becomes no longer available, I get a "broken pipe" and it retries that same IP forever. Is it possible to have it try the connection over (playing by the TTL rules of DNS and getting an updated IP address in the process?
I don't mind forking and adding this in myself, just wanted to make sure there wasn't a flag or anything I could set.

Thanks!

Collection Interval

Found it in the code, but thought it might be useful for the docs since i have my graphite resolution at 1 minute

60

thanks for doing an amazing job with this project.

Min/avg/max aggregation?

The use case is as follows:

When polling values it is easy to miss spikes for things such as CPU utilization or memory usage. The obvious fix when using polling is to poll more frequently. However, as jmxtrans-agent is built now, that also means metrics are sent to the backend more frequently, which results in increased load and storage requirements.

Hence it would be useful to be able to poll values frequently (think every second) but only send a max/min/avg/median aggregate more rarely.

While the functionality would be useful I'm not sure it fits within jmxtrans-agent. Other solutions would be to use statsd (although that introduces another moving part). Another solution would to instead build a separate agent that exposes new MBeans that keep a rolling min/max/avg and query those beans from jmxtrans-agent.

What are your thoughts on this? Should we look into implementing this in jmxtrans-agent or should we try to limit the scope to what it currently does? @cyrille-leclerc do you have any thoughts?

Agent fails to load configuration from opaque classpath resource

The io.resource framework introduced in jmxtrans-agent 1.2.3 causes a load-time failure if the classpath URI is not hierarchical. For example:

java -javaagent:org.jmxtrans.agent.jmxtrans-agent-1.2.3.jar=classpath:jmxtrans-config.xml ...
SEVERE [main] org.jmxtrans.agent.JmxTransAgent - Exception loading JmxTransExporter from 'classpath:jmxtrans-config.xml'
java.lang.IllegalArgumentException: URI is not hierarchical
    at java.io.File.<init>(File.java:418)
    at org.jmxtrans.agent.util.io.ClasspathResource.getFile(ClasspathResource.java:68)
    at org.jmxtrans.agent.util.io.IoUtils.getFileAsDocument(IoUtils.java:153)
    at org.jmxtrans.agent.JmxTransConfigurationXmlLoader.loadConfiguration(JmxTransConfigurationXmlLoader.java:80)
    at org.jmxtrans.agent.JmxTransExporter.loadNewConfiguration(JmxTransExporter.java:74)
    at org.jmxtrans.agent.JmxTransExporter.<init>(JmxTransExporter.java:70)
    at org.jmxtrans.agent.JmxTransAgent.initializeAgent(JmxTransAgent.java:99)
    at org.jmxtrans.agent.JmxTransAgent.premain(JmxTransAgent.java:84)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
    at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401)

Question: 2 Query constructors assume attribute name as the resultAlias

Following 2 constructors for Query assume resultAlias as the attributeName.

This works fine if we are polling limited set of attributes and attribute names are self-explanatory. However, with the #34 we are allowing capability to export all the attributes and that IMO makes this assumption unnecessary.

If user does not specify the resultAlias, then we can rely on ResultNameStrategy to generate the appropriate name comprised of <objectname>.<attribute-name>.

    /**
     * @see #Query(String, String, String, Integer, String, String, ResultNameStrategy)
     */
    public Query(@Nonnull String objectName, @Nullable String attribute, @Nonnull ResultNameStrategy resultNameStrategy) {
        this(objectName, attribute, null, null, null, attribute, resultNameStrategy);
    }

    /**
     * @see #Query(String, String, String, Integer, String, String, ResultNameStrategy)
     */
    public Query(@Nonnull String objectName, @Nullable String attribute, int position, @Nonnull ResultNameStrategy resultNameStrategy) {
        this(objectName, attribute, null, position, null, attribute, resultNameStrategy);
    }

Sponsored by: Lookout, Inc.

Is this possible to use wildcard search?

For example, I am querying activemq and want to have a list of queues back, and in graphite, I want to see them in their respective treeview folders with meaningful hierarchy.
is this possible to do with jmxtrans-agent?

Regards

RollingFileOutputWriter

Hey,
I wrote a RollingFileOutputWriter. It acts very much like a log4j log. It has the following parameters:

  • RollingFileOutputWriter
  • <fileName> is the file you want to write to.
  • <maxFileSize> is the maximium size in mb before file rolls over. It is optional. Default value is 10mb and maximum file size is 10mb.
  • <maxBackupIndex> is the number of files you want to keep. It is an optional value that defaults to 5. When a file becomes larger the <maxFileSize> it then becomes <fileName>.1 this continues up until <maxbackupIndex> (default <fileName>.4). Much like log4j rolling file appendender

Here is a sample config:

   <outputWriter class="org.jmxtrans.agent.RollingFileOutputWriter">
      <fileName>rollingJMXOutputFile</fileName>
      <maxFileSize>10</maxFileSize>
      <maxBackupIndex>4</maxBackupIndex>
   </outputWriter>

Sample output looks like below. Time is in GMT:

[May 16, 2014 1:19:44 PM] os.systemLoadAverage 1.35400390625
[May 16, 2014 1:19:44 PM] jvm.heapMemoryUsage.used 246736032
[May 16, 2014 1:19:44 PM] jvm.heapMemoryUsage.committed 389021696
[May 16, 2014 1:19:44 PM] jvm.nonHeapMemoryUsage.used 30183280
[May 16, 2014 1:19:44 PM] jvm.nonHeapMemoryUsage.committed 30867456
[May 16, 2014 1:19:44 PM] jvm.loadedClasses 4161
[May 16, 2014 1:19:44 PM] jvm.thread 25
[May 16, 2014 1:19:44 PM] tomcat.requestCount 1
[May 16, 2014 1:19:44 PM] tomcat.errorCount 0
[May 16, 2014 1:19:44 PM] tomcat.processingTime 261
[May 16, 2014 1:19:44 PM] tomcat.bytesSent 47142
[May 16, 2014 1:19:44 PM] tomcat.bytesReceived 0
[May 16, 2014 1:19:44 PM] application.activeSessions 0

My company deploys this with every tomcat instance deployed in AWS then consume the logs using AWS CloudWatch, its working very well. It requires java 1.7 though since I uses java.nio.File (but that can be changed to work with 1.6). Let me know if you want a pull request, I just have to undo a change int he FileOverwriter that prints the timestamp.

Float values causes agent to crash when sent via InfluxDbOutputWriter

Hello Guys,

I encountered a problem with the InfluxDbOutputWriter. The floating point metrics, for example :
java.lang:type=OperatingSystem > ProcessCpuLoad
causes jmxtrans-agent to crash.

Try this :
<query objectName="java.lang:type=OperatingSystem" attribute="ProcessCpuLoad" resultAlias="ProcessCpuLoad"/>

You will obtain :
WARNING [jmxtrans-agent-1] org.jmxtrans.agent.JmxTransExporter - Ignore exception flushing metrics java.lang.RuntimeException: Failed to write metrics, response code: 400, response message: Bad Request ...

My environment :
jmxtrans-agent : 1.2.4
InfluxDB : 1.1
Tomcat : 7.0.73
JRE : Oracle 1.8.0_72-b15

Thank you for your help

Configuring jmxtrans-agent.xml with jvm MemoryPool beans

Hi,

I am trying to send some beans related to the memory pool, but I do not get them, so my guess is I am doing something wrong in the jmxtrans-agent config file (other beans does received): The beans are in the form of:

where XXX is one of {"Eden Space", "Survivor Space", "Tenured Gen" ... }

My guess is I am doing something wrong with 'objectName' value. Does anyone see the problem?

Thanks.

writeQueryResult method update

This is an enhancement request.

It would be so much easier to further develop the API if the OutputWriter interface method

    /**
     * @param metricName
     * @param metricType see {@link Query#type}
     * @param value
     * @throws IOException
     */
    void writeQueryResult(@Nonnull String metricName, @Nullable String metricType, @Nullable Object value) throws IOException;

Could be replaced with

   /**
     * @param metricName
     * @param queryResult see {@link QueryResult}
     * @param value
     * @throws IOException
     */
    void writeQueryResult(@Nonnull String metricName, @Nullable QueryResult queryResult) throws IOException;

In that case the backward incompatible changes to the API (such as including extra fields, which are required by the new Writers) would be so much easier to implement.

Happy to assist with rewriting.

Namespacing conversion

I was wondering if converting all the dots in a bean name (using %name% in the configuration) being transformed to underscores is a specific design decision?

At this line adding the optional second parameter to false would keep the dots.

It made querying Graphite unpractical since names were really long instead of having easy namespaces, for example service.http-server.akka-http-server_200_count instead of service.http-server.akka-http-server.200.count

Edit: here are the type of beans I am currently exporting:

screen shot 2017-07-31 at 15 23 47

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.