Metrics integration about cm_ext HOT 8 OPEN

I'm also asking this question: see
http://community.cloudera.com/t5/Cloudera-Manager-Installation/Process-monitoring-for-a-custom-service-in-Cloudera-Manager/m-p/37150
for the latest of what we have heard from Cloudera

from cm_ext.

@ujjwal @cmacdonald Looks like KAFKA got an .mdl file.

from cm_ext.

@alxn thanks. their .kdl file is not where one might expect it, i.e. its in src/main/resources, rather than src/descriptor. The referenced counters are defined in Kafka through Codahale, but I'm not sure how the CSD can calculate the JMX name.

Clearly the Java source now included in the CSD is related somehow - any ideas?

from cm_ext.

The java source in the KAFKA csd generates the final src/descriptor/service.mdl. I cannot get the KAFKA generator to run, as it relies on a com.cloudera.csd package that is apparently not open source.

On the other hand, the generated service.mdl for KAFKA can be found in a Cloudera distribution:

unzip -c  /usr/share/cmf/csd/KAFKA-5.7.0.jar descriptor/service.mdl | less
<snip>
"metricDefinitions" : [ {
      "context" : "kafka.controller.KafkaController::ActiveControllerCount::value",
      "counter" : false,
      "description" : "Will be 1 if this broker is the active controller, 0 otherwise",
      "label" : "Active Controller",
      "name" : "kafka_active_controller",
      "numeratorUnit" : "controller"
    }

I believe the context attribute to be some form of pattern for a JMX query. HDFS has more complex formats, e.g.:

{
      "context" : "REGEX$$Hadoop:service=DataNode,name=DataNodeActivity::BlockReportsAvgTime",
      "counter" : false,
      "description" : "Block Reports Average Time",
      "label" : "Block Reports Average Time",
      "name" : "hdfs_block_reports_avg_time",
      "numeratorUnit" : "ms",
      "weightingMetricName" : "hdfs_block_reports_num_ops"
    }

from cm_ext.

@cmacdonald They did update the wiki with the SMDL format... but I didn't notice the repo:

<repository>
      <id>cdh.releases.repo</id>
      <url>http://maven.jenkins.cloudera.com:8081/artifactory/libs-release-local</url>
      <name>Releases Repository</name>
    </repository>

That's brutal...

from cm_ext.

The repo not accessible.

I also think the documentation has been there a while, but the crucial "context" parameter was not documented.

from cm_ext.

After more investigation, I can confirm that "context" does not entirely refer to something that can be directly obtained over JMX....

In fact, metrics appears to work is as follows

• the Cloudera Manager agent process contacts the process over an HTTP connection, e.g. http://namenode:50070/jmx
• the exact URL definition and how to parse the output is customised in the agent using an adaptor for each service.

Until the generic adaptor of Cloudera Manager has more functionality by default ( in file /usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.7.0-py2.7.egg/cmf/monitor/generic/adapter.py ), no useful statistics can be distributed in the a custom CSD. Cloudera Manager is hard-coded to the services they currently support (see /usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.7.0-py2.7.egg/cmf/monitor/generic/adapter_factory.py).

from cm_ext.

Same thing for Kafka. curl -H "Content-Type: application/json" http://localhost:24042/api/metrics | jq . gives you the Kafka metrics in JSON that is slightly massaged Yammer(Codahale) metrics.

Then this script monitor/generic/kafka_adapters.py reads the JSON and ingest the metric.

CDH Kafka uses the 3rd party metric exporter class. This class is loaded by the Kafka Broker, and runs the HTTP server on a separate thread.

from cm_ext.