Git Product home page Git Product logo

snap-plugin-collector-elasticsearch's Introduction

DISCONTINUATION OF PROJECT.

This project will no longer be maintained by Intel.

This project has been identified as having known security escapes.

Intel has ceased development and contributions including, but not limited to, maintenance, bug fixes, new releases, or updates, to this project.

Intel no longer accepts patches to this project.

DISCONTINUATION OF PROJECT

This project will no longer be maintained by Intel. Intel will not provide or guarantee development of or support for this project, including but not limited to, maintenance, bug fixes, new releases or updates. Patches to this project are no longer accepted by Intel. If you have an ongoing need to use this project, are interested in independently developing it, or would like to maintain patches for the community, please create your own fork of the project.

Snap collector plugin - Elasticsearch

This plugin collects Elasticsearch cluster and nodes statistics using Snap telemetry framework.

The intention for this plugin is to collect metrics for Elasticsearch nodes and cluster health.

This plugin is used in the [Snap framework] (http://github.com/intelsdi-x/snap).

  1. Getting Started
  1. Documentation
  1. Community Support
  2. Contributing
  3. License
  4. Acknowledgements

Getting Started

In order to use this plugin you need the Elasticsearch cluster that you can collect metrics from.

System Requirements

Operating systems

All OSs currently supported by Snap:

  • Linux/amd64
  • Darwin/amd64

Installation

Install Elasticsearch:

To install sysstat package from the official repositories simply use:

  • For Darwin/amd64: brew install elasticsearch
  • For Docker installation:
Get and run the latest elasticsearch
$ docker pull elasticsearch
Using default tag: latest
latest: Pulling from library/elasticsearch
77e39ee82117: Pull complete 
...
Digest: sha256:c8ca4802e3ee5d6165da4c2158eee14d4dc7f4f39e1edbc62a003d659be35f3c
Status: Downloaded newer image for elasticsearch:latest

$ docker run -d --name snap-elasticsearch -p 9200:9200 -p 9300:9300 elasticsearch
2878aeba854c7f65c52f86b8ccf6f5ebf221cb63ff021d519ffd4572065fd183
Check Elasticsearch cluster health
$ http://DOCKERHOST:9200/_cluster/health?pretty
{
  "cluster_name" : "elasticsearch",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

To build the plugin binary:

Fork https://github.com/intelsdi-x/snap-plugin-collector-elasticsearch

Clone repo into $GOPATH/src/github.com/intelsdi-x/:

$ git clone https://github.com/<yourGithubID>/snap-plugin-collector-elasticsearch.git

Build the Snap elasticsearch plugin by running make within the cloned repo:

$ make

This builds the plugin in ./build/

Builds

You can also download prebuilt binaries for OS X and Linux (64-bit) at the releases page

Configuration and Usage

Documentation

To learn more about this plugin:

Collected Metrics

This plugin can gather Elasticsearch node and cluster level statistics. The node level statistics are similar for each node except that each node has a different node id. To show the statistics for all nodes inside a cluster, using a wildcard * to represent node id in the task manifest. Otherwise, a particular node id may be specified.

In the node level, this plugin collects metrics listed the next catalog.

Metric Name Description
indices Indices stats about size, document count, indexing and deletion times, search times, field cache size, merges and flushes
os Operating system stats, load average, mem, swap
process Process statistics, memory consumption, cpu usage, open file descriptors
jvm JVM stats, memory pool information, garbage collection, buffer pools, number of loaded/unloaded classes
thread_pool Statistics about each thread pool, including current size, queue and rejected tasks
fs File system information, data path, free disk space, read/write stats
http HTTP connection information
breaks Statistics about the field data circuit breaker
script Computing the grades stats based on a script

Examples

Example running snap-plugin-collector-elasticsearch and writing data to a file.

Dockerized example

In one terminal window, open the snap daemon (in this case with logging set to 1 and trust disabled):

$ snapteld -l 1 -t 0

In another terminal window:

Download and load Snap plugins:

$ wget http://snap.ci.snap-telemetry.io/plugins/snap-plugin-collector-elasticsearch/latest/linux/x86_64/snap-plugin-collector-elasticsearch
$ wget http://snap.ci.snap-telemetry.io/plugins/snap-plugin-publisher-file/latest/linux/x86_64/snap-plugin-publisher-file
$ chmod 755 snap-plugin-*
$ snaptel plugin load snap-plugin-collector-elasticsearch
$ snaptel plugin load snap-plugin-publisher-file

See available metrics for your system (this is just part of the list)


$ snaptel metric list
NAMESPACE 									 VERSIONS
/intel/elasticsearch/cluster/active_primary_shards 				 1
/intel/elasticsearch/cluster/active_shards 					 1
/intel/elasticsearch/cluster/active_shards_percent_as_number 			 1
/intel/elasticsearch/cluster/cluster_name 					 1
/intel/elasticsearch/cluster/delayed_unassigned_shards 				 1
/intel/elasticsearch/cluster/number_of_data_nodes 							 1
/intel/elasticsearch/cluster/number_of_in_flight_fetch 							 1
/intel/elasticsearch/cluster/number_of_nodes 								 1
/intel/elasticsearch/cluster/number_of_pending_tasks 							 1
/intel/elasticsearch/cluster/relocating_shards 								 1
/intel/elasticsearch/cluster/status 									 1
/intel/elasticsearch/cluster/task_max_waiting_in_queue_millis 						 1
/intel/elasticsearch/cluster/timed_out 									 1
/intel/elasticsearch/cluster/unassigned_shards 								 1
/intel/elasticsearch/node/*/thread_pool/warmer/largest 				 1
/intel/elasticsearch/node/*/thread_pool/warmer/queue 				 1
/intel/elasticsearch/node/*/thread_pool/warmer/rejected 			 1

Create a task manifest file (e.g. elasticsearch-task.json).

{
    "version": 1,
    "schedule": {
        "type": "simple",
        "interval": "1s"
    },
    "workflow": {
        "collect": {
            "metrics": {
                "/intel/elasticsearch/node/*/timestamp": {},
		        "/intel/elasticsearch/node/*/host": {},
                "/intel/elasticsearch/node/*/indices/merges/total_throttled_time_in_millis": {},
                "/intel/elasticsearch/node/*/jvm/classes/total_loaded_count": {},
                "/intel/elasticsearch/node/*/os/mem/total_in_bytes": {},
                "/intel/elasticsearch/cluster/status": {}
            },
            "config": {
                "/intel/elasticsearch/node": {
                    "server": "192.168.99.100",
                    "port": 9200
                }
            },
            "publish": [
                {
                    "plugin_name": "file",
                    "config": {
                        "file": "/tmp/published_elasticsearch"
                    }
                }
            ]
        }
    }
}

Create task:

$ snaptel task create -t elaticsearch-task.json
Using task manifest to create task
Task created
ID: 5aadafc8-a7a1-427c-892c-87e680235563
Name: Task-5aadafc8-a7a1-427c-892c-87e680235563
State: Running

This data is published to a file /tmp/published_elasticsearch.log per task specification:

48178-05-24 14:55:10 -0800 PST|[intel elasticsearch node F2fP3bedSsqK8S_v440enA os mem total_in_bytes]|1044631552|172.17.0.6
48178-05-24 14:55:10 -0800 PST|[intel elasticsearch node F2fP3bedSsqK8S_v440enA timestamp]|1458196124110|172.17.0.6
2016-03-16 23:28:53.281902987 -0700 PDT|[intel elasticsearch cluster status]|green|egu-mac01.lan
48178-05-24 15:11:51 -0800 PST|[intel elasticsearch node F2fP3bedSsqK8S_v440enA host]|172.17.0.6|172.17.0.6
48178-05-24 15:11:51 -0800 PST|[intel elasticsearch node F2fP3bedSsqK8S_v440enA indices merges total_throttled_time_in_millis]|0|172.17.0.6
48178-05-24 15:11:51 -0800 PST|[intel elasticsearch node F2fP3bedSsqK8S_v440enA jvm classes total_loaded_count]|6636|172.17.0.6
48178-05-24 15:11:51 -0800 PST|[intel elasticsearch node F2fP3bedSsqK8S_v440enA os mem total_in_bytes]|1044631552|172.17.0.6
48178-05-24 15:11:51 -0800 PST|[intel elasticsearch node F2fP3bedSsqK8S_v440enA timestamp]|1458196125111|172.17.0.6
2016-03-16 23:28:54.282481719 -0700 PDT|[intel elasticsearch cluster status]|green|egu-mac01.lan
48178-05-24 15:28:31 -0800 PST|[intel elasticsearch node F2fP3bedSsqK8S_v440enA host]|172.17.0.6|172.17.0.6
48178-05-24 15:28:31 -0800 PST|[intel elasticsearch node F2fP3bedSsqK8S_v440enA indices merges total_throttled_time_in_millis]|0|172.17.0.6

Roadmap

This plugin is still in active development. As we launch this plugin, we have a few items in mind for the next few releases:

  • Additional error handling

If you have a feature request, please add it as an issue and/or submit a pull request.

Community Support

This repository is one of many plugins in the Snap, a powerful telemetry agent framework. See the full project at http://github.com/intelsdi-x/snap. To reach out to other users, head to the main framework.

Contributing

We love contributions!

There is more than one way to give back, from examples to blogs to code updates.

License

Snap, along with this plugin, is an Open Source software released under the Apache 2.0 License.

Acknowledgements

snap-plugin-collector-elasticsearch's People

Contributors

candysmurf avatar izabellaraulin avatar katarzyna-z avatar kindermoumoute avatar marcin-krolik avatar mkleina avatar nanliu avatar nguyenddn avatar rashmigottipati avatar rdower avatar taotod avatar woodsaj avatar zhengqingyuan avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

snap-plugin-collector-elasticsearch's Issues

zero type of plugin.MetricType returned for all collected metrics

steps to reproduce.

  1. start up snap with elasticsearch plugin configured

---
log_level: 1
control:
  plugin_trust_level: 0
  plugins:
    collector:
      elasticsearch:
        all:
          server: 127.0.0.1
          port: 9200
  1. create task to collect metrics
curl -X POST http://localhost:8181/v1/tasks -d '{
  "name": "es-test",
  "start": true,
  "workflow": {
    "collect": {
      "metrics": {
        "/intel/elasticsearch/*": {}
      }
    }
  },
  "schedule": {
    "type": "simple",
    "interval": "10s"
  }
}'
  1. watch task

All data is the zero type of plugin.MetricType

data: {"type":"stream-open","message":"Stream opened"}

data: {"type":"metric-event","message":"","event":[{"namespace":"/","data":null,"timestamp":"0001-01-01T00:00:00Z","tags":{"plugin_running_on":"anthony-desktop"}},
...

Support for elasticsearch 5.0

Snap version (use snapctl -v): 0.17.0

What happened:

The latest elasticsearch update to 5.0 has broken medium/large test. I've pinned all test to version 2.4 per #30.

Steps to reproduce it (as minimally and precisely as possible):

  1. Patch medium test to use elasticsearch 5.0 and new health endpoint:
diff --git a/scripts/medium.sh b/scripts/medium.sh
index 900506e..61f9d7a 100644
--- a/scripts/medium.sh
+++ b/scripts/medium.sh
@@ -4,7 +4,7 @@ set -e
 set -u
 set -o pipefail

-id=$(docker run -d --name elasticsearch -p 9200:9200 -p 9300:9300 elasticsearch:2.4)
+id=$(docker run -d --name elasticsearch -p 9200:9200 -p 9300:9300 elasticsearch:5.0)
 DOCKER_HOST=${DOCKER_HOST-}
 if [[ -z "${DOCKER_HOST}" ]]; then
   SNAP_ELASTICSEARCH_HOST="127.0.0.1"
@@ -16,7 +16,7 @@ export SNAP_ELASTICSEARCH_HOST
 _info "Elasticsearch Host: ${SNAP_ELASTICSEARCH_HOST}"

 _info "Waiting for elasticsearch docker container"
-while ! curl -sG "http://${SNAP_ELASTICSEARCH_HOST}:9200/_status" > /dev/null 2>&1; do
+while ! curl -sG "http://${SNAP_ELASTICSEARCH_HOST}:9200/_cluster/health" > /dev/null 2>&1; do
   sleep 1
   echo -n "."
 done
  1. make test-medium
bash -c "./scripts/test.sh medium"
2016-10-31 17:06:32 UTC [     info] Elasticsearch Host: 127.0.0.1
2016-10-31 17:06:32 UTC [     info] Waiting for elasticsearch docker container
...............
2016-10-31 17:06:49 UTC [     info] running go_test
2016-10-31 17:06:49 UTC [     info] running test type: medium
?   	github.com/intelsdi-x/snap-plugin-collector-elasticsearch	[no test files]
=== RUN   TestESCollectMetrics

  Elasticsearch collector
    collect metrics ✔✘
    collect cluster metrics ✔✔✔✔


Failures:

  * /Users/nanliu/gopath/src/github.com/intelsdi-x/snap-plugin-collector-elasticsearch/elasticsearch/elasticsearch_integration_test.go
  Line 77:
  Expected [] to NOT be empty (but it was)!


6 total assertions

--- FAIL: TestESCollectMetrics (0.12s)
FAIL
coverage: 57.3% of statements
exit status 1
FAIL	github.com/intelsdi-x/snap-plugin-collector-elasticsearch/elasticsearch	0.139s
make: *** [test-medium] Error 1

Anything else do we need to know (e.g. issue happens only occasionally):

Large test on jenkins also appears to fail on 5.0:
https://ci.snap-telemetry.io/job/snap-plugin-collector-elasticsearch-large-prb-k8s/5/console

No Name specified for the wildcard namespace

Snap version (use snapctl -v):
Latest master

What happened:
Hostname was not filtered out when publish metrics to the DB.

What you expected to happen:
Namespace without the hostname published in DB.

Steps to reproduce it (as minimally and precisely as possible):

  1. Publish to Influxdb

Coherence: tests and builds

As a plugin developer, I want to run consistency tests and builds in all my Snap plugin repos:

  1. Run the test with make test-small.
  2. Same with make check.
  3. When building, I want my binary to go in build/$OS/$ARCH/.

NB:
Influxdb plugin has the most updated scripts. For this plugin:
Makefile
scripts/test.sh
scripts/deps.sh
scripts/build.sh
scripts/common.sh

EDIT: may need to change .Travis.yml too.

snap-plugin-collector-elasticsearch will not load with snapctl plugin load

Problem: Built snap-plugin-collector-elasticsearch from latest commit
When attempting to load snap-plugin-collector-elasticsearch into snap with snapctl plugin load I receive the following error:

<029-zwzca:/go/bin# snapctl plugin load snap-plugin-collector-elasticsearch
Error loading plugin:
unexpected EOF

Environment:

  • snap in tribe mode in kubernetes on gce
  • snap plugins loaded:
NAME             VERSION         TYPE            SIGNED          STATUS          LOADED TIME
heapster         1               publisher       false           loaded          Fri, 09 Sep 2016 17:09:50 UTC
docker           8               collector       false           loaded          Fri, 09 Sep 2016 17:09:56 UTC
cassandra        3               collector       false           loaded          Fri, 09 Sep 2016 17:10:01 UTC
passthru         1               processor       false           loaded          Fri, 09 Sep 2016 17:09:41 UTC
influx           15              publisher       false           loaded          Fri, 09 Sep 2016 17:09:46 UTC

METRICS.md file

Could you add "data type" and "description" columns to collected metrics tables ? It would be also nice to have one file with collected metrics like here.

panic during initialization

  * /home/anthony/go/src/github.com/intelsdi-x/snap-plugin-collector-elasticsearch/elasticsearch/elasticsearch_integration_test.go 
  Line 39: - reflect: call of reflect.Value.NumField on zero Value 
  goroutine 6 [running]:
  panic(0x87dbe0, 0xc820165fe0)
    /usr/local/go/src/runtime/panic.go:426 +0x4e9
  reflect.flag.mustBe(0x0, 0x19)
    /usr/local/go/src/reflect/value.go:202 +0xa6
  reflect.Value.NumField(0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/reflect/value.go:1153 +0x2a
  github.com/intelsdi-x/snap-plugin-collector-elasticsearch/elasticsearch.(*ESMetric).parseData(0xc82011b4a0, 0x7acc40, 0xc82015a3b0, 0x196, 0xc82011b338, 0x0, 0x0)
    /home/anthony/go/src/github.com/intelsdi-x/snap-plugin-collector-elasticsearch/elasticsearch/nodeClient.go:128 +0x562

This was tracked down to the "classes" section missing for some nodes in the response from /_nodes.
Is it possible that "classes" is only filled in for the queried node and not other nodes in the cluster?

In any event, missing sections should probably be handled more gracefully then with a panic.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.