snap-plugin-publisher-elasticsearch
This plugin supports publishing metrics to Elasticsearch endpoint
It is used in the [Snap framework] (http://github.com/intelsdi-x/snap).
Getting Started
System Requirements
- golang 1.6+ (needed only for building)
- Elasticsearch instance to publish data (docker run elasticsearch)
Installation
To build the plugin binary:
Fork https://github.com/intelsdi-x/snap-plugin-publisher-elasticsearch
Clone repo into $GOPATH/src/github.com/intelsdi-x/
:
$ git clone https://github.com/<yourGithubID>/snap-plugin-publisher-elasticsearch.git
Build the plugin by running make within the cloned repo:
$ make
This builds the plugin in ./build/
Configuration and Usage
- Set up the Snap framework
Documentation
The plugin expects you to provide following parameters:
protocol
http, https, tlsaddress
address of elasticsearch endpointport
elasticsearch portindex
index for publishing datatype
mapping type
You can also set following options if needed:
index_timestamp
default is set totrue
(boolean). Set to false if not want to extend index prefix with timestamppublish_fields
default is set toNamespace|Config|Data|Description|Timestamp|Unit|Version
(string). Remove field from metric structure which shouldn't be published.
Plugin publishes dynamic elements from metric namespace.
Examples
You can find exemplary task manifests for using this plugin in folder examples.
Example running logs collector plugin, logs-regexp processor, logs-openstack processor, and writing data to elasticsearch.
Prepare docker with elasticsearch
$ docker pull elasticsearch
$ docker run --name elasticsearch -p 9200:9200 -p 9300:9300 elasticsearch
In one terminal window, open the Snap daemon:
$ snapteld -t 0 -l 1 -m 999
The option "-l 1" it is for setting the debugging log level, "-t 0" is for disabling plugin signing, and "-m 999" is for the maximum number of instances of a loaded plugin to run.
In another terminal window:
Download and load collector, processor and publisher plugins:
$ wget http://snap.ci.snap-telemetry.io/plugins/snap-plugin-collector-logs/latest/linux/x86_64/snap-plugin-collector-logs
$ wget http://snap.ci.snap-telemetry.io/plugins/snap-plugin-processor-logs-regexp/latest/linux/x86_64/snap-plugin-processor-logs-regexp
$ wget http://snap.ci.snap-telemetry.io/plugins/snap-plugin-processor-logs-openstack/latest/linux/x86_64/snap-plugin-processor-logs-openstack
$ wget http://snap.ci.snap-telemetry.io/plugins/snap-plugin-publisher-elasticsearch/latest/linux/x86_64/snap-plugin-publisher-elasticsearch
$ snaptel plugin load snap-plugin-collector-logs
$ snaptel plugin load snap-plugin-processor-logs-regexp
$ snaptel plugin load snap-plugin-processor-logs-openstack
$ snaptel plugin load snap-plugin-publisher-elasticsearch
Find logs to collect or use exemplary logs available in logs.
Create task manifests, see examples:
- to use snap-plugin-collector-logs, snap-plugin-processor-logs-regexp, snap-plugin-publisher-elasticsearch - logs-regexp-elasticsearch.json
- to use snap-plugin-collector-logs, snap-plugin-processor-logs-openstack, snap-plugin-publisher-elasticsearch - logs-openstack-elasticsearch.json
If you are using exemplary logs you need to modify "log_dir"
in task manifests and set appropriate paths to directories with logs.
If you want to collect your own logs you need to adjust config sections in task manifests to your environment.
Create tasks:
$ snaptel task create -t logs-regexp-elasticsearch.json
$ snaptel task create -t logs-openstack-elasticsearch.json
See running tasks:
$ snaptel task list
Check indexes in elasticsearch:
$ curl 'http://localhost:9200/_cat/indices?v'
See data saved in elasticsearch:
$ curl -XGET 'http://localhost:9200/snap*/message/_search?pretty=true&size=9999'
Exemplary data saved in elasticsearch has the form which is shown in outputs.
Roadmap
If you have a feature request, please add it as an issue and/or submit a pull request.
Community Support
This repository is one of many plugins in the Snap, a powerful telemetry agent framework. See the full project at http://github.com/intelsdi-x/snap. To reach out to other users, head to the main framework.
Contributing
We love contributions! ๐
There is more than one way to give back, from examples to blogs to code updates. See our recommended process in CONTRIBUTING.md.
License
Snap, along with this plugin, is an Open Source software released under the Apache 2.0 License.
Acknowledgements
- Author: Patryk Matyjasek
And thank you! Your contribution, through code and participation, is incredibly important to us.