Snap plugin intended to process logs using regular expressions.
You can get the pre-built binaries for your OS and architecture from the plugin's GitHub Releases page.
Download the plugin from the latest release and load it into snapteld
(/opt/snap/plugins
is the default location for Snap packages).
Fork https://github.com/intelsdi-x/snap-plugin-processor-logs-regexp
Clone repo into $GOPATH/src/github/intelsdi-x/
:
$ git clone https://github.com/<yourGithubID>/snap-plugin-processor-logs-regexp
Build the plugin by running make in repo:
$ make
This builds the plugin in ./build
- Set up the Snap framework
The intention of this plugin is to parse logs according to the form defined by the user in task manifest.
The plugin processes log line using regular expressions with names. These names indicate tags which are added to metric.
The regular expression can contain special names (timestamp
and message
) to replace metric timestamp and value.
The plugin can be configured by following parameters (all parameters are optional):
regexp_log
- regular expression with method to parse log line, special names:message
- the main message in log which will be set as a metric valuetimestamp
- the log timestamp which will be set as a metric timestamp.
regexp_message
- regular expression with method to parse the main message in log (indicated bymessage
).regexp_time
- regular expression with method to parse the log timestamp (indicated bytimestamp
), this expression needs to contain following names:day
,month
,year
,hour
,seconds
,minutes
,seconds
,timezone
.
Notice: Special characters in regular expressions needs to be escaped.
On default plugin processes following log line (metric value which is returned by collector plugin):
127.0.0.1 - - [07/Dec/2016:06:00:12 -0500] "GET /v3/users/fa2b2986c200431b8119035d4a47d420/projects HTTP/1.1" 200 446 21747 "-" "python-keystoneclient"
to Snap's metric with:
- Data:
"GET /v3/users/fa2b2986c200431b8119035d4a47d420/projects HTTP/1.1" 200 446 21747 "-" "python-keystoneclient"
- Timestamp:
07/Dec/2016:06:00:12 -0500
- Tags:
client_ip
:127.0.0.1
http_method
:GET
http_url
:/v3/users/fa2b2986c200431b8119035d4a47d420/projects
http_version
:1.1
http_status
:200
http_response_size
:446
http_response_time
:21747
This is an example running snap-plugin-collector-logs, processing collected regexp-logs and writing post-processed data to a file. It is assumed that you are using the latest Snap binary and plugins.
In one terminal window, open the Snap daemon (in this case with logging set to 1 and trust disabled) with appropriate configuration needed by logs collector. To do that properly, please follow the instruction on snap-plugin-collector-logs.
$ snapteld -l 1 -t 0 --config config.json
In another terminal window:
Download and load plugins:
$ wget http://snap.ci.snap-telemetry.io/plugins/snap-plugin-collector-logs/latest/linux/x86_64/snap-plugin-collector-log
$ wget http://snap.ci.snap-telemetry.io/plugins/snap-plugin-processor-logs-regexp/latest/linux/x86_64/snap-plugin-processor-logs-regexp
$ wget http://snap.ci.snap-telemetry.io/plugins/snap-plugin-publisher-file/latest/linux/x86_64/snap-plugin-publisher-file
$ snaptel plugin load snap-plugin-collector-logs
$ snaptel plugin load snap-plugin-processor-logs-regexp
$ snaptel plugin load snap-plugin-publisher-file
Create a task manifest - see examplary task manifests in examples/tasks:
{
"version": 1,
"schedule": {
"type": "simple",
"interval": "15s"
},
"workflow": {
"collect": {
"metrics": {
"/intel/logs/*": {}
},
"process": [
{
"plugin_name": "logs-regexp",
"config": {
"regexp_log": "(?P<client_ip>\\S+) (\\S{1,}) (\\S{1,}) [[](?P<timestamp>\\d{2}[/]\\S+[/]\\d{4}[:]\\d{2}[:]\\d{2}[:]\\d{2} \\S\\d+)[]] (?P<message>.*)",
"regexp_message": "(?P<http_method>[A-Z]{3,}) (?P<http_url>/\\S*) HTTP/(?P<http_version>\\d+.\\d+)\" (?P<http_status>\\d*) (?P<http_response_size>\\S*) (?P<http_response_time>\\S*)",
"regexp_time" : "(?P<day>\\d{2})/(?P<month>[a-zA-Z]+)/(?P<year>\\d{4}):(?P<hour>\\d{2}):(?P<minutes>\\d{2}):(?P<seconds>\\d{2}) (?P<timezone>.\\d+)"
},
"process": null,
"publish": [
{
"plugin_name": "file",
"config": {
"file": "/tmp/published_logs_with_config.log"
}
}
]
}
]
}
}
}
Create a task:
$ snaptel task create -t task-config.json
To stop task:
$ snaptel task stop <task_id>
There isn't a current roadmap for this plugin, but it is in active development. As we launch this plugin, we do not have any outstanding requirements for the next release.
If you have a feature request, please add it as an issue and/or submit a pull request.
This repository is one of many plugins in Snap, a powerful telemetry framework. See the full project at http://github.com/intelsdi-x/snap To reach out to other users, head to the main framework
We love contributions!
There's more than one way to give back, from examples to blogs to code updates. See our recommended process in CONTRIBUTING.md.
Snap, along with this plugin, is an Open Source software released under the Apache 2.0 License.
- Author: Katarzyna Kujawa
And thank you! Your contribution, through code and participation, is incredibly important to us.