Snap plugin intended to process logs using regular expressions.

1. Getting Started

• System Requirements
• Installation
• Configuration and Usage

2. Documentation

• Examples
• Roadmap

3. Community Support
4. Contributing
5. License
6. Acknowledgements

You can get the pre-built binaries for your OS and architecture from the plugin's GitHub Releases page. Download the plugin from the latest release and load it into snapteld (/opt/snap/plugins is the default location for Snap packages).

Fork https://github.com/intelsdi-x/snap-plugin-processor-logs-regexp

Clone repo into $GOPATH/src/github/intelsdi-x/:

$ git clone https://github.com/<yourGithubID>/snap-plugin-processor-logs-regexp

Build the plugin by running make in repo:

$ make

This builds the plugin in ./build

• Set up the Snap framework

The intention of this plugin is to parse logs according to the form defined by the user in task manifest.

The plugin processes log line using regular expressions with names. These names indicate tags which are added to metric. The regular expression can contain special names (timestamp and message) to replace metric timestamp and value.

The plugin can be configured by following parameters (all parameters are optional):

• regexp_log - regular expression with method to parse log line, special names:
  • message - the main message in log which will be set as a metric value
  • timestamp - the log timestamp which will be set as a metric timestamp.
• regexp_message - regular expression with method to parse the main message in log (indicated by message).
• regexp_time - regular expression with method to parse the log timestamp (indicated by timestamp), this expression needs to contain following names: day, month, year, hour, seconds, minutes, seconds, timezone.

Notice: Special characters in regular expressions needs to be escaped.

On default plugin processes following log line (metric value which is returned by collector plugin):

127.0.0.1 - - [07/Dec/2016:06:00:12 -0500] "GET /v3/users/fa2b2986c200431b8119035d4a47d420/projects HTTP/1.1" 200 446 21747 "-" "python-keystoneclient"

to Snap's metric with:

• Data: "GET /v3/users/fa2b2986c200431b8119035d4a47d420/projects HTTP/1.1" 200 446 21747 "-" "python-keystoneclient"
• Timestamp: 07/Dec/2016:06:00:12 -0500
• Tags:
  • client_ip: 127.0.0.1
  • http_method:GET
  • http_url: /v3/users/fa2b2986c200431b8119035d4a47d420/projects
  • http_version: 1.1
  • http_status: 200
  • http_response_size: 446
  • http_response_time: 21747

This is an example running snap-plugin-collector-logs, processing collected regexp-logs and writing post-processed data to a file. It is assumed that you are using the latest Snap binary and plugins.

In one terminal window, open the Snap daemon (in this case with logging set to 1 and trust disabled) with appropriate configuration needed by logs collector. To do that properly, please follow the instruction on snap-plugin-collector-logs.

$ snapteld -l 1 -t 0 --config config.json

In another terminal window:

Download and load plugins:

$ wget http://snap.ci.snap-telemetry.io/plugins/snap-plugin-collector-logs/latest/linux/x86_64/snap-plugin-collector-log
$ wget http://snap.ci.snap-telemetry.io/plugins/snap-plugin-processor-logs-regexp/latest/linux/x86_64/snap-plugin-processor-logs-regexp
$ wget http://snap.ci.snap-telemetry.io/plugins/snap-plugin-publisher-file/latest/linux/x86_64/snap-plugin-publisher-file
$ snaptel plugin load snap-plugin-collector-logs
$ snaptel plugin load snap-plugin-processor-logs-regexp
$ snaptel plugin load snap-plugin-publisher-file

Create a task manifest - see examplary task manifests in examples/tasks:

{
  "version": 1,
  "schedule": {
    "type": "simple",
    "interval": "15s"
  },
  "workflow": {
    "collect": {
      "metrics": {
        "/intel/logs/*": {}
      },
      "process": [
        {
          "plugin_name": "logs-regexp",
          "config": {
            "regexp_log": "(?P<client_ip>\\S+) (\\S{1,}) (\\S{1,}) [[](?P<timestamp>\\d{2}[/]\\S+[/]\\d{4}[:]\\d{2}[:]\\d{2}[:]\\d{2} \\S\\d+)[]] (?P<message>.*)",
            "regexp_message": "(?P<http_method>[A-Z]{3,}) (?P<http_url>/\\S*) HTTP/(?P<http_version>\\d+.\\d+)\" (?P<http_status>\\d*) (?P<http_response_size>\\S*) (?P<http_response_time>\\S*)",
            "regexp_time" : "(?P<day>\\d{2})/(?P<month>[a-zA-Z]+)/(?P<year>\\d{4}):(?P<hour>\\d{2}):(?P<minutes>\\d{2}):(?P<seconds>\\d{2}) (?P<timezone>.\\d+)"
          },
          "process": null,
          "publish": [
            {
              "plugin_name": "file",
              "config": {
                "file": "/tmp/published_logs_with_config.log"
              }
            }
          ]
        }
      ]
    }
  }
}

Create a task:

$ snaptel task create -t task-config.json

To stop task:

$ snaptel task stop <task_id>

There isn't a current roadmap for this plugin, but it is in active development. As we launch this plugin, we do not have any outstanding requirements for the next release.

If you have a feature request, please add it as an issue and/or submit a pull request.

This repository is one of many plugins in Snap, a powerful telemetry framework. See the full project at http://github.com/intelsdi-x/snap To reach out to other users, head to the main framework

We love contributions!

There's more than one way to give back, from examples to blogs to code updates. See our recommended process in CONTRIBUTING.md.

Snap, along with this plugin, is an Open Source software released under the Apache 2.0 License.

• Author: Katarzyna Kujawa

And thank you! Your contribution, through code and participation, is incredibly important to us.