A logstash community-driven site for documentation, shared experiences, etc.
♥ Fork and contribute ♥
Community-driven cookbook of extra logstash documentation, tricks, etc.
Home Page: http://cookbook.logstash.net
A logstash community-driven site for documentation, shared experiences, etc.
♥ Fork and contribute ♥
The website is not there. Bummer.
The site http://cookbook.logstash.net (linked from this repo's description) is no longer resolving. It's been removed from DNS.
The logstash agent Cpu usage :
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
27729 logstash 20 0 1643m 221m 16m S 101.1 1.4 0:36.80 java
Thank you for your support!
Hi,
Using logstash on a RHEL 6.4 host, In the logstash.sh script do_stop() function, there's the line:
checkpid $pid && sleep $delay &&
but $delay is not defined and we get:
sleep: missing operand
Try `sleep --help' for more information.
You can check the killproc function in /etc/init.d/functions as an example.
The query '*' resulted the following error:
org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to execute phase [query], total failure; shardFailures {[yOyNYFaTYaQfkKXq7QaRw][mcare][1]: RemoteTransportException[[es-1][inet[/10.1.3.10:9300]][search/phase/query]]; nested: SearchParseException[[mcare][1]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [Failed to parse source [{"from":0,"size":50,"query":{"query_string":{"query":"","default_operator":"and"}},"sort":[{"@timestamp":{"order":"desc"}}]}]]]; nested: SearchParseException[[mcare][1]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }{[03aCVBiST0in9Nvuhpd4Fw][knowledge][3]: RemoteTransportException[[es-1][inet[/10.1.3.9:9300]][search/phase/query]]; nested: SearchParseException[[knowledge][3]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [Failed to parse source [{"from":0,"size":50,"query":{"query_string":{"query":"","default_operator":"and"}},"sort":[{"@timestamp":{"order":"desc"}}]}]]]; nested: SearchParseException[[knowledge][3]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }{[yOyNYFaTYaQfkKXq7QaRw][graylog2][0]: RemoteTransportException[[es-1][inet[/10.1.3.10:9300]][search/phase/query]]; nested: SearchParseException[[graylog2][0]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [Failed to parse source [{"from":0,"size":50,"query":{"query_string":{"query":"","default_operator":"and"}},"sort":[{"@timestamp":{"order":"desc"}}]}]]]; nested: SearchParseException[[graylog2][0]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }{[yOyNYFaTYaQfkKXq7QaRw][mcare][0]: RemoteTransportException[[es-1][inet[/10.1.3.10:9300]][search/phase/query]]; nested: SearchParseException[[mcare][0]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [Failed to parse source [{"from":0,"size":50,"query":{"query_string":{"query":"","default_operator":"and"}},"sort":[{"@timestamp":{"order":"desc"}}]}]]]; nested: SearchParseException[[mcare][0]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }{[03aCVBiST0in9Nvuhpd4Fw][knowledge][1]: RemoteTransportException[[es-1][inet[/10.1.3.9:9300]][search/phase/query]]; nested: SearchParseException[[knowledge][1]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [Failed to parse source [{"from":0,"size":50,"query":{"query_string":{"query":"","default_operator":"and"}},"sort":[{"@timestamp":{"order":"desc"}}]}]]]; nested: SearchParseException[[knowledge][1]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }{[03aCVBiST0in9Nvuhpd4Fw][graylog2][3]: RemoteTransportException[[es-1][inet[/10.1.3.9:9300]][search/phase/query]]; nested: SearchParseException[[graylog2][3]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [Failed to parse source [{"from":0,"size":50,"query":{"query_string":{"query":"","default_operator":"and"}},"sort":[{"@timestamp":{"order":"desc"}}]}]]]; nested: SearchParseException[[graylog2][3]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }{[yOyNYFaTYaQfkKXq7QaRw][graylog2][4]: RemoteTransportException[[es-1][inet[/10.1.3.10:9300]][search/phase/query]]; nested: SearchParseException[[graylog2][4]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [Failed to parse source [{"from":0,"size":50,"query":{"query_string":{"query":"","default_operator":"and"}},"sort":[{"@timestamp":{"order":"desc"}}]}]]]; nested: SearchParseException[[graylog2][4]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }{[yOyNYFaTYaQfkKXq7QaRw][mcare][3]: RemoteTransportException[[es-1][inet[/10.1.3.10:9300]][search/phase/query]]; nested: SearchParseException[[mcare][3]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [Failed to parse source [{"from":0,"size":50,"query":{"query_string":{"query":"","default_operator":"and"}},"sort":[{"@timestamp":{"order":"desc"}}]}]]]; nested: SearchParseException[[mcare][3]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }{[yOyNYFaTYaQfkKXq7QaRw][mcare][4]: RemoteTransportException[[es-1][inet[/10.1.3.10:9300]][search/phase/query]]; nested: SearchParseException[[mcare][4]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [Failed to parse source [{"from":0,"size":50,"query":{"query_string":{"query":"","default_operator":"and"}},"sort":[{"@timestamp":{"order":"desc"}}]}]]]; nested: SearchParseException[[mcare][4]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }{[03aCVBiST0in9Nvuhpd4Fw][knowledge][2]: RemoteTransportException[[es-1][inet[/10.1.3.9:9300]][search/phase/query]]; nested: SearchParseException[[knowledge][2]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [Failed to parse source [{"from":0,"size":50,"query":{"query_string":{"query":"","default_operator":"and"}},"sort":[{"@timestamp":{"order":"desc"}}]}]]]; nested: SearchParseException[[knowledge][2]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }{[yOyNYFaTYaQfkKXq7QaRw][graylog2][1]: RemoteTransportException[[es-1][inet[/10.1.3.10:9300]][search/phase/query]]; nested: SearchParseException[[graylog2][1]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [Failed to parse source [{"from":0,"size":50,"query":{"query_string":{"query":"","default_operator":"and"}},"sort":[{"@timestamp":{"order":"desc"}}]}]]]; nested: SearchParseException[[graylog2][1]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }{[yOyNYFaTYaQfkKXq7QaRw][knowledge][0]: RemoteTransportException[[es-1][inet[/10.1.3.10:9300]][search/phase/query]]; nested: SearchParseException[[knowledge][0]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [Failed to parse source [{"from":0,"size":50,"query":{"query_string":{"query":"","default_operator":"and"}},"sort":[{"@timestamp":{"order":"desc"}}]}]]]; nested: SearchParseException[[knowledge][0]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }{[03aCVBiST0in9Nvuhpd4Fw][knowledge][4]: RemoteTransportException[[es-1][inet[/10.1.3.9:9300]][search/phase/query]]; nested: SearchParseException[[knowledge][4]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [Failed to parse source [{"from":0,"size":50,"query":{"query_string":{"query":"","default_operator":"and"}},"sort":[{"@timestamp":{"order":"desc"}}]}]]]; nested: SearchParseException[[knowledge][4]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }{[yOyNYFaTYaQfkKXq7QaRw][graylog2][2]: RemoteTransportException[[es-1][inet[/10.1.3.10:9300]][search/phase/query]]; nested: SearchParseException[[graylog2][2]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [Failed to parse source [{"from":0,"size":50,"query":{"query_string":{"query":"","default_operator":"and"}},"sort":[{"@timestamp":{"order":"desc"}}]}]]]; nested: SearchParseException[[graylog2][2]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }{[yOyNYFaTYaQfkKXq7QaRw][mcare][2]: RemoteTransportException[[es-1][inet[/10.1.3.10:9300]][search/phase/query]]; nested: SearchParseException[[mcare][2]: query[ConstantScore(NotDeleted(:))],from[0],size[50]: Parse Failure [Failed to parse source [{"from":0,"size":50,"query":{"query_string":{"query":"","default_operator":"and"}},"sort":[{"@timestamp":{"order":"desc"}}]}]]]; nested: SearchParseException[[mcare][2]: query[ConstantScore(NotDeleted(:*))],from[0],size[50]: Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }
There is "Server not found" error from yesterday.
Host lookup results using Google's nameservers:
dig @google-public-dns-a.google.com cookbook.logstash.net
; <<>> DiG 9.7.0-P1 <<>> @google-public-dns-a.google.com cookbook.logstash.net
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 245
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0
;; QUESTION SECTION:
;cookbook.logstash.net. IN A
;; AUTHORITY SECTION:
logstash.net. 979 IN SOA ns1.dreamhost.com. hostmaster.dreamhost.com. 2012110200 20085 1800 1814400 14400
;; Query time: 97 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Sun Nov 4 15:48:46 2012
;; MSG SIZE rcvd: 103
Hi,
timestamp format seems wrong, elasticsearch doesn't like it.
This is the recipe:
LogFormat "{ "@timestamp": "%{%Y/%m/%dT%H:%M:%S%z}t"
but shouldn't it be
LogFormat "{ "@timestamp": "%{%Y-%m-%dT%H:%M:%S%z}t" ?
I like this cookbook, but also would love a working example of a filter allowing to skip all 200 statuses. I tried but couldn't succeed in grepping the status field...
thanks
Instead of code being included in the page, the include_code block is visible
example in http://cookbook.logstash.net/recipes/rsyslog-agent/
configure rsyslog
The rsyslog daemon is useful for both taking local syslog messages as well as for pulling logs from files.
To watch files with rsyslog, you want to use the imfile rsyslog module.
For example, let’s say we want to forward local syslog as well as apache and mysql log files to logstash.
{% include_code rsyslog.conf %}
configure logstash
Now, logstash needs to be told to accept syslog input. This is simple enough. Here is an example config that takes syslog and emits it to stdout:
{% include_code logstash.conf %}
I am facing a, possible, issue with logstash 1.2.1. Sporadically, when starting it up, I receive the following error:
java -jar logstash-1.2.1-flatjar.jar agent -f conf/logstash.conf -- web
----------
Using milestone 2 input plugin 'file'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.2.1/plugin-milestones {:level=>:warn}
+---------------------------------------------------------+
| An unexpected error occurred. This is probably a bug. |
| You can find help with this problem in a few places: |
...
The error reported is:
pattern %{GREEDYDATA:message_id} not defined
My configuration file looks like this:
input {
file {
type => "my-component"
path => [ "/path/to/my/log/directory/*.log" ]
add_field => [ "API", "mycomponent"]
}
...
}
filter {
if [type] == "my-component" {
grok {
match => [ "message", "(%{GREEDYDATA:message_id}) %{TIMESTAMP_ISO8601:log_timestamp} %{LOGLEVEL:loglevel} %{GREEDYDATA:message_remainder}" ]
add_field => ["raw_message", "%{@message}"]
}
mutate {
replace => ["message", "%{message_remainder}" ]
}
multiline {
pattern => "^\s"
what => "previous"
}
}
}
output {
elasticsearch { embedded => true }
}
It works if I kill it and start it again.
if you don't explicitely set it the embedded elasticsearch tries to create it's data dir in the logstash user's dir (which happened for me because I had created the UID with a home dir)
better to explicitly set it in the upstart config IMO
i.e. something like https://github.com/logstash/logstash/wiki/Testing-your-filters but not isolated to filters.
Rather an integrated end-to-end test of loading a custom (output) plugin that provides a set of sample files, specifies inputs, some filters and an output parameters for that plugin.
Ideally it would include stubbing out the backend implementation of the output so that the test can ensure that the filter+output is invoking the right sequence of operations on the backend.
I realise this might be a lot to ask for - portions would still be useful, especially the basics of an integrated test.
Thanks, M.
when :hover on a featurelet button, the border fades in smoothly over 0.5s, but the background gradient snaps from A to B instantly, creating a distracting flash effect.
hi :)
how about changing the whole theme that is matches logstash.net and integrates seamless into it?
i could give it a try if you don't actually want it to look that different and dark ^^
Can someone post here the error messages (if any) that are being generated so I can help research this gh-pages issue
thanks
You may already be aware of this, but cookbook.logstash.net just returns a 404 not found page.
When I follow the instructions here: http://cookbook.logstash.net/recipes/rsyslog-agent/
I get this error:
{"message":"Using experimental plugin 'syslog'. This plugin is untested and may change in the future. For more information about plugin statuses, see http://logstash.net/docs/1.1.1-pre/plugin-status ","level":"warn"}
{"message":"Missing required parameter 'type' for input/syslog","level":"error"}
{"message":"Config validation failed.","level":"error"}
We noticed a problem with our logstash-client whereby it didn't seem to be tailing over 50% of the log files that we'd configured it to watch.
Eventually we tracked it down to a problem in ruby-filewatcher in the watch.rb file in '_discover_file'.
On the first line of that function is a call to 'Dir.glob' which was returning an empty array even for files which we know exist and are accessible by the logstash user.
We're running logstash 1.1.1 (which internally uses the jRuby 1.6.7 interpretor, but on trying out 'Dir.glob' on a local instance of that interpretor we couldn't replicate the problem).
We wrote a hack to help us get around it here -> alphagov/ruby-filewatch@9daaab8 - but presumably there's a cleaner solution to this.
Has anyone else experienced this problem?
Mark
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.