Ability to exclude files about logstash-forwarder HOT 25 CLOSED

+1

from logstash-forwarder.

+1

from logstash-forwarder.

+1 (Even though this is over a year old)

Some justification for this would be the log-rotations which are stored as a backup, but shouldn't be transmitted

from logstash-forwarder.

^ +1

from logstash-forwarder.

+1

from logstash-forwarder.

+1, would love to see this, because of the log rotation issue

from logstash-forwarder.

+1
The logstash agent that this is intended to replace supports 'exclude', and it's very useful.

from logstash-forwarder.

Considering that I'll most likely need this functionality in the near future, I've started working on this feature. Hopefully I'll have this completed and tested shortly!

from logstash-forwarder.

I can see your commit and have a suggestion.

Use filepath.Match to check if a file name matches any exclude pattern.
http://golang.org/pkg/path/filepath/#Match

It'll save doing two deep directory listings (which I think glob does) and it'll prevent race condition where a file appears between generating excluded files list and then doing path glob.

from logstash-forwarder.

Thanks for that tip @driskell. I hadn't considered that possibility of a race condition which will most likely happen at some point. I've applied that modification locally and I'll test it shortly before I push the changes.

from logstash-forwarder.

👍

from logstash-forwarder.

👍

from logstash-forwarder.

+1

from logstash-forwarder.

+1

from logstash-forwarder.

👍

from logstash-forwarder.

++

from logstash-forwarder.

Awesome addition, was just looking for this. Eagerly awaiting a merge :)

from logstash-forwarder.

+1

from logstash-forwarder.

👍

from logstash-forwarder.

It's worth noting that some simple exclusions can be done with a glob regex, detailed here https://blog.vladionescu.com/excluding-from-a-path-glob/

from logstash-forwarder.

@rmoff Thanks for that example, I guess it can also be accomplished with that query. Keep in mind if you have files that you don't want indexed or that aren't necessary, then they shouldn't be sent to Logstash to begin with. No point in having unused data being processed by Logstash or stored by Elasticsearch.

from logstash-forwarder.

+1

from logstash-forwarder.

+1

from logstash-forwarder.

big +1

from logstash-forwarder.

Thanks for helping make logstash-forwarder better!

Logstash-forwarder is going away and is replaced by https://github.com/elastic/filebeat/. If this is still an issue with filebeat, would you mind opening a ticket there?

from logstash-forwarder.