Comments (8)
Can you show us an example JSON file and tell us which program is generating this?
We are trying to minimize the processing on the collector so this is probably something we will not implement except there are good reasons.
from collector.
I think I have the same feature request.
Currently we are using log2gelf to send messages already stored in JSON format from varnish. The varnishncsa daemon can be configured to store one message per line, so we already formatted it as GELF, we just need something to pick them up and send them to graylog reliably (when graylog is down, queue them, resume where it left last time etc)
The problem with current solution is that it uses more CPU than varnish itself... and needs to be manually restarted every time graylog goes down.
I don't see how this feature adds more processing time, since the message is already formatted as it should, it just needs to be somehow defined as json file format and attributes need to be correctly detected.
Thank you!
from collector.
Since Graylog 1.2.0, we ship a JSON Extractor which can be used to "expand" a JSON payload received as plaintext into a proper GELF message. You can read in the varnish log file with the Graylog Collector and use a JSON Extractor on the GELF TCP/UDP input.
from collector.
OK, thank you, I will try this solution, but I still think this should be the collector's job, this way we would avoid unnecessary processing power on the server.
from collector.
We designed the Graylog Collector exactly the other way round. The goal was to do as little preprocessing on the client machines as possible and move that complexity into the Graylog server nodes.
from collector.
I tried the solution, it works great, thank you!
from collector.
I'm struggling with this like crazy. I just want to send in some flat json to Graylog and have it fill in GELF info by extractor. I can almost send data in but it's like its discarding it as I get no recent message errors from the Extract page when clicking load message.
The documentation really is lacking here. A full fledged example of how to send JSON directly into Graylog would be extremely useful.
I'm trying to send AWS CloudTrail logs directly from S3 into Graylog (via Lambda). No, I'm not using the plugin which assumes a horribly cost inefficient model of CloudTrail -> SNS -> SQS <- Graylog (via pull).
from collector.
@et304383 We are using GitHub issues for tracking bugs in Graylog itself, but this doesn't look like one. Please post this issue to our discussion forum or join the #graylog channel on freenode IRC.
Thank you!
from collector.
Related Issues (20)
- Ship a log4j2 configuration file
- Allow collector internal logs to be shipped via GELF
- Allow multiple outputs HOT 2
- Grok is not working about input file with content-splitter=PATTERN HOT 1
- Collector cant handle File deletions
- Collector stops working on Windows server 2k12 with exception HOT 4
- Graylog Collector - Unable to read event HOT 1
- NEWLINE content-splitter does not work on IBM1047 charset files HOT 1
- Add functionality to track log file position HOT 1
- FILTER / GROK support on client side HOT 1
- Collector registration/heartbeat failing with Graylog 2.0.0 Alpha 5 HOT 5
- Graylog Collector: collector.utils.CollectorId - No node ID file found
- secret S3 access-key on github? HOT 1
- Open source windows binaries HOT 4
- Windows eventlog not working HOT 2
- Graylog collector is not working with graylog server 2.0
- release .deb for latest release HOT 1
- Unquoted service path privilege escalation vulnerability HOT 3
- Collector is not notified by the change or creation of log files HOT 1
- No messages after collector config HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from collector.