graylog-labs / collector Goto Github PK

View Code? Open in Web Editor NEW

40.0 40.0 17.0 3.32 MB

[DEPRECATED] The Graylog Collector

Home Page: https://www.graylog.org

License: GNU General Public License v3.0

Shell 0.46% Java 97.75% Batchfile 1.78%

collector's People

Contributors

Stargazers

Watchers

Forkers

rickheil sheh1000 howkj1 rationalbean csm monaghans-andrew-munro lgscofield wanghe4096 agolpl kira8565 spitfire1900 rlugojr jiangwh rhadoo tilaka3 kovkolia isabella232

collector's Issues

Support reading in old logfiles

Currently reading starts at the end of any given logfile - it should be possible to read in old data as well.

3rd party package

create msi/exe for windows with java embedded inside if possible or a java-less agent.
due to java insecurity some customer have policy that block java installation.
also create rpm and deb for linux installation

Documentation tasks

This issue collects missing documentation topics. Feel free to submit missing topics via the comments.

Document required Java version. (See #34)
Configuration for multiline messages. (See #21)
TLS configuration for GELF output
Charset configuration for file input
Complete configuration file documentation with all available options for inputs, outputs, and global settings.
Input/Output routing.

GELF messages should contain agent id metadata

Similar to how radio nodes send their input and node id as metadata (gl2_source_radio reserved field) agents should also identify themselves to allow tracing messages back (and allow to group message volumes by agent in a later version).

"dynamic" memory usage - memoryleak?

refering to issue #39
@bernd sry for the late reply

0.2.5 works fine now but with the addional config entry we see the following:

The graylog-collector-service-x64 process uses about 70MB of ram which starts growing about 1MB/s until it reaches ~133MB and falls back to ~70MB.

As mentioned before the additional config entry (now more specific):

 }
  test-mcafee {
    type = "file"
    path = "C:/ProgramData/McAfee/DesktopProtection/UpdateLog.txt"
  }

The logfile looks like that:

03.07.2015  11:02:15    domain\admin.accountname    Task wird gestartet: AutoUpdate
....
03.07.2015  11:02:22    domain\admin.accountname    Aktualisierung beendet

Evaluate izpack for Windows packaging

http://izpack.org/

leave filed name as original

previously i use nxlog to send gelf syslog and it correctly send filed like EventID with filed name "EventID" while graylog collector change this in "event_id" and broke existend correlation rules

Java Error on execute graylog-collector

hi,
i installed the graylog-collector 0.2.2 on an SLED 11, Kernel "3.0.13-0.27-pae" in /usr/share/graylog-collector, updated collector.conf and want to start with "graylog-collector run -f ../config/collector.conf". Immediately i get the following error messages:

Exception in thread "main" java.lang.UnsupportedClassVersionError: org/graylog/collector/cli/Main : Unsup                       ported major.minor version 51.0
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:634)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:277)
        at java.net.URLClassLoader.access$000(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:212)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
Could not find the main class: org.graylog.collector.cli.Main. Program will exit.

On the Linux machine is some Java installed: java-1_6_0-openjdk

What's going wrong? what to do?

graylog-web unresponsive after start collector

Tested on graylog 1.1 beta2 and beta3

After start a collector 0.2.1 (on the same graylog-server or other server), the graylog-web interface stops responding (a graylog-server restart is needed).

The config (very simple):

// Graylog Collector example configuration.

// URL to REST API of Graylog server this collector registers at
server-url = "http://server-name:12900"

// Enable registration with the Graylog server. (enabled by default)
//enable-registration = true

message-buffer-size = 128

// The id used to identify this collector. Can be either a string which is used as id,
// or the location of a file if prefixed with "file:". If the file does not exist,
// an id will be generated and written to that file. If it exists, it is expected
// to contain a single string without spaces which will be used for the id.
// Defaults to "file:config/collector-id" if not specified.
collector-id = "file:config/collector-id"

metrics {
  enable-logging = false
  log-duration = "60s"
}

inputs {
  local-syslog {
    type = "file"
    path = "/var/log/syslog"
    charset = "utf-8"
    content-splitter = "newline"
  }
  test-log {
    type = "file"
    path = "/tmp/test.log"
  }
}

outputs {
  gelf-tcp {
    type = "gelf"
    host = "127.0.0.1"
    port = 12201
    client-tls = false
    client-tls-cert-chain-file = "/path/to/cert-chain.pem"
    client-tls-verify-cert = true
    client-queue-size = 512
    client-connect-timeout = 5000
    client-reconnect-delay = 1000
    client-tcp-no-delay = true
    client-send-buffer-size = 32768
  }
  console {
    type = "stdout"
  }
}

Windows Server: Graylog 0.4.0 stdout leaking

I have installed release 0.4.0 the Graylog collector on a Windows Server 2008 server, after a few days of running we noticed a low disk issue which we tracked back to the .\logs\graylog-collector-stdout file hitting 388GB.

Sadly the file was too large to open and we had to delete it to recover disk space so I am unsure of the contents but this didn't seem like a file that should be reaching this size, or in fact retaining any data.

The server is running several roles, among them is AD DS, File and Print Services, IIS, DHCP and DNS. If there are any other details that I can submit to assist with this please let me know. The server was submitting logs to Graylog and from what I can tell there are no logs which were not forwarded correctly.

Windows service script problem on Windows 2003

There seems to be a problem with the graylog-collector-service.bat script on Windows 2003.

C:\collector\bin>graylog-collector-service.bat install GA
Installing service for Graylog Collector

Service name: "GA"
JAVA_HOME:    "C:\Program Files\Java\jre7\"
ARCH:         "x86"

WARNING: JAVA_HOME points to a JRE and not JDK installation; a client (not
a server) JVM will be used...
[2015-05-27 16:00:35] [error] [ 2796] Unrecognized cmd option
C:\collector\bin\\windows\graylog-collector-service-x86.exe
[2015-05-27 16:00:35] [error] [ 2796] Invalid command line arguments
[2015-05-27 16:00:35] [error] [ 2796] Commons Daemon procrun failed with
exit value: 1 (Failed to parse command line arguments)
ERROR: Failed to install service: GA

C:\collector\bin>

Source: https://groups.google.com/forum/#!msg/graylog2/XxLxcq4PI1A/tRjKW5UshuEJ

Mac OS X has no "readlink -f"

Fix the shell startup script to work on Mac OS X. There is no readlink -f available.

Log the operating system version on startup

This will be helpful for debugging.

Accept JSON as an input to the collector

There is currently no capability to use JSON data as an input - the collector sees the entire JSON string as as single message.

Instead of having to re-parse the JSON'ized message using a Graylog server extractor, provide a capability to add the JSON data as additional fields in the GELF message before sending it to Graylog server.

Support client-side parsing of certain log files

Many logfiles are written in CSV format - given the header of such a file the collector should stuff the correct values into fields and append those to the GELF message.

Perhaps you should even support GROK patterns for arbitrary logs ...

Message from Windows EventLog is cut off.

We have multilie messages, that start with a servicenaem in brackets

Example:

( IB.NODE.01.default ) Ausführungsgruppe wurde mit Konfigurationsnachricht beendet.

An den Broker wird eine Antwort auf einen Befehl gesendet.

Keine Benutzeraktion erforderlich.

The message sent to graylog is just the content inside the brackets -> IB.NODE.01.default

windows collector 0.2.4 log transfer to the server broken?

Windows Graylog Collector 0.2.4 on Windows2008R2std.x64SP1 & Java 1.8.0_45

config:

server-url = "http://server:no default port/"

message-buffer-size = 128

inputs {
 win-eventlog-application {
    type = "windows-eventlog"
    source-name = "Application"
    poll-interval = 1s
  }
  win-eventlog-system {
    type = "windows-eventlog"
    source-name = "System"
    poll-interval = 1s
  }
  win-eventlog-security {
    type = "windows-eventlog"
    source-name = "Security"
    poll-interval = 1s
  }
}

outputs {
  gelf-udp {
    type = "gelf"
    host = "server"
    port = 12204
  }
}

C:\Program Files\graylog-collector-0.2.4\logs\graylog-collector-stderr.yyyy-mm-dd.log

yyyy-mm-dd hh:mm:ss Commons Daemon procrun stderr initialized
Exception in thread "EventLogThread" Exception in thread "EventLogThread" Exception in thread "EventLogThread" java.lang.UnsatisfiedLinkError: org.hyperic.sigar.win32.EventLog.close()V
    at org.hyperic.sigar.win32.EventLog.close(Native Method)
    at org.hyperic.sigar.win32.EventLogThread.run(EventLogThread.java:175)
    at java.lang.Thread.run(Unknown Source)
java.lang.UnsatisfiedLinkError: org.hyperic.sigar.win32.EventLog.close()V
    at org.hyperic.sigar.win32.EventLog.close(Native Method)
    at org.hyperic.sigar.win32.EventLogThread.run(EventLogThread.java:175)
    at java.lang.Thread.run(Unknown Source)
java.lang.UnsatisfiedLinkError: org.hyperic.sigar.win32.EventLog.close()V
    at org.hyperic.sigar.win32.EventLog.close(Native Method)
    at org.hyperic.sigar.win32.EventLogThread.run(EventLogThread.java:175)
    at java.lang.Thread.run(Unknown Source)

We installed the collector via script which now works fine even with whitespaces.
There is a connection openend to the server viewable in netstat but we don't see any logs incoming from the collector.

If we add an additional log file entry, which worked with 0.2.2, there seems to be somewhat like a memoryleak which gets cleaned up by the garbagecollector from time to time.
additional config entry

  test-file {
    type = "file"
    path = "C:/ProgramData/testfolder1/testfolder2/Log.txt"
  }

Collector Does not Pick Up New Files in Grep Path

Using JDK 1.8.0-25 and collector 0.4.1 on Centos 6.5 Configuration here (sensitive details replaced by XXXX).

Steps to reproduce:

Start collector using configuration
Observe message "Configured files for input "tdb-system" do not exist yet. They will be followed once they are created" (Only appears if log directory is empty)
Create files in pertinent paths and append to them
No logs are submitted to Graylog
Restart collector
Any logs appended to files are now submitted to Graylog
Create new file in path and append to it
Logs from new file are not submitted to Graylog

I'm running as a non-root user with read/write access to the log files in question.

EventId from Windows Eventlog is not the same in graylog

Example: In the EventLog the ID ist 7036.
In the corresponding graylog entry I can only find
event_id 1073748860
event_record_number 29773.

hmmm...

Evaluate msitools for Windows packaging

https://wiki.gnome.org/msitools

Graylog Collector on Windows Server 2012 R2

I tried to install the Graylog Collector with the instruction on http://docs.graylog.org/en/1.2/pages/collector.html#windows. I do all the steps. But if i want execute bin\graylog-collector-service.bat install GraylogCollector the Batch dont show all the messages it should show, nothing. After that, there is no Service of Graylog in the ControlPanel. The same for bin\graylog-collector-service.bat start GraylogCollector. I use Graylog Collector 0.4.1 on Windows Server 2012 R2 and tried to install in C:\Programme (x86)\GraylogCollector.
What else must i write?

Support new Windows EventLog API

Windows has a new EventLog API since Vista / Server 2008. The library we are using the read the event log (sigar) does not support the new API yet.

We have to find another library that supports reading the from the new API to be able to read the new eventlog types correctly.

Make sure the eventlog reader can handle manual eventlog deletion/cleaning. The current sigar based reader throws an exception in that case. (2015-09-03T08:17:27.431+0200 ERROR [EventLogThread] sigar.win32.EventLogThread - Unable to read event id 250667: org.hyperic.sigar.win32.Win32Exception: Error reading from the event log: 1503)
Save the current state for each followed eventlog stream and start reading from that position after a restart to avoid losing logs.

documentation: setting reader- or poll-intervall

Here is some question for documentation....
When i start graylog collector, i see for each log file reader-intervall ='100' What this means? seconds, milliseconds?
can i set reading-interval for my logfiles in config/collector.conf?
in colector.conf.example i see in the windows section: poll-interval = 1s. Can i use this also for other log entries in conf file?

Support request: Collector not sending messages

Hi,
I configured it to send messages from text files to remote Graylog instance. Could you please help me diagnose what happens?

My configuration:

server-url = "http://x.x.x:5901"
enable-registration = true
collector-id = "janpoboril@webfaction"
inputs {
  wf-frontend {
    type = "file"
    path-glob-root = "/home/xxx/logs/frontend"
    path-glob-pattern = "*.log"
  }
}
outputs {
  gelf-tcp {
    type = "gelf"
    host = "x.x.x"
    port = 5902
  }
  stdout {
    type = "stdout"
  }
}

Config of input:

recv_buffer_size: 1048576
port: 12201
tls_key_file:
tls_key_password: *******
max_message_size: 2097152
override_source:
bind_address: 0.0.0.0
tls_cert_file:

Port is not same because it is translated by Docker hosting Graylog instance.

Heartbeat is working (I can see collector in Gaylog) and in Inputs I can see active connection. I can see messages in stdout from running collector, but they are not send to gelf-tcp output.

I tried "nc x.x.x 5902" (in same environment with the collector) to send somethink to Graylog input and this messages was received and saved successfully. Everything seems collector is not sending messages to TCP.

Thank you for hint :-)

crash is hostname has hyphen in host name

Exception in thread "main" com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) Error injecting constructor, java.net.UnknownHostException: proxy-xxx: proxy-xxx: Nome o servizio sconosciuto
  at org.graylog.collector.heartbeat.CollectorRegistrationRequestProvider.<init>(CollectorRegistrationRequestProvider.java:29)
  while locating org.graylog.collector.heartbeat.CollectorRegistrationRequestProvider
  while locating org.graylog.collector.heartbeat.CollectorRegistrationRequest
    for parameter 1 at org.graylog.collector.heartbeat.HeartbeatService.<init>(HeartbeatService.java:45)
  while locating org.graylog.collector.heartbeat.HeartbeatService
  while locating com.google.common.util.concurrent.Service annotated with @com.google.inject.multibindings.Element(setName=,uniqueId=13, type=MULTIBINDER, keyType=)
  at org.graylog.collector.guice.CollectorModule.registerService(CollectorModule.java:41) (via modules: org.graylog.collector.buffer.BufferModule -> com.google.inject.multibindings.Multibinder$RealMultibinder)
  while locating java.util.Set<com.google.common.util.concurrent.Service>
    for parameter 0 at org.graylog.collector.services.ServiceManagerProvider.<init>(ServiceManagerProvider.java:33)
  while locating org.graylog.collector.services.ServiceManagerProvider
  while locating com.google.common.util.concurrent.ServiceManager
    for parameter 0 at org.graylog.collector.services.CollectorServiceManager.<init>(CollectorServiceManager.java:31)
  while locating org.graylog.collector.services.CollectorServiceManager

1 error
        at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1025)
        at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1051)
        at org.graylog.collector.cli.commands.Run.run(Run.java:62)
        at org.graylog.collector.cli.Main.main(Main.java:50)
Caused by: java.net.UnknownHostException: proxy-xxx: proxy-xxx: Nome o servizio sconosciuto
        at java.net.InetAddress.getLocalHost(InetAddress.java:1494)
        at org.graylog.collector.heartbeat.CollectorRegistrationRequestProvider.<init>(CollectorRegistrationRequestProvider.java:31)
        at org.graylog.collector.heartbeat.CollectorRegistrationRequestProvider$$FastClassByGuice$$89039544.newInstance(<generated>)
        at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
        at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:61)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:105)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267)
        at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:61)
        at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
        at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:104)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267)
        at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56)
        at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103)
        at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012)
        at com.google.inject.multibindings.Multibinder$RealMultibinder.get(Multibinder.java:375)
        at com.google.inject.multibindings.Multibinder$RealMultibinder.get(Multibinder.java:258)
        at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81)
        at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:53)
        at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:61)
        at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:45)
        at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
        at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:104)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267)
        at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:61)
        at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
        at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:104)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267)
        at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
        at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012)
        ... 3 more
Caused by: java.net.UnknownHostException: proxy-bazoli: Nome o servizio sconosciuto
        at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
        at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:922)
        at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1314)
        at java.net.InetAddress.getLocalHost(InetAddress.java:1490)
        ... 40 more

Change heartbeat-interval setting to duration

The heartbeat-interval setting is currently an integer. Changes this to a duration so users can use 5s or 1h.

Windows file reader file locking/rotation issue

Hello, I'm using Collector 0.4.0 on Windows 2008 R2 SP1 x64, Oracle jre 1.8.0_60 (win,x64). I was able to setup file-based logging for the DHCP service on windows, and it correctly tailed my logs for the first day and I got events in graylog. However when the DHCP service rotated its logs collector stopped reading, and DHCP service was no longer able to write to the next day's logs (they remained timestamped at the previous rotation date and were locked, unable to open with Notepad).

I do a search in process explorer and see the graylog collector with an open file handle on the log files (as expected), so I'm wondering if there's some best practice or other config that can be done with Windows file readers to prevent the lock from interfering with Windows rotating the logs?

I've replicated this on 3 different servers, all with the same configuration. I'm not sure this is an issue with Collector more than an issue with the way Windows DHCP service is handling log file locking, but any help would be appreciated.

This is my configuration below:

inputs {
DHCP-logs {
type = "file"
path-glob-root = "c:\\windows\\system32\\dhcp"
path-glob-pattern = "DhcpSrvLog-*.log"
content-splitter = "NEWLINE"
poll-interval = "1s"
outputs = "gelf-file"
}}
outputs {
gelf-file{
type="gelf"
host = "<my graylog server ip>"
port = 12202
}}

too many files opened / Cannot read modified file

since one week i use the collector to collect logfiles from my processmanagmentsystems (about 70 linux machines) and send to graylog server. the collector ist running on SLES 11.x. i had to log about 60 - 80 logfiles. in my collector.conf there are all files and paths defined, even if they are not exist.
i started the collector by a selfcreatet start/stop file in /etc/init.d/. all output from collector himself is going to a logfile in /var /log.
now it happened, that sometimes some machines get an error like this:

"2015-07-01T07:24:17.520Z,""    at org.graylog.collector.file.FileReaderService$FsChangeListener.pathCreated(FileReaderService.java:180) [graylog-collector.jar:?]"",""x0x4001.nbg"""
"2015-07-01T07:24:17.520Z,""    at org.graylog.collector.file.FileReaderService.followFile(FileReaderService.java:130) ~[graylog-collector.jar:?]"",""x0x4001.nbg"""
"2015-07-01T07:24:17.520Z,""    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91) ~[?:1.7.0_21]"",""x0x4001.nbg"""
"2015-07-01T07:24:17.520Z,""    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.7.0_21]"",""x0x4001.nbg"""
"2015-07-01T07:24:17.521Z,""    at java.nio.channels.AsynchronousFileChannel.open(AsynchronousFileChannel.java:248) ~[?:1.7.0_21]"",""x0x4001.nbg"""
"2015-07-01T07:24:17.521Z,""    at org.graylog.collector.file.FileReaderService.access$400(FileReaderService.java:41) ~[graylog-collector.jar:?]"",""x0x4001.nbg"""
"2015-07-01T07:24:17.521Z,""    at org.graylog.collector.file.FileReaderService.access$400(FileReaderService.java:41) ~[graylog-collector.jar:?]"",""x0x4001.nbg"""
"2015-07-01T07:24:17.521Z,""    at org.graylog.collector.file.FileReaderService.followFile(FileReaderService.java:130) ~[graylog-collector.jar:?]"",""x0x4001.nbg"""
2015-07-01T07:24:17.521Z,"java.nio.file.FileSystemException: /home/users/Pmsx_3.3/system/bin/dbie.log: Zu viele offene Dateien","x0x4001.nbg"

or like this:

2015-07-01T07:21:25.326Z,"2015-07-01T09:17:21.587+0200 ERROR [FileObserver] collector.file.FileReaderService - Cannot read newly created file /home/users/Pmsx_3.3/system/bin/timesr.log","x0x4001.nbg"
"2015-07-01T07:21:25.326Z,""    at sun.nio.fs.UnixFileSystemProvider.newAsynchronousFileChannel(UnixFileSystemProvider.java:195) ~[?:1.7.0_21]"",""x0x4001.nbg"""
"2015-07-01T07:21:25.326Z,""    at org.graylog.collector.file.FileReaderService.followFile(FileReaderService.java:130) ~[graylog-collector.jar:?]"",""x0x4001.nbg"""
"2015-07-01T07:21:25.326Z,""    at com.google.common.util.concurrent.AbstractExecutionThreadService$1$2.run(AbstractExecutionThreadService.java:60) [graylog-collector.jar:?]"",""x0x4001.nbg"""
"2015-07-01T07:21:25.326Z,""    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91) ~[?:1.7.0_21]"",""x0x4001.nbg"""
"2015-07-01T07:21:25.326Z,""    at sun.nio.fs.UnixFileSystemProvider.newAsynchronousFileChannel(UnixFileSystemProvider.java:195) ~[?:1.7.0_21]"",""x0x4001.nbg"""
"2015-07-01T07:21:25.326Z,""    at org.graylog.collector.file.FileReaderService.access$400(FileReaderService.java:41) ~[graylog-collector.jar:?]"",""x0x4001.nbg"""
2015-07-01T07:21:25.326Z,"2015-07-01T09:17:21.588+0200 ERROR [FileObserver] collector.file.FileReaderService - Cannot read modified file /home/users/Pmsx_3.3/system/bin/qlibri.log","x0x4001.nbg"

Add support for wildcards for file input configuration

It can be useful to support wildcards/globbing for file names in file input. This makes it easy to collect generated files and also to avoid excessive configuration if you have a lot of files. (i.e. /var/log/apache2/*.log or /var/log/apps/**/*.log)

using sigar-loader

https://github.com/kamon-io/sigar-loader

java embedded installer

create an executable installer with embedded java
actually manual install is an heavy task and i cannot download 130 mb of java jdk per client

Fix react warning on System/Agents page

React prints:
Warning: Each child in an array or iterator should have a unique "key" prop. Check the render method of AgentList. See http://fb.me/react-warning-keys for more information.

The table rows are probably missing the key attributes.

(Filing here to make it easier to keep track of agent related issues).

Generate sequence numbers for each message

To be able to absolutely order messages emitted from a single source, we should think about if we need a sequence number, or if the timestamps are enough.

The file we read from and convert sections into messages can come from the same millsecond, thus we would lose the ability to reliably order them.

Add help command to CLI

There is currently no help command or flag to show the usage.

Needs documentation for Windows

What are the dependency requirements? (through trial and error found it asks for JDK vs. Client Java)
Instructions for installation, can be as simple as
1. Install JDK
2. run command line etc
3. create config file
Need documentation on WHERE config files live, where log files live.
NXLog install notes as a point of reference http://nxlog.org/node/295#quickstart_windows

Decide which log framework to use

logback or log4j.

Also create example configuration file.

Log-File input fails with content-splitter = PATTERN

Hi,

with the following collector config

  graylog-server-log {
    type = "file"
    path = "/var/log/graylog-server/server.log"
    content-splitter = "PATTERN"
    content-splitter-pattern = "^\\d{4}-\\d{2}-\\d{2}T"
  }

the collector transmitted only the line 1 to 3. The log-line 4 would never send, because the pattern does not match.

2015-06-22T15:05:31.715+02:00 INFO  [Log] Rolled new log segment for 'messagejournal-0' in 1 ms.
2015-06-22T15:06:11.851+02:00 INFO  [Log] Scheduling log segment 2470544171 for log messagejournal-0 for deletion.
2015-06-22T15:07:11.851+02:00 INFO  [Log] Deleting segment 2470544171 from log messagejournal-0.
2015-06-22T15:07:11.894+02:00 INFO  [OffsetIndex] Deleting index /var/lib/graylog-server/journal/messagejournal-0/00000000002470544171.index.deleted

These log-line will only send if another log-line will be written to the logfile.

In this case for example it could be happen, that a panic message from an application never transmit to the graylog server.

Is there a solution for this case like a configuration item or something?

No message if file input cannot access actual file

Steps to reproduce:

configure file input to read from non-existent file or set permissions to deny access
start agent
no message about the file is printed, at least not on INFO or higher

Expectation:
Print warning or error message if the file cannot be opened for reading.

using collector for forwarding syslog messages: no facility nor level is shown in graylog server

i want to use graylog collector for forwarding my syslog messages ( and some other logfiles too). all seems fine, but in graylog server all messages come with facility "6" and no level is shown.
if i send syslog messages directly by port 514 to graylog, graylog server shows me facilty and level correct.
need i some additional filters or plugins ??

Add git commit id to assets and version command

This will help identifying running versions.

Defining custom field in the message

I wanted top set custom fields like "facility" and "environment" while sending the logs to graylog. How do I do that using the unix version of collector.

Whitespace issue in startup scripts

There seems to be an issue with blanks in directories.

c:\test test\graylog-collector-0.2.2\bin>graylog-collector-service.bat start GraylogCollector
The data area passed to a system call is too small. Failed to start service
ERROR: Failed to start service: GraylogCollector

File changes by rsync are not detected

I'm wanting to feed our (multiple) squid server logs into graylog and want to simply rsync the logs into a staging directory on the server, and have the collector pipe them in via the GELF connector. (ie I don't want them put into syslog, nor do I want to install java on the proxies just so I can run the collector)

For testing I'm running it in a shell with the output going to stdout.

inputs {
  squid {
    type = "file"
    path-glob-root = "/var/spool/squid-logs"
    path-glob-pattern = "*access.log"
  }
}

I have a server1-access.log file in there, and if I echo squidline > server1-access.log it triggers graylog-collector nicely and I see the GELF. However, rsync doesn't work like that: it creates a new file with a temporary filename, copies the original file to that, appends the new data and then renames it over the original file. End result is the file is updated, but has a new inode. It appears graylog-collector doesn't notice that change, nor the fact the file is now a different size?

Source: https://groups.google.com/forum/?hl=en#!topic/graylog2/g1NuVz571FE

This might be related to the fact that we do not read old data yet.

Cosmetic changes for UI/wording on System/Agents page

Consider making some minor cosmetic changes:

Node Id -> Host name
ID vs Id
margins for inactive agents
use element for last seen

Multiline log message support via REGEX pattern

Some application logs have a single log entry on multiple lines. It won't be helpful to have the log lines shipped as separate messages and should be treated as a single message.

force source hostname

how can i overwrite souce hostname collector? by default graylog show host sysname collector taken by os, how can i manually change the hostname description?

Missing Log-File is not configuration error

We use a lot of applications, that write logs only in case of error. In the consequence there is not a log all the time. But in this case the collector wont start.

Example error-message:

2015-06-18T09:52:27.376+0200 ERROR [main] cli.commands.Run - Configuration Error: /var/log/apache2/access.log or /var/log/apache2 is not accessible (check if directory exists and permissions are correct) (path)

It is possible to change this behaviour of the collector? The collector could poll in intervall if the log exists.

installing graylog-collector-latest-repository-debian8_latest.deb make apt-get crash

before this package no issue

root@proxy:~# dpkg -i graylog-collector-latest-repository-debian8_latest.deb
(Lettura del database... 72427 file e directory attualmente installati.)
Preparativi per estrarre graylog-collector-latest-repository-debian8_latest.deb...
Estrazione di graylog-collector-latest-repository-debian8 (1.0.0-1) su (1.0.0-1)...
Configurazione di graylog-collector-latest-repository-debian8 (1.0.0-1)...
root@proxy-bazoli:~# apt-get update
Trovato http://security.debian.org jessie/updates InRelease
Trovato http://repo.zabbix.com jessie InRelease
Trovato http://debian.saltstack.com jessie-saltstack InRelease
Trovato http://security.debian.org jessie/updates/main Sources
Scaricamento di:1 http://security.debian.org jessie/updates/main i386 Packages [131 kB]
Trovato http://httpredir.debian.org jessie InRelease
Scaricamento di:2 http://security.debian.org jessie/updates/main Translation-en [71,8 kB]
Scaricamento di:3 http://repo.zabbix.com jessie/main Sources [1.214 B]
Ign https://packages.graylog2.org jessie InRelease
Scaricamento di:4 http://repo.zabbix.com jessie/main i386 Packages [2.707 B]
Scaricamento di:5 http://httpredir.debian.org jessie/main i386 Packages [6.767 kB]
Trovato http://debian.saltstack.com jessie-saltstack/main i386 Packages
Ign http://repo.zabbix.com jessie/main Translation-it_IT
Scaricamento di:6 http://httpredir.debian.org jessie/main Sources [7.059 kB]
Ign http://repo.zabbix.com jessie/main Translation-it
Scaricamento di:7 http://httpredir.debian.org jessie/main Translation-en [4.585 kB]
Ign http://repo.zabbix.com jessie/main Translation-en
Trovato https://packages.graylog2.org jessie Release.gpg
Ign http://debian.saltstack.com jessie-saltstack/main Translation-it_IT
Ign http://debian.saltstack.com jessie-saltstack/main Translation-it
Ign http://debian.saltstack.com jessie-saltstack/main Translation-en
E: Method https has died unexpectedly!
E: Il sottoprocesso https ha ricevuto un segmentation fault.

Linux proxy 3.16.0-4-686-pae #1 SMP Debian 3.16.7-ckt11-1+deb8u3 (2015-08-04) i686 GNU/Linux

Please add basic auth support

Because we are running Graylog at EC2 we've placed a proxy in front which is handling the Gelf requests, but first you need to authenticate with basic auth. Could you please add support for that in the collector?

Support for GELF UDP output

By default the output type "gelf" appears to be GELF TCP only. It'd be nice if we could configure a GELF UDP output for the graylog-collector as well.