That's weird, having no certificate would result in 'connection refused' as far as I've seen. How do I debug or fix it? I mean, logstash is flat jar with all necessary stuff in it and I've built lumberjack from source and installed it on a similar machine and it all worked fine, but I suddenly got this now.

What information might be relevant to help resolve it?

ship metrics about errors, event counts, etc.

Using this in my run script:
exec /opt/lumberjack/bin/lumberjack.sh --host logstash.internal --port 5005 --ssl-ca-path /etc/lumberjack/syslog/ca.crt --field type=syslog /var/log/secure /var/log/messages /var/log/*.log

I only see /var/log/secure streaming into logstash

Fixed by doing this (ghetto):
exec tail -F /var/log/secure /var/log/messages /var/log/*.log | /opt/lumberjack/bin/lumberjack.sh --host logstash.internal --port 5005 --ssl-ca-path /etc/lumberjack/syslog/ca.crt --field type=syslog -

sudo su lumberjack -c "/opt/lumberjack/bin/lumberjack --host logstash.example.com --port 50670 --ssl-ca-path /etc/pki/real/path/to/cert /var/log/*.log"
Watching 2 files, setting open file limit to 106
Watching 2 files, setting memory usagelimit to 256 bytes
Assertion failed harvester.c:59 in harvest(), insist(fd >= 0): open(/var/log/yum.log) failed: Permission denied
Aborted

• I would like to know when packages were changed
• Running lumberjack as root to read the log is less attractive.
• Yum pushes these permissions when yum update is run.

Thinking as I write this, the file permissions issue isn't really a lumberjack issue, but I imagine this would be a common use case and would love to suggest a work around or best practice in the readme / faq

The assertion / abort issue is tricky. I respect that the program exits when it is unable to do what it was told to do, but would a warning suffice like logstash agent? A flag to enable warnings when files are unreadable?

Thoughts...

• If we are shipping a low message rate, we should drop the window size to something around 5-seconds of events.
• If we are encountering frequent errors, we should drop the window size to avoid the disconnect-and-retry sending too many duplicate events.

See: https://logstash.jira.com/browse/LOGSTASH-1288

Jordan mentioned this could be a lumberjack issue.

Having the following issue building the code on linux mint 15... I'm no expert at compiling code but I assume I'm missing a package somewhere.

Makefile:22: warning: overriding commands for target `clean'
Makefile.ext:15: warning: ignoring old commands for target `clean'
PKG_CONFIG_PATH=$PWD/build/lib/pkgconfig \
    go build -ldflags '-r $ORIGIN/../lib' -v -o build/bin/lumberjack
_/home/geoffrey/Temp/lumberjack
# _/home/geoffrey/Temp/lumberjack
./lumberjack.go:58: undefined: syslog.LOG_DAEMON
make: *** [build/bin/lumberjack] Error 2

Thanks.

seems to be the go edition that's the default/current. but under go/ - it says go wasn't too good.. in the go/readme.

What's one to think? :)

In the README, you list several popular protocols which fail to meet your needs. However, the BEEP protocol might be what you're looking for. See http://en.wikipedia.org/wiki/BEEP as a starting point.

It would be neat to be able to exclude files.

So one could do something like this:
{
"paths": [
"/var/log/messages",
"/var/log/*.log"
],
"exclude": [
"/var/log/uninteresting.log",
"/var/log/alotofjunk.log"
],
"fields": { "type": "syslog" }
}

The init script is referencing non-existing files on Fedora 17.

Running it shows:
/etc/init.d/lumberjack: line 27: /lib/init/vars.sh: No such file or directory

It also references /lib/lsb/init-functions, which doesn't exist.

We are starting via monit and our own init scripts, so its not much of an issue. I usually do that anyway. But, the documentation does refer people to create the RPM, so it could cause frustration for some.

Perhaps some documentation on systems the init script is known to work on would be helpful and a disclaimer to change it on your own if you run something else. Not sure if you want to have to support init scripts for every os out there.

I was thinking maybe this needs to go in FPM, but It looks like its referencing lumberjack.init in the Makefile.

config file:
{
"network": {
"servers": [ "log01.dev:8514" ],
"ssl ca": "/etc/lumberjack/lumberjackCA.pem"
},
"files": [
{
"paths": [ "/var/log/messages" ],
"fields": { "type": "syslog" }
}, {
"paths": [ "/tmp/foo" ],
"fields": { "type": "foo" }
}
]
}

commandline:
/opt/lumberjack/bin/lumberjack.sh --config test.conf

steps

$ date >> /tmp/foo
$ cat .lumberjack
{"/tmp/foo":{"source":"/tmp/foo","offset":406,"inode":494459,"device":2049}}
[stop lumberjack]
$ date >> /tmp/foo
$ date >> /tmp/foo
$ cat .lumberjack
{"/tmp/foo":{"source":"/tmp/foo","offset":406,"inode":494459,"device":2049}}
[start lumberjack]
$ cat .lumberjack
{"/tmp/foo":{"source":"/tmp/foo","offset":406,"inode":494459,"device":2049}}
$ date >> /tmp/foo
$ cat .lumberjack
{"/tmp/foo":{"source":"/tmp/foo","offset":493,"inode":494459,"device":2049}}

I've found one here:
https://github.com/ahbit/public/blob/master/lumberjack/templates/startup-RedHat.erb
Should work.

Ruby has (in theory) a decent pure-ruby implementation of protobuf decoder (beefcake), which makes protobufs at least somewhat appealing for logstash integration.

Experiment with protobufs and swap out the internals maybe.

> Building zeromq PATH=$PWD:$PATH make -C vendor/zeromq/ install PREFIX=$PWD/build DEBUG= make -C ../libuuid PREFIX=/Users/seanarnold/projects/lumberjack/build /Users/seanarnold/projects/lumberjack/build/lib/libuuid.dylib Makefile:33: warning: overriding commands for target clean'
Makefile.ext:15: warning: ignoring old commands for target clean' Makefile:23: warning: overriding commands for target zeromq-2.2.0/Makefile'
../Makefile.ext:34: warning: ignoring old commands for target zeromq-2.2.0/Makefile' Makefile:20: warning: overriding commands for target util-linux-ng-2.17.2/Makefile'
../Makefile.ext:34: warning: ignoring old commands for target util-linux-ng-2.17.2/Makefile' make[2]: *** No rule to make target /Users/seanarnold/projects/lumberjack/build/lib/libuuid.dylib'. Stop.
make[1]: *** [/Users/seanarnold/projects/lumberjack/build/lib/libuuid.dylib] Error 2
make: *** [build/include/zmq.h] Error 2`

(rsyslog)[http://www.rsyslog.com/features/](and presumably %28syslog-ng%29[http://www.balabit.com/network-security/syslog-ng/opensource-logging-system/features/summary]) seem to match the requirements you've outlined.

It would be useful to add a section comparing lumberjack with those to help people (including me) see where lumberjack shines in contrast to them.

Hello and thank you for a perfect log shipper and aggregator!

I'm evaluating logstash as an aggregator of logs from two dozens of servers. To manage which users can search which logs, I ship logs from different sources to different indices and restrict access to them with nginx.

To accomplish that, I need several simple grep filters, which (as I've noticed) are prone to break the application. In order not to lose my events, I run two instances of logstash and lumberjacks are configured to connect to both of them.

If a logstash crashes, I can launch it again pretty quickly with puppet (ensure => running), but by that time all of lumberjacks would be shipping logs to another instance.

Is there a good way to make lumberjack reconnect so that they are shipping to randomly picked logstashes somehow? I wouldn't like to use cron here because I think some events would duplicate this way because of that window setting.

Discovered when reading a file with very long lines as STDIN.

Hello,

I suspect this is something dumb but I cant for the life of me figure out why lumberjack has started to throw these errors.

2013/09/19 12:36:37.093618 Failed to tls handshake with 192.168.51.61:5043 read tcp 192.168.51.61:5043: i/o timeout
2013/09/19 12:36:38.093839 Connecting to 192.168.51.61:5043
2013/09/19 12:36:53.094597 Failed to tls handshake with 192.168.51.61:5043 read tcp 192.168.51.61:5043: i/o timeout
2013/09/19 12:36:54.094838 Connecting to 192.168.51.61:5043

We have been using it for a while with no issues, then elastic search ran out of disk space. Cleaned up the disk and now we are seeing these errors.

Restarted logstash, elastic search and the indexer and lumberjack with no update.

thanks
Luke

Problem overview
I'm running lumberjack on Ubuntu 10.04 and 12.04. My logstash server is running on Ubuntu 12.04 and configured for lumberjack.

Whenever there is something to send to the logstash server, lumberjack is aborted with the one of the following messages:

Assertion failed harvester.c:158 in harvest(), insist(rc == 0): Assertion failed emitter.c:99 in emitter(), insist(rc == 0): zmq_recv(inproc://emitter) failed (returned 135): Resource temporarily unavailablezmq_send(event) failed: Unknown error 135

Aborted

Assertion failed harvester.c:158 in harvest(), insist(rc == 0): zmq_send(event) failed: Unknown error 135
Aborted

Assertion failed harvester.c:158 in harvest(), insist(rc == 0): Assertion failed emitter.c:99 in emitter(), insist(rc == 0): zmq_send(event) failed: Unknown error 135
zmq_recv(inproc://emitter) failed (returned 135): Resource temporarily unavailable
Aborted

The error number varies.

Some setup details
Logstash Server configuration:

lumberjack {
    port => 6782
    ssl_certificate => "/path/to/logstash.crt"
    ssl_key => "/path/to/logstash.key"

    type => "lumberjack"
  }

According to the logstash log file everything was loaded and is ready, additionally "telneting" to the port from the client works.

That's how lumberjack is started:

sudo /opt/lumberjack/bin/lumberjack --host x.y.z.x --port 6782 --ssl-ca-path /path/to/logstash.crt /var/log/auth.log

Any idea what's going on here? If you need further information, just let me know.
thanks!

Hi Jordan,

Kudos on your Logstash and Lumberjack work!

Wouldn't it be cool to have lumberjack send fresh timber directly to Redis?
I am currently designing a Logstash -> Elasticsearch architecture in which Redis plays a major role.

My high-level design looks like this:

• logging - local on ~200 servers (not on shared storage, to avoid nfs mayhem)
• solution X

--- from here on its centralized/scaled ---

• redis - a buffer between input and parsing LS instances
• logstash (parsing) - centralized, N instances with M workers
• redis - buffer between parsing & forwarding LS instances
• logstash (forwarding) - doing nothing more than redis.blpop and push to ES
• elasticsearch
• kibana

I could run another tier of LS instances as "solution X", but running ~200 LS instances seems like overkill and wasteful. I am sure you agree.

So I was thinking: wouldn't it be cool to use Lumberjack with Redis support?

What are your ideas?

Regards,
Renzo

output:

gethostbyname(demo.logstash.net) failed: No such file or directory
Connection attempt to demo.logstash.net:5111 failed: Success

strace:

[pid 38149] stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=73, ...}) = 0
[pid 38149] stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=73, ...}) = 0
[pid 38149] open("/etc/hosts", O_RDONLY|O_CLOEXEC) = -1 EMFILE (Too many open files)
[pid 38149] stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=73, ...}) = 0
[pid 38149] socket(PF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = -1 EMFILE (Too many open files)
[pid 38149] socket(PF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = -1 EMFILE (Too many open files)
[pid 38149] write(1, "gethostbyname(demo.logstash.net)"..., 67) = 67
[pid 38149] write(1, "Connection attempt to demo.logst"..., 61) = 61
[pid 38149] nanosleep({15, 0}, ^C <unfinished ...>

$ make
Makefile:31: warning: overriding commands for target `clean'
Makefile.ext:15: warning: ignoring old commands for target `clean'
mkdir build
mkdir build/include
PATH=$PWD:$PATH fetch.sh -o build/include/insist.h https://raw.github.com/jordansissel/experiments/master/c/better-assert/insist.h
-o build/include/insist.h https://raw.github.com/jordansissel/experiments/master/c/better-assert/insist.h
-o build/include/insist.h -- https://raw.github.com/jordansissel/experiments/master/c/better-assert/insist.h
URL: https://raw.github.com/jordansissel/experiments/master/c/better-assert/insist.h
--2013-01-25 17:46:40--  https://raw.github.com/jordansissel/experiments/master/c/better-assert/insist.h
Resolving raw.github.com... 199.27.75.193
Connecting to raw.github.com|199.27.75.193|:443... connected.
ERROR: certificate common name “*.a.ssl.fastly.net” doesn’t match requested host name “raw.github.com”.
To connect to raw.github.com insecurely, use ‘--no-check-certificate’.
make: *** [build/include/insist.h] Error 5

running make again then breaks

lumberjack.c: In function ‘set_resource_limits’:
lumberjack.c:84: error: implicit declaration of function ‘insist’

the file is empty because the file was created, but the download failed

$ cat ./build/include/insist.h
$ 

As the title says, there is no option in lumberjack to disabling auto-addition of fields while shipping logs. It would be great if we have an option to do that.

In my case, I am sending all logs to central syslog server and from there I'm using lumberjack to send to multiple logstash instances. Now, in this case, the source_host field is not relevant to me as all logs will have it as the hostname of syslog server.

MartinCleaver: does logstash lumberjack work on Windows?
[4:57pm] MartinCleaver: can see how to do a deb and rpm only
[4:58pm] attractiveape: MartinCleaver: no, not for windows.
[4:58pm] MartinCleaver: is there an equivalent for windows?

I recently built lumberjack in the past day or two and encountered a problem.

I have it seeking for files in /var/log/apache2/some_rotated_log*

When apache rotates to a new log index, lumberjack takes a few seconds to pickup the new file and doesn't seek from the beginning. This appears to cause missed events while it tails the new log.

You can reproduce this issue by setting lumberjack to parse for a wildcard and echo "some line" > watched_file, wait 20 seconds, then echo another line. It'll send the second, but not the first. It would be awesome to implement a flag to have the harvester seek from the beginning of a new file.

some folks don't want or need ssl (nor the overheads of SSL cert management)

Currently lumberjack fails to start if it can't find at least one file that matches a wildcard that's passed to it. It seems like it should continue to start and watch for new files much like logstash does.

Thanks!

Hello,

We have a java app which uses log4j to output to a log file. The log file is rotated based on size. (200mb).

Lumberjack appears to be locking the file after it has been deleted by Log4J. This means the file is never deleted and eventually we run out of disk space.

[root@server folder]# lsof |grep -i del|grep -i lum
lumberjac 18491 root 9r REG 253,5 403973659 66690 /usr/local/APP/logs/dailyShell.log.4 (deleted)
lumberjac 18491 root 14r REG 253,5 209760944 65034 /usr/local/APP/logs/dailyShell.log.4 (deleted)
lumberjac 18491 root 15r REG 253,5 209715244 65036 /usr/local/APP/logs/dailyShell.log.4 (deleted)
lumberjac 18491 root 16r REG 253,5 209771415 65037 /usr/local/APP/logs/dailyShell.log.4 (deleted)
lumberjac 18491 root 17r REG 253,5 209799901 65041 /usr/local/APP/logs/dailyShell.log.4 (deleted)
lumberjac 18491 root 18r REG 253,5 209728019 65060 /usr/local/APP/logs/dailyShell.log.4 (deleted)
lumberjac 18491 root 19r REG 253,5 209720933 65061 /usr/local/APP/logs/dailyShell.log.4 (deleted)
[root@server folder]#

Restarting lumberjack removes the file locks and frees the disk space up. however the app creates 1GB's of log per hour.

The RPM we built is lumberjack-0.1.2-1.x86_64.rpm

Red Hat Enterprise Linux Server release 6.3 (Santiago)

Linux server.common 2.6.32-279.22.1.el6.x86_64 #1 SMP Sun Jan 13 09:21:40 EST 2013 x86_64 x86_64 x86_64 GNU/Linux

Additionally we see lumberjack start harvesting on the new log file as it is rotated.

2013/09/05 23:28:33.594797 Saving registrar state.
2013/09/05 23:28:36.231139 Launching harvester on rotated file: /usr/local/APP/logs/dailyShell.log
2013/09/05 23:28:36.231265 Starting harvester: /usr/local/APP/logs/dailyShell.log
2013/09/05 23:28:37.531868 Registrar received 1000 events
2013/09/05 23:28:37.532050 Saving registrar state.
2013/09/05 23:28:38.651929 Registrar received 1000 events

Thanks
Luke

If zeromq is used as a transport, both 2.x and 3.x should be supported.

Currently when a file specified on the command line does not exist, lumberjack dies with an assertion. It's also impossible to detect new files as they appear. Lumberjack would be even more useful if it could monitor transient logfiles matching a pattern specified on the command line. Example

$ /opt/lumberjack/bin/lumberjack --host 127.0.0.1 \
   --port 1234 \
   --ssl-ca-path /path/to/cert \
    /path/to/logs/job-\*.log

In the config for the Go version of Lumberjack, this works:
"fields": { "type": "gearman_manager_log", "tags": "gearman_manager" }
But this fails:
"fields": { "type": "gearman_manager_log", "tags": [ "gearman_manager", "foo", "bar" ] }

With the following error:
2013/08/19 06:11:17 Failed unmarshalling json: json: cannot unmarshal array into Go value of type string

Tags are supposed to be an array!

This was particularily confusing for me since I only know FPM from php-fpm, and not your fpm (which looks great btw).

I think a small "prerequisites" or "getting started" paragraph on with a link to the FPM repo would be helpful.

Hello,

Love lumberjack and logstash but suddenly having problems, lumberjack on two new servers (all RHEL 6) fails to connect, never gives the Connecting to or Connected Successfully message, this is a production logstash server other clients are sending to.
I tried to install the latest lumberjack but the new version requires go which seems to have no RHEL/Centos support. Tried to add error logging in the lumberjack_tcp_connect function in proto.c in the old c version but it won't build after a make clean, gives a message error: jemalloc/jemalloc.h: No such file or directory even though I can see the jemalloc.h file in the directory cc calls out in the -Ibuild/include command line switch.
Any suggestions on how to debug the connection problem or how to proceed to use lumberjack on RHEL/Centos 6?
Thanks again!

Compiling with gcc 4.6.3 on HP-UX 11.23:
lumberjack $ gmake
Makefile:33: warning: overriding recipe for target clean' Makefile.ext:15: warning: ignoring old recipe for targetclean'
gcc -D_POSIX_C_SOURCE=199309 -std=c99 -Wall -Wextra -Werror -pipe -g -Wno-unused-function -Ibuild/include -c -o lumberjack.o lumberjack.c
In file included from /usr/local/include/openssl/ssl.h:1361:0,
from proto.h:7,
from lumberjack.c:16:
/usr/local/include/openssl/dtls1.h:236:17: error: field 'next_timeout' has incomplete type
lumberjack.c:37:13: error: 'no_argument' undeclared here (not in a function)
lumberjack.c:46:14: error: 'required_argument' undeclared here (not in a function)
lumberjack.c: In function 'set_resource_limits':
lumberjack.c:80:17: error: storage size of 'limits' isn't known
lumberjack.c:83:3: error: implicit declaration of function 'nice' [-Werror=implicit-function-declaration]
lumberjack.c:84:3: error: implicit declaration of function '__STRING' [-Werror=implicit-function-declaration]
lumberjack.c:84:3: error: format '%s' expects argument of type 'char ', but argument 6 has type 'int' [-Werror=format]
lumberjack.c:101:9: error: request for member 'rlim_cur' in something not a structure or union
lumberjack.c:101:27: error: request for member 'rlim_max' in something not a structure or union
lumberjack.c:101:3: error: statement with no effect [-Werror=unused-value]
lumberjack.c:103:28: error: request for member 'rlim_max' in something not a structure or union
lumberjack.c:103:10: error: format '%ld' expects argument of type 'long int', but argument 3 has type 'struct option_doc *' [-Werror=format]
lumberjack.c:104:3: error: implicit declaration of function 'setrlimit' [-Werror=implicit-function-declaration]
lumberjack.c:104:18: error: 'RLIMIT_NOFILE' undeclared (first use in this function)
lumberjack.c:104:18: note: each undeclared identifier is reported only once for each function it appears in
lumberjack.c:105:3: error: format '%s' expects argument of type 'char *', but argument 6 has type 'int' [-Werror=format]
lumberjack.c:105:3: error: request for member 'rlim_max' in something not a structure or union
lumberjack.c:120:9: error: request for member 'rlim_cur' in something not a structure or union
lumberjack.c:120:27: error: request for member 'rlim_max' in something not a structure or union
lumberjack.c:120:3: error: statement with no effect [-Werror=unused-value]
lumberjack.c:123:18: error: 'RLIMIT_RSS' undeclared (first use in this function)
lumberjack.c:124:3: error: format '%s' expects argument of type 'char *', but argument 6 has type 'int' [-Werror=format]
lumberjack.c:124:3: error: request for member 'rlim_max' in something not a structure or union
lumberjack.c:80:17: error: unused variable 'limits' [-Werror=unused-variable]
lumberjack.c: In function 'main':
lumberjack.c:143:61: error: invalid application of 'sizeof' to incomplete type 'struct option'
lumberjack.c:143:52: error: invalid operands to binary * (have 'int' and 'struct option_doc *')
lumberjack.c:143:68: error: passing argument 2 of 'realloc' makes integer from pointer without a cast [-Werror]
/usr/local/include/jemalloc/jemalloc.h:50:23: note: expected 'size_t' but argument is of type 'struct option_doc *'
lumberjack.c:144:5: error: invalid use of undefined type 'struct option'
lumberjack.c:144:19: error: dereferencing pointer to incomplete type
lumberjack.c:144:22: error: request for member 'name' in something not a structure or union
lumberjack.c:144:5: error: statement with no effect [-Werror=unused-value]
lumberjack.c:145:5: error: invalid use of undefined type 'struct option'
lumberjack.c:145:19: error: dereferencing pointer to incomplete type
lumberjack.c:145:22: error: request for member 'has_arg' in something not a structure or union
lumberjack.c:145:5: error: statement with no effect [-Werror=unused-value]
lumberjack.c:146:5: error: invalid use of undefined type 'struct option'
lumberjack.c:147:19: error: dereferencing pointer to incomplete type
lumberjack.c:147:22: error: request for member 'val' in something not a structure or union
lumberjack.c:147:5: error: statement with no effect [-Werror=unused-value]
lumberjack.c:151:59: error: invalid application of 'sizeof' to incomplete type 'struct option'
lumberjack.c:151:50: error: invalid operands to binary * (have 'int' and 'struct option_doc *')
lumberjack.c:151:66: error: passing argument 2 of 'realloc' makes integer from pointer without a cast [-Werror]
/usr/local/include/jemalloc/jemalloc.h:50:23: note: expected 'size_t' but argument is of type 'struct option_doc *'
lumberjack.c:152:3: error: invalid use of undefined type 'struct option'
lumberjack.c:152:17: error: dereferencing pointer to incomplete type
lumberjack.c:152:20: error: request for member 'name' in something not a structure or union
lumberjack.c:152:3: error: statement with no effect [-Werror=unused-value]
lumberjack.c:155:3: error: implicit declaration of function 'getopt_long_only' [-Werror=implicit-function-declaration]
lumberjack.c:159:9: error: implicit declaration of function 'strdup' [-Werror=implicit-function-declaration]
lumberjack.c:159:36: error: assignment makes pointer from integer without a cast [-Werror]
lumberjack.c:168:29: error: assignment makes pointer from integer without a cast [-Werror]
lumberjack.c:185:48: error: assignment makes pointer from integer without a cast [-Werror]
lumberjack.c:187:50: error: assignment makes pointer from integer without a cast [-Werror]
lumberjack.c:191:9: error: format '%s' expects argument of type 'char *', but argument 6 has type 'int' [-Werror=format]
lumberjack.c:221:3: error: format '%s' expects argument of type 'char *', but argument 6 has type 'int' [-Werror=format]
cc1: all warnings being treated as errors
gmake: ** [lumberjack.o] Error 1

The dependencies in vendor/ build mostly fine. (I skipped libuuid, zeromq builds fine without it, jemalloc had a few issues, but I think I solved those).

I've compiled lumberjack-go on a SuSE 11 system. When trying to install lumberjack on a SuSE 10 system, i'm getting the following error message:

"2013/09/09 11:52:33.095986 Failed to open syslog: Unix syslog delivery error
startproc: exit status of parent of /opt/lumberjack/bin/lumberjack.sh: 1"

Do i have do compile an extra lumberjack-client for SuSE 10 on a SuSE 10 system?

Fails to compile on OS X with a type mismatch:

matt@momiji:~/code/lumberjack$ make
Makefile:22: warning: overriding commands for target `clean'
Makefile.ext:15: warning: ignoring old commands for target `clean'
PKG_CONFIG_PATH=$PWD/build/lib/pkgconfig \
        go build -ldflags '-r $ORIGIN/../lib' -v -o build/bin/lumberjack
_/Users/matt/code/lumberjack
# _/Users/matt/code/lumberjack
./prospector.go:56: invalid operation: fstat.Dev == state.Device (mismatched types int32 and uint64)
./registrar.go:30: cannot use fstat.Dev (type int32) as type uint64 in field value
make: *** [build/bin/lumberjack] Error 2
matt@momiji:~/code/lumberjack$ make
Makefile:22: warning: overriding commands for target `clean'
Makefile.ext:15: warning: ignoring old commands for target `clean'
PKG_CONFIG_PATH=$PWD/build/lib/pkgconfig \
        go build -ldflags '-r $ORIGIN/../lib' -v -o build/bin/lumberjack
_/Users/matt/code/lumberjack
# _/Users/matt/code/lumberjack
./prospector.go:56: invalid operation: fstat.Dev == state.Device (mismatched types int32 and uint64)
./registrar.go:30: cannot use fstat.Dev (type int32) as type uint64 in field value
make: *** [build/bin/lumberjack] Error 2

Building jemalloc with TLS (thread library storage) causes all sorts of noise in the logs of Xen para-virtualized domUs.

Feb 15 13:15:03 smokeping kernel: printk: 1240 messages suppressed.
Feb 15 13:15:03 smokeping kernel: 4gb seg fixup, process lumberjack (pid 13806), cs:ip 73:00f84e2d
Feb 15 13:15:08 smokeping kernel: printk: 437 messages suppressed.
Feb 15 13:15:08 smokeping kernel: 4gb seg fixup, process lumberjack (pid 13813), cs:ip 73:00980e2d
Feb 15 13:15:13 smokeping kernel: printk: 26 messages suppressed.
Feb 15 13:15:13 smokeping kernel: 4gb seg fixup, process lumberjack (pid 13813), cs:ip 73:00980e2d
Feb 15 13:15:18 smokeping kernel: printk: 14 messages suppressed.
Feb 15 13:15:18 smokeping kernel: 4gb seg fixup, process lumberjack (pid 13806), cs:ip 73:00f8542f
Feb 15 13:15:23 smokeping kernel: printk: 4 messages suppressed.
Feb 15 13:15:23 smokeping kernel: 4gb seg fixup, process lumberjack (pid 13806), cs:ip 73:00f84e2d

This is a known issue with the system libc packages, and most of the common distros have fixes for this. Since lumberjack is using the custom-compiled jemalloc, it is bypassing those fixes, and the those messages spam logs with great regularity. Adding "--disable-tls" to the jemalloc configure line fixes this.

• Add libiconv
• Convert lines that are invalid UTF-8, assumedly ascii, to valid UTF-8?

Currently idle-flush is 'time since last log read' which can cause logs to buffer locally until the buffer is full on a slow-but-steady stream of events.

Ideal would be flushing:

• when buffer full (done already)
• when last flush was too long ago and there's data in the buffer.

I have installed FPM and Go and receive the following error message after issueing make

Makefile:21: warning: overriding commands for target `clean'
Makefile.ext:15: warning: ignoring old commands for target `clean'
go get -d github.com/alecthomas/gozmq
PKG_CONFIG_PATH=$PWD/build/lib/pkgconfig \
      go install -tags zmq_3_x github.com/alecthomas/gozmq
go install -ldflags '-r $ORIGIN/../lib' lumberjack
can't load package: package lumberjack: import "lumberjack": cannot find package
make: *** [bin/lumberjack] Error 1

SSL_write and SSL_read should both timeout and cause connection aborts if appropriate.

The message received here https://github.com/jordansissel/lumberjack/blob/master/emitter.c#L55 seems to have the correct size for the payload but is blank. I've verified that the message sent here (the line variable) is correct: https://github.com/jordansissel/lumberjack/blob/master/harvester.c#L136

My hunch is some sort of ZMQ problem but I still haven't been able to narrow it down.

I tried to execute the lumberjack.sh with nohup but there is no log on nohup.out:

$ nohup /opt/lumberjack/bin/lumberjack --ssl-ca-path /etc/ssl/logstash.pub --host 10.196.51.168 --port 5000 /var/log/messages&
$ ll nohup
-rw------- 1 root root 0 Jul 12 18:19 nohup.out

When I execute without nohup, I can see the lumberjack log on console, like this:

$ /opt/lumberjack/bin/lumberjack --ssl-ca-path /etc/ssl/logstash.pub --host 10.196.51.168 --port 5000 /var/log/messages
2013-07-12T18:22:28.336+0000 Watching 1 files, setting open file limit to 103
2013-07-12T18:22:28.337+0000 Watching 1 files, setting memory usage limit to 1048576 bytes
2013-07-12T18:24:48.617+0000 Connecting to 10.196.51.168(10.196.51.168):5000

Any idea?

here's full *FLAGS that i used in my distro:

x86_64-pld-linux-gcc -O2 -fwrapv -pipe -Wformat -Werror=format-security -gdwarf-4 -fno-debug-types-section -fvar-tracking-assignments -g2 -Wp,-D_FORTIFY_SOURCE=2 -fstack-protector --param=ssp-buffer-size=4 -fPIC -march=x86-64 -gdwarf-4 -fno-debug-types-section -fvar-tracking-assignments -g2  -D_POSIX_C_SOURCE=199309 -std=c99 -Wall -Wextra -Werror -pipe -g -Wno-unused-function -Ibuild/include    -c -o proto.o proto.c
proto.c: In function 'rand_uint32':
proto.c:49:3: error: dereferencing type-punned pointer will break strict-aliasing rules [-Werror=strict-aliasing]
cc1: all warnings being treated as errors
make: *** [proto.o] Error 1

ps: why you don't tag in git when you have released new release, currently only two tags exist, 0.0.1 and 0.0.8

Building jemalloc on HP-UX is a big pain - if at all possible.
It would be great to be able to build lumberjack without having to go through that pain.

Currently, if I want to pass different --field options for two files watched by lumberjack, I need to run two instances of lumberjack. Supporting custom fields per file makes messages easier to parse using Logstash.

I'm currently using lumberjack as a resque logger with daily log rotation. When the log file needs to be rotated, the workers tend to hang indefinitely. Checking the process with strace reveals the worker is waiting for an exclusive lock on the logfile.

I have Lumberjack eating a python log. It would spontaneously stop sending traffic to logstash (but not crash, process is still running). I've tracked it down to the fact that it hits a single absurdly long line in my log file. Restarting lumberjack makes it resume from current time (not from where it left off, though I believe LJ doesn't track its place right now?)

I've exported and measured the long line in question:

me@Server:~# wc longline.txt
1 1453 27840 longline.txt

I'm assuming this is too large for LJ to handle, but it would be nice for it to handle it gracefully rather than silently hang.

need to verify event order on retransmission