multiple processes to a single forwarder about oklog HOT 4 CLOSED

A forwarder does nothing more than netcat would. Why would it be overkill to have one per process? This is the pattern employed by e.g. svlogd and many other logging pipeline systems.

from oklog.

Is there a reason not to have one process for multiple writers?

The reasoning is that our deployments can have very specific allocations on them. Each process needs a certain amount of CPU, disk space, memory, etc. The overhead of adding a forward for each process can cause a lot of context switching for kernel.

Is it more expensive to have a many processes select from a single pipe or have a single process select from many pipes? That is an honest question, I don't know.

The work we've done with the metron forwarder is about having one process read from multiple sources.

I could also be spewing a lot of nonsense I know nothing about. 😛

from oklog.

The overhead of adding a forward for each process can cause a lot of context switching for kernel.

I'll take your word for it—but I'm not really convinced without concrete e.g. profiling data :)

The main reason to use a single forwarder per process (service) is because it's the most straightforward way to abide the 12factor rule of logs as streams. There's certainly nothing wrong with running a single forwarder for multiple processes (services), if it's not too complicated. But I don't have any specific recommendations.

from oklog.

@jtarchie you could write a plugin for your existing forwarder that connects to and forwards to the ingest nodes. This would allow you to continue to capture the logs from multiple processes (or harvest from log files) and forward to oklog.

from oklog.