agnivade / funnel Goto Github PK

Just a pure golang alternative to elastic.
It also means you could index client side in the agent if you wanted to of course.

You can bring up a huge cluster by just denting to S3 or NATS and then let the server side bleve do the indexing and searching also.

It's very easy to get this going to replace elastic since it's just logs.

If you want a sample I can point you to some as others have written log indexers for bless e already

So far Iog to file can only be conditioned on max-lines or max-bytes, but sometimes daily rotation is more appreciated.

In logrotate there are also options for daily/weekly/monthly.

So maybe consider adding daily roll over?

	tick := time.Tick(time.Second)
	for {
		select {
		case <-tick:
			log.Info().Msg("tick")
			os.Stderr.WriteString(fmt.Sprintf("stderr - now :%v\n", time.Now()))
		}
	}

funnel kafka brokers set to invalid ports to mock kafka brokers failure.

we expect demo process running 7x24. but funnel stops it when kafka brokers down.

In the list of supported targets, could you add support for NATS (https://nats.io/) ?
It's an alternate solution to Kafka written in go.

This would be really nice to have this run on routers running openwrt. I can help with testing on ar71xx and ramips targets (both are MIPS I think)

Could you provide an example of how I could setup my docker services to log to funnel automatically, using docker-compose configuration?

I could explain more if needed!
Thanks.

In which OS can Funnel be used ? Can it be used on Windows ? A few words on this in the readme would help.

Looking at the prebuild binaries, it seam that funnel can't be used on windows. Is that right ?

If not, how difficult would it be to add support for windows ?

From the README, to capture stderr, you need to do:

$/etc/myapp/bin 2>&1 | funnel

What if instead you could do: funnel -cmd '/etc/myapp/bin'? Then it could capture all stdout and stderr on its own.

I would be nice if funnel had an option to add ISO 8601-style timestamp prefix (eg.: 2018-01-02 14:34:00)