chiwanpark / flume-ng-redis Goto Github PK

Add source and sink based on List Data Structure (command LPUSH, RPUSH, LPOP, RPOP)

First off, thx for providing the plugin, great job.

We have a slight issue with it though. Using the RawMessageHandler we found that the plugin always parses the incoming flume events binary body as UTF-8 and also sends the body along as UTF-8. This breaks bodies which rely on the binary encoding. In our case we have binary thrift in the event body, which when interpreted as UTF-8 and put into Redis as UTF-8 changed the binary encoding.

We got this fixed for us by modifying the plugin a bit. Before opening a pull request, I would like to know if this change actually makes sense to you. In that case, I'm happy to share our changes.

There are some API changes in PollableSource since Flume 1.7.0 to allow backoff times to be configurable. We have to change the class hierarchy of RedisListDrivenSource. The source implementation should be derived by AbstractPollableSource.

Add PSUBSCRIBE-based Source to RedisSubscribeDrivenSource.

Thie repository is already connected with Travis-CI. But there is no test case. We will add unit test code as soon as possible.

Hi there,

Thanks for this awesome plug-in!

I've been trying to inject fields like the source and environment names for our deployment via the Flume configuration using interceptors, but it seems like this is not supported by this plug-in. Is there any chance this can be added?

I'm mostly interested in the static interceptor.

Hi there!

Thanks for this awesome plug-in!

I noticed that Flume sources using this plug-in do not appear in the flume metrics page. Is there any chance this can be added?

Thanks!

Hi, I clone this repository and run 'mvn package' on my computer, but there is no .jar file generated.

Currently, although Jedis is support multiple channel subscription, there is no multiple subscription feature in RedisSubscribeDrivenSource.

Subject says it all, add it please.

can you build and publish the latest version?

publish ---->redis---->flume-ng-redis(source)
when the flume-ng-redis(source) stop for some reason,the data will lose during this period。

Currently, there is no sink to redis.

Can elaborate on how to use the the Redis Sink and the Redis source. Am I suppose to modify the config properties file, or the java file and then package with mvn ? Thanks!

bin/flume-ng agent -n -agent -c conf -f conf/example-PublishDrivenSink.properties error

bin/flume-ng agent -n agent -c conf -f conf/example-PublishDrivenSink.properties correct

One more -

Hi
I have 1277 events/s from redis cluster in Pub/Sub mode to flume redis sub source. But just 2 min,data link will break. Is something wrong? or some by limited?

when 400 events/s is ok.

( redis cluster -----> flume redis source -----> kafka channel )