onyx-platform / onyx-redis Goto Github PK

View Code? Open in Web Editor NEW

6.0 3.0 8.0 425 KB

Onyx plugin for Redis

Home Page: http://www.onyxplatform.org/

Clojure 97.02% Shell 2.98%

onyx-redis's Introduction

onyx-redis

Onyx plugin for redis.

Installation

In your project file:

[org.onyxplatform/onyx-redis "0.14.5.0"]

In your peer boot-up namespace:

(:require [onyx.plugin.redis])

Functions

Write to Redis

Example:

{:onyx/name :out-to-redis
 :onyx/plugin :onyx.plugin.redis/writer
 :onyx/type :output
 :onyx/medium :redis
 :redis/uri "redis://127.0.0.1:6379"
 :redis/allowed-commands [:set :get]
 :onyx/batch-size batch-size}

Attributes

key	type	description
`:redis/uri`	`string`	Redis uri
`:redis/read-timeout-ms`	`int`	Time to wait (in ms) before giving up on trying to write to Redis.
`:redis/allowed-commands`	`vector` of `keyword`	Optional list of Redis commands that are allowed (default: all).

Segments should be supplied to the plugin in the form:

{:op :sadd :args [redis-key redis-value]}

Where op is a keyword that corresponds 1:1 to a Redis command function in taoensso.carmine, e.g. :sadd -> taoensso.carmine/sadd. Other examples are :set, :del, :pfadd, :zadd, :lpush. For more information on the commands supported by carmine, see its documentation.

Inject Connection Spec lifecycle

Injects an carmine connection spec into the event map. Will also inject as an :onyx/fn param if :onyx/param? is true.

{:lifecycle/task :use-redis-task
 :lifecycle/calls :onyx.plugin.redis/reader-conn-spec
 :redis/uri "redis://127.0.0.1:6379"
 :redis/read-timeout-ms <<optional-timeout>>
 :onyx/param? true
 :lifecycle/doc "Initialises redis conn spec into event map, or as a :onyx.core/param"}

When used with task :use-redis-task and with :onyx/param? true, you can use it from your function e.g.

(ns your.ns
  (:require [taoensso.carmine :as car :refer (wcar)]))

(defn use-redis-task-fn [conn segment]
  (wcar conn
        [(car/smembers (:key segment)])))

Alternatively, the conn that is created is placed under :redis/conn in the event map, for use within lifecycles e.g. :before-batch. Use in this way allows requests to be batched.

Please see the Carmine documentation for examples for how to use Carmine.

Contributing

Pull requests into the master branch are welcomed.

License

Distributed under the Eclipse Public License, the same as Clojure.

onyx-redis's People

Contributors

Stargazers

Watchers

Forkers

bcambel teaforthecat rasom tcoupland clojurians-org workfloapp arnaudbos cogfor

onyx-redis's Issues

Does it support using redis as queue?

Hello,

Thanks for plugin!

I want to use redis as a message queue. Carmine library, that you use for communication with redis supports it. It even supports durable queues. So, is it possible to adopt this plugin to support such types of communication with redis?

Carmine message queue
Carmine pub sub

Thank you,
Aleh

Enable connection pooling

Connection pooling is currently disabled, as https://github.com/gardnervickers had some issues with it in the past. It should be re-enabled if possible as connection pooling can be important

Improve error reporting for unsupported Redis ops

Currently, if an operation is not supported, onyx-redis throws a NullPointerException (see below). It would be great if it would fail with a more specific exception and a message describing that the operation is not currently supported.

16-07-05 14:07:02 columba WARN [onyx.peer.task-lifecycle] - 
                                                            java.lang.Thread.run              Thread.java:  745
                              java.util.concurrent.ThreadPoolExecutor$Worker.run  ThreadPoolExecutor.java:  617
                               java.util.concurrent.ThreadPoolExecutor.runWorker  ThreadPoolExecutor.java: 1142
                                                                             ...                               
                                               clojure.core.async/thread-call/fn                async.clj:  439
                                       onyx.peer.task-lifecycle.TaskLifeCycle/fn       task_lifecycle.clj:  512
                                     onyx.peer.task-lifecycle/run-task-lifecycle       task_lifecycle.clj:  354
onyx.lifecycles.lifecycle-invoke/invoke-task-lifecycle-gen/invoke-task-lifecycle     lifecycle_invoke.clj:   53
                                                                             ...                               
                         onyx.lifecycles.lifecycle-invoke/restartable-invocation     lifecycle_invoke.clj:   21
                         onyx.lifecycles.lifecycle-invoke/restartable-invocation     lifecycle_invoke.clj:   23
                                                              clojure.core/apply                 core.clj:  651
                                                                             ...                               
                                  onyx.peer.task-lifecycle/eval28131/write-batch       task_lifecycle.clj:  252
                               onyx.peer.task-lifecycle/eval28131/write-batch/fn       task_lifecycle.clj:  254
                                       onyx.plugin.redis.RedisWriter/write-batch                redis.clj:   43
                                                onyx.plugin.redis.RedisWriter/fn                redis.clj:   43
                                             onyx.plugin.redis.RedisWriter/fn/fn                redis.clj:   43
                                         taoensso.carmine.protocol/with-replies*             protocol.clj:  341
                                          onyx.plugin.redis.RedisWriter/fn/fn/fn                redis.clj:   44
                                                              clojure.core/doall                 core.clj: 3114
                                                              clojure.core/dorun                 core.clj: 3099
                                                                clojure.core/seq                 core.clj:  137
                                                                             ...                               
                                                             clojure.core/map/fn                 core.clj: 2719
                                       onyx.plugin.redis.RedisWriter/fn/fn/fn/fn                redis.clj:   46
java.lang.NullPointerException: 
    clojure.lang.ExceptionInfo: Handling uncaught exception thrown inside task lifecycle - killing this job. -> Exception type: java.lang.NullPointerException. Exception message: null
       job-id: #uuid "61ef65d8-bea5-44c1-b95f-c9a3351e27ba"
     metadata: {}
      peer-id: #uuid "d1ac53a3-c39c-439d-b035-167c5237f12f"
      task-id: #uuid "44bb74b9-c95d-418c-b95e-1cf96b14c383"
    task-name: :redis

Completion/retry mechanism will lose segments on peer failure

Writes are requeued on retry, but if a peer crashes between the read and the retry the segment will be lost

Add support for HSET, HMSET etc.

A lot of the commands supported by carmine are currently not supported by onyx-redis. It would be nice to have a more complete coverage of the Redis API.