Git Product home page Git Product logo

lua-resty-redis-connector's Introduction

lua-resty-redis-connector

Build Status

Connection utilities for lua-resty-redis, making it easy and reliable to connect to Redis hosts, either directly or via Redis Sentinel.

Synopsis

Quick and simple authenticated connection on localhost to DB 2:

local redis, err = require("resty.redis.connector").new({
    url = "redis://[email protected]:6379/2",
}):connect()

More verbose configuration, with timeouts and a default password:

local rc = require("resty.redis.connector").new({
    connect_timeout = 50,
    send_timeout = 5000,
    read_timeout = 5000,
    keepalive_timeout = 30000,
    password = "mypass",
})

local redis, err = rc:connect({
    url = "redis://127.0.0.1:6379/2",
})

-- ...

local ok, err = rc:set_keepalive(redis)  -- uses keepalive params

Keep all config in a table, to easily create / close connections as needed:

local rc = require("resty.redis.connector").new({
    connect_timeout = 50,
    send_timeout = 5000,
    read_timeout = 5000,
    keepalive_timeout = 30000,

    host = "127.0.0.1",
    port = 6379,
    db = 2,
    password = "mypass",
})

local redis, err = rc:connect()

-- ...

local ok, err = rc:set_keepalive(redis)

connect can be used to override some defaults given in new, which are pertinent to this connection only.

local rc = require("resty.redis.connector").new({
    host = "127.0.0.1",
    port = 6379,
    db = 2,
})

local redis, err = rc:connect({
    db = 5,
})

DSN format

If the params.url field is present then it will be parsed to set the other params. Any manually specified params will override values given in the DSN.

Note: this is a behaviour change as of v0.06. Previously, the DSN values would take precedence.

Direct Redis connections

The format for connecting directly to Redis is:

redis://USERNAME:PASSWORD@HOST:PORT/DB

The USERNAME, PASSWORD and DB fields are optional, all other components are required.

Use of username requires Redis 6.0.0 or newer.

Connections via Redis Sentinel

When connecting via Redis Sentinel, the format is as follows:

sentinel://USERNAME:PASSWORD@MASTER_NAME:ROLE/DB

Again, USERNAME, PASSWORD and DB are optional. ROLE must be either m or s for master / slave respectively.

On versions of Redis newer than 5.0.1, Sentinels can optionally require their own password. If enabled, provide this password in the sentinel_password parameter. On Redis 6.2.0 and newer you can pass username using sentinel_username parameter.

A table of sentinels must also be supplied. e.g.

local redis, err = rc:connect{
    url = "sentinel://mymaster:a/2",
    sentinels = {
        { host = "127.0.0.1", port = 26379 },
    },
    sentinel_username = "default",
    sentinel_password = "password"
}

Proxy Mode

Enable the connection_is_proxied parameter if connecting to Redis through a proxy service (e.g. Twemproxy). These proxies generally only support a limited sub-set of Redis commands, those which do not require state and do not affect multiple keys. Databases and transactions are also not supported.

Proxy mode will disable switching to a DB on connect. Unsupported commands (defaults to those not supported by Twemproxy) will return nil, err immediately rather than being sent to the proxy, which can result in dropped connections.

discard will not be sent when adding connections to the keepalive pool

Disabled commands

If configured as a table of commands, the command methods will be replaced by a function which immediately returns nil, err without forwarding the command to the server

Default Parameters

{
    connect_timeout = 100,
    send_timeout = 1000,
    read_timeout = 1000,
    keepalive_timeout = 60000,
    keepalive_poolsize = 30,

    -- ssl, ssl_verify, server_name, pool, pool_size, backlog
    -- see: https://github.com/openresty/lua-resty-redis#connect
    connection_options = {},

    host = "127.0.0.1",
    port = "6379",
    path = "",  -- unix socket path, e.g. /tmp/redis.sock
    username = "",
    password = "",
    sentinel_username = "",
    sentinel_password = "",
    db = 0,

    master_name = "mymaster",
    role = "master",  -- master | slave
    sentinels = {},

    connection_is_proxied = false,

    disabled_commands = {},
}

API

new

syntax: rc = redis_connector.new(params)

Creates the Redis Connector object, overriding default params with the ones given. In case of failures, returns nil and a string describing the error.

connect

syntax: redis, err = rc:connect(params)

Attempts to create a connection, according to the params supplied, falling back to defaults given in new or the predefined defaults. If a connection cannot be made, returns nil and a string describing the reason.

Note that params given here do not change the connector's own configuration, and are only used to alter this particular connection operation. As such, the following parameters have no meaning when given in connect.

  • keepalive_poolsize
  • keepalive_timeout
  • connection_is_proxied
  • disabled_commands

set_keepalive

syntax: ok, err = rc:set_keepalive(redis)

Attempts to place the given Redis connection on the keepalive pool, according to timeout and poolsize params given in new or the predefined defaults.

This allows an application to release resources without having to keep track of application wide keepalive settings.

Returns 1 or in the case of error, nil and a string describing the error.

Utilities

The following methods are not typically needed, but may be useful if a custom interface is required.

connect_via_sentinel

syntax: redis, err = rc:connect_via_sentinel(params)

Returns a Redis connection by first accessing a sentinel as supplied by the params.sentinels table, and querying this with the params.master_name and params.role.

try_hosts

syntax: redis, err = rc:try_hosts(hosts)

Tries the hosts supplied in order and returns the first successful connection.

connect_to_host

syntax: redis, err = rc:connect_to_host(host)

Attempts to connect to the supplied host.

sentinel.get_master

syntax: master, err = sentinel.get_master(sentinel, master_name)

Given a connected Sentinel instance and a master name, will return the current master Redis instance.

sentinel.get_slaves

syntax: slaves, err = sentinel.get_slaves(sentinel, master_name)

Given a connected Sentinel instance and a master name, will return a list of registered slave Redis instances.

Author

James Hurst [email protected]

Licence

This module is licensed under the 2-clause BSD license.

Copyright (c) James Hurst [email protected]

All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

  • Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

  • Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

lua-resty-redis-connector's People

Contributors

alasdairnicol avatar andrewsharpe avatar hamishforbes avatar invizory avatar pintsized avatar piotrp avatar rainest avatar ryaneorth avatar stockrt avatar tangwz avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

lua-resty-redis-connector's Issues

read_timeout cannot be set

In connector.lua you have:

r:set_timeout(self, config.read_timeout)

which passes connector object instance as timeout. It should be changed to:

r:set_timeout(config.read_timeout)

I have two redis one master and one from use , how to Read and write separation use lua-resty-redis-connector

local rc = require("resty.redis.connector").new({
    connect_timeout = 100,
    keepalive_poolsize = 100
})
local hosts = {
    {host = "127.0.0.1", db = 7, role="master",port = 7379}, --master 
    {host = "127.0.0.1", db = 7, role="slave",port = 18379} --slave  it's not work 
}

local redis, err, previous_errors = rc:try_host(hosts)
redis:set("dog4","1")
ngx.say(cjson.encode(previous_errors)),      why previouse_errors is nil ???

I have two redis one master and one from use , how to Read and write separation use lua-resty-redis-connector

sentinel.get_master method

Please describe how to use method sentinel.get_master method
after I get connected redis after connect_via_sentinel

ERR invalid DB index in 0.09

I believe this commit introduced a regression in the public connect interface that throws the following error when a url without a trailing /N exists where N is a valid DB index.

ERR invalid DB index

connect is passed a url matching redis://redis:6379 only.

Additionally, the connection object that is returned is a boolean.

Pinning the version to 0.08 resolves the issue.

why my Redis connection closed?

Hello,
I use this lua-resty-redis-connector to connect redis sentinel. But when I subscribe the redis channel and read_reply in a while true loop, I received closed error sometimes. My code:
The function redis_conn which return a redis-sentinel connection.

redis_conn = function(self)
    local rc = connector.new()
    rc:set_connect_timeout(1000) -- Sets the cosocket connection timeout, in ms.
    rc:set_read_timeout(5000)

    local red, err = rc:connect {
        sentinels = self.config['sentinels'],
        master_name = self.config['master_name'],
        -- db = self.conffig['db'],
    }

    if not red then
        return nil, err
    end

    local ok, err = red:psubscribe(unpack(self.config['channels']))

    if not ok then
        return nil, err
    end

    return red, nil
end

And the main loop.

while true do

    if wb.fatal then
        red:punsubscribe(pusher.config["channels"])
        ngx.log(ngx.ERR, "websocket has disconnected:  ", err)
        return ngx.exit(444)
    end

    if ngx.worker.exiting() then
        return ngx.exit(444)
    end

    local res, err = red:read_reply()

    if not res then
        if err ~= "timeout" then
            ngx.log(ngx.ERR, "Redis read error: ", err)  -- Sometimes I receive "Redis read error: closed,"
            return ngx.exit(444)
        end
    else
        if g_list[res[3]] then
            local ok, err = pusher.ws_send(wb, "msg", res[4])
            if not ok then
                red:punsubscribe(pusher.config["channels"])
                ngx.log(ngx.ERR, "failed to send text: ", err)
                return ngx.exit(444)
            end
        end
    end
end

sentinel set_keepalive without parameters

The following invocation in connect.lua (currently line 247) has no parameters:
sentnl:set_keepalive()

Could someone kindly tell me what is the default parameter under open-resty, and is this expected?

use sentinel connect find the master IP is the pod ip

I used the sentinel to connect, is the pod IP.
Also, how to use get_master method. my Code is
local rc = require("resty.redis.connector").new(redis)

local redis, err = rc:connect{
url = "sentinel://mymaster:a/2",
sentinels = {
{ host = "10.62.108.88", port = 32641 },
}
}
if not redis then
ngx.say("close redis error : ", err)
end

local master, err = require("resty.redis.sentinel").get_master(sentinels, master_name)
if not master then
ngx.say("close redis error : ", err)
end

I already changed the sentinels in the default table. like:
sentinels = {
{ host = "10.62.108.88", port = 32641 },
}
the Output is

2019/02/27 02:09:43 [error] 32187#32187: *46 lua tcp socket connect timed out, when connecting to 172.30.63.147:6379, client: 127.0.0.1, server: , request: "GET /lua_redis_basic HTTP/1.1", host: "localhost:800"
2019/02/27 02:09:43 [error] 32187#32187: *46 lua entry thread aborted: runtime error: ...y/lua-resty-redis-connector/lib/resty/redis/sentinel.lua:18: attempt to index local 'sentinel' (a nil value)
stack traceback:
coroutine 0:
...y/lua-resty-redis-connector/lib/resty/redis/sentinel.lua: in function 'get_master'
/usr/local/openresty/nginx/conf/test.lua:14: in function </usr/local/openresty/nginx/conf/test.lua:1>, client: 127.0.0.1, server: , request: "GET /lua_redis_basic HTTP/1.1", host: "localhost:800"
curl: (52) Empty reply from server

what can I do? Thanks! @pintsized @andrewsharpe @stockrt

Example of sentinel master failover

I'm trying to understand the best way to deal with master failover. Is this something that the module does inherently? Or does every application need to implement this?

What is the best way to do this? And example would be very helpful

rc = require("resty.redis.connector").new() get stack traceback

6 function set_ngx_cache()
7 local rc = require("resty.redis.connector").new()
8 local sentinel, err = rc:connect {
9 url = "redis://172.16.1.21:26371",
10 url = "redis://172.16.1.31:26372",
11 url = "redis://172.16.1.32:26373"
12 }
13 local redis_monitor = 'zpredis'
14 assert(sentinel and not err, "sentinel should connect without errors")
15
16 local slaves, err = require("resty.redis.sentinel").get_slaves(sentinel,redis_monitor)
17 assert(slaves and not err, "slaves should be returned without error")
18 sentinel:close()

After running for a few hours, the following error occurred.
[error] 19806#0: *143951211 lua entry thread aborted: runtime error: /usr/local/openresty/scripts/init.lua:14: sentinel should connect without errors
stack traceback:
coroutine 0:
[C]: in function 'assert'
/usr/local/openresty/scripts/init.lua:14: in function 'set_ngx_cache'

where is the problem?

closed socket when I enable set_keepalive

This is my config:

local rc = require("resty.redis.connector").new({
                connect_timeout = 50,
                read_timeout = 5000,
                keepalive_timeout = 30000
            })

            local redis, err1 = rc:connect({
                url = "redis://127.0.0.1:6379",
            })

            if not redis then
                ngx.say( err1)    
            else
                ngx.say('connection fine')    
            end

            local ok, err = rc:set_keepalive(redis)  

I'm getting this error:
[error] 4044#0: *1068056 attempt to send data on a closed socket: u:00007F2BABA05D48, c:0000000000000000, ft:0 eof:0, client: MY IP, server: , request: "GET /foo HTTP/1.1", host: "SERVER IP"

When I disable keepalive setting, get this error:
[error] 4147#0: *1089971 lua tcp socket connect timed out, when connecting to 127.0.0.1:6379,

When I comment out the line related to rc:set_keepalive it works on single requests and crashes on 100 requests.

Sentinels & Connection Pooling

This seems to go around for some time, but nothing i found suggests a reliable fix.

I am connecting via sentinels

local rc = require("resty.redis.connector").new({
connect_timeout = 1000,
read_timeout = 5000,
master_name = "redis-cluster",
sentinels = redisSentinels,
role = "master"
})

local redis, err = rc:connect()

Whenever i do

local ok, err = rc:set_keepalive(redis)

I am no longer able to send commands due to

attempt to send data on a closed socket

This is used in a location access_by_lua_file

redis:get_reused_times() returns the number of reuses correctly, but i cannot figure out why i get the closed socket issue

I'm using v0..11 of the lib and openresty 1.19.9.1

Any suggestions ?

ERR max number of clients reached

local args = ngx.req.get_uri_args();
local tvid = args["tvid"]
local page = args["page"]
local message = "success"
local code = "A000000"
local time_now = ngx.time()
local timestamp = os.date("%Y%m%d%H%M%S",time_now)
local resp={}
resp["timestamp"]= timestamp

local rc = require("resty.redis.connector").new({
connect_timeout = 5000,
read_timeout = 5000,
keepalive_timeout = 30000,
keepalive_poolsize = 3000;
})

local sentinel_name = config.sentinel_name
local sentinel_passwd = config.sentinel_passwd
local hash_db_index = config.hash_db_index
local sentinel_nodes = config.sentinel_nodes
local sentinel_url = "sentinel://"..sentinel_passwd.."@"..sentinel_name..":m/"..tostring(hash_db_index)
local redis, err = rc:connect{
url = sentinel_url,
sentinels = sentinel_nodes
}

if err then
ngx.log(ngx.ERR,"connect redis sentinel failed,reason:",err)
message="failed"
code = "A000001"
resp["message"] = message
resp["code"] = code
resp["data"] = {}
ngx.print(cjson.encode(resp))
return 500
end

local h_key = tvid.."_"..page
local results, err = redis:hgetall(h_key)

if err or #results<=0 then
message="failed"
code = "A000001"
resp["message"] = message
resp["code"] = code
resp["data"] = {}
ngx.print(cjson.encode(resp))
ngx.log(ngx.ERR,"hgetall:",err)
else
resp["message"] = message
resp["code"] = code
local array={}
for i, res in ipairs(results) do
if i%2==0 then
print(i)
k = results[i-1]
v = results[i]
array[k]=v
end
end
resp["data"] = array
--print(cjson.encode(resp))
ngx.print(cjson.encode(resp))
end
--[[
local ok, err = redis:set_keepalive(10000,1000)
if not ok then
ngx.log(ngx.ERR,"failed to set_keepalive: ", err)
--return 200
end
]]
local ok, err = rc:set_keepalive(redis)
if not ok then
ngx.log(ngx.ERR,"failed to set_keepalive: ", err)
return 200
end

hi,i use ab command press the sentinel ,and the total connections is 50k,concurrent connections is 10k,nginx log appear error " ERR max number of clients reached ",and the error's count is very much.

Client ssl certificate authentication with sentinel

Hi,

Is it possible to send certificates as part of the call below when connecting to sentinel? I have setup redis and sentinel using TLS, but don't know how can clients connects to TLS Sentinel.

local redis, err = rc:connect {
url = "sentinel://mymaster:m",
sentinels = {},
password = "password",
-- tls = {
-- cert = "",
-- key = "",
-- cacert = ""
-- }
}

I am trying to do the following using lua-resty-redis-connector.
redis-cli -p 26379
--tls
--cert /etc/redis/tls/redis.crt
--key /etc/redis/tls/redis.key
--cacert /etc/redis/tls/ca.crt

Bug when more than two slaves on localhost

We use a master and two slaves on our staging server (using Redis Sentinel), but I recently added two more slaves to test something. redis-connector (0.03-0) is crashing when there are more than two slaves running on localhost:

2017/07/12 08:07:15 [error] 19066#0: *10806 lua entry thread aborted: runtime error: /usr/share/luajit/share/lua/5.1/resty/redis/connector.lua:169: invalid order function for sorting
stack traceback:
coroutine 0:
        [C]: in function 'tbl_sort'
        /usr/share/luajit/share/lua/5.1/resty/redis/connector.lua:169: in function 'connect'

This shouldn't be a problem in production, since running multiple slaves on the same host doesn't make a lot of sense.

What is the current status of the connector?

Hi, thanks for working on this.

Does "Redis Cluster support." under TODO in the README mean that it cannot be used to do queries yet? There's no example in the README.

Thanks in advance

auth/select of connection reuse issue

Hello,

I have an auth/select of connection reuse issue.

This my env:
redis-server: 127.0.0.1:6379
auth: 123456

when I connect to redis-server. every connection method all to auth, but the connection may be a pool connection, and it needn't to auth.

I can create a single redis-connector for my code, however, the openresty not allowed to do that, please see:
openresty/lua-resty-redis#44

BTW, Could we do something on get_reused_times() of lua-resty-redis. If the return times > 0, then the connection is reused, so we ignore auth and select?

Please tell me how to working it. Thanks!

no request object found

at code

local redis, err = rc:connect{ url = 'redis://h:[email protected]:999' }

i've got following error

2016-06-24T17:02:33.635965+00:00 app[web.1]: 2016/06/24 17:02:32 [error] 62#0: [lua] kong.lua:168: init(): Startup error: /app/.heroku/share/lua/5.1/kong/tools/utils.lua:211: /app/.heroku/share/lua/5.1/resty/redis/connector.lua:70: no request object found

i don't get it. is the url string malformed ?

Re-create and subscribe to the same PUBSUB channel in case there is a Redis failover

We use this code to connect to HA Redis, create/subscribe a PUBSUB channel and read out the published message:
local sub = assert(redis_conf.redis_connector:connect()) local _, err = sub:subscribe(redis_conf.channel) ... local bytes, error = sub:read_reply()

When testing on a HA Redis failover scenario, the existing PUBSUB channels are gone when the master Redis goes down. We were expecting the PUBSUB channel with the same name would be created and subscribed automatically by lua-resty-redis-connector in the newly promoted master node, but it didn't happen.

Is this something lua-resty-redis-connector handles? Or our code needs to explicitly handle this failure scenario?

Thanks.

Missing luarocks release

The current release in github is 0.03, whereas it is 0.02 in luarocks. Can you please make a new luarocks release?

nginx: [error] init_by_lua_file error: /tmp/a/resty/redis/connector.lua:70: no request object found for sentinel connection

Hi @pintsized I'm getting "no request object found" error on sentinel connection. can you please help me on this error.

nginx: [error] init_by_lua_file error: /tmp/a/resty/redis/connector.lua:70: no request object found stack traceback: [C]: in function 'ngx_re_match' /tmp/a/resty/redis/connector.lua:70: in function 'parse_dsn' /tmp/a/resty/redis/connector.lua:104: in function 'connect' /opt/nginx/conf/script/push_data.lua:30: in main chunk nginx: configuration file /opt/nginx//conf/nginx.conf test failed

connection string as below

local redis, err = rc:connect{ url = "sentinel://mymaster:a", sentinel = { host = "127.0.0.1", port = "26379" } }

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.