Two serious issues about redisson HOT 31 CLOSED

1.Sentinel not support auth password.（I have solve this problem）

fixed

from redisson.

1. When redis sentinel occur failover，redisson can not failover to new master automatic

Could you provide more details about this case? Sentinel not switching master at a moment it waits down-after-milliseconds

Have you seen "+switch-master ..." command in your sentinel log?

from redisson.

@lianghaijian can't reproduce it

from redisson.

Wait a moment,I will reproduce it,and provide more details.

from redisson.

My test code:

  @Test
    public void testSentinelWithAuth() throws InterruptedException {
        Config config = new Config();
        config.useSentinelConnection().setPassword("foobared")
                .setMasterName("mymaster")
                .addSentinelAddress("192.168.27.111:26379", "192.168.27.112:26379");
        Redisson redisson = Redisson.create(config);
        final List<User> list = redisson.getList("cms_list");
        Runnable r = new Runnable() {
            @Override
            public void run() {
                long begin = System.currentTimeMillis();
                for(int i=0;i<10000;i++){
                    try{
                        list.add(new User());
                    }catch (Exception e){
                        e.printStackTrace();
                    }
                }
                System.out.println("coast:" + (System.currentTimeMillis() - begin) / 1000);
            }
        };
        List<Thread> threads = new ArrayList<Thread>(Runtime.getRuntime().availableProcessors()*2);
        for(int i=0;i<Runtime.getRuntime().availableProcessors()*2;i++){
            Thread t = new Thread(r);z
            t.start();
            threads.add(t);
        }
        for(Thread t : threads){
            t.join();
        }
    }

After i killed the master redis server process,redis sentinel log:

[19545] 14 Jul 11:17:03.882 # +failover-end master mymaster 192.168.27.111 6379
[19545] 14 Jul 11:17:03.882 # +switch-master mymaster 192.168.27.111 6379 192.168.27.112 6379
[19545] 14 Jul 11:17:03.882 * +slave slave 192.168.27.111:6379 192.168.27.111 6379 @ mymaster 192.168.27.112 6379
[19545] 14 Jul 11:17:08.969 # +sdown slave 192.168.27.111:6379 192.168.27.111 6379 @ mymaster 192.168.27.112 6379

list.add(new User()); will throw timeout exception,next loop can not get new master connection.

com.lambdaworks.redis.RedisException: Command timed out
    at com.lambdaworks.redis.RedisAsyncConnection.await(RedisAsyncConnection.java:1195)
    at com.lambdaworks.redis.RedisConnection.await(RedisConnection.java:852)
    at com.lambdaworks.redis.RedisConnection.rpush(RedisConnection.java:489)
    at org.redisson.RedissonList.addAll(RedissonList.java:134)
    at org.redisson.RedissonList.add(RedissonList.java:85)
    at org.redisson.SentinelAuthTest$1.run(SentinelAuthTest.java:48)
    at java.lang.Thread.run(Thread.java:744)

from redisson.

My computer :2.7 GHz Intel Core i7 16 GB 1600 MHz DDR3,
The test result is not satisfactory,an average of 750 times per second

from redisson.

It's a bug,when a connection can not be used,it always be return back to pool

@Override
    public boolean addAll(Collection<? extends V> c) {
        RedisConnection<String, Object> conn = connectionManager.connectionWriteOp();
        try {
            conn.rpush(getName(), c.toArray());
            return true;
        } finally {
            connectionManager.releaseWrite(conn);
        }
    }

Because the pool have connections,i can't get a new master connection from pool

    @Override
    public <K, V> RedisConnection<K, V> connectionWriteOp() {
        acquireMasterConnection();

        RedisConnection<K, V> conn = masterConnections.poll();
        if (conn != null) {
            return conn;
        }

        conn = masterClient.connect(codec);
        if (config.getPassword() != null) {
            conn.auth(config.getPassword());
        }
        return conn;
    }

from redisson.

Oh, could you update from master? i think i fixed this issue. All info about master server now in holds in one entry-object.

from redisson.

here is my change - 9bd40f1

from redisson.

Sorry,i will update and try immediately

from redisson.

com.lambdaworks.redis.RedisException: Command timed out
    at com.lambdaworks.redis.RedisAsyncConnection.await(RedisAsyncConnection.java:1195)
    at com.lambdaworks.redis.RedisConnection.await(RedisConnection.java:852)
    at com.lambdaworks.redis.RedisConnection.rpush(RedisConnection.java:489)
    at org.redisson.RedissonList.addAll(RedissonList.java:134)
    at org.redisson.RedissonList.add(RedissonList.java:85)
    at org.redisson.SentinelTest$1.run(SentinelTest.java:49)
    at java.lang.Thread.run(Thread.java:744)

When redis occur failover,the exception above coast too much time,do you have any plan to resolve it.

from redisson.

Yes, i'll think how to resolve it. What about bug with master change?

from redisson.

The bug has been resolved, your solution is the same as I think, hahahaha.

from redisson.

What about exception handling how did you done it?

from redisson.

Try again a few times,if still failed,warning.I have seen your code,if a connection is closed,this connection can still return to the queue?I know in the lastest version jedis,jedis client close() will check if client is close before return to commons pool.

from redisson.

If channel enters to "Inactive" state then com.lambdaworks.redis.protocol.ConnectionWatchdog going to work.

from redisson.

Check the last commit. Did you mean this problem?

from redisson.

@lianghaijian i just complete operation auto-repeat for RedissonMap, you don't need to catch exceptions anymore then master or slave is down. Check it out ;)

from redisson.

I am facing same issue. One of the sentinels in my config is down and i can not start server as redisson always try to connect the last one in config.

from redisson.

@prarthanb

always try to connect the last one in config

and not the server which has down? it's never connects to sentinel which has down?

from redisson.

Always connects to last one even if it is down. And never retries other sentinels in the config.

from redisson.

ok, i'll fix it

from redisson.

Having sentinel failing over to slave but redisson refuse to reconnect to new master issue:

I have two nodes: both runs sentinel and redis server on default ports. When I send a DEBUG SEGFAULT to the master I get the following:

On slave node:
10.211.55.23:26379> SUBSCRIBE +switch-master
Reading messages... (press Ctrl-C to quit)

1. "subscribe"
2. "+switch-master"
3. (integer) 1
4. "message"
5. "+switch-master"
6. "redis 10.211.55.24 6379 10.211.55.23 6379"

On master node:
10.211.55.24:26379> SUBSCRIBE +sdown
Reading messages... (press Ctrl-C to quit)

1. "subscribe"
2. "+sdown"
3. (integer) 1
4. "message"
5. "+sdown"
6. "master redis 10.211.55.24 6379"
7. "message"
8. "+sdown"
9. "slave 10.211.55.24:6379 10.211.55.24 6379 @ redis 10.211.55.23 6379"

However on the redisson output I am getting "WARN org.redisson.connection.SentinelConnectionManager:onSlaveDown(180) - Invalid message: master redis 10.211.55.24 6379 from Sentinel localhost:26379 " warning message but the connections is not re-establishing.

from redisson.

@jackygurui I fixed your problem. Please check

from redisson.

Hi @mrniko

Thanks for getting back so quick. I have tried the same scenario with the latest master code, upon having sent DEBUG SEGFAULT to the master, I am still getting an error message saying it can't connect to the already dead node. It has not failed over to the new master.

20:06:09,240 WARN org.redisson.connection.RoundRobinLoadBalancer:nextConnection(197) - Can't connect to /10.211.55.24:6379, trying next connection!
20:06:11,440 WARN org.redisson.connection.RoundRobinLoadBalancer:nextConnection(197) - Can't connect to /10.211.55.24:6379, trying next connection!
20:06:13,640 WARN org.redisson.connection.RoundRobinLoadBalancer:nextConnection(197) - Can't connect to /10.211.55.24:6379, trying next connection!
20:06:15,839 WARN org.redisson.connection.RoundRobinLoadBalancer:nextConnection(197) - Can't connect to /10.211.55.24:6379, trying next connection!

from redisson.

could you decrease the timeout and try again? because after timeout i see the master is in the game and requests go forward

from redisson.

@jackygurui please check it again

from redisson.

@prarthanb

I am facing same issue. One of the sentinels in my config is down and i can not start server as redisson always try to connect the last one in config.

So the problem appears during redisson start only, right?

from redisson.

@prarthanb i fixed this case. please check too

from redisson.

thanks.

On Fri, Aug 14, 2015 at 7:51 AM, Nikita Koksharov [email protected]
wrote:

@prarthanb https://github.com/prarthanb i fixed this case. please check
too

—
Reply to this email directly or view it on GitHub
#41 (comment).

from redisson.

@mrniko Thanks. I think it is working now.

from redisson.