eleme / corvus Goto Github PK
View Code? Open in Web Editor NEWA fast and lightweight Redis Cluster Proxy for Redis 3.0
License: MIT License
A fast and lightweight Redis Cluster Proxy for Redis 3.0
License: MIT License
client : Lettuce,a aync client base netty nio
corvus version : a Secondary development version base eleme/corvus master(update to date)
redis: redis cluster
problem:
when run my code some time, i got some exceptions like this:
tcpdump:
I find some things wrong by tcpdump.
*2 $-1
should be *2 $-1 $-1
reproducer code:
public class RedisOptionV2 {
private static RedisAsyncCommands<String, String> ASYNC;
private static boolean flag = false;
private static AtomicLong index = new AtomicLong(0);
static {
RedisURI redisURI = RedisURI.create("corvus ip", 6666);
RedisClient redisClient = RedisClient.create(redisURI);
StatefulRedisConnection<String, String> connection = redisClient.connect();
ASYNC = connection.async();
}
@PostConstruct
void init() {
MyExecutorService.schedule(this::lettuceTestTask, 30, TimeUnit.SECONDS);
}
private void lettuceTestTask() {
for (long i = 0; i < Long.MAX_VALUE; i++) {
try {
MyExecutorService.submit(this::lettuceDemo);
} catch (Throwable t) {
t.printStackTrace();
}
}
}
public void lettuceDemo() {
try {
List<String> keys = new ArrayList<>();
keys.add("debugLettuceKey" + index.getAndIncrement());
if (flag) {
keys.add("debugLettuceKey2");
flag = false;
} else {
flag = true;
}
//core code
RedisFuture<List<KeyValue<String, String>>> future = ASYNC.mget(keys.toArray(new String[0]));
future.whenCompleteAsync((e, t) -> {
if (t == null) {
log.info("{} res: {}", keys, e);
} else {
log.error("调用redis失败, {} res : {}", keys, e, t);
}
}, MyExecutorService.getExecutorService());
} catch (Throwable t) {
log.error("fail to execute task", t);
}
}
}
Looking forward to reply!Thanks
Version: 0.2.3
Requests are slow during this period, because the redis instance processing cluster slots
is busy sending result back to corvus.
startup log:
2016-06-30 16:49:20.284 info corvus: [1757] starting slot manager thread
2016-06-30 16:49:20.288 info corvus: [1760] updating slot map using 10.0.15.60:7113
2016-06-30 16:49:20.300 info corvus: [1757] starting stats thread
2016-06-30 16:49:20.301 info corvus: [1757] serve at 0.0.0.0:9301
slot update logs:
https://gist.githubusercontent.com/wooparadog/ed40c5e284b95605e57851d789dbcb4a/raw/-
Is there any plan on this kind of commands supporting?
They are available when a redis node is in a cluster.
Corvus seems a good job, but I cannot understand what corvus' config file, and what corvus' cluster exactly means, and I also can't understand the logic of corvus.
there are some questions,
bind 1000 node localhost:6301,localhost:7000 thread 4
Could you kindly explain to us. Many Thanks!
增加了一些特性, 可以提交到主干. 现在这里的corvus还有人维护吗?
corvus metrics信息, 有没有命令直接取出来?
类似redis的info命令,看配置里面是要配合statsd才能取到数据,但是我们的监控环境没有statsd。能否提供命令或者api?
From my perspective, the limitation of not being able to get all KEYS
is a total dealbreaker.
Being sure, that this is only a design limitation made willingly, I want to ask politely if you really don't see any chance of implementing this anyhow.
Technically, it's just that all cluster nodes have to be SCAN
from corvus and responded to the client just as if it was one single instance. This may be slower, but better than not available.
Since SCAN
is also not supported, there seems to be no alternative for getting all keys in a cluster.
Is there any technical reason why this can't be done?
Currently corvus always listens on 0.0.0.0, making it globally reachable over the machine's interfaces.
Additionally, the configuration parameter is called bind
but sets the port corvus listens on.
It would be desirable to be able to configure the interface to listen to for incoming connection (e.g. 127.0.0.1 aka localhost only).
Redis also uses bind
for the interfaces and port
for the actual port it is listening to.
In addition, this would make it possible to add IPv6 support as well.
from two options read-slave
and read-master-slave
to one option like:
read-strategy
with values being:
master
defaultread-slave-only
send all read ops to slaves.both
send read ops to both master and slaves.I would like to point out that identifiers like "__CORVUS_H
" and "__PARSER_H
" do not fit to the expected naming convention of the C language standard.
Would you like to adjust your selection for unique names?
We should implement SELECT
command for compatibility. #113
Hello,
With v0.2.7 tag and redis-cluster which is configured with requirepass.
Found that the corvus doesn't send AUTH command to the backend redis cluster instance.
So I got "(error) NOAUTH Authentication required."
Twemproxy has redis_auth configuration which implements "Authenticate to the Redis server on connect."
Thanks!
Situation:
Metrics recorded by corvus is higher than it actually is. We tried to reproduce it in offline environment, but failed.
Sometimes connected_clients
and remote_latency
get negative value.
My node by cluster machine configurationed password ,but I don't know how to configuration password by corvus.
我创建了3个主节点 3个从节点 ,然后配置如下:
bind 12345
node 127.0.0.1:6379,127.0.0.1:6380,127.0.0.1:6381
thread 4
启动后用redis-benchmark压测发现tps降了
不用corvus是这样 GET: 33046.93 requests per second
用了corvus后 GET: 5262.88 requests per second
Just came across corvus and was wondering if you could clarify how to use it ?
I use my redis-generator.sh to create 3 stand alone redis servers locally. But is there anything else I need to do before starting up corvus ? Do I need to setup an actual redis cluster first or just standalone redis server instances ?
On CentOS 7.3, created 3 redis 3.2.8 based servers on TCP ports 6479, 6480, and 6481
./redis-generator.sh 3
Creating redis servers starting at TCP = 6479...
-------------------------------------------------------
creating redis server: redis6479.service [increment value: 0]
redis TCP port: 6479
create systemd redis6479.service
cp -a /usr/lib/systemd/system/redis.service /usr/lib/systemd/system/redis6479.service
create /etc/redis6479/redis6479.conf config file
mkdir -p /etc/redis6479
cp -a /etc/redis.conf /etc/redis6479/redis6479.conf
-rw-r----- 1 redis root 46K Mar 13 21:22 /etc/redis6479/redis6479.conf
-rw-r--r-- 1 root root 249 Sep 14 08:43 /usr/lib/systemd/system/redis6479.service
Created symlink from /etc/systemd/system/multi-user.target.wants/redis6479.service to /usr/lib/systemd/system/redis6479.service.
## Redis TCP 6479 Info ##
# Server
redis_version:3.2.8
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:dd923e72e9efa6d8
redis_mode:standalone
os:Linux 3.10.0-514.10.2.el7.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.8.5
process_id:28820
run_id:75012c9827a5090346297b039e405fbe8e20263e
tcp_port:6479
uptime_in_seconds:0
uptime_in_days:0
hz:10
lru_clock:13154149
executable:/etc/redis6479/redis-server
config_file:/etc/redis6479/redis6479.conf
-------------------------------------------------------
creating redis server: redis6480.service [increment value: 1]
redis TCP port: 6480
create systemd redis6480.service
cp -a /usr/lib/systemd/system/redis.service /usr/lib/systemd/system/redis6480.service
create /etc/redis6480/redis6480.conf config file
mkdir -p /etc/redis6480
cp -a /etc/redis.conf /etc/redis6480/redis6480.conf
-rw-r----- 1 redis root 46K Mar 13 21:22 /etc/redis6480/redis6480.conf
-rw-r--r-- 1 root root 249 Sep 14 08:43 /usr/lib/systemd/system/redis6480.service
Created symlink from /etc/systemd/system/multi-user.target.wants/redis6480.service to /usr/lib/systemd/system/redis6480.service.
## Redis TCP 6480 Info ##
# Server
redis_version:3.2.8
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:dd923e72e9efa6d8
redis_mode:standalone
os:Linux 3.10.0-514.10.2.el7.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.8.5
process_id:28887
run_id:05e6b7f63f98d9bb1e6b80c072b78aa3126dfdd4
tcp_port:6480
uptime_in_seconds:0
uptime_in_days:0
hz:10
lru_clock:13154149
executable:/etc/redis6480/redis-server
config_file:/etc/redis6480/redis6480.conf
-------------------------------------------------------
creating redis server: redis6481.service [increment value: 2]
redis TCP port: 6481
create systemd redis6481.service
cp -a /usr/lib/systemd/system/redis.service /usr/lib/systemd/system/redis6481.service
create /etc/redis6481/redis6481.conf config file
mkdir -p /etc/redis6481
cp -a /etc/redis.conf /etc/redis6481/redis6481.conf
-rw-r----- 1 redis root 46K Mar 13 21:22 /etc/redis6481/redis6481.conf
-rw-r--r-- 1 root root 249 Sep 14 08:43 /usr/lib/systemd/system/redis6481.service
Created symlink from /etc/systemd/system/multi-user.target.wants/redis6481.service to /usr/lib/systemd/system/redis6481.service.
## Redis TCP 6481 Info ##
# Server
redis_version:3.2.8
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:dd923e72e9efa6d8
redis_mode:standalone
os:Linux 3.10.0-514.10.2.el7.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.8.5
process_id:28954
run_id:2019276e1ec4c6a359f9b6e952c31b467e7a04fb
tcp_port:6481
uptime_in_seconds:0
uptime_in_days:0
hz:10
lru_clock:13154150
executable:/etc/redis6481/redis-server
config_file:/etc/redis6481/redis6481.conf
then setup corvus.conf file
bind 12345
node 127.0.0.1:6479,127.0.0.1:6480,127.0.0.1:6481
thread 4
# debug, info, warn, error
loglevel debug
syslog 0
# Close the connection after client idle for `client_timeout` seconds.
# No response after `server_timeout` seconds waiting for redis-server
# connection to redis-server will be closed.
#
# Value 0 means never timeout.
#
# Default 0
#
# client_timeout 30
# server_timeout 5
# Statsd config
# metrics:
# corvus.<cluster>.<host-port>.<value label>
#
# corvus.default.localhost-12345.connected_clients
# corvus.default.localhost-12345.completed_commands
# corvus.default.localhost-12345.used_cpu_sys
# corvus.default.localhost-12345.used_cpu_user
# corvus.default.localhost-12345.latency
# corvus.default.localhost-12345.redis-node.127-0-0-1-8000.bytes.{send,recv}
#
# Cluster annotation. Using `cluster` to add a cluster name to metrics.
#
# cluster default
#
# Metrices are sent using udp socket. Using `statsd` config to
# set statsd server address. Every `metric_interval` senconds
# will produce a set of metrices.
#
# statsd localhost:8125
# metric_interval 10
# Buffer size allocated each time avoiding fregments
# Buffer used in processing data recieving or sending
# Min buffer size is limited to 64 Bytes
# Default value is 16KBytes (16384)
#
# bufsize 16384
#
# Client should send AUTH <PASSWORD> if `requirepass` setted.
# Corvus will not forward this command, and do authentication just by itself.
# If it is given empty, it will be no effect and you can access the proxy with no password check.
#
# requirepass password
#
# Use `read-strategy` to config how to read from the cluster. It has three valid
# values:
#
# * `master`, forward all reading commands to master, the default
# * `read-slave-only`, forward all reading commands to slaves
# * `both`, forward reading commands to both master and slaves
#
# If new slaves are added to the cluster, `PROXY UPDATESLOTMAP` should be emmited
# to tell corvus to use the newly added slaves.
#
# read-strategy master
# Slowlog
# The following two configs are almost the same with redis.
# Every command whose lantency exceeds `slowlog-log-slower-than` will be considered a slow command,
# which will be sent to statsd and can also be retrieved using `slowlog get`.
# Note that the lantency here is the time spent in proxy,
# including redirection caused by MOVED and ASK.
# Both slowlog command and sending slowlog to statsd are disabled by default.
# You can enable both or either of them.
#
# A zero value will log every command.
# slowlog-log-slower-than 10000
#
# Set this to positive value to enable slowlog command.
# slowlog-max-len 1024
#
# Set this to 1 if you want to send slowlog to statsd.
# slowlog-statsd-enabled 0
then start corvus
src/corvus /etc/corvus/corvus.conf
2017-03-15 03:41:10,813 DEBUG [default 12345 28999 28999]: Registering signal[Interrupt] (corvus.c:62)
2017-03-15 03:41:10,814 DEBUG [default 12345 28999 28999]: Registering signal[Terminated] (corvus.c:62)
2017-03-15 03:41:10,814 DEBUG [default 12345 28999 28999]: Registering signal[Segmentation fault] (corvus.c:62)
2017-03-15 03:41:10,814 INFO [default 12345 28999 28999]: starting slot manager thread (slot.c:533)
2017-03-15 03:41:10,814 DEBUG [default 12345 28999 29001]: slowlog enabled (corvus.c:256)
2017-03-15 03:41:10,814 DEBUG [default 12345 28999 29002]: slowlog enabled (corvus.c:256)
2017-03-15 03:41:10,816 DEBUG [default 12345 28999 29003]: slowlog enabled (corvus.c:256)
2017-03-15 03:41:10,816 INFO [default 12345 28999 28999]: serve at 0.0.0.0:12345 (corvus.c:481)
2017-03-15 03:41:10,816 DEBUG [default 12345 28999 29004]: slowlog enabled (corvus.c:256)
2017-03-15 03:41:10,817 INFO [default 12345 28999 29000]: updating slot map using 127.0.0.1:6481 (slot.c:290)
2017-03-15 03:41:10,819 ERROR [default 12345 28999 29000]: parse_cluster_nodes: expect data type 2 got 5 (slot.c:189)
2017-03-15 03:41:10,819 INFO [default 12345 28999 29000]: updating slot map using 127.0.0.1:6480 (slot.c:290)
2017-03-15 03:41:10,820 ERROR [default 12345 28999 29000]: parse_cluster_nodes: expect data type 2 got 5 (slot.c:189)
2017-03-15 03:41:10,820 INFO [default 12345 28999 29000]: updating slot map using 127.0.0.1:6479 (slot.c:290)
2017-03-15 03:41:10,820 ERROR [default 12345 28999 29000]: parse_cluster_nodes: expect data type 2 got 5 (slot.c:189)
2017-03-15 03:41:10,820 WARN [default 12345 28999 29000]: can not update slot map (slot.c:351)
DEBUG output suggests not working well ?
Should send calculated metrics per 10s instead of sending a total number and let graphite calculate the increment.
So we can use banshee for alerting.
edit:
Add an command to retrieve top n accessed keys in corvus.
When a multiple key command is considered slow, besides adding itself to slowlog command queue, it should also add its slowest sub command so that we can determine whether there is a key or redis node slow down the whole command.
Would you like to add more error handling for return values from functions like the following?
The number of slot_create_job(SLOT_UPDATE)
can tell some very fundamental information about the health of redis cluster (if the number is too high, there might be a partition or resharding).
So, giving out this information is helpful for monitoring
Hello,
are there any plans to support Redis 5.0 or higher?
Thank you,
Apostolos
Hello,
When I set the slowlog-log-slower-than with big values like e.g. "24000", the slowlogs I get are also big e.g.
localhost:6380> config set slowlog-log-slower-than 24000
OK
localhost:6380> slowlog reset
OK
localhost:6380> slowlog get 2
In the same environment, when I set the slowlog-log-slower-than with small values "24", the slowlogs I get are also small and the big delays being displayed previously are disappeared:
localhost:6380> config set slowlog-log-slower-than 24
OK
localhost:6380> slowlog reset
OK
localhost:6380> slowlog get 2
So does Corvus understands the desired time unit (micro, mill, etc) from the config setting it so does display the result accordingly?
By the way, in my environment, the latencies are below 24ms so I can't interpret correctly the numbers like 282 and 318 ...
Thank you
Any suggestions on how to implement a simple prometheus health check for corvus?
I tried a simple TCP 'PING' request and expected 'PONG', but that didn't work.
Please share a simple 'telnet' or 'netcat' example command line for the suggestion. I can use that to automate the check on my end.
For better management, we need to have auth. We can put password/auth info in config file, instead of using the native auth in redis itself.
I see that the max clients I was able to reach is 468. Is it something hardcoded or I can change it via settings?
So that we can put alerts on them.
Great software, Does corvus support redis AUTH for instances or does it just accept a master like twemporxy? i.e.
myredis.com:6379 - AUTH mypass
myredis2.com:6379 - AUTH mypass2
thank you.
We should count how many requests are slow in corvus, and send it to statsd.
Thoughts:
For a specific $cluster, if one $cmd (like GET, SET...) has cost more than 10ms(can be configured),
increase the inmemory counter for that slow log category by 1. And send the combined result to statsd
every 10s
corvus 本身的 流量监控
slow query 的监控.
指标思路:
1. statsd.corvus.$cluster.slow_query.$cmd
2. statsd.corvus.$cluster.nodes.$node.slow_query.$cmd
corvus是否支持ssdb或者pika?
An extra null pointer check is not needed in functions like the following.
I am using this from a php client.
If some slots move to another node... redis-cluster usually tells me with MOVED ip:port response.
Does corvus maintain these informations so requests for this slot will then be redirected by corvus to the correct new node?
Idea is to prevent the redirect/connect overhead if slots are moved...
thanks max
Awesome stuff @maralla @xinming90 !
Corvus will be shared in our next RedisWeekly! Once published (in a matter of minutes) you will be able to see your project here :)
I wonder if supporting higher versions of redis is on the roadmap? it'd be awesome to see the new features introduced by redis 3.2.
Hi,
First, thanks for you great work! 🎉
Why do you do command splitting ?
Is it to reduce redis lock time of large MGET/MSET/DEL/... ?
Did you make some measurements ?
phpredis
Whats the status of using latest redis 4 with this proxy?
Cheers Max
问题:
1 qps非常高
2 corvus检测到redis连接不上时,会触发refresh
3 refresh的时候应该停掉server,因为如果不停掉server的端口,在qps非常高的情况下,请求堆积,corvus内存瞬间就打爆了
4如果在corvus和redis连接不可用的情况下,关闭server端口,客户端能做到识别从而不往他发请求
Do you have any plan do below function?
ERROR [30942]: cmd_parse_token: fail to parse command QUIT (command.c:716)
Hi,
I'm trying to install using the package from the release page, but I cannot see the package there for v0.2.7, the last one is v0.2.4.
I'm trying to compile from the source code(downloaded from the release page), but it is failing:
$ make
/Library/Developer/CommandLineTools/usr/bin/make jemalloc -C ../deps
cd jemalloc && ./configure --with-jemalloc-prefix=je_ --enable-cc-silence
/bin/sh: ./configure: No such file or directory
make[2]: *** [jemalloc] Error 127
make[1]: *** [../deps/jemalloc/lib/libjemalloc.a] Error 2
make: *** [corvus] Error 2
And I cannot run git submodule update --init
because this is not a git repository.
Can you provide corvus-0.2.7.tar.bz2. So we can install just running make
?
Hi.
We are currently working with nutcracker (twemproxy) in order to create our own redis cluster. However we are experiencing some scaling problems due to twemproxy is not multithreaded. In order to start testing with Corvus using the same existing cluster, we would like to know which key distribution algorithm is being used currently in order to know if we can use the same existing cluster to test in parallel both solutions.
If corvus use ketama as algorithm we would use is without any complex migration procedure for our persistent keys.
Thanks in advance.
The function "exit" does not belong to the list of async-signal-safe functions.
I guess that a different program design will be needed for your function "sig_handler".
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.