stickermule / rump Goto Github PK

Hello,

rump is a great tool however in certain situations having ttl set to 0 does not work okay (session data etc). May I suggest adding a separate flag for ttl with 0 default, so that default behavior does not change. Please see my local change at master...eugenetaranov:ttl, if you find it useful, I'll be happy to make a PR.

Hy,
I have an interesting issue by some redis clusters.
We have 3 redis clusters in AWS.
Old, New1, New2
When I do sync from old to new1, then everything goes fine.
When I do sync from old to new 2 I get following error:
redigo: unexpected element type for Strings, got type redis.Error

I can log into new2 with redis-cli, so don't see what could be wrong, compared to new1. new1 and new 2 are identical clusters, same instance size, count, etc..

Thank you

• I have read the contributing guide.

Report

• Dump remote Redis with rump and with AUTH to local rump file

$ ./rump -from redis://h:[email protected]:PORT/0 -to ./hero-redis.rump -ttl

rwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrw

SUCCESS

• Connect to the destination ElastiCache using redis_uri format and redis-cli

echo $REDIS_URL 
redis://[email protected]:6379

redis-cli -u ${REDIS_URL} INFO SERVER
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
# Server
redis_version:4.0.10
redis_git_sha1:0
redis_git_dirty:0
redis_build_id:0
redis_mode:standalone
os:Amazon ElastiCache
...

• [!] Restore data to the the destination ElastiCache

./rump -from ./hero-redis.rump -to redis://[email protected]:6379/0

rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr
signal: exit
NOAUTH Authentication required.

Notes

I have seen #26 and I have downloaded rump 1.0.0 from GitHub Release, but still facing the issue

• I have read the contributing guide.

Report

./rump -from /opt/Rump/data/redisu0.rump -to redis://:[email protected]:6379/2
signal: exit
w
redis write: exit
bufio.Scanner: token too long

Hello.
thank you very much for this. Very cool.

As the title says，in this case , it will lose some keys，will you consider covering this case?

In a test environment we did a test transfer between two redis cluster, and we found that there was a bigger number of differences between key numbers.
Dest node 1625684
Source node 1673187

How can this happen, or how can we mitigate these issues? How can we debug what cases them?
Thank you

• I have read the contributing guide.

Report

Sync with ttl terminated by 'ERR Invalid TTL value, must be >= 0'
The source AWS ElastiCache redis version is 3.2.10
rump -from redis://redis-002.ruf311.0001.cnw1.cache.amazonaws.com.cn:6379/0 -to redis://127.0.0.1:6379/0 -ttl
signal: exit

redis read: exit
ERR Invalid TTL value, must be >= 0

• I have read the contributing guide.

Report

I am hitting an error when running rump in a Kubernetes pod(if that matters). The failure appears to happen here but also reports a TCP i/o timeout.

Can you confirm whether this is a filesystem i/o issue or an issue talking to Redis?

Is it possible to set timeouts higher and prevent this issue?

I see around a 7x increase in CPU usage but still not over 55%.

Node type: cache.m3.medium
Engine: Redis
Engine Version Compatibility: 2.8.24

file write: exit

signal: exit
read tcp 10.X.X.10:57824->10.Z.Z.145:6379: i/o timeout

• I have read the contributing guide.

Report

Please update the docs for the ttl option. It is given without explanation and the example in the README appears incorrect (it lacks a value, which is apparently required).

1. What happens to records with a TTL when the ttl option is not given?
2. What happens when it is given?

Should rump include a "daemon mode"?

Hello.

First of all, thank you very much for this. Very cool.

I noticed that there's no test coverage and, if it's okay, I'd like to add/increase that? I'm on vacation for the next fortnight, but I can get a PR in before the end of the month?

Anthony.

• I have read the contributing guide.

Report

Currently rump will only deal with the specified URI database numbers, or, if those are omitted, it will use 0 as both source and destination. Having a way of moving all the source database numbers to the cluster would be nice

• I have read the contributing guide.

Report

Trying to migrate from one Redis system to a Digital Ocean managed Redis, I get EOF as the only output and no keys in the destination system, I've tried system to system, and I've also tried writing to a backup.rump and source that to a destination and get the same output.

Any suggestions?

rump -from redis://:password@test-redis:6379/0 -to redis://default:[email protected]:25061/0
EOF

Dumping to a file:

rump -from redis://:password@test-redis:6379/0 -to backup.rump
rwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwwrrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrwrw
signal: exit
done

Trying to restore from the file:

rump -from backup.rump -to rediss://default:[email protected]:25061/0
EOF

• I have read the contributing guide.

Report

I have been using RUMP last few months to Sync a primary redis endpoint with a failover (in the even of DR). Ive noticed that RUMP is actually copying keys across and not deleted keys that have been removed from primary. For example primary endpoint has around 166 thousand keys, while the failover now has 26 million. is there a flag i am missing to do this or is it not a-part of the functionality? Thank you

• I have read the contributing guide.

Report

Question: I have a redis cluster created from the bitnami helm chart here: https://github.com/bitnami/charts/tree/master/bitnami/redis-cluster/#installing-the-chart with 3 master nodes and 3 replica nodes. I am trying to copy data from another redis cluster (with no replicas) into this cluster but I keep getting errors like this:

./rump -from redis://redis-telem-cache-stg-ksat-m-1-redis-cluster:6379 -to redis://10.255.8.12:6379 -ttl
r
signal: exit
rr
redis read: exit
MOVED 4042 10.255.8.14:6379

Note: Above 10.255.8.12:6379 is one of the master nodes.

$ kk exec rails-cache-stg-redis-ksat-m-1-redis-cluster-0 -- redis-cli --cluster check 10.255.8.10:6379
Defaulting container name to rails-cache-stg-redis-ksat-m-1-redis-cluster.
Use 'kubectl describe pod/rails-cache-stg-redis-ksat-m-1-redis-cluster-0 -n default-minim-private' to see all of the containers in this pod.
10.255.8.15:6379 (fbdd5a4e...) -> 14029 keys | 5462 slots | 1 slaves.
10.255.8.12:6379 (b3718ac7...) -> 568 keys | 5461 slots | 1 slaves.
10.255.8.14:6379 (8790c578...) -> 487 keys | 5461 slots | 1 slaves.
[OK] 15084 keys in 3 masters.
0.92 keys per slot on average.
>>> Performing Cluster Check (using node 10.255.8.10:6379)
S: 8500263462d98613bc6e1172106ddcd37f5976a6 10.255.8.10:6379
   slots: (0 slots) slave
   replicates b3718ac766349261d819c9923cd915fc43d70352
M: fbdd5a4eebe77b49ec984ec953ec098fe262d320 10.255.8.15:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 4b371b26a9b5b3d0f23d2c965197f0bf3cbc355b 10.255.8.11:6379
   slots: (0 slots) slave
   replicates 8790c57879627039f3d74cc599e61707d33d93d4
M: b3718ac766349261d819c9923cd915fc43d70352 10.255.8.12:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: f01bd775618fd6de9a18160186a48a37b469da03 10.255.8.13:6379
   slots: (0 slots) slave
   replicates fbdd5a4eebe77b49ec984ec953ec098fe262d320
M: 8790c57879627039f3d74cc599e61707d33d93d4 10.255.8.14:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

Does rump currently support clusters with replicas? If it does can you recommend a way to diagnose/debug this further?

Thanks.

I believe that adding the pb library will help rump's UX

• I have read the contributing guide.

Report

I had a situation where I had a Redis full of stuff and I wanted to only transfer the keys matching some-pattern* to another Redis. For now, Rump doesn't support dumping only a pattern; however, both Redis and Radix supports it.

It would be great if Rump supported dumping keys matching a pattern.

The demo link(https://github.com/stickermule/rump#demo) contains an instance of docker image running with rump already installed. However, not able to find an image for the same online

• I have read the contributing guide.

Report

First of all thanks for a nice and robust tool!

I thought it could use some performance improvements. Currently scanning and dumping data from bigger data sets takes quite a bit of time and there are a couple reasons for that:

• When observing the reads and writes they happen almost serially. Making them concurrent would certainly improve the situation. By making the reads concurrent we should also be able to scan ahead (currently when scanning we're idling while fetching the next chunk of data).

• Another thing is the default COUNT returned when scanning is 10. Increasing it is definitely a low-hanging fruit, but for obvious reasons it cannot be set too high and since it wouldn't make sense to hard-code an arbitrary value I feel that giving the user an option to increase the COUNT manually is the right thing to do.

• I have read the contributing guide.

Report

The current version of rump does not work with an Elasticache Redis cache cluster where in-transit encryption (TLS) is enabled. If you try it, you will only get errors about blocked connections.
(More about in-transit encryption: https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/in-transit-encryption.html)

Would you be interested in a PR to add support for TLS encrypted connections? I have a private version where this works for my use case. If you are interested in a PR, I could create a more general solution that works with all combinations of encrypted / not encrypted connections and files.

It requires a change to the URL validation, an upgrade of the radix library and some changes to the run package where the connections are managed.

I wonder if it would make sense to have support for yaml config's. That way you could do:

$ rump -bookmark local
$ rump -b prod

_config.yml:

bookmark:
  local:
    from: 'redis://127.0.0.1:6379/1'
    to: 'redis://127.0.0.1:6379/2'
  prod:
    from: 'redis://production.cache.amazonaws.com:6379/1'
    to: 'redis://127.0.0.1:6379/1'

• I have read the contributing guide.

Report

In some cases I need to sync redis where tls is enabled and with auth token. And will be good to have an option to sync redis like this: radix support this option.

rump -from redis://redis-1:6379 -to redis://auth-token-string@redis-2:6379

I've used this tool it in the past and it works great, but now I have a case in which I need to dump keys that only match a certain pattern instead of dumping all the keys of a DB.

MATCH option of SCAN command can be used to add this support.

The following commit fzunino@67bc2e1 adds this support adding an optional command line argument called match and using * as default value, maintaining the current semantic.

Let me know and I can submit a PR with this commit.

I've just tried to make a run between SRC=a regular redis and DEST=an elasticache server.

• Rump detects the keys as I see progress on the screen, proportionate to the keys
• but when I look at the keys in the DEST elasticache I only see
  1) "ElastiCacheMasterReplicationTimestamp"

Is this supported?

The elasticserver info:

# Server
redis_version:2.8.24
redis_git_sha1:0
redis_git_dirty:0
redis_build_id:0
redis_mode:standalone
os:Amazon ElastiCache
arch_bits:64

You can check the https://github.com/microredis/migrator

I just noticed that if dumping TO a cluster with AUTH enabled, and NOT specifying the auth password in the "connection string", rump will not show any error, leading the user to believe that a dump has been performed (when it in fact has not)