Comments (17)
@xaratt this is really odd.
what happens when you try to connect to the AWS cache servers from "my.proxy.server" directly. So, in one try you use
printf "get foo\r\n" | nc myserver.0001.use1.cache.amazonaws.com 11211
And in the next try you do
printf "get foo\r\n" | nc ec2-xx-xx-xx-xx.compute-1.amazonaws.com 11211
do both of them work for you?
Also does ping of "ec2-xx-xx-xx-xx.compute-1.amazonaws.com" and " myserver.0001.use1.cache.amazonaws.com" resolve to different addresses?
from twemproxy.
Thank you for response.
Yes, when I try to connect directly to cache servers, both of them works fine:
[email protected]:~$ printf "get foo\r\n" | nc myserver.0001.use1.cache.amazonaws.com 11211 END [email protected]:~$ printf "get foo\r\n" | nc ec2-xx-xx-xx-xx.compute-1.amazonaws.com 11211 END
And ping of both server names resolve to the same address. I can give you real IP of "ec2-xx-xx-xx-xx", but Amazon's FAQ say that "Currently, all clients to an ElastiCache Cluster must be within the Amazon EC2 network".
I want to try to run twemproxy with memcached on my non-AWS server using CNAME and IP for connection. I'll post results here.
from twemproxy.
I made few new attempts and found configuration which allow me use twemproxy with AWS Elasticaches. I created own CNAMEs which points on Amazon's myserver.000x.use1.cache.amazonaws.com servers and twemproxy works fine with this strange scheme:
cache1.example.com -> myserver.0001.use1.cache.amazonaws.com -> ec2-xx-xx-xx-xx.compute-1.amazonaws.com
Is this bug (feature?) in Amazon DNS system? I don't know.
from twemproxy.
Xaratt did you ever get to the bottom of this?
We recently saw a similar-sounding issue (twemproxy appears to connect but then commands result in "ERR Connection timed out" with an Elasticache Redis write endpoint. It was intermittent but I will attempt to gather more info if it happens again.
from twemproxy.
@tom-dalton-fanduel, sorry for delay in responding, but we didn't found source or solution for this problem. We only added CNAMEs for each of our memcache nodes.
from twemproxy.
No problem - it looks like this was unrelated to an issue we were looking at!
from twemproxy.
Confused. Isn't EC Cluster endpoint a proxy, like twemproxy? Why deal with two proxies, rather set your app to connect directly to the EC Cluster endpoint.
from twemproxy.
Twemproxy is more than just a proxy, it provides transparent sharding too. In my case (and I'm guessing @xaratt 's too?) twemproxy is used to shard across multiple EC [write] endpoints.
from twemproxy.
@tom-dalton-fanduel I am having similar issue as you. Did you find out what the problem was. Thanks
from twemproxy.
I'm afraid I don't even remember the context for my comment, let alone if we ever solved it. We've since moved away from Twemproxy.
from twemproxy.
from twemproxy.
We use AWS elasticache as well but intermittently we get "ERR Connection timed out" Could not see anything in the logs.
from twemproxy.
from twemproxy.
I have not. Can you point me to it. Thanks!
from twemproxy.
from twemproxy.
I wonder if #567 was related - until that's merged, ketama_max_hostlen is 86. (if it works for short names but not long names)
But I don't see how that'd possibly be the issue, it'd just hash requests to the wrong host. (snprintf still appends null characters)
(looking at this issue while looking into whether timeouts are more likely with elasticache in general - doesn't seem like it)
from twemproxy.
Leaving a note on this to refer back to later in case anyone else has issues with elasticache memcached - the issue I'm looking into is unrelated
https://docs.aws.amazon.com/AmazonElastiCache/latest/mem-ug/ParameterGroups.Memcached.html suggests there's no timeout, so I'm confused
- It could be a bug parsing requests or responses in an older version
- It could be intermittent issues with connectivity from a datacenter or application to elasticache
Elasticache itself doesn't have an idle timeout according to recent documentation for 1.4, not sure if old versions were different
idle_timeout | Default: 0 (disabled)Type: integerModifiable: YesChanges Take Effect: At Launch | The minimum number of seconds a client will be allowed to idle before being asked to close. Range of values: 0 to 86400. |
---|
I think https://github.com/twitter/twemproxy/pull/324/files#diff-01600ca8f8e542768f785de1842f38b3aeeb315531c63b9d2ce8730a21f72a80 may help (related to the redis sentinel support proposal), but I still get occasional timeouts to elasticache when there's low traffic anyway
from twemproxy.
Related Issues (20)
- Cross Region Replication Support HOT 1
- Monotonically Increasing CPU usage for redis proxy HOT 3
- Twemproxy does not release memory for mbufs: Free list garbage collection approach HOT 1
- Not working redis-sentinel-feature HOT 2
- Building release 0.5.0 on OpenBSD 7.0 fails with `ld: error: undefined symbol: backtrace` HOT 1
- we have Redis Cluster ,what's twemproxy usage for Redis now?
- add HRANDFIELD HOT 5
- twemproxy支持blpop指令么? HOT 2
- Question: Can we pass/upload configuration file through web browser?
- error logs: NC_ request HOT 2
- possible bug in slots count reported in verbose logging statement HOT 1
- Q: still copy after each msg_parsed?? HOT 1
- memcached replication (with consistant hash) HOT 2
- Do not ship/install "test_all"
- Murmur implementation is not endian/alignment-neutral
- Getting error "" Redis connection lost and commands aborted in uncertain state. They might have been processed."'
- nutcracker sets umask only upon daemonization HOT 1
- fuzzing integration in oss-fuzz HOT 1
- 这个工具支持 redis 双向实时同步吗? HOT 1
- How to open HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from twemproxy.