Connection timed out to AWS elasticache about twemproxy HOT 17 CLOSED

@xaratt this is really odd.

what happens when you try to connect to the AWS cache servers from "my.proxy.server" directly. So, in one try you use

printf "get foo\r\n" | nc myserver.0001.use1.cache.amazonaws.com 11211

And in the next try you do

printf "get foo\r\n" | nc ec2-xx-xx-xx-xx.compute-1.amazonaws.com 11211

do both of them work for you?

Also does ping of "ec2-xx-xx-xx-xx.compute-1.amazonaws.com" and " myserver.0001.use1.cache.amazonaws.com" resolve to different addresses?

from twemproxy.

Thank you for response.

Yes, when I try to connect directly to cache servers, both of them works fine:

[email protected]:~$ printf "get foo\r\n" | nc myserver.0001.use1.cache.amazonaws.com 11211
END
[email protected]:~$ printf "get foo\r\n" | nc ec2-xx-xx-xx-xx.compute-1.amazonaws.com 11211
END

And ping of both server names resolve to the same address. I can give you real IP of "ec2-xx-xx-xx-xx", but Amazon's FAQ say that "Currently, all clients to an ElastiCache Cluster must be within the Amazon EC2 network".

I want to try to run twemproxy with memcached on my non-AWS server using CNAME and IP for connection. I'll post results here.

from twemproxy.

I made few new attempts and found configuration which allow me use twemproxy with AWS Elasticaches. I created own CNAMEs which points on Amazon's myserver.000x.use1.cache.amazonaws.com servers and twemproxy works fine with this strange scheme:

cache1.example.com -> myserver.0001.use1.cache.amazonaws.com -> ec2-xx-xx-xx-xx.compute-1.amazonaws.com

Is this bug (feature?) in Amazon DNS system? I don't know.

from twemproxy.

Xaratt did you ever get to the bottom of this?

We recently saw a similar-sounding issue (twemproxy appears to connect but then commands result in "ERR Connection timed out" with an Elasticache Redis write endpoint. It was intermittent but I will attempt to gather more info if it happens again.

from twemproxy.

@tom-dalton-fanduel, sorry for delay in responding, but we didn't found source or solution for this problem. We only added CNAMEs for each of our memcache nodes.

from twemproxy.

No problem - it looks like this was unrelated to an issue we were looking at!

from twemproxy.

Confused. Isn't EC Cluster endpoint a proxy, like twemproxy? Why deal with two proxies, rather set your app to connect directly to the EC Cluster endpoint.

from twemproxy.

Twemproxy is more than just a proxy, it provides transparent sharding too. In my case (and I'm guessing @xaratt 's too?) twemproxy is used to shard across multiple EC [write] endpoints.

from twemproxy.

@tom-dalton-fanduel I am having similar issue as you. Did you find out what the problem was. Thanks

from twemproxy.

I'm afraid I don't even remember the context for my comment, let alone if we ever solved it. We've since moved away from Twemproxy.

from twemproxy.

from twemproxy.

We use AWS elasticache as well but intermittently we get "ERR Connection timed out" Could not see anything in the logs.

from twemproxy.

from twemproxy.

I have not. Can you point me to it. Thanks!

from twemproxy.

from twemproxy.

I wonder if #567 was related - until that's merged, ketama_max_hostlen is 86. (if it works for short names but not long names)

But I don't see how that'd possibly be the issue, it'd just hash requests to the wrong host. (snprintf still appends null characters)

(looking at this issue while looking into whether timeouts are more likely with elasticache in general - doesn't seem like it)

from twemproxy.

Leaving a note on this to refer back to later in case anyone else has issues with elasticache memcached - the issue I'm looking into is unrelated

https://docs.aws.amazon.com/AmazonElastiCache/latest/mem-ug/ParameterGroups.Memcached.html suggests there's no timeout, so I'm confused

• It could be a bug parsing requests or responses in an older version
• It could be intermittent issues with connectivity from a datacenter or application to elasticache

Elasticache itself doesn't have an idle timeout according to recent documentation for 1.4, not sure if old versions were different

I think https://github.com/twitter/twemproxy/pull/324/files#diff-01600ca8f8e542768f785de1842f38b3aeeb315531c63b9d2ce8730a21f72a80 may help (related to the redis sentinel support proposal), but I still get occasional timeouts to elasticache when there's low traffic anyway

from twemproxy.