Comments (9)
@yagnik I've been pretty interested in that for a while. I started down that path with the config generators refactor, but I didn't finish the job. There are only really two blockers:
- There is still some coupling between haproxy as the output format and the service watchers, but it should be straightforward to remove.
- We'd have to make it ok to have 'nginx' or w.e. at the top level instead of just haproxy
Neither of these is tough, but it is tough to say "this works" without running it in production, and while I can't speak for airbnb I can say that we (Yelp) are reasonably happy with haproxy as the core routing component now that we figured out how to restart it without dropping traffic.
from synapse.
@jolynch what was your reason for using nginx instead of haproxy ?
The part that got me concerned is your post http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html which I believe is not a problem in nginx land cause process swapping doesn't happen. Is that the same reason you thought about using nginx ?
from synapse.
@yagnik Yes we considered switching to nginx due to the reload issue, especially as PaaSTA matured and we found out that SmartStack would be restarting HAProxy basically all the time.
However, since our zero downtime reload strategy shipped 8 months ago and we invested serious effort in performance tuning our HAProxy configs we've had relatively few problems with HAProxy itself. We're trying to work with the HAProxy maintainers to complement their currently supported dynamic changing of hostnames (in 1.6) with dynamic changing of ports, which would mean that we could theoretically pre-populate and then swap in and out as we get backends, although I'd still be worried about some of the stuff we can only do on reload. On the other hand Linux 4.4 will ship with a sweet refactor to the TCP SYN handling stack which will fix the 3WHS bug in Linux (causes most of HAProxy's dropped packets with SO_REUSEPORT) and if Eric and co. can figure out this patch then HAProxy will be 100% hitless on a reload.
I guess what I'm trying to say is that we've spent a lot of effort finely tuning HAProxy for high availability and performance, so investing in nginx seems like a hard move for us especially when Linux might finally fix their SO_REUSEPORT implementation. It's worth noting that "switching to nginx" isn't a silver bullet. Nginx is still a relatively young load balancer (i.e. just recently gained the capability to load balance TCP in the open source version, last I checked healthchecks are still part of nginx+ only, etc ...), and there is no guarantee that it won't have some other show stopping problem.
All that being said, I'd love if there was a nginx output format for Synapse, it's just hard for me to find time to work on it, but I'd be happy to help in any way I can.
from synapse.
thanks @jolynch sorry about the delayed reply. Also curious do you guys use chef/puppet for continuous deployment to all nodes for synapse/nerve ? How do you guys manage versioning in that case ?
In the mean time I'll pull away some time for getting nginx support, wait for my pr :)
from synapse.
@yagnik Yea we basically build debian packages of synapse, nerve, synapse-tools and nerve-tools using Docker. We have some glue puppet modules that tie together a synapse and nerve puppet module which basically install the aforementioned packages and setup cron jobs to run the jobs from synapse-tools
and nerve-tools
.
One of these days we'll get our act together and make a "how to smartstack at scale" website or something with a single binary you can curl and pipe to ba^H^H... I mean a debian package or something.
from synapse.
Never got around to doing this and we went ahead with haproxy too and now haproxy doesn't have a reload issue: https://www.haproxy.com/blog/truly-seamless-reloads-with-haproxy-no-more-hacks/ so happy with that.
As an aside we ended up building an rpm using omnibus: https://github.com/yagnik/omnibus-smartstack and use dendrite:https://github.com/yagnik/dendrite to build nerve and synapse config.
Going to close this issue for now.
from synapse.
Thanks for all the help @jolynch !
from synapse.
@yagnik oh also fwiw Synapse supports this now through config generators :-)
See the readme for how to do it and https://github.com/jolynch/synapse-nginx for an example of a proxy plugin.
from synapse.
👍
from synapse.
Related Issues (20)
- Extending cookie_value_method HOT 1
- In DNS service watcher, re-resolve every $TTL HOT 1
- AWS Auto Scaling Group watcher HOT 1
- Allow name in the DNS service watcher's ping method to configurable
- HAProxy with ActiveMQ HOT 2
- Obtain AWS credentials from IAM Instance Profile HOT 1
- ec2tag watcher needs to back off if Client.RequestLimitExceeded is returned HOT 1
- ZookeeperWatcher does not handle NoNode failures HOT 4
- docker-api v1.32 dropped support for Ruby 1.9 HOT 1
- discovery fails if Zookeeper tree is read only
- service_conf_dir no longer a valid top level config key HOT 2
- Unexpected error when migrating from Synapse-0.13.8 to Synapse-0.14.3 HOT 6
- ec2tag watcher fails if metadata service is unavailable during a credential refresh HOT 1
- No support for relative server_port overrides HOT 1
- used Synapse to replace RDS HOT 1
- Synapse and HAProxy seamless reloads HOT 10
- Provide a way to disable datadog stats
- Zookeeper watcher with autoscale servers
- Doc: README uses YAML config in Example Migration but only mentions JSON format in Configuration
- Gem does not build on arm64 platform
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from synapse.