Comments (7)
Thanks for reporting this.
This is major problem for usage within web app servers, where you may have many processes coming in and out of existence at any time due to scaling behavior or processes sporadically dying and being resurrected.
As far as I understand - please @phillbaker correct me if I'm wrong - the Redis storage is intended to be used whenever you want to share state between breaker instances, making those act like a single unit due to the shared state between them. So, unless you create different CircuitRedisStorage
(and providing different values for the namespace
argument), this is expected behavior.
Maybe for the mentioned use case (circuit breakers within web servers, each breaker with its own state), you should use the default in-memory storage.
from pybreaker.
Hm, I think some more information would help. If the following is happening, then this does seem like a bug:
this will cause the fail counter to be reset anytime a new CircuitBreaker is instantiated using the same shared storage (like the provided CircuitRedisStorage).
I would have thought these two lines prevented overriding the shared counter and state in redis:
Lines 450 to 451 in c84055a
@jcwilson - what impacts of this are you seeing in your environment?
from pybreaker.
Sorry for the double post, but I just saw #25. Would that also help address this issue?
from pybreaker.
@jcwilson Thanks for posting this, I am just getting started with pybreaker
and ran into the same issue that you described.
the Redis storage is intended to be used whenever you want to share state between breaker instances, making those act like a single unit due to the shared state between them. So, unless you create different
CircuitRedisStorage
(and providing different values for thenamespace
argument), this is expected behavior.
Are you suggesting that the way that pybreaker is currently written is correct? My expectation would be that the library wouldn't reset the counter when you spin up a new instance of the breaker.
Maybe for the mentioned use case (circuit breakers within web servers, each breaker with its own state), you should use the default in-memory storage.
I think this is the opposite of the described use case. @jcwilson seems to be suggesting that he wants to share state between web servers but this is not really possible as each circuit breaker resets the global state of the breaker.
I would have thought these two lines prevented overriding the shared counter and state in redis:
Those lines are indeed written to respect pre-existing state, however it's trampled over when initiating a circuit breaker due to the line posted in the issue's body:
Lines 41 to 62 in c84055a
Lines 713 to 728 in c84055a
from pybreaker.
Please see #27 for my suggestion on how to resolve this issue. Comments/critique welcome!
from pybreaker.
what impacts of this are you seeing in your environment?
Thanks a lot for taking a look. I neglected to mention that we started on version ~0.3.1 and have written our own redis backend that stores things in a slightly different manner, so it doesn't make use of the setnx()
code in CircuitRedisStorage
. That being said, I believe the current implementation as shipped with this library will still produce the same undesirable behavior. I believe the issue is best resolved at the CircuitBreaker
level, rather than at the CircuitBreakerStorage
level.
I think @alukach has expressed the problem pretty well. To keep things simple, let's speak in terms of a single circuit breaker instantiation in the code, but the code might be run in several processes (ie. uwsgi worker processes). We're using redis storage to provide a central shared counter for each circuit breaker's context so that an error on any box/process will increment the counter for all the rest, too. All of our processes will use the same namespace. I think if we were going to assign each process its own namespace, we'd just stick with the in-memory storage option and forego the overhead of the redis storage.
from pybreaker.
Closed by #27, feel free to re-open this if you have problems.
from pybreaker.
Related Issues (20)
- Error thrown when state is missing on Redis HOT 1
- proposal: adding a name property to CircuitBreaker HOT 1
- Chaining exception into the CircuitBreakerError HOT 2
- New release HOT 1
- Is the lock effective even across multiple server processes?
- Allow installing CircuitBreakerStorageBasedTestCase as a pip extra
- Exceptions in generators have wrong location
- Old versions of Python
- Python 3 support HOT 5
- Performance issues for multithreading
- Monitoring two or more funcions HOT 1
- CircuitBreakerError hiding a real ConnectTimeout HOT 3
- Maintainership HOT 1
- TTL on Redis HOT 1
- pybreaker should provide getter for the time remaining till the circuit changes state from open to half-open
- README sentence left hanging HOT 2
- Missing py.typed Marker HOT 3
- Contextmanager / `with` API? HOT 2
- define two CircuitBreaker problem HOT 1
- (🐞) incorrect typing for exclude argument HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pybreaker.