danielfm / pybreaker Goto Github PK
View Code? Open in Web Editor NEWPython implementation of the Circuit Breaker pattern.
License: BSD 3-Clause "New" or "Revised" License
Python implementation of the Circuit Breaker pattern.
License: BSD 3-Clause "New" or "Revised" License
Can you make a new release version and add it to PyPI? Thanks!
Exclude is defined as:
exclude: Sequence[Type[ExceptionType]] | None = None,
but it can also be a callable. Mypy complains about following code (when not ignoring the type error):
pybreaker.CircuitBreaker(
fail_max=CIRCUIT_BREAKER_MAX_FAIL,
reset_timeout=CIRCUIT_BREAKER_RESET_TIMEOUT,
# don't trip circuit on client errors, except 429 (too Many Requests)
exclude=[ # type: ignore[arg-type]
lambda e: isinstance(e, HTTPError)
and e.response.status_code < 500
],
)
If you want to maintain support for older versions of Python, you could change the download source for those versions to python.org directly instead of the seemingly now missing files from the TravisCI. Is that something you care about, or do you want to drop support for these older versions that are no longer supported overall?
Example: Python 3.2.6 - https://www.python.org/downloads/release/python-326/
Happy to PR either way @danielfm
Would a PR to add external state be accepted?
We're looking at using pybreaker to manage the healthiness of an external API and we'd like to aggregate exceptions/failures across multiple processes/machines. For a first pass we'll probably throw data in redis, but I'd understand not wanting to add that as a dependency to this library. We were thinking of something similar to the pattern described here: https://github.com/sittercity/breaker_box#persistence-factories, which abstracts the storage a bit. What do you think would be a good approach that fits?
I'll be opening some PRs on this project, as I intend to use it extensively.
I'd like to know if this package is/will be actively maintained, if I should fork it - I'll still intend to submit the PRs here if that's the case - or if I can help here with the maintenance. ๐
In any case, thanks so much for this package! ๐
Hello! ๐
I'm discovering the library, thanks for sharing! =)
Here to report an unfinished sentence in the README: https://github.com/danielfm/pybreaker/blob/main/README.rst?plain=1#L204
By default, when the circuit breaker trips, it w
I'm curious, what's the end of that sentence? ๐
I'm trying use the decorator to monitoring two functions saving the state in redis, but when the first function is ok and the second function is failed, and i make a request again, the first function when is ok reset the counter of fails and the circuit never open
When the circuit breaker's state is closed and the call to the wrapped function fails the listener.failure function is not called because an exception is thrown in CircuitClosedState.on_failure. Is this behaviour intentional?
The block below is from the CircuitBreakerState class:
def _handle_error(self, exc):
"""
Handles a failed call to the guarded operation.
"""
if self._breaker.is_system_error(exc):
self._breaker._inc_counter()
self.on_failure(exc)
for listener in self._breaker.listeners:
listener.failure(self._breaker, exc)
else:
self._handle_success()
raise exc
Hi danielfm,
Does the redis storage is intended as the shared breaker state storage for different workers? it seems like the current implementation would be stayed in open state.
For example:
start 32 workers, each worker with state as closed, if one worker enter state open, it will store the state in the redis. when another worker receive the request, it will compare its state name with redis storage, and enter open state, but in the CircuitOpenState:
def __init__(self, cb, prev_state=None, notify=False):
"""
Moves the given circuit breaker `cb` to the "open" state.
"""
super(CircuitOpenState, self).__init__(cb, STATE_OPEN)
self._breaker._state_storage.opened_at = datetime.utcnow()
if notify:
for listener in self._breaker.listeners:
listener.state_change(self._breaker, prev_state, self)
It did not read redis storage opened_at, but set it as current time, it cause the closed worker has no way to enter half-open state?
The CircuitBreakerStorage
is a great way to share state across multiple servers. But is the lock effective for that?
We currently monitor the state of several circuit breakers via the current_state
attribute. However, we've noticed that if some of these connections have spiky traffic and trip the breaker, they can stay in the open state until another request is made which clears the state.
The breaker itself is operating as expected, however, our monitoring records in correct data. What do you think of including a check of _state_storage.opened_at
in current_state
? We're including this check in our code currently, but if this behavior at the library level seems incorrect I can open a PR.
Hi! I was testing how pybreaker would behave in various scenarios where state is being stored on Redis and the Redis server misbehaves.
CircuitRedisStorage will handle intermittent and complete connection loss with the Redis server in an acceptable manner. But it doesn't handle data loss on the Redis server gracefully. When no value has been set for a key, the Redis library returns None. This results in an AttributeError being thrown on this line:
Line 476 in c5cb8fa
I opened a PR (#25) which aims to resolve this by doing the following when it finds missing data on Redis:
Also, perhaps the circuit breaker's listeners should be notified with a custom exception when this occurs? I'm open to suggestions.
See PR:
#8
For the use case I'm working on, I would prefer to specify a list of exceptions that WILL count against the breaker's fail_counter
, rather than a list of exceptions that shouldn't be counted against it. That would prevent any unexpected run-time error from tripping the breaker.
What do you think?
In:
Line 512 in 945b3ff
state = state_bytes.decode('utf-8')
This fails in python 3 with:
AttributeError: 'str' object has no attribute 'decode'
Any plans to fix this, will a PR help on this one?
Hi,
if redis package is not installed then this code raise error:
try:
from redis.exceptions import RedisError
HAS_REDIS_SUPPORT = True
except ImportError:
HAS_REDIS_SUPPORT = FALSE
please, rewrite FALSE
to False
.
Are there plans to make a new release including the functionality to define the excluded errors as a callable?
Hello, thanks for the great package.
When using it with mypy I will get the missing py.typed marker as this is not being exported alongside the package;
https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
Skipping analyzing "pybreaker": module is installed, but missing library stubs or py.typed marker [import] See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
Any way that could be included with the next version?
Locking the thread at the moment of calling the decorated function will lead to performance degradation in the multithread setup.
I think it worth mentioning the issue in the description.
Line 213 in 7f6dd06
I wrote a custom CircuitBreakerStorage class for a different cache backend than Redis, and I'd like to test it using CircuitBreakerStorageBasedTestCase. But the tests module is not included in setup. Please allow installing CircuitBreakerStorageBasedTestCase as a pip extra, preferably separate from the rest of the tests stuff.
I define two CircuitBreaker But if one of them changes the state of all CircuitBreaker change!!!
I think , need to change the code when set in Redis and use the name of CircuitBreaker to solve it
breaker = pybreaker.CircuitBreaker(name='breaker', fail_max=3, reset_timeout=2, listeners=[NotifyListener()],
state_storage=pybreaker.CircuitRedisStorage(pybreaker.STATE_CLOSED, redis))
other_breaker = pybreaker.CircuitBreaker(name='other_breaker', fail_max=3, reset_timeout=2, listeners=[NotifyListener()],
state_storage=pybreaker.CircuitRedisStorage(pybreaker.STATE_CLOSED, redis))
Hi All,
it would be nice if function is_system_error
can work with self._exceptions
as well. I am about to implement ES circuit breaker for just one exception (ES BROKEN CONNECTION) so that I have to create my own Breaker (subclassing CircuitBreaker) and rewrite function is_system_error
instead of just define one exception in the new list self._exceptions
.
Regards,
Vojtech.
It would be useful to configure an effective retry policy. There is no use in retrying before the reset_timeout has elapsed.
This chaining exception feature is added into Python 3.6 https://www.python.org/dev/peps/pep-3134/, however, if we used this library in Python2, the CircuitBreakerError doesn't carry the trigger exception and the trace. This maybe an issue we actually lost the trace when the threshold is breached.
Due to these lines:
Line 62 in c84055a
Line 728 in c84055a
Also, this will cause the fail counter to be reset anytime a new CircuitBreaker is instantiated using the same shared storage (like the provided CircuitRedisStorage
).
This is major problem for usage within web app servers, where you may have many processes coming in and out of existence at any time due to scaling behavior or processes sporadically dying and being resurrected.
It seems like a possible solution might involve rethinking what the notify
argument is being used for. It appears to be set to True
in all cases except for this initial default value being set on line 62. Perhaps move the state change effect into the if notify:
block as well?
A better solution might be to not assume that a CircuitBreaker
should start in the Closed state, but instead ask the self._state_storage
object for the current state and initialize the CircuitBreaker
without affecting the state storage.
which can be easily triggered by running the tests under a heavy loaded system.
An easy/possible way to achieve this is to start many(sufficiently) instance of the tests in parallel :
gstarck@taf $ for x in $(seq 20)
> do
> (python setup.py test &> res$x || echo "res$x failed .." ) &
> done | grep -c failed
5
โ ~/work/public/python/pybreaker [master|โฆ20]
gstarck@taf $
We're seeing unusual behavior when wrapping a generator (a contextmanager specifically) that has a try
/except
in a circuit breaker. Specifically, the try
/except
in the wrapped function isn't being executed.
A simplified example is:
from contextlib import contextmanager
from pybreaker import CircuitBreaker
class ExceptionA(Exception): pass
class ExceptionB(Exception): pass
breaker = CircuitBreaker()
class Foo:
@contextmanager
@breaker
def wrapper(self):
try:
yield
except ExceptionA as e:
raise ExceptionB()
def foo(self):
with self.wrapper():
raise ExceptionA()
try:
Foo().foo()
except ExceptionB as e:
print('caught ExceptionB', e)
This example should print "caught ExceptionB", however, it raises:
Traceback (most recent call last):
File "context.py", line 67, in <module>
Foo().foo()
File "context.py", line 63, in foo
raise ExceptionA()
__main__.ExceptionA
At the root, I believe that these lines:
Lines 744 to 745 in 42b05c7
should not have _handle_error
invoke raise
but call wrapped_generator.throw(e)
.
Would it make sense to submit a PR for this? Perhaps a new argument to _handle_error can make raising optional?
Hi,
not an issue but a question:
is it safe to use the same breaker instance (CircuitBreaker) by multiple threads, possibly concurrently (modulo GIL ofcourse) ?
thx for the lib :)
This behavior I feel is incorrect.
The failure which passes the fail threshold throws a CircuitBreakerError instead of the actual underlying exception. To me, this is semantically incorrect. A real error happened, and the circuit was not bypassed, so why are we showing a CircuitBreakerError?
We are using this in cases of ReadTimeout and ConnectTimeout when connecting to external services, and which one is the case matters. We store the exception name in the DB, and we erroneously get CircuitBreakerError now when it was actually a ReadTimeout. This matters. The difference between a ReadTimeout and a CircuitBreakerError is the difference between an action that was potentially successful, and definitely not successful.
I've looked through the code and this is a fairly easy fix, which I can open a PR for, but I need to know if there's agreement on whether this is semantically the way it should behave or not.
At the very least it should be something like
CircuitBreakerPopped and CircuitBreakerOpen so that the cases of the breaker popping, and skipping operation are differentiable. Then in the CircuitBreakerPopped case, I can grab the next exception up on the stack for logging
Would you be open to accepting a PR to make CircuitBreaker
also possible to use like so:
with circuit_breaker:
do_the_thing_that_can_fail()
I think this is slightly nicer than the .call(...)
API.
Thanks for creating & maintaining this library!
Hey, pybreaker doesn't behave as expected with async code on the asyncio event loop. Is it because it isn't supported or am I doing something wrong?
Here's my code:
data_services_breaker = pybreaker.CircuitBreaker(fail_max=2, reset_timeout=60)
@data_services_breaker
async def get_all_bookings():
async with aiohttp.ClientSession() as session:
async with session.get('https://api.githubXXXX.com/events') as resp:
return await resp.text()
CircuitBreakerError is never thrown...
Thanks!
Currently, there's no expire time on the keys created on Redis. I'm concerned about orphan keys during the lifetime of the application. A full clean can be performed from time to time... But I guess we can also enable a TTL parameter on those.
To be specific, image I create a namespace "potato", then I'll have the key "potato:pybreaker:state". If I decide to change the namespace, I'll have the previous key being orphan.
Thanks for the great library. I'm adding logging to our use of circuit breakers so that we can see just how often states are changing. To make our logs more informative I've subclassed CircuitBreaker
to add a "name" property that I can use in logs, statsd counters, etc. Is there any interest in adding this to CircuitBreaker
? If so, I'll create a PR.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.