hynek / stamina Goto Github PK

View Code? Open in Web Editor NEW

743.0 4.0 7.0 809 KB

Production-grade retries for Python

Home Page: https://stamina.hynek.me/

License: MIT License

Python 100.00%

retry python retrying reliability

stamina's People

Contributors

Stargazers

Watchers

Forkers

simonegiacomelli rohankumardubey occoder ffengill abhinavm24 sthagen sysujayce

stamina's Issues

Please modify imports to reduce runtime impact

Start up time which is important for CLIs:

❯ python -m timeit -n 1 -r 1 "import structlog"
1 loop, best of 1: 185 msec per loop
❯ python -m timeit -n 1 -r 1 "from prometheus_client import Counter"
1 loop, best of 1: 97.2 msec per loop

RSS which is important for long-running processes on users' machines:

❯ python -c "import psutil;f=lambda p=psutil.Process(): print(p.memory_info().rss / 1024 ** 2);f();import structlog;f()"
14.984375
29.125
❯ python -c "import psutil;f=lambda p=psutil.Process(): print(p.memory_info().rss / 1024 ** 2);f();from prometheus_client import Counter;f()"
14.6796875
22.03515625

Retries made if no error happened, when set_active is set to False

In [11]: stamina.set_active(False)
    ...: for attempt in stamina.retry_context(on=ValueError, wait_initial=1, wait_max=2, timeout=40, wait_jitter=0, wait_exp_base=2, attempts=0):
    ...:     with attempt:
    ...:         print(attempt)
    ...:         time.sleep(1)
    ...:         print("test")
<Attempt num=1>
test
<Attempt num=1>
test

When set active is set to True it behaves like it should.

In [11]: stamina.set_active(True)
    ...: for attempt in stamina.retry_context(on=ValueError, wait_initial=1, wait_max=2, timeout=40, wait_jitter=0, wait_exp_base=2, attempts=0):
    ...:     with attempt:
    ...:         print(attempt)
    ...:         time.sleep(1)
    ...:         print("test")
<Attempt num=1>
test

Before and After Retry, and Logging

I'd like to log retries similar to what is done in the tenacity documentation.

https://tenacity.readthedocs.io/en/latest/#before-and-after-retry-and-logging

Stamina raises retriable exception before reaching max retries

I'm seeing some strange behavior from stamina, where a retriable exception is raised before the max attempts is reached.

Python: 3.10.4
stamina: 24.2.0

Sentry reports this log before the exception is raised:

stamina.retry_scheduled
{
  stamina.args: [],
  stamina.callable: <context block>,
  stamina.caused_by: ReadTimeout(''),
  stamina.kwargs: {},
  stamina.retry_num: 3,
  stamina.wait_for: 0.69,
  stamina.waited_so_far: 1.64
}

stamina.retry_num is 3, but attempts is set to 10 in retry_context

Code:

async def handle_async_request(self, request: httpx.Request) -> httpx.Response:
        for attempt in stamina.retry_context(
            on=(
                httpx.RemoteProtocolError,
                httpx.ReadError,
                httpx.ReadTimeout,
            ),
            attempts=settings.HTTP_CLIENT_RETRIES,
        ):
            with attempt:
                return await self._wrapper.handle_async_request(request)

        # This code should be unreachable, since stamina will raise an exception
        # if the maximum number of retries is exceeded.
        raise Exception("Max retries exceeded.")  # pragma: no cover

Stack trace:

ReadTimeout: null
  File "httpx/_transports/default.py", line 69, in map_httpcore_exceptions
    yield
  File "httpx/_transports/default.py", line 373, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "httpcore/_async/connection_pool.py", line 216, in handle_async_request
    raise exc from None
  File "httpcore/_async/connection_pool.py", line 196, in handle_async_request
    response = await connection.handle_async_request(
  File "httpcore/_async/connection.py", line 101, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "httpcore/_async/http11.py", line 143, in handle_async_request
    raise exc
  File "httpcore/_async/http11.py", line 113, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
  File "httpcore/_async/http11.py", line 186, in _receive_response_headers
    event = await self._receive_event(timeout=timeout)
  File "httpcore/_async/http11.py", line 224, in _receive_event
    data = await self._network_stream.read(
  File "httpcore/_backends/anyio.py", line 31, in read
    with map_exceptions(exc_map):
  File "contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
ReadTimeout: null
  File "app/services/external/gamification.py", line 25, in get_entity_for_action
    entity_response = await get_blocks_pivot_data(field=EBlockField.ID, value=action.entity_id)
  File "app/http/courses.py", line 243, in get_blocks_pivot_data
    response = await pool.courses.get(
  File "httpx/_client.py", line 1801, in get
    return await self.request(
  File "httpx/_client.py", line 1574, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "httpx/_client.py", line 1661, in send
    response = await self._send_handling_auth(
  File "httpx/_client.py", line 1689, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "httpx/_client.py", line 1726, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "httpx/_client.py", line 1763, in _send_single_request
    response = await transport.handle_async_request(request)
  File "app/http/pool.py", line 16, in handle_async_request
    for attempt in stamina.retry_context(
  File "stamina/_core.py", line 439, in __iter__
    for r in _t.Retrying(
  File "__init__.py", line 347, in __iter__
    do = self.iter(retry_state=retry_state)
  File "__init__.py", line 325, in iter
    raise retry_exc.reraise()
  File "__init__.py", line 158, in reraise
    raise self.last_attempt.result()
  File "concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "app/http/pool.py", line 26, in handle_async_request
    return await self._wrapper.handle_async_request(request)
  File "httpx/_transports/default.py", line 372, in handle_async_request
    with map_httpcore_exceptions():
  File "contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "httpx/_transports/default.py", line 86, in map_httpcore_exceptions
    raise mapped_exc(message) from exc

Is there any direction you could point me to debug this issue, or is it a bug in the library?
Any help is appreciated.

httpx.RemoteProtocolError not retry correctly

Hi there,
I use stamina to retry my async function with httpx.AsyncClient, but once the request received httpx.RemoteProtocolError, the program died without retry.

Here is how I use stamina:

@stamina.retry(on=Exception, attempts=3, timeout=30)
async def func(url: str, params: dict, client: httpx.AsyncClient):
    response: httpx.Response = await client.post(url, json=params)
    response.raise_for_status()

Sorry I can not give you a snippet code to reproduce the problem because it just happens almost in 1/100k. I think the problem is that the server drop the connection and httpx still use that connection to send request. If this is the situation, maybe retry will not solve the problem either.
stamina version: 23.3.0
httpx version: 0.26.0

Below is the traceback:

Traceback (most recent call last):
  File "/opt/conda/envs/cms-test/lib/python3.12/site-packages/httpx/_transports/default.py", line 67, in map_httpcore_exceptions
    yield
  File "/opt/conda/envs/cms-test/lib/python3.12/site-packages/httpx/_transports/default.py", line 371, in handle_async_request
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/cms-test/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 268, in handle_async_request
    raise exc
  File "/opt/conda/envs/cms-test/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 251, in handle_async_request
    response = await connection.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/cms-test/lib/python3.12/site-packages/httpcore/_async/connection.py", line 103, in handle_async_request
    return await self._connection.handle_async_request(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/cms-test/lib/python3.12/site-packages/httpcore/_async/http11.py", line 133, in handle_async_request
    raise exc
  File "/opt/conda/envs/cms-test/lib/python3.12/site-packages/httpcore/_async/http11.py", line 111, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/cms-test/lib/python3.12/site-packages/httpcore/_async/http11.py", line 176, in _receive_response_headers
    event = await self._receive_event(timeout=timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/cms-test/lib/python3.12/site-packages/httpcore/_async/http11.py", line 226, in _receive_event
    raise RemoteProtocolError(msg)
httpcore.RemoteProtocolError: Server disconnected without sending a response.   

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/cms-test/text/netease.py", line 187, in query_text_async_helper
    response: httpx.Response = await client.post(
                               ^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/cms-test/lib/python3.12/site-packages/httpx/_client.py", line 1877, in post
    return await self.request(
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/cms-test/lib/python3.12/site-packages/httpx/_client.py", line 1559, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/cms-test/lib/python3.12/site-packages/httpx/_client.py", line 1646, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/cms-test/lib/python3.12/site-packages/httpx/_client.py", line 1674, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/cms-test/lib/python3.12/site-packages/httpx/_client.py", line 1711, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/cms-test/lib/python3.12/site-packages/httpx/_client.py", line 1748, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/cms-test/lib/python3.12/site-packages/httpx/_transports/default.py", line 370, in handle_async_request
    with map_httpcore_exceptions():
  File "/opt/conda/envs/cms-test/lib/python3.12/contextlib.py", line 155, in __exit__
    self.gen.throw(value)
  File "/opt/conda/envs/cms-test/lib/python3.12/site-packages/httpx/_transports/default.py", line 84, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.RemoteProtocolError: Server disconnected without sending a response.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

Prevent exponential retry explosion

Originally, I wanted to get this into 23.1.0, but between work, having a life, and EP preparation I had make a decision.

I think it should be possible with contextvars, but it's a bit more involved, than one would expect.

This is a test that needs to pass:

def test_retry_block_no_recursive_by_default():
    """
    When retrying context managers, don't retry recursively by default.
    """
    inner = outer = 0

    with pytest.raises(ValueError):
        for o_a in stamina.retry_context(
            on=ValueError, attempts=2, wait_max=0
        ):
            with o_a:
                outer += 1
                for i_a in stamina.retry_context(
                    on=ValueError, attempts=2, wait_max=0
                ):
                    with i_a:
                        inner += 1
                        raise ValueError

    assert 2 == inner
    assert 2 == outer

Pluggable observability?

At current work, we use datadog, not prometheus. At previous work, we used an internal metrics tool.

Since the _instrumentation needs are pretty modest, how would you feel about defining an "observation sink" protocol and having a way for stamina to be configured with a dynamic callable or module to receive the observable event?

I see the implementation already uses an INSTRUMENTS list, so I think this might just mean stabilizing the interface and documenting how to add to INSTRUMENTS?

hynek / stamina Goto Github PK

stamina's People

Contributors

Stargazers

Watchers

Forkers

stamina's Issues

Please modify imports to reduce runtime impact

Retries made if no error happened, when set_active is set to False

Before and After Retry, and Logging

Stamina raises retriable exception before reaching max retries

httpx.RemoteProtocolError not retry correctly

Prevent exponential retry explosion

Pluggable observability?

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent