Git Product home page Git Product logo

aiokeydb-py's Introduction

aiokeydb

Latest Version: PyPI version

Unified Syncronous and Asyncronous Python client for KeyDB and Redis.

It implements the majority of redis-py API methods and is fully compatible with redis-py API methods, while also providing a unified Client for both sync and async methods.

Additionally, newer apis such as KeyDBClient are strongly typed to provide a more robust experience.


Usage

aiokeydb can be installed from pypi using pip or from source.

Installation

# Install from pypi
pip install aiokeydb

# Install from source
pip install git+https://github.com/trisongz/aiokeydb-py.git

Examples

Below are some examples of how to use aiokeydb. Additional examples can be found in the tests directory.

import time
import asyncio
import uuid
from aiokeydb import KeyDBClient

# The session can be explicitly initialized, or
# will be lazily initialized on first use
# through environment variables with all 
# params being prefixed with `KEYDB_`

keydb_uri = "keydb://localhost:6379/0"

# Initialize the Unified Client
KeyDBClient.init_session(
    uri = keydb_uri,
)

# Cache the results of these functions
# cachify works for both sync and async functions
# and has many params to customize the caching behavior
# and supports both `redis` and `keydb` backends
# as well as `api` frameworks such as `fastapi` and `starlette`

@KeyDBClient.cachify()
async def async_fibonacci(number: int):
    if number == 0: return 0
    elif number == 1: return 1
    return await async_fibonacci(number - 1) + await async_fibonacci(number - 2)

@KeyDBClient.cachify()
def fibonacci(number: int):
    if number == 0: return 0
    elif number == 1: return 1
    return fibonacci(number - 1) + fibonacci(number - 2)

async def test_fib(n: int = 100, runs: int = 10):
    # Test that both results are the same.
    sync_t, async_t = 0.0, 0.0

    for i in range(runs):
        t = time.time()
        print(f'[Async - {i}/{runs}] Fib Result: {await async_fibonacci(n)}')
        tt = time.time() - t
        print(f'[Async - {i}/{runs}] Fib Time: {tt:.2f}s')
        async_t += tt

        t = time.time()
        print(f'[Sync  - {i}/{runs}] Fib Result: {fibonacci(n)}')
        tt = time.time() - t
        print(f'[Sync  - {i}/{runs}] Fib Time: {tt:.2f}s')
        sync_t += tt
    
    print(f'[Async] Cache Average Time: {async_t / runs:.2f}s | Total Time: {async_t:.2f}s')
    print(f'[Sync ] Cache Average Time: {sync_t / runs:.2f}s | Total Time: {sync_t:.2f}s')
        
async def test_setget(runs: int = 10):
    # By default, the client utilizes `pickle` to serialize
    # and deserialize objects. This can be changed by setting
    # the `serializer`

    sync_t, async_t = 0.0, 0.0
    for i in range(runs):
        value = str(uuid.uuid4())
        key = f'async-test-{i}'
        t = time.time()
        await KeyDBClient.async_set(key, value)
        assert await KeyDBClient.async_get(key) == value
        tt = time.time() - t
        print(f'[Async - {i}/{runs}] Get/Set: {key} -> {value} = {tt:.2f}s')
        async_t += tt

        value = str(uuid.uuid4())
        key = f'sync-test-{i}'
        t = time.time()
        KeyDBClient.set(key, value)
        assert KeyDBClient.get(key) == value
        tt = time.time() - t
        print(f'[Sync  - {i}/{runs}] Get/Set: {key}  -> {value} = {tt:.2f}s')
        sync_t += tt
    
    print(f'[Async] GetSet Average Time: {async_t / runs:.2f}s | Total Time: {async_t:.2f}s')
    print(f'[Sync ] GetSet Average Time: {sync_t / runs:.2f}s | Total Time: {sync_t:.2f}s')


async def run_tests(fib_n: int = 100, fib_runs: int = 10, setget_runs: int = 10):
    
    # You can explicitly wait for the client to be ready
    # Sync version
    # KeyDBClient.wait_for_ready()
    await KeyDBClient.async_wait_for_ready()

    # Run the tests
    await test_fib(n = fib_n, runs = fib_runs)
    await test_setget(runs = setget_runs)


    # Utilize the current session
    await KeyDBClient.async_set('async_test_0', 'test')
    assert await KeyDBClient.async_get('async_test_0') == 'test'

    KeyDBClient.set('sync_test_0', 'test')
    assert KeyDBClient.get('sync_test_0') == 'test'


    # you can access the `KeyDBSession` object directly
    # which mirrors the APIs in `KeyDBClient`

    await KeyDBClient.session.async_set('async_test_1', 'test')
    assert await KeyDBClient.session.async_get('async_test_1') == 'test'

    KeyDBClient.session.set('sync_test_1', 'test')
    assert KeyDBClient.session.get('sync_test_1') == 'test'

    # The underlying client can be accessed directly
    # if the desired api methods aren't mirrored

    # KeyDBClient.keydb
    # KeyDBClient.async_keydb
    # Since encoding / decoding is not handled by the client
    # you must encode / decode the data yourself
    await KeyDBClient.async_keydb.set('async_test_2', b'test')
    assert await KeyDBClient.async_keydb.get('async_test_2') == b'test'

    KeyDBClient.keydb.set('sync_test_2', b'test')
    assert KeyDBClient.keydb.get('sync_test_2') == b'test'

    # You can also explicitly close the client
    # However, this closes the connectionpool and will terminate
    # all connections. This is not recommended unless you are
    # explicitly closing the client.

    # Sync version
    # KeyDBClient.close()
    await KeyDBClient.aclose()

asyncio.run(run_tests())

Additionally, you can use the previous APIs that are expected to be present

from aiokeydb import KeyDB, AsyncKeyDB, from_url

sync_client = KeyDB()
async_client = AsyncKeyDB()

# Alternative methods
sync_client = from_url('keydb://localhost:6379/0')
async_client = from_url('keydb://localhost:6379/0', asyncio = True)

Setting the Global Client Settings

The library is designed to be explict and not have any global state. However, you can set the global client settings by using the KeyDBClient.configure method. This is useful if you are using the KeyDBClient class directly, or if you are using the KeyDBClient class in a library that you are developing.

For example, if you initialize a session with a specific uri, the global client settings will still inherit from the default settings.

from aiokeydb import KeyDBClient
from lazyops.utils import logger

keydb_uris = {
    'default': 'keydb://127.0.0.1:6379/0',
    'cache': 'keydb://localhost:6379/1',
    'public': 'redis://public.redis.db:6379/0',
}

sessions = {}

# these will now be initialized
for name, uri in keydb_uris.items():
    sessions[name] = KeyDBClient.init_session(
        name = name,
        uri = uri,
    )
    logger.info(f'Session {name}: uri: {sessions[name].uri}')

# however if you initialize another session
# it will use the global environment vars

sessions['test'] = KeyDBClient.init_session(
    name = 'test',
)
logger.info(f'Session test: uri: {sessions["test"].uri}')

By configuring the global settings, any downstream sessions will inherit the global settings.

from aiokeydb import KeyDBClient
from lazyops.utils import logger

default_uri = 'keydb://public.host.com:6379/0'
keydb_dbs = {
    'cache': {
        'db_id': 1,
    },
    'db': {
        'uri': 'keydb://127.0.0.1:6379/0',
    },
}

KeyDBClient.configure(
    url = default_uri,
    debug_enabled = True,
    queue_db = 1,
)

# now any sessions that are initialized will use the global settings

sessions = {}
# these will now be initialized

# Initialize the first default session
# which should utilize the `default_uri`
KeyDBClient.init_session()

for name, config in keydb_dbs.items():
    sessions[name] = KeyDBClient.init_session(
        name = name,
        **config
    )
    logger.info(f'Session {name}: uri: {sessions[name].uri}')

KeyDB Worker Queues

Released since v0.1.1

KeyDB Worker Queues is a simple, fast, and reliable queue system for KeyDB. It is designed to be used in a distributed environment, where multiple KeyDB instances are used to process jobs. It is also designed to be used in a single instance environment, where a single KeyDB instance is used to process jobs.

import asyncio
from aiokeydb import KeyDBClient
from aiokeydb.queues import TaskQueue, Worker
from lazyops.utils import logger


# Configure the KeyDB Client - the default keydb client will use 
# db = 0, and queue uses 2 so that it doesn't conflict with other
# by configuring it here, you can explicitly set the db to use
keydb_uri = "keydb://127.0.0.1:6379/0"

# Configure the Queue to use db = 1 instead of 2
KeyDBClient.configure(
    url = keydb_uri,
    debug_enabled = True,
    queue_db = 1,
)

@Worker.add_cronjob("*/1 * * * *")
async def test_cron_task(*args, **kwargs):
    logger.info("Cron task ran")
    await asyncio.sleep(5)

@Worker.add_function()
async def test_task(*args, **kwargs):
    logger.info("Task ran")
    await asyncio.sleep(5)

async def run_tests():
    queue = TaskQueue("test_queue")
    worker = Worker(queue)
    await worker.start()

asyncio.run(run_tests())

Requirements

  • deprecated>=1.2.3

  • packaging>=20.4

  • importlib-metadata >= 1.0; python_version < "3.8"

  • typing-extensions; python_version<"3.8"

  • async-timeout>=4.0.2

  • pydantic

  • anyio

  • lazyops


Major Changes:

  • v0.0.8 -> v0.0.11:

    Fully Migrates away from aioredis to redis-py @ v4.3.4.

    This inherits all API methods from redis-py to enforce compatability since deprecation of aioredis going forward

    This fork of redis-py has some modified semantics, while attempting to replicate all the API methods of redis-py to avoid compatibility issues with underlying libraries that require a pinned redis version.

    Notably, all async Classes and methods are prefixed by Async to avoid name collisions with the sync version.

  • v0.0.11 -> v0.1.0:

    Migration of aiokeydb.client -> aiokeydb.core and aiokeydb.asyncio.client -> aiokeydb.asyncio.core However these have been aliased to their original names to avoid breaking changes.

    Creating a unified API available through new KeyDBClient class that creates sessions which are KeyDBSession inherits from KeyDB and AsyncKeyDBClient class that inherits from AsyncKeyDB class

aiokeydb-py's People

Contributors

dmitrybabanovforreal avatar trisongz avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar

aiokeydb-py's Issues

python 3.11 support

Hi! First of all, huge thanks for this repo and all the work that you put into it!

I just discovered this, and it seems like you don't officially support python versions newer than 3.8, importing library in my python3.11 project results in a following error

Traceback (most recent call last):
  File "core.py", line 7, in <module>
    from aiokeydb import from_url
  File "venv/lib/python3.11/site-packages/aiokeydb/__init__.py", line 57, in <module>
    from aiokeydb.client.serializers import SerializerType
  File "venv/lib/python3.11/site-packages/aiokeydb/client/__init__.py", line 39, in <module>
    from aiokeydb.client.serializers import SerializerType, BaseSerializer
  File "venv/lib/python3.11/site-packages/aiokeydb/client/serializers/__init__.py", line 7, in <module>
    from aiokeydb.client.serializers._pickle import PickleSerializer, DillSerializer
  File "venv/lib/python3.11/site-packages/aiokeydb/client/serializers/_pickle.py", line 29, in <module>
    class DefaultProtocols:
  File "venv/lib/python3.11/site-packages/aiokeydb/client/serializers/_pickle.py", line 32, in DefaultProtocols
    dill: int = dill.HIGHEST_PROTOCOL
                ^^^^^^^^^^^^^^^^^^^^^
AttributeError: type object 'object' has no attribute 'HIGHEST_PROTOCOL'

The easy fix for me was installing dill, but i am not sure if I will face more problems further down the road with my 3.11 install :) Do you happen to have any suggestions or tips for me?

My intention is to use this in production, it will also be my first attempt at using KeyDb instead of Redis :) Thanks a lot!

Pydantic V2 migration

With Pydantic v2 there have been code changes and this module won't load. I fixed it for my use case with "pydantic<2" in requirements but I guess that's not a long term solution.

Error:

File "/usr/local/lib/python3.10/dist-packages/aiokeydb/__init__.py", line 57, in <module>
  from aiokeydb.client.serializers import SerializerType
File "/usr/local/lib/python3.10/dist-packages/aiokeydb/client/__init__.py", line 40, in <module>
  from aiokeydb.client.config import KeyDBSettings
File "/usr/local/lib/python3.10/dist-packages/aiokeydb/client/config.py", line 8, in <module>
  from aiokeydb.client.types import BaseSettings, validator, lazyproperty, KeyDBUri
File "/usr/local/lib/python3.10/dist-packages/aiokeydb/client/types.py", line 9, in <module>
  from pydantic import BaseSettings as _BaseSettings
File "/usr/local/lib/python3.10/dist-packages/pydantic/__init__.py", line 206, in __getattr__
  return _getattr_migration(attr_name)
File "/usr/local/lib/python3.10/dist-packages/pydantic/_migration.py", line 279, in wrapper
  raise PydanticImportError(
pydantic.errors.PydanticImportError: `BaseSettings` has been moved to the `pydantic-settings` package. See https://docs.pydantic.dev/2.0.1/migration/#basesettings-has-moved-to-pydantic-settings for more details.

Issue when using with pypy

When using with pypy I am getting this error

Traceback (most recent call last):
  File "c:\Users\User\Desktop\Python Programs\redis\keydbtesting.py", line 4, in <module>
    from aiokeydb import KeyDBClient
  File "C:\Users\User\Downloads\pypy3.10-v7.3.12-win64\pypy3.10-v7.3.12-win64\lib\site-packages\aiokeydb\__init__.py", line 3, in <module>     
    from aiokeydb.core import KeyDB, AsyncKeyDB
  File "C:\Users\User\Downloads\pypy3.10-v7.3.12-win64\pypy3.10-v7.3.12-win64\lib\site-packages\aiokeydb\core.py", line 28, in <module>        
    from aiokeydb.connection import (
  File "C:\Users\User\Downloads\pypy3.10-v7.3.12-win64\pypy3.10-v7.3.12-win64\lib\site-packages\aiokeydb\connection.py", line 39, in <module>  
    from aiokeydb.utils import set_ulimits
  File "C:\Users\User\Downloads\pypy3.10-v7.3.12-win64\pypy3.10-v7.3.12-win64\lib\site-packages\aiokeydb\utils\__init__.py", line 1, in <module>
    from .base import (
  File "C:\Users\User\Downloads\pypy3.10-v7.3.12-win64\pypy3.10-v7.3.12-win64\lib\site-packages\aiokeydb\utils\base.py", line 7, in <module>   
    from aiokeydb.types import ENOVAL
  File "C:\Users\User\Downloads\pypy3.10-v7.3.12-win64\pypy3.10-v7.3.12-win64\lib\site-packages\aiokeydb\types\__init__.py", line 1, in <module>
    from .base import *
  File "C:\Users\User\Downloads\pypy3.10-v7.3.12-win64\pypy3.10-v7.3.12-win64\lib\site-packages\aiokeydb\types\base.py", line 8, in <module>   
    from .compat import validator, root_validator, Field, PYD_VERSION, get_pyd_field_names, get_pyd_dict, pyd_parse_obj, get_pyd_schema        
  File "C:\Users\User\Downloads\pypy3.10-v7.3.12-win64\pypy3.10-v7.3.12-win64\lib\site-packages\aiokeydb\types\compat.py", line 12, in <module>    from lazyops.utils.imports import resolve_missing
  File "C:\Users\User\Downloads\pypy3.10-v7.3.12-win64\pypy3.10-v7.3.12-win64\lib\site-packages\lazyops\utils\__init__.py", line 1, in <module>    from .logs import logger
  File "C:\Users\User\Downloads\pypy3.10-v7.3.12-win64\pypy3.10-v7.3.12-win64\lib\site-packages\lazyops\utils\logs.py", line 9, in <module>    
    from lazyops.libs.logging import (
  File "C:\Users\User\Downloads\pypy3.10-v7.3.12-win64\pypy3.10-v7.3.12-win64\lib\site-packages\lazyops\libs\logging\__init__.py", line 6, in <module>
    from .main import (
  File "C:\Users\User\Downloads\pypy3.10-v7.3.12-win64\pypy3.10-v7.3.12-win64\lib\site-packages\lazyops\libs\logging\main.py", line 21, in <module>
    from loguru import _defaults
  File "C:\Users\User\Downloads\pypy3.10-v7.3.12-win64\pypy3.10-v7.3.12-win64\lib\site-packages\loguru\__init__.py", line 31, in <module>      
    logger.add(_sys.stderr)
  File "C:\Users\User\Downloads\pypy3.10-v7.3.12-win64\pypy3.10-v7.3.12-win64\lib\site-packages\loguru\_logger.py", line 979, in add
    exception_formatter = ExceptionFormatter(
  File "C:\Users\User\Downloads\pypy3.10-v7.3.12-win64\pypy3.10-v7.3.12-win64\lib\site-packages\loguru\_better_exceptions.py", line 157, in __init__
    self._lib_dirs = self._get_lib_dirs()
  File "C:\Users\User\Downloads\pypy3.10-v7.3.12-win64\pypy3.10-v7.3.12-win64\lib\site-packages\loguru\_better_exceptions.py", line 166, in _get_lib_dirs
    paths = {sysconfig.get_path(name, scheme) for scheme in schemes for name in names}
  File "C:\Users\User\Downloads\pypy3.10-v7.3.12-win64\pypy3.10-v7.3.12-win64\lib\site-packages\loguru\_better_exceptions.py", line 166, in <setcomp>
    paths = {sysconfig.get_path(name, scheme) for scheme in schemes for name in names}
  File "C:\Users\User\Downloads\pypy3.10-v7.3.12-win64\pypy3.10-v7.3.12-win64\Lib\sysconfig.py", line 596, in get_path
    return get_paths(scheme, vars, expand)[name]
KeyError: 'platstdlib'

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.