producer-consumer-fastapi-celery's Introduction

A producer-consumer example with FastAPI and Celery.

How to play

Server (Option 1 - On your Local)

Install Redis, and run the following commands:

$ make env        # create a conda environment (need only once)
$ conda activate producer-consumer-fastapi-celery  # activate the env
$ make setup      # setup packages (need only once)
$
$ make broker     # run redis broker
$ make worker     # run celery worker
$ make api        # run fastapi server
$ make dashboard  # run dashboard that monitors celery

Server (Option 2 - Docker Compose)

Install Docker & Docker Compose, and run the following command:

$ docker-compose up

[Optional] Additional Workers

You can start up additional workers on other devices.

$ export BROKER_URL=redis://redis-broker-ip:6379
$ export BACKEND_URL=redis://redis-backend-ip:6379
$ make worker

Client

Option 1: In WebBrowser

http://0.0.0.0:8000/produce

Option 2: Test Client

Sending a number of requests simultaneously.

$ python src/client.py --n-req [N_REQUSTS]

Dashboard for Celery (Flower)

http://0.0.0.0:5555/

Issue Handling

Redis Error 8 connecting localhost:6379. nodename nor servname provided, or not known.

$ ulimit -n 1024

Docker's `network_mode=bridge`degrades the network performance.

We recommend to use Linux server if you would like to run docker-compose up.

For Developers

$ make setup-dev      # setup for developers
$ make format         # format scripts
$ make lint           # lints scripts
$ make utest          # runs unit tests
$ make cov            # opens unit test coverage information

Recommend Projects