Project		Status
Community
Meta

Note Starlite has been renamed to Litestar

This is an API performance test comparing:

1. Litestar
2. Starlite v1.5x
3. Starlette
4. FastAPI
5. Sanic
6. BlackSheep

Using the bombardier HTTP benchmarking tool.

You can view the last run results under the /results folder - it contains json files with the output. The plotting is done using pandas - script is under /analysis.

Note: PRs improving the analysis script are welcome.

Setup is identical for all frameworks.

• Applications reside in the frameworks folder and consist of a single file named <framework_name>_app.py

All tests are run sync and async

• Sending 100 bytes plaintext
• Sending 1kB plaintext
• Sending 10kB plaintext
• Sending 100kB plaintext
• Sending 500kB plaintext
• Sending 1MB plaintext
• Sending 5MB plaintext

Serializing a dictionary into JSON

• Serializing and sending 1kB JSON
• Serializing and sending 10kB JSON
• Serializing and sending 100kB JSON
• Serializing and sending 500kB JSON
• Serializing and sending 1MB JSON

(only supported by Litestar and FastAPI)

• Serializing 50 dataclass objects each referencing 2 more dataclass objects
• Serializing 100 dataclass objects each referencing 5 more dataclass objects
• Serializing 500 dataclass objects each referencing 3 more dataclass objects
• Serializing 50 pydantic objects each referencing 2 more pydantic objects
• Serializing 100 pydantic objects each referencing 5 more pydantic objects
• Serializing 500 pydantic objects each referencing 3 more pydantic objects

• Sending a 100 bytes binary file
• Sending a 1kB bytes binary file
• Sending a 50kB binary file
• Sending a 1MB bytes binary file

All responses return "No Content"

• No path parameters
• Single path parameter, coerced into an integer
• Single query parameter, coerced into an integer
• A path and a query parameters, coerced into integers

(not supported by Starlette)

• Resolving 3 nested synchronous dependencies
• Resolving 3 nested asynchronous dependencies (only supported by Litestar and FastAPI)
• Resolving 3 nested synchronous, and 3 nested asynchronous dependencies (only supported by Litestar and FastAPI)

All responses return "No Content"

• Setting response headers
• Setting response cookies

• docker
• poetry
• Python 3.11

1. Clone this repo
2. Run poetry install
3. Run tests with bench run --rps --latency

After the run, the results will be stored in results/run_<run_mumber>.json

To select a framework, simply pass its name to the run command:

bench run --rps litestar starlette fastapi

• A version available on PyPi: bench run --rps [email protected]
• A version from git: bench run --rps litestar@git+branch_or_tag_name
• A version from a specific git repository: bench run --rps litestar@git+https://github.com/litestar-org/litestar.git@branch_or_tag_name
• A local file: bench run --rps litestar@file+/path/to/litestar

You can run a single test by specifying its full name and category:

bench run --rps litestar -t json:json-1K

• Run bench results to generate plots from the latest test results
• Run bench results -s to generate plots from the latest test results and split them into separate files for each category

PRs are welcome.

Please make sure to install pre-commit on your system, and then execute pre-commit install in the repository root - this will ensure the pre-commit hooks are in place.

After doing this, add a PR with your changes and a clear description of the changes.