qweeze / uring_file Goto Github PK

First of all, thank you for your example code. Would it be possible that you add a license file for your project (see e.g. the Liburing project)?

Hello,

Thank you for your example code.

I was looking for a library that can operate files using io_uring for my personal interest, and finally I found this repository.

Your example code for integrating io_uring and asyncio was very interesting.
So I forked this repository and I've been writing a new library based on your code.
https://github.com/tree4096/uring_file

Now I achieved excellent R/W performance compared with the other libraries such as aiofiles, aiofile, and anyio.

I tested writing 100 lines of text to 1000 files in 1 second and measured the time to completion.
And I also tested reading 1000 files in 0.1 second.
https://github.com/tree4096/uring_file/blob/master/performance_test.py

Here is the results.

Writing Test

Reading Test

*aiofile has Linux AIO mode, but does not work on Ubuntu due to the bug. So it run on C threading mode. See mosquito/caio#17 (comment)

Environment

OS: Ubuntu 20.04.4 LTS on WSL2
Kernel ver.: 5.10.102.1-microsoft-standard-WSL2
CPU: AMD Ryzen 7 5800U
SSD: MTFDHBA1T0TDV-1AZ1AABHA
Python ver.: 3.10.5

In these results, uring_file in my repository is much faster than other libs and its process time (the sum of CPU time not include sleeping) is very small because it may run on single thread.

In addition, my library also provides the low level APIs to easily access the io_uring asynchronously.
Here is the sample code.

# Create new Uring
uring = uring_file.Uring(sq_size=8, cq_size=64, session_sq_size=4)

# Get SQE and submit (Open file)
async with uring.session() as session:
    sqe = session.get_sqe()
    sqe.prep_openat(liburing.AT_FDCWD, b"hello.txt", os.O_RDONLY, 0o644)

# Get result of CQE
fd = sqe.result()
print("FD:", fd)

# Close file
async with uring.session() as session:
    # Raw SQE object can be accessed from the _sqe property
    liburing.io_uring_prep_close(session.get_sqe()._sqe, fd)
    # Same expression as session.get_sqe().prep_close(fd)

# Get default Uring
default_uring = uring_file.get_default_uring()

These APIs enable to access the io_uring like a database session.
The SQEs of multiple sessions are automatically submitted together under high frequency access, and I call it "auto bulk submit".
The auto bulk submit is realized by controlling the task scheduling of asyncio in a little hacky way and it makes greate performance improvement.
And the overflow protection for the CQ is also implemented to prevent the error message, "Device or resource busy".

If your are interested in my library, would you give me some comments or advice?

Awaiting for your answer.

Hi, I tried running this code:

import string
import asyncio
import uring_file
import os

async def main():
    path = 'test.txt'

    f = uring_file.File(path)
    await f.open(os.O_CREAT | os.O_WRONLY)
    await f.write(b'hello\nworld')
    await f.close()


if __name__ == "__main__":
    asyncio.run(main())

I get the following error:

ubuntu@pc:~$ python3 main.py 
Exception in callback Uring.eventfd_callback()
handle: <Handle Uring.eventfd_callback()>
Traceback (most recent call last):
  File "/usr/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/home/ubuntu/uring_file.py", line 48, in eventfd_callback
    result = liburing.trap_error(cqe.res)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/liburing/wrapper.py", line 27, in trap_error
    raise OSError(-no, strerror(-no))
OSError: [Errno 9] Bad file descriptor

These are the system details. It is an AWS machine.

Distributor ID: Ubuntu
Description:    Ubuntu 20.04.2 LTS
Release:        20.04
Codename:       focal

5.4.0-1038-aws

I installed liburing from source. Could you tell me if I'm doing something wrong?