how to implement flow-control about during HOT 5 CLOSED

Hello @ikod, definitelly an interesting question.
I've tried this sample on my fedora box with 5.6.19-200.fc31.x86_64 and it just hung completely (still full and no return from io_uring_enter in consumer).
On Ubuntu 20.04 with 5.4.0-39-generic it works as you describe.

Normally (IIRC) completion queue is set to have two times the capacity of submission queue and are somewhat independent.
My guess about overflows would be that kernel processes submission queue faster than results are consumed from completion queue and so the overflows.

I've looked for similar questions in axboe/liburing:

• axboe/liburing#103
• axboe/liburing#109
• axboe/liburing#125
• axboe/liburing#127
• axboe/liburing#129
• axboe/liburing#140

But I'm still not sure how it should behave in this case and if this is even supported.

I've tried to reproduce same behaviour with liburing directly in C to avoid some possible issues in this lib:

#include <assert.h>
#include <pthread.h>
#include <signal.h>
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <unistd.h>

#include "liburing.h"

#define ITERATIONS 10000000
#define RING_SIZE 32

struct io_uring ring;
static void sig_int(int sig) {
    exit(0);
}

static void *consumer(void *data) {
    unsigned long completions = 0;
    struct io_uring_cqe *cqe;
    int ret;

    while (completions < ITERATIONS) {
        ret = io_uring_wait_cqe(&ring, &cqe);
        if (ret < 0) {
            fprintf(stderr, "io_uring_wait_cqe=%d (%s)\n", ret, strerror(-ret));
            return NULL;
        }
        if (cqe) {
            io_uring_cqe_seen(&ring, cqe);
            completions++;
        }
    }
    printf("completions=%lu\n", completions);
    return NULL;
}

int main(int argc, char *argv[]) {
    pthread_t t;
    struct io_uring_sqe *sqe;
    int ret;
    unsigned long submissions = 0;

    signal(SIGINT, sig_int);

    assert(io_uring_queue_init(RING_SIZE, &ring, 0) >= 0);

    pthread_create(&t, NULL, consumer, NULL);

    while (submissions < ITERATIONS) {
        sqe = NULL;
        while (sqe = io_uring_get_sqe(&ring), !sqe) {
            // short sleep while sq is full
            usleep(10);
        }

        io_uring_prep_nop(sqe);
        sqe->user_data = submissions;
        ret = io_uring_submit(&ring);
        if (ret != 1) {
            fprintf(stderr, "io_uring_submit=%d (%s) - submissions = %lu\n", ret, strerror(-ret), submissions);
            continue;
        }
        submissions++;
    }

    printf("submissions=%lu\n", submissions);
    printf("overflow=   %d\n", *ring.cq.koverflow);

    pthread_join(t, NULL);
    return 0;
}

It's basically same story.
On my Fedora box it hangs for the same reason. On ubuntu there are completion queue overflows.
Plus it outputs some Device or resource busy errors on submit (which doesn't happen with this lib).

Personally I'd try to avoid this usage and use the io_uring just from one thread and figure out some other means to pass data between threads that doesn't involve kernel syscalls.

We're using uring in a project where in one thread we're handling TCP server requests and file reads/writes at the same time and also using lockfree queue (SPSC) to pass data to consumer thread.
Eventfd can be used to help synchronizing threads with separate rings.

Hope this helps.

from during.

Hello, @tchaloupka
Thanks for the clarifications. most likely this mean that uring is not developed to support this use case.

I tried to implement http server following scheme: first thread use io_uring to handle all io completions and using lockfree queue to pass all these completed events(accepted connections, send/receive completions) to "computing" thread, where these events are handled.

So main idea is to have one thread which only wait on IO completions, and second thread which only execute "computations". Here is single problem - how to send reply when "computation" finished. I hoped that I can send data directly from from "computing" thread using same shared io_uring, but looks like this will not work.

Thanks anyway for nice library!

UPD: Forgot to mention that maybe some memory barriers can fix the problem.

from during.

Yeah some synchronization primitives to avoid using same ring from multiple threads at the same time might help, but it'll loose the main benefit of lockfree kernel queue :)

If it's just for http server, I'll go with a worker threads with own uring in each using listen on shared port and make it all isolated.
We're using it in a similar way, just subprocesses instead of threads. Each subprocess handles connected client from start to end and that's it.

If you want to manually decide which worker should handle which client, we're again using similar case (we need to send client to worker that handles concrete resource - each worker handles different subset). For that we have one listener thread that accepts the clients, based on some initial client data decides to which worker it should be passed to and uses unix socket with sendmsg to send client socket to concrete worker subprocess.

from during.

from during.

I guess I can close this as it's more a usecase problem with io_uring itself than with this lib.

from during.