`block_in_place` + `block_on` can hang on runtime shutdown / runtime drop about tokio HOT 10 OPEN

When you call block_in_place, then the current thread ceases to be a worker thread, and a new worker thread is spawned using spawn_blocking.

from tokio.

I would expect IO resources and timers to return errors once runtime shutdown start, so it surprises me that it is hanging. Could you give more details?

from tokio.

Hmmm...

Maybe it is specific to what I'm doing.

impl Drop for ProcessHandleInner {
    fn drop(&mut self) {
        let Some(child) = &mut self.child else {
            return;
        };
        let name = self.name.clone();
        block_in_place(move || {
            tokio::runtime::Handle::current().block_on(async move {
                debug!(
                    target: LOG_DEVIMINT,
                    "sending SIGKILL to {name} and waiting for it to exit"
                );
                send_sigkill(child);
                if let Err(e) = child.wait().await {
                    warn!(target: LOG_DEVIMINT, "failed to wait for {name}: {e:?}");
                }
            })
        })
    }
}

@Darksonn the child is tokio::process::Child. Could it be it's just this one particular case does doesn't get handled?

from tokio.

Oh, shoot. Now I see it's not actually a send_sigkill and I'm not 100% sure if the process didn't hang. I don't think it did, because they all get killed on ctrl+c all the time reliably, but I'll try to verify.

Edit:
Nah, I changed to send_sigkill and I get the same result.

from tokio.

I pasted relevant part of gdb session: https://pastebin.com/VzHF0B5T , including list of threads and stackstrace that is mostly tokio functions if it's of any help.

from tokio.

@dpc I once also met this issue and got more debugging hints via this tool: https://github.com/tokio-rs/console

@Darksonn According to my own experience last time, is it possible that tokio is running out of worker threads? Last time, I noticed that every time I tokio::spawn 33rd task (my machine has 32 logical cores) and used block_in_place to escape to the sync world, the whole runtime would hang and console says all threads are waiting for signals to wake up. I can't give the exact reproduction unfortunately because I have workarounded it by wrapping the code with tokio::spawn_blocking.

from tokio.

Yes, it's possible to run out of worker threads when you use block_in_place, but the limit would be the number of blocking threads, which is 512 by default. The fact that you're already blocked at 32 tasks tells me that you're probably blocking the thread outside of block_in_place.

from tokio.

Yes, it's possible to run out of worker threads when you use block_in_place, but the limit would be the number of blocking threads, which is 512 by default. The fact that you're already blocked at 32 tasks tells me that you're probably blocking the thread outside of block_in_place.

Cool, that indeed makes sense. My code base is mixing sync and async almost everywhere, though I know it's bad practice.

Is it possible to configure this limit or at least check if we are reaching the limit to avoid any hang?

from tokio.

Quick googling gives this which resolves my questions. =)

from tokio.

block_in_place blocks the worker threads, not blocking threads, no? The whole point of block_in_place is that not having to Send anything, because the current worker thread is taken out of scheduling to avoid moving to a different thread, no?

from tokio.