Hang in dataset iterator about returnn HOT 5 CLOSED

I realize, the zombie subproc is definitely not one of the MPD workers, as you see that all the others are complete.
The first unnamed subproc is multiprocessing/resource_tracker.
The second unnamed subproc is multiprocessing/managers.
The main proc hangs in starting the subproc for PyTorch _MultiProcessingDataLoaderIter. So this is the subproc which raised this exception and then crashed and which hangs in zombie state.

The corresponding code from the main proc in PyTorch dataloader _MultiProcessingDataLoaderIter (torch/utils/data/dataloader.py:1039) is:

            w = multiprocessing_context.Process(
                target=_utils.worker._worker_loop,
                args=(self._dataset_kind, self._dataset, index_queue,
                      self._worker_result_queue, self._workers_done_event,
                      self._auto_collation, self._collate_fn, self._drop_last,
                      self._base_seed, self._worker_init_fn, i, self._num_workers,
                      self._persistent_workers, self._shared_seed))
            w.daemon = True
            # NB: Process.start() actually take some time as it needs to
            #     start a process and pass the arguments over via a pipe.
            #     Therefore, we only add a worker to self._workers list after
            #     it started, so that we do not call .join() if program dies
            #     before it starts, and __del__ tries to join but will get:
            #     AssertionError: can only join a started process.
            w.start()   # <---- hang here

The w is a NonDaemonicSpawnProcess, and it has an instance of _DataLoaderWorkerPreInitFunc for pre_init_func. The unpickling of this fails.

from returnn.

As the code happens in our custom pre_init_func handling logic (not directly there, but we can catch it there), we will catch the exception there and try to do some proper cleanup such that the parent should properly get the SIGPIPE.

from returnn.

I found the bug in CPython itself, reported here: python/cpython#118981

I'm not sure that we can really find a good workaround for this. We could try to delay the exception. I need to think more about it.

from returnn.

(Note, the PR #1515 just adds a workaround for the hang. So it does not hang anymore now. The CPython PR python/cpython#118982 should fix the hang in another cleaner way as well. But we still get the UnicodeDecodeError here.)

from returnn.

(Note, the UnicodeDecodeError was fixed in aa41c4a. But it's really unrelated to the hang.)

from returnn.