DistributeFilesDataset Sharding with PT Dataloader breaks about returnn HOT 3 CLOSED

Two things:

AttributeError: 'DistributeFilesDataset' object has no attribute '_workers'

Because of early exception in __init__. But anyway, we should avoid this error by checking hasattr or so in __del__. But that's not really the main issue here.

The main issue is that we call _get_rank_and_size in the sub proc. Instead, this should be passed from the parent to the sub proc in serialization. We can pass some _rank and _size as args and if they are passed, use them. That should already fix it.

from returnn.

I think I know what this is

from returnn.

The main issue is that we call _get_rank_and_size in the sub proc. Instead, this should be passed from the parent to the sub proc in serialization. We can pass some _rank and _size as args and if they are passed, use them. That should already fix it.

I feel like we should rather fix the DistributedContext to properly work in and child process, no?

Never mind, I don't think this is feasible because torch will always try to "properly" initialize a process and make it part of the worker group, even if we just want to access some worker group metadata like the size and rank of the parent.

from returnn.