--num-processes causes build to error about datasets HOT 6 CLOSED

Parallelization is done by processing each config in a separate process:

The way multiprocessing works in Python it would pickle each builder (including references to Mk0 module) in the main process and then unpickle it in the child processes. During unpickling it must be able to import Mk0, which is most probably not installed in your Python environment.

You could make your dataset builder code discoverable by Python (e.g. including it into TFDS and reinstalling the library locally). Or perhaps you could consider implementing parallelization in your dataset builder or use Beam.

from datasets.

#5279 should make children processes aware of your dataset builder.

from datasets.

#5279 should make children processes aware of your dataset builder.

I can now run with the --num-processes flag which is pretty cool. Thank you for fixing that!

This means that there's no reason to set -num-processes higher than the number of distinct configs for your dataset.

Is it possible to effectively split my dataset into different configs then merge them together when done?

Or perhaps you could consider implementing parallelization in your dataset builder

My code unfortunately isn't the slow part. Its my abuse of TFDS due to me having 138 features in my dataset . My code takes 2.5 seconds per 3000 examples meanwhile my generation happens at 80 examples per second.

from datasets.

I can now run with the --num-processes flag which is pretty cool. Thank you for fixing that!

Great, really glad that it worked for you! I'll be closing this issue then.

Is it possible to effectively split my dataset into different configs then merge them together when done?

TFDS doesn't natively support mixing datasets, but you can use some other tools for that, e.g. https://github.com/google/seqio

My code unfortunately isn't the slow part. Its my abuse of TFDS due to me having 138 features in my dataset . My code takes 2.5 seconds per 3000 examples meanwhile my generation happens at 80 examples per second.

It's usually very straightforward to parallelize examples generation:

Or with Beam:

from datasets.

Ok. Thanks for you for your help with this.

I'll look into parallel example generation. and see if I can get that working with my dataset.

from datasets.

Hi, I have unfortunately encountered the same issues and had no luck on trying tf-nightly or multiprocessing with python 3.9/3.10 (py3.10 was necessary to use tensorflow-datasets==4.9.4 and nightly versions with the fix) with MacOS.

However, I think I managed to bypass this using BUILDER_CONFIGS. Let's say I want to generate the datasets in parallel with 10 workers, I simply need to define 10 configs as below,

class Builder(tfds.core.GeneratorBasedBuilder, skip_registration=True):
    VERSION = tfds.core.Version("1.0.0")
    BUILDER_CONFIGS: ClassVar[list[tfds.core.BuilderConfig]] = [
        tfds.core.BuilderConfig(name=str(group)) for group in range(1,11)
    ]

then run the CLI per group one by one.

tfds build my_dataset --config 1

Later we can load and concatenate the datasets as below

ds1 = tfds.load('my_dataset/1', split='train')
ds2 = tfds.load('my_dataset/2', split='train')
ds = tf.data.Dataset.sample_from_datasets([ds1, ds2])

I tested locally, it does seem to work. Please let me know if there are any hidden traps 🙌 If not, hope it could help others who are facing same issues!

from datasets.