Cannot load huggingface:imagenet-1k dataset due to parse error about datasets HOT 5 OPEN

Hello @christian-steinmeyer !

As HF and TFDS have different naming rules, you will have to adapt the dataset name to follow TFDS' naming: in this case, the correct name would be huggingface:imagenet_1k

As a pointer, you can refer to the from_hf_to_tfds function under:

We will update our documentation so that this is clearer for users!

from datasets.

That worked, thanks! And yes, an update in the documentation would be very helpful!

from datasets.

@ccl-core Quick follow-up question: Downloading the dataset worked - however, after generating splits, the load function also includes the step of generating tfrecords (Output "Generating training examples..."), which is pretty slow for me (~20 examples/s). Is there any way to speed this up? I couldn't find anything in the builder config or the download and prepare config. The number of available CPUs doesn't seem to be a factor. For Imagenet-1k, this is taking many hours.

from datasets.

Hi again! I found the tfds_num_proc argument of the hugginface dataset builder. However, it doesn't seem to be what I'm looking for. Using a number equal to my cpu count or half / quarter times that, there is no progress printed in the generating training examples... step, only my ram fills up and then at some point it crashes.

tfds.load(
    'huggingface:imagenet_1k',
    data_dir=IMAGE_DIR,
    shuffle_files=True,
    builder_kwargs={"tfds_num_proc": N_JOBS}
)

In the meantime, my original try ran through (without builder_kwargs). However, when I use this in a training run, I get tons of warnings like W tensorflow/core/lib/png/png_io.cc:88] PNG warning: 1CCP: known incorrect profile or profile 'ICC PRofile': 'RGB ': RGB color space not permitted on grayscale PNG. Both of which to me seem like a misconfiguration of the dataset somehow. Or is this expected?

from datasets.

@ccl-core Quick follow-up question: Downloading the dataset worked - however, after generating splits, the load function also includes the step of generating tfrecords (Output "Generating training examples..."), which is pretty slow for me (~20 examples/s). Is there any way to speed this up? I couldn't find anything in the builder config or the download and prepare config. The number of available CPUs doesn't seem to be a factor. For Imagenet-1k, this is taking many hours.

Same problem here. It runs ~20 examples/s and eventually after a day or so it crashes.

from datasets.