Hi @rom1504!
Thanks for open-sourcing the code! Great work!
The project page mentions that 3456 samples of the LAION-400M were annotated for identifying NSFW content. Is is possible to download these annotations with the corresponding image-text pairs?

Thanks!

Hi! Awesome repo, thanks for building this.

I have 10TB of the Laion dataset downloaded, thanks to your scripts! However, I'm trying to use your data loader, and ran into an issue.

In your WebdatasetReader object, you have a preprocess argument passed. However, this isn't defined in the script.

It seems to come from a ghost function, load_clip(). I don't see this defined, or used anywhere in your repo for that matter. Could you explain? Thanks.

Hi, I'm looking for original urls for each image
http://the-eye.eu/eleuther_staging/cah/ is not accessible anymore
is there any raw form of dataset available for LAION-400m, 2B?

Thanks

• what is cah
• how was it crawled
• how to use it
• what can be done with it

options:

• https://huggingface.co/docs/datasets/add_dataset.html
• https://www.kaggle.com/datasets
• https://github.com/activeloopai/Hub

Those should make it easy to distribute the dataset of url+captions

Worth investigating about the images

it is done currently by bash + aria2c
it takes about 30min
Could easily take only 5min if done with the same techniques as img2dataset

Can you report the md5 value for 32 .parquet files?
To verify that all meta data is downloaded successfully.

• https://gist.github.com/rom1504/f427b1c82df26c9993daa36fca7f9881 download csv
• https://gist.github.com/rom1504/0b846b2dc64c5e0604e1d532c09cbff6 make them unique parquets
• http://3080.rom1504.fr/cah/cah_dataframe_unique/
• visualize https://colab.research.google.com/drive/14Hc_fUUOrG9260VzD_XsTxWX7f5cptyL
• download with img2dataset rom1504/img2dataset#17 (comment)

Then add:

• how to make clip embeddings out of that
• how to make knn indices
• what to train using that (clip, dalle)
• how to get interesting datasets

Hi, @rom1504
I know there are three versions of the parquet files as below.

So I wonder know if the parquet files in different versions are one-to-one correspondence.
I download the 400 million version dataset. What should I do if I'd like to download the newest version of the dataset without downloading the duplicate files?

of https://huggingface.co/laion

Hi @rom1504 !
If I'd like to download images from 'part-[00000-00031]-03f11a48-0c63-4b59-a590-c03169a0d265-c000.snappy.parquet', how to set the '--url_list' parameter? Should I make a dir named 'laion400m-meta' and put all the *.parquet in this dir?

Another question, can I set '--url_list' as one of the '*.parquet' files to download part of this dataset? Like,
img2dataset --url_list part-00000-03f11a48-0c63-4b59-a590-c03169a0d265-c000.snappy.parquet --input_format "parquet"\ --url_col "URL" --caption_col "TEXT" --output_format webdataset\ --output_folder your_output_folder --processes_count 16 --thread_count 128 --image_size 256\ --save_additional_columns '["NSFW","similarity","LICENSE"]' --enable_wandb True

Issue

Our team requires removal of all nsfw content (especially nudity)

Fix

I see here - https://laion.ai/laion-5b-a-new-era-of-open-large-scale-multi-modal-datasets/ we are pointed at this script:

https://github.com/rom1504/laion-prepro/blob/main/laion5B/safety/join.py

However, I see references to 2B rather than 5B

Question

Is the script above usable for non-en langs? Or does the script only work for en langs?

• package installation instructions
• add path options
• make it easy to switch on/off incremental mode
• add end to end scheduling script

Hi, @rom1504
I download the 32 parquet files and compute the total of url. I find about 26760000 urls in every parquet, and 32*26760000 = 800 million. But you said the number of this dataset is 400m?
So what is the difference?