Comments (2)
rom1504
commented on August 19, 2024
Hi,
All three versions you mention are free of duplicate and are subset of each other, ie version 2 contains 1, 3 contains 2.
Only the 400M version (the first one) is properly released by us (that's the one we call laion400m) and you can get it from https://the-eye.eu/public/AI/cah/laion400m-met-release/laion400m-meta/ or https://www.kaggle.com/romainbeaumont/laion400m
The other 2 versions you mention are work in progress, and are not yet fully ready for use (for example these versions 2 and 3 are not fully randomly shuffled unlike version 1, which is an important property for use of the dataset)
We will release a larger version of the dataset with a few billions samples in a few months.
Do you have any deadlines / uses of the larger dataset (larger than 400m) on your side?
from laion-prepro.
qiaogh97
commented on August 19, 2024
It doesn't matter, I don't have any deadlines.
from laion-prepro.
Related Issues (20)
- add content HOT 1
- consider shuffling the dataset
- add how to make clip embeddings out of it HOT 1
- add how to make knn indices HOT 1
- add what to train using that (clip, dalle)
- add how to get interesting subsets
- make this more user friendly
- make listing and download of csv files faster and cleaner
- Consider ways to distribute the dataset HOT 3
- Add more information to make this a good home page for the dataset HOT 2
- add command line calls for clip retrieval for cah HOT 1
- How to set '--url_list' parameter in download_images.sh? HOT 2
- How many about the dataset? HOT 3
- md5 check for `.parquet` files HOT 4
- Does https://github.com/rom1504/laion-prepro/blob/main/laion5B/safety/join.py work for non-en langs? HOT 13
- add description of the schema of all laion datasets
- laion400m/download_csv is not available. HOT 1
- Define process and load_clip in data loader HOT 1
- Link to download annotated data
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from laion-prepro.