Comments (5)
Hey why did you close it ?
I think it's a good improvement and people will review the PRs soon
from datacomp.
from datacomp.
from datacomp.
Hello! I closed the issue because it wasn't quite actionable, but rather a βnote to my future selfβ that could eventually become documentation. π I'll reopen it if you wish, though.
from datacomp.
Alternative version, without containers.
cluster_name: datacomp-downloader
min_workers: 0
max_workers: 10
upscaling_speed: 1.0
provider:
type: aws
region: us-east-1
cache_stopped_nodes: false
available_node_types:
ray.head.default:
resources: {}
node_config:
InstanceType: m5.12xlarge
ImageId: ami-068d304eca3399469
BlockDeviceMappings:
- DeviceName: /dev/sda1
Ebs:
DeleteOnTermination: true
VolumeSize: 200
VolumeType: gp2
ray.worker.default:
resources: {}
node_config:
InstanceType: m5.12xlarge
ImageId: ami-068d304eca3399469
BlockDeviceMappings:
- DeviceName: /dev/sda1
Ebs:
DeleteOnTermination: true
VolumeSize: 200
VolumeType: gp2
initialization_commands:
# Knot Resolver
- wget https://secure.nic.cz/files/knot-resolver/knot-resolver-release.deb
- sudo dpkg --install knot-resolver-release.deb
- rm knot-resolver-release.deb
- sudo apt-get update
- sudo apt-get install --yes knot-resolver
- echo $(hostname --all-ip-addresses) $(hostname) | sudo tee --append /etc/hosts
- sudo systemctl start kresd@{1..48}.service
- echo nameserver 127.0.0.1 | sudo tee /etc/resolv.conf
- sudo systemctl stop systemd-resolved
# Anaconda
- sudo mkdir /opt/miniconda3 && sudo chown $USER /opt/miniconda3
- wget https://repo.anaconda.com/miniconda/Miniconda3-py39_22.11.1-1-Linux-x86_64.sh
- bash Miniconda3-py39_22.11.1-1-Linux-x86_64.sh -f -b -p /opt/miniconda3
- rm Miniconda3-py39_22.11.1-1-Linux-x86_64.sh
- /opt/miniconda3/bin/conda init bash
# Ray
- conda create --yes --name=ray python=3.10.8
- echo conda activate ray >> ~/.bashrc
- pip install ray[all]==2.7.0
setup_commands:
- sudo apt-get update
- sudo apt-get install --yes build-essential ffmpeg
from datacomp.
Related Issues (20)
- Get 400 error when submitting jsonl to firebase, but successfully submit Slack notification HOT 2
- How to precompute and save model-based metric during download? HOT 2
- Workshop submission deadline HOT 3
- Appendix in the workshop paper submission HOT 2
- Will the training framework do upsampling when train-num-samples is far more than the amount of actual data HOT 8
- Not able to push data to google cloud storage HOT 1
- Tried evaluate the model on a local network only machine HOT 4
- FMoW dataset and results variance HOT 1
- Dataset Size on Leaderboard HOT 1
- Conda environment build issue HOT 3
- 14% of SHA256 hashes not matching HOT 32
- the normal success rate and downloading speed? HOT 1
- `zeroshot_templates` split error for FairFace / UTKFace HOT 9
- Deduplication against evaluation sets HOT 1
- Remove CSAM, if present HOT 2
- Metadata for datacomp-large text-based filter HOT 1
- Pretraining dataset HOT 1
- Training log HOT 1
- Frequency of Leaderboard Updates HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from datacomp.