Here we will run benchmarks for different data-versioning tools.
TODO
Create a XetHub repository and three github repositories for DVC, git LFS and git-annex with readme's.
Here I named them xethub-py
, xethub-git
, versioning-dvc
, versioning-lfs
, versioning-lfs-github
,
We clone them locally and setup the remotes: Setup your git user name and python environment:
GITUSER=$(git config --global user.name) # or manually set your GitHub/XetHub user name
python -m venv .venv \
&& source .venv/bin/activate \
&& pip install -r requirements.txt
# Download data - takes time!
python src/download.py --dir=data --download=all --limit=2
# For quick testing. Seed can by any number.
python src/generate.py --path=0.csv --rows=1000 [--seed=1]
python src/generate.py --path=0.parquet--rows=1000 [--seed=1]
-
git xet clone https://xethub.com/$GITUSER/xethub-py.git xethub-pyxet
# use your own repository -
git xet clone https://xethub.com/xdssio/xethub-git.git xethub-git
-
Get token and setup as environment variables:
export XET_USER_NAME=<user-name> export XET_USER_TOKEN=<xethub-token>
-
pip install pyxet
-
(Optional) Install CLI
git clone https://github.com/$GITUSER/versioning-dvc dvc
- Install CLI
pip install dvc dvc-s3
- setup remote:
cd dvc dvc init dvc remote add -d versioning-article s3://<your-bucket-name>/dvc dvc remote modify versioning-article region us-west-2
Warning: THIS WILL COST YOU MONEY! Limitations:
- GitHub Free and GitHub Pro have a maximum file size limit of 2 GB
- GitHub Team has a maximum file size limit of 4 GB
- GitHub Enterprise Cloud has a maximum file size limit of 5 GB
- Bitbucket Cloud has a maximum file upload limit of 10 GB
Setup:
git clone https://github.com/xdssio/versioning-lfs-github.git lfs-github
- Install CLI
cd lfs-github
git lfs install
git lfs track '*.parquet'
git lfs track '*.csv'
git add .gitattributes && git commit -m "Enable LFS" && git push
git clone https://github.com/$GITUSER/versioning-lfs lfs-s3
- Install CLI
cd lfs-s3
git lfs install
git lfs track '*.parquet'
git lfs track '*.csv'
git add .gitattributes && git commit -m "Enable LFS" && git push
cd ..
# so we can setup the server- LFS server setup - Reference
-
Generating a random key is easy:
openssl rand -hex 32
- Keep this secret and save it in a password manager so you don't lose it. We will pass this to the server below.
-
Create a lfs-server/.env file with the following contents:
AWS_ACCESS_KEY_ID=XXXXXXXXXXXXXXXXXXXX AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX AWS_DEFAULT_REGION=us-west-2 LFS_ENCRYPTION_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx # the result of the openssl command above LFS_S3_BUCKET=my-bucket LFS_MAX_CACHE_SIZE=10GB
-
Improve performance (optional)
# Increase the number of worker threads git config --global lfs.concurrenttransfers 64 # Use a global LFS cache to make re-cloning faster git config --global lfs.storage ~/.cache/lfs
-
- Update the lfs-s3/.lfsconfig file:
[lfs]
url = "http://http://0.0.0.0:8081/api/my-org/my-project"
─────────┬──────── ──┬─ ─┬─ ───┬── ─────┬────
│ │ │ │ └ Replace with your project's name
│ │ │ └ Replace with your organization name
│ │ └ Required to be "api"
│ └ The port your server started with
└ The host name of your server
- Run local :
docker-compose up
- On mac:
brew tap treeverse/lakefs && brew install lakefs
- Run using docker metadata and credentials
mkdir ~/lakefs/metadata # for persistency docker run --pull always -p 8000:8000 -e LAKEFS_BLOCKSTORE_TYPE='s3' -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -e LAKEFS_DATABASE_LOCAL_PATH=/etc/lakefs/metadata -v ~/lakefs/metadata:/etc/lakefs/metadata treeverse/lakefs run --local-settings
- Copy credentials ands save to
~/.lakefs.yaml
. - Create a repository and connect to S3 in the UI
# Terminal 1
(cd lfs-server && docker-compose up)
# Terminal 2
export AWS_ACCESS_KEY_ID
docker run --pull always -p 8000:8000 -e LAKEFS_BLOCKSTORE_TYPE='s3' -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -e LAKEFS_DATABASE_LOCAL_PATH=/etc/lakefs/metadata -v ~/lakefs/metadata:/etc/lakefs/metadata treeverse/lakefs run --local-settings
export PYTHONPATH="$(pwd):$PYTHONPATH"
XET_LOG_LEVEL=debug XET_LOG_PATH=pwd
/numeric.log python src/main.py numeric -i=20 --show --upload
export PYTHONPATH="$(pwd):$PYTHONPATH"
XET_LOG_LEVEL=debug XET_LOG_PATH=`pwd`/append.log python src/main.py append -i=30 --show --upload
export PYTHONPATH="$(pwd):$PYTHONPATH"
python src/generate.py --dir=mock --count=10 --rows=10
python src/main.py --dir=mock --show --upload
export PYTHONPATH="$(pwd):$PYTHONPATH"
python src/download.py --dir=data --download=all --limit=40
python src/main.py --dir=data --show --upload
PYTHONPATH="$(pwd):$PYTHONPATH" pytest tests
python src/generate.py --dir=mock --count=10 --rows=10
python src/main.py --dir=mock --show # for quick testing
# or
export PYTHONPATH="$(pwd):$PYTHONPATH"
python src/main.py --dir=mock --rows=10000
Visualize the results with snakeviz:
snakeviz profile.prof
Use jupyter notebook to analyse the results:
pip install jupyter
jupyter notebook
# open the notebook in the browser
All data is pointers
# deprecated
AWS_ACCESS_KEY_ID='LAKEFS_ACCESS_KEY' AWS_SECRET_ACCESS_KEY='LAKEFS_SECRET' aws s3 ls --endpoint http://localhost:8000
Upload a file:
lakectl fs upload -s mock/0.parquet lakefs://versioning-article/main/0.parquet
Downloading:
lakectl fs download lakefs://<REPO>/<BRANCH>/path/to/object <DESTINATION>
You can also leverage the --recursive flag to download/upload dirs