versioning-benchmarks

Here we will run benchmarks for different data-versioning tools.

Setup

S3 setup

TODO

Repositories

Create a XetHub repository and three github repositories for DVC, git LFS and git-annex with readme's. Here I named them xethub-py, xethub-git, versioning-dvc, versioning-lfs, versioning-lfs-github,

We clone them locally and setup the remotes: Setup your git user name and python environment:

GITUSER=$(git config --global user.name) # or manually set your GitHub/XetHub user name

python -m venv .venv \
&& source .venv/bin/activate \
&& pip install -r requirements.txt

# Download data - takes time! 
python src/download.py --dir=data --download=all --limit=2

# For quick testing. Seed can by any number. 
python src/generate.py --path=0.csv --rows=1000 [--seed=1]
python src/generate.py --path=0.parquet--rows=1000 [--seed=1]

XetHub setup

git xet clone https://xethub.com/$GITUSER/xethub-py.git xethub-pyxet # use your own repository
git xet clone https://xethub.com/xdssio/xethub-git.git xethub-git

Get token and setup as environment variables:

  export XET_USER_NAME=<user-name>
  export XET_USER_TOKEN=<xethub-token>

pip install pyxet
(Optional) Install CLI

DVC setup

git clone https://github.com/$GITUSER/versioning-dvc dvc
Install CLI
pip install dvc dvc-s3

setup remote:

cd dvc
dvc init
dvc remote add -d versioning-article s3://<your-bucket-name>/dvc
dvc remote modify versioning-article region us-west-2

LFS - natural setup

Warning: THIS WILL COST YOU MONEY! Limitations:

GitHub Free and GitHub Pro have a maximum file size limit of 2 GB
GitHub Team has a maximum file size limit of 4 GB
GitHub Enterprise Cloud has a maximum file size limit of 5 GB
Bitbucket Cloud has a maximum file upload limit of 10 GB

Setup:

git clone https://github.com/xdssio/versioning-lfs-github.git lfs-github
Install CLI
cd lfs-github
git lfs install
git lfs track '*.parquet'
git lfs track '*.csv'
git add .gitattributes && git commit -m "Enable LFS" && git push

LFS setup + S3

git clone https://github.com/$GITUSER/versioning-lfs lfs-s3
Install CLI
cd lfs-s3
git lfs install
git lfs track '*.parquet'
git lfs track '*.csv'
git add .gitattributes && git commit -m "Enable LFS" && git push
cd .. # so we can setup the server

LFS server setup - Reference

Generating a random key is easy: openssl rand -hex 32
- Keep this secret and save it in a password manager so you don't lose it. We will pass this to the server below.

Create a lfs-server/.env file with the following contents:

AWS_ACCESS_KEY_ID=XXXXXXXXXXXXXXXXXXXX
AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
AWS_DEFAULT_REGION=us-west-2
LFS_ENCRYPTION_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx # the result of the openssl command above
LFS_S3_BUCKET=my-bucket
LFS_MAX_CACHE_SIZE=10GB

Improve performance (optional)

# Increase the number of worker threads
git config --global lfs.concurrenttransfers 64
# Use a global LFS cache to make re-cloning faster
git config --global lfs.storage ~/.cache/lfs

Update the lfs-s3/.lfsconfig file:

[lfs]
url = "http://http://0.0.0.0:8081/api/my-org/my-project"
           ─────────┬──────── ──┬─ ─┬─ ───┬── ─────┬────
                    │           │   │     │        └ Replace with your project's name
                    │           │   │     └ Replace with your organization name   
                    │           │   └ Required to be "api"
                    │           └ The port your server started with
                    └ The host name of your server

Run local : docker-compose up

LakeFS

Install CLI

On mac: brew tap treeverse/lakefs && brew install lakefs

Run using docker metadata and credentials

mkdir ~/lakefs/metadata  # for persistency 
docker run --pull always -p 8000:8000 -e LAKEFS_BLOCKSTORE_TYPE='s3' -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -e LAKEFS_DATABASE_LOCAL_PATH=/etc/lakefs/metadata -v ~/lakefs/metadata:/etc/lakefs/metadata treeverse/lakefs run --local-settings

Copy credentials ands save to ~/.lakefs.yaml.
Create a repository and connect to S3 in the UI

Run

Prepare docker servers

# Terminal 1
(cd lfs-server && docker-compose up)
# Terminal 2
export AWS_ACCESS_KEY_ID
docker run --pull always -p 8000:8000 -e LAKEFS_BLOCKSTORE_TYPE='s3' -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -e LAKEFS_DATABASE_LOCAL_PATH=/etc/lakefs/metadata -v ~/lakefs/metadata:/etc/lakefs/metadata treeverse/lakefs run --local-settings

Workflows

Numeric non-git

export PYTHONPATH="$(pwd):$PYTHONPATH" XET_LOG_LEVEL=debug XET_LOG_PATH=pwd/numeric.log python src/main.py numeric -i=20 --show --upload

Append to blog csv

export PYTHONPATH="$(pwd):$PYTHONPATH"
XET_LOG_LEVEL=debug XET_LOG_PATH=`pwd`/append.log python src/main.py append -i=30 --show --upload

Mock data

export PYTHONPATH="$(pwd):$PYTHONPATH"
python src/generate.py --dir=mock --count=10 --rows=10
python src/main.py --dir=mock --show --upload

Taxi

export PYTHONPATH="$(pwd):$PYTHONPATH"
python src/download.py --dir=data --download=all --limit=40
python src/main.py --dir=data --show --upload

Tests


PYTHONPATH="$(pwd):$PYTHONPATH" pytest tests

python src/generate.py --dir=mock --count=10 --rows=10
python src/main.py --dir=mock --show # for quick testing

# or
export PYTHONPATH="$(pwd):$PYTHONPATH" 
python src/main.py --dir=mock --rows=10000

Analyse results

Visualize the results with snakeviz:

snakeviz profile.prof

Use jupyter notebook to analyse the results:

pip install jupyter
jupyter notebook

# open the notebook in the browser

All data is pointers

# deprecated
AWS_ACCESS_KEY_ID='LAKEFS_ACCESS_KEY' AWS_SECRET_ACCESS_KEY='LAKEFS_SECRET' aws s3 ls --endpoint http://localhost:8000

CLI

Upload a file:
lakectl fs upload -s mock/0.parquet lakefs://versioning-article/main/0.parquet
Downloading:
lakectl fs download lakefs://<REPO>/<BRANCH>/path/to/object <DESTINATION>
You can also leverage the --recursive flag to download/upload dirs

ylow / versioning-article Goto Github PK

versioning-article's Introduction