The original repository is here; Thank YuanJianhao508 for doing such amazing work.
This repository only focus on implementing the program on cluster: CSC
/RAG-DRIVER
/llava
- videollava/models
- This folder contains checkpoint models from LangugeBind./LanguageBind_Image
- download from huggingface./LanguageBind_Video_merge
- download from huggingface./Video-LLaVA-7B
- download from huggingface./Video-LLaVA-Pretrain-7B
- download from huggingface.
/rag_env
- RAG-Driver virtual environment for containerizationREADME.md
- Overview and general information about the project..gitignore
- Specifies patterns of files to ignore in Git operations.
NOTE: The virtual environment require load 2 modules: tykky and cuda.
git clone https://github.com/timbrist/RAG-Driver.git
cd RAG-Driver # we will continue use this folder as WORKSPACE, This will be the only change directory of all.
export CW_DEBUG_KEEP_FILES=${pwd}
bash rag_env/create_rag_env.sh
After the installation, we need to add additional package.
The reason to do it sperately is because tykky can not install pip install flash-attn --no-build-isolation
and can not use extra parameter to create new environment.
Please check the create_rag_env.sh file to make sure the version of cuda is above 11.7
module spider cuda #use this to check which cuda your system support, use cuda/11.7+ please.
module cuda
conda-containerize update --post-install rag_env/restpackages.sh ./rag_env
This step is for people who cannot use git lfs to download files from hugginface. This script will automatically download the checkpoint models.
NOTE: The checkpoint models will take up at least: 46GB. If you don't have much space in current directory, please change MODELS_DIR in download.sh to desired diretory: export MODELS_DIR=path/to/models. In this case, you will have to specify your model path in scripts/finetune.sh too.
bash models/download.sh
Processed Version of BDD-X is available from here
If you want to download the dataset manually, Please unzip the file into video_process folder
Then run the following command:
bash video_process/download_bdd.sh
Cannot find a way to automate the process. A lot of things need to config in this step.
NOTE: Please remember to change:
MODELS_DIR in scripts/finetune.sh
CACHESPACE in run_rag.sh and test_rag.sh
--account=<project>
specify your project name such as project_2010795
the script is follow the exmaple on Puhti
We will testing if everything will be ok before we submitted to expensive GPU Cluster.
sbatch ./slurm_jobs/test_rag.sh
tail -f slurm-*
sbatch ./slurm_jobs/run_rag.sh
tail -f slurm-*
Wait about 12 hours