• 06/08/2024: 🎇 Awesome!!! OV-SAM = OV-DINO + SAM2. We update OV-SAM marrying OV-DINO with SAM2 on the online demo.

• 16/07/2024: We provide the online demo, click and enjoy !!! NOTE: You uploaded image will be stored for failure analysis.

• 16/07/2024: We release the web inference demo, try to deploy it by yourself.

• 15/07/2024: We release the fine-tuning code, try to fine-tune on your custom dataset. Feel free to raise issue if you encounter some problem.

• 15/07/2024: We release the local inference demo, try to deploy OV-DINO on you local machine and run inference on images.

• 14/07/2024: We release the pre-trained models and the evaluation code.

• 11/07/2024: We release OV-DINO paper on arxiv. Code and pre-trained model are coming soon.

This project contains the official PyTorch implementation, pre-trained models, fine-tuning code, and inference demo for OV-DINO.

• OV-DINO is a novel unified open vocabulary detection approach that offers superior performance and effectiveness for practical real-world application.

• OV-DINO entails a Unified Data Integration pipeline that integrates diverse data sources for end-to-end pre-training, and a Language-Aware Selective Fusion module to improve the vision-language understanding of the model.

• OV-DINO shows significant performance improvement on COCO and LVIS benchmarks compared to previous methods, achieving relative improvements of +2.5% AP on COCO and +12.7% AP on LVIS compared to G-DINO in zero-shot evaluation.

NOTE: AP^mv denotes the zero-shot evaluation results on LVIS MiniVal, AP^val denotes the zero-shot evaluation results on LVIS Val, AP^coco denotes (zero-shot / fine-tune) evaluation results on COCO, respectively.

OV-DINO
├── datas
│   ├── coco
│   │   ├── annotations
│   │   ├── train2017
│   │   └── val2017
│   ├── lvis
│   │   ├── annotations
│   │   ├── train2017
│   │   └── val2017
│   └── custom
│       ├── annotations
│       ├── train
│       └── val
├── docs
├── inits
│   ├── huggingface
│   ├── ovdino
│   ├── sam2
│   └── swin
├── ovdino
│   ├── configs
│   ├── demo
│   ├── detectron2-717ab9
│   ├── detrex
│   ├── projects
│   ├── scripts
│   └── tools
├── wkdrs
│   ├── ...
│

# clone this project
git clone https://github.com/wanghao9610/OV-DINO.git
cd OV-DINO
export root_dir=$(realpath ./)
cd $root_dir/ovdino

# create conda env for ov-dino
conda create -n ovdino -y
conda activate ovdino
conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.6 -c pytorch -c nvidia -y
conda install gcc=9 gxx=9 -c conda-forge -y # Optional: install gcc9
python -m pip install -e detectron2-717ab9
pip install -e ./

# Optional: create conda env for ov-sam, it may not compatible with ov-dino, so we create a new env.
# ov-sam = ov-dino + sam2
conda create -n ovsam -y
conda activate ovsam
conda install pytorch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 pytorch-cuda=12.1 -c pytorch -c nvidia -y
# install the sam2 following the sam2 project.
# please refer to https://github.com/facebookresearch/segment-anything-2.git
# download sam2 checkpoints and put them to inits/sam2
python -m pip install -e detectron2-717ab9
pip install -e ./

• Download COCO from the official website, and put them on datas/coco folder.

  cd $root_dir
  wget http://images.cocodataset.org/zips/train2017.zip -O datas/coco/train2017.zip
  wget http://images.cocodataset.org/zips/val2017.zip -O datas/coco/val2017.zip
  wget http://images.cocodataset.org/annotations/annotations_trainval2017.zip -O datas/coco/annotations_trainval2017.zip

• Extract the ziped files, and remove them:

  cd $root_dir
  unzip datas/coco/train2017.zip -d datas/coco
  unzip datas/coco/val2017.zip -d datas/coco
  unzip datas/coco/annotations_trainval2017.zip -d datas/coco
  rm datas/coco/train2017.zip datas/coco/val2017.zip datas/coco/annotations_trainval2017.zip

• Download LVIS annotation files:

  cd $root_dir
  wget https://huggingface.co/hao9610/OV-DINO/resolve/main/lvis_v1_minival_inserted_image_name.json -O datas/lvis/annotations/lvis_v1_minival_inserted_image_name.json
  wget https://huggingface.co/hao9610/OV-DINO/resolve/main/lvis_v1_val_inserted_image_name.json -O datas/lvis/annotations/lvis_v1_val_inserted_image_name.json

• Soft-link COCO to LVIS:

  cd $root_dir
  ln -s $(realpath datas/coco/train2017) datas/lvis
  ln -s $(realpath datas/coco/val2017) datas/lvis

Download the pre-trained model from Model Zoo, and put them on inits/ovdino directory.

cd $root_dir/ovdino
sh scripts/eval.sh path_to_eval_config_file path_to_pretrained_model output_directory

cd $root_dir/ovdino
sh scripts/eval.sh \
  projects/ovdino/configs/ovdino_swin_tiny224_bert_base_eval_coco.py \
  ../inits/ovdino/ovdino_swint_og-coco50.6_lvismv39.4_lvis32.2.pth \
  ../wkdrs/eval_ovdino

cd $root_dir/ovdino
sh scripts/eval.sh \
  projects/ovdino/configs/ovdino_swin_tiny224_bert_base_eval_lvismv.py \
  ../inits/ovdino/ovdino_swint_ogc-coco50.2_lvismv40.1_lvis32.9.pth \
  ../wkdrs/eval_ovdino

sh scripts/eval.sh \
  projects/ovdino/configs/ovdino_swin_tiny224_bert_base_eval_lvis.py \
  ../inits/ovdino/ovdino_swint_ogc-coco50.2_lvismv40.1_lvis32.9.pth \
  ../wkdrs/eval_ovdino

cd $root_dir/ovdino
sh scripts/train.sh \
  projects/ovdino/configs/ovdino_swin_tiny224_bert_base_ft_coco_24ep.py \
  ../inits/ovdino/ovdino_swint_og-coco50.6_lvismv39.4_lvis32.2.pth

• Prepare your custom dataset as the COCO annotation format.

• Refer the following command to run fine-tuning.

  cd $root_dir/ovdino
  sh scripts/train.sh \
    projects/ovdino/configs/ovdino_swin_tiny224_bert_base_ft_custom_24ep.py \
    ../inits/ovdino/ovdino_swint_ogc-coco50.2_lvismv40.1_lvis32.9.pth

• Local inference on a image or folder give the category names.

  # for ovdino: conda activate ovdino
  # for ovsam: conda activate ovsam
  cd $root_dir/ovdino
  sh scripts/demo.sh demo_config.py pretrained_model category_names input_images_or_directory output_directory

  Examples:

  cd $root_dir/ovdino
  # single image inference
  sh scripts/demo.sh \
    projects/ovdino/configs/ovdino_swin_tiny224_bert_base_infer_demo.py \
    ../inits/ovdino/ovdino_swint_ogc-coco50.2_lvismv40.1_lvis32.9.pth \
    "class0 class1 class2 ..." img0.jpg output_dir/img0_vis.jpg
  
  # multi images inference
  sh scripts/demo.sh \
    projects/ovdino/configs/ovdino_swin_tiny224_bert_base_infer_demo.py \
    ../inits/ovdino/ovdino_swint_ogc-coco50.2_lvismv40.1_lvis32.9.pth \
    "class0 long_class1 long_class2 ..." "img0.jpg img1.jpg" output_dir
  
  # image folder inference
  sh scripts/demo.sh \
    projects/ovdino/configs/ovdino_swin_tiny224_bert_base_infer_demo.py \
    ../inits/ovdino/ovdino_swint_ogc-coco50.2_lvismv40.1_lvis32.9.pth \
    "class0 long_class1 long_class2 ..." image_dir output_dir

  NOTE: the input category_names are separated by spaces, and the words of single class are connected by underline (_).

• Web inference demo.

  cd $root_dir/ovdino
  sh scripts/app.sh \
    projects/ovdino/configs/ovdino_swin_tiny224_bert_base_infer_demo.py \
    ../inits/ovdino/ovdino_swint_ogc-coco50.2_lvismv40.1_lvis32.9.pth

  After the web demo deployment, you can open the demo on your browser.

  We also provide the online demo, click and enjoy.

• Release the pre-trained model.
• Release the fine-tuning and evaluation code.
• Support the local inference demo.
• Support the web inference demo.
• Support OV-DINO in 🤗 transformers.
• Release the pre-training code.

This project has referenced some excellent open-sourced repos (Detectron2, detrex, GLIP, G-DINO, YOLO-World). Thanks for their wonderful works and contributions to the community.

If you find OV-DINO is helpful for your research or applications, please consider giving us a star 🌟 and citing it by the following BibTex entry.

@article{wang2024ovdino,
  title={OV-DINO: Unified Open-Vocabulary Detection with Language-Aware Selective Fusion}, 
  author={Hao Wang and Pengzhen Ren and Zequn Jie and Xiao Dong and Chengjian Feng and Yinlong Qian and Lin Ma and Dongmei Jiang and Yaowei Wang and Xiangyuan Lan and Xiaodan Liang},
  journal={arXiv preprint arXiv:2407.07844},
  year={2024}
}