This is the official pytorch implementation of BCNet built on the open-source detectron2.
Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers
Lei Ke, Yu-Wing Tai, Chi-Keung Tang
CVPR 2021
- Two-stage instance segmentation with state-of-the-art performance.
- Image formation as composition of two overlapping layers.
- Bilayer decoupling for the occluder and occludee.
- Efficacy on both the FCOS and Faster R-CNN detectors.
Under construction. Our code and pretrained model will be fully released in two months.
![]() Qualitative instance segmentation results of our BCNet, using ResNet-101-FPN and FCOS detector. |
(Check Table 8 of the paper for full results, all methods are trained on COCO train2017)
Detector | Backbone | Method | mAP(mask) |
---|---|---|---|
Faster R-CNN | ResNet-50 FPN | Mask R-CNN | 34.2 |
Faster R-CNN | ResNet-50 FPN | MS R-CNN | 35.6 |
Faster R-CNN | ResNet-50 FPN | PointRend | 36.3 |
Faster R-CNN | ResNet-50 FPN | PANet | 36.6 |
Faster R-CNN | ResNet-50 FPN | BCNet | 38.4 |
Faster R-CNN | ResNet-101 FPN | Mask R-CNN | 36.1 |
Faster R-CNN | ResNet-101 FPN | BMask R-CNN | 37.7 |
Faster R-CNN | ResNet-101 FPN | MS R-CNN | 38.3 |
Faster R-CNN | ResNet-101 FPN | BCNet | 39.8, [Pretrained Model] |
FCOS | ResNet-101 FPN | SipMask | 37.8 |
FCOS | ResNet-101 FPN | BlendMask | 38.4 |
FCOS | ResNet-101 FPN | CenterMask | 38.3 |
FCOS | ResNet-101 FPN | BCNet | 39.6, [Pretrained Model] |
Segmenting highly-overlapping objects is challenging, because typically no distinction is made between real object contours and occlusion boundaries. Unlike previous two-stage instance segmentation methods, BCNet models image formation as composition of two overlapping layers, where the top GCN layer detects the occluding objects (occluder) and the bottom GCN layer infers partially occluded instance (occludee). The explicit modeling of occlusion relationship with bilayer structure naturally decouples the boundaries of both the occluding and occluded instances, and considers the interaction between them during mask regression. We validate the efficacy of bilayer decoupling on both one-stage and two-stage object detectors with different backbones and network layer choices. The network of BCNet is as follows:
![]() |
conda create -n bcnet python=3.7 -y
source activate bcnet
conda install pytorch==1.4.0 torchvision==0.5.0 cudatoolkit=10.1 -c pytorch
# FCOS and coco api and visualization dependencies
pip install ninja yacs cython matplotlib tqdm
pip install opencv-python==4.4.0.40
export INSTALL_DIR=$PWD
# install pycocotools. Please make sure you have installed cython.
cd $INSTALL_DIR
git clone https://github.com/cocodataset/cocoapi.git
cd cocoapi/PythonAPI
python setup.py build_ext install
# install BCNet
cd $INSTALL_DIR
git clone https://github.com/lkeab/BCNet.git
cd BCNet/
python3 setup.py build develop
unset INSTALL_DIR
Prepare for coco2017 dataset following this instruction. And use our converted mask annotations to replace original annotation file for bilayer decoupling training.
mkdir -p datasets/coco
ln -s /path_to_coco_dataset/annotations datasets/coco/annotations
ln -s /path_to_coco_dataset/train2017 datasets/coco/train2017
ln -s /path_to_coco_dataset/test2017 datasets/coco/test2017
ln -s /path_to_coco_dataset/val2017 datasets/coco/val2017
bash all.sh
Or
CUDA_VISIBLE_DEVICES=0,1 python3 tools/train_net.py --num-gpus 2 \
--config-file configs/fcos/fcos_imprv_R_50_FPN_1x.yaml 2>&1 | tee log/train_log.txt
TBD
mkdir pretrained_models
#And put the downloaded pretrained models in this directory.
TBD
bash eval.sh
If you find BCNet useful in your research, please star this repository and consider citing:
@inproceedings{ke2021bcnet,
author = {Ke, Lei and Tai, Yu-Wing and Tang, Chi-Keung},
title = {Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers},
booktitle = {CVPR},
year = {2021},
}
BCNet is released under the MIT license. See LICENSE for additional details. Thanks to the Third Party Libs detectron2