This is an implementation of Mask R-CNN on Python 3, Keras, and TensorFlow.Privacy is protected by detecting sensitive objects and segmenting those from video.The model generates bounding boxes and segmentation masks for each instance of an object in the image. It's based on Feature Pyramid Network (FPN) and a ResNet101 backbone as feature extractor. Later better version was developed using different framework. Link of that repository.

Some example of the work :

The post processing part after instance segmentation can be described as :

The repository includes:

• Source code of Mask R-CNN built on FPN and ResNet101.
• Jupyter notebooks to visualize the detection pipeline at every step
• Source code for Instance Segmentation in a Video
• Source code for Privacy protection for single image
• Source code for Privacy protection in a video

1. Clone this repository

2. Install dependencies

   pip3 install -r requirements.txt

3. Run setup from the repository root directory

   python3 setup.py install

4. Download pre-trained COCO weights (mask_rcnn_coco.h5) from the releases page.Download it from Mask R-CNN 2.0 -> assets -> mask_rcnn_coco.h5

5. (Optional) To train or test on MS COCO install pycocotools from one of these repos. They are forks of the original pycocotools with fixes for Python3 and Windows (the official repo doesn't seem to be active anymore).

• Linux: https://github.com/waleedka/coco
• Windows: https://github.com/philferriere/cocoapi. You must have the Visual C++ 2015 build tools on your path (see the repo for additional details)

You can follow Mask_RCNN_Dependencies Installation to install every dependency required for this project.

• visualize_cv2.ipynb Is the easiest way to start.It's an example of using a model pre-trained on MS COCO to segment objects in your own video(You should change the working directory of input image in code.For instance,a line in code capture = cv2.VideoCapture('working_directory\video name.file type') is desired.You should also change directory in every code in this repo)

• demo+BlurredImage.ipynb It shows an example of using a model pre-trained on MS COCO to segment objects in your own images first and then it blur those sensitive images to provide privacy. It includes code to run object detection and instance segmentation on arbitrary images.

• (model.py, utils.py, config.py,coco.py): These files contain the main Mask RCNN implementation.(Must keep these in the directory of main code)

• Privacy_Protected.ipynb The main source code of our repository.It is the outcome of some modifications on Mask_RCNN.After instance segmentation we are able to detect sensitive objects in video and blur for privacy.Care should be taken as privacy is the main concern.You can observe the level of privacy protection whether it is noticable,distracting,disturbing or not.2 videos are provided to check its performance.We use these video as dataset as it was provided by IEEESPS for VIP Cup 2019.

• Privacy_Protected_Save.ipynb This is the source code to make this task portable.A privacy protected MP4 video can be generated by it from a specific input folder containing multiple videos to a output folder.

Python 3.4, TensorFlow 1.3, Keras 2.0.8,opencv and other common packages listed in requirements.txt.

Tons of code was downloaded from theirs repo

• https://github.com/matterport/Mask_RCNN
• https://github.com/michaelnation26/sunglasses_of_invisibility.git

If you find our approach and code useful for your research (such as privacy aware action recognition and many more applications), please consider giving us citation :

@article{ghosh2020privacy,
  title={Privacy-Aware Activity Classification from First Person Office Videos},
  author={Ghosh, Partho and Istiak, Md and Rashid, Nayeeb and Akash, Ahsan Habib and Abrar, Ridwan and Dastider, Ankan Ghosh and Sushmit, Asif Shahriyar and Hasan, Taufiq and others},
  journal={arXiv preprint arXiv:2006.06246},
  year={2020}
}