sungbeomchoi / blurfacedetection Goto Github PK

This project was carried out by YAI 10th, in cooperation with Alchera

  👑 KIM MINSU, YAI 7th
  🚀 KIM HYUNJIN, YAI 8th
  🎓 PARK JUNYOUNG, YAI 9th
  🌵 LEE SUMIN, YAI 9th
  🐯 JIN HYUNBIN, YAI 9th
  😀 CHOI SUNGBEOM, YAI 9th

conda create -n "environment name" python=3.7
conda activate "environment name"

pip install -U Cython cmake numpy
pip install onnxruntime-gpu
pip install -U insight-face

pip install torch>=1.8.1 
pip install torchvision>=0.9.1
pip install pytorch-lightning
pip install numpy
pip install scipy
pip install opencv-python
conda install scikit-image
pip install tqdm

git clone https://github.com/minsu1206/BlurFaceDetection.git

You can just clone this repo into your own computer

And finally the directory hierarchy is configured as,

FaceBlurring
├── config
│      ├── resnet18_regression.yaml
│      └── .....
├── data
├── data_samples
├── dataset
│      ├── blur.py
│      ├── create_blurring.py
│      ├── dataset.py
│      ├── utils.py
│      └── .....
├── experiments
│      ├── results
│      ├── sample_code
│      └── .....
├── legacy
├── models
│      ├── utils # dir for yolov5n.py 
│      ├── edgenext.py
│      ├── mobilenetv2.py
│      └── .....
├── loss.py
├── model_factory.py
├── recorder.py
├── sample.sh
├── test.py
├── train.py
└── utils.py

• FFHQ

```
cd /data
wget https://raw.githubusercontent.com/NVlabs/ffhq-dataset/master/download_ffhq.py
python ./download_ffhq.py --images
cd ../
```

• Our processed data (resolution : 112px)

  You can download the blurred images we created from the link below.

  • https://drive.google.com/drive/folders/1zSfqyeqSlpENTpi6BRcuV6hW9VIkOZsR?usp=sharing

I made two methods to create blur images

• DeblurGAN
  • paper : https://openaccess.thecvf.com/content_cvpr_2018/html/Kupyn_DeblurGAN_Blind_Motion_CVPR_2018_paper.html
  • github : https://github.com/KupynOrest/DeblurGAN
• Defocus and Motion Blur Detection with Deep Contextual Features
  • paper : https://onlinelibrary.wiley.com/doi/full/10.1111/cgf.13567
  • github : https://github.com/Imalne/Defocus-and-Motion-Blur-Detection-with-Deep-Contextual-Features

You have two options to create blur images. The first option is to apply blur iteratively to an clean image. Second option is to apply blur method only once. As blur label, we use 1-cosine similarity.

I show an example command to create blurred images and save them with label information.

cd ./dataset
python create_blurimg_iterative.py --path ../data/FFHQ_1024/clean --n 4
python create_blur_label.py --path ../data/FFHQ_1024/clean 

Above command would generate set of blurred images which were applied blur method four times iterative.

cd ./dataset
python create_blurimg_iterative.py --path ../data/FFHQ_1024/clean --n 4
python create_blur_label.py --path ../data/FFHQ_1024/clean --wo --multi

Above command would generate set of blurred images using multiprocess to generate faster.

cd ./dataset
python create_blur_image.py --blur defocus --iter 1

It’s how to generate blurred images with Defocus method. One blur image is generated for one clean image.

cd ./dataset
python create_blur_image.py --blur deblurgan --iter 1

This command would use DeblurGAN blur method to generate blur images.

cd ./dataset
python create_blur_image.py --blur defocus --iter 1 --scrfd True

This command would generate blur images using defocus blur method and SCRFD inference. SCRFD module is used to detect face in an image.

All generated blur images are stored in the “data” folder.

data
├── FFHQ_1024
│      ├── blur_deblurGAN
│      │      ├── 00000
│      │      │      ├── 00000.png
│      │      │      ├── 00001.png
│      │      │      ├── .....
│      │      │      └── 00999.png
│      │      ├── 01000
│      │      │      ├── 01000.png
│      │      │      ├── 01001.png
│      │      │      ├── .....
│      │      │      └── 01999.png
│      │      └── .....
│      ├── blur_defocus
│      │      ├── 00000
│      │      │      ├── 00000.png
│      │      │      ├── .....
│      │      │      └── 00999.png
│      │      └── .....
│      └── blur_Random
│             ├── 00000
│             │      ├── 00000.png
│             │      ├── .....
│             │      └── 00999.png
│             └── .....
├── label_deblurGAN
│      └── label
│             └── data_label.csv
├── label_defocus
│      └── label
│             └── data_label.csv
├──label_random
│      └── label
│             └── data_label.csv
└──label_val.csv

The following code is used to plot the distribution of the generated blur images.(Below is an example using the deblurgan method)

python data_distribution.py --path ../data/label_deblurGAN/label/data_label.csv

The distribution of the data we provided is as follows. (The x-axis is the blur label, and the y-axis is the number of images. The graph is sequentially using DeblurGAN method, Defocus method, and both methods.)

About 210,000 image samples were generated with random kernel-based methods according to the DeblurGAN and Defocus methods. And we extracted 100,000 samples among them, so that the overall dataset samples were evenly distributed. Training and validation dataset were matched through random split applied with the same random seed in each experiment. The training/validation dataset distribution is as follows.

Basically, we provide a models such as resnet, and also provide light weight backbones which show a fast interference speed.

If you want to train the code, please refer to the training script below.

> python train.py --config config/{}.yaml --save {} --device {} --viz

optional arguments:
	--config								select yaml file to run (in config folder)
	--save									set a path to save checkpoint and graph
	--device								select a device (ex cuda:@)
	--viz										add if you want to visualize

EX)
> python train.py --config mobilenetv2_0.5_regression --save checkpoint/mobilenetv2_0.5 --device cuda:0 --viz

This figure shows that our designed model predicts motion blur well and their error is close to zero when compared to GT whether the blur angle is fixed or not. (Also whatever the backbone is!) Each model’s result is the mean of result about 30 people.

This figure shows that ResNet with simple structure predicts better than one with complex structure. Furthermore, the stack of linear layers increases the inference speed and model size. Therefore, we don’t fix any regressor (fc layer) of all the models we used at this project.

(Upper) : ResNet trained by classfication
(Bottom) : EdgeNext_xx_samll trained by classification

We divide 0 ~ 1 into $N(20, 40)$ classes. $i^{th}$ Class (i=0~N-1) means GT blur degree is between i/N ~ (i+1)/N, so regression label can be changed into classification label.

We train ResNet and EdgeNext_xx_small with cross entropy + MSE(CMSE) or crossentropy + probability based MSE (WeightMSE, WMSE). These figures show that solving this task as classification is also valid approach.

You can detect the blur of the video or image with trained model. First Download a video/image file to the "data" folder path that you want to detect a blur.

Below command detects a blur in the video and generates a result video.

python demo.py --device cpu --pretrained_path {pretrained_model.pt} --mode video --file_path ./data/sample.mp4 --save_path ./data/result_sample.mp4

Below command detects a blur in the image and generates a result image.

python demo.py --device cpu --pretrained_path {pretrained_model.pt} --mode image --file_path ./data/sample.png --save_path ./data/result_sample.png

Be careful you have to match following image specification(for facial cropped image) in test/inference with pre-trained model.

• model.eval() and torch.no_grad()
• Image spatial dimension should be $112 \times 112$
• Image value should be normalized to $0 \sim 1$, not $0 \sim 255$