[Paper Link (IEEE Access)] [Paper Link (EI 2020)]

In public spaces such as zoos and sports facilities, the presence of fences often annoys tourists and professional photographers. There is a demand for a post-processing tool to produce a non-occluded view from an image or video. This “de-fencing” task is divided into two stages: one to detect fence regions and the other to fill the missing part. For over a decade, various methods have been proposed for video-based de-fencing. However, only a few single-image-based methods are proposed. In this paper, we focus on single-image fence removal. Conventional approaches suffer from inaccurate and non-robust fence detection and inpainting due to less content information. To solve these problems, we combine novel methods based on a deep convolutional neural network (CNN) and classical domain knowledge in image processing. In the training process, we are required to obtain both fence images and corresponding non-fence ground truth images. Therefore, we synthesize natural fence images from real images. Moreover, spacial filtering processing (e.g. a Laplacian filter and a Gaussian filter) improves the performance of the CNN for detection and inpainting. Our proposed method can automatically detect a fence and generate a clean image without any user input. Experimental results demonstrate that our method is effective for a broad range of fence images.

Please cite this paper if you use this code.

@ARTICLE{8933392, 
author={T. {Matsui} and M. {Ikehara}}, 
journal={IEEE Access}, 
title={Single-Image Fence Removal Using Deep Convolutional Neural Network}, 
year={2020}, 
volume={8}, 
number={}, 
pages={38846-38854},}

Run demo_defence.m

• MATLAB R2019a
• caffe version 1.0.0-rc3
• CPU or NVIDIA GPU + CUDA CuDNN

To train U-Net, we collect 545 real-world fence images and binary masks created by Du et al.. From these images, we cropped 128 × 128 × 3 patches. In order to increase the amount of data for training improvement, the cropped patches are randomly flipped, rotated, zoomed and brightened.

Fence images are created by combining fence masks with the clean outdoor images from UCID dataset and from the BSD dataset in the directry /dataset/train/. If you would like to create a new pair of a fence image and a fence mask, please run as following:

[fence_image, fence_mask] = add_fence(img, theta, scale, color_num, noise, real).

takuro-matsui

If you have any questions, please feel free to send us an e-mail [email protected].