Code repo for the paper "Spatio-Temporal Filter Adaptive Network for Video Deblurring".  [Paper]   [Project Page] 

The proposed filter adaptive convolutional (FAC) layer applies generated pixel variant filters to the features, In theory, the element-wise adaptive filters is five-dimensional . In practice, the dimension of the generated filter is and we reshape it into the five-dimensional filter. For each position of input feature, a specific local filter (reshape from ) is applied to the region centered around .

➤ The forward pass of the proposed Filter Adaptive Convolutional (FAC) Layer is as follows:

➤ The backward pass can be presented as:

The frame alignment and deblurring are both spatially variant tasks. Using the proposed FAC layer, we consider these two processes as two filter adaptive convolution in feature domain. The convolution operation can transform the pixels of features, which can be used for frames alignment (a) and deblurring (b) using estimated corresponding filters.

You could download the pretrained model (21.5MB) of STFAN from here.

(Note that the model does not need to unzip, just load it directly.)

• Linux (tested on Ubuntu 14.04/16.04)
• CUDA 8.0/9.0/10.0
• gcc 4.9+
• Python 2.7+
• PyTorch 1.0+
• easydict
• tensorboardX Note that if your CUDA is 10.2+, you need to modify KernelConv2D_cuda.cpp:

1. Uncomment: #include <c10/cuda/CUDAStream.h>
2. Modify: at::cuda::getCurrentCUDAStream() -> c10::cuda::getCurrentCUDAStream()

pip install -r requirements.txt
bash install.sh

Download DeepVideoDeblurring dataset from this link, and modify the data index tree as following:

├── [DATASET_ROOT]
│   ├── train
│   │   ├── input
│   │   ├── GT
│   ├── test
│   │   ├── input
│   │   ├── GT

Use the following command to train the neural network:

python runner.py 
        --phase 'train'\
        --data [dataset path]\
        --out [output path]

Use the following command to test the neural network:

python runner.py \
        --phase 'test'\
        --weights './ckpt/best-ckpt.pth.tar'\
        --data [dataset path]\
        --out [output path]

Use the following command to resume training the neural network:

python runner.py 
        --phase 'resume'\
        --weights './ckpt/best-ckpt.pth.tar'\
        --data [dataset path]\
        --out [output path]

You can also use the following simple command, with changing the settings in config.py:

python runner.py

Some results are shown in [Project Page].

If you find STFANet, or FAC layer useful in your research, please consider citing:

@inproceedings{zhou2019stfan,
  title={Spatio-Temporal Filter Adaptive Network for Video Deblurring},
  author={Zhou, Shangchen and Zhang, Jiawei and Pan, Jinshan and Xie, Haozhe and  Zuo, Wangmeng and Ren, Jimmy},
  booktitle={Proceedings of the IEEE International Conference on Computer Vision},
  year={2019}
}

We are glad to hear if you have any suggestions and questions.

Please send email to [email protected]

This project is open sourced under MIT license.