This is the implementation of paper "Xue, S., et al., Faster Image Super-Resolution by Improved Frequency Domain Neural Networks. Signal, Image and Video Processing, submitted, 2019."

• OS: CentOS 7 Linux kernel 3.10.0-514.el7.x86_64
• CPU: Intel Xeon(R) CPU E5-2667 v4 @ 3.20GHz x 32
• Memory: 251.4 GB
• GPU: NVIDIA Tesla P4, 8 GB

• Cuda 8.0 (Cudnn installed)
• Caffe (pycaffe and matcaffe interface required)
• Python 2.7.5
• Matlab 2017b

These datasets are the same as other paper provided. Readers can directly use them or download them from here:

BSDS100, BSDS200, General-100, Set5, Set14, T91, Train_291, Urban100, and DIV2K.

1. Copy the 'train' directory to '.../Caffe_ROOT/examples/', and rename the directory to 'IFNNSR'.
2. Prepare datasets into your own directory.
3. (optional) run 'data_aug.m' in Matlab for data augmentation; e.g., data_aug('data/BSDS200'), which will generates a new directory 'BSDS-200-aug'.
4. Run 'generate_train_data.m' and 'generate_test_data.m' in Matlab to generate 'train_IFNNSR_x[2,3,4].h5' and 'test_IFNNSR_x[2,3,4].h5'. (modify the data path in *.m files)
5. (optional) Modify the parameters in 'create_IFNNSR_x[2,3,4].py'.
6. Run in command line: 'python create_IFNNSR_[2,3,4].py'. It will regenerate 'train_x[2,3,4].prototxt' and 'test_x[2,3,4].prototxt'.
7. (optional) modify the parametes in 'solver_x[2,3,4].prototxt'.
8. Run in command line './examples/IFNNSR/run_train_x[2,3,4].sh' at 'Caffe_ROOT' path.
9. Waiting for the training procedure completed.

• net: "examples/IFNNSR/train_x2.prototxt"
• test_iter: 1000
• test_interval: 100
• base_lr: 1e-4
• lr_policy: "step"
• gamma: 0.5
• stepsize: 10000
• momentum: 0.9
• weight_decay: 1e-04
• display: 100
• max_iter: 100000
• snapshot: 5000
• snapshot_prefix: 'examples/IFNNSR/model/x2/x2_d_c_k_'
• solver_mode: GPU
• type: "SGD"

1. Prepare datasets into 'data' directory.
2. Copy 'test_x[2,3,4].prototxt' from training directory to 'test' directory.
3. Copy '*.caffemodel' from training directory to 'test/model' directory.
4. Modify some paths in 'test_IFNNSR_main.m' if necessary.
5. Run 'test_IFNNSR_main.m' in Matlab.
6. Metrics will be printed and reconstrcuted images will be saved into 'result' directory.

Readers can use 'Netscope' to visualize the network architecture

We define a new layer named "ElementWiseProduct" in Caffe, which implements the elementwise product opreation to data with a weight matrix:

input: $X \in \mathbb{R}^{n \times c \times h \times w}$, weight: $W \in \mathbb{R}^{1 \times 1 \times h \times w}$, and output $Y \in \mathbb{R}^{n \times c \times h \times w}$. $$ Y_{ij} = W \circ X_{ij}, \ i = 1,2,\ldots,n, \ j = 1, 2, \ldots, c,$$ where $\circ$ is the Hadamard product (i.e., elementwise product) and $W$ contains parameters that can be learned during training. Caffe files:

• /caffe/include/caffe/layers/elementwise_product_layer.hpp
• /caffe/src/caffe/layers/elementwise_product_layer.cpp
• /caffe/src/caffe/layers/elementwise_product_layer.cu

Place the three files above to corresponding positions of your own caffe directory. Do not forget to add a few definitions into '.../caffe/src/caffe/proto/caffe.proto' file. Detailed steps are provided in 'warning do NOT replace.txt' in our directory along with 'caffe.proto'.

We propose the weighted Euclidean loss, which assigns larger values of weights to high-frequency parts and smaller values of weights to low-frequency parts. Precisely, the loss function is defined as \begin{align} {W}{\rm e} &= \begin{bmatrix} w{11} & w_{12} & \cdots & w_{1n} \ w_{21} & w_{22} & \cdots & w_{2n} \ \vdots & \vdots & \ddots & \vdots \ w_{m1} & w_{m2} & \cdots & w_{mn} \end{bmatrix} \in \mathbb{R}^{m \times n}, \ w_{ij} &= \exp\left ( \alpha \left (\frac{|m/2-i|}{m/2} \right )^2 + \beta \left (\frac{|n/2-j|}{n/2} \right )^2 \right ), \nonumber \ i &= 1,2,\ldots,m, \ \ j = 1,2,\ldots,n, \ e &= \frac{1}{2} | {W}{\rm e} \circ (\hat{I}{\rm out} - \hat{I}{\rm HR})|{\rm F}^2, \end{align} where $\alpha$ and $\beta$ are constants, ${W}{\rm e}$ is the weight matrix, $\hat{I}{\rm out}$ is the output of our network, $\hat{I}{\rm HR}$ is the corresponding label, and $|\cdot|{\rm F}$ denotes the Frobenius norm. Caffe files:

• /caffe/include/caffe/layers/weight_l2_loss_layer.hpp
• /caffe/src/caffe/layers/weight_l2_loss_layer.cpp
• /caffe/src/caffe/layers/weight_l2_loss_layer.cu

If you have any suggestion or question, please do not hesitate to contact me.

Ph.D. candidate, Shengke Xue

College of Information Science and Electronic Engineering

Zhejiang University, Hangzhou, P.R. China

Email: [email protected], [email protected]