This repository provides the tensorflow implementation of the Video Super-resolution model FANI for mobile devices.

• Feature Aggregating Network with Inter-Frame Interaction for Efficient Video Super-Resolution
  • Overview
    • Contents
    • Requirements
    • Dataset preparation
    • Training and Validation
      • Configuration
      • Training
      • Validation
    • Testing
    • Convert to tflite
    • TFLite inference on Mobile
    • License
    • Citation

• Python: 3.8.10
• Python packages: numpy, imageio and pyyaml
• TensorFlow = 2.11.0 + CUDA cuDNN
• GPU for training (e.g., Nvidia GeForce RTX 3090)

[back]

• Download REDS dataset and extract it into data folder.

  _{The REDS dataset folder should contain three subfolders: train/, val/ and test/. Please find the download links to above files in website.}

[back]

Before training and testing, please make sure the fields in config.yml is properly set.

log_dir: snapshot -> The directory which records logs and checkpoints. 

dataset:
    dataloader_settings: -> The setting of different splits dataloader.
        train:
            batch_size: 4
            drop_remainder: True
            shuffle: True
            num_parallel_calls: 6
        val:
            batch_size: 1
    data_dir: data/ -> The directory of REDS dataset.
    degradation: sharp_bicubic -> The degradation of images.
    train_frame_num: 10 -> The number of image frame(s) for per training step.
    test_frame_num: 100 -> The number of image frame(s) for per testing step.
    crop_size: 64 -> The height and width of cropped patch.

model:
    path: model/mobile_rrn.py -> The path of model file.
    name: MobileRRN -> The name of model class.

learner:
    general:
        total_steps: 1500000 -> The number of training steps.
        log_train_info_steps: 100 -> The frequency of logging training info.
        keep_ckpt_steps: 10000 -> The frequency of saving checkpoint.
        valid_steps: 100000 -> The frequency of validation.

    optimizer: -> Define the module name and setting of optimizer
        name: Adam
        beta_1: 0.9
        beta_2: 0.999

    lr_scheduler: -> Define the module name and setting of learning rate scheduler
        name: ExponentialDecay
        initial_learning_rate: 0.0001
        decay_steps: 1000000
        decay_rate: 0.1
        staircase: True

    saver:
        restore_ckpt: null -> The path to checkpoint where would be restored from.

To train the model, use the following command:

sh train.sh

The main arguments are as follows:

process :   Process type should be train or test.
config_path :   Path of yml config file of the application.

After training, the checkpoints will be produced in log_dir.

To valid the model, use the following command:

sh valid.sh

After validating, the output images will be produced in log_dir/output.

[back]

To generate testing outputs, use the following command:

sh test.sh

The main arguments are as follows:

model_path :   Path of model file.
model_name :   Name of model class.
ckpt_path :   Path of checkpoint.
data_dir :   Directory of testing frames in REDS dataset.
output_dir :   Directory for saving output super-resolution images.

[back]

To convert the keras model to tflite, use the following command:

sh convert_to_tflite.sh

The main arguments are as follows:

model_path :   Path of model file.
model_name :   Name of model class.
input_shape :   Series of the input shapes split by `:`.
ckpt_path :   Path of checkpoint.
output_tflite :   Path of output tflite.

[back]

• AI benchmark: An app allowing you to load your model and run it locally on your own Android devices with various acceleration options (e.g. CPU, GPU, APU, etc.).

[back]

FANI is released under the MIT license.

[back]

@inproceedings{DBLP:conf/icdm/LiZZFZX23,
  author       = {Yawei Li and
                  Zhao Zhang and
                  Suiyi Zhao and
                  Jicong Fan and
                  Haijun Zhang and
                  Mingliang Xu},
  editor       = {Guihai Chen and
                  Latifur Khan and
                  Xiaofeng Gao and
                  Meikang Qiu and
                  Witold Pedrycz and
                  Xindong Wu},
  title        = {Feature Aggregating Network with Inter-Frame Interaction for Efficient
                  Video Super-Resolution},
  booktitle    = {{IEEE} International Conference on Data Mining, {ICDM} 2023, Shanghai,
                  China, December 1-4, 2023},
  pages        = {329--338},
  publisher    = {{IEEE}},
  year         = {2023},
  url          = {https://doi.org/10.1109/ICDM58522.2023.00042},
  doi          = {10.1109/ICDM58522.2023.00042},
  timestamp    = {Tue, 13 Feb 2024 13:16:49 +0100},
  biburl       = {https://dblp.org/rec/conf/icdm/LiZZFZX23.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}

[back]