holmes-alan / dsrvae Goto Github PK

View Code? Open in Web Editor NEW

118.0 10.0 14.0 14.26 MB

Unsupervised Real Image Super-Resolution via Variational AutoEncoder in CVPR2020

Python 100.00%

dsrvae's Introduction

dSRVAE (Generative Variational AutoEncoder for Real Image Super-Resolution)

By Zhi-Song Liu, Li-Wen Wang, Chu-Tak Li, Marie-Paule Cani and Wan-Chi Siu

This repo only provides simple testing codes, pretrained models and the network strategy demo.

We propose a joint image denoising and Super-Resolution model by using generative Variational AutoEncoder (dSRVAE)

We participate CVPRW NTIRE2020 Real Image Super-Resolution Challenge

Please check our paper

BibTex

    @InProceedings{Liu2020dsrvae,
        author = {Zhi-Song Liu, Wan-Chi Siu and Li-Wen Wang and Chu-Tak Li and Marie-Paule Cani and Yui-Lam Chan},
        title = {Unsupervised Real Image Super-Resolution via Generative Variational AutoEncoder},
        booktitle = {IEEE International Conference on Computer Vision and Pattern Recognition Workshop(CVPRW)},
        month = {June},
        year = {2020}
    }

For proposed dSRVAE model, we claim the following points:

• First working on using Variational AutoEncoder for image denoising.

• Then the Super-Resolution Sub-Network (SRSN) is attached as a small overhead to the DAE which forms the proposed dSRVAE to output super-resolved images.

Dependencies

Python > 3.0
OpenCV library
Pytorch > 1.0
NVIDIA GPU + CUDA
pytorch-gan-metrics

Complete Architecture

The complete architecture is shown as follows,

Implementation

1. Quick testing

Download pre-trained models from

https://drive.google.com/open?id=1SUZGE04vw5_yDYiw6PJ4sbHAOIEV6TJ7

and copy them to the folder "models"

Copy your image to folder "Test" and run

$ python test.py

The SR images will be in folder "Result" 3. For self-ensemble, run

$ python test_enhance.py

GAN feature evaluation

# download statistics.npz from http://bioinf.jku.at/research/ttur/
from pytorch_gan_metrics import get_inception_score, get_fid

images = ... # [N, 3, H, W] normalized to [0, 1]
IS, IS_std = get_inception_score(images)        # Inception Score
FID = get_fid(images, 'path/to/statistics.npz') # Frechet Inception Distance

2. Testing for NTIRE 20202

s1. Testing images on NTIRE2020 Real World Super-Resolution Challenge - Track 1: Image Processing artifacts can be downloaded from the following link:

https://drive.google.com/open?id=10ZutE-0idGFW0KUyfZ5-2aVSiA-1qUCV

s2. Testing images on NTIRE2020 Real World Super-Resolution Challenge - Track 2: Smartphone Images can be downloaded from the following link:

https://drive.google.com/open?id=1_R4kRO_029g-HNAzPobo4-xwp86bMZLW

s3. Validation images on NTIRE2020 Real World Super-Resolution Challenge - Track 1 and Track 2 can be downloaded from the following link:

https://drive.google.com/open?id=1nKEJ4N2V-0NFicfJxm8AJqsjXoGMYjMp

3. Training

s1. Download the training images from NTIRE2020.

https://competitions.codalab.org/competitions/22220#learn_the_details

s2. Start training on Pytorch

Train the Denoising VAE by running

$ python main_denoiser.py

Train the super-resolution SRSN overhead by running

$ python main_GAN.py

Partial image visual comparison

1. Visualization comparison

Results on 4x image SR on Track 1 dataset

Reference

You may check our newly work on General image super-resolution using VAE

You may also check our work on Reference based face SR using VAE

You may also check our work on Reference based General image SR using VAE

Special thanks to the contributions of Jakub M. Tomczak for their VAE with a VampPrior on KL loss calculation.

dsrvae's People

Contributors

Stargazers

Watchers

Forkers

super-resolution-list mgrankin tom666tom666 killsking peterzs liuguoyou ikasumi tamwaiban ashbt cv-ip mariomosko muchway2019 zhangjiewen123 katymq

dsrvae's Issues

Kl loss is different from original code?

i haved scaned original code about kl loss，it is different from this，I'm confused，thanks.

Newbie Concerns

Hello!

I'm trying to replicate your results but I had a little issues/concerns about the published code.

1.- I downloaded the dataset called "Corrupted-tr-x" for the code called main_denoiser.py, in line 108 the run search for 2 folders stored on /data/NTIRE2020/ for HR and SR, the first concern is: the images on zip file should be the HR so I need to process all the folder to get the SR ('bicubic';1/4*scale)?
2. In test_enhace is called from prepare_images import * is the file needed or is from a library?

In advance thanks for your time and for share the code.

denoising and some questions

There is no change in the processing of the image with the denoising model
in main_GAN, it is not HR_feat_extractor, Lap_loss and train_pretrained function

i am facing this error when i try to run test.py file. would you please give me any suggestions

Traceback (most recent call last):
File "test.py", line 58, in
denoiser = torch.nn.DataParallel(denoiser, device_ids=gpus_list)
File "/home/ali/anaconda3/envs/torch/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 136, in init
_check_balance(self.device_ids)
File "/home/ali/anaconda3/envs/torch/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 19, in _check_balance
dev_props = [torch.cuda.get_device_properties(i) for i in device_ids]
File "/home/ali/anaconda3/envs/torch/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 19, in
dev_props = [torch.cuda.get_device_properties(i) for i in device_ids]
File "/home/ali/anaconda3/envs/torch/lib/python3.7/site-packages/torch/cuda/init.py", line 317, in get_device_properties
raise AssertionError("Invalid device id")
AssertionError: Invalid device id

TypeError: TypeError: eval() takes 0 positional arguments but 2 were given

Hello i ws wondering if you had any insight about the error that i get when running the Test.py code

Traceback (most recent call last):
File "C:\Program Files\PyScripter\Lib\rpyc.zip\rpyc\core\protocol.py", line 329, in _dispatch_request
res = self._HANDLERS[handler](self, *args)
File "C:\Program Files\PyScripter\Lib\rpyc.zip\rpyc\core\protocol.py", line 590, in _handle_call
return obj(*args, **dict(kwargs))
File "", line 433, in evalcode
TypeError: eval() takes 0 positional arguments but 2 were given

if you have any idea or suggestions
Please let me know
thank you in advance

self.idle_input is not defined in modules.py

Unsupervised training

Hello! Could you please clarified why do we have clean and noise images during the train process? It is unsupervised training, so we must have only one type of image, isn't it?

loss used for generator in main_GAN.py seems confusing

In main_GAN.py:
line 85 :down_ref = down(ref) #down_ref is not used latter,ref is used to refer
line 92 :SR_tar = G(target) #target is loaded from HR files , so G(target) looks incorrect
Therefore i think codes should be corrected as :
line 85 :down_target = down(target)
line 92 :SR_tar = G(down_target )
did i get wrong with something? Please tell me ,thank you!
And i want to ask : is the code main_GAN.py a survised version?

list index out of range

During launch of python main_denoiser.py

Traceback (most recent call last):
  File "main_denoiser.py", line 142, in <module>
    train(epoch)
  File "main_denoiser.py", line 52, in train
    for iteration, batch in enumerate(training_data_loader, 1):
  File "/home/u/.anaconda/envs/fast_tabnet/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/home/u/.anaconda/envs/fast_tabnet/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 856, in _next_data
    return self._process_data(data)
  File "/home/u/.anaconda/envs/fast_tabnet/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 881, in _process_data
    data.reraise()
  File "/home/u/.anaconda/envs/fast_tabnet/lib/python3.7/site-packages/torch/_utils.py", line 394, in reraise
    raise self.exc_type(msg)
IndexError: Caught IndexError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/u/.anaconda/envs/fast_tabnet/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/u/.anaconda/envs/fast_tabnet/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/u/.anaconda/envs/fast_tabnet/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/u/dSRVAE/datasets.py", line 105, in __getitem__
    ref = load_img(self.hr_image_filenames[2649-index])
IndexError: list index out of range

Request for testing

Hi,
this is a really good method. So, I try to run your code on my machine, but I fail to complete it. I want to reference your method in my paper, and want to get the results from your method. So could you help me test my image? I mean I can send you my testing image, and you can run it in your machine, and then you can send me the result back.

In testing, did you use only noisy image as input or clean image also used as input for denoise the image? and in testing u mentioned u don't use encoder for denoising image only used decoder to denoise the image. Why? Please let know. Thank u in advance..

NameError: name 'LR_filename' is not defined

Running test.py gives the following error:

Namespace(chop_forward=True, gpu_mode=True, gpus=1, input='Test', model_SR='models/VAE_SR.pth', model_denoiser='models/VAE_denoiser.pth', model_type='VAE', output='Result', patch_size=128, seed=123, stride=8, testBatchSize=64, threads=6, upscale_factor=4)
===> Building model  VAE
===> Loading datasets
Pre-trained Denoiser model is loaded.
Pre-trained SR model is loaded.
Traceback (most recent call last):
  File "test.py", line 151, in <module>
    eval()
  File "test.py", line 86, in eval
    SR_image = [join(opt.Result, x) for x in listdir(LR_filename) if is_image_file(x)]
NameError: name 'LR_filename' is not defined

Not sure what LR_filename is since it's not defined anywhere else in the code. How can I get your test to run?

prepare_images

A file named prepare_images is called in test_enhance.py. But I couldn't able to find the file.

TypeError: forward() missing 1 required positional argument: 'down_ref'

Traceback (most recent call last):
File "main_denoiser.py", line 142, in
train(epoch)
File "main_denoiser.py", line 63, in train
Denoise_LR, KL = model(HR_feat, input)
File "D:\Anaconda3\envs\SR\lib\site-packages\torch\nn\modules\module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "D:\Anaconda3\envs\SR\lib\site-packages\torch\nn\parallel\data_parallel.py", line 150, in forward
return self.module(*inputs[0], **kwargs[0])
File "D:\Anaconda3\envs\SR\lib\site-packages\torch\nn\modules\module.py", line 532, in call
result = self.forward(*input, **kwargs)
TypeError: forward() missing 1 required positional argument: 'down_ref

I don't know how to fix the problem,please help