This repo hosts the official implementary of:
Xingqian Xu, Shant Navasardyan, Vahram Tadevosyan, Andranik Sargsyan, Yadong Mu and Humphrey Shi, Image Completion with Heterogeneously Filtered Spectral Hints, Paper arXiv Link.
- [2022.11.12]: Evaluation code and pretrained model released.
- [2022.11.07]: Our paper is accepted in WACV23.
- [2022.11.06]: Repo initiated.
Spectral Hint GAN (SH-GAN) is an high-performing inpainting network enpowered by CoModGAN and novel spectral processing techniques. SH-GAN reaches state-of-the-art on FFHQ and Places2 with freeform masks.
The overall structure of our SH-GAN shows in the following figure:
The sturcture of our Spectral Hint Unit shows in the following graph:
Heterogeneous Filtering Explaination:
- 1x1 Convolution in Fourier domain leads a uniform (homogeneous) transform from one spectral space to another.
- ReLU in Fourier domain is like a value-dependend band pass filter that zero out some frequency values.
- We promote the heterogeneous transforms in spectral space, in which the frequency value transformations are depended on the frequency bands.
Gaussian Split Algorithm Explaination:
- Gaussian Split is a spectral space downsampling method that well-suit deep learning structures. A quick intuition is that it likes Wavelet Transform that can pass information in different frequency band to its corresponding resolution.
We use FFHQ and Places2 as our main dataset. Download these dataset from the following official link: FFHQ, Places2
Directory of FFHQ data for our code:
├── data
│ └── ffhq
│ └── ffhq256x256.zip
│ └── ffhq512x512.zip
Directory of Places2 data for our code:
- Download the data_challenge.zip from Places2 official website and decompress it to /data/Places2
- Same for val_large.zip
├── data
│ └── Places2
│ └── data_challenge
│ ...
│ └── val_large
│ ...
conda create -n shgan python=3.8
conda activate shgan
conda install pytorch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0 cudatoolkit=11.1 -c pytorch -c conda-forge
pip install -r requirement.txt
DIM | DATA | FID | LPIPS | PSNR | SSIM | Download | |
---|---|---|---|---|---|---|---|
CoModGAN | 256 | FFHQ | 4.7755 | 0.2568 | 16.24 | 0.5913 | |
SH-GAN | 256 | FFHQ | 4.3459 | 0.2542 | 16.37 | 0.5911 | link |
CoModGAN | 512 | FFHQ | 3.6996 | 0.2469 | 18.46 | 0.6956 | |
SH-GAN | 512 | FFHQ | 3.4134 | 0.2447 | 18.43 | 0.6936 | link |
CoModGAN | 256 | Places2 | 9.3621 | 0.3990 | 14.50 | 0.4923 | |
SH-GAN | 256 | Places2 | 7.5036 | 0.3940 | 14.58 | 0.4958 | link |
CoModGAN | 512 | Places2 | 7.9735 | 0.3420 | 16.00 | 0.5953 | |
SH-GAN | 512 | Places2 | 7.0277 | 0.3386 | 16.03 | 0.5973 | link |
Here are the one-line shell commends to evaluation SH-GAN on FFHQ 256/512 and Places2 256/512.
python main.py --experiment shgan_ffhq256_eval --gpu 0 1 2 3 4 5 6 7 --eval 99999
python main.py --experiment shgan_ffhq512_eval --gpu 0 1 2 3 4 5 6 7 --eval 99999
python main.py --experiment shgan_places256_eval --gpu 0 1 2 3 4 5 6 7 --eval 99999
python main.py --experiment shgan_places512_eval --gpu 0 1 2 3 4 5 6 7 --eval 99999
Also you need to:
- Download the data, put them as the directories mentioned in Data session.
- Create
./pretrained
and move all downloaded pretrained models in it. - Create
./log/shgan_ffhq/99999_eval
and./log/shgan_places2/99999_eval
Some simple things to do to resolve the issues:
- The evaluation code caches and later relys on
.cache/****_real_feat.npy
for FID calculation. If it corrupts, numbers will be wrong. But you can simple remove it and the code will auto recompute one. - The final stage of FID computation requires CPU resource so it is normal to be slow, so be patient.
coming soon
@inproceedings{xu2022image,
title={Image Completion with Heterogeneously Filtered Spectral Hints},
author={Xingqian Xu, Shant Navasardyan, Vahram Tadevosyan, Andranik Sargsyan, Yadong Mu, and Humphrey Shi},
booktitle={WACV},
year={2023}
}
Part of the codes reorganizes/reimplements code from the following repositories: Comodgan official Github and Stylegan2-ADA official Github.