yangxy / gpen Goto Github PK

Python 22.65% C++ 0.18% Cuda 1.16% Jupyter Notebook 75.88% Shell 0.14%

gpen's Introduction

GAN Prior Embedded Network for Blind Face Restoration in the Wild

Paper | Supplementary | Demo | ModelScope

Tao Yang¹, Peiran Ren¹, Xuansong Xie¹, Lei Zhang^1,2
¹DAMO Academy, Alibaba Group, Hangzhou, China
²Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China

Face Restoration

Selfie Restoration

Face Colorization

Face Inpainting

Conditional Image Synthesis (Seg2Face)

News

(2023-02-15) GPEN-BFR-1024 and GPEN-BFR-2048 are now publicly available. Please download them via [ModelScope2].

(2023-02-15) We provide online demos via [ModelScope1] and [ModelScope2].

(2022-05-16) Add x1 sr model. Add --tile_size to avoid OOM.

(2022-03-15) Add x4 sr model. Try --sr_scale.

(2022-03-09) Add GPEN-BFR-2048 for selfies. I have to take it down due to commercial issues. Sorry about that.

(2021-12-29) Add online demos . Many thanks to CJWBW and AK391.

(2021-12-16) Release a simplified training code of GPEN. It differs from our implementation in the paper, but could achieve comparable performance. We strongly recommend to change the degradation model.

(2021-12-09) Add face parsing to better paste restored faces back.

(2021-12-09) GPEN can run on CPU now by simply discarding --use_cuda.

(2021-12-01) GPEN can now work on a Windows machine without compiling cuda codes. Please check it out. Thanks to Animadversio. Alternatively, you can try GPEN-Windows. Many thanks to Cioscos.

(2021-10-22) GPEN can now work with SR methods. A SR model trained by myself is provided. Replace it with your own model if necessary.

(2021-10-11) The Colab demo for GPEN is available now .

Download models from Modelscope

Install modelscope:

pip install "modelscope[cv]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html

Run the following codes:

import cv2
from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks
from modelscope.outputs import OutputKeys

portrait_enhancement = pipeline(Tasks.image_portrait_enhancement, model='damo/cv_gpen_image-portrait-enhancement-hires')
result = portrait_enhancement('https://modelscope.oss-cn-beijing.aliyuncs.com/test/images/marilyn_monroe_4.jpg')
cv2.imwrite('result.png', result[OutputKeys.OUTPUT_IMG])

It will automatically download the GPEN models. You can find the model in the local path ~/.cache/modelscope/hub/damo. Please note pytorch_model.pt, pytorch_model-2048.pt are respectively the 1024 and 2048 versions.

Usage

Clone this repository:

git clone https://github.com/yangxy/GPEN.git
cd GPEN

Download RetinaFace model and our pre-trained model (not our best model due to commercial issues) and put them into weights/.

RetinaFace-R50 | ParseNet-latest | model_ir_se50 | GPEN-BFR-512 | GPEN-BFR-512-D | GPEN-BFR-256 | GPEN-BFR-256-D | GPEN-Colorization-1024 | GPEN-Inpainting-1024 | GPEN-Seg2face-512 | realesrnet_x1 | realesrnet_x2 | realesrnet_x4
Restore face images:

python demo.py --task FaceEnhancement --model GPEN-BFR-512 --in_size 512 --channel_multiplier 2 --narrow 1 --use_sr --sr_scale 4 --use_cuda --save_face --indir examples/imgs --outdir examples/outs-bfr

Colorize faces:

python demo.py --task FaceColorization --model GPEN-Colorization-1024 --in_size 1024 --use_cuda --indir examples/grays --outdir examples/outs-colorization

Complete faces:

python demo.py --task FaceInpainting --model GPEN-Inpainting-1024 --in_size 1024 --use_cuda --indir examples/ffhq-10 --outdir examples/outs-inpainting

Synthesize faces:

python demo.py --task Segmentation2Face --model GPEN-Seg2face-512 --in_size 512 --use_cuda --indir examples/segs --outdir examples/outs-seg2face

Train GPEN for BFR with 4 GPUs:

CUDA_VISIBLE_DEVICES='0,1,2,3' python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 train_simple.py --size 1024 --channel_multiplier 2 --narrow 1 --ckpt weights --sample results --batch 2 --path your_path_of_croped+aligned_hq_faces (e.g., FFHQ)

When testing your own model, set --key g_ema.

Please check out run.sh for more details.

Main idea

Citation

If our work is useful for your research, please consider citing:

@inproceedings{Yang2021GPEN,
    title={GAN Prior Embedded Network for Blind Face Restoration in the Wild},
    author={Tao Yang, Peiran Ren, Xuansong Xie, and Lei Zhang},
    booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
    year={2021}
}

License

Acknowledgments

We borrow some codes from Pytorch_Retinaface, stylegan2-pytorch, Real-ESRGAN, and GFPGAN.

Contact

If you have any questions or suggestions about this paper, feel free to reach me at [email protected].

gpen's People

Contributors

Stargazers

Watchers

Forkers

lotayou scott-mao mowshon mornydew zhhezhhe kiku-jw cvlinks bycloudai deepvoodoofx helixngc7293 arukaito repo-collection darkknight2223 c00renut jymcheong deepmagic-io mlbcp atomeyang espressoandcode ayanbabi90 mohammedgomaa githubcrj zeratel xiaohaipeng perseverancelx lllgoyour shiluooulihs astoncpou tiansong1991 8secz-johndpope energy888666 xiusdk mjq11302010044 nnnnai templeblock crzaizxw1314 rubo77 boboyiyi kuijiang94 molo32 gessyoo kuchavo cagery c1a1o1 alexsevas zt706 wallzfe sadam1195 kkodoo lillllllll wubangjunjava jmu201521121021 suchaogao h1-ti fyushan bencoster kyocen antoine-cha xuewengeophysics robotpin wcy-cs xijunjun unitingcoders didadidadidad lxzatwowone1 gbriones1 borisplus super-zoe z2labplus gv3d little-purple-fish mickeygit haroonshakeel fodereas shovelingpig kozlovip 950r janfschr deepartist best-love wl0521 lyntai harisreedhar bertwang kropotovoleg mostafaahmedzaki yueyedeai chinasilva dudrill gozarck knightxmare hadryan conansherry lincong666 zhaojia1984 rostyslavbryiovskyi mrbananahuman popmeshgrid criticalpulsar rococostudio

gpen's Issues

ONNX conversion and inference

Nice work buddy. Don't know about your priorities right now but if you can add onnx conversion,inference script for your lightweight model that would give tremendous boost to this repo. Take care.

the effect of pretrained model

Hi yangxy
Thanks for GPEN
What is the comparison between the effect of without/with stylegan2 pretrained model?
no pretrained model vs stylegan2 pretrained model ft

GPEN-BFR-512-D

Hello, I want to test the GPEN-BFR-512-D model, could you tell me how to set parameters?

About batchsize when training

Thanks for sharing. Brilliant work!
I notice that batchsize is set to 1 as mentioned in your paper. Is it by design or just some concession to hardware limitation？Thank you

Excuse me，If I only use the CPU, how do I run the script?

will the train code release?

i have test the model you release, the result is so good. but there is some black points on face sometimes.
will you release the trainning code ?

the problem of the real-world LQ faces

I appreciate your work. Could you release the 1,000 real-world LQ faces as soon as possible?

How can i test 8x face SR which input image is 16x16

Thank you.

应用于视频

这个效果非常好，我在测试的时候发现人脸增强应用于视频，会有人脸临近帧某些位置不一致问题即人脸抖动，请问关于这种现在有什么解决方案吗？谢谢大佬～

subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

Hi there!
When I try to apply the command "python face_enhancement.py" I get the following error.

(gpen) F:\Anaconda3\GPEN-main>python face_enhancement.py
F:\Anaconda3\envs\gpen\lib\site-packages\torch\utils\cpp_extension.py:274: UserWarning: Error checking compiler version for cl: 'utf-8' codec can't decode byte 0xa2 in position 24: invalid start byte
warnings.warn('Error checking compiler version for {}: {}'.format(compiler, error))
Traceback (most recent call last):
File "F:\Anaconda3\envs\gpen\lib\site-packages\torch\utils\cpp_extension.py", line 1522, in _run_ninja_build
env=env)
File "F:\Anaconda3\envs\gpen\lib\subprocess.py", line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "face_enhancement.py", line 13, in
from face_model.face_gan import FaceGAN
File "F:\Anaconda3\GPEN-main\face_model\face_gan.py", line 13, in
from model import FullGenerator
File "F:\Anaconda3\GPEN-main\face_model\model.py", line 15, in
from op import FusedLeakyReLU, fused_leaky_relu, upfirdn2d
File "F:\Anaconda3\GPEN-main\face_model\op_init_.py", line 1, in
from .fused_act import FusedLeakyReLU, fused_leaky_relu
File "F:\Anaconda3\GPEN-main\face_model\op\fused_act.py", line 14, in
os.path.join(module_path, 'fused_bias_act_kernel.cu'),
File "F:\Anaconda3\envs\gpen\lib\site-packages\torch\utils\cpp_extension.py", line 980, in load
keep_intermediates=keep_intermediates)
File "F:\Anaconda3\envs\gpen\lib\site-packages\torch\utils\cpp_extension.py", line 1185, in _jit_compile
with_cuda=with_cuda)
File "F:\Anaconda3\envs\gpen\lib\site-packages\torch\utils\cpp_extension.py", line 1283, in _write_ninja_file_and_build_library
error_prefix="Error building extension '{}'".format(name))
File "F:\Anaconda3\envs\gpen\lib\site-packages\torch\utils\cpp_extension.py", line 1537, in _run_ninja_build
message += ": {}".format(error.output.decode()) # type: ignore
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe1 in position 2472: invalid continuation byte

Any idea why this happens to me? Do I need to install something?

GPEN on local machine? (without google colab)

Is there a way to make GPEN run on local machine using Windows 10 + Anaconda?

If it's not possible using the official version or future update,
Can someone please make a branch with guided tutorial?

After installing all the requirements, trying to run GPEN using:

python face_enhancement.py

I get these errors:

CUDA out of memory error

Hi
I am trying your code on a group photo with approximately 90 faces in it. I am getting the following CUDA error...

RuntimeError: CUDA out of memory. Tried to allocate 6.27 GiB (GPU 0; 8.00 GiB total capacity; 7.82 GiB already allocated; 0 bytes free; 7.83 GiB reserved in total by PyTorch)

Is there a flag that will allow me to run with my RTX 2070 SUPER card (8GB dedicated GPU memory, 8GB shared GPU memory)? I noticed for another question about CUDA, you referred the questioner to https://github.com/rosinality/stylegan2-pytorch.
I looked at that repo and saw for a CUDA memory issue, they recommended to use torch.no_grad. Will that help here, and if so, how should that be done?

Any help would be appreciated!
Thanks!

轻量级的模型更新了吗？

轻量级的模型更新了吗？我一直在关注，期待更新模型，提升速度

Alternative model links?

Are there any alternative download links for the models?
Unfortunately the download speed is very slow for me, and I am sure I have fast internet.

discriminator pretrain model

Do you have the plan to release the pretrained model of discriminator?

how to

is there a way to put a guide of how to use it, for newbie users, the more detailed explain the better, that would be super useful, thank you.

ONNX/TorchScript convertion

Hello!

Wanted to know if anyone managed to convert model with custom layers from stylegan2 to onnx/torchscript serialization format?

About pretrained model of Face Inpainting

Do you have the plan to release the pretrained model of Face Inpainting? Thank you very much!

About dataset

Thank you for releasing your code. I know you "synthesize degraded faces from the HQ images in FFHQ using the degradation model" from the paper. Maybe can you release the code of degradation?

Training code?

when will you release the training code?

Source code

Hi!
Great results! Will you publish the source code?

from op import FusedLeakyReLU, fused_leaky_relu, upfirdn2d

Your compiler (c++) is not compatible with the compiler Pytorch was
built with for this platform, which is g++ on linux. Please
use g++ to to compile your extension. Alternatively, you may
compile PyTorch from source using c++, and then you can also use
c++ to compile your extension.

import err when import ops

Reproduce training

Thanks for your awesome work @yangxy

Questions about training:

How many iterations did you trian the model?
What about the speed of training?

No CUDA runtime is found, using CUDA_HOME

I installed all the libraries in requirements.txt and my CUDA toolkit version is 11.3.1. I'm on a RTX 3090.

Does not run on Windows

Hi all,

When I try to run this on Windows I get the error:

Ninja is required to load C++ extensions

Ninja is already installed. Is it possible to allow this code to run on Windows without Docker?

Thanks!

Learning rate setting

求尽快提供训练代码

how many epoches you set when you train

nice work ,

how many epoches you set when you train, and what GPUs you used and how long time

win10下anaconda创建GPEN环境需要VS那些组件？

请问大佬，win10下用anaconda创建GPEN虚拟环境，这个需要VS哪些组件，下载完整的太大了，我想能不能只下载必要的节省时间和空间。

colorization process

Hi yangxy
Thanks for GPEN, we experiment great results in our tests in most of the cases
Some wrinkles around eyes seems to be identify as glasses, but it's maybe due to pretrained datasets

Could you explain how did you get colorization result in your supplementary pdf ?

[removed]

Real-time inference

Is there a way to speed up inference to get it done in (almost) real-time?

Why the learning rate varies for the encoder, the decoder and the discriminator?

Hi. Thanks for your excellent research work.
I have a question why the learning rate varies for the encoder, the decoder and the discriminator?
In my opinion, the discriminator does not use the GAN prior weight, should the LR of discriminator be larger than that of decoder(generator)?

excentlent work, can you offer the pretrained_model ffhq-512 model（pytorch）? thanks

Not working with CPU getting error "OSError: CUDA_HOME environment variable is not set. Please set it to your CUDA install root."

Smaller and Faster model

Hello!

Thank out for a great project.

Model is already extremely fast, but here you mentioned faster and lighter model, wanted to know if you are still going to release it.

The restoration result is over-smooth

您好，我在复现您的论文时，发现复原的结果过于平滑，头发以及脸上的细节纹理没有被很好的复原出来。请问您觉得可能是哪里出了问题，可以给出一点修改意见吗？谢谢！

face_enhancement 处理图片的速度非常慢 face_ Enhancement is very slow to process pictures

I have a video that is fuzzy. I use GPEN to optimize it. The video generated 405 pictures in total. It took 11 minutes to complete GPEN processing. My graphics card is a quarter of T4 graphics card. How can I optimize the processing speed? Looking forward to your reply?

我有一个视频比较模糊，我使用了GPEN 来优化。视频一共生成了405张图片，GPEN 处理完成耗时 11分钟，我的显卡是 1/4 的 T4 显卡。我该如何优化处理速度？非常期待您的回复?

Error when trying to run it

I have tried to install it but when I try to launch the python file I get this error on Windows 10.

(base) C:\better\GPEN>python face_enhancement.py Traceback (most recent call last): File "face_enhancement.py", line 12, in <module> from retinaface.retinaface_detection import RetinaFaceDetection File "C:\better\GPEN\retinaface\retinaface_detection.py", line 6, in <module> import torch File "C:\Users\hadex\anaconda3\lib\site-packages\torch\__init__.py", line 123, in <module> raise err OSError: [WinError 127] No se encontró el proceso especificado. Error loading "C:\Users\hadex\anaconda3\lib\site-packages\torch\lib\caffe2_detectron_ops.dll" or one of its dependencies.

Runtime error happened when running face_colorization.py.

Hi, I just want to run the demo to see the effect on my machine and I got the result below.

Traceback (most recent call last):
File "/home/suoych/.local/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 1672, in _run_ninja_build
env=env)
File "/usr/lib/python3.6/subprocess.py", line 438, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "face_colorization.py", line 12, in
from face_model.face_gan import FaceGAN
File "/home/suoych/GPEN/face_model/face_gan.py", line 13, in
from model import FullGenerator
File "/home/suoych/GPEN/face_model/model.py", line 15, in
from op import FusedLeakyReLU, fused_leaky_relu, upfirdn2d
File "/home/suoych/GPEN/face_model/op/init.py", line 1, in
from .fused_act import FusedLeakyReLU, fused_leaky_relu
File "/home/suoych/GPEN/face_model/op/fused_act.py", line 14, in
os.path.join(module_path, 'fused_bias_act_kernel.cu'),
File "/home/suoych/.local/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 1092, in load
keep_intermediates=keep_intermediates)
File "/home/suoych/.local/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 1303, in jit_compile
is_standalone=is_standalone)
File "/home/suoych/.local/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 1408, in write_ninja_file_and_build_library
error_prefix=f"Error building extension '{name}'")
File "/home/suoych/.local/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 1682, in run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error building extension 'fused': [1/2] :/usr/local/cuda-11.2/bin/nvcc -DTORCH_EXTENSION_NAME=fused -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /home/suoych/.local/lib/python3.6/site-packages/torch/include -isystem /home/suoych/.local/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -isystem /home/suoych/.local/lib/python3.6/site-packages/torch/include/TH -isystem /home/suoych/.local/lib/python3.6/site-packages/torch/include/THC -isystem :/usr/local/cuda-11.2/include -isystem /usr/include/python3.6m -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_BFLOAT16_CONVERSIONS -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_61,code=compute_61 -gencode=arch=compute_61,code=sm_61 --compiler-options '-fPIC' -std=c++14 -c /home/suoych/GPEN/face_model/op/fused_bias_act_kernel.cu -o fused_bias_act_kernel.cuda.o
FAILED: fused_bias_act_kernel.cuda.o
:/usr/local/cuda-11.2/bin/nvcc -DTORCH_EXTENSION_NAME=fused -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /home/suoych/.local/lib/python3.6/site-packages/torch/include -isystem /home/suoych/.local/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -isystem /home/suoych/.local/lib/python3.6/site-packages/torch/include/TH -isystem /home/suoych/.local/lib/python3.6/site-packages/torch/include/THC -isystem :/usr/local/cuda-11.2/include -isystem /usr/include/python3.6m -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_61,code=compute_61 -gencode=arch=compute_61,code=sm_61 --compiler-options '-fPIC' -std=c++14 -c /home/suoych/GPEN/face_model/op/fused_bias_act_kernel.cu -o fused_bias_act_kernel.cuda.o
/bin/sh: 1: :/usr/local/cuda-11.2/bin/nvcc: not found
ninja: build stopped: subcommand failed.

I got GPU on my machine and my python version is 3.6, torch version 1.9, CUDA version 10.2, GCC version 7.5. I also checked the $PATH and the nvcc path is included.

It would be great if someone could help me with the problem. Thanks so much!

Testing code error

I used python ver : 3.6 / cuda ver : 10.0 / pytorch ver : 1.8.0, as README metioned,

While debugging the test code as followed,

from face_model.face_gan import FaceGAN at face_enhancement.py

from model import FullGenerator at face_gan.py

from op import FusedLeakyReLU, fused_leaky_relu, upfirdn2d at model.py

fused = load(
'fused',
sources=[
os.path.join(module_path, 'fused_bias_act.cpp'),
os.path.join(module_path, 'fused_bias_act_kernel.cu'),
],
) at fused_act.py

then I face following error

Please help me!

error

Traceback (most recent call last):
File "D:\GPEN-main\face_enhancement.py", line 13, in
from face_model.face_gan import FaceGAN
File "D:\GPEN-main\face_model\face_gan.py", line 13, in
from model import FullGenerator
File "D:\GPEN-main\face_model\model.py", line 15, in
from op import FusedLeakyReLU, fused_leaky_relu, upfirdn2d
File "D:\GPEN-main\face_model\op_init_.py", line 1, in
from .fused_act import FusedLeakyReLU, fused_leaky_relu
File "D:\GPEN-main\face_model\op\fused_act.py", line 10, in
fused = load(
File "D:\Anaconda3\envs\torch\lib\site-packages\torch\utils\cpp_extension.py", line 1122, in load
return _jit_compile(name,
File "D:\Anaconda3\envs\torch\lib\site-packages\torch\utils\cpp_extension.py", line 1367, in _jit_compile
return _import_module_from_library(name, build_directory, is_python_module)
File "D:\Anaconda3\envs\torch\lib\site-packages\torch\utils\cpp_extension.py", line 1751, in _import_module_from_library
file, path, description = imp.find_module(module_name, [path])
File "D:\Anaconda3\envs\torch\lib\imp.py", line 296, in find_module
raise ImportError(_ERR_MSG.format(name), name=name)
ImportError: No module named 'fused'

python3.6才可以运行，python3.7报错。

Runnin GPEN on Windows

Hi! I tried to run your repo on Windows 10 environment but it just remain in loop for long time. How can I solve it?

Black spots on the face & Inference without insightface

1.I tried GPEN in some low-res face-swap pics, find that black spots always show on generated faces, the lines of the face become deeper, makes face not real enough.
2.I modified GPEN codes, find that inference without insightface (but used mediapipe regardless the crop rule), the results also looks good to me.

Just share my using feeling, thx for your codes.

某些图片识别出错，输出的图像和输入图像一毛一样；
2.某些人脸识别不准，放着明显的脸不找，偏偏在头发、眼睛等位置硬生生搞出人脸的轮廓；
针对以上情况，作者有好的解决办法么？