csxmli2016 / dfdnet Goto Github PK

fyi
The image size is 512x512, my GPU is GTX1080 with 8GB VRAM

I use a side view of face and it says "No face is detected".Is the repo designed for almost front face or I just pick a bad image.
Is it possible to deal with videos in the future ? I think it's super useful to repair highly compressed video face.

I tried running the get_face_landmark.py on the supplied test images but the script returns "#########No face" for all images.

Here is a Google Colab notebook of DFDNet I put together. Thanks for your great work.
https://github.com/hopperrr/DFDNet_Colab

After running "python test_FaceDict.py", I get this message, but I can't seem to fix it. My GPU(0) is a 2080ti

THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1595629427286/work/torch/csrc/cuda/Module.cpp line=59 error=101 : invalid device ordinal
Traceback (most recent call last):
File "test_FaceDict.py", line 88, in
opt = TestOptions().parse()
File "/home/nextface/DFDNet/options/base_options.py", line 114, in parse
torch.cuda.set_device(opt.gpu_ids[0])
File "/home/nextface/miniconda3/envs/DFD/lib/python3.6/site-packages/torch/cuda/init.py", line 281, in set_device
torch._C._cuda_setDevice(device)
RuntimeError: cuda runtime error (101) : invalid device ordinal at /opt/conda/conda-bld/pytorch_1595629427286/work/torch/csrc/cuda/Module.cpp:59

Were the examples unseen data? Were they cherry picked?

Hi, I ran this in colab and am getting a weird output where the faces are grey and not coloured correctly.
You can check out the colab here https://colab.research.google.com/drive/1EvZeY66KeLQSTbCeRUpTciAwKJSjXa5T?usp=sharing

Any idea what is going on?
Thanks.

Lets say i want to just enhance the face without upscaling....

What value i should put in UpScaleWhole = ? Just leave it empty or put 0?

Or the enhance is only possible when upscaling?

Is it possible to visualize the original dictionaries?

Im using nvdia 2070 and amd 2700, and i got this error report😭How to solve it.
我现在是N卡2070和AU2700，第三步就报错显存不够，CPU模式用不了，请问要怎么解决啊😭泪，拉了出来。

Error in enhancing this image: CUDA out of memory. Tried to allocate 758.00 MiB (GPU 0; 8.00 GiB total capacity; 3.78 GiB already allocated; 674.04 MiB free; 4.99 GiB reserved in total by PyTorch)

How could DFDNet be changed to allow temporal coherence? https://streamable.com/wwmvuk

Hi Sir
I try to clone your work and loaded the model provided but when i try the test script (test_FaceDict.py), it produce some weird result. Could you provide some guidance on how to correct this?

I am running the project from pop os 20.04

The follow are my python packages
python 3.7.7
pytorch 1.6.0 (py3.7_cuda10.1.243_cudnn7.6.3_0)
dlib 19.20.99
dominate 2.4.0
opencv-python 4.2.0.34
scikit-image 0.17.2
face_alignment 1.1.0

The model I downloaded are from Google Drive.
https://drive.google.com/drive/folders/1bayYIUMCSGmoFPyd4Uu2Uwn347RW-vl5?usp=sharing

The following are the result I gotten.

Thanks.

Hi, How can I get a VGG-19 model file which is pretrained VGG-Face2 dataset??

Because I want to try construct dictionaries but I can find only VGG-16 model on the VGGFace2 website.

hey! I am enhancing my own videos but it takes a long time.
Is there a chance you have a folder with some examples of enhanced videos/frames sequences?
Thanks!

Hello guys!

Tell me what could be wrong? How can I enable debugging?

Or does the code not work under Windows?

Hi :)

I tried it out and and the result is awesome :)
But: i need only step 3 and 4. Means no aligment, no landmark.

How can i do that?

Thanks a lot :)

//edit: the source comes already aligned and cropped to 512x512 :) (DFL from iperov)

Hi, Excuse me,
when I just Run the Face_Restoration function with aligned image(512*512), it occured to a question :
"################ Error in enhancing this image: input and output sizes should be greater than 0, but got input (H: 16, W: 0) output (H: 40, W: 40)"
I checked the class UNetDictFace in network.py carefully, i find:
cur_feature is the shape of [1, 128, 256, 256];
mo_location is the shape of [4],and its value is [205, 288, 242, 325];
then ;
MO_feature = cur_feature[:,:,mo_location[1]:mo_location[3],mo_location[0]:mo_location[2]].clone();
it didn't work that the value would cross the border of cur_feature.

some pics work, but some pics didn't work.
Could u tell me the reason?
thx very much!!

Hi, can you release your train code?

I was never able to run the test on a GPU, using a 2070 gpu.
I was wondering if anybody was able to find where the problem is with the allocation of memory: I thought it was the number of test images, or the dictionary, but I used my own small dictionary and just 10 images, and it still gives me a cuda out of memory error.

(base) test@test-XPS-8700:~/DFDNet$ python test_FaceDict.py --test_path ./TestData/TestWhole --results_dir ./Results/TestWholeResults --upscale_factor 4 --gpu_ids 0

###################### Now Running the X 4 task ##############################

###############################################################################
####################### Step 1: Crop and Align Face ###########################
###############################################################################

Crop and Align test1.jpg image
Crop and Align test2.jpg image
Crop and Align test3.jpg image
Crop and Align test4.jpg image
Crop and Align test5.jpg image

###############################################################################
####################### Step 2: Face Landmark Detection #######################
###############################################################################

Detecting test1.jpg
Detecting test2.jpg
Detecting test3.jpg
Detecting test4.jpg
Detecting test5.jpg

###############################################################################
####################### Step 3: Face Restoration ##############################
###############################################################################

Traceback (most recent call last):
File "test_FaceDict.py", line 242, in
model.setup(opt)
File "/home/test/DFDNet/models/base_model.py", line 43, in setup
self.load_networks(opt.which_epoch)
File "/home/test/DFDNet/models/base_model.py", line 131, in load_networks
state_dict = torch.load(load_path, map_location=str(self.device))
File "/home/test/.local/lib/python3.7/site-packages/torch/serialization.py", line 585, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "/home/test/.local/lib/python3.7/site-packages/torch/serialization.py", line 772, in _legacy_load
deserialized_objects[key]._set_from_file(f, offset, f_should_read_directly)
RuntimeError: storage has wrong size: expected 131262 got 512

Just out of curiosity, is it possible to skip the face detection stage when running DFDNet? Would it crash or just keep working? I love experimenting and I'm curious how the result would look like since I'm looking for a tool that would help me enhance hair in photos where subjects have long hairs. So far I tested a lot of solutions such as DFDNet, Remini, and other apps and scripts of this type and they only enhance hairs around faces, leaving the rest of the hair looking like the subject put a load of glue on their head

Hi, I am getting this error:

File "test_FaceDict.py", line 13, in
import cv2
File "C:Users\User\anaconda3\envs\dfdNet\lib\site-packages\cv2_init_.py", line 5, in
from .cv2 import *
ImportError: DLL load failed: The specified module could not be found.

Thanks.

Hi! Thanks for open-sourcing such a perspective work!

I'm trying to test it in Colab and having some issues with CUDA on the last step (test_FaceDict.py).
Can you please provide us with some environment and requirements that can be used to reevaluate results?

Error:

----------------- Options ---------------
<Some options>
----------------- End -------------------
THCudaCheck FAIL file=/pytorch/torch/csrc/cuda/Module.cpp line=59 error=101 : invalid device ordinal
Traceback (most recent call last):
  File "test_FaceDict.py", line 88, in <module>
    opt = TestOptions().parse()
  File "/content/DFDNet/options/base_options.py", line 114, in parse
    torch.cuda.set_device(opt.gpu_ids[0])
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 281, in set_device
    torch._C._cuda_setDevice(device)
RuntimeError: cuda runtime error (101) : invalid device ordinal at /pytorch/torch/csrc/cuda/Module.cpp:59

Thanks for the great paper and for sharing the code.

Currently, the code does not seem to be able to load the pre-train model.
I modified model/network.py as follows and it generated successfully.

netG = UNetDictFace(64)

↓

netG = UNetDictFace(64)
netG.load_state_dict(torch.load('./weights/latest_net_G.pth'))

thank you!

1.what is the size of your gaussianblur kernel ?
2.can you provide blur kernels used in your experiment or relative codes?
3.why down-sampler r is not integer but decimal?for get images of any size?
and the most important question is : your job is of good generalization to real LQ images,what is the most key element?model structure ? use of face features dicts?

Just download the update zip files, extract and replace?

Or should be done in another way?

Thx in advance!

Hi!
Step 3
Restoring test1.jpg
################ Error in enhancing this image: CUDA out of memory. Tried to allocate 758.00 MiB (GPU 0; 3.81 GiB total capacity; 2.10 GiB already allocated; 46.31 MiB free; 2.33 GiB reserved in total by PyTorch)
################ continue...
Step 4
error
missing image, Step4_FinalResults map empty

Im use Ubuntu 20.04
GeForce GTX 1650 / 4GB
Ubuntu default install: cuda 10.1 and libcudnn8_8.0.4.30-1+cuda10.1_amd64

Just like deoldify would be awesome if there was a docker build file to not require native pytorch.

Why my cpu and gpu process time is the same?

Hi, thank you very much for your open source projiect. I found it have wonderful performance in many test image. But also i found the algorithm does not pay too much attention to the details, resulting in partial animation and painting style.

I really hope to reproduce the training process, so can you tell me more about the image processing and its related code？

Looking forward to your reply!

Hi,
I am trying to run python get_face_landmark.py with 2 test images n000024_0042_02 and n000056_0060_01
but it returned with:

(DFD) G:\DFDNet\DFDNet\FaceLandmarkDetection>python get_face_landmark.py
(0, 'Landmarks')
Traceback (most recent call last):
  File "get_face_landmark.py", line 28, in <module>
    imgs = io.imread(os.path.join(FilePath,name))
  File "E:\Anaconda3\envs\DFD\lib\site-packages\skimage\io\_io.py", line 48, in imread
    img = call_plugin('imread', fname, plugin=plugin, **plugin_args)
  File "E:\Anaconda3\envs\DFD\lib\site-packages\skimage\io\manage_plugins.py", line 209, in call_plugin
    return func(*args, **kwargs)
  File "E:\Anaconda3\envs\DFD\lib\site-packages\skimage\io\_plugins\imageio_plugin.py", line 10, in imread
    return np.asarray(imageio_imread(*args, **kwargs))
  File "E:\Anaconda3\envs\DFD\lib\site-packages\imageio\core\functions.py", line 265, in imread
    reader = read(uri, format, "i", **kwargs)
  File "E:\Anaconda3\envs\DFD\lib\site-packages\imageio\core\functions.py", line 182, in get_reader
    "Could not find a format to read the specified file in %s mode" % modename
ValueError: Could not find a format to read the specified file in single-image mode

How can I get past this error?
Thanks in advance

I want to use my picture to train my eyes model dictionary in the folder DictionaryCenter512

Hi.

I want you ask about component dictionaries and vgg 19 weights file that you provided.

which resolution of images are used for training?? is it 256 x 256 or 512 x 512 ??

In the paper, for handling 256 x 256 images, left/right eyes, nose and mouth on scale-1 are set to 40/40, 25, 55 respectively. So I think 256 x 256 resolution images are used for training.
But the provided model is making 512 x 512 resolution image.
I'm just wondering about that.
Thank you in advance :)

RuntimeError: Error while calling cudaOccupancyMaxPotentialBlockSize(&num_blocks,&num_threads,K)
shows up

I just tested your work on some real face images of arbitrary sizes, and the results look really good. However, the output images are always squared to 512x512px. Is it possible to perform true x4 upscaling without changing the proportions of the input images?

First appreciate your great works. I have simply checked the code in this repository and found a question on the component feature extraction and insertion.

I find you mentioned that using the ROIAlign for getting patches from feature maps in paper, but the code seems to simply crop patches using specified coordinates from the landmarks.

Is that a unsupported feature in this version of code 😃 .

I am using colab

!python test_FaceDict.py --gpu_ids=-1

###################### Now Running the X 4 task ##############################

###############################################################################
####################### Step 1: Crop and Align Face ###########################
###############################################################################

Traceback (most recent call last):
  File "test_FaceDict.py", line 166, in <module>
    detector = dlib.cnn_face_detection_model_v1('./packages/mmod_human_face_detector.dat')
RuntimeError: Error while calling cudaGetDevice(&the_device_id) in file /tmp/pip-wheel-66glv9rf/dlib/dlib/cuda/gpu_data.cpp:201. code: 100, reason: no CUDA-capable device is detected

Hi,

I noticed while trying to change the batch size that the only way the option can be parsed is by writing batchSize, capitalizing Size, but not batch. I don't know if this is fixable, but thought it might be good to put in a details section if not.

is there any way to make these run on gpu? what is the reason behind the fact that they only run on cpu?

In most cases I get an error while merging back to the original frame.
I can see the enhanced face in the : step3_restorecropface folder.

Sometimes it works, but in most cases I just get this error.

###############################################################################
############### Step 4: Paste the Restored Face to the Input Image ############
###############################################################################

Final Restoring 04377.png
Traceback (most recent call last):
File "test_FaceDict.py", line 270, in
reverse_align(WholeInputPath, FaceResultPath, ParamPath, SaveWholePath, UpScaleWhole)
File "test_FaceDict.py", line 59, in reverse_align
input_img = dlib.load_rgb_image(input_path)
RuntimeError: Unable to open file: /home/dfl2/Desktop/DFDNet-whole/TestData/TestWhole/04377.png

I notice there does not seem to be any indicator as to how we'd go about training/fine tuning the net on our own datasets, this is something I (and I suspect other people) would be very interested in. I'm quite curious as to how the model would perform with say further segmentation of facial components (e.g handling eyebrows and eyes separately) as well as fine tuning models trained on general face datasets on datasets of a specific person.

With that said I'd like to request that if it is possible, the steps for training be added to the readme page :D

I'm running through the first 3 steps of whole frames, I can see the RestoreCropFace folder and everything looks fine.

At step 4 it just dies.

###############################################################################
############### Step 4: Paste the Restored Face to the Input Image ############
###############################################################################

Final Restoring 002124.jpg
^C

I've worked back through running prints and think the offending line is here as nothing gets printed after it, but does before.
io.imsave(save_path, merge_img.astype(np.uint8))
line 76 test_FaceDict.py

I've checked file paths and they are all correct.

Pls use Google Drive mirror
I upload one map to zip

https://drive.google.com/file/d/1fPze7JAJUHVUYJspRXcEQGkiMc7Vvv8-/view?usp=sharing

I have tried to use official pix2pixHD repo as the template to reproduce your code on 2080Ti but found OOM ERROR.

Noticed that you used Tesla V100, so i am questioning about how much memory will costed (on 512x512) when training when i use --fp16 or not use --fp16 respectively？

Hi, thanks for this amazing work. I wanted to try training my own model with a custom dataset are there any plans to release the training script?

When I run the test with

python test_FaceDict.py --test_path ./TestData/TestWhole --results_dir ./Results/TestWholeResults --upscale_factor 4 --gpu_ids 0

this error occurs.
I am using python 3.6, torch 1.1.0, torchvision 0.3.0

where could the problem be?

Hi, is it a possibility to process a face in an entire frame (1920*1080) rather than cropping and scaling it to just the face?

Thanks.

what's the problem here?

Crop and Align Dx8jGPoU0AAXapa.jpg image
Traceback (most recent call last):
File "test_FaceDict.py", line 187, in
align_and_save(ImgPath, SavePath, SaveInput, SaveParam, UpScaleWhole)
File "test_FaceDict.py", line 44, in align_and_save
img = dlib.load_rgb_image(img_path)
AttributeError: module 'dlib' has no attribute 'load_rgb_image'

I have the question because my gpu doesn't have enough memory.

Hi, I'm getting the following error, any help on it? I've read it's an issue with the models, but I can't find them anywhere...

(DFDNet) C:\DFDNet-whole>python test_FaceDict.py --test_path ./TestData/TestWhole --results_dir ./Results/TestWholeResults --upscale_factor 4 --gpu_ids 0

###################### Now Running the X 4 task ##############################

###############################################################################
####################### Step 1: Crop and Align Face ###########################
###############################################################################

Crop and Align asd2 002.jpg image

###############################################################################
####################### Step 2: Face Landmark Detection #######################
###############################################################################

Traceback (most recent call last):
File "test_FaceDict.py", line 197, in
FD = face_alignment.FaceAlignment(face_alignment.LandmarksType._2D,device='cuda:0', flip_input=False)
File "E:\anaconda3\envs\DFDNet\lib\site-packages\face_alignment-1.0.1-py3.6.egg\face_alignment\api.py", line 69, in init
File "E:\anaconda3\envs\DFDNet\lib\site-packages\face_alignment-1.0.1-py3.6.egg\face_alignment\detection\sfd\sfd_detector.py", line 22, in init
File "E:\anaconda3\envs\DFDNet\lib\site-packages\torch\hub.py", line 495, in load_state_dict_from_url
return torch.load(cached_file, map_location=map_location)
File "E:\anaconda3\envs\DFDNet\lib\site-packages\torch\serialization.py", line 585, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "E:\anaconda3\envs\DFDNet\lib\site-packages\torch\serialization.py", line 772, in _legacy_load
deserialized_objects[key]._set_from_file(f, offset, f_should_read_directly)
RuntimeError: unexpected EOF, expected 9325805 more bytes. The file might be corrupted.