csbhr / ffhq-uv Goto Github PK

I tried to train a face reconstruction model on the basis of hifi3d++ based on Deep3DFaceRecon_pytorch, but serious deformation appeared around the mouth and eyes, and it seems that the effect of training is not enough, do you have encountered such a problem, do you have any good solutions?

您好，感谢您分享的工作！我刚接触该方向，有几个关于代码中投影矩阵以及坐标系变换的基础问题向您请教：

1、在render_nvdiffrast.py文件中的line19-20，为什么需要将ndc_projection得到的投影矩阵跟torch.diag(torch.tensor([1., -1, -1, 1]))的对角阵相乘？这一步的物理意义是什么？

2、在render_nvdiffrast.py文件中的line53，为什么需要将vertex的y坐标取负号？这一步的物理意义是什么？

期待您的回复，谢谢！

thanks your job!
Can this work train a model to do UV mapping inference? If possible, how many images are needed as a training set

Thanks for your great work! But I wonder why the '01223_front.png' in your repo is totally front. I didn't found align code in 'run_styleflow_edit.py'. It seems like you load the inversed code to generate front face directly.

您好，感谢您分享的工作！我想问一下关于眼球安装的问题。我现在遇到一个问题，run_mesh_add_eyeball中先按照包络关键点进行刚性对齐，再按照眼眶进行位置微调的方法很棒。但我发现hifi3d生成结果中不同的人会存在左右眼球包络并不对称的情况，进而导致左右眼安装不对称的问题，不知道您是否遇到过。

您好，感谢您出色的工作!
我注意到该项目有部分环境要求为pytorch1.7.1+tensorflow1.5+cuda10.0，但是同时该项目又要安装pytorch3d，但是目前最新版本pytorch3d(0.7.4)支持的pytorch范围为PyTorch 1.10.0, 1.10.1, 1.10.2, 1.11.0, 1.12.0, 1.12.1, 1.13.0, 2.0.0 or 2.0.1，请问应该使用哪一个版本的pytorch3d才能在该项目环境下正常使用呢？

请问下眼球的模型和贴图都是统一的吗，还是可以生成不同的贴图和模型。

Hello,Thanks for the great work.

Installation requirement
Python 3.7
git clone https://github.com/facebookresearch/pytorch3d

but pytorch3d github Installation is :

it require Python 3.8, 3.9 or 3.10
how to do it?

when install pytorch3d failed, the error part message is like
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File "/home/migu/anaconda3/envs/ffhq-uv/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1539, in _run_ninja_build
env=env)
File "/home/migu/anaconda3/envs/ffhq-uv/lib/python3.7/subprocess.py", line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception

Can you give me some suggestion,Thanks in advance

你好，请问[Microsoft Face API]不能使用，有没有替代方案呢？

您好，我想请问一下这个项目可以测试自己的图片吗？就是用自己的照片，生产UV，拟合到shape中或者mesh中。还有一个就是可以获得最终的.obj文件吗？非常期待您的回复！

Thank you for release your awesome project.
I am novice of 3D reconstruction, and want to ask two questions after finished RGB fitting.

First question is,
if I want to change the color of the eyebrows of generated 3D face after finished RGB fitting script,
It seems that I can only manually check which faces belong to the eyebrows,
then change corresponding pixels on texture picture by UV coordinates.
Do I ignore anything or maybe you can teach me which faces belong which parts of generated 3D face ?

Second one is,
if the above is true,
can you teach me how to find corresponding pixels on the texture by 3D vertices ?

2023-07-13 15:01:07.375403: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2023-07-13 15:01:07.375417: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2023-07-13 15:01:07.375429: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2023-07-13 15:01:07.375440: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2023-07-13 15:01:07.375451: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2023-07-13 15:01:07.375462: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2023-07-13 15:01:07.375474: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2023-07-13 15:01:07.376538: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2023-07-13 15:01:07.376562: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2023-07-13 15:01:07.376571: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2023-07-13 15:01:07.376577: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2023-07-13 15:01:07.377548: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10322 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:40:00.0, compute capability: 7.5)
/root/miniconda3/envs/ffhq-uv/lib/python3.7/site-packages/tensorflow_core/python/client/session.py:1750: UserWarning: An interactive session is already active. This can cause out-of-memory errors in some cases. You must explicitly call InteractiveSession.close() to release resources held by the other session(s).
warnings.warn('An interactive session is already active. This can '
WARNING:tensorflow:From /root/FFHQ-UV-main/DataSet_Step4_UV_Texture/third_party/mtcnn/detect_face_with_mtcnn.py:289: The name tf.global_variables_initializer is deprecated. Please use tf.compat.v1.global_variables_initializer instead.

loading the recon model from ../checkpoints/deep3d_model/epoch_latest.pth
2023-07-13 15:01:15.146694: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2023-07-13 15:01:16.773960: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
/root/FFHQ-UV-main/DataSet_Step4_UV_Texture/preprocess/preprocess_func.py:45: FutureWarning: rcond parameter will change to the default of machine precision times max(M, N) where M and N are the input matrix dimensions.
To use the future default and silence this warning we advise to pass rcond=None, to keep using the old, explicitly pass rcond=-1.
k, _, _, _ = np.linalg.lstsq(A, b)
/root/FFHQ-UV-main/DataSet_Step4_UV_Texture/preprocess/init.py:82: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
trans_params = np.array([w0, h0, s, t[0], t[1], self.target_size])
Unwrap texture 01223 took 9.9289 seconds.
Unwrap texture process done!

Failed List: []
为什么我的第四步跑完只生成了一个demo的uv，而我在image里面放了三张图，dataset_examples里的attributes attributes_ms_api edit这三个文件夹里也没有生成对应的另外两张图片的内容。是漏了什么步骤吗

Thanks for the script to modify a FLAME mesh to enable reading from a UV texture made for HiFi3D++.

I'm trying to convert a UV texture made for HiFi3D++ into a UV texture for FLAME, such that I can use it on the original/unmodified FLAME mesh. I need to basically copy the pixels from one UV texture into the corresponding location in the new UV texture. Any tips on how to get this correspondence?

I run uv_unwarp code and found the hair is incorrpondent with your released one.

Is there a way to obtain the shape/face meshes corresponding to the ~50K texture UVs in the dataset?

If I simply take an FFHQ image and apply RGBFitting to it, the obtained texture doesn't look as highly detailed as the one in the provided dataset.

One option is to apply RGBFitting on an FFHQ image, and then simply take the output identity mesh and match it to the corresponding texture in the dataset. But this might not be entirely correct, since the dataset textures are combinations of textures from 3 different views, right?

I'm trying to generate a texture from a custom image. Is there any difference between these two options?

1. Run run_rgb_fitting.sh on my image
2. Place my image into the dataset folder and run run_ffhq_uv_dataset.sh

When testing both options using an FFHQ image, the texture generated by run_rgb_fitting.sh seems to be lower fidelity.

When I run python run_ms_api_attr.py --proj_data_dir ../examples/dataset_examples. I got his error

Error: 'Endpoint'.
MS API error, sleep 2 minutes and retry 1-st.

Thanks for the great work.
According to the paper, the released dataset has been filtered.

Is there any metohd to get the correspondence of the dataset with FFHQ?

请问下REALY这个步骤是做什么的

您好，感谢您的工作！
我有一个关于您使用的HIFI3d 3DMM中uv坐标的问题跟您请教：

请问在hifi3dpp.py文件中，
1、vtx_vt，也就是mesh几何顶点的uv坐标是如何获得的？

原始的hifi3d 3DMM的参数dict中似乎没有这个key。
2、类似地，vt_vtx_idx，也就是mesh的纹理坐标vt所对应的几何顶点的index是如何获得的？

原始的hifi3d 3DMM的参数dict中似乎也没有这个key。
3、如果我有一个新的3DMM，我应该如何得到上述两个参数的值呢？

期待您的回复，谢谢！

Hi, I downloaded the one that generates the picture from latent code dataset，but there is still some light on the picture, is that normal
?
Is this due to styleflow can not remove cleanly?

Thank you for your great work

My goal is to create my face in Unreal

Download: tin_0_1024.zip

你好，关于render我有一个疑问，

为什么这里对于UV texture的渲染选择的点映射呢，而不是直接使用dr.texture进行uv渲染，后者会有更好的渲染效果，但是我不确定对于收敛的影响，请问有做过相关的收敛对比吗

如题

Hello,
In your code, Is there any process for generating 3D head obj ? Or you just download the 3D head.obj as datasets from others ?

请教个问题，如果只想将obj模型的脸生成，不需要生成脖子部位这个可以实现吗

我想将生成的模型面数减少到8000有什么方法可以实现吗

Hey @csbhr ,
Thanks a lot for releasing the code. I tried the given run_rgb_fitting.sh and it's working fine.
But output face shape is not matching with the input as it's giving round faces in most of the cases.

Any idea how to improve the shape optimization part running in step 3 of ours_fit_model.py module?

Reference input,

Reference output,

I want to get the depth map

How to get depth map?

Thank you very much!

Hello,

Thanks for the great work. I am trying to reproduce the REALY benchmark results from your paper. Unfortunately I am unable to find the corresponding HIFI3D++ template mesh (.obj) file + the barycentric keypoints for it. Could you provide the files and instructions necessary to replicate the REALY benchmark results?

Thanks in advance

Thanks ,It looks useful for me

作者你好，请问下为什么会报这个错误：
/home/akira/anaconda3/envs/ffhq-uv/lib/python3.7/site-packages/requests/init.py:114: RequestsDependencyWarning: urllib3 (1.26.14) or chardet (None)/charset_normalizer (3.1.0) doesn't match a supported version!
RequestsDependencyWarning,
Traceback (most recent call last):
File "run_e4e_inversion.py", line 10, in
from models.psp import pSp # we use the pSp framework to load the e4e encoder.
File "/home/akira/FFHQ-UV/DataSet_Step1_Inversion/models/psp.py", line 6, in
from models.encoders import psp_encoders
File "/home/akira/FFHQ-UV/DataSet_Step1_Inversion/models/encoders/psp_encoders.py", line 9, in
from models.stylegan2.model import EqualLinear
File "/home/akira/FFHQ-UV/DataSet_Step1_Inversion/models/stylegan2/model.py", line 7, in
from models.stylegan2.op import FusedLeakyReLU, fused_leaky_relu, upfirdn2d
File "/home/akira/FFHQ-UV/DataSet_Step1_Inversion/models/stylegan2/op/init.py", line 1, in
from .fused_act import FusedLeakyReLU, fused_leaky_relu
File "/home/akira/FFHQ-UV/DataSet_Step1_Inversion/models/stylegan2/op/fused_act.py", line 6, in
from torch.utils.cpp_extension import load
File "/home/akira/anaconda3/envs/ffhq-uv/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 19, in
from .hipify import hipify_python
File "/home/akira/anaconda3/envs/ffhq-uv/lib/python3.7/site-packages/torch/utils/hipify/hipify_python.py", line 34, in
from .cuda_to_hip_mappings import CUDA_TO_HIP_MAPPINGS
File "/home/akira/anaconda3/envs/ffhq-uv/lib/python3.7/site-packages/torch/utils/hipify/cuda_to_hip_mappings.py", line 34, in
rocm_path = subprocess.check_output(["hipconfig", "--rocmpath"]).decode("utf-8")
File "/home/akira/anaconda3/envs/ffhq-uv/lib/python3.7/subprocess.py", line 411, in check_output
**kwargs).stdout
File "/home/akira/anaconda3/envs/ffhq-uv/lib/python3.7/subprocess.py", line 488, in run
with Popen(*popenargs, **kwargs) as process:
File "/home/akira/anaconda3/envs/ffhq-uv/lib/python3.7/subprocess.py", line 800, in init
restore_signals, start_new_session)
File "/home/akira/anaconda3/envs/ffhq-uv/lib/python3.7/subprocess.py", line 1551, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
NotADirectoryError: [Errno 20] Not a directory: 'hipconfig'
我是nvidia的gpu，怎么会用amd去编译

作者您好，首先感谢您做出的这份令人惊叹的工作。如果我想要使用您开源的代码，生成一套基于BFM系数（您论文中提到使用的是HIFI3D++系数）的纹理uvmap数据集，请问是可行的吗？具体地我需要在您这里面更改多少地方呢？多谢！

It is not possible to run the model because there is no account on Azure. What is the output of the detect_face function in your project? Are there any similar open libraries for solving this problem without using the api?

I've followed README to constuct the required environment and run the run_rgb_fitting.sh. But I got the following errors, could you please help me solve it?

Thanks for your great work. But could you please kindly share the codes about how to generate UV texture map by using .obj file information?

Hello!

Thanks for great work! I am currently trying to download a dataset from OneDrive (can not use Baidu as far as I'm not living in China). I have tried to generate link from network using method provided here to download dataset directly to my remote server. But loading always interrupts, and I get only 15-17 GBs from 180. Manual downloading is challenging too, because a little network issue ruins all loading.

You provided a script for downloading the dataset from AWS here but later you deprecated it. If there is some link to it, could you, please, share it?

Could you, please, provide some script/method for automatic downloading from OneDrive or maybe you have some S3 link?

Thanks your work. I want to train my own texgan model and hope to get some guidance.

Hi, I have already read your paper and I think the three stages reconstruction method mentioned in your paper is very great!
But I’m still a little confused in stage 2 that how to converted the parameter {Pid, Pexp, Ppose, Plight} to z？

您好，感谢您杰出的工作带来的灵感，想了解您在对比试验中提到了ganfit和avatarme，但是找了很久没找到开源代码，想了解是否有其他推理方式？或者您推荐类似的相关方法还有哪些？

The position of the output UV diagram is in the middle, how to lower the position of the face

Hello, I have a problem when I run mesh_add_eyeball.py, it just creates the two eyeball but it is not applied to the mesh head, so the final mesh head obj is still blank !

Hi and thank you very much for you contribution!

Although you provide extensive documentation for some things, I still cannot understand how to export 3d meshes for the FFHQ subjects.
Are there latent codes that correspond to HIFI3D 3dmm?
If I understood correctly, the run_gen_face_from_latent.sh script generates images and not 3D objects.

Could you help me identify what I'm missing?

(and btw the objs created with script for adding eyeballs miss the 'usemtl' line in the resulting .obj)

Thanks for your time.

Can you please host the data set on something other then baidu, I cant get it to download the site is so slow

Traceback (most recent call last):
File "run_styleflow_editing.py", line 328, in
set_normal(sf_model, args.proj_data_dir, output_edit_dir, fn, edit_items)
File "run_styleflow_editing.py", line 204, in set_normal
cur_latent = torch.load(os.path.join(proj_data_dir, 'latents', f'{basename}.pt'))
File "/root/miniconda3/envs/ffhq-uv/lib/python3.7/site-packages/torch/serialization.py", line 581, in load
with _open_file_like(f, 'rb') as opened_file:
File "/root/miniconda3/envs/ffhq-uv/lib/python3.7/site-packages/torch/serialization.py", line 230, in _open_file_like
return _open_file(name_or_buffer, mode)
File "/root/miniconda3/envs/ffhq-uv/lib/python3.7/site-packages/torch/serialization.py", line 211, in init
super(_open_file, self).init(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: '../examples/dataset_examples/latents/01223.pt'
为什么里面传入01223.pt了过后运行脚本的时候这个文件会被删除

Hi, Unable to download from AWS CloudFront.

how can i get a key and endpoint in ms_api?