fengju514 / face-pose-net Goto Github PK
View Code? Open in Web Editor NEWEstimate 3D face pose (6DoF) or 11 parameters of 3x4 projection matrix by a Convolutional Neural Network
Estimate 3D face pose (6DoF) or 11 parameters of 3x4 projection matrix by a Convolutional Neural Network
Hi, thank you for your great work.
In all the tests I've done, I never reached a yaw greater than 1 (or -1 in the negative) even though the subject was clearly viewed from profile. Hence I'm wondering what the unit of pose is.
Is it rad? Or has it been squeezed to [-1, 1]?
Thanks
when running main_fpn.py I currently get:
"Model restored.
Predicted pose for: subject10_a
Predicted pose for: subject10_a_flip
Predicted pose for: subject3_a
Predicted pose for: subject3_a_flip
and so on..."
Somehow the angles of the pose are not predicted and so far I couldn't find out why. Any knowledge on how I could fix this? Or where could this error come from?
I don't get any errors while running the code. Only some warnings from which I am not sure if they could result in the non-predicted pose?
"WARNING:tensorflow:From /home/usr/.local/lib/python2.7/site-packages/tensorflow/python/util/decorator_utils.py:127: VARIABLES (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.GraphKeys.GLOBAL_VARIABLES
instead.
2018-12-13 09:58:49.386090: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
INFO:tensorflow:Restoring parameters from ./fpn_new_model/model_0_1.0_1.0_1e-07_1_16000.ckpt
Model restored."
in the esimatePose
function:
## Force TF to use CPU oterwise we set the ID of the string of GPU we wanna use but here we are going use CPU
os.environ['CUDA_VISIBLE_DEVICES'] = '1' #e.g. str(FLAGS.gpu_id)# '7'
if use_gpu == False:
dev = '/cpu:0'
print("Using CPU")
elif use_gpu == True:
dev = '/gpu:0'
print("Using GPU " + os.environ['CUDA_VISIBLE_DEVICES'])
else:
raise ValueError('Only support 0 or 1 gpu.')
the code change the dev
variable, but is dev
be used anywhere else?
Hello!
In your paper, you said "Specifically, FPN was applied to the bounding boxes returned by the detector of
Yang and Nevatia [48], following expansion of its dimensions by 25%."
And <FACE_X, FACE_y, FACE_WIDTH, FACE_HEIGHT> is the face bounding box information, either obtained manually or by the face detector.
So if i want to change the face image, how can i get bounding boxes , and how to expansion of its dimensions by 25%. I can not find any code from [48], besides can you explain what is the meaning of FACE_X, FACE_y, FACE_WIDTH, FACE_HEIGHT?
Thanks a lot!
In the file network_shape.py --> class Network_Shape --> def feed. The variable basestring is not defined.
Has this already been done?
Hi Fengju, I got problem running the code. The following error shown during testing. I have checked the variables 'prj' in function warpImg. I used temp. variable to check the problem as following. The size of the elements, say [0] and [1], in the array are the same 50176. How could the problem be solved???
tmpX = np.squeeze(np.asarray( prj[0,:] )).astype('float32')
tmpY = np.squeeze(np.asarray( prj[1,:] )).astype('float32')
File "main_fpn.py", line 50, in <module> renderer_fpn.render_fpn(outpu_proc, output_pose_db, output_render) File "E:\PROJECTs\FaceReconstruction\FacePose\Face-Pose-Net-master\renderer_fpn.py", line 88, in render_fpn model3D.ref_U, eyemask, model3D.facemask, opts) File "E:\PROJECTs\FaceReconstruction\FacePose\Face-Pose-Net-master/face_renderer\renderer.py", line 120, in render frontal_raw = warpImg(img, ref_U.shape[0], ref_U.shape[1], prj_jnt, ind_jnt) File "E:\PROJECTs\FaceReconstruction\FacePose\Face-Pose-Net-master/face_renderer\renderer.py", line 32, in warpImg pixels = cv2.remap(img, tmpX, tmpY, cv2.INTER_CUBIC) cv2.error: OpenCV(3.4.2) D:\Build\OpenCV\opencv-3.4.2\modules\imgproc\src\imgwarp.cpp:1728: error: (-215:Assertion failed) dst.cols < 32767 && dst.rows < 32767 && src.cols < 32767 && src.rows < 32767 in function 'cv::remap'
Hello Feng,
I tried to run the code and the following issue came out. Do I knew more RAM on this code???
Traceback (most recent call last): File "main_fpn.py", line 48, in <module> getRts.esimatePose( model_folder, outpu_proc, output_pose_db, model_used, lr_rate_scalar, if_dropout, keep_rate, use_gpu=True ) File "F:\PROJECTS\Face Reconstruction\Face-Pose-Net-master\get_Rts.py", line 121, in esimatePose run_pose_estimation( root_model_path, inputFile, outputDB, model_used, lr_rate_scalar, if_dropout, keep_rate ) File "F:\PROJECTS\Face Reconstruction\Face-Pose-Net-master\get_Rts.py", line 61, in run_pose_estimation pose_lmdb_env = lmdb.Environment(outputDB, map_size=1e12) lmdb.Error: ./output_pose.lmdb: There is not enough space on the disk.
First of all, @fengju514 thank you for sharing this work. I'm a little bit confuse about the using of the rendering models of kind "model3D_aug_-......mat". By reading the code I saw that the yaw angle is used for rendering. My question is: if I want to frontalize the face (bring yaw, pitch, roll angle to 0) it is enough to run the demo code (from face_renderer) with the following model "models3d_new/model3D_aug_-00_00_01.mat"?
Thank you
when i run main_predict_6DoF.py , i get a error with
File "main_predict_6DoF.py", line 222, in extract_3dmm_pose
img_new = pu.preProcessImage_v2(image.copy(), bbox.copy(), factor, _resNetSize, 1)
AttributeError: 'module' object has no attribute 'preProcessImage_v2'
and pose_utils.py does not have func preProcessImage_v2.
and this problem is same in main_predict_ProjMat.py
Hello!
Thanks very much for your paper ,and now I can solve most of the problems in my database,but there are still a little,.
I think pose estimation will be very useful because I have both 3D point clloud and 2D images。That will be great if I can estimate the angle of the face deflection .
Now I have two questions about your code as below:
Hello!
I have a question than how do I get the value of pitch and roll ?
Could you give me some advise ?
Thanks a lot.
I tried using this on an infant child face and the render gave the shape of an adult head. How would I go about training a new set of mat files which better resemble the head shape I am creating?
Secondary question, how would I go about retraining for a larger resolution model? Currently the output is 227x227 and I want to generate higher resolution faces.
I am doing some research in 3d face point cloud,I have only a little experience,Now I have a problem that I can't find the nose position in my point cloud data,I see maybe your library can help me but I don't find the right program,could you help me? or tell me wether it can do this,many thanks!
Hi, thanks for your amazing work! One question, you just say that 'The detected landmarks were then used to compute the 6DoF labels for the images in this sets' in your paper, so I want to ask that how can you get the 6Dof labels using landmarks? In some issues, you mention that by using generic 3D face model, but may be you can explain it in more detail. Looking for your reply~
It shows a runtime error at
File "Face-Pose-Net/get_Rts.py", line 53, in run_pose_estimation
net_data = np.load(root_model_path +"PAM_frontal_ALexNet.npy").item()
with the message
ValueError: Object arrays cannot be loaded when allow_pickle=False
and if I set allow_pickle=True, it just keeps on printing endless bytes of strings and doesn't give any output
I got only 5 images from the program. I would like to get more than 100 images.Is it possible?
Hello,
I have a couple of quick questions reg the pose vector generated. Is my understanding of the code correct?
Pose vector is in the variable predRts
The vector has 3 Rotational and 3 translational parameters
The Rotation parameters are in radian and stored in the order Pitch (PredRts[0][0]) Yaw (PredRts[0][1]) and Roll (PredRts[0][2]) and the translation parameters are the remaining three in mm
Is there a utility function to plot the pose vectors generated on the input image (Superimpose the three axes on the image). If not, is there a utility script that exists somewhere which I can leverage? I want to visualize the results to check my understanding of the code.
Thanks
Hi,
I have a few questions about the translation ground truth:
Best Regards
the results printed by your code is:
Model restored.
Predicted pose for: subject10_a
[ 8.27930197e-02 1.95288494e-01 1.03690195e+00 -1.98086071e+01
8.39025688e+00 3.26982227e+03]
Predicted pose for: subject10_a_flip
[ 1.37187794e-01 -1.48106769e-01 -1.06753802e+00 2.50120773e+01
4.28642845e+00 3.29182764e+03]
then how to convert the results to raw,roll,and pitch angle?
much thx:)
i'm sorry to find that it's hard for me in china to get the fpn_model download for some reasons . can u provide some other ways to download the fpn_model? thank u!
Hi,
How are the metrics generated for: FILE, FACE_X, FACE_y, FACE_WIDTH, FACE_HEIGHT
In the readme it says obtained manually or by the face detector?
Is the face detector included in your repository?
Thanks,
JP
I am trying to get the frame pose ,the RT matrix is R+T
the T is tx,ty,tz,
the 6DOF is scale, pitch, yaw , roll, translation_x, translation_y ,missing translation_z,seems the translation_z is always 0,how can I add it, if I have RGBD datas
Hello, could you please give your comments on whether it is possible to write the obtained pred_shape and pred_lmks as obj file?
Traceback (most recent call last):
File "main_fpn.py", line 39, in
data_dict = myparse.parse_input(input_file)
File "/home/zyechun/project/Face-Pose-Net/myparse.py", line 7, in parse_input
for line in reader:
File "/usr/lib/python2.7/csv.py", line 107, in next
self.fieldnames
File "/usr/lib/python2.7/csv.py", line 90, in fieldnames
self._fieldnames = self.reader.next()
_csv.Error: line contains NULL byte
I enter "python main_fpn.py ./images/input1.jpg
How can I solve it?
Hello, I'd like to know how to display photos marked with feature points.
Best regards.
i want run main_predict_6DoF.py with python 3.5.4 and on windows 10. i fixed some syntax error about python version but this error occur :
D:\Artificial Intelligence\Face Detection_Recognition\Face\Face-Pose-Net-master\Face-Pose-Net-master>python main_predict_6DoF.py 0 new_input_list.txt
C:\Users\127051\AppData\Local\Programs\Python\Python35\lib\site-packages\h5py\__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
from ._conv import register_converters as _register_converters
/gpu:0
True
Traceback (most recent call last):
File "main_predict_6DoF.py", line 333, in <module>
tf.app.run()
File "C:\Users\127051\AppData\Roaming\Python\Python35\site-packages\tensorflow\python\platform\app.py", line 125, in run
_sys.exit(main(argv))
File "main_predict_6DoF.py", line 328, in main
extract_3dmm_pose()
File "main_predict_6DoF.py", line 107, in extract_3dmm_pose
net_shape = resnet101_shape({'input': x2}, trainable=True) # False: Freeze the ResNet Layers
File "./kaffe/tensorflow\network_shape.py", line 48, in __init__
self.setup()
File "./ResNet\ThreeDMM_shape.py", line 11, in setup
(self.feed('input')
File "./kaffe/tensorflow\network_shape.py", line 140, in feed
if isinstance(fed_layer, basestring):
NameError: name 'basestring' is not defined
when running "python main-fpn.py input.csv" I receive the "ValueError: too many values to unpack" when loading the pose model in model3D_aug_-00_00_01.mat, any idea what I'm doing wrong here?
I get the same error when trying to run "python main_fpn.py subject1_a,images/input1.jpg,108.2642,119.6774,170,179"
error is in line 10 of main_fpn.py
line 34 in renderer_fpn.py when loading the models
Hi. Could you give some more details how did you prepare the data?
Hello devs and all,
I am really charmed by your idea of head-pose estimation without landmarking and I appreciate all the work you have already done. Unfortunately, I have not (yet) been able to run main_predict_6DoF.py in either MacOs or Windows. Getting all dependencies right is frustrating...
Anyway. I am looking for a fast and accurate way to determine the 6 DOF headpose of "the user" in front of the webcam and so I came across FAME and FPN. My goal is to create apps for both Windows and MacOs and hence my question: how can I use FPN in either Visual Studio or XCode? Are there any C++/Objective-C versions (planned)?
Maybe integration in OpenCV would even be possible?
Hi, congratulations on your great work! I am wondering if you have the plan to release your training label (the generated 6Dofs from landmark).
Hi, I run your code, but I got the error, I post here, would you please help me? Thank you very much, I appreciate.
The OpenCV version is 3.4.0 in my machine. The error shows as follows:
OpenCV Error: Assertion failed (dst.cols < 32767 && dst.rows < 32767 && src.cols < 32767 && src.rows < 32767) in remap, file /io/opencv/modules/imgproc/src/imgwarp.cpp, line 1749
Traceback (most recent call last):
File "main_fpn.py", line 50, in
renderer_fpn.render_fpn(outpu_proc, output_pose_db, output_render)
File "/home1/caoleilei/faceRec/Face-Pose-Net/renderer_fpn.py", line 88, in render_fpn
model3D.ref_U, eyemask, model3D.facemask, opts)
File "/home1/caoleilei/faceRec/Face-Pose-Net/face_renderer/renderer.py", line 109, in render
frontal_raw = warpImg(img, ref_U.shape[0], ref_U.shape[1], prj_jnt, ind_jnt)
File "/home1/caoleilei/faceRec/Face-Pose-Net/face_renderer/renderer.py", line 25, in warpImg
np.squeeze( np.asarray( prj[1,:] ) ).astype('float32'), cv2.INTER_CUBIC)
cv2.error: /io/opencv/modules/imgproc/src/imgwarp.cpp:1749: error: (-215) dst.cols < 32767 && dst.rows < 32767 && src.cols < 32767 && src.rows < 32767 in function remap
Hello,
I used git clone --recursive https://github.com/fengju514/Face-Pose-Net.git to get both Face Pose Net and the renderer.
I downloaded the model file and put it in Face-Pose-Net/fpn_new_model
I then used the command python main_fpn.py input.csv to try out the model on the provided data and I keep getting the message :
**** Have you forgeten to "git clone --recursive"? ****
**** You have to do that to also download the face renderer ****
Am I missing a step here to run the code?
Hello,
I am facing the below error,
/programs/Face-Pose-Net-master/face_renderer/ThreeD_Model.py", line 65, in createMask
contours, hierarchy = cv2.findContours(eyemask,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE)
ValueError: too many values to unpack (expected 2)
Any help? Thanks
Hello,
Does the result returned by the pre-trained model depend on how tight the bounding box is to the face? Does it also depend on the aspect ratio of the box ? I'm seeing an almost frontal face with very little tilt register a large roll ~60 degrees.
As for the results generated by the pre-trained model, what are the sign conventions of the pitch, yaw and roll angles? For example, if a person's face is turned to the right, is that considered +ve or is that -ve? Similarly, what are the conventions for looking up, looking down, tilted right, tilted left?
Thanks!
Hello! I was wondering what is the license of this project and the "face renderer" dependency?
thank u share the 3d model for us, if i want to train model with my own data ,how to do it ?
Hi,
@iacopomasi
I am a new beginner. I have checked the code from the example usage of https://www.openu.ac.il/home/hassner/projects/poses/, but i don't know how to get the reference textured 3D model and how to set the parameters distance, elevation and azimuth.
Could you please tell me how to do?
Thanks.
I make a better one [https://github.com/WIKI2020/FacePose_pytorch]
Your paper's name is "landmark-free", I want to know if there is a step to extract landmarks?
Is the landmark extracted but not displayed, or is the landmark not extracted at all?
By the way, in your paper, you said the training set is produced by running on existing landmark detector on VGG dataset. Are all 2.6 million images used for training?
I want to render the input to a particular angle, is it possible?
Dear Developer:
Hello,I have download your python open source project :
Face-Pose-Net https://github.com/fengju514/Face-Pose-Net
Why it pop up this bug:
ModuleNotFoundError: No module named 'camera_calibration'
I have 1G of picture about human face,I need to get 6 DOF of their rotation,now I have to do the job hand by hand.I sincerely need your project to automatically do my job,please help me!
Xia Zhenghan
2018.2.26
Hello,
For the results generated by the pre-trained model, what are the sign conventions of the pitch, yaw and roll angles?
For example, if a person's face is turned to the right, is that considered +ve or is that -ve?
similarly, what are the conventions for looking up, looking down, tilted right, tilted left?
Thanks!
Hello,
Please help me to solve this problem, thanks very much!
Traceback (most recent call last):
File "/home/zheng/Face-Pose-Net/main_fpn.py", line 23, in
input_file = str(sys.argv[1]) #'input.csv'
IndexError: list index out of range
the render only limit the yaw angle, is there a way to limit the other 2 angles?
where is the dnn model file. and so that get 3d pos and 3d face shape ?
This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint.
I want to generate the images of certain poses without background, but have no idea how to achieve it.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.