arthur151 / romp Goto Github PK

Hey @Arthur151,
I was able to understand the joint positions and order in SMPL24 and I know that the pose (72,) is the rotation values for 24 joints in SMPL.
I am trying to convert the output into a BVH file, but I don't understand, how to use these rotation values while converting into BVH. Do i need to change the format of the joint rotations. If yes, then to which format and how?
Can you please guide me on this?

作者你好，请问在做训练数据的时候，camera map（1x3）是通过3dpw的ground truth中的哪些数据通过什么方式换算过来的呢？可否给你一个具体的公式呢？

您好！
我在 3DPW比赛网站上找到的两个 track都是不允许用 2D pose对图片进行crop的，但是看到您这边说 follow的 VIBE 数据预处理都是用的 官方提供的2D pose得到 bbox的，这样是不是就少了一个tracking的误差? 还是在比赛之外大家评测的时候就是给定了对应关系?
感谢!

Can you share the video / ppt for your presentation at eccv 2020 workshop? I am also interested in solutions from other teams. But I can not find related materials on the internet.

Hello! I have recently run your work. I want to get one main person from my video. However there are some other persons showing up during the video sometimes. How should i set the configs? I have tried set "multi_person" to False, but seems it won't work.
It would be so nice if you could help me here. Thank you.

Hello. Thanks for your work. I am currently conducting some researches on 3d human pose estimation, and I want to consult some concepts.

I noticed that there are lots of information containing in the pkl files. I am a newcomer in this field, and the values make me confused. For example, I wonder that what's the difference between "pose", "pose2d", "poses_60Hz" and "jointPositions"? And also, there are 5 items containing the information of SMPL including "betas", "betas_clothed" and "pose", and what's their exact meaning?

Or to be simplified, are there any references which can explain such items with exact meanings in details? It will be my pleasure if you are willing to help me!

File "core/test.py", line 4, in
from utils.multiprocess import * #Multiprocess
File "/root/lj28/project/git-code/CenterHMR/src/utils/multiprocess.py", line 9, in
from utils.demo_utils import OpenCVCapture, Open3d_visualizer
File "/root/lj28/project/git-code/CenterHMR/src/utils/demo_utils.py", line 5, in
import open3d as o3d
File "/opt/conda/lib/python3.6/site-packages/open3d/init.py", line 56, in
_CDLL(next((_Path(file).parent / 'cpu').glob('pybind*')))
File "/opt/conda/lib/python3.6/ctypes/init.py", line 348, in init
self._handle = _dlopen(self._name, mode)
OSError: /opt/conda/lib/python3.6/site-packages/open3d/cpu/pybind.cpython-36m-x86_64-linux-gnu.so: undefined symbol: glPushGroupMarkerEXT

Hello. I am coming from another details about the implementation.
In 2d pose estimation, we always use heatmap regression method, which generates a third attribute "confidence" which can be used for detected non-existent keypoints. As for the keypoint whose maximum value of the whole heatmap is too low, the corresponding key-point can be recognised as "not exist", which is maybe out of the image or occluded.
However, in 3d pose estimation, there's neither an attribute called "confidence", nor a heatmap. Does it means that every point are assumed to be existent? Are there any tricks for detecting the occluded body-parts?

We use 256 image input to your trained 512 input model, and adjust the coordconv size, but the result seems not right. Does your network need the same input size on train and test stage?

Thanks for your work. I am running your demo with CUDA_VISIBLE_DEVICES=0 python core/test.py --gpu=0 --configs_yml=configs/single_image.yml
But an error occurs.
I am running with ubuntu16.04. The python verison is 3.6.9. How should I check what's happening?

(romp) jack@jack-System-Product-Name:~/Documents/ROMP/src$ CUDA_VISIBLE_DEVICES=0 python core/test.py --configs_yml=configs/single_image.yml
pygame 2.0.1 (SDL 2.0.14, Python 3.6.13)
Hello from the pygame community. https://www.pygame.org/contribute.html
INFO:root:{'tab': 'hrnet_cm64_single_image_test', 'configs_yml': 'configs/single_image.yml', 'demo_image_folder': '/path/to/image_folder', 'local_rank': 0, 'model_version': 1, 'multi_person': True, 'collision_aware_centermap': False, 'collision_factor': 0.2, 'kp3d_format': 'smpl24', 'eval': False, 'max_person': 64, 'input_size': 512, 'Rot_type': '6D', 'rot_dim': 6, 'centermap_conf_thresh': 0.25, 'centermap_size': 64, 'deconv_num': 0, 'model_precision': 'fp32', 'backbone': 'hrnet', 'gmodel_path': '../trained_models/ROMP_hrnet32.pkl', 'print_freq': 50, 'fine_tune': True, 'gpu': '0', 'batch_size': 64, 'val_batch_size': 1, 'nw': 4, 'calc_PVE_error': False, 'dataset_rootdir': '/home/jack/Documents/dataset/', 'high_resolution': True, 'save_best_folder': '/home/jack/Documents/checkpoints/', 'log_path': '/home/jack/Documents/log/', 'total_param_count': 85, 'smpl_mean_param_path': '/home/jack/Documents/ROMP/models/satistic_data/neutral_smpl_mean_params.h5', 'smpl_model': '/home/jack/Documents/ROMP/models/statistic_data/neutral_smpl_with_cocoplus_reg.txt', 'smplx_model': True, 'cam_dim': 3, 'beta_dim': 10, 'smpl_joint_num': 22, 'smpl_model_path': '/home/jack/Documents/ROMP/models', 'smpl_J_reg_h37m_path': '/home/jack/Documents/ROMP/models/smpl/J_regressor_h36m.npy', 'smpl_J_reg_extra_path': '/home/jack/Documents/ROMP/models/smpl/J_regressor_extra.npy', 'kernel_sizes': [5], 'GPUS': 0, 'use_coordmaps': True, 'webcam': False, 'video_or_frame': False, 'save_visualization_on_img': True, 'output_dir': '/path/to/outputdir', 'save_mesh': True, 'save_centermap': True, 'save_dict_results': True, 'multiprocess': False}
INFO:root:------------------------------------------------------------------
INFO:root:start building model.
Using ROMP v1
INFO:root:using fine_tune model: ../trained_models/ROMP_hrnet32.pkl
INFO:root:finished build model.
Traceback (most recent call last):
File "core/test.py", line 225, in
main()
File "core/test.py", line 205, in main
demo = Demo()
File "core/test.py", line 7, in init
self.prepare_modules()
File "core/test.py", line 14, in prepare_modules
self.visualizer = Visualizer(resolution=self.vis_size, input_size=self.input_size,with_renderer=True)
File "/home/jack/Documents/ROMP/src/core/../lib/models/../utils/../maps_utils/../dataset/../dataset/../dataset/../visualization/visualization.py", line 23, in init
self.renderer = get_renderer(resolution=resolution)
File "/home/jack/Documents/ROMP/src/core/../lib/models/../utils/../maps_utils/../dataset/../dataset/../dataset/../visualization/../visualization/renderer.py", line 142, in get_renderer
renderer = Renderer(faces,resolution=resolution[:2])
File "/home/jack/Documents/ROMP/src/core/../lib/models/../utils/../maps_utils/../dataset/../dataset/../dataset/../visualization/../visualization/renderer.py", line 72, in init
point_size=1.0)
File "/home/jack/anaconda3/envs/romp/lib/python3.6/site-packages/pyrender/offscreen.py", line 31, in init
self._create()
File "/home/jack/anaconda3/envs/romp/lib/python3.6/site-packages/pyrender/offscreen.py", line 149, in _create
self._platform.init_context()
File "/home/jack/anaconda3/envs/romp/lib/python3.6/site-packages/pyrender/platforms/egl.py", line 188, in init_context
EGL_NO_CONTEXT, context_attributes
File "/home/jack/anaconda3/envs/romp/lib/python3.6/site-packages/OpenGL/platform/baseplatform.py", line 402, in call
return self( *args, **named )
File "/home/jack/anaconda3/envs/romp/lib/python3.6/site-packages/OpenGL/error.py", line 232, in glCheckError
baseOperation = baseOperation,
OpenGL.error.GLError: GLError(
err = 12297,
baseOperation = eglCreateContext,
cArguments = (
<OpenGL._opaque.EGLDisplay_pointer object at 0x7ff367d4e268>,
<OpenGL._opaque.EGLConfig_pointer object at 0x7ff367d4e1e0>,
<OpenGL._opaque.EGLContext_pointer object at 0x7ff367e84d08>,
<OpenGL.arrays.lists.c_int_Array_7 object at 0x7ff367e64d08>,
),
result = <OpenGL._opaque.EGLContext_pointer object at 0x7ff367d311e0>
)

I removed Coordconv when training centermap task independently, the training loss similar to the network with Coordconv, but the centermap seems to get bad on the test set.

First of all thanks a lot for this great repo

How can I export .obj file of a video ?

by change save_mesh to True nothing generate?

How can I save mesh of all video frames In one .fbx file? (Just like vibe repo https://github.com/mkocabas/VIBE
https://github.com/mkocabas/VIBE/blob/c0c3f77d587351c806e901221a9dc05d1ffade4b/lib/utils/fbx_output.py)

or How can I concat all .obj file by exporting via image config (not video) as one .fbx file?

I've try to use the .npy file of video to ROMP/src/lib/utils/convert_fbx.py and generate .fbx file but the generated file doesn't consist of any animation in belender.

Could you please help to solve this issue.

Hi, thanks for the great work.

Could you please hint me how to convert from rotation matrix representation (Bx3x3) to axis-angle representation (Bx3) (used in original SMPL)?

Thank you!

Hi, is it possible if you can post a step-by-step tutorial on how to run and install the program? Thank you in advance.

I didn't find the download link of resnet model. Have you ever published it? Thanks.

you can add a function to export to bvh and use in blender

@Arthur151 I have a question. In the final outputs of 25 3D op_joints, I want to revert/reverse the values of Z. Kindly update me that how to do it?
And if I want to revert the values of all axis (X, Y and Z) then how I can change it?

Waiting for response.

Visualization bug: OpenGL.error.GLError: GLError(err = 12289,baseOperation = eglMakeCurrent ....

The bug of pyrender is really a pain in the ass... ``bash

按照下面关于 centos 的解决办法，对我的linux系统（redHat） 无效，依然报上面的错
for centos
sudo yum install libXext libSM libXrender freeglut-devel

Hello. I feel a little concerned about your "camera map".

As illustrated in your paper, the camera map contains the 3-dim camera parameters (s, tx , ty ) that describe the 2D scale s and translation t = (tx , ty ) of the person in the image. The scale s reflects the body size and the depth to some extent. tx and ty , ranging in (−1, 1), reflect the normalized translation of the human body relative to the image center on the x and y axis, respectively.

From my personal perspective, the camera map may contain 6 parameters: rotation and translation of 3 axis, which comprises the extrinsic parameters of the camera position. Maybe I may recognise the parameter s as the translation of z dimension, but I am wondering that how can we obtain the rotation of the xyz dimension?

Hello Im trying to run the colab demo, under the "Run the demo code " heading, when i run the first command of
%cd /content/ROMP/src !CUDA_VISIBLE_DEVICES=0 python core/test.py --gpu=0 --configs_yml=configs/single_image.yml

I get the following error :
/content/ROMP/src pygame 2.0.1 (SDL 2.0.14, Python 3.7.10) Hello from the pygame community. https://www.pygame.org/contribute.html Traceback (most recent call last): File "core/test.py", line 1, in <module> from base import * File "/content/ROMP/src/core/base.py", line 20, in <module> from evaluation import compute_error_verts, compute_similarity_transform, compute_similarity_transform_torch, \ File "/content/ROMP/src/core/../lib/models/../utils/../maps_utils/../evaluation/__init__.py", line 1, in <module> from .evaluation_matrix import compute_error_verts, compute_similarity_transform, compute_similarity_transform_torch, \ File "/content/ROMP/src/core/../lib/models/../utils/../maps_utils/../evaluation/evaluation_matrix.py", line 6, in <module> from smplx import SMPL File "/usr/local/lib/python3.7/dist-packages/smplx/__init__.py", line 17, in <module> from .body_models import ( File "/usr/local/lib/python3.7/dist-packages/smplx/body_models.py", line 27, in <module> from loguru import logger ModuleNotFoundError: No module named 'loguru'

Cell "!CUDA_VISIBLE_DEVICES=0 python core/test.py --gpu=0 --configs_yml=configs/basic_test_video.yml"
Outputs:

Configuration:
{'tab': 'test', 'configs_yml': 'configs/basic_test_video.yml', 'demo_image_folder': '../demo/videos/Messi_1', 'multi_person': True, 'use_coordmaps': True, 'head_conv_block_num': 2, 'kp3d_format': 'smpl24', 'eval': False, 'max_person': 16, 'BN_type': 'BN', 'Rot_type': '6D', 'center_idx': 1, 'centermap_size': 64, 'HMloss_type': 'MSE', 'model_precision': 'fp32', 'baseline': 'hrnetv5', 'input_size': 512, 'gmodel_path': '../trained_models/pw3d_81.8_58.6.pkl', 'best_save_path': '', 'print_freq': 50, 'epoch': 300, 'fine_tune': True, 'lr': 0.0003, 'weight_decay': 1e-05, 'gpu': '0', 'batch_size': 64, 'val_batch_size': 8, 'nw': 4, 'dataset_rootdir': '/content/dataset/', 'dataset': 'h36m,mpii,coco,aich,up,ochuman,lsp,movi', 'voc_dir': '/content/dataset/VOCdevkit/VOC2012/', 'high_resolution': True, 'save_best_folder': '/content/checkpoints/', 'log_path': '/content/log/', 'total_param_count': 85, 'smpl_model_path': '/content/CenterHMR/models', 'adjust_lr_epoch': [], 'kernel_sizes': [5], 'GPUS': 0, 'save_mesh': False, 'save_centermap': False, 'save_dict_results': False, 'webcam': False}

Traceback (most recent call last):
File "core/test.py", line 4, in
from utils.multiprocess import * #Multiprocess
File "/content/CenterHMR/src/utils/multiprocess.py", line 9, in
from utils.demo_utils import OpenCVCapture, Open3d_visualizer
File "/content/CenterHMR/src/utils/demo_utils.py", line 4, in
import open3d as o3d
File "/usr/local/lib/python3.6/dist-packages/open3d/init.py", line 64, in
_CDLL(next((_Path(file).parent / 'cpu').glob('pybind*')))
File "/usr/lib/python3.6/ctypes/init.py", line 348, in init
self._handle = _dlopen(self._name, mode)
OSError: libc++.so.1: cannot open shared object file: No such file or directory"

Regards

Hello! I´ve been using your work recently and its very good! However, I´m stuck trying to do the segmentation of the different body parts in the mesh of the results.

So far, I tried to use the SMPL format and this labeler: https://github.com/ZhengZerong/SmplLabeling, but I do not find some information in the results as, in example, the J regressor.

There is a way with the current code to do this? If I didn´t explain myself well enough, do not hesitate to ask me.

Thank you in advance.

I have to try loading the project on VM machine and this issue occurs,
my machine is UBUNTU 18.04, and 3D is enabled in VMware.
Indeed I tried with multiple pyrender and OpenGL Versions but doesn't solve it.

INFO - 2021-05-02 18:43:37,442 - base - finished build model.
Traceback (most recent call last):
File "/root/Desktop/final/m3d/src/core/test.py", line 230, in
main()
File "/root/Desktop/final/m3d/src/core/test.py", line 210, in main
demo = Demo()
File "/root/Desktop/final/m3d/src/core/test.py", line 7, in init
self.prepare_modules()
File "/root/Desktop/final/m3d/src/core/test.py", line 15, in prepare_modules
self.visualizer = Visualizer(resolution=self.vis_size, input_size=self.input_size,with_renderer=True)
File "/root/Desktop/final/m3d/src/core/../lib/models/../utils/../maps_utils/../dataset/../dataset/../dataset/../visualization/visualization.py", line 23, in init
self.renderer = get_renderer(resolution=resolution)
File "/root/Desktop/final/m3d/src/core/../lib/models/../utils/../maps_utils/../dataset/../dataset/../dataset/../visualization/../visualization/renderer.py", line 142, in get_renderer
renderer = Renderer(faces,resolution=resolution[:2])
File "/root/Desktop/final/m3d/src/core/../lib/models/../utils/../maps_utils/../dataset/../dataset/../dataset/../visualization/../visualization/renderer.py", line 72, in init
point_size=1.0)
File "/usr/local/lib/python3.6/dist-packages/pyrender/offscreen.py", line 31, in init
self._create()
File "/usr/local/lib/python3.6/dist-packages/pyrender/offscreen.py", line 149, in _create
self._platform.init_context()
File "/usr/local/lib/python3.6/dist-packages/pyrender/platforms/egl.py", line 188, in init_context
EGL_NO_CONTEXT, context_attributes
File "/usr/local/lib/python3.6/dist-packages/OpenGL/platform/baseplatform.py", line 409, in call
return self( *args, **named )
File "src/errorchecker.pyx", line 58, in OpenGL_accelerate.errorchecker._ErrorChecker.glCheckError
OpenGL.error.GLError: GLError(
err = 12297,
baseOperation = eglCreateContext,
cArguments = (
<OpenGL._opaque.EGLDisplay_pointer object at 0x7f3dba91e2f0>,
<OpenGL._opaque.EGLConfig_pointer object at 0x7f3dba91e378>,
<OpenGL._opaque.EGLContext_pointer object at 0x7f3dbcaabe18>,
<OpenGL.arrays.lists.c_int_Array_7 object at 0x7f3dbcaab1e0>,
),
result = <OpenGL._opaque.EGLContext_pointer object at 0x7f3dba940268>
)

Hi,
thanks for releasing the code.
When I run the code on the google colab, I get this error

Configuration:
{'tab': 'test', 'configs_yml': 'configs/basic_test.yml', 'demo_image_folder': 'None', 'multi_person': True, 'use_coordmaps': True, 'head_conv_block_num': 2, 'kp3d_format': 'smpl24', 'eval': False, 'max_person': 16, 'BN_type': 'BN', 'Rot_type': '6D', 'center_idx': 1, 'centermap_size': 64, 'HMloss_type': 'MSE', 'model_precision': 'fp32', 'baseline': 'hrnetv5', 'input_size': 512, 'gmodel_path': '../trained_models/pw3d_81.8_58.6.pkl', 'best_save_path': '', 'print_freq': 50, 'epoch': 300, 'fine_tune': True, 'lr': 0.0003, 'weight_decay': 1e-05, 'gpu': '0', 'batch_size': 64, 'val_batch_size': 2, 'nw': 4, 'dataset_rootdir': '/content/dataset/', 'dataset': 'h36m,mpii,coco,aich,up,ochuman,lsp,movi', 'voc_dir': '/content/dataset/VOCdevkit/VOC2012/', 'high_resolution': True, 'save_best_folder': '/content/checkpoints/', 'log_path': '/content/log/', 'total_param_count': 85, 'smpl_model_path': '/content/CenterHMR/models', 'adjust_lr_epoch': [], 'kernel_sizes': [5], 'GPUS': 0, 'save_mesh': False, 'save_centermap': True, 'save_dict_results': True, 'webcam': False, 'multiprocess': False}

pygame 2.0.0 (SDL 2.0.12, python 3.6.9)
Hello from the pygame community. https://www.pygame.org/contribute.html
model_type smpl
start building model.

using fine_tune model: ../trained_models/pw3d_81.8_58.6.pkl
missing parameters of layers:
[]

finished build model.
Initialization finished!
gathering datasets
Loading 3 internet data from:/content/CenterHMR/demo/images
Size should be int or sequence. Got <class 'numpy.ndarray'>
Size should be int or sequence. Got <class 'numpy.ndarray'>
Traceback (most recent call last):
File "core/test.py", line 194, in
main()
File "core/test.py", line 190, in main
demo.run(demo_image_folder)
File "core/test.py", line 28, in run
for test_iter,data_3d in enumerate(loader_val):
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 435, in next
data = self._next_data()
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 1085, in _next_data
return self._process_data(data)
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 1111, in _process_data
data.reraise()
File "/usr/local/lib/python3.6/dist-packages/torch/_utils.py", line 428, in reraise
raise self.exc_type(msg)
TypeError: Caught TypeError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/content/CenterHMR/src/dataset/internet.py", line 84, in getitem
return self.get_item_single_frame(index)
File "/content/CenterHMR/src/dataset/internet.py", line 43, in get_item_single_frame
torchvision.transforms.Resize(resized_image_size, interpolation=3),
File "/usr/local/lib/python3.6/dist-packages/torchvision/transforms/transforms.py", line 253, in init
raise TypeError("Size should be int or sequence. Got {}".format(type(size)))
TypeError: Size should be int or sequence. Got <class 'numpy.ndarray'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py", line 198, in _worker_loop
data = fetcher.fetch(index)
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/content/CenterHMR/src/dataset/mixed_dataset.py", line 19, in getitem
return self.dataset[index]
File "/content/CenterHMR/src/dataset/internet.py", line 88, in getitem
return self.get_item_single_frame(index)
File "/content/CenterHMR/src/dataset/internet.py", line 43, in get_item_single_frame
torchvision.transforms.Resize(resized_image_size, interpolation=3),
File "/usr/local/lib/python3.6/dist-packages/torchvision/transforms/transforms.py", line 253, in init
raise TypeError("Size should be int or sequence. Got {}".format(type(size)))
TypeError: Size should be int or sequence. Got <class 'numpy.ndarray'>

I'm trying to animate 3D character in Unity using SMPL 24 joints data. Everything is going well but the only problem is that there is different in Unity and SMPL final data axis.

Below axis are from SMPL data (I draw 3D skeleton using 24 joints from CenterHMR)

Below image is about axis in Unity

Seems that we need to change the Y and Z axis in SMPL results.

So suggest me that we should swap/change axis (Y and Z) in j3d_smpl24 list that I got from CenterHMR from here or in Unity side?

Hello @Arthur151,
Congratulations on the great work and Thanks for making the code available for use.

I tried running the inference script on a custom video via:
CUDA_VISIBLE_DEVICES=0 python core/test.py --gpu=0 --configs_yml=configs/video.yml

But the result is strange..
Is this because of the camera view ??
If yes then, Is there any way to tune the model for such camera views??

Hi! Is it possible yet to render the model mesh like in the inference with the single_image config but with model texture (png)?
I see that you have it working on inference with the webcam config already, that's why I'm asking.

Thanks for this project! :)

First of all thanks @Arthur151. I'm able to generate 3D meshes for my videos however I'm curious to know if we can also run this on CPU. it would be very kind of you if you can guide me on this.

Hey, thanks for letting this open source!

I was trying to merge your project with another one and for that other project I needed a 207-dim Pose Blend Shape as input. Checking your code the close I can get, it's a 72-dim pose parameter... Is there any way to convert those parameters to the 207 vector that I need?

Thanks

你好，我看你论文提到使用了Movi数据集，原始的Movi数据集只提供了3d以及AMASS通过fitting得到的关于SMPL-H的mesh参数。SMPL-H关于shape有16个值，pose有52个，我想知道你在使用Movi的数据集时候，是否使用了AMASS提供的pose和beta。我目前将beta的前10个值和pose前22个值导入到我们当前的SMPL里面，然后借助Movi提供的相机内参和外参以及参考Movi提供的教程，始终无法在我们这个程序中可视化成功。求指教。

如果没有使用，是不是因为AMASS提供的beta和pose不适用我们这个SMPL模型，但是看AMASS论文好像是可以的。
MoVi-Toolbox：MoVi-Toolbox https://github.com/saeed1262/MoVi-Toolbox/blob/master/MoCap/utils.py

�按照Movi的教程以及将其mesh系数按照beta[0:10] pose[0:22] + 两个手部pose设置为0传入到我们当前的SMPL
smpl_model_path = './centerHMR/models/smpl/SMPL_NEUTRAL.pkl'
self.smplx = smpl_model.create(smpl_model_path, batch_size=self.batch_size,model_type=self.model_type, gender='neutral', use_face_contour=False, ext='npz', joint_mapper=joint_mapper,flat_hand_mean=True, use_pca=False).cuda()

@Arthur151

Colab notebook: https://colab.research.google.com/drive/18Y9-cLJftQZMi45YS7vHFf6-cyoQrFV-?usp=sharing

Used Google Colab GPU runtime.
Also had to edit config.py because the config file path string kept getting corrupted.

Error when running run.sh:

/content/CenterHMR/src
: not found
: not found
cat: 'configs/basic_test.yml'$'\r': No such file or directory
Error: invalid path 'ARGS.GPUS', can't query subvalue 'ARGS' of a leaf (leaf value is None).
Namespace(BN_type='BN', HMloss_type='MSE', Rot_type='6D', baseline='hrnetv4', batch_size=64, best_save_path='', center_idx=1, centermap_size=64, configs_yml='configs/basic_test.yml\r', dataset='h36m,mpii,coco,aich,up,ochuman,lsp,movi', dataset_rootdir='/content/dataset/', demo_image_folder='None', epoch=300, eval=False, fine_tune=False, gmodel_path='', gpu='\r', head_conv_block_num=2, high_resolution=True, input_size=512, kp3d_format='smpl24', log_path='/content/log/', lr=0.0003, max_person=16, model_precision='fp32', multi_person=True, nw=4, print_freq=50, save_best_folder='/content/checkpoints/', smpl_model_path='/content/CenterHMR/models', tab='CenterHMR', total_param_count=85, use_coordmaps=True, val_batch_size=64, voc_dir='/content/dataset/VOCdevkit/VOC2012/', weight_decay=1e-05)
Namespace(BN_type='BN', HMloss_type='MSE', Rot_type='6D', baseline='hrnetv4', batch_size=64, best_save_path='', center_idx=1, centermap_size=64, configs_yml='configs/basic_test.yml', dataset='h36m,mpii,coco,aich,up,ochuman,lsp,movi', dataset_rootdir='/content/dataset/', demo_image_folder='None', epoch=300, eval=False, fine_tune=False, gmodel_path='', gpu='\r', head_conv_block_num=2, high_resolution=True, input_size=512, kp3d_format='smpl24', log_path='/content/log/', lr=0.0003, max_person=16, model_precision='fp32', multi_person=True, nw=4, print_freq=50, save_best_folder='/content/checkpoints/', smpl_model_path='/content/CenterHMR/models', tab='CenterHMR', total_param_count=85, use_coordmaps=True, val_batch_size=64, voc_dir='/content/dataset/VOCdevkit/VOC2012/', weight_decay=1e-05)
----------------
Configuration:
{'tab': 'test', 'configs_yml': 'configs/basic_test.yml', 'demo_image_folder': '/content/CenterHMR/demo/images', 'multi_person': True, 'use_coordmaps': True, 'head_conv_block_num': 2, 'kp3d_format': 'smpl24', 'eval': False, 'max_person': 16, 'BN_type': 'BN', 'Rot_type': '6D', 'center_idx': 1, 'centermap_size': 64, 'HMloss_type': 'MSE', 'model_precision': 'fp32', 'baseline': 'hrnetv5', 'input_size': 512, 'gmodel_path': '../trained_models/pw3d_81.8_58.6.pkl', 'best_save_path': '', 'print_freq': 50, 'epoch': 300, 'fine_tune': True, 'lr': 0.0003, 'weight_decay': 1e-05, 'gpu': '\r', 'batch_size': 64, 'val_batch_size': 2, 'nw': 4, 'dataset_rootdir': '/content/dataset/', 'dataset': 'h36m,mpii,coco,aich,up,ochuman,lsp,movi', 'voc_dir': '/content/dataset/VOCdevkit/VOC2012/', 'high_resolution': True, 'save_best_folder': '/content/checkpoints/', 'log_path': '/content/log/', 'total_param_count': 85, 'smpl_model_path': '/content/CenterHMR/models', 'adjust_lr_epoch': [], 'kernel_sizes': [5], 'GPUS': 0, 'save_mesh': False, 'save_centermap': False}
----------------
model_type smpl
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=47 error=100 : no CUDA-capable device is detected
Traceback (most recent call last):
  File "core/test.py", line 50, in <module>
    main()
  File "core/test.py", line 41, in main
    demo = Demo()
  File "core/test.py", line 7, in __init__
    self.set_up_smplx()
  File "/content/CenterHMR/src/core/base.py", line 53, in set_up_smplx
    self.smplx = smpl_mdoel.create(args.smpl_model_path, batch_size=self.batch_size,model_type=self.model_type, gender='neutral', use_face_contour=False, ext='npz', joint_mapper=joint_mapper,flat_hand_mean=True, use_pca=False).cuda()
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 458, in cuda
    return self._apply(lambda t: t.cuda(device))
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 354, in _apply
    module._apply(fn)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 397, in _apply
    self._buffers[key] = fn(buf)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 458, in <lambda>
    return self._apply(lambda t: t.cuda(device))
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 190, in _lazy_init
    torch._C._cuda_init()
RuntimeError: cuda runtime error (100) : no CUDA-capable device is detected at /pytorch/aten/src/THC/THCGeneral.cpp:47

Impressive project, I want to run on Windows, however windows seems donot have the EGL library as required.
However I have encountered this issue:
ImportError: ('Unable to load EGL library', "Could not find module 'EGL' (or one of its dependencies). Try using the full path with constructor syntax.", 'EGL', None)
Does the openGL library necessary and do the code can run on windows ?

After runing the demo code, I can get the .obj and .npz files.
How to put these files in MAYA for further usage.

Thanks

Thank you for your awesome work!
I have one question.
Do you have plans to upload the training code?

Hi there,

Really awesome work. Unfortunately, it can't run with the provided instructions. Trying to sh run.sh results in multiple errors.

First:

----------------
Traceback (most recent call last):
  File "core/test.py", line 2, in <module>
    from base import *
  File "/home/jb/Documents/python/CenterHMR/src/core/base.py", line 20, in <module>
    from dataset.mixed_dataset import SingleDataset
  File "/home/jb/Documents/python/CenterHMR/src/dataset/mixed_dataset.py", line 5, in <module>
    from dataset.internet import Internet
  File "/home/jb/Documents/python/CenterHMR/src/dataset/internet.py", line 9, in <module>
    import smplx
ModuleNotFoundError: No module named 'smplx'

I installed smplx from here: https://github.com/vchoutas/smplx

Next:

----------------
In Ubuntu, using osmesa mode for rendering
Traceback (most recent call last):
  File "/home/jb/anaconda3/envs/centerhmr/lib/python3.7/site-packages/OpenGL/platform/osmesa.py", line 25, in GL
    mode=ctypes.RTLD_GLOBAL 
  File "/home/jb/anaconda3/envs/centerhmr/lib/python3.7/site-packages/OpenGL/platform/ctypesloader.py", line 45, in loadLibrary
    return dllType( name, mode )
  File "/home/jb/anaconda3/envs/centerhmr/lib/python3.7/ctypes/__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: ('OSMesa: cannot open shared object file: No such file or directory', 'OSMesa', None)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "core/test.py", line 2, in <module>
    from base import *
  File "/home/jb/Documents/python/CenterHMR/src/core/base.py", line 21, in <module>
    from visualization.visualization import Visualizer
  File "/home/jb/Documents/python/CenterHMR/src/visualization/visualization.py", line 15, in <module>
    from .renderer import get_renderer
  File "/home/jb/Documents/python/CenterHMR/src/visualization/renderer.py", line 19, in <module>
    import pyrender
  File "/home/jb/anaconda3/envs/centerhmr/lib/python3.7/site-packages/pyrender/__init__.py", line 3, in <module>
    from .light import Light, PointLight, DirectionalLight, SpotLight
  File "/home/jb/anaconda3/envs/centerhmr/lib/python3.7/site-packages/pyrender/light.py", line 10, in <module>
    from OpenGL.GL import *
  File "/home/jb/anaconda3/envs/centerhmr/lib/python3.7/site-packages/OpenGL/GL/__init__.py", line 3, in <module>
    from OpenGL import error as _error
  File "/home/jb/anaconda3/envs/centerhmr/lib/python3.7/site-packages/OpenGL/error.py", line 12, in <module>
    from OpenGL import platform, _configflags
  File "/home/jb/anaconda3/envs/centerhmr/lib/python3.7/site-packages/OpenGL/platform/__init__.py", line 35, in <module>
    _load()
  File "/home/jb/anaconda3/envs/centerhmr/lib/python3.7/site-packages/OpenGL/platform/__init__.py", line 32, in _load
    plugin.install(globals())
  File "/home/jb/anaconda3/envs/centerhmr/lib/python3.7/site-packages/OpenGL/platform/baseplatform.py", line 92, in install
    namespace[ name ] = getattr(self,name,None)
  File "/home/jb/anaconda3/envs/centerhmr/lib/python3.7/site-packages/OpenGL/platform/baseplatform.py", line 14, in __get__
    value = self.fget( obj )
  File "/home/jb/anaconda3/envs/centerhmr/lib/python3.7/site-packages/OpenGL/platform/osmesa.py", line 66, in GetCurrentContext
    function = self.OSMesa.OSMesaGetCurrentContext
  File "/home/jb/anaconda3/envs/centerhmr/lib/python3.7/site-packages/OpenGL/platform/baseplatform.py", line 14, in __get__
    value = self.fget( obj )
  File "/home/jb/anaconda3/envs/centerhmr/lib/python3.7/site-packages/OpenGL/platform/osmesa.py", line 60, in OSMesa
    def OSMesa( self ): return self.GL
  File "/home/jb/anaconda3/envs/centerhmr/lib/python3.7/site-packages/OpenGL/platform/baseplatform.py", line 14, in __get__
    value = self.fget( obj )
  File "/home/jb/anaconda3/envs/centerhmr/lib/python3.7/site-packages/OpenGL/platform/osmesa.py", line 28, in GL
    raise ImportError("Unable to load OpenGL library", *err.args)
ImportError: ('Unable to load OpenGL library', 'OSMesa: cannot open shared object file: No such file or directory', 'OSMesa', None)

I had to install Mesa using the instructions from PyRender: https://pyrender.readthedocs.io/en/latest/install/

Next:

Traceback (most recent call last):
  File "core/test.py", line 50, in <module>
    main()
  File "core/test.py", line 46, in main
    demo.run(demo_image_folder)
  File "core/test.py", line 20, in run
    self.visualizer = Visualizer(model_type=self.model_type,resolution =vis_size, input_size=self.input_size, result_img_dir = test_save_dir,with_renderer=True)
  File "/home/jb/Documents/python/CenterHMR/src/visualization/visualization.py", line 23, in __init__
    self.renderer = get_renderer(model_type=model_type,resolution=self.resolution)
  File "/home/jb/Documents/python/CenterHMR/src/visualization/renderer.py", line 138, in get_renderer
    renderer = Renderer(faces,resolution=resolution[:2])
  File "/home/jb/Documents/python/CenterHMR/src/visualization/renderer.py", line 69, in __init__
    point_size=1.0)
  File "/home/jb/anaconda3/envs/centerhmr/lib/python3.7/site-packages/pyrender/offscreen.py", line 31, in __init__
    self._create()
  File "/home/jb/anaconda3/envs/centerhmr/lib/python3.7/site-packages/pyrender/offscreen.py", line 149, in _create
    self._platform.init_context()
  File "/home/jb/anaconda3/envs/centerhmr/lib/python3.7/site-packages/pyrender/platforms/osmesa.py", line 19, in init_context
    from OpenGL.osmesa import (
ImportError: cannot import name 'OSMesaCreateContextAttribs' from 'OpenGL.osmesa' (/home/jb/anaconda3/envs/centerhmr/lib/python3.7/site-packages/OpenGL/osmesa/__init__.py)

This one stumped me, couldn't get it to work. Maybe my Mesa installation is still not correct?

I would like to visualize the body landmarks projected onto the 2d image however I have some problems understanding how the values are stored. When I run the demo script I get a file called .npz and I can load it with numpy. Inside it I found f->results where two dictionaries are stored one for each person in the image. Inside each dictionary I can see 'cam' (3,), 'pose' (72,), 'betas' (10,), 'j3d_smpl24' (24, 3), 'j3d_op25' (25, 3), and 'verts' (6890, 3) variables (I put their corresponding shapes in parentheses). From your paper I thought 'pose' variable would be of length 132 (22 landmarks x 6D). Anyway, I assume you have saved 24 landmarks in 3D which makes an array of length 72. From the 'cam' variable (tx, ty = cam[1:]) I calculated the center for each people trivially since those numbers are normalized between -1 and 1 just as you mentioned in your paper. However when it comes to visualizing 'pose', 'j3d_smpl24' or 'verts' variables I ran into some problems. Could you explain how each of these variables store their data? Apparently they are not normalized between -1 and 1. Also I'm having some problems understanding the first number in 'cam' variable which should correspond to scale according to the paper. In the paper it is said that this variable "reflects the size and depth of the human body to some extent.". How can this scale be used to visualize pose points? In my example I get 5.51 for one person and -5.156 for the other. Would you also explain what does a negative scale represent?

Hi,
I need not the real depth but the order of the different depths. in your paper you mention ..." For people of different scales, the people with greater scale are in the front. For people of similar scale, the people with greater center confidence are in the front"...
My doubt is how you classify "different scales" & "similar scales".
(Maybe you have already coded that issue when selecting the rendering order of the different masks/meshs; if so could you point where that code is?)
Many Thanks in advance!

Hi, kudos for the great work!
When are you planning to release the training code?
Thanks

Hi,

Can I know if the currently released pretrained model has been trained on 3dpw or not? Thanks!

Thank you for sharing your amazing work. I have tried to visualize the projected 2d keypoints (i.e. pj2d) on the original image.
Taking the 3dpw_sit_on_street.jpg sample image as an example. Since pj2d are normalized (-1~1) coordinates on the input image size (512x 512), I first compute their corresponding image coordinate in the original image size (1920x1080) by:
pj2d= (pj2d+ [1,1]) / 2 * [1920, 1080]

Then I plotted those keypoints on the original image. However, it seems there is an offset on the y-axis (i.e. height). All the keypoints are lower than the positions where they should be.

Any thoughts on how to compute and visualize projected 2D keypoints correctly would be really appreciated.

I haven't read the source code yet.
I ran the demo, and found that the head pose estimation is not supported, which confirmed in the results of the experimental section of paper.
Would you like to know whether the support for head pose estimation will be increased in the future?
Thanks.

I'm trying to draw body joints in 3D space using 3D SMPL joints and 3D openpose joints using CenterHMR. I followed SMPL joints order from here and openpose order from here

I'm getting wrong 3D skeleton, could you share the right order of joints that CenterHMR outputs? Joints order is same with original SMPL or you changed joints order in output?

Poor results on some images, one of the examples is attached below:

How can we improve it

The model runs perfectly but I noticed there is a lot of jittering in the output. Is there any way to remove the jittering?

HI!

Awesome work! When do you plan to release the code?

hey @Arthur151, great framework.
but I have one issue, i tried executing it in the google colab environment on a video of 446 frames, took 153 seconds.
The fps is < 3fps, also tried on AWS got almost same around 3.5 fps there.
Is there something i am missing or that's all we can get ?

Hello. Thank you for your work.
I wonder that what's the unit of 3d pose? I found it completely different from the annotations in 2d which are the pixel value. Instead, they are the value at around [-5,5] like the screenshot.
What's the actual meaning of these values and how to measure them in the real world?

Hey bro. I am coming again for the another problem.
I want to run the new feature which can export the mesh to .fbx file, and it seems that "bpy" package is needed to download. But when I use pip install bpy, some CMake error always occurs.
I wonder that which version of CMake you are using when you install bpy?